AWS Lambda & Perl/Vendoring Libraries - A Case Study w/Image::Magick (Part II)

In my last blog, I showed how to create a zipfile that contains a working Image Magick Perl environment you can use when creating Perl Lambdas.

This is part two of a two part blog that discusses how to package libraries for your Perl Lambdas.
If you recall, in my last blog we created a Docker image and installed the Perl module Image::Magick  by downloading and compiling the latest Image Magick source code.  The trick it seemed was to make sure that the environment had no conflicting Image Magick include files.  We also used the --with-perl flag to install build PerlMagick rather than using cpanm to download install it from CPAN.

In order to make Image::Magick  work in the Lambda runtime environment, you'll need one more ingredient - the shared libraries specifically used by the Image Magick Perl module and possibly any shared libraries used by Image Magick shared libraries.

When Image Magick was built in our Docker image, the shared libraries were installed to /usr/local/lib.  To use Image Magick with Lambda you'll need these shared libraries installed, in the Lambda runtime environment but you won't be able to install them to /usr/local/lib.  Using Lambda Layers however you can install libraries to /opt/lib. In the Lambda execution environment LD_LIBRARY_PATH (one of the places where the dynaloader ld will look for shared libraries) points to /opt/lib.

From the Docker recipe in my previous blog, one of of the last steps is to copy the files created during the Image Magick build to /opt/lib.  I've copied all of the files from the build into /opt/lib however I believe that the Perl module only uses the libMagickCore-6.Q16.so.6.0.0.

RUN mkdir /opt/lib; \
    cp /usr/local/lib/libMagickCore-6.Q16.so.6.0.0 /opt/lib; \
    ln -s /opt/lib/libMagickCore-6.Q16.so.6.0.0 /opt/lib/libMagickCore-6.Q16.so; \
    ln -s /opt/lib/libMagickCore-6.Q16.so.6.0.0 /opt/lib/libMagickCore-6.Q16.so.6; \
    cp /usr/local/lib/libMagickWand-6.Q16.so.6.0.0 /opt/lib; \  
    ln -s /opt/lib/libMagickWand-6.Q16.so.6.0.0 /opt/lib/libMagickWand-6.Q16.so.6; \
    ln -s /opt/lib/libMagickWand-6.Q16.so.6.0.0 /opt/lib/libMagickWand-6.Q16.so; \
    cp /usr/local/lib/libMagick++-6.Q16.so.8.0.0 /opt/lib; \ 
    ln -s /opt/lib/libMagick++-6.Q16.so.8.0.0 /opt/lib/libMagick++-6.Q16.so.8; \
    ln -s /opt/lib/libMagick++-6.Q16.so.8.0.0 /opt/lib/libMagick++-6.Q16.so;

To see what shared libraries we need we use ldd. First, let's find the shared object that was built for the Perl module.

bash-4.2# locate Magick | grep perl

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/Image/Magick

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/Image/Magick.pm

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/Image/Magick/Q16.pm

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/auto/Image/Magick

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/auto/Image/Magick/.packlist

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/auto/Image/Magick/Q16

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/auto/Image/Magick/Q16/Q16.so

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/auto/Image/Magick/Q16/autosplit.ix

/usr/local/share/doc/ImageMagick-6/www/perl-magick.html

Next, let's see what shared libraries it requires.

bash-4.2# ldd /opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/auto/Image/Magick/Q16/Q16.so

        linux-vdso.so.1 =>  (0x00007fffd1db8000)

        libMagickCore-6.Q16.so.6 => /tmp/ImageMagick-6.9.10-28/PerlMagick/quantum/../../magick/.libs/libMagickCore-6.Q16.so.6 (0x00007ff1d351b000)

        libm.so.6 => /lib64/libm.so.6 (0x00007ff1d3219000)

        libc.so.6 => /lib64/libc.so.6 (0x00007ff1d2e4c000)

        liblcms2.so.2 => /usr/lib64/liblcms2.so.2 (0x00007ff1d2bf4000)

        ...

This produces a rather lengthy list of shared librararies with libMagickCore-Q16.so.6 near the top.  In general, when building libraries to include in your Lambda Layers, you'll need to identify the shared libraries that do not exist in the standard Lambda runtime and copy those to the /opt/lib directory.  In this case, only the one shared library was not found on the standard runtime.  I've actually copied more of the Image Magick libraries that are built than is necessary for Image::Magick, to work in my Docker recipe, however there may be other utilities I may want to take advantage of in the future that require those, therefore it made sense to copy of all of them.

For example, if I want to use the version of the Image Magick convert utility that was built from source, I would copy that from /usr/local/bin.  Taking a look at that binary, we find it also requires libMagickCore-6.Q16.so.6 along with libMagickWand-6.Q16.so.6.

bash-4.2# /usr/local/bin/convert --version

Version: ImageMagick 6.9.10-28 Q16 x86_64 2019-02-24 https://imagemagick.org

Copyright: © 1999-2019 ImageMagick Studio LLC

License: https://imagemagick.org/script/license.php

Features: Cipher DPC OpenMP

Delegates (built-in): bzlib fontconfig freetype gslib jng jpeg lcms lzma png ps tiff x xml zlib

bash-4.2# ldd /usr/local/bin/convert | grep -i magick

        libMagickCore-6.Q16.so.6 => /usr/local/lib/libMagickCore-6.Q16.so.6 (0x00007f2075897000)

        libMagickWand-6.Q16.so.6 => /usr/local/lib/libMagickWand-6.Q16.so.6 (0x00007f207556d000)

Now with a Lambda Layer that contains Perl 5.28.1, Image::Magick and its shared libraries we're ready to proceed to the next task in creating our Lambda that interprets QR codes.  Next time - Building and Installing the zbar Library.  

AWS Lambda & Perl/Vendoring Libraries - A Case Study w/Image::Magick (Part I)

Now here's where things get interesting.

This is a two part blog that discusses how to package libraries for your Perl Lambdas.

So you'd like to use Image::Magick in your Perl Lambda? And you thought you would just use Image::Magick? Or perhaps you were smarter than that because you are using a Lambda Layer and have Perl 5.28.1 installed, so you thought might just install it via your Perl 5.28.1 Docker image that mimics the Lambda runtime environment using cpanm? Well, not so fast...

The first challenge is installing Image::Magick, the next challenge is actually running it in your Lambda.  Spoiler alert...both of these challenges are possible to overcome! In part 1 we'll install Image::Magick.

AWS Lambda & Perl - Using Docker to Build Lambda Layers

As the Perl Lambda adventure continues, I'm learning more about the Perl toolchain, Lambdas and Docker.  After running into an issue with OpenSSL versioning on the supposed AMI that AWS documents as being the official Lambda execution environment it appears that it is not.

What does appear to be true is that the image is similar to the runtime environment. For a truer representation of the Lambda runtime there is a Docker image lambci/lambda that has proven to be a closer representation of the Lambda runtime environment.  In a previous post I noted that the version of OpenSSL on the AMI is 1.0.2 while the Lambda runtime environment version is 1.0.1.  The Docker image has OpenSSL version 1.0.1 installed, so empirically the Docker image does appear to be similar to the Lambda execution environment.  Why is this important?

AWS Lambda & Perl - The Journey Begins

Perl has been the odd man out as more modern programming languages take hold and gain the support of the software development community.  There is no doubt that Perl 5 has been left in the dust with regard to support from vendors of all kinds.  It is rare to see an API written for Perl for any of the cloud services.  It seems to be left up to the Perl community itself to provide support for many of these APIs.

A few weeks ago I released version 0.0.1 of a framework for creating AWS Lambdas in Perl.   It is a challenge to continue to develop in Perl and becomes more and more challenging as other languages continue to gain support, however I don't think the Perl community is giving up.  While I'm quite willing to learn Python and embrace new methods, I also see the need for supporting legacy environments with a large investment in Perl code.  I also enjoy a challenge. :-)

AWS Lambda Gotchas - libcrypt.so.10 Hell and Back

This is part of continuing series of blog posts regarding AWS Lambda and Perl.

As I continue my deep dive into making Perl work effectively as a Lambda runtime, I hit the libcrypto wall that so many other people on the wire have encountered...even Pythonistas!

CodeBuild 2019 or "How I built perl for less than 3 cents"

This is my second (or third) dalliance with AWS CodeBuild.  I think I'm starting to like it now though.  I recently wrote a bash script to compile a version of perl to create Lambdas as part of my Perl Lambda project.  My so-called `make-a-perl` script will instantiate an EC2, download the perl source code, compile it, zip it up and copy it to an S3 bucket.

I suspected I could use AWS CodeBuild to do all of the aforementioned operations, but did not want to descend down the CodeBuild rabbit hole once more.  But alas, the temptation was just too much!

RPM Virtual Packages and Ordering

I recently decided to use an RPM virtual package (one that offers no files but merely Requires others) in order to lock down the version of packages that were being installed when an EC2 was provisioned.  Much to my chagrin it failed miserably when yum loaded packages in an unexpected order.  So, can you tell yum what order to load your RPMs?