AWS Lambda & Perl/Vendoring Libraries - A Case Study w/Image::Magick (Part II)

In my last blog, I showed how to create a zipfile that contains a working Image Magick Perl environment you can use when creating Perl Lambdas.

This is part two of a two part blog that discusses how to package libraries for your Perl Lambdas.
If you recall, in my last blog we created a Docker image and installed the Perl module Image::Magick  by downloading and compiling the latest Image Magick source code.  The trick it seemed was to make sure that the environment had no conflicting Image Magick include files.  We also used the --with-perl flag to install build PerlMagick rather than using cpanm to download install it from CPAN.

In order to make Image::Magick  work in the Lambda runtime environment, you'll need one more ingredient - the shared libraries specifically used by the Image Magick Perl module and possibly any shared libraries used by Image Magick shared libraries.

When Image Magick was built in our Docker image, the shared libraries were installed to /usr/local/lib.  To use Image Magick with Lambda you'll need these shared libraries installed, in the Lambda runtime environment but you won't be able to install them to /usr/local/lib.  Using Lambda Layers however you can install libraries to /opt/lib. In the Lambda execution environment LD_LIBRARY_PATH (one of the places where the dynaloader ld will look for shared libraries) points to /opt/lib.

From the Docker recipe in my previous blog, one of of the last steps is to copy the files created during the Image Magick build to /opt/lib.  I've copied all of the files from the build into /opt/lib however I believe that the Perl module only uses the libMagickCore-6.Q16.so.6.0.0.

RUN mkdir /opt/lib; \
    cp /usr/local/lib/libMagickCore-6.Q16.so.6.0.0 /opt/lib; \
    ln -s /opt/lib/libMagickCore-6.Q16.so.6.0.0 /opt/lib/libMagickCore-6.Q16.so; \
    ln -s /opt/lib/libMagickCore-6.Q16.so.6.0.0 /opt/lib/libMagickCore-6.Q16.so.6; \
    cp /usr/local/lib/libMagickWand-6.Q16.so.6.0.0 /opt/lib; \  
    ln -s /opt/lib/libMagickWand-6.Q16.so.6.0.0 /opt/lib/libMagickWand-6.Q16.so.6; \
    ln -s /opt/lib/libMagickWand-6.Q16.so.6.0.0 /opt/lib/libMagickWand-6.Q16.so; \
    cp /usr/local/lib/libMagick++-6.Q16.so.8.0.0 /opt/lib; \ 
    ln -s /opt/lib/libMagick++-6.Q16.so.8.0.0 /opt/lib/libMagick++-6.Q16.so.8; \
    ln -s /opt/lib/libMagick++-6.Q16.so.8.0.0 /opt/lib/libMagick++-6.Q16.so;

To see what shared libraries we need we use ldd. First, let's find the shared object that was built for the Perl module.

bash-4.2# locate Magick | grep perl

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/Image/Magick

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/Image/Magick.pm

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/Image/Magick/Q16.pm

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/auto/Image/Magick

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/auto/Image/Magick/.packlist

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/auto/Image/Magick/Q16

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/auto/Image/Magick/Q16/Q16.so

/opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/auto/Image/Magick/Q16/autosplit.ix

/usr/local/share/doc/ImageMagick-6/www/perl-magick.html

Next, let's see what shared libraries it requires.

bash-4.2# ldd /opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux/auto/Image/Magick/Q16/Q16.so

        linux-vdso.so.1 =>  (0x00007fffd1db8000)

        libMagickCore-6.Q16.so.6 => /tmp/ImageMagick-6.9.10-28/PerlMagick/quantum/../../magick/.libs/libMagickCore-6.Q16.so.6 (0x00007ff1d351b000)

        libm.so.6 => /lib64/libm.so.6 (0x00007ff1d3219000)

        libc.so.6 => /lib64/libc.so.6 (0x00007ff1d2e4c000)

        liblcms2.so.2 => /usr/lib64/liblcms2.so.2 (0x00007ff1d2bf4000)

        ...

This produces a rather lengthy list of shared librararies with libMagickCore-Q16.so.6 near the top.  In general, when building libraries to include in your Lambda Layers, you'll need to identify the shared libraries that do not exist in the standard Lambda runtime and copy those to the /opt/lib directory.  In this case, only the one shared library was not found on the standard runtime.  I've actually copied more of the Image Magick libraries that are built than is necessary for Image::Magick, to work in my Docker recipe, however there may be other utilities I may want to take advantage of in the future that require those, therefore it made sense to copy of all of them.

For example, if I want to use the version of the Image Magick convert utility that was built from source, I would copy that from /usr/local/bin.  Taking a look at that binary, we find it also requires libMagickCore-6.Q16.so.6 along with libMagickWand-6.Q16.so.6.

bash-4.2# /usr/local/bin/convert --version

Version: ImageMagick 6.9.10-28 Q16 x86_64 2019-02-24 https://imagemagick.org

Copyright: © 1999-2019 ImageMagick Studio LLC

License: https://imagemagick.org/script/license.php

Features: Cipher DPC OpenMP

Delegates (built-in): bzlib fontconfig freetype gslib jng jpeg lcms lzma png ps tiff x xml zlib

bash-4.2# ldd /usr/local/bin/convert | grep -i magick

        libMagickCore-6.Q16.so.6 => /usr/local/lib/libMagickCore-6.Q16.so.6 (0x00007f2075897000)

        libMagickWand-6.Q16.so.6 => /usr/local/lib/libMagickWand-6.Q16.so.6 (0x00007f207556d000)

Now with a Lambda Layer that contains Perl 5.28.1, Image::Magick and its shared libraries we're ready to proceed to the next task in creating our Lambda that interprets QR codes.  Next time - Building and Installing the zbar Library.