AWS Lambda Gotchas - libcrypt.so.10 Hell and Back

This is part of continuing series of blog posts regarding AWS Lambda and Perl.

As I continue my deep dive into making Perl work effectively as a Lambda runtime, I hit the libcrypto wall that so many other people on the wire have encountered...even Pythonistas!

I'll start with the answer (inspired by this forum post) -  build your Lambda packages against version 1.0.1 of OpenSSL if you are working in the custom runtime environment (i.e. you have selected "provided" as the compatible runtime).  Lock your repo to a specific release and downgrade rpm packages.  Then re-build your libraries against version openssl 1.0.1.

sed -i 's;^releasever.*;releasever=2017.03;;' /etc/yum.conf
yum clean all
yum -y downgrade curl libcurl
yum -y downgrade openssl-1.0.1k
yum -y downgrade openssl-devel openssl

In my case, I wanted to use LWP with HTTPS.  This requires LWP::Protocol:https which in turn requires Net::SSLeay.  Unfortunately, building this even on the recommended Amazon Linux AMI created a dependency on openssl 1.0.2.  It appears that although the AWS documentation tells us that the AMI used for the Lambda runtime is...

• Operating system – Amazon Linux
• AMI – amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2
• Linux kernel – 4.14.77-70.59.amzn1.x86_64
• AMI id:   ami-4fffc834

...something is amiss because the Lambda runtime uses openssl version 1.0.1 and the AMI above uses openssl 1.0.2.  When you attempt to use Net::SSLeay the dynaloader fails to find 1.0.2 in /lib64 and fails to load.

After downgrading openssl, rebuilding Net::SSLeay and reinstalling my Lambda, I was able to get my Lambda working.

Update: I have confirmed with AWS support that it appears they are NOT using the documented environment for custom runtimes. 

It's pretty clear the AMI documented is not EXACTLY the Lambda runtime.  I have found that using the Docker image lambci/lambda:build is a better approach to building runtimes.  I will blog on this as I gain more clarity on exactly WTF is going on.

I will continue to update this post when they confirm the actual environment being used.

Update: See my latest blog post in which I discuss abandoning the use of an EC2 in favor of a Docker image (lambci/lambda) which is closer in flavor to the Lambda execution environment and does have version 1.0.1 of OpenSSL installed.