Follow-up: Using AWS Simple Email Service (SES) for Inbound Mail

As I alluded to in my first post on this subject, it's possible to use AWS Lambda to process mail you receive as well.   In this final post on AWS Simple Email Service, I'll show you how to forward your email using a Lambda function.


I'm not going to go into any of the details of how you actually set up your Lambda functions in the AWS console, I'll just include the snippet of code I was able to get working and give you a few pointers.

console.log("Loading event...");

exports.handler = function(event, context) {
   
    var AWS = require('aws-sdk');
    
    var Sns = event.Records[0].Sns;
    var Message = Sns.Message;
    
    Message = JSON.parse(Message)

    var bucketName = Message.receipt.action.bucketName;
    var objectKey = Message.receipt.action.objectKey;
    
    var ses = new AWS.SES({apiVersion: '2010-12-01'});

    var s3 = new AWS.S3();
    
    var params = { Bucket: bucketName, Key: objectKey };

    s3.getObject(params,function(err, data) {
        if (err) {
            console.log(err, err.stack);
            }
        else {
        var params = {
                RawMessage: {
                    Data: data.Body
                },
                Destinations: ['rlauer6@comcast.net']
            };
            
            ses.sendRawEmail(params, function(err, data) {
                if (err) console.log(err, err.stack); // an error occurred
                else     console.log(data);           // successful response
            });    
        }
    });
};

It's actually pretty compelling when you look at how easy it seems to be to create a serverless solution to processing incoming email.  As a forwarding agent, there are a few caveats to consider though.  From the AWS SDK docs regarding the SendRawEmail() method:

Sends an email message, with header and content specified by the client. The SendRawEmail action is useful for sending multipart MIME emails. The raw text of the message must comply with Internet email standards; otherwise, the message cannot be sent.

There are several important points to know about SendRawEmail:

• You can only send email from verified email addresses and domains; otherwise, you will get an "Email address not verified" error. If your account is still in the Amazon SES sandbox, you must also verify every recipient email address except for the recipients provided by the Amazon SES mailbox simulator. For more information, go to the Amazon SES Developer Guide.
• The total size of the message cannot exceed 10 MB. This includes any attachments that are part of the message.
• Amazon SES has a limit on the total number of recipients per message. The combined number of To:, CC: and BCC: email addresses cannot exceed 50. If you need to send an email message to a larger audience, you can divide your recipient list into groups of 50 or fewer, and then call Amazon SES repeatedly to send the message to each group.
• The To:, CC:, and BCC: headers in the raw message can contain a group list. Note that each recipient in a group list counts towards the 50-recipient limit.
• Amazon SES overrides any Message-ID and Date headers you provide.
• For every message that you send, the total number of recipients (To:, CC: and BCC:) is counted against your sending quota - the maximum number of emails you can send in a 24-hour period. For information about your sending quota, go to the Amazon SES Developer Guide.

That doesn't mean it still isn't compelling as a solution to process your incoming email if what you want is a bit of organization and perhaps automation around incoming email. The gotchas of developing more complex solutions with Lambda are many right now.

For one thing, it's a rather steep learning curve if you're coming from more traditional programming environments. And while there are many resources available on the internet to help get you started, be prepared to be dazed and confused with the volume of documentation. Some of it is very good, some is confusing and most of it will seem disorganized to you and lack context. At least that was my experience, especially when you start to run into problems. Essentially I think I just described the internet itself. ;-)

Of course, as we all know, all of the issues I encountered led me to a better (albeit still incomplete) understanding of Lambdas. Embrace the problems and prepare to be enlightened.

You'll also need to consider things like available resources (memory), the time your task takes to run (Lambda's are best for short tasks and event handlers), ancillary libraries and helpers you might need to accomplish the task, not to mention the versions of those libraries that are available to you as part of the Lambda environment. While you can add your own libraries as part of your deployment package that process can be a bit cumbersome and potentially a time sync to someone accustomed to a more polished development environment.

You'll find that creating a Lambda should include understanding all of the places your code might fail and handling those cases, even something as trivial as a syntax error in your javascript could cause you a lot of headaches. If you search the interweb you'll see a lot of people who have identified many of these issues as well and a lot of people who have trudged through them too in order to advance their use of serverless technologies. As always YMMV depending on your experience, patience and time you have to build your application.

Resources

Javascript SDK - http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/index.html
Lambdas & SNS - http://docs.aws.amazon.com/sns/latest/dg/sns-lambda.html

Tips

First, you should know that you're going to have to setup a role for the Lambda function and then give that role certain privileges if you want it to be able to read your S3 bucket and use SES.  To get this working, I attached canned policies to the role for reading my S3 bucket and for accessing SES. You'll want to be more restrictive and make sure your Lambda can only access the resources it needs.

You're also going to have to make sure your SES configuration can actually send mail.

I ran into some issues with Lambda timing out.  As it turns out, the default timeout is 3 seconds and for some reason this little snippet was timing out after 3 seconds.  Timing out, means full stop, exit and abort.  This prevented me from seeing the fact that I was getting a permissions error accessing my S3 bucket.  Why this function takes 5300ms to complete is a mystery.  I suspect startup time, plus connecting to the SES service and sending mail through that service may be the culprit despite my very short email tests.

In my experience, debugging Lambdas is a nightmare.  console.log() is helpful, but when your program is just exiting without any clue as to why, you're sort of left to psychic debugging and endless frobbing.

Update: Upon further reflection of my experience debugging this small snippet, I have come to the conclusion that the timeouts I was experiencing compounded my frustration.  Despite inserting copious log messages everywhere, I found at times the logs provided no clues.  After a good sleep, a walk and replaying the entire sordid debugging session in my head, I'm theorizing (although I have no direct evidence of this fact) that there must be some buffering of the log messages going on such that, upon an abrupt and rude halting of the Lambda, those messages get lost. That could explain some of the confusing results I seemed to have encountered when logs seemed to be missing messages I thought should appear.

I think this has some promise though so I'll be looking into using more Lambdas to create a serverless application environment in future applications.