 |
| Delete, delete, delete, delete, forward... |
In part one of my two part blog on Amazon's Simple Email Service, we set up the necessary resources to receive and process inbound email. In part two, we'll create a worker that reads an SQS queue and forwards the mail to another email address.
To recap from our previous blog we set up and configured several resources in order to receive email using Amazon's Simple Email Service. Recall that we:
 |
| A simple email processor for Simple Email Service |
When email is sent to any address @openbedrock.net the mail will be stored in our S3 bucket and an SNS notification will be sent to our SQS queue. So, in order to forward mail for someone to another email address for example, we'll need to perform these steps:
- Read the SQS queue and see if we have any messages
- Interpret the message contents
- Download the mail
- Forward the mail
- Delete the message from the queue
- Delete the message from the S3 bucket
Here's an example Perl script that accomplishes all of those steps.
#!/usr/bin/perl
use strict;
use warnings;
use Amazon::S3;
use Amazon::SQS::Client;
use Amazon::SQS::Model::DeleteMessageRequest;
use Amazon::SQS::Model::ReceiveMessageRequest;
use Amazon::SQS::Model::ListQueuesRequest;
use File::Temp qw/tempfile/;
use Email::Filter;
use JSON;
my $sqs = Amazon::SQS::Client->new($ENV{AWS_ACCESS_KEY_ID}, $ENV{AWS_SECRET_ACCESS_KEY});
my $queue_url = 'https://sqs.us-east-1.amazonaws.com/' . $ENV{AWS_ACCOUNT} . '/' . $ENV{AWS_SQS_QUEUE};
my $request = Amazon::SQS::Model::ReceiveMessageRequest->new({ QueueUrl => $queue_url, MaxNumberOfMessages => 1});
# 1. Look for a message
my $response = $sqs->receiveMessage($request);
my $messageList;
if ( $response->isSetReceiveMessageResult() ) {
my $receiveMessageResult = $response->getReceiveMessageResult();
$messageList = $receiveMessageResult->getMessage();
}
if (@$messageList) {
foreach my $msg (@$messageList) {
my $body = from_json($msg->getBody);
my $message = from_json($body->{Message});
# 2. Interpret the message contents
print STDERR to_json($message, { pretty => 1} );
my ($bucketName, $objectKey) = @{$message->{receipt}->{action}}{qw/bucketName objectKey/};
print STDERR sprintf("S3: %s %s\n", $bucketName, $objectKey);
# 3. Download the mail from S3
my $s3 = Amazon::S3->new(
{
aws_access_key_id => $ENV{AWS_ACCESS_KEY_ID},
aws_secret_access_key => $ENV{AWS_SECRET_ACCESS_KEY},
retry => 1
}
);
my $bucket = $s3->bucket($bucketName);
my (undef, $filename) = tempfile();
$bucket->get_key_filename($objectKey, 'GET', $filename);
# 4. Forward the email
open STDIN, '<' . $filename;
my $mail = Email::Filter->new();
$mail->exit(0);
$mail->pipe('sendmail', $ENV{FORWARD_MAIL_TO});
# remove temp file
unlink $filename;
# 5. Delete the message from the queue
my $receiptHandle = $msg->getReceiptHandle();
my $deleteMessageRequest = Amazon::SQS::Model::DeleteMessageRequest->new(
{
QueueUrl => $queue_url,
ReceiptHandle => $receiptHandle
}
);
$sqs->deleteMessage($deleteMessageRequest);
#6. Delete the message from the S3 bucket
$bucket->delete_key($objectKey);
}
}
Perl Modules
There are several useful Perl modules for using Amazon web services. I'm using two of the rather light-weight modules for reading and deleting messages from SQS queues and for interacting with S3. There are several other good modules available on CPAN that can be used, but I prefer to keep the dependencies simple. I'm also using a Perl module for forwarding the mail (Email::Filter) although you could pipe the email file we download from S3 directly to sendmail. Let's go through the script...
Perl Script
 |
| Just the meat & potatoes please... |
I've numbered the comments in the script to correspond with the steps in the algorithm we outlined in the preamble of this blog so it should be fairly straight forward to follow.
The SNS message that is placed in the queue is a JSON string that contains information about the notification. The actual relevant information we need is embedded in the 'Message' element and itself is a JSON string that we need to decode in order to get at the email message particulars.
my $body = from_json($msg->getBody);
my $message = from_json($body->{Message});
# 2. Interpret the message contents
print STDERR to_json($message, { pretty => 1} );
Here's what the SNS 'Message' key looks like after we've deserialized the SQS message, dug out the SNS message and prettified it:
{
"notificationType" : "Received",
"mail" : {
"headersTruncated" : false,
"messageId" : "2tlb8ad9l26vr2d736e7hsitvgenhf0tu261s801",
"source" : "rlauer@openbedrock.net",
"headers" : [
{
"value" : "<rlauer@openbedrock.net>",
"name" : "Return-Path"
},
{
"value" : "from openbedrock.net (ec2-NN-NN-NNN-NNN.compute-1.amazonaws.com [NN.NN.NNN.NNN]) by inbound-smtp.us-east-1.amazonaws.com with SMTP id 2tlb8ad9l26vr2d736e7hsitvgenhf0tu261s801 for rlauer@openbedrock.net; Thu, 29 Dec 2016 16:30:19 +0000 (UTC)",
"name" : "Received"
},
{
"value" : "PASS",
"name" : "X-SES-Spam-Verdict"
},
{
"value" : "PASS",
"name" : "X-SES-Virus-Verdict"
},
{
"value" : "none (spfCheck: NN.NN.NNN.NNN is neither permitted nor denied by domain of openbedrock.net) client-ip=NN.NN.NNN.NNN; envelope-from=rlauer@openbedrock.net; helo=ec2-NN-NN-NNN-NNN.compute-1.amazonaws.com;",
"name" : "Received-SPF"
},
{
"value" : "amazonses.com; spf=none (spfCheck: NN.NN.NNN.NNN is neither permitted nor denied by domain of openbedrock.net) client-ip=NN.NN.NNN.NNN; envelope-from=rlauer@openbedrock.net; helo=ec2-NN-NN-NNN-NNN.compute-1.amazonaws.com;",
"name" : "Authentication-Results"
},
{
"value" : "AEFBQUFBQUFBQUFHMlRlVExuTEh6Y09ySHRsdnRzVjFuK0llL2FLbkhnRUdTVlQxT1NycGtPQ0JBVUpCckI3REs4US9CK282K0N6amxTMHhCdlZxWFpwNmRwenFia3krSDY3QzFac1ZKZWkzeHBOQjc1VEJ0OU9lTkdVcGpHNkd3NytQeXV1ZFpzcG5JVzhxTHZyUUJSR0ZKRStsRXpmWjFnK3d6NloxbGQ1U2RuUW5uNFdUNHdLT3Fqb0J1RDY1RWtCMzhpTzJKcXNOOVVYcmd0OUFDK0w3ekpaSCtpaDBLejZkZ0JXeldsMkM0ejZYV0NQTllIS0tUMm9qc0x2b3RhWG9PUitmUWVvNW5nMmtlVlo3aExtM1AyNGR0RFoveC91amlaZkozVUEzWlRFYTNjWGF3U1E9PQ==",
"name" : "X-SES-RECEIPT"
},
{
"value" : "v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=6gbrjpgwjskckoa6a5zn6fwqkn67xbtw; d=amazonses.com; t=1483028894; h=X-SES-RECEIPT:Date:To:Subject:MIME-Version:Content-Type:Message-Id:From; bh=zmlq5rQulATvu8Ut0Tt6HTQL1eElci/3DyAMQjE5CQU=; b=dMjxAZOSI5b0xkPHipUHu6UjisZQHY6kAoatyf7Y8XEhTr5LuVUF0pvW9j0pUfdk p1I/MBwENYj2g+Ra0bF7u7eG1QSuAF6xJ+T+aWOVhqG12oJF/OF9PRxrqY4sX3vauVc oEJ+U+pUcunZIHEWknzcGaC5yZXs5XXbtCM3O47k=",
"name" : "X-SES-DKIM-SIGNATURE"
},
{
"value" : "by openbedrock.net (Postfix, from userid 501) id 0F3CEDB93; Thu, 29 Dec 2016 16:28:14 +0000 (UTC)",
"name" : "Received"
},
{
"value" : "Thu, 29 Dec 2016 11:28:14 -0500",
"name" : "Date"
},
{
"value" : "rlauer@openbedrock.net",
"name" : "To"
},
{
"value" : "test",
"name" : "Subject"
},
{
"value" : "Heirloom mailx 12.4 7/29/08",
"name" : "User-Agent"
},
{
"value" : "1.0",
"name" : "MIME-Version"
},
{
"value" : "text/plain; charset=us-ascii",
"name" : "Content-Type"
},
{
"value" : "7bit",
"name" : "Content-Transfer-Encoding"
},
{
"value" : "<20161229162814.0F3CEDB93@openbedrock.net>",
"name" : "Message-Id"
},
{
"value" : "rlauer@openbedrock.net",
"name" : "From"
}
],
"timestamp" : "2016-12-29T16:30:19.961Z",
"commonHeaders" : {
"messageId" : "<20161229162814.0F3CEDB93@openbedrock.net>",
"to" : [
"rlauer@openbedrock.net"
],
"returnPath" : "rlauer@openbedrock.net",
"subject" : "test",
"date" : "Thu, 29 Dec 2016 11:28:14 -0500",
"from" : [
"rlauer@openbedrock.net"
]
},
"destination" : [
"rlauer@openbedrock.net"
]
},
"receipt" : {
"spfVerdict" : {
"status" : "GRAY"
},
"processingTimeMillis" : 383,
"spamVerdict" : {
"status" : "PASS"
},
"dkimVerdict" : {
"status" : "GRAY"
},
"timestamp" : "2016-12-29T16:30:19.961Z",
"virusVerdict" : {
"status" : "PASS"
},
"action" : {
"objectKey" : "mail/2tlb8ad9l26vr2d736e7hsitvgenhf0tu261s801",
"objectKeyPrefix" : "mail",
"bucketName" : "mail.openbedrock.net",
"topicArn" : "arn:aws:sns:us-east-1:<your-account-number>:ses-openbedrock",
"type" : "S3"
},
"recipients" : [
"rlauer@openbedrock.net"
]
}
}
We're particularly interested in the key 'receipt', where we'll find the recipients and an 'action' key that contains the information we'll need to find the email message in our S3 bucket. The 'objectKey' and the 'bucketName' will allow us to find and download the message.
my ($bucketName, $objectKey) = @{$message->{receipt}->{action}}{qw/bucketName objectKey/};
print STDERR sprintf("S3: %s %s\n", $bucketName, $objectKey);
# 3. Download the mail from S3
my $s3 = Amazon::S3->new(
{
aws_access_key_id => $ENV{AWS_ACCESS_KEY_ID},
aws_secret_access_key => $ENV{AWS_SECRET_ACCESS_KEY},
retry => 1
}
);
my $bucket = $s3->bucket($bucketName);
my (undef, $filename) = tempfile();
$bucket->get_key_filename($objectKey, 'GET', $filename);
# 4. Forward the email
open STDIN, '<' . $filename;
my $mail = Email::Filter->new();
$mail->exit(0);
$mail->pipe('sendmail', $ENV{FORWARD_MAIL_TO});
To make sure we don't end up processing this message again, the standard protocol regarding working with SQS messages is to delete the message after it has been successfully processed.
# 5. Delete the message from the queue
my $receiptHandle = $msg->getReceiptHandle();
my $deleteMessageRequest = Amazon::SQS::Model::DeleteMessageRequest->new(
{
QueueUrl => $queue_url,
ReceiptHandle => $receiptHandle
}
);
$sqs->deleteMessage($deleteMessageRequest);
Finally, we remove the email message from the S3 bucket.
#6. Delete the message from the S3 bucket
$bucket->delete_key($objectKey);
To Do
There are lot more things you might want to consider in writing a script that interprets your SES mail. For example:
- Check the virus and spam status and remove mail that does not pass those tests
- Only forward mail for known recipients
- Filter attachments
- Archive mail using S3 lifecycle rules
Using SES for inbound email is not that hard. Hopefully this blog will help you create your own inbound mail processor. Good luck!
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.