openec2 Article Description
Part 8
Configure an SES/S3 Bucket for forwarding email
We will create four Lambda functions under yur email regions (e.g. Oregon).We will then edit and remove our general Lambda function from your SES Receiving rules, and add the four new functions.
These functions need to use the following order – you can play around with re-ordering when editing the SES rule.
We will use the example domain name.
Lambda functions:
mydomain_com_receipt
mydomain_com_dkim
mydomain_com_dmarc
mydomain_com_forward
Dkim looks at valid SPF/DKIM content in the email.
The SES rule will need the email only to go into the S3 bucket after the dmarc test has been successful. So use the order:
receipt, dkim, dmarc, S3 bucket, forward.
The rules prior to the bucket need “RequestResponse invocation”. The last function needs “Event invocation”.
By this stage you should be able to work things out sufficiently without as many diagrams. Just ensure you are in your correct region.
Create each function using Node.js.16x, arm64, and the Role we created in IAM, e.g. mydomain_com_inbox that we created in an earlier lesson.
The receipt function’s code:
'use strict';
const AWS = require('aws-sdk');
exports.handler = (event, context, callback) => {
console.log('Blocking email filter starting');
const sesNotification = event.Records[0].ses;
const messageId = sesNotification.mail.messageId;
const receipt = sesNotification.receipt;
const mail = sesNotification.mail;
// Convert the environment variable into array. Clean spaces from it.
var blockingListString = process.env.blockingList;
blockingListString = blockingListString.replace(/\s/g,'');
var blockingListArray = blockingListString.split(",");
// Check if the mail source matches with any of the email addresses or domains defined in the environment variable
function isListed() {
var length = blockingListArray.length;
for(var i = 0; i < length; i++) {
if (mail.source.endsWith(blockingListArray[i]))
return true;
}
return false;
}
console.log('Processing message:', messageId);
// Processing the message
if (isListed()) {
callback(null, {'disposition':'STOP_RULE_SET'});
console.log('Rejecting messageId: ', messageId, ' - Source: ', mail.source, ' - Recipients: ',receipt.recipients,' - Subject: ', mail.commonHeaders['subject']);
}
else {
console.log('Accepting messageId:', messageId, ' - Source: ', mail.source, ' - Recipients: ',receipt.recipients,' - Subject: ', mail.commonHeaders['subject']);
callback();
}
};
After deploying the code, go to the configurations tab, and then the General Configuration menu on the left, and change the Timeout to 5 seconds. The memory should be 128MB.
Then go to the Environment variables menu on the left, and add a rule, (optional) to block email coming from eny specific email address, a country, or a domain top level domain type. No world cards are allowed.
To do this, click Edit, and in the Key field add:
blockingList
(This has a capital “L” in it)
Then add your list. e.g.:
.online,.io.,.ru,.cn,.biz,.info
The dkim function’s code:
exports.handler = function(event, context, callback) {
console.log('Spam filter');
var sesNotification = event.Records[0].ses;
console.log("SES Notification:\n", JSON.stringify(sesNotification, null, 2));
// Check if any spam check failed
if (sesNotification.receipt.spfVerdict.status === 'FAIL'
|| sesNotification.receipt.dkimVerdict.status === 'FAIL'
|| sesNotification.receipt.spamVerdict.status === 'FAIL'
|| sesNotification.receipt.virusVerdict.status === 'FAIL') {
console.log('Dropping spam');
// Stop processing rule set, dropping message
callback(null, {'disposition':'STOP_RULE_SET'});
} else {
callback(null, null);
}
};The dmarc function’s code:
'use strict';
const AWS = require('aws-sdk');
// Assign the emailDomain environment variable to a constant.
const emailDomain = 'johnrigbyart.com.au';
exports.handler = (event, context, callback) => {
console.log('Spam filter starting');
const sesNotification = event.Records[0].ses;
const messageId = sesNotification.mail.messageId;
const receipt = sesNotification.receipt;
console.log('Processing message:', messageId);
// If DMARC verdict is FAIL and the sending domain's policy is REJECT
// (p=reject), bounce the email.
if (receipt.dmarcVerdict.status === 'FAIL'
&& receipt.dmarcPolicy.status === 'REJECT') {
// The values that make up the body of the bounce message.
const sendBounceParams = {
BounceSender: `mailer-daemon@${emailDomain}`,
OriginalMessageId: messageId,
MessageDsn: {
ReportingMta: `dns; ${emailDomain}`,
ArrivalDate: new Date(),
ExtensionFields: [],
},
// Include custom text explaining why the email was bounced.
Explanation: "Unauthenticated email is not accepted due to the sending domain's DMARC policy.",
BouncedRecipientInfoList: receipt.recipients.map((recipient) => ({
Recipient: recipient,
// Bounce with 550 5.6.1 Message content rejected
BounceType: 'ContentRejected',
})),
};
console.log('Bouncing message with parameters:');
console.log(JSON.stringify(sendBounceParams, null, 2));
// Try to send the bounce.
new AWS.SES().sendBounce(sendBounceParams, (err, data) => {
// If something goes wrong, log the issue.
if (err) {
console.log(`An error occurred while sending bounce for message: ${messageId}`, err);
callback(err);
// Otherwise, log the message ID for the bounce email.
} else {
console.log(`Bounce for message ${messageId} sent, bounce message ID: ${data.MessageId}`);
// Stop processing additional receipt rules in the rule set.
callback(null, {
disposition: 'stop_rule_set',
});
}
});
// If the DMARC verdict is anything else (PASS, QUARANTINE or GRAY), accept
// the message and process remaining receipt rules in the rule set.
} else {
console.log('Accepting message:', messageId);
callback();
}
};
We now add our mydomain_com_forward function:
"use strict";
var AWS = require('aws-sdk');
console.log("AWS Lambda SES Forwarder // @arithmetric // Version 4.2.0");
// Configure the S3 bucket and key prefix for stored raw emails, and the
// mapping of email addresses to forward from and to.
//
// Expected keys/values:
//
// - fromEmail: Forwarded emails will come from this verified address
//
// - subjectPrefix: Forwarded emails subject will contain this prefix
//
// - emailBucket: S3 bucket name where SES stores emails.
//
// - emailKeyPrefix: S3 key name prefix where SES stores email. Include the
// trailing slash.
//
// - forwardMapping: Object where the key is the lowercase email address from
// which to forward and the value is an array of email addresses to which to
// send the message.
//
// To match all email addresses on a domain, use a key without the name part
// of an email address before the "at" symbol (i.e. `@example.com`).
//
// To match a mailbox name on all domains, use a key without the "at" symbol
// and domain part of an email address (i.e. `info`).
var defaultConfig = {
fromEmail: "admin@mydomain.com",
subjectPrefix: "",
emailBucket: "mydomain.com.inbox",
emailKeyPrefix: "",
forwardMapping: {
"contact@mydomain.com": [
"me@gmail.com"
],
"admin@mydomain.com": [
"me@gmail.com"
],
"abuse@mydomain.com": [
"me@gmail.com"
],
"postmaster@mydomain.com": [
"me@gmail.com"
],
"webmaster@mydomain.com": [
"me@gmail.com"
],
"dmarc@mydomain.com": [
"me@gmail.com"
],
"noreply@mydomain.com": [
"me@gmail.com"
]
}
};
exports.parseEvent = function(data) {
// Validate characteristics of a SES event record.
if (!data.event ||
!data.event.hasOwnProperty('Records') ||
data.event.Records.length !== 1 ||
!data.event.Records[0].hasOwnProperty('eventSource') ||
data.event.Records[0].eventSource !== 'aws:ses' ||
data.event.Records[0].eventVersion !== '1.0') {
data.log({message: "parseEvent() received invalid SES message:",
level: "error", event: JSON.stringify(data.event)});
return Promise.reject(new Error('Error: Received invalid SES message.'));
}
data.email = data.event.Records[0].ses.mail;
data.recipients = data.event.Records[0].ses.receipt.recipients;
return Promise.resolve(data);
};
/**
* Transforms the original recipients to the desired forwarded destinations.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.transformRecipients = function(data) {
var newRecipients = [];
data.originalRecipients = data.recipients;
data.recipients.forEach(function(origEmail) {
var origEmailKey = origEmail.toLowerCase();
if (data.config.forwardMapping.hasOwnProperty(origEmailKey)) {
newRecipients = newRecipients.concat(
data.config.forwardMapping[origEmailKey]);
data.originalRecipient = origEmail;
} else {
var origEmailDomain;
var origEmailUser;
var pos = origEmailKey.lastIndexOf("@");
if (pos === -1) {
origEmailUser = origEmailKey;
} else {
origEmailDomain = origEmailKey.slice(pos);
origEmailUser = origEmailKey.slice(0, pos);
}
if (origEmailDomain &&
data.config.forwardMapping.hasOwnProperty(origEmailDomain)) {
newRecipients = newRecipients.concat(
data.config.forwardMapping[origEmailDomain]);
data.originalRecipient = origEmail;
} else if (origEmailUser &&
data.config.forwardMapping.hasOwnProperty(origEmailUser)) {
newRecipients = newRecipients.concat(
data.config.forwardMapping[origEmailUser]);
data.originalRecipient = origEmail;
}
}
});
if (!newRecipients.length) {
data.log({message: "Finishing process. No new recipients found for " +
"original destinations: " + data.originalRecipients.join(", "),
level: "info"});
return data.callback();
}
data.recipients = newRecipients;
return Promise.resolve(data);
};
/**
* Fetches the message data from S3.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.fetchMessage = function(data) {
// Copying email object to ensure read permission
data.log({level: "info", message: "Fetching email at s3://" +
data.config.emailBucket + '/' + data.config.emailKeyPrefix +
data.email.messageId});
return new Promise(function(resolve, reject) {
data.s3.copyObject({
Bucket: data.config.emailBucket,
CopySource: data.config.emailBucket + '/' + data.config.emailKeyPrefix +
data.email.messageId,
Key: data.config.emailKeyPrefix + data.email.messageId,
ACL: 'private',
ContentType: 'text/plain',
StorageClass: 'STANDARD'
}, function(err) {
if (err) {
data.log({level: "error", message: "copyObject() returned error:",
error: err, stack: err.stack});
return reject(
new Error("Error: Could not make readable copy of email."));
}
// Load the raw email from S3
data.s3.getObject({
Bucket: data.config.emailBucket,
Key: data.config.emailKeyPrefix + data.email.messageId
}, function(err, result) {
if (err) {
data.log({level: "error", message: "getObject() returned error:",
error: err, stack: err.stack});
return reject(
new Error("Error: Failed to load message body from S3."));
}
data.emailData = result.Body.toString();
return resolve(data);
});
});
});
};
/**
* Processes the message data, making updates to recipients and other headers
* before forwarding message.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.processMessage = function(data) {
var match = data.emailData.match(/^((?:.+\r?\n)*)(\r?\n(?:.*\s+)*)/m);
var header = match && match[1] ? match[1] : data.emailData;
var body = match && match[2] ? match[2] : '';
// Add "Reply-To:" with the "From" address if it doesn't already exists
if (!/^Reply-To: /mi.test(header)) {
match = header.match(/^From: (.*(?:\r?\n\s+.*)*\r?\n)/m);
var from = match && match[1] ? match[1] : '';
if (from) {
header = header + 'Reply-To: ' + from;
data.log({level: "info", message: "Added Reply-To address of: " + from});
} else {
data.log({level: "info", message: "Reply-To address not added because " +
"From address was not properly extracted."});
}
}
// SES does not allow sending messages from an unverified address,
// so replace the message's "From:" header with the original
// recipient (which is a verified domain)
header = header.replace(
/^From: (.*(?:\r?\n\s+.*)*)/mg,
function(match, from) {
var fromText;
if (data.config.fromEmail) {
fromText = 'From: ' + from.replace(/<(.*)>/, '').trim() +
' <' + data.config.fromEmail + '>';
} else {
fromText = 'From: ' + from.replace('<', 'at ').replace('>', '') +
' <' + data.originalRecipient + '>';
}
return fromText;
});
// Add a prefix to the Subject
if (data.config.subjectPrefix) {
header = header.replace(
/^Subject: (.*)/mg,
function(match, subject) {
return 'Subject: ' + data.config.subjectPrefix + subject;
});
}
// Replace original 'To' header with a manually defined one
if (data.config.toEmail) {
header = header.replace(/^To: (.*)/mg, () => 'To: ' + data.config.toEmail);
}
// Remove the Return-Path header.
header = header.replace(/^Return-Path: (.*)\r?\n/mg, '');
// Remove Sender header.
header = header.replace(/^Sender: (.*)\r?\n/mg, '');
// Remove Message-ID header.
header = header.replace(/^Message-ID: (.*)\r?\n/mig, '');
// Remove all DKIM-Signature headers to prevent triggering an
// "InvalidParameterValue: Duplicate header 'DKIM-Signature'" error.
// These signatures will likely be invalid anyways, since the From
// header was modified.
header = header.replace(/^DKIM-Signature: .*\r?\n(\s+.*\r?\n)*/mg, '');
data.emailData = header + body;
return Promise.resolve(data);
};
/**
* Send email using the SES sendRawEmail command.
*
* @param {object} data - Data bundle with context, email, etc.
*
* @return {object} - Promise resolved with data.
*/
exports.sendMessage = function(data) {
var params = {
Destinations: data.recipients,
Source: data.originalRecipient,
RawMessage: {
Data: data.emailData
}
};
data.log({level: "info", message: "sendMessage: Sending email via SES. " +
"Original recipients: " + data.originalRecipients.join(", ") +
". Transformed recipients: " + data.recipients.join(", ") + "."});
return new Promise(function(resolve, reject) {
data.ses.sendRawEmail(params, function(err, result) {
if (err) {
data.log({level: "error", message: "sendRawEmail() returned error.",
error: err, stack: err.stack});
return reject(new Error('Error: Email sending failed.'));
}
data.log({level: "info", message: "sendRawEmail() successful.",
result: result});
resolve(data);
});
});
};
/**
* Handler function to be invoked by AWS Lambda with an inbound SES email as
* the event.
*
* @param {object} event - Lambda event from inbound email received by AWS SES.
* @param {object} context - Lambda context object.
* @param {object} callback - Lambda callback object.
* @param {object} overrides - Overrides for the default data, including the
* configuration, SES object, and S3 object.
*/
exports.handler = function(event, context, callback, overrides) {
var steps = overrides && overrides.steps ? overrides.steps :
[
exports.parseEvent,
exports.transformRecipients,
exports.fetchMessage,
exports.processMessage,
exports.sendMessage
];
var data = {
event: event,
callback: callback,
context: context,
config: overrides && overrides.config ? overrides.config : defaultConfig,
log: overrides && overrides.log ? overrides.log : console.log,
ses: overrides && overrides.ses ? overrides.ses : new AWS.SES(),
s3: overrides && overrides.s3 ?
overrides.s3 : new AWS.S3({signatureVersion: 'v4'})
};
Promise.series(steps, data)
.then(function(data) {
data.log({level: "info", message: "Process finished successfully."});
return data.callback();
})
.catch(function(err) {
data.log({level: "error", message: "Step returned error: " + err.message,
error: err, stack: err.stack});
return data.callback(new Error("Error: Step returned error."));
});
};
Promise.series = function(promises, initValue) {
return promises.reduce(function(chain, promise) {
if (typeof promise !== 'function') {
return Promise.reject(new Error("Error: Invalid promise item: " +
promise));
}
return chain.then(promise);
}, Promise.resolve(initValue));
};
Under the Configuration tab, general configuration, use 10 seconds, and 128MB.
Now edit the forwarding function to use your values. Where you see …@mydomain.com use your own domain, and don’t forget we are still in sandbox mode, and change me@gmail.com to your own address that was verified in SES. You can edit these at any time.
You can have multiple forwarding of emails. e.g.:
"contact@mydomain.com": [ "me@gmail.com", "you@hotmail.com" ],
These edits are only in this portion of the code. No need to worry about further sections below it or above it.
These scripts are provided by Amazon AWS documents and Internet Forums.
Now, this is important… Where you see the bucket name: emailBucket:
“mydomain.com.inbox”,
emailKeyPrefix: “.in/”,
The emailKey Prefix is “” (blank) for emails to go into the top level folder. If you create a subfolder, (I use .name with the dot, so that any shell scripting can easily know what I’m looking for) this goes in both the Prefix with the forward slash, and in the SES Receiving rule, we enter the same (with the slash) into the S3 bucket receiving rule:
Object key prefix – optional
.in/
You will see this option in the SES bucket rule, but have it in both the Lambda function as above, and SES. If this is the case you do not want a global life cycle expiry rule in the S3 bucket, so remove any global life cycle rule, then create a new rule to configure it to expire on a prefix only.
E.g., under these headings, select:
Limit the scope of this rule using one or more filters
And enter the prefix in the Prefix field. e.g. .in/ (or whatever name you created in the bucket as a subfolder). Obviously you monitor that subfolder for testing incoming emails once this is in the Lambda function and the SES rule. Get familiar with how to refresh an Amazon service page.
It seems a lot to understand here in writing, but just work it through from the configuration screens. If you have done the previous lessons, this will work out okay. Then, when you send a test email, check it in the CloudWatch logs.
The following screen shots show how your SES Receiving rule should look. You access these screens by editing the rule you created under the default-rule-set.
You may have skipped some content, but this is the “whole” content I have developed for client solutions since commencing this in 2012.
We can take time to build our knowledge.
Developing shell scripts is separate to these constructions, so would be in separate lessons if I add them later.
