openec2 Article Description

Dependencies on previous articles around SES/Rules/Lambda etc.

It has been some time before SDK3 functions have become available to forward emails.
The following has been tested on two live sites:

– First, we forward the emails in SES Email Rules. We check for spam, nominated bad domains/addresses, and IP blocking.
– Then we run a shell script once a night to process the emails that are still in the S3 Bucket.

Some points to keep in mind:

You will need to keep in mind the shell script “email.sh” uses a single character ^M with the sed command, (use control-v-M ) so after you cut and paste the code, edit it, or download the script which already has the ^M as one character.

This is based on the previous SDK2 (node16) work. You do need to understand how to use these rules and the IAM permissions, S3 Bucket permissions, and Lambda functions as described in detail on my website.

We use three Lambda functions. The spam function is needed as Amazon SES inserts headers that show if an email fails, but does still forwards spam unless we have the function to stop it. The second function blocks any nominated bad domains or email addresses. For instance, I block China – .cn, and what I call mosquito domains such as .mx, .io, .tech, .info, .biz. The third function forwards the email to one or more people, including multiple addresses like admin@, contact@ and so on.

The dmarc@ emails are trapped by a second SES Email Rule. Keep in mind your forwarding addresses should really be verified addresses and not aliases. (You can test otherwise, but this has worked best for me.)

The forwarding function may contain multiple domains, such as mydomain.com.au and mydomain.com, placing into the same bucket – e.g. a bucket name could be mydomain.inbox.

The S3 email Bucket requires a working end-point. In my case it works if the bucket is created in us-west-2 (Oregon). It fails if the bucket is created in Sydney.

LAMBDA FUNCTION to Drop Spam - e.g. call it SDK3_mydomain_com_spam

export const handler = async (event, context, callback) => {
  console.log('Spam filter');
  
  const sesNotification = event.Records[0].ses;
  console.log("SES Notification:\n", JSON.stringify(sesNotification, null, 2));
  
  // Check if any spam check failed
  if (sesNotification.receipt.spfVerdict.status === 'FAIL'
          || sesNotification.receipt.dkimVerdict.status === 'FAIL'
          || sesNotification.receipt.spamVerdict.status === 'FAIL'
          || sesNotification.receipt.virusVerdict.status === 'FAIL') {
              
      console.log('Dropping spam');

      // Stop processing rule set, dropping message
      callback(null, {'disposition':'STOP_RULE_SET'});
  } else {
      callback(null, {'disposition':'CONTINUE'});   
  }
};

[save the code]
Under configuration change the time from 3s to 5s.


LAMBDA SDK3 Function to Drop bad domains or email addresses - e.g. call it SDK3_mydomain_com_domains

'use strict';

export const handler = async (event) => {
    console.log('Blocking email filter starting');
    const sesNotification = event.Records[0].ses;
    const messageId = sesNotification.mail.messageId;
    const receipt = sesNotification.receipt;
    const mail = sesNotification.mail;

    // Convert the environment variable into array. Clean spaces from it.
    const blockingListString = process.env.blockingList;
    const blockingListArray = blockingListString.replace(/\s/g, '').split(",");

    // Check if the mail source matches with any of the email addresses or domains defined in the environment variable
    const isListed = () => {
        return blockingListArray.some(item => mail.source.endsWith(item));
    };

    console.log('Processing message:', messageId);

    // Processing the message
    if (isListed()) {
        console.log('Rejecting messageId: ', messageId, ' - Source: ', mail.source, ' - Recipients: ', receipt.recipients, ' - Subject: ', mail.commonHeaders['subject']);
        return { disposition: 'STOP_RULE_SET' };
    } else {
        console.log('Accepting messageId:', messageId, ' - Source: ', mail.source, ' - Recipients: ', receipt.recipients, ' - Subject: ', mail.commonHeaders['subject']);
        return { disposition: 'CONTINUE' };
    }
};


[save the code]

Change the Configuration tab to show 5s instead of 3s.
Create the blocking rules via the Configuration > Environment variables tab:
Call the key: blockingList. (with a capital L)
In the values field use things like this: .io,.info,.biz,.cn,.ru,.tech,.mx, baddomain.com, jo@youareapest.com

I want to provide the Lambda code up front. You need to edit the fields near the top for your own Oregon (assumed) bucket name, but do have the Bucket subfolders .archive, .other, .dmarc, .ses so the email.sh script works.
The example code lists one email primary domain with some aliases and multiple forwarding – so change to suite your addresses.
LAMBDA FORWARDING FUNCTION - e.g. call it SDK3_mydomain_com_forwarding (use your own name) with Node22, ARM, and your Lambda access from IAM as per my other articles.

"use strict";

import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import { SESClient, SendRawEmailCommand } from "@aws-sdk/client-ses";

console.log("AWS Lambda SES Forwarder // @arithmetric // Version 5.0.0");

// Configure the S3 bucket and key prefix for stored raw emails, and the
// mapping of email addresses to forward from and to.
//
// Expected keys/values:
//
// - fromEmail: Forwarded emails will come from this verified address
//
// - subjectPrefix: Forwarded emails subject will contain this prefix
//
// - emailBucket: S3 bucket name where SES stores emails.
//
// - emailKeyPrefix: S3 key name prefix where SES stores email. Include the
//   trailing slash.
//
// - allowPlusSign: Enables support for plus sign suffixes on email addresses.
//   If set to `true`, the username/mailbox part of an email address is parsed
//   to remove anything after a plus sign. For example, an email sent to
//   `example+test@example.com` would be treated as if it was sent to
//   `example@example.com`.
//
// - forwardMapping: Object where the key is the lowercase email address from
//   which to forward and the value is an array of email addresses to which to
//   send the message.
//
//   To match all email addresses on a domain, use a key without the name part
//   of an email address before the "at" symbol (i.e. `@example.com`).
//
//   To match a mailbox name on all domains, use a key without the "at" symbol
//   and domain part of an email address (i.e. `info`).
//
//   To match all email addresses matching no other mapping, use "@" as a key.
var defaultConfig = {
  fromEmail: "contact@mydomain.com.au",
  subjectPrefix: "",
  emailBucket: "mydomain.inbox",
  emailKeyPrefix: ".ses/",
  allowPlusSign: true,
  forwardMapping: {
    "contact@mydomain.com.au": [
      "me@gmail.com",
      "you@gmail.com"
     ],
    "admin@mydomain.com.au": [
      "me@gmail.com"
    ],
    "postmaster@mydomain.com.au": [
      "me@gmail.com"
    ],
    "abuse@mydomain.com.au": [
      "me@gmail.com"
    ],
    "contact@mydomain.com": [
      "me@gmail.com"
     ],
    "admin@mydomain.com": [
      "me@gmail.com"
    ],
    "postmaster@mydomain.com": [
      "me@gmail.com"
    "abuse@mydomain.com": [
      "me@gmail.com"
    ]
  }
};

/**
 * Parses the SES event record provided for the `mail` and `receipients` data.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
async function parseEvent(data) {
  // Validate characteristics of a SES event record.
  if (!data.event ||
      !data.event.hasOwnProperty('Records') ||
      data.event.Records.length !== 1 ||
      !data.event.Records[0].hasOwnProperty('eventSource') ||
      data.event.Records[0].eventSource !== 'aws:ses' ||
      data.event.Records[0].eventVersion !== '1.0') {
    return Promise.reject(new Error('Error: Received invalid SES message.'));
  }

  data.email = data.event.Records[0].ses.mail;
  data.recipients = data.event.Records[0].ses.receipt.recipients;
  return Promise.resolve(data);
}

/**
 * Transforms the original recipients to the desired forwarded destinations.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */

async function transformRecipients(data) {
  const newRecipients = new Set(); // Use a Set for efficient uniqueness checks
  data.originalRecipients = data.recipients;

  for (const origEmail of data.originalRecipients) {
    const origEmailKey = data.config.allowPlusSign
      ? origEmail.toLowerCase().split('+')[0]
      : origEmail.toLowerCase();

    // Check for direct email mappings
    if (data.config.forwardMapping[origEmailKey]) {
      data.config.forwardMapping[origEmailKey].forEach(email => newRecipients.add(email));
      continue; // Skip further checks if a direct mapping is found
    }

    // Check for domain and username mappings
    const [origEmailUser, origEmailDomain] = origEmailKey.split('@');
    if (origEmailDomain && data.config.forwardMapping[`@${origEmailDomain}`]) {
      data.config.forwardMapping[`@${origEmailDomain}`].forEach(email => newRecipients.add(email));
    } else if (origEmailUser && data.config.forwardMapping[origEmailUser]) {
      data.config.forwardMapping[origEmailUser].forEach(email => newRecipients.add(email));
    } else if (data.config.forwardMapping['@']) {
      data.config.forwardMapping['@'].forEach(email => newRecipients.add(email));
    }
  }

  data.recipients = Array.from(newRecipients); // Convert Set back to Array
  return data;
}


/**
 * Fetches the message data from S3.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
async function fetchMessage(data) {
    // Copying email object to ensure read permission
    const s3Params = {
        Bucket: data.config.emailBucket,
        Key: data.config.emailKeyPrefix + data.email.messageId,
    };

    try {
        const s3Response = await data.s3.send(new GetObjectCommand(s3Params));
        // Stream the S3 object contents
        const stream = s3Response.Body;
        const chunks = [];
        for await (const chunk of stream) {
            chunks.push(chunk);
        }
        data.emailData = Buffer.concat(chunks).toString('utf-8');
        return data;
    } catch (error) {
        console.error("Error fetching message from S3:", error);
        throw new Error('Error: Failed to load message body from S3.');
    }
}

/**
 * Processes the message data, making updates to recipients and other headers
 * before forwarding message.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
async function processMessage(data) {
  var match = data.emailData.match(/^((?:.+\r?\n)*)(\r?\n(?:.*\s+)*)/m);
  var header = match && match[1] ? match[1] : data.emailData;
  var body = match && match[2] ? match[2] : '';

  // Add "Reply-To:" with the "From" address if it doesn't already exists
  if (!/^reply-to:[\t ]?/mi.test(header)) {
    match = header.match(/^from:[\t ]?(.*(?:\r?\n\s+.*)*\r?\n)/mi);
    var from = match && match[1] ? match[1] : '';
    if (from) {
      header = header + 'Reply-To: ' + from;
    }
  }

  // SES does not allow sending messages from an unverified address,
  // so replace the message's "From:" header with the original
  // recipient (which is a verified domain)
  header = header.replace(
    /^from:[\t ]?(.*(?:\r?\n\s+.*)*)/mgi,
    function(match, from) {
      var fromText;
      if (data.config.fromEmail) {
        fromText = 'From: ' + from.replace(/<(.*)>/, '').trim() +
        ' <' + data.config.fromEmail + '>';
      } else {
        fromText = 'From: ' + from.replace('<', 'at ').replace('>', '') +
        ' <' + data.originalRecipient + '>';
      }
      return fromText;
    });

  // Add a prefix to the Subject
  if (data.config.subjectPrefix) {
    header = header.replace(
      /^subject:[\t ]?(.*)/mgi,
      function(match, subject) {
        return 'Subject: ' + data.config.subjectPrefix + subject;
      });
  }

  // Replace original 'To' header with a manually defined one
  if (data.config.toEmail) {
    header = header.replace(/^to:[\t ]?(.*)/mgi, () => 'To: ' + data.config.toEmail);
  }

  // Remove the Return-Path header.
  header = header.replace(/^return-path:[\t ]?(.*)\r?\n/mgi, '');

  // Remove Sender header.
  header = header.replace(/^sender:[\t ]?(.*)\r?\n/mgi, '');

  // Remove Message-ID header.
  header = header.replace(/^message-id:[\t ]?(.*)\r?\n/mgi, '');

  // Remove all DKIM-Signature headers to prevent triggering an
  // "InvalidParameterValue: Duplicate header 'DKIM-Signature'" error.
  // These signatures will likely be invalid anyways, since the From
  // header was modified.
  header = header.replace(/^dkim-signature:[\t ]?.*\r?\n(\s+.*\r?\n)*/mgi, '');

  data.emailData = header + body;
  return Promise.resolve(data);
}

/**
 * Send email using the SES sendRawEmail command.
 *
 * @param {object} data - Data bundle with context, email, etc.
 *
 * @return {object} - Promise resolved with data.
 */
async function sendMessage(data) {
  var params = {
    Destinations: data.recipients,
    Source: data.originalRecipient,
    RawMessage: {
      Data: Buffer.from(data.emailData)
    }
  };
  return new Promise(function(resolve, reject) {
    data.ses.send(new SendRawEmailCommand(params), function(err, result) {
      if (err) {
        return reject(new Error('Error: Email sending failed.'));
      }
      resolve(data);
    });
  });
}

/**
 * Handler function to be invoked by AWS Lambda with an inbound SES email as
 * the event.
 *
 * @param {object} event - Lambda event from inbound email received by AWS SES.
 * @param {object} context - Lambda context object.
 * @param {object} callback - Lambda callback object.
 * @param {object} overrides - Overrides for the default data, including the
 * configuration, SES object, and S3 object.
 */
export const handler = async (event, context, callback, overrides) => {
  try {
    let steps = overrides?.steps || [
      parseEvent,
      transformRecipients,
      fetchMessage,
      processMessage,
      sendMessage
    ];

    let data = {
      event,
      context,
      config: overrides?.config || defaultConfig,
      ses: overrides?.ses || new SESClient(),
      s3: overrides?.s3 || new S3Client({ signatureVersion: 'v4' })
    };

    for (const step of steps) {
      data = await step(data);
    }

    console.log("Process finished successfully.");
    callback(null, 'Success');
  } catch (error) {
    console.error("Error in processing:", error);
    callback(error);
  }
};

[save the code]
Under configuration change the time from 3s to 10s.

As per other articles, you will have SMTP in place, your IAM rules, an S3 bucket in Oregon (or a valid end point) where the bucket has the SES/S3/Lambda permissions and subfolders previously mentioned, verified domain and validated forwarding addresses.
You then create an Email Receiving rule in SES in the default set.
It would basically add the email addresses you want, except dmarc@, where the first rule puts the email into your nominated bucket, then the three lambda functions, where only the last function checks Event Invocation (the other Lambdas use RequestResponse invocation).
It will likely ask to add permissions when saving. You do test emails by looking at the bucket and Cloudwatch logs.
The SES bucket rule is the first, but you tell it to put your emails onto .ses/ so that we can have more processing later with email.sh shell script.

You create a second Enail receiving rule to accept dmarc@ emails. You simply place them into the bucket with the subfolder .dmarc/

You then have a short life cycle rule on that folder.

If you find MS Exchange or Google dmarc emails are still going through to your inbox, you can capture these in this rule as well, and modify the email.sh script to capture them also. A bit more work.

This script checks for up to four failure conditions. You can modify. These emails go to the .other subfolder, with a short life cycle.

These are registered in register_black.csv.

It archives “good” emails into the bucket’s .archive folder. These are registered in register.csv.

Your own website could have a password protected page that downloads these .csv files.

If any dmarc email escape the SES rules, you can edit the script for these as well.

Remember, the ^M shown is a single character, so I have a link to download the file, rather than giving the code here.

Download email.tar from the button below, and in the Linux editor (e.g. vi) you can see the ^M characters with the sed command. You cannot edit in a desktop editor to get these characters.

Once done, you can have a line like this in crontab:

30 2 * * * /home/ec2-user/email.sh mydomain.inbox me mydomain.com >/dev/null 2>&1

[save and exit]

Where mydomain.inbox is any name yuo created and configured to an S3 email bucket, "me" and mydomain.com are arbotrary subfolders you create where the shell script will be processing the bucket emails.

Now, create this directory (using your own names)

cd /
mkdir data
chmod 2775 data
cd data
mkdir ses
cd ses
mkdir mydomain.com
cd mydomain.com
mkdir me
cd me

mkdir archive
mkdir outdir
mkdir workdir
echo "Date, Time, Size, Email, Client IP, From, To, Subject" > register.csv
echo "Date, Time, Size, Email, Client IP, From, To, Subject" > register_block.csv

echo ".ses" > content.dat
vi forward.dat

[save and exit with one blank line]

echo ".ses" > sets.dat

Now ls -l and check these are all there.

cd /data
chmod -R 777 ses
chgrp -R ec2-user ses

cd ses

vi block.dat

[save and exit with one blank line]
chmod 777 block.dat
chgrp ec2-user block.dat
cp -p block.dat language.dat domain.dat ip.dat spam.dat

You are asking why all this? My original email.sh script provided a lot of options and use of the mutt email command, none of which I use now, but if I keep these it ensures the code does not fail, so it is easier to just do these commands.