SES Email Forwading 2024

This is complicated stuff. The example below is in production. I have various versions of this.

SES Email Forwading 2024

As with all technical posts, things change over time. This was written in February 2024.

Amazon sunset Lambda Node16, so we lost functionality around email forwarding. The example here will use Node18.

The content below shows how to forward an email message to let one person know an email has arrived in an S3 bucket from SES.

After crontab processes the email, the same person receives it as an attachment, where the shell script uses “mutt” to send the file. Things like “Sendmail” do not work. This is not foolproof. An email notification may also relate to a junk email.

I developed scripts to deal with various spams, bad domain names, blocking etc. but it became highly complex and at times a little faulty. As a result, I am only providing a simplified version of scripting.

index.mjs Lambda function example

Here is the code to forward a notification: (I use us-west-2 for Oregon. Use your own email address and subject Text message.)

// Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: Apache-2.0

import { SESClient, SendEmailCommand } from "@aws-sdk/client-ses";
const ses = new SESClient({ region: "us-west-2" });

export const handler = async(event) => {
  const command = new SendEmailCommand({
    Destination: {
      ToAddresses: ["YOUR_EMAIL@gmail.com"],
    },
    Message: {
      Body: {
        Text: { Data: "Email has arrived for YOU" },
      },

      Subject: { Data: "New Email SES" },
    },
    Source: "YOUR_EMAIL@gmail.com",
  });

  try {
    let response = await ses.send(command);
    // process data.
    return response;
  }
  catch (error) {
    // error handling.
  }
  finally {
    // finally.
  }
};

SES default_rule_set

You need to know how to add an SES rule that identifies inbound emails, places them into an S3 bucket, and runs the Lambda function to notify you. I have other articles on this. The bucket needs specific permissions. This also involves use of IAM. The domain name will need to have been verified in SES as well.

An example of bucket permissions (user your own account and bucket details). The bucket has to be set up before adding the SES rule, as the rule will want to add some further permissions to access the bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowSESPuts-1674865967951",
            "Effect": "Allow",
            "Principal": {
                "Service": "ses.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::MY_EMAIL_BUCKET/*",
            "Condition": {
                "StringEquals": {
                    "AWS:SourceAccount": "MY_AWS_ACCOUNT_NUMBER"
                },
                "StringLike": {
                    "AWS:SourceArn": "arn:aws:ses:*"
                }
            }
        }
    ]
}

A shell script to process the bucket files

This is a simplified version of my scripting.

Where you see “^M” you must replace with CTRLVM to get the special end of line character. THis was not needed in Linux2, but is needed in Linux2023.

The script looks at configuration files under you own /data directory.

cd /home/ec2-user
vi email.sh

#!/bin/sh

# email-ses.sh FORWARD S3 BUCKET EMAILS via POSTIX and mutt 
# Crontab root entry example: * * * * * /home/ec2-user/email-ses.sh domain.com.inbox fred domain.com >/dev/null 2>&1
# S3 bucket minimum subfolders: .ses, .archive, .dmarc, .other

IFS=$'\n'
bucket="$1"
user="$2"
domain="$3"
work="/data/ses/$domain/$user/workdir"
out="/data/ses/$domain/$user/outdir"
dest_register="/data/ses/$domain/$user/register.csv"
adir="/data/ses/$domain/$user/archivedir"
dest_block="/data/ses/$domain/$user/register_block.csv"
bfile="/data/ses/block.dat"
dfile="/data/ses/domain.dat"
ffile="/data/ses/$domain/$user/forward.dat"
ifile="/data/ses/ip.dat"
lfile="/data/ses/language.dat"
sfile="/data/ses/spam.dat"
cfile="/data/ses/$domain/$user/content.dat"

unset IFS

pf=""
pf=`sudo ps -ef|grep "postfix/master"|awk '{print $9}'|head -1`
if [ "$pf" = "-w" ] ; then
:
else
/usr/bin/systemctl start postfix
fi

for sets in `cat /data/ses/$domain/$user/sets.dat`
do

content=""
content=`grep $sets $cfile`
if [ "$content" != "" ] ;
then
content="y"
fi

cd $work 

aws s3 sync s3://$bucket/$sets/ $work
if [ -f "$work/AMAZON_SES_SETUP_NOTIFICATION" ] ; 
then
aws s3 rm s3://$bucket/$sets/AMAZON_SES_SETUP_NOTIFICATION
rm -f $work/AMAZON_SES_SETUP_NOTIFICATION
fi

lf=""
lf=`ls *`
if [ "$lf" != "" ] ;
then
 let fcount=0
  for file in `ls *` 
  do
   let fcount=$fcount+1
   earray[$fcount]="$file"
   size=""
   time=""
   day=""
   sta=""
   sta=`stat $file | grep "Access:"|tail -1`
   day=`echo $sta | grep "Access:"|tail -1|awk '{print $2}'|awk -F- '{print $1,$2,$3}'`
   time=`echo $sta | grep "Access:"|tail -1|awk -F. '{print $1}'|awk '{print $NF}'`
   day_array[$fcount]="$day"
   time_array[$fcount]="$time"
   size=`stat $file | grep "Size:"|head -1|awk '{print $2}'`
   size_array[$fcount]="$size"
   chmod 777 $file
   chgrp ec2-user $file
   done
fi

dmarc_function()
{
    m=""
        m=`grep "Subject: " $work/$efile | grep -i "Report Domain"`
        if [ "$m" != "" ] ;
        then
        aws s3 cp $work/$efile s3://$bucket/.other/$efile.eml
        dmarc_ret=1
    fi
    return "$dmarc_ret"
}

fail_function()
{
    flag=""
     o=""
     o=`head -15 $work/$efile|grep -i "Received-SPF: pass"| awk '{print $1,$2}'`
      if [ "$o" = "" ] ;
      then
         block_ret=1
      fi
        o=""
        o=`grep "X-SES-Spam-Verdict:" $work/$efile`
        p=""
        p=`grep "X-SES-Virus-Verdict:" $work/$efile`

        if [ "$o" != "" ] ;
        then
        k=""
        k=`echo $o|grep PASS`
         if [ "$k" = "" ] ;
          then
          block_ret=1
         fi
        fi

        if [ "$p" != "" ] ;
        then
        k=""
        k=`echo $p|grep PASS`
         if [ "$k" = "" ] ;
          then
          block_ret=1
         fi
        fi

return "$block_ret"
}

block_function()
{
        IFS=$'\n'
        for x in `cat $bfile`
        do
        y=""
        y=`grep "From: " $work/$efile|grep -v "Date:" | grep -v Message-ID | head -1 | grep -i "$x"`
        if [ "$y" != "" ] ;
        then
                block_ret=2
        fi
        done

    return "$block_ret"
}

ip_function()
{
         IFS=$'\n'

for ip in `cat $ifile`
do
ipf=""
ipf=`echo $ip|sed 's/\./\\\./g'`
ipa=""
ipa=`grep "client-ip=" $work/$efile | grep -E $ipf | head -1`
if [ "$ipa" != "" ] ;
then
block_ret=3
fi
done
        return "$block_ret"
}

domain_function()
{
 IFS=$'\n'
 y="";yy=""

 yy=`grep "From: " $work/$efile|grep -v "Date:"|grep -v Message-ID|head -1|awk -F@ '{print $NF}'|awk -F. '{print $NF}'|tr "<" " "|tr ">" " "`
 y=`echo $yy|awk '{print $1}'`

 for x in `cat $dfile`
 do
 j=`echo $x|awk -F\. '{print $NF}'|awk '{print $1}'`
 if [ "$y" = "$j" ] ;
 then
 block_ret=4
 fi
 done 
        
return "$block_ret"
}

forward_email()
{
day=""; time=""; size="";from="";to="";subject=""
day=${day_array[$email]}
time=${time_array[$email]}
size=${size_array[$email]}
from=`grep "From: " $work/$efile|awk '{print $NF}'|sed 's/^M//g'`
to=`grep "To:[ ]" $work/$efile|sed 's/,/-/g'|awk -FTo: '{print $NF}'|sed 's/^M//g'`
subject=`grep "Subject: " $work/$efile|sed 's/,/-/g'|awk -F: '{print $NF}'|awk '{$1=$1}1'`
echo $day , $time , $size , $efile, $from, $to, $subject >> $dest_register

IFS=$'\n'

gr="\\s\\"$sets
for l in `cat $ffile| grep "$gr"`
do
 if [ "$l" != "" ] ;
then
forward=`echo $l|awk '{print $1}'`
# /usr/sbin/sendmail -f $forward $forward < $work/$efile

echo "An e-mail attachment has arrived from Amazon SES for domain $domain" | EMAIL="mail $domain <admin@$domain>" /usr/bin/mutt -s "New e-mail from $domain" $forward -a $out/$efile.eml
aws s3 cp $adir/$efile.eml s3://$bucket/.archive/$efile.eml
else
aws s3 cp $adir/$efile.eml s3://$bucket/.archive/$efile.eml
fi
done
}

block_email()
{
day=""; time=""; size="";from="";to="";subject=""
day=${day_array[$email]}
time=${time_array[$email]}
size=${size_array[$email]}
from=`grep "From: " $work/$efile|awk '{print $NF}'|sed 's/^M//g'`
to=`grep "To:[ ]" $work/$efile|sed 's/,/-/g'|awk -FTo: '{print $NF}'|sed 's/^M//g'`
subject=`grep "Subject: " $work/$efile|sed 's/,/-/g'|awk -F: '{print $NF}'|sed 's/^M//g'`
echo $day , $time , $size , $efile, $from, $to, $subject >> $dest_block
aws s3 cp $work/$efile s3://$bucket/.other/$efile.eml
echo "A BLOCKED e-mail attachment has arrived from Amazon SES for domain $domain" | EMAIL="mail $domain <admin@$domain>" /usr/bin/mutt -s "A BLOCKED e-mail from $domain" $forward -a $out/$efile.eml
}

content_header()
{
cat $work/$efile | sed '0,/Subject: /{s//Subject: [content advice] /}' | sed 's/^M//g' > $work/$efile.tmp
mv $work/$efile.tmp $work/$efile
}

archive_header()
{
cat $work/$efile | sed '0,/Subject: /{s//Subject: [archive] /}' | sed 's/^M//g' > $adir/$efile.tmp
mv $adir/$efile.tmp $adir/$efile.eml
}

click_function()
{
        m=""
        m=`grep -i "click" $work/$efile|grep href|head -1`
        if [ "$m" != "" ] ;
        then    
        content_ret=2
        fi
        return "$content_ret"
}

language_function()
{
IFS=$'\n'
for v in `cat $lfile`
do
        w=""
        w=`grep -w -i "$v|$v[[:space:]]|$v[[:space:]]\|$v\|\b$v\|$v\b" $work/$efile |grep -v -i "$v="|grep -v -i "=$v"|grep -v -i "$v/"|grep -v -i "/$v"|grep -v -i "+$v"|grep -v -i "$v+"|grep -v -i '\-$v'|grep -v -i '$v-'`
        if [ "$w" != "" ] ;
        then 
         content_ret=1
        fi
done
return "$content_ret"
}

spam_function()
{
IFS=$'\n'
for x in `cat $sfile`
do
        y=""
        y=`grep -i "$x" $work/$efile`
        if [ "$y" != "" ] ;
        then
                content_ret=3
        fi
done
return "$content_ret"
}

reset_function()
{
dmarc_ret=0
block_ret=0
content_ret=0
}

# START PROGRAM

dmarc_ret=0
block_ret=0
content_ret=0

# FOR LOOP
if [ "$lf" != "" ] ;
then
let email=0
for efile in ${earray[*]}
do
let email=$email+1
cp $efile $out/$efile.eml
chmod 777 $out/$efile.eml
chgrp ec2-user $out/$efile.eml
dmarc_function
dmarc_ret=$?
# uncomment block function to block specific bad externaladdresses - see block.dat
# block_function
# fail function is for some spf dmarc failures
fail_function
# ip function does not really work
# ip_function
# domain function to block .biz, .info and so on
domain_function
block_ret=$?
# If you want to test content and insert [content advice] into the subject line
# language_function
content_ret=$?
# spam_function
content_ret=$?
# click_function
content_ret=$?
proc=""
if [ "$dmarc_ret" = "0" ] ;
then
:
else
proc="dmarc"
fi

if [ "$block_ret" = "1" ] || [ "$block_ret" = "2" ] || [ "$block_ret" = "3" ] || [ "$block_ret" = "4" ] ;
then
proc="block"
block_email
else
:
fi

archive_header

if [ "$content_ret" = "1" ] || [ "$content_ret" = "2" ] || [ "$content_ret" = "3" ] ;
then
 if [ "$content" = "y" ] ;
 then
 content_header
 fi
fi

if [ "$proc" = "" ] ;
then
forward_email
proc="email"
fi

 if [ "$work" != "" ] ;
 then
   if [ "$efile" != "" ] ;
   then
     if [ -f $work/$efile ] ;
     then
     rm -f $work/$efile
     aws s3 rm s3://$bucket/$sets/$efile
     fi

     if [ -f $adir/$efile.eml ] ;
     then
     rm -f $adir/$efile.eml 
     fi


     if [ -f $out/$efile.eml ] ;
     then
     rm -f $out/$efile.eml
     fi

   fi
 fi

reset_function

done
# lf not null
fi
# end sets
done

# systemctl stop postfix - if this is required as a precaution

exit


[save and exit]

The script is run with the command:

[crontab run every 30 minutes in this example]

30 * * * * /home/ec2-user/email.sh YOUR_S3_BUCKET YOUR_SUBDIRECTORY_USER YOUR_SUBDIRECTORY >/dev/null 2>&1

[For example:]

/home/ec2-user/email.sh mydomain.com.inbox fred mydomain.com

We would then have these directories on Linux: (all your own naming choice)

/data/ses/mydomian.com/fred

The ses directory has some basic configuration files, which for simplification are:

cd /
mkdir data
chmod 2775 data
cd data
mkdir ses
chmod 2775 ses
cd ses
mkdir mydomain.com
chmod 2775 mydomain.com
touch block.dat
touch domain.dat
touch ip.dat
touch language.dat
touch spam.dat
chmod 777 *.dat

[As I have not refined this work, simply create the above files. You can put specific email addresses you want to block in block.dat without any arrows. e.g., you coud block bad.person@baddomain.com]

cd mydomain.com
mkdir fred
chmod 2775 fred
cd fred
mkdir archivedir workdir outdir
vi content.dat
.ses
[save and exit - this was originally intended to alert to bad content in the email. I still need it to run the script regarldess.]
vi sets.dat
.ses
.set
[save and exit]

vi forward.dat

[If you include .set, it means you have an SES rule to place specific emails such as john@gmail.com or julie@gmail.com to the S3 bucket undr the subfolder .set.
They will both receive the emails. If you only want emails to yourself, such as me@, dmarc@, admin@, webmaster@ and so forth, these can be identified by SES and placed into the .ses bucket subfolder.
In the follwing forward.dat file, you could have:]

me@gmail.com .ses

[If you have the .set subfoldre receiving emails, you could send a copy to yourself and others like this:]

me@gmail.com .set
john@gmail.com .set
julie@gmail.com .set

[save and exit]

[Under your email bucket, such as mydomain.com.inbox, create the subfolders:]

.ses
.set
.archive
.other

[You can put life cycles on these if you wish. Emails are archived as .eml format under .other or .archive. You can manually download archived files. I keep archives for up to 365 days. Cyberduch can also be used to access.]

[We also have a register.csv file to record some details. You can create a softlink from /var/www/html to /data in order to download the file to you PC, using a password protected webpage.]

touch register.csv
touch register_block.csv

chmod 777 *

[Your SES rules should catch at least the following for the .ses subfolder:]
dmarc@mydomain.com
admin@
webmaster@
postmaster@

[and others you like, such as ME@, contact@ etc.] [and the .set SES rule should catch people you want to forward copies to, such as:] contact@
mail@

Your Content Goes Here

SES Email Forwading 2024

SES Email Forwading 2024

SES Email Forwading 2024

SES Email Forwading 2024

EC2 Menu

EC2 Menu

SES Email Forwading 2024

index.mjs Lambda function example

SES default_rule_set

A shell script to process the bucket files

Privacy & Cookies