openec2 Article Description

Scripts – Part 2 – Bits and Pieces

Sort a .csv MS Excel spreadsheet by a column. Asserts no commas (,)  star (*), or tilde (~) characters in the row contents.
For example, an email list that needs a sort by name, or who is subscribed or not confirmed and so on.

The email list can have double quotes around each element, or no double quotes. Assumes no double quotes within an element’s content.

The -f option ignores upper and lower case.

Handy for plugins like MailPoet – e.g. bounced, subscribed, surname and so on.

vi csv.sh

#!/bin/sh
IFS=$'\n'
tail -n +2 $1|tr " " "*"|tr "\"" "~"|tr "," " "|sort -f -k$2|tr " " ","|tr "*" " "|tr "~" "\""
exit

[save and exit]
chmod 777 csv.sh

Use - e.g. sort on column 1

./csv.sh email.csv 1

This script will fix file ownerships and permissions for WordPress.You can modify as you wish. For instance, chown -R apache and so forth if using httpd or Apache2.

(For Debain, cd to /home/admin)
cd /home/ec2-user

vi chdir.sh

#!/bin/sh
chown -R nginx *
chgrp -R nginx *
find . -type d -exec chmod 2775 {} \;
find . -type f -exec chmod 0664 {} \;
# in case any hidden files:
chown nginx .??*
chgrp nginx .??*
chmod 664 .??*
chmod 777 *.sh
chown root chdir.sh
chgrp root chdir.sh
chmod 770 chdir.sh
exit

[save and exit]

chmod 777 chdir.sh; chown root chdir.sh; chgrp ec2-user chdir.sh

[You really only need execute permission from a root login]

[To run the script for WordPress files in your domain, for instance, under /var/www/html:]
cp -p chdir.sh /var/www/html
cd /var/www/html
./chdir.sh

This script is provided on the Internet for the public to use. It may block an IP address you do not wish, but generally is good.

One can use IP2 Location’s website to download specific country lists, or to get current versions, but obviously you should take some care around how many countries and which ones.

If the script fails, it would be due to being outdated. It does not take too long to execute is you are not using the “sh -x” option.

[Debian would be /home/admin]

[Replace mydomain.com in your own blacklists below, and as shown in the script, place the country files into your nominated domain URL directory]
[We will come back to more details in a moment]

cd /home/ec2-user

vi blacklist.sh

#!/bin/sh

# IP blacklisting script for Linux servers
# Pawel Krawczyk 2014-2015
# documentation https://github.com/kravietz/blacklist-scripts

# iptables logging limit
LIMIT="10/minute"

# try to load config file
# it should contain one blacklist URL per line

config_file="/etc/ip-blacklist.conf"
if [ -f "${config_file}" ]; then
    source ${config_file}
else
    # if no config file is available, load default set of blacklists
    # URLs for further blocklists are appended using the classical
    # shell syntax:  "$URLS new_url"

    # Emerging Threats lists offensive IPs such as botnet command servers
    URLS="http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt"

    # Blocklist.de collects reports from fail2ban probes, listing password brute-forces, scanners and other offenders
    URLS="$URLS https://www.blocklist.de/downloads/export-ips_all.txt"

    # MY OWN BLACKLIST(S)
 # URLS="$URLS https://mydomain.com/russia.txt"
 # URLS="$URLS https://mydomain.com/china.txt"
 # URLS="$URLS https://mydomain.com/iran.txt"
 # URLS="$URLS https://mydomain.com/turkey.txt"
 # URLS="$URLS https://mydomain.com/northkorea.txt"



    # iblocklist.com is also supported
    # URLS="$URLS http://list.iblocklist.com/?list=srzondksmjuwsvmgdbhi&fileformat=p2p&archiveformat=gz&username=USERNAMEx$&pin=PIN"
fi

link_set () {
    if [ "$3" = "log" ]; then
        iptables -A "$1" -m set --match-set "$2" src,dst -m limit --limit "$LIMIT" -j LOG --log-prefix "BLOCK $2 "
    fi
    iptables -A "$1" -m set --match-set "$2" src -j DROP
    iptables -A "$1" -m set --match-set "$2" dst -j DROP
}

# This is how it will look like on the server

# Chain blocklists (2 references)
#  pkts bytes target     prot opt in     out     source               destination         
#     0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set manual-blacklist src,dst limit: avg 10/min burst 5 LOG flags 0 level 4 prefix "BLOCK manual-blacklist "
#     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set manual-blacklist src,dst
#     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set rules.emergingthreats src
#     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set rules.emergingthreats dst
#     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set www.blocklist.de src
#     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set www.blocklist.de dst
#     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set www.badips.com src
#     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set www.badips.com dst
blocklist_chain_name=blocklists

# check for dependencies - ipset and curl
if [ -z "$(which ipset 2>/dev/null)" ]; then
    echo "Cannot find ipset"
    echo "Run \"apt-get install ipset\" (Debian/Ubuntu) or \"yum install ipset\" (RedHat/CentOS/Fedora) or \"opkg install ipset\" (OpenWRT/LEDE)"
    exit 1
fi
if [ -z "$(which curl 2>/dev/null)" ]; then
    echo "Cannot find curl"
    echo "Run \"apt-get install curl\" (Debian/Ubuntu) or \"yum install curl\" (RedHat/CentOS/Fedora) or \"opkg install curl\" (OpenWRT/LEDE)"
    exit 1
fi

# check if we are on OpenWRT
if [ "$(which uci 2>/dev/null)" ]; then
    # we're on OpenWRT
    wan_iface=pppoe-wan
    IN_OPT="-i $wan_iface"
    INPUT=input_rule
    FORWARD=forwarding_rule
    COMPRESS_OPT=""
else
    COMPRESS_OPT="--compressed"
    INPUT=INPUT
    FORWARD=FORWARD
fi

# create main blocklists chain
if ! iptables -nL | grep -q "Chain ${blocklist_chain_name}"; then
    iptables -N ${blocklist_chain_name}
fi

# inject references to blocklist in the beginning of input and forward chains
if ! iptables -nL ${INPUT} | grep -q ${blocklist_chain_name}; then
  iptables -I ${INPUT} 1 ${IN_OPT} -j ${blocklist_chain_name}
fi
if ! iptables -nL ${FORWARD} | grep -q ${blocklist_chain_name}; then
  iptables -I ${FORWARD} 1 ${IN_OPT} -j ${blocklist_chain_name}
fi                                                                 

# flush the chain referencing blacklists, they will be restored in a second
iptables -F ${blocklist_chain_name}

# create the "manual" blacklist set
# this can be populated manually using ipset command:
# ipset add manual-blacklist a.b.c.d
set_name="manual-blacklist"
if ! ipset list | grep -q "Name: ${set_name}"; then
    ipset create "${set_name}" hash:net
fi
link_set "${blocklist_chain_name}" "${set_name}" "$1"

# download and process the dynamic blacklists
for url in $URLS
do
    # initialize temp files
    unsorted_blocklist=$(mktemp)
    sorted_blocklist=$(mktemp)
    new_set_file=$(mktemp)
    headers=$(mktemp)

    # download the blocklist
    set_name=$(echo "$url" | awk -F/ '{print substr($3,0,21);}') # set name is derived from source URL hostname
    curl -L -v -s ${COMPRESS_OPT} -k "$url" >"${unsorted_blocklist}" 2>"${headers}"

    # this is required for blocklist.de that sends compressed content regardless of asked or not
    if [ -z "$COMPRESS_OPT" ]; then
        if grep -qi 'content-encoding: gzip' "${headers}"; then
            mv "${unsorted_blocklist}" "${unsorted_blocklist}.gz"
            gzip -d "${unsorted_blocklist}.gz"
        fi
    fi
    # autodetect iblocklist.com format as it needs additional conversion
    if echo "${url}" | grep -q 'iblocklist.com'; then
        if [ -f /etc/range2cidr.awk ]; then
            mv "${unsorted_blocklist}" "${unsorted_blocklist}.gz"
            gzip -d "${unsorted_blocklist}.gz"
            awk_tmp=$(mktemp)
            awk -f /etc/range2cidr.awk <"${unsorted_blocklist}" >"${awk_tmp}"
            mv "${awk_tmp}" "${unsorted_blocklist}"
        else
            echo "/etc/range2cidr.awk script not found, cannot process ${unsorted_blocklist}, skipping"
            continue
        fi
    fi

    sort -u <"${unsorted_blocklist}" | egrep "^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}(/[0-9]{1,2})?$" >"${sorted_blocklist}"

    # calculate performance parameters for the new set
    if [ "${RANDOM}" ]; then
        # bash
        tmp_set_name="tmp_${RANDOM}"
    else
        # non-bash
        tmp_set_name="tmp_$$"
    fi
    new_list_size=$(wc -l "${sorted_blocklist}" | awk '{print $1;}' )
    hash_size=$(expr $new_list_size / 2)

    if ! ipset -q list ${set_name} >/dev/null ; then
        ipset create ${set_name} hash:net family inet
    fi

    # start writing new set file
    echo "create ${tmp_set_name} hash:net family inet hashsize ${hash_size} maxelem ${new_list_size}" >>"${new_set_file}"

    # convert list of IPs to ipset statements
    while read line; do
        echo "add ${tmp_set_name} ${line}" >>"${new_set_file}"
    done <"$sorted_blocklist" # replace old set with the new, temp one - this guarantees an atomic update echo "swap ${tmp_set_name} ${set_name}" >>"${new_set_file}"

    # clear old set (now under temp name)
    echo "destroy ${tmp_set_name}" >>"${new_set_file}"

    # actually execute the set update
    ipset -! -q restore < "${new_set_file}"
    
    link_set "${blocklist_chain_name}" "${set_name}" "$1"

    # clean up temp files
    rm "${unsorted_blocklist}" "${sorted_blocklist}" "${new_set_file}" "${headers}"
done


[save and exit]

chmod 777 blacklist.sh; chown root blacklist.sh; chgrp ec2-user blacklist.sh

Download a working set of country files from this link, or create your own files:

countries.tar

Now that you are ready to run the script – with any of your URL files in place, install the dnf packages (or apt packages in Debian).

Simply try dnf install ipset iptables. I think from memory you only need to install one of these on Debian.

Run the script:

/home/ec2-user/blacklist.sh
iptables -L -vn

We will now enhance the scripting with limit.sh to help stopping anyone who opens a port without closing it, so that they can then flood the server until it freezes:

cd /home/ec2-user

vi limits.sh

#!/bin/sh
iptables -A INPUT -p tcp --syn --dport 443 -m connlimit --connlimit-above 5 -j DROP
iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 2 -j DROP
iptables -L -vn
exit

[save and exit]

chown root limits.sh; chgrp ec2-user limits.sh; chmod 777 limits.sh;

./limits.sh

This is a crucial shell script if using email ports such as port 587 in postfix. You must have it. The values I use in limits.sh is kind of arbitrary. I do not know the best values to use but they seem to be okay so far.

*** These rules are cleared if you reboot the server. You have to manually run the scripts. I have not developed an automated way to do this so far. ***

How to refresh the blacklists if you modify the IP addresses. For instance, you may find some viscous IP address attacks on WordPress. We cannot continually edit our files as there are so many hacking attempts out there. But if you need to:

cd //home/ec2-user

vi clear.sh

#!/bin/bash
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -L -vn
exit

[save and exit]

[Change the ownerships and permissions of clear.sh as per normal practice, examples shown above on other scripts/]

Note: you can play around with /binsh or /bin/bash as you please. There are differences between Linux 2023 and Debian.
In fact, crontab in Debian may need an entry that says "... /usr/bin/bash my_script.sh ..." to execute a bash script. Handy to keep this in mind.

When you run the script, you see there are no blacklists remaining. Simply re-run blacklist.sh and limits.sh to get the rules back. Handy if you want to add a new IP address for WordPress concerns or another country, etc. This way you do not need to reboot the server.

If the free -m space becomes overloaded, you can run a crontab script each night.

cd /home/ec2-user
vi services.sh

#!/bin/sh
g=350
f=0
h=`free -m|grep Swap|awk '{print $3}'`
f=$(expr $h + 1)
if [ $f -le $g ] ; then
 d=`date`
 echo services.sh: date: $d freespace $h >> /home/ec2-user/info.log
else
 d=`date`
 /usr/bin/systemctl stop nginx
 /usr/bin/systemctl stop mariadb
 /usr/bin/systemctl stop memcached
 /usr/sbin/swapoff -a
 /usr/sbin/swapon -a
 /usr/bin/systemctl reload php8.2-fpm
 /usr/bin/systemctl start mariadb
 /usr/bin/systemctl start memcached
 /usr/bin/systemctl start nginx

# if using axigen on Debian 11:
# /usr/bin/systemctl restart spamassassin
# /usr/bin/systemctl restart axigen

k=`free -m|grep Swap|awk '{print $3}'`
echo services.sh: date: $d freespace before: $h freespace after: $k>> /home/ec2-user/info.log
fi
exit

[save and exit]

chmod 777 services.sh; chown root services.sh; chgrp ec2-user services.sh

crontab -e

0 0 * * * /home/ec2-user/services.sh

[save and exit]

You nominate the free -m value you are comfortable with. Thins slow down too much in my view after 300. Purchasing higher resources of course is necessary if they are overloaded.

This is helpful if making edits that benefit from a restart of all your services. Very convenient.

This example as with others is using php8.2 for a php-fpm restart, but Linux 2023 only needs to specify php-fpm.

#!/bin/sh
 /usr/bin/systemctl stop nginx
 /usr/bin/killall nginx
 /usr/bin/systemctl stop mariadb
 /usr/bin/systemctl stop memcached
 /usr/bin/systemctl reload php8.2-fpm
 /usr/bin/systemctl start memcached
 /usr/bin/systemctl start mariadb
 /usr/bin/systemctl start nginx
exit

# if using axigen, you can have a script with this as well for midnight
# It is not good to have axigen on the same server as other services if you are stopping and starting all the time.
# It is better to purchase an inexpensive service from VentraIP for a few dollars a month
/usr/bin/systemctl stop spamassassin
/usr/bin/systemctl stop axigen
/usr/bin/systemctl start spamassassin
/usr/bin/systemctl start axigen

exit

[save and exit, and set permissions]

The killall nginx line may not be needed. I put it in, just in case.

This script resets swap space:

vi swap.sh
#!/bin/sh
 d=`date`
 h=`free -m|grep Swap|awk '{print $3}'`
 /usr/sbin/swapoff -a
 /usr/sbin/swapon -a
 k=`free -m|grep Swap|awk '{print $3}'`
 echo services.sh: date: $d freespace before: $h freespace after: $k>> /home/ec2-user/info.log
exit

[save and exit, and set permissions]