I was asked if there was a way to extract the FTP activity to be emailed to someone. The server had a typical xferlog, but the box was being used for shared hosting and the reports didn’t need to include results for all of the other sites.

I put together the following script to extract the activity and transpose it to a more friendlier output.

#!/bin/bash
 
if [[ -z $1 || -z $2 ]]
then
        echo "Usage: $0 <PATTERN> <XFERLOG>"
        exit 1
fi
 
if [ ! -f $2 ]
then
        echo "Log file does not exist"
        exit 1
fi
 
awk "/$1/"' {
        gsub(" d "," deleted ");
        gsub(" o "," downloaded ");
        gsub(" i "," uploaded ");
        printf("%s %s %s %s: %s@%s - %s %s\n",$1,$2,$3,$4,$14,$7,$12,$9);
     }' "$2"

Feel free to download it here. Make sure you chmod +x xfer-report.sh before you try to use it.

The first argument, PATTERN, is for the regex that awk will use to isolate the appropriate entries. The second argument, XFERLOG, is the path to the xferlog to be processed. For example, the following would match any entries containing “rugmonster.org” from the log at /var/log/xferlog.1:

./xfer-report.sh 'rugmonster\.org' /var/log/xferlog.1

The resulting output would then give you something like:

Fri Feb 27 20:23:00: user@12.34.56.78 - downloaded /path/to/file
|-------DATE------|       |SOURCE IP|   |-ACTION-| |----FILE---|

The results can be pretty long if there’s been a lot of FTP activity, but this was prompted as a result of some files be deleted by someone that shouldn’t have been only to be discovered too late to be restored from backup. The bigger lesson, of course, is that you should ensure only those that need access to your server have it and measure twice, cut once.

All you need to do is create a file with one address per line to be read in by the command.

cd /var/qmail/mailnames; \
IFS="@"; \ 
while read LINE ; \
do \
    ADDR=( ${LINE} ); \
    echo "${ADDR[0]}@${ADDR[1]}"; \
    find ${ADDR[1]}/${ADDR[0]}/Maildir/ \
        \( -path "*/cur/*" -o -path "*/new/*" \) \
        ! -type d -mtime +7 | \
        while read FILE; \
        do \
            DIR=$( dirname ${FILE} ); \
            mkdir -p /path/to/dst/${DIR}; \
            mv ${FILE} /path/to/dst/${DIR}; \
        done; \
done < /path/to/address_file

find /path/to/src/ -user foo -exec cp -a {} /path/to/dst/

…you end up with all of the files in /path/to/dst/ with the original directory structure flattened out.

I came up with the following to solve the problem:

find /path/to/src/ -user foo | while read FILE \
do \
    DIR=$( dirname ${FILE} ); \
    mkdir -p /path/to/dst/${DIR}; \
    cp -d --preserve=mode,ownership,timestamp ${FILE} /path/to/dst/${DIR}; \
done

Here’s a brief explanation. Since the find command returns the relative path of the matching item in it’s STDOUT, we can use the dirname command to get the original relative directory name. We then create the directory if it doesn’t exist, and finally do the copy.

Query: CREATE "Sent"
Reason Given: Invalid mailbox name.

To resolve the issue, I had to change some of the Squirrelmail folder settings in /etc/squirrelmail/config.php.

$default_folder_prefix          = 'mail/';
$trash_folder                   = 'INBOX.Trash';
$sent_folder                    = 'INBOX.Sent';
$draft_folder                   = 'INBOX.Drafts';

This was on a RHEL 5.3 box, so the config.php may be located at another path if you’re using another distro.

To begin, I setup the necessary DNS records for a test blacklist containing my laptop’s IP, 10.20.50.40.

40.50.20.10.rugmonster.org. 600	IN	A	127.0.0.2
40.50.20.10.rugmonster.org. 600	IN	TXT	"go away"

Using a VM running CentOS 5.3 with Plesk 8.6, I setup rugmonster.org as a DNSBL for incoming mail. I connected to TCP port 25 on the VM using telnet to simulate an SMTP exchange.

1.  rblsmtpd: 10.20.50.40 pid 8586: 451 go away
2.  220 rblsmtpd.local
3.  EHLO daniel-nb.home
4.  250 rblsmtpd.local
5.  AUTH PLAIN AGRhbmllbEBwbGVzazg2LmhvbWUAcGFzc3dvcmQ=
6.  451 go away

[1-2] Initial server response
[3] I send the reqired client identification
[4] Server responds, but doesn’t give any service extensions
[5] I tried to authenticate
[6] And the server

Once I looked at /etc/xinetd.d/smtp_psa, I realized why this was happening.

server_args  = -Rt0 /usr/sbin/rblsmtpd -r rugmonster.org
	/var/qmail/bin/relaylock /var/qmail/bin/qmail-smtpd
	/var/qmail/bin/smtp_auth /var/qmail/bin/true
	/var/qmail/bin/cmd5checkpw /var/qmail/bin/true

When the SMTP connection is made, rblsmtpd is handed the connection and does its DNSBL checking. It’s then responsible for either executing the next program (relaylock) if there’s no match, or handling the connection itself if it does find a match. It was rblsmtpd that I was interacting with above, which had no way of handling authenticiation, not qmail-smtpd.

I did find that you can patch qmail to use plugins thanks to qmail-spp and the . This solution would require qmail’s source to be patched and a custom compile done, which makes upgrading in the future much more complicated.

I also found spamdyke, “a drop-in connection-time filter for qmail…[that] does not require patching or recompiling qmail.” I also found a HOWTO for installing spamdyke and integrating it with Plesk. While I haven’t tried it, from what I saw on their site, spamdyke would be a great tool to ensure users can send mail properly while keeping spam out.

Finally, Plesk 9.x has included Postfix, which has no problem handling this sort of situation out of the box. I am making another assumption though, since I haven’t tested Plesk 9.x with Postfix yet. I’ve heard that Parallels hasn’t done the best job in with bringing Postfix into the mix. At work, there are only a handful of clients running Plesk 9 and Plesk 9.2.x seems to be less problematic than the 9.0.x releases, but I’m not giving any recommendations to upgrade yet.

I’ve used the following configuration for many systems that were being plagued by spam problems. In many cases, SpamAssassin was doing the job, but it was having to process so much junk that it was putting an amazing amount of load on the server. After adding this to the Postfix configuration, Postfix was able to reject mail before it came into the queue, thereby reducing the amount of mail that made it through to SpamAssassin.

Add to /etc/postfix/main.cf

smtpd_delay_reject = yes
smtpd_helo_required = yes
disable_vrfy_command = yes

smtpd_helo_restrictions =
	permit_mynetworks,
	reject_non_fqdn_helo_hostname,
	reject_invalid_helo_hostname,
	reject_unknown_helo_hostname,
	permit

smtpd_sender_restrictions =
	permit_sasl_authenticated,
	permit_mynetworks,
	reject_non_fqdn_sender,
	reject_unknown_sender_domain,
	permit

smtpd_recipient_restrictions =
	permit_mynetworks,
	permit_sasl_authenticated,
	reject_unauth_pipelining,
	reject_non_fqdn_recipient,
	reject_unknown_recipient_domain,
	reject_unauth_destination,
	reject_rbl_client bl.spamcop.net,
	reject_rbl_client zen.spamhaus.org,
	permit

On average, only about two or three spam messages a day actually get accepted for delivery, which SpamAssassin then handles appropriately. You can see some stats on how well this is working through my Postfix stats*.

* I just started tracking the stats, so they may not show much depending on when you’re getting here.

yum install httpd-devel
wget http://www.fastcgi.com/dist/mod_fastcgi-current.tar.gz
tar xzf mod_fastcgi-current.tar.gz
cd mod_fastcgi-
cp Makefile.AP2 Makefile

The next step will depend on your architecture:

ia-32

make top_dir=/usr/lib/httpd
make top_dir=/usr/lib/httpd install

ia-64

make top_dir=/usr/lib64/httpd
make top_dir=/usr/lib64/httpd install

You’ll find the module is now installed in /etc/httpd/modules. You can now add the configuration directive to load the module. I suggest adding the following to /etc/httpd/conf.d/fastcgi.conf to follow RHEL convention:

LoadModule fastcgi_module modules/mod_fastcgi.so

I may get around to creating an RPM for this, but that will be time permitting.

mysql -e 'show processlist\G' |\
egrep -b5 'Time: [0-9]{2,}' |\
grep 'Id:' |\
cut -d':' -f2 |\
sed 's/^ //' |\
while read id
do
    mysql -e "kill $id;"
done

This goes with the assumption that you’re MySQL authentication credentials are in ~/.my.cnf. You aren’t running this as root with no root password set, are you?