I'm a bleeding edge kind of guy. No, I'm not. When it comes to software, I'm generally a bleeding edge kind of guy. I really like Chromium. I hate going to the Chromium build site, downloading it, closing my browser, then moving it in place. I'm a lazy admin.

Here's a couple quick-n-dirty bash script that retrieves the latest Chromium build.

Mac Version

Special Notes:

#!/bin/bash
echo -n "Retrieving latest build: "
VER=$( curl -s http://build.chromium.org/f/chromium/snapshots/chromium-rel-mac/LATEST )
echo "$VER"  
 
TMPDIR=$( mktemp -d /tmp/chromium.XXXXX )  
 
echo "Downloading Chromium: "
curl -o ${TMPDIR}/chromium.zip http://build.chromium.org/f/chromium/snapshots/chromium-rel-mac/${VER}/chrome-mac.zip  
 
echo -n "Decompressing archive: "  
unzip -qqx ${TMPDIR}/chromium -d ${TMPDIR}  
echo "done"  
 
rm -rf /Applications/Chromium.app  
mv ${TMPDIR}/chrome-mac/Chromium.app /Applications/  
 
rm -rf ${TMPDIR}  
echo "New Chromium installed!"

Linux Version

Special Notes:

#!/bin/bash
echo -n "Retrieving latest build: "
VER=$( curl -s http://build.chromium.org/f/chromium/snapshots/chromium-rel-linux-64/LATEST )
echo "$VER"  
 
TMPDIR=$( mktemp -d /tmp/chromium.XXXXX )  
 
echo "Downloading Chromium: "
curl -o ${TMPDIR}/chrome-linux.zip http://build.chromium.org/f/chromium/snapshots/chromium-rel-linux-64/${VER}/chrome-linux.zip  
 
echo -n "Decompressing archive: "  
7z x -tzip -o${TMPDIR} ${TMPDIR}/chrome-linux.zip > /dev/null
echo "done"  
 
if [ -d /opt/chrome-linux.1 ]
then
        rm -rf /opt/chrome-linux.1
fi
 
if [ -d /opt/chrome-linux ]
then
        mv /opt/chrome-linux /opt/chrome-linux.1 
fi
 
mv ${TMPDIR}/chrome-linux /opt/ 
find /opt/chrome-linux -type d -exec chmod 2775 {} \;
find /opt/chrome-linux -type f -perm /100 -exec chmod 775 {} \;
find /opt/chrome-linux -type f ! -perm /100 -exec chmod 664 {} \;
chgrp -R opt /opt/chrome-linux
 
rm -rf ${TMPDIR}  
echo "New Chromium installed!"

Enjoy!

8 Nov 2010: Updated to match new URLs being used

Warning: It looks like your PortIndex file for rsync://rsync.macports.org/release/ports/ may be corrupt.

I couldn't find a clear answer on what was going on, until I found a bug report on the MacPorts site. The fix for this will be put into MacPorts 1.9.0, but I'm on 1.8.2 now, so that doesn't help much.

The issue sounds like it has to do with a cache of the PortIndex can build up stale information. My brute force resolution was to remove that cache. I wasn't terribly sure where that was, but I did find it.

$ locate PortIndex.quick
/opt/local/var/macports/sources/rsync.macports.org/release/ports/PortIndex.quick
$ sudo rm /opt/local/var/macports/sources/rsync.macports.org/release/ports/PortIndex.quick

Alternatively, you could move the file out of the way if that makes you more comfortable.

Once that was done, I went ahead and gave it try.

$ sudo port sync
Warning: No quick index file found, attempting to generate one for source: rsync://rsync.macports.org/release/ports/
$ sudo port upgrade outdated
...

All worked as it should! So until we see MacPorts 1.9.0, this will be my solution should I see this come up again.

What?!

So I dug in my mail server's logs and found that SpamCop had listed some of the Facebook mail servers on their DNS Blacklist.

Example from today:
May 4 18:49:14 sh-srv2 postfix/smtpd[28197]: NOQUEUE: reject: RCPT from outmail006.snc1.tfbnw.net[69.63.178.165]: 554 5.7.1 Service unavailable; Client host [69.63.178.165] blocked using bl.spamcop.net; Blocked - see http://www.spamcop.net/bl.shtml?69.63.178.165; from=<notification+mu_miidm@facebookmail.com> to=<daniel@*don't spam me*> proto=ESMTP helo=<mx-out.facebook.com>


As you can see, Postfix did what it was supposed to do according to my configuration. The problem is, apparently, Facebook doesn't check the actual server responses and assumes that if a message is undeliverable for any reason, the address is no longer valid. Over the past month, I've had 101 messages from Facebook mail servers blocked based on the listing from SpamCop. I can't tell you how many times I've had to click the "reconfirm your current email" link to get the message to go away.

As most experienced Postfix admins know, you can always manipulate Postfix's access control mechanisms just about any way you can imagine. If you're an experienced Postfix admin, you can probably move along since you probably already know what I'm going to explain from here.

I don't know why, but today was the final straw. Once again, SpamCop had listed one of the Facebook mail relays and a message had been rejected. While I had tolerated reconfirming my address before, today I decided that this could be fixed and I was going to share it with the world.

In looking at the server names for all of the blocked Facebook servers, they had a standard naming convention of outmail###.snc1.tfbnw.net. Obviously, tfbnw.net stands for The Facebook Network. My guess is that snc1 is an identifier for datacenter or something along those lines. I figured that it would be safe enough to whitelist all subdomains of tfbnw.net.

First, I created a new file I called /etc/postfix/rbl_whitelist, which contains the following lines:

# Facebook
/.*.tfbnw.net/ OK


In the first field, I have a regular expression, /.*.tfbnw.net, which should match anything sent from tfbnw.net domain and any subdomain. The second field specifies what action Postfix should take when it encounters that pattern. Easy enough. Since I'm using a regexp type table, there's no need to use postmap.

Next, I had to tell Postfix that it needed to use this file to check for "client" access, or remote servers sending mail destined for my server. Since Postfix evaluates smtpd_client_restrictions in order they are defined, I simply needed to drop check_client_access hash:/etc/postfix/wl_servers in before my other restrictions. This made my smtpd_recipient_restrictions look like this:

smtpd_client_restrictions =
     permit_mynetworks,
     permit_sasl_authenticated,
     check_client_access regexp:/etc/postfix/rbl_whitelist,
     reject_unauth_pipelining,
     reject_rbl_client bl.spamcop.net,
     reject_rbl_client zen.spamhaus.org,
     permit

This causes Postfix to catch any thing sent from a Facebook mail relay and accept it before any DNSBL checks are done. Problem solved.

Updated 20 May 2010: I realized that having the check_client_access in the smtpd_recipient_restrictions was not giving the desired results. The check_client_access was being ignored since it was really in the wrong spot. Once I realized that, I also figured out that the reject_rbl_client directives could be moved under smtpd_client_restrictions as well.

Updated 29 May 2010: I got prompted again to confirm my address. I checked if the inbound hostnames were actually matching on the hash table I was previously using, but it was not, so I moved it to a regexp table instead. I couldn't figure out why the hash wasn't working as was expected. Perhaps due to the multiple levels of subdomains off of tfbnw.net, but the regexp works fine.

My IMAP Folders

I use IMAP folders combined with fairly extensive Sieve rules to sort my incoming mail into the appropriate folders. This works great for me, because my Facebook notifications go to "Websites", bill notifications go into "Bills". It makes it easy for me to find what I'm looking for without having to sort through a single, massive Inbox.

When I got my iPhone, I was very disappointed in its handling of folders. It simply lacks the ability to check all folders for new mail. I didn't want to have to click through each folder to see if I got new mail in one of them. I wanted to be able to see my little mail badge with the number of new messages I had, when I got a new message, no matter what folder it happened to reside in. I thought of various solutions, such as writing a server-side IMAP proxy that would do the checking for me and present a virtual mailbox of only new messages. While it may have been a fun project, a friend pointed out a more obvious solution: setup a second mailbox and have all of my mail forwarded to it. Here are the details of what I setup.

I created an account named "iphone" on my mail server. While I could have just added a .forward in my primary user's home directory and called it done, I didn't want to get everything sent to the mailbox on my iPhone. For example, SpamAssassin is setup to flag spam, and while I keep a "Spam" folder to periodically go back through for false-positives, I am not interested in seeing those on my iPhone. Also, I have several servers sending me logs occasionally. Again, I don't care to see those on my iPhone. To keep the unnecessary stuff from being forwarded on, I implemented some rules to exclude the unwanted messages and forward the rest.

Here is an example for handling the messages marked as spam from my ~/.dovecot.sieve file:

require ["imapflags","fileinto","copy"];

if  header :is "X-Spam-Flag" "YES" {
    addflag "\Seen";
    fileinto "Spam";
    stop;
} else {
    redirect :copy "iphone@rugmonster.org";
}

I put all of the rules for things I don't want to be sent to the iPhone's mailbox before the "redirect" statement.

I happen to like Sieve for setting up mail delivery rules, but this could also be accomplished with procmail if your local delivery agent doesn't support Sieve. You could also do this with any mail client that supports mail processing rules.

To prevent mail from being sent directly to the iPhone's mailbox, I added a recipient check in my Postfix config. First, I created a file I called "denied_recipients" with the following:

iphone@	REJECT	This address doesn't accept mail directly

I updated my smtpd_recipient_restrictions so Postfix would check that file. It looks like this now:

smtpd_recipient_restrictions =
     permit_mynetworks,
     permit_sasl_authenticated,
     check_recipient_access hash:/etc/postfix/denied_recipients,
     reject_unauth_pipelining,
     reject_non_fqdn_recipient,
     reject_unknown_recipient_domain,
     reject_unauth_destination,
     reject_rbl_client bl.spamcop.net,
     reject_rbl_client zen.spamhaus.org,
     permit

Since mail is being forwarded locally, everything gets through perfectly fine. And because I'm not accepting mail for that address, I can comfortably put iphone@rugmonster.org up on here and not worry about spammers sending me a bunch of crap.

While that all works great, I did one more thing to make things a bit easier for myself. I don't always delete messages out of the mailbox on the iPhone on a regular basis. In fact, I would commonly delete the messages straight out of the maildir from the shell when the mailbox got too big. To keep the mailbox fairly tidy, I've setup a cronjob to clean things up for me.

0 0 * * * find /home/iphone/Maildir/ ! -type d -type f
-path "*/cur/*" -daystart -ctime +1 -exec rm -f {} ; 2> /dev/null/

Note that the newline is only for formatting on this page. cron does not permit commands to be split across multiple lines.

This just goes through the iphone mailbox's Maildir and removes and messages that are older than 1 day. It gets not only the messages in the inbox, but also anything that I may have manually deleted. This isn't necessary by any means, but I'm lazy. What sys admin isn't, though.

I was asked if there was a way to extract the FTP activity to be emailed to someone. The server had a typical xferlog, but the box was being used for shared hosting and the reports didn't need to include results for all of the other sites.

I put together the following script to extract the activity and transpose it to a more friendlier output.

#!/bin/bash
 
if [[ -z $1 || -z $2 ]]
then
        echo "Usage: $0 <PATTERN> <XFERLOG>"
        exit 1
fi
 
if [ ! -f $2 ]
then
        echo "Log file does not exist"
        exit 1
fi
 
awk "/$1/"' {
        gsub(" d "," deleted ");
        gsub(" o "," downloaded ");
        gsub(" i "," uploaded ");
        printf("%s %s %s %s: %s@%s - %s %s\n",$1,$2,$3,$4,$14,$7,$12,$9);
     }' "$2"

Feel free to download it here. Make sure you chmod +x xfer-report.sh before you try to use it.

The first argument, PATTERN, is for the regex that awk will use to isolate the appropriate entries. The second argument, XFERLOG, is the path to the xferlog to be processed. For example, the following would match any entries containing "rugmonster.org" from the log at /var/log/xferlog.1:

./xfer-report.sh 'rugmonster\.org' /var/log/xferlog.1

The resulting output would then give you something like:

Fri Feb 27 20:23:00: user@12.34.56.78 - downloaded /path/to/file
|-------DATE------|       |SOURCE IP|   |-ACTION-| |----FILE---|

The results can be pretty long if there's been a lot of FTP activity, but this was prompted as a result of some files be deleted by someone that shouldn't have been only to be discovered too late to be restored from backup. The bigger lesson, of course, is that you should ensure only those that need access to your server have it and measure twice, cut once.