Lawn Bowls

Cortex, where I work, is four years old and on Friday we celebrated by going to play lawn bowls (Diggers on the park). It was a change of pace from the usual Cortex social occasion. Our guide characterised lawn bowls as “low, slow, with the flow” and so it turned out to be.

I think the results were essentially random – too long, too short, too far to the right or too far to the left. It was lots of fun. I’d say it is a sport where it is easy to be OK and hard to be good. I’ll be ready to try it again in about 20 years

Good Names

One of the hardest things in software development is coming up with good names. Whether it is the name for a variable, a class or a project, a good name is really important to succinctly convery the role or purpose of the thing being named.

BTW, let me go on record to say that I don’t like the use of so-called "Hungarian" notation. For me names should be pronounceable (Pronounceable is a funny word – it is hardly pronounceable). lpszBlah does not work for me.

The reason I mention this is I am having trouble naming my little Java based alternative to fetchmail. I originally called it popper but there is already a tool by that name. My thoughts of a play on codefeed, "mailfeed", is also already in use. Is this like domain names – everything you can think of is already gone. I’ll have to sleep on it.

I just remembered that I once had to work on a system where the source files were part of a bill of materials and had to have part numbers. The part number was reflected in the file name – e.g. 123456.h. I’d rather have the problem of coming up with good names …

More on Spam

This is a follow up on my previous blog on spam blocking. Here I’ll detail my setup (pretty straight forward) and give some idea of the problems I had.

Stefan suggests I try Mozilla 1.3. I did know about it but I wanted to move my email delivery away from an all-in-one approach to one where I can control the processing more easily. If I think there is a better alternative to bogofilter, I want to be able to swap it in. If I want to add additional filtering, I want to be able to do that. Anyway, I think kMail is superior to Mozilla-Mail. I have also been experiencing crashes when browsing with Mozilla 1.2.1 and 1.3. It’s annoying when these take out your email too.

My system is a standard RedHat 8.0 system. I started by installing fetchmail. I’ve had some problems with fetchmail which I’ll detail further below. Configuring fetchmail is pretty easy. Fetchmail retrieves mail from my pop server and forwards it onto the local SMTP port. Sendmail picks this email up and forwards it to procmail. This is all standard on RedHat and I did not have to change any config there.

To invoke bogofilter, I needed a .procmailrc file. Stripped of its comments this file looks like this

PMDIR=$HOME/.procmail
LOGFILE=$PMDIR/pmlog
MAILDIR=$HOME/.procmail       # Make sure this directory exists!
INCLUDERC=$PMDIR/rc.bogo
INCLUDERC=$PMDIR/rc.main

The rc.bogo file contains procmail rules to run bogofilter. It looks like this

:0fw
| bogofilter -2 -u -e -p

# if bogofilter failed, return the mail to the queue, the MTA will
# retry to deliver it later
# 75 is the value for EX_TEMPFAIL in /usr/include/sysexits.h
:0e
{ EXITCODE=75 HOST }

# file the mail to spam-bogofilter if it's spam.
:0:
* ^X-Bogosity: Yes, tests=bogofilter
spam-bogofilter

This is pretty much from the bogofiklter docs. It runs bogofilter and causes it to classify mail as spam or not. It also updates the word lists (good and spam). Mail that is determined to be spam is placed into mbox file “spam-bogofilter”

The rc.main file is:

:0:
inbox

This just transfers mail from my system spool file to a file, “inbox”, in the .procmail directory. The next step is to configure kMail to pick up its email from these mbox files. Each mbox file, the inbox and spam files, is added as a receiving source with procmail locking. There is a lot of warnings in the documentation about locking mail files in kMail. It is important that the procmail rules use lock files, which the above do. kMail can send each receiving source into a separate folder which I do.

Fetchmail problems.

The problems I have had with Fetchmail come about mostly because I access my POP mailbox from both my work system and my home system. For this setup to work I leave mail on the server when it is downloaded. I like to have access to complete threads on each of my systems. Unfortunately I found that fetchmail would not download all mail on both systems. It seems to decide that some mail has already been read and not download it. If I direct it to fetch all mail, it would duplicate some email even though I have asked it to use UIDL to avoid that.

Also, I need to clean up the old mail to stop my POP account from overflowing. I do that by manually flushing the account. I miss the Outlook feature that allows you to delete files which are over a certain age. According to the fetchmail FAQ this feature is often requested but won’t be implemented as it is apparently out of fetchmail’s scope.

Faced with these problems, I have written a small Javamail based utility to perform the role of fetchmail with a few added features. I called it “popper” and it seems to be working quite nicely. I’ll blog about that in a few day’s time.

So, my email system now works as follows. Popper pops the mail from my POP servers and passes it to my local SMTP server, which is sendmail. Sendmail invokes procmail. Procmail invokes bogofilter to classify the email and finally kMail fetches the processed email and allows me to at last read it. It probably sounds more complicated than it is but it’s better than wading through spam.

My First Program

Matt blogs about the the first program he ever wrote. His reverie has prompted me to relate the story of my own first program and early programming experiences.

My first program was written in BASIC in 1979 and executed on my brother’s Unix account on a PDP-11. He had given me some class notes he had for BASIC as part of his Chemical Engineering course. I was absorbed and wrote the progam on paper, executing it with paper virtual variables and writing it out many times. The program was, of course, a game – a ballistic cannon ball thing – enter the amount of powder and the angle to hit a target at a random distance. It took a while before I got access to the actual BASIC interpreter. It worked first go which was pretty cool.

I was hooked, then and there. My parents spent a fortune to buy a Tandy Model 1 TRS-80. Thinking of the machine’s specs now is almost funny (1 MHz 8 bit processor, 16k of RAM, 1k video RAM, 128 x 48 graphics res, cassette tape storage). Yet, as pathetic as it now seems, I would learn an immense amount from getting to know that machine.

I remember using POKE and VARPTR in a program to perform a mysterious operation called a “left shift”. I was disappointed to find it just multiplied by 2 – I could do that in BASIC already. Later I would get into hardware. through the machine’s expansion port and a 40-way edge connector. I remember slowing down programs by pulling the HALT line low for a split second. If you left it too long, it would blank the DRAM since the refresh also stopped, causing a crash.

I’m glad I started with such humble stuff because I squeezed the most I could from it. It was simple enough to understand everything about it – something which is probably not true today. I even still like to occasionally write assembler programs.

From that day I have used a variety of machines, rarely the most powerful available. The idea is to make the most of what you’ve got, not to covet the latest and greatest available. I did that once – anyone want a PowerMac 8100/80AV?

Disclaimer: I may be off on the MHz values, etc – all a bit hazy now.

Spam blocking

I’ve recently been playing with my email system in an attempt to reduce spam.

Unfortunately I’m inundated with spam. I have had the same email address for over 8 years now and in the early days I posted it about the place without much care. In particular I was a regular poster to Usenet, particularly the C++ and Mac news groups. As a result I get lots and lots of spam. It’s pretty annoying.

My work email address also appears in a lot of Apache Ant documentation and source code, so I get quite a few virus emails too. In fact, since these viruses spoof the sending email address I sometimes get virus email from myself.

I have gone so far as to set up a Spamcop account. It has been easy to use, has a nice webmail interface and is reasonably effective. It is not, however, effective enough for me even when I selected every available blacklist.

While I am happy to see spammers prosecuted, ultimately a technical solution is going to be a better answer for me than after-the-fact legal action (as if I could afford the time or money for that anyway). After reading Paul Graham’s article on Spam, I decided to update my email system. I selected Bogofilter as a easy to manage Bayesian filter.

Up to now I have always popped my email direct from the pop server into Mozilla Mail. To get Bogofilter into the chain that needed to change. I installed fetchmail to pop the mail which then sends it to sendmail/procmail. I configured procmail to invoke Bogofilter and separate out the spam.

Mozilla mail does not support local mail. Actually it does support it in some sort of half-arsed fashion but I didn’t want to risk that so I have switched to KDE’s KMail. I have to say I am wrapped with KMail. It has the easiest filter config setup I have ever used especially when you use lots of mailing lists. You can keep your email system fresh by setting expiry conditions on folders, multiple identities, etc.

So far it is working well to reduce spam. I have turned off the spamcop filters to give it the stress test. I think it is catching more spam than spamcop did so that’s good. I have setup two folders – one for false positives (none yet) and the other for false negatives (a few). Bogofilter is setup to update its word lists when it processes each email. I then occasionally tell Bogofilter to reverse its thinking on the false negatives.

I have run into some problems with fetchmail and the fact that I pop email from both work and home accounts. I have developed a javamail based alternative about which I will blog a bit later (It’s getting late …)