In this issue

According to MessageLabs, 86.3% of all e-mails they filtered in June were spam, up from 75.2% in May.  MessageLabs filters emails for other companies so they have a  pretty good handle on it.  Other estimates vary from 65% upwards.

If we don't do something spam is going to kill email as a useful service.  What strategies do we have at present?

The filtering programs

Filtering programs read mails and on the basis of the content decide if it is likely to be spam or not.  They use various criteria such as the words, the colour and type of text, the number of images etc.  The end-result is a score, and you decide how high a score you are willing to accept.

I use spamassassin, but set the score fairly low because I don't want to risk losing real mails from real people.  The more paranoid you are the higher you set the score.  I also use the filtering in Outlook 2003 over the top of that. 

I analysed my mail for a 24 hour period:

Spam caught by spamassassin: 420 (68%)
Spam subsequently caught by Outlook 2003 59 (9%)
Bounced emails from people I never sent mail to: 45 (7%)
Spam manually removed by me: 39 (6%)
Genuine mail: 64 (10%)

So filtering is pretty good, but somehow manually removing spam still seems quite a chore.   

Why do so many mails advertising generic Viagra still get past spamassasin? 

• They are generally avertising something not quite Viagra - like V1agra
• They put information into graphics that Spamassassin can't read
• Then puff out the mail with random text to make it look like a regular mail.  

The spammers are getting cleverer month on month.

Blacklisting services

The idea is simple.  You get what you think is spam from bob@textor.com, you report it to a service, and they put textor.com on a blacklist.  Thereafter blacklist subscribers who get mail from anyone at textor.com will reject them.  There are several of these services and many large companies use them.

In my opinion these services are a menace:

1. They assume that what some punter thinks is spam really is spam.  Maybe said punter opted in to a mailing list and forgot, maybe he is totally paranoid and will classify anything he doesn't like as spam.   
2. They assume that mail that says it is from bob@textor.com is really from bob@textor.com.  In practice hard-core spammers rarely use their real email address.  In fact they often use email addresses of totally innocent third parties.  One of my clients found itself on such a list and found their mails blocked by a major client because a spammer somewhere decided to use their address. 
3. The people who run the blacklists are frequently (with some honourable exceptions) totally focused on spam as the biggest problem facing the world.  Forget war, pestilence, death and famine for a moment - they are unimportant, the horseman at the front is spam.    So a bit of collateral damage to innocent bystanders is completely acceptable to them.  Some blacklist ranges of internet addresses, even though the range may include completely innocent parties. 

If you find yourself on this sort of list, check their website and follow whatever procedure they have set up for removal. In extremis we have considered creating a whole new domain just for outgoing email.

Challenge-response

This sounds like a good idea.  If you send a mail you immediately get a mail back asking you to respond in some fairly simple way to prove that you really exist. Only when you have verified your existence does the original mail get through.  You only have to do this once and it is generally fairly painless.

This is a great idea for some people.  It wouldn't suite me because I don't want any sort of barrier between myself and prospective clients. 

Central filtering

If you could check the incoming mail for enough people, you could spot spam in a fairly foolproof way.  You would see patterns of mail that would point to a spam attack.  Also if you really focus on spam filtering as a task you can get very good at it. 

People like AOL, Hotmail, Yahoo and special services such as MessageLabs are in a good position to do this.   If you want it on your regular email then there is of course a cost attached.

Do we have the answer?

At present there is no complete answer.  However a powerful new tool is being developed which will be a major part of the solution. This is called email authentication.

What is authentication? 

The purpose of email authentication is to guarantee that mail from bob@textor.com really comes from me.  At present a mail stands on its own.  Inside the mail is some information you don't normally see that says who the mail came from and what route if used to get to you.  This can all be forged.

Email authentication requires that you have a database somewhere that email systems can check back with and say - Hey I have  a mail here routed via computer such and such from someone called bob@textor.com .  Could this have come from you?  Then the central database replies with - Yes he is one of my people, and we route mail through that computer so the mail is probably from him. (or her - computers being non-sexist)

There are different levels of authentication being worked on, but the most likely to be used in the short term is called Sender ID.  It is a collaborative effort between Microsoft and another team that developed an earler standard called SPF.  Sender ID is not by any means foolproof but is a big leap forward from where we are now.

What difference will Sender ID make?

• Filter programs like Spamassassin will give a huge negative weight to unauthenticated emails which by definition are suspicious
• Blacklists will actually work!
• Spammers can be tracked back to their source if they start using authenticated emails.

What do we need to do?

Nothing yet - it is early days.  But you will need to make sure your email is compliant in due course.  The database that holds routing information for mails is held in the same place as your domain.  So if you are joe@mydomain.com you will need to set the database up wherever mydomain.com is hosted and managed.  I will expand on this closer to the time when you will need to do something. 

The good news is that action is really being taken which should be very effective in combating spam.