
=======================================
Integrating Spambayes with mail systems
=======================================

General
-------

Spambayes is a tool used to segregate unwanted mail (spam) from the mail you
want (ham).  Before Spambayes can be your spam filter of choice you need to
train it on representative samples of email you receive.  After it's been
trained, you use Spambayes to classify new mail according to its spamminess
and hamminess qualities.

To train Spambayes (which you don't need to do if you're going to be using
the POP3 proxy to classify messages, but you'll get better results from
the outset if you do) you need to save your incoming email for awhile,
segregating it into two piles, known spam and known ham (ham is our nickname
for good mail).  It's best to train on recent email, because your interests
and the nature of what spam looks like change over time.  Once you've
collected a fair portion of each (anything is better than nothing, but it
helps to have a couple hundred of each), you can tell Spambayes, "Here's my
ham and my spam".  It will then process that mail and save information about
different patterns which appear in ham and spam.  That information is then
used during the filtering stage.  See the "Command-line training" section
below for details.

When Spambayes filters your email, it compares each unclassified message
against the information it saved from training and makes a decision about
whether it thinks the message qualifies as ham or spam, or if it's unsure
about how to classify the message.  It adds its classification to the message
by adding a header, X-Spambayes-Classification: spam|ham|unsure.  You can
then filter on this header, to file away suspected spam into its own mail
folder for example.

In the sections below, are gathered notes about how Spambayes can be
integrated into your mail processing system.


Requirements
------------

As a general requirement, you must have a recent version of Python
installed on your computer, version 2.2 or later.  Python 2.2.3 is
recommended.  (Don't ask about backporting it to earlier versions of
Python.  It's almost a certainty this won't happen.)  If you need to
install Python on your system, check the Python download page for the
version appropriate to your computer:

    http://www.python.org/download/

You also need version 2.4.3 or above of the Python "email" package.
If you're running Python 2.2.3 or Python 2.3 (which at the time of
writing is available from SourceForge CVS, or as the beta version
2.3b1, available from python.org) then you already have a good version
of the email package.

If not, you can download email version 2.5 (as of this writing) from
http://mimelib.sf.net and install it - unpack the archive, cd to the
email-2.5 directory and type "python setup.py install" (YMMV on
different platforms).  This will install it into your Python
site-packages directory.  You'll also need to move aside the standard
"email" library - go to your Python "Lib" directory and rename "email"
to "email_old".


Overview
--------

There are eight main components to the Spambayes system:

 o A database.  Loosely speaking, this is a collection of words and
   associated spam and ham probabilities.  The database says "If a message
   contains the word 'Viagra' then there's a 98% chance that it's spam, and
   a 2% chance that it's ham."  This database is created by training - you
   give it messages, tell it whether those messages are ham or spam, and it
   adjusts its probabilities accordingly.  How to train it is covered
   below.  By default it lives in a file called "hammie.db".

 o The tokeniser/classifier.  This is the core engine of the system.  The
   tokenizer splits emails into tokens (words, roughly speaking), and the
   classifier looks at those tokens to determine whether the message looks
   like spam or not.  You don't use the tokeniser/classifier directly -
   it powers the other parts of the system.

 o The POP3 proxy.  This sits between your email client (Eudora, Outlook
   Express, etc) and your incoming email server, and adds the
   classification header to emails as you download them.  A typical
   user's email setup looks like this:

       +-----------------+                              +-------------+
       | Outlook Express |      Internet or intranet    |             |
       |  (or similar)   | <--------------------------> | POP3 server |
       |                 |                              |             |
       +-----------------+                              +-------------+

   The POP3 server runs either at your ISP for internet mail, or somewhere
   on your internal network for corporate mail.  The POP3 proxy sits in the
   middle and adds the classification header as you retrieve your email:

       +-----------------+        +------------+        +-------------+
       | Outlook Express |        | Spambayes  |        |             |
       |  (or similar)   | <----> | POP3 proxy | <----> | POP3 server |
       |                 |        |            |        |             |
       +-----------------+        +------------+        +-------------+

   So where you currently have your email client configured to talk to
   say, "pop3.my-isp.com", you instead configure the *proxy* to talk to
   "pop3.my-isp.com" and configure your email client to talk to the proxy.
   The POP3 proxy can live on your PC, or on the same machine as the POP3
   server, or on a different machine entirely, it really doesn't matter.
   Say it's living on your PC, you'd configure your email client to talk
   to "localhost".  You can configure the proxy to talk to multiple POP3
   servers, if you have more than one email account.

 o The SMTP proxy.  This sits between your email client (Eudora, Outlook
   Express, etc) and your outgoing email server.  Any mail sent to
   spambayes_spam@localhost or spambayes_ham@localhost is intercepted
   and trained appropriately.  A typical user's email setup looks like
   this:

       +-----------------+                              +-------------+
       | Outlook Express |      Internet or intranet    |             |
       |  (or similar)   | <--------------------------> | SMTP server |
       |                 |                              |             |
       +-----------------+                              +-------------+

   The SMTP server runs either at your ISP for internet mail, or somewhere
   on your internal network for corporate mail.  The SMTP proxy sits in the
   middle and checks for mail to train on as you send your email:

       +-----------------+        +------------+        +-------------+
       | Outlook Express |        | Spambayes  |        |             |
       |  (or similar)   | <----> | SMTP proxy | <----> | SMTP server |
       |                 |        |            |        |             |
       +-----------------+        +------------+        +-------------+

   So where you currently have your email client configured to talk to
   say, "smtp.my-isp.com", you instead configure the *proxy* to talk to
   "smtp.my-isp.com" and configure your email client to talk to the proxy.
   The SMTP proxy can live on your PC, or on the same machine as the SMTP
   server, or on a different machine entirely, it really doesn't matter.
   Say it's living on your PC, you'd configure your email client to talk
   to "localhost".  You can configure the proxy to talk to multiple SMTP
   servers, if you have more than one email account.

 o The web interface.  This is a server that runs alongside the POP3 proxy,
   SMTP proxy, and IMAP filter (see below) and lets you control it through
   the web.  You can upload emails to it for training or classification,
   query the probabilities database ("How many of my emails really *do*
   contain the word Viagra"?), find particular messages, and most
   importantly, train it on the emails you've received.  When you start
   using the system, unless you train it using the Hammie script it will
   classify most things as Unsure, and often make mistakes.  But it keeps
   copies of all the email's its seen, and through the web interface you
   can train it by going through a list of all the emails you've received
   and checking a Ham/Spam box next to each one.  After training on a few
   messages (say 20 spams and 20 hams), you'll find that it's getting it
   right most of the time.   The web training interface automatically
   checks the Ham/Spam boxes according to what it thinks, so all you need
   to do it correct the odd mistake - it's very quick and easy.

 o The Outlook plug-in.  For Outlook 2000 and Outlook XP users (not Outlook
   Express) this lets you manage the whole thing from within Outlook.  You
   set up a Ham folder and a Spam folder, and train it simply by dragging
   messages into those folders.  Alternatively there are buttons to do the
   same thing. And it integrates into Outlook's filtering system to make it
   easy to file all the suspected spam into its own folder, for instance.

 o The Hammie script.  This does three jobs: command-line training,
   procmail filtering, and XML-RPC.  See below for details of how to use
   Hammie for training, and how to use it as procmail filter.  Hammie can
   also run as an XML-RPC server, so that a programmer can write code that
   uses a remote server to classify emails programmatically - see
   hammiesrv.py.

 o The IMAP filter.  This is a cross between the POP3 proxy and the Outlook
   plugin.  If your mail sits on an IMAP server, you can use the this to
   filter your mail.  You can designate folders that contain mail to train
   as ham and folders that contain mail to train as spam, and the filter
   does this for you.  You can also designate folders to filter, along with
   a folder for messages Spambayes is unsure about, and a folder for
   suspected spam. When new mail arrives, the filter will move mail to the
   appropriate location (ham is left in the original folder).

Where things live
-----------------

The Hammie script is called hammie.py.  The POP3 proxy lives in pop3proxy.py,
and the smtpproxy lives in smtpproxy.py.  The IMAP filter lives in
imapfilter.py.  The Outlook plug-in lives in the Outlook2000 subdirectory
- see the README.txt in that directory for more information on that.

As well as these components, there's also a whole pile of utility scripts,
test harnesses and so on - see README.txt and TESTING.txt in the spambayes
distribution for more information.


Configuration
-------------

The system is configured through a file called "bayescustomize.ini".  In
here you can configure the name and type of your database, the POP3
server(s) you want to proxy to, the ports you want the proxy and the web
interface to run on, and so on.  You can also control details like how sure
you want the system to be that message really is spam before it marks it as
such.  The default values for all the options, and the documentation for
them, all lives in Options.py.  To change an option, create a
bayescustomize.ini and add the option to that - don't edit Options.py.
If you are using the POP3 proxy, SMTP proxy or IMAP filter, you can also
change most of the options you will need to access via the web user
interface.  You will probably find this at <http://localhost:8880>. To
configure the Outlook plugin, you should click on the Anti-Spam button on
the toolbar.


Command-line training
---------------------

Given a pair of Unix mailbox format files (each message starts with a line
which begins with 'From '), one containing nothing but spam and the other
containing nothing but ham, you can train Spambayes using a command like

    hammie.py -g ~/tmp/newham -s ~/tmp/newspam

The above command is command-line-centric (eg. unix, or Windows command
prompt).  You can also use the web interface for training as detailed below.


Minimal setup for using the POP3 proxy and web interface
--------------------------------------------------------

To setup the POP3 and SMTP proxies, run
    pop3proxy.py -b
from the command line.  The web interface should open in your default
browser.  You need to click on the "Configuration Link" to go to the setup
page.  The minimum you need to do to get started is enter the servers and
ports information in the POP3 proxy and SMTP proxy sections.

The POP3 proxy is then ready for your email client to connect to it on
port 110 and the SMTP proxy is ready for connections on port 25.


Reading emails and training the classifier
------------------------------------------

You now need to configure your email client to talk to the proxies instead of
the real email servers.  Change your equivalent of "pop3.my-isp.com" to
"localhost" (or to the name of the machine you're running the proxy on) in
your email client's setup, and do the same with your equivalent of
"smtp.my-isp.com". Hit "Get new email" and look at the headers of
the emails (send yourself an email if you don't have any!) - there should
be an X-Spambayes-Classification header there.  It probably says "unsure",
if you haven't done any training yet.  You should be able to create a
mail folder called "Suspected spam" and set up a filtering rule that puts
emails with an "X-Spambayes-Classification: spam" heading into that folder.
(Eventually we should publish instructions on how to do this in all the
popular email clients).

You can now train the system through the web interface - follow the "Review
messages" link and you'll see a list of the emails that the system has seen
so far.  Check the appropriate boxes and hit Train.  The messages disappear
(eventually you'll be able to get back to them, for instance to correct any
training mistakes) and if you go back to the home page you'll see that the
"Total emails trained" has increased.

Alternatively, when you receive an incorrectly classified message, you can
forward it to the SMTP proxy for training.  If the message should have been
classified as spam, forward or bounce the message to spambayes_spam@localhost,
and if the message should have been classified as ham, forward it to
spambayes_ham@localhost.  You can still review the training through the web
interface, if you wish to do so.

Note that some mail clients (particularly Outlook Express) do not forward
all headers when you bounce, forward or redirect mail.  For these clients,
you will need to set (via the web interface) the "add mail id to" option
to body, which will add a unique id to the body of each message you
receive.  You can also use this id to find a particular message via the
web interface.

Once you've done this on a few spams and a few hams, you'll find that the
X-Spambayes-Classification header is getting it right most of the time.  The
more you train it the more accurate it gets.  There's no need to train it on
every message you receive, but you should train on a few spams and a few
hams on a regular basis.  You should also try to train it on about the same
number of spams as hams.

You can train it on lots of messages in one go by either using the Hammie
script as explained in the "Command-line training" section, or by giving
messages to the web interface via the "Train" form on the Home page.  You
can train on individual messages (which is tedious) or using mbox files.


Procmail filtering
------------------

Many people on Unix-like systems have procmail available as an optional or
as the default local delivery agent.  Integrating Spambayes checking with
Procmail is straightforward.  Once you've trained Spambayes on your
collection of know ham and spam, you can use the hammie.py script to
classify incoming mail like so:

    :0 fw:hamlock
    | /usr/local/bin/hammie.py -f -d -p $HOME/hammie.db

The above Procmail recipe tells it to run /usr/local/bin/hammie.py in filter
mode (-f), and to use the training results stored in the dbm-style file
~/hammie.db.  While hammie.py is runnning, Procmail uses the lock file
hamlock to prevent multiple invocations from stepping on each others' toes.
(It's not strictly necessary in this case since no files on-disk are
modified, but Procmail will still complain if you don't specify a lock
file.)

The result of running hammie.py in filter mode is that Procmail will use the
output from the run as the mail message for further processing downstream.
Hammie.py inserts an X-Spambayes-Classification header in the output message
which looks like:

    X-Spambayes-Classification: ham; 0.00; '*H*': 1.00; '*S*': 0.00; 'python': 0.00;
	'linux,': 0.01; 'desirable': 0.01; 'cvs,': 0.01; 'perl.': 0.02;
	...

You can then use this to segregate your messages into various inboxes, like
so:

    :0
    * ^X-Spambayes-Classification: spam
    spam

    :0
    * ^X-Spambayes-Classification: unsure
    unsure

The first recipe catches all messages which hammie.py classified as spam.
The second catches all messages about which it was unsure.  The combination
allows you to isolate spam from your good mail and tuck away messages it was
unsure about so you can scan them more closely.


VM and Gnus
-----------

VM and Gnus are mail readers distributed with Emacs and XEmacs.  The
spambayes.el file in the contrib directory contains code and
instructions for VM and Gnus integration.



Things to watch out for
-----------------------

While Spambayes does an excellent job of classifying incoming mail, it is
only as good as the data on which it was trained.  Here are some tips to
help you create a good training set:

 * Don't use old mail.  The characteristics of your email change over time,
   sometimes subtly, sometimes dramatically, so it's best to use very recent
   mail to train Spambayes.  If you've abandoned an email address in the
   past because it was getting spammed heavily, there are probably some
   clues in mail sent to your old address which would bias Spambayes.

 * Check and recheck your training collections.  While you are manually
   classifying mail as spam or ham, it's easy to make a mistake and toss a
   message or ten in the wrong file.  Such miscategorized mail will throw
   off the classifier.
