> > - A program that acts both as a pop client and a pop server.  You
> >   configure it by telling it about your real pop servers.  You then
> >   point your mail reader to the pop server at localhost.  When it
> >   receives a connection, it connects to the remote pop servers, reads
> >   your mail, and gives you only the non-spam.
>
> FYI, I'll never trust such a scheme:  I have no tolerance for false
> positives, and indeed do nothing to try to block spam on any of my email
> accounts now for that reason.  Deliver all suspected spam to a Spam folder
> instead and I'd love it.
Another config parameter.
The filter could add a header file.  Or a ~ to the subject if you like
that style. :-)
--Guido van Rossum (home page: http://www.python.org/~guido/)

[Guido]
>   ...
>   I don't know how big that pickle would be, maybe loading it each time
>   is fine.  Or maybe marshalling.)
My tests train on about 7,000 msgs, and a binary pickle of the database is
approaching 10 million bytes.  I haven't done anything to try to reduce its
size, and know of some specific problem areas (for example, doing character
5-grams of "long words" containing high-bit characters generates a lot of
database entries, and I suspect they're approximately worthless).  OTOH,
adding in more headers will increase the size.  So let's call it 10 meg
<wink>.

So then, Tim Peters <tim.one@comcast.net> is all like:
> [Guido]
> >   ...
> >   I don't know how big that pickle would be, maybe loading it each time
> >   is fine.  Or maybe marshalling.)
>
> My tests train on about 7,000 msgs, and a binary pickle of the database is
> approaching 10 million bytes.
My paltry 3000-message training set makes a 6.3MB (where 1MB=1e6 bytes)
pickle.  hammie.py, which I just checked in, will optionally let you
write stuff out to a dbm file.  With that same message base, the dbm
file weighs in at a hefty 21.4MB.  It also takes longer to write:
Using a database:
real    8m24.741s
user    6m19.410s
sys     1m33.650s
Using a pickle:
real    1m39.824s
user    1m36.400s
sys     0m2.160s
This is on a PIII at 551.257MHz (I don't know what it's *supposed* to
be, 551.257 is what /proc/cpuinfo says).
For comparison, SpamOracle (currently the gold standard in my mind, at
least for speed) on the same data blazes along:
real    0m29.592s
user    0m28.050s
sys     0m1.180s
Its data file, which appears to be a marshalled hash, is 448KB.
However, it's compiled O'Caml and it uses a much simpler tokenizing
algorithm written with a lexical analyzer (ocamllex), so we'll never be
able to outperform it.  It's something to keep in mind, though.
I don't have statistics yet for scanning unknown messages.  (Actually, I
do, and the database blows the pickle out of the water, but it scores
every word with 0.00, so I'm not sure that's a fair test. ;)  In any
case, 21MB per user is probably too large, and 10MB is questionable.
On the other hand, my pickle compressed very well with gzip, shrinking
down to 1.8MB.
Neale