You missed the part that said that spam is kept in the "eThunk" and was
viewable by a simple viewer for final disposition?
Of course, with Outbloat, you could fire up PythonWin and stuff the spam
into the Junk Email folder... but then you loose the ability to retrain on
> [mailto:spambayes-bounces+whisper=oz.net@python.org]On Behalf Of Tim
> Sent: Friday, September 06, 2002 12:24

So then, Tim Peters <tim.one@comcast.net> is all like:
> [Guido]
> >   ...
> >   I don't know how big that pickle would be, maybe loading it each time
> >   is fine.  Or maybe marshalling.)
>
> My tests train on about 7,000 msgs, and a binary pickle of the database is
> approaching 10 million bytes.
My paltry 3000-message training set makes a 6.3MB (where 1MB=1e6 bytes)
pickle.  hammie.py, which I just checked in, will optionally let you
write stuff out to a dbm file.  With that same message base, the dbm
file weighs in at a hefty 21.4MB.  It also takes longer to write:
Using a database:
real    8m24.741s
user    6m19.410s
sys     1m33.650s
Using a pickle:
real    1m39.824s
user    1m36.400s
sys     0m2.160s
This is on a PIII at 551.257MHz (I don't know what it's *supposed* to
be, 551.257 is what /proc/cpuinfo says).
For comparison, SpamOracle (currently the gold standard in my mind, at
least for speed) on the same data blazes along:
real    0m29.592s
user    0m28.050s
sys     0m1.180s
Its data file, which appears to be a marshalled hash, is 448KB.
However, it's compiled O'Caml and it uses a much simpler tokenizing
algorithm written with a lexical analyzer (ocamllex), so we'll never be
able to outperform it.  It's something to keep in mind, though.
I don't have statistics yet for scanning unknown messages.  (Actually, I
do, and the database blows the pickle out of the water, but it scores
every word with 0.00, so I'm not sure that's a fair test. ;)  In any
case, 21MB per user is probably too large, and 10MB is questionable.
On the other hand, my pickle compressed very well with gzip, shrinking
down to 1.8MB.
Neale

[Tim]
> My tests train on about 7,000 msgs, and a binary pickle of the database is
> approaching 10 million bytes.
That shrinks to under 2 million bytes, though, if I delete all the WordInfo
records with spamprob exactly equal to UNKNOWN_SPAMPROB.  Such records
aren't needed when scoring (an unknown word gets a made-up probability of
UNKNOWN_SPAMPROB).  Such records are only needed for training; I've noted
before that a scoring-only database can be leaner.
In part the bloat is due to character 5-gram'ing, part due to that the
database is brand new so has never been cleaned via clearjunk(), and part
due to plain evil gremlins.