So then, Tim Peters <tim.one@comcast.net> is all like:
> [Tim]
> > My tests train on about 7,000 msgs, and a binary pickle of the database is
> > approaching 10 million bytes.
>
> That shrinks to under 2 million bytes, though, if I delete all the WordInfo
> records with spamprob exactly equal to UNKNOWN_SPAMPROB.  Such records
> aren't needed when scoring (an unknown word gets a made-up probability of
> UNKNOWN_SPAMPROB).  Such records are only needed for training; I've noted
> before that a scoring-only database can be leaner.
That's pretty good.  I wonder how much better you could do by using some
custom pickler.  I just checked my little dbm file and found a lot of
what I would call bloat:
>>> import anydbm, hammie
>>> d = hammie.PersistentGrahamBayes("ham.db")
>>> db = anydbm.open("ham.db")
>>> db["neale"], len(db["neale"])
('ccopy_reg\n_reconstructor\nq\x01(cclassifier\nWordInfo\nq\x02c__builtin__\nobject\nq\x03NtRq\x04(GA\xce\xbc{\xfd\x94\xbboK\x00K\x00K\x00G?\xe0\x00\x00\x00\x00\x00\x00tb.', 106)
>>> d.wordinfo["neale"], len(`d.wordinfo["neale"]`)
(WordInfo'(1031337979.16197, 0, 0, 0, 0.5)', 42)
Ignoring the fact that there are too many zeros in there, the pickled
version of that WordInfo object is over twice as large as the string
representation.  So we could get a 50% decrease in size just by using
the string representation instead of the pickle, right?
Something about that logic seems wrong to me, but I can't see what it
is.  Maybe pickling is good for heterogeneous data types, but every
value of our big dictionary is going to have the same type, so there's a
ton of redundancy.  I guess that explains why it compressed so well.
Neale

[Spambayes] understanding high false negative rate

>> > ##Remove: jeremy@alum.mit.edu##
Tim> Yuck: it got two 0.01's from embedding your email address at the
Tim> bottom here.
Which suggests that tagging email addresses in To/CC headers should be
handled differently than in message bodies?
Skip

[Spambayes] [ANN] Trained classifier available

http://sf.net/project/showfiles.php?group_id=61702
This is the binary pickle of my classifier after training on
my first spam/ham corpora pair.  All records with
spamprob == UNKNOWN_SPAMPROB have been purged.
It's in a zip file, and is only half a meg.
Jeremy, it would be interesting if you tried that on your data.  The false
negative rates across my other 4 test sets when run against this are:
0.364%
0.400%
0.400%
0.909%