>     >> > ##Remove: jeremy@alum.mit.edu##
>
>     Tim> Yuck: it got two 0.01's from embedding your email address at the
>     Tim> bottom here.
>
> Which suggests that tagging email addresses in To/CC headers should be
> handled differently than in message bodies?
I don't know whether it suggests that, but they would be tagged differently
in to/cc if I were tagging them at all right now.  If I were tagging To:
addresses, for example, the tokens would look like
'to:email addr:mit'
instead of
'email addr:mit'
as they appear when an email-like thingie is found in the body.  Whether
email addresses should be stuck in as one blob or split up as they are now
is something I haven't tested.

>> The false positive rate is 0-3%.  (Finally!  I had to scrub a
>> bunch of previously unnoticed spam from my inbox.)  Both
>> collections have about 1100 messages.
TP> Does this mean you trained on about 1100 of each?
The total collections are 1100 messages.  I trained with 1100/5
TP> Can't guess.  You're in a good position to start adding more
TP> headers into the analysis, though.  For example, an easy start
TP> would be to uncomment the header-counting lines in tokenize()
TP> (look for "Anthony").  Likely the most valuable thing it's
TP> missing then is some special parsing and tagging of Received
I tried the "Anthony" stuff, but it didn't make any appreciable
difference that I could see from staring at the false negative rate.
The numbers are big enough that a quick eyeball suffices.
Then I tried a dirt simple tokenizer for the headers that tokenize the
words in the header and emitted like this "%s: %s" % (hdr, word).
That worked too well :-).  The received and date headers helped the
classifier discover that most of my spam is old and most of my ham is
So I tried a slightly more complex one that skipped received, data,
and x-from_, which all contained timestamps.  I also skipped the X-VM-
if k in self.skip or k.startswith('x-vm'):
This did moderately better.  The false negative rate is 7-21% over the
tests performed so far.  This is versus 11-28% for the previous test
run that used the timtest header tokenizer.
It's interesting to see that the best descriminators are all ham
discriminators.  There's not a single spam-indicator in the list.
Most of the discriminators are header fields.  One thing to note is
that the presence of Mailman-generated headers is a strong non-spam
indicator.  That matches my intuition: I got an awful lot of
Mailman-generated mail, and those lists are pretty good at surpressing
spam.  The other thing is that I get a lot of ham from people who use
XEmacs.  That's probably Barry, Guido, Fred, and me :-).
One final note.  It looks like many of the false positives are from
people I've never met with questions about Shakespeare.  They often
> May I please take some of your precious time to ask you to help me to find a
> solution to a problem that is worrying me greatly. I am old science student