> > Maybe.  I batch messages using fetchmail (don't ask why), and adding
> > .4 seconds per message for a batch of 50 (not untypical) feels like a
> > real wait to me...
>
> Yeesh.  Sounds like what you need is something to kick up once and score
> an entire mailbox.
>
> Wait a second...  So *that's* why you wanted -u.
>
> If you can spare the memory, you might get better performance in this
> case using the pickle store, since it only has to go to disk once (but
> boy, does it ever go to disk!)  I can't think of anything obvious to
> speed things up once it's all loaded into memory, though.  That's
> profiler territory, and profiling is exactly the kind of optimization
> I just said I wasn't going to do :)
We could have a server mode (someone described this as an SA option).
--Guido van Rossum (home page: http://www.python.org/~guido/)

>   TP> I'm reading this now as that you trained on about 220 spam and
>   TP> about 220 ham.  That's less than 10% of the sizes of the
>   TP> training sets I've been using.  Please try an experiment: train
>   TP> on 550 of each, and test once against the other 550 of each.
Possibly.  I checked in a change to classifier.py overnight (getting rid of
MINCOUNT) that gave a major improvement in the f-n rate too, independent of
> Here are results with the stock tokenizer:
Unsure what "stock tokenizer" means to you.  For example, it might mean
> Training on <mbox: /home/jeremy/Mail/INBOX 0> & <mbox:
> Training on <mbox: /home/jeremy/Mail/INBOX 2> & <mbox:
Note that those rates remain much higher than I got using just 220 of ham
and 220 of spam.  That remains A Mystery.
> And results from the tokenizer that looks at all headers except Date,
Unsure what that means too.  For example, "looks at" might mean you enabled
Anthony's count-them gimmick, and/or that you're tokenizing them yourself,
> Is it safe to conclude that avoiding any cleverness with headers is a
Since I don't know what you did, exactly, I can't guess.  What you seemed to
show is that you did *something* clever with headers and that doing so
helped (the "after" numbers are better than the "before" numbers, right?).
Assuming that what you did was override what's now
tokenizer.Tokenizer.tokenize_headers with some other routine, and didn't
call the base Tokenizer.tokenize_headers at all, then you're missing
carefully tested treatment of just a few header fields, but adding many
dozens of other header fields.  There's no question that adding more header
fields should help; tokenizer.Tokenizer.tokenize_headers doesn't do so only
because my testing corpora are such that I can't add more headers without
getting major benefits for bogus reasons.
Apart from all that, you said you're skipping Received.  By several
accounts, that may be the most valuable of all the header fields.  I'm
(meaning tokenizer.Tokenizer.tokenize_headers) skipping them too for the
reason explained above.  Offline a week or two ago, Neil Schemenauer
for header in msg.get_all("received", ()):
This makes a lot of sense to me; I just checked it in, but left it disabled