So then, Guido van Rossum <guido@python.org> is all like:
> Maybe.  I batch messages using fetchmail (don't ask why), and adding
> .4 seconds per message for a batch of 50 (not untypical) feels like a
> real wait to me...
Yeesh.  Sounds like what you need is something to kick up once and score
an entire mailbox.
Wait a second...  So *that's* why you wanted -u.
If you can spare the memory, you might get better performance in this
case using the pickle store, since it only has to go to disk once (but
boy, does it ever go to disk!)  I can't think of anything obvious to
speed things up once it's all loaded into memory, though.  That's
profiler territory, and profiling is exactly the kind of optimization
I just said I wasn't going to do :)
Neale

Your message of "Fri, 06 Sep 2002 22:48:17 PDT."
             <w537khybba6.fsf@woozle.org>

> > Maybe.  I batch messages using fetchmail (don't ask why), and adding
> > .4 seconds per message for a batch of 50 (not untypical) feels like a
> > real wait to me...
>
> Yeesh.  Sounds like what you need is something to kick up once and score
> an entire mailbox.
>
> Wait a second...  So *that's* why you wanted -u.
>
> If you can spare the memory, you might get better performance in this
> case using the pickle store, since it only has to go to disk once (but
> boy, does it ever go to disk!)  I can't think of anything obvious to
> speed things up once it's all loaded into memory, though.  That's
> profiler territory, and profiling is exactly the kind of optimization
> I just said I wasn't going to do :)
We could have a server mode (someone described this as an SA option).
--Guido van Rossum (home page: http://www.python.org/~guido/)

>> The total collections are 1100 messages.  I trained with 1100/5
TP> I'm reading this now as that you trained on about 220 spam and
TP> about 220 ham.  That's less than 10% of the sizes of the
TP> training sets I've been using.  Please try an experiment: train
TP> on 550 of each, and test once against the other 550 of each.
This helps a lot!  Here are results with the stock tokenizer:
Training on <mbox: /home/jeremy/Mail/INBOX 0> & <mbox: /home/jeremy/Mail/spam 8>
Training on <mbox: /home/jeremy/Mail/INBOX 0> & <mbox: /home/jeremy/Mail/spam 0>
Training on <mbox: /home/jeremy/Mail/INBOX 2> & <mbox: /home/jeremy/Mail/spam 3>
Training on <mbox: /home/jeremy/Mail/INBOX 2> & <mbox: /home/jeremy/Mail/spam 0>
And results from the tokenizer that looks at all headers except Date,
Is it safe to conclude that avoiding any cleverness with headers is a
TP> Do that a few times making a random split each time (it won't be
TP> long until you discover why directories of individual files are
TP> a lot easier to work -- e.g., random.shuffle() makes this kind
You haven't looked at mboxtest.py, have you?  I'm using
random.shuffle(), too.  You don't need to have many files to make an
arbitrary selection of messages from an mbox.
I'll report some more results when they're done.