[Neale Pickett]
> ...
> If you can spare the memory, you might get better performance in this
> case using the pickle store, since it only has to go to disk once (but
> boy, does it ever go to disk!)  I can't think of anything obvious to
> speed things up once it's all loaded into memory, though.
On my box the current system scores about 50 msgs per second (starting in
memory, of course).  While that can be a drag while waiting for one of my
full test runs to complete (one of those scores a message more than 120,000
times, and trains more than 30,000 times), I've got no urge to do any speed
optimizations -- if I were using this for my own email, I'd never notice the
drag.  Guido will bitch like hell about waiting an extra second for his
50-msg batches to score, but he's the boss so he bitches about everything
<wink>.
> That's profiler territory, and profiling is exactly the kind of
> optimization I just said I wasn't going to do :)
I haven't profiled yet, but *suspect* there aren't any egregious hot spots.
5-gram'ing of long words with high-bit characters is likely overly expensive
*when* it happens, but it doesn't happen that often, and as an approach to
non-English languages it sucks anyway (i.e., there's no point speeding
something that ought to be replaced entirely).

[Guido]
> There seem to be two "drivers" for the classifier now: Neale Pickett's
> hammie.py, and the original GBayes.py.  According to the README.txt,
> GBayes.py hasn't been kept up to date.
It seemed that way to me when I ripped the classifier out of it -- I don't
think anyone has touched it after.
> Is there anything in there that isn't covered by hammie.py?
Someone else will have to answer that (I don't use GBayes or hammie, at
least not yet).
> About the only useful feature of GBayes.py that hammie.py doesn't (yet)
> copy is -u, which calculates spamness for an entire mailbox.  This
> feature can easily be copied into hammie.py.
That's been done now, right?
> (GBayes.py also has a large collection of tokenizers; but timtoken.py
> rules, so I'm not sure how interesting that is now.)
Those tokenizers are truly trivial to rewrite from scratch if they're
interesting.  The tiny spam/ham collections in GBayes are also worthless
now.  The "self test" feature didn't do anything except print its results;
Tester.py since became doctest'ed and verifies that some basic machinery
actually delivers what it's supposed to deliver.
> Therefore I propose to nuke GBayes.py, after adding a -u feature.
+1 here.
> Anyone against?