So then, Neale Pickett <neale@woozle.org> is all like:
> Maybe there's some subtle interaction between generators and lists
> that I can't understand.  Or something.  Being as I'm baffled, I don't
> imagine any theory I come up with will be anywhere close to reality.
And then, just as I was about to fall asleep, I figured it out.  The
tokenizer now has an extra [:], and all is well.  I feel like a real
chawbacon for not realizing this earlier.  :*)
Blaming it on staying up past bedtime,
Neale

> 5 sets, each of 1800ham/1550spam, just ran the once (it matched all 5 to
> Training on Data/Ham/Set1 & Data/Spam/Set1 ... 1798 hams & 1548 spams
> Training on Data/Ham/Set2 & Data/Spam/Set2 ... 1798 hams & 1546 spams
> Training on Data/Ham/Set3 & Data/Spam/Set3 ... 1798 hams & 1547 spams
> Training on Data/Ham/Set4 & Data/Spam/Set4 ... 1803 hams & 1547 spams
> Training on Data/Ham/Set5 & Data/Spam/Set5 ... 1798 hams & 1550 spams
How were these msgs broken up into the 5 sets?  Set4 in particular is giving
the other sets severe problems, and Set5 blows the f-n rate on everything
it's predicting -- when the rates across runs within a training set vary by
as much as a factor of 25, it suggests there was systematic bias in the way
the sets were chosen.  For example, perhaps they were broken into sets by
arrival time.  If that's what you did, you should go back and break them
into sets randomly instead.  If you did partition them randomly, the wild
variance across runs is mondo mysterious.
>> I expect hammie will do a much better job on this already than hand
>> grepping.  Be sure to stare at the false positives and get the
> Yah, but there's a chicken-and-egg problem there - I want stuff that's
> _known_ to be right to test this stuff,
Then you have to look at every message by eyeball -- any scheme has non-zero
> so using the spambayes code to tell me whether it's spam is not
Trust me <wink> -- it helps a *lot*.  I expect everyone who has done any
testing here has discovered spam in their ham, and vice versa.  Results
improve as you improve the categorization.  Once the gross mistakes are
straightened out, it's much less tedious to scan the rest by eyeball.
> Yep, it shows up in a lot of spam, but also in different forms in hams.
> But the hams each manage to pick a different variant of
> or whatever - so they don't end up counteracting the various bits in the
> Looking further, a _lot_ of the bad skip rubbish is coming from
For whatever reason, there appear to be few of those in BruceG's spam
collection.  I added code to strip uuencoded sections, and pump out uuencode
summary tokens instead.  I'll check it in.  It didn't make a significant
difference on my usual test run (a single spam in my Set4 is now judged as
ham by the other 4 sets; nothing else changed).  It does shrink the database
size here by a few percent.  Let us know whether it helps you!
Before and after stripping uuencoded sections:
total unique fn went from 89 to 90 lost    +1.12%