[Guido]
> Perhaps more useful would be if Tim could check in the pickle(s?)
> generated by one of his training runs, so that others can see how
> Tim's training data performs against their own corpora.
I did that yesterday, but seems like nobody bit.  Just in case <wink>, I
uploaded a new version just now.  Since MINCOUNT went away, UNKNOWN_SPAMPROB
is much less likely, and there's almost nothing that can be pruned away (so
the file is about 5x larger now).
http://sf.net/project/showfiles.php?group_id=61702
> This could also be the starting point for a self-contained distribution
> (you've got to start with *something*, and training with python-list data
> seems just as good as anything else).
The only way to know anything here is for someone to try it.

Before we get too far down this road, what do people think of creating a
spambayes package containing classifier and tokenizer?  This is just to
minimize clutter in site-packages.
Skip

> Before we get too far down this road, what do people think of creating a
> spambayes package containing classifier and tokenizer?  This is just to
> minimize clutter in site-packages.
Too early IMO (if you mean to leave the various other tools out of
it).  If and when we package this, perhaps we should use Barry's trick
from the email package for making the package itself the toplevel dir
of the distribution (rather than requiring an extra directory level
just so the package can be a subdir of the distro).
--Guido van Rossum (home page: http://www.python.org/~guido/)

> I'd prefer to strip HTML tags from everything, but last time I
> tried that it still had bad effects on the error rates in my
> corpora (the full test results with and without HTML tag stripping
> is included in the "What about HTML?" comment block).  But as the
> # XXX So, if another way is found to slash the f-n rate, the decision here
> # XXX not to strip HTML from HTML-only msgs should be revisited.
> and we've since done several things that gave significant f-n rate
> reductions.  I should test that again now.
I did so.  Alas, stripping HTML tags from all text still hurts the f-n rate
Last time I tried this (see tokenizer.py comments), the f-n rate after
stripping tags ranged from 0.982% to 1.781%, with a median of about 1.34%,
so we've made tons of progress on the f-n rate since then.  But the mere
presence of HTML tags still remains a significant clue for c.l.py traffic,
If we want to take the focus of this away from c.l.py traffic, I can't say
what effect HTML stripping would have (I don't have suitable test data to