[Anthony Baxter]
> I've got a test set here that's the last 3 and a bit years email to
> info@ekit.com and info@ekno.com - it's a really ugly set of 20,000+
> messages, currently broken into 7,000 spam, 9,000 ham, 9,000 currently
> unclassified. These addresses are all over the 70-some different
> ekit/ekno/ISIConnect websites, so they get a LOT of spam.
>
> As well as the usual spam, it also has customers complaining about
> credit card charges, it has people interested in the service and
> asking questions about long distance rates, &c &c &c. Lots and lots
> of "commercial" speech, in other words. Stuff that SA gets pretty
> badly wrong.
Can this corpus be shared?  I suppose not.
> I'm currently mangling it by feeding all parts (text, html, whatever
> else :) into the filters, as well as both a selected number of headers
> (to, from, content-type, x-mailer), and also a list of (header,
> count_of_header). This is showing up some nice stuff - e.g. the
> X-uidl that stoopid spammers blindly copy into their messages.
If we ever <wink> have a shared corpus, an easy refactoring of timtest
should allow to plug in different tokenizers.  I've only made three changes
to Graham's algorithm so far (well, I've made dozens -- only three survived
testing as proven winners), all the rest has been refining the tokenization
to provide better clues.
> I did have Received in there, but it's out for the moment, as it causes
> rates to drop.
That's ambiguous.  Accuracy rates or error rates, ham or spam rates?
> I'm also stripping out HTML tags, except for href="" and src="" - there's
> so so much goodness in them (note that I'm only keeping the contents of
> the attributes).
Mining embedded http/https/ftp thingies cut the false negative rate in half
in my tests (not keying off href, just scanning for anything that "looked
like" one); that was the single biggest f-n improvement I've seen.  It
didn't change the false positive rate.  So you know whether src added
additional power, or did you do both at once?

Guido> Takers?  How is ESR's bogofilter packaged?  SpamAssassin?  The
Guido> Perl Bayes filter advertised on slashdot?
Dunno about the other tools, but SpamAssassin is a breeze to incorporate
into a procmail environment.  Lots of people use it in many other ways.  For
performance reasons, many people run a spamd process and then invoke a small
C program called spamc which shoots the message over to spamd and passes the
result back out.  I think spambayes in incremental mode is probably fast
enough to not require such tricks (though I would consider changing the
pickle to an anydbm file).
Basic procmail usage goes something like this:
:0fw
| spamassassin -P
:0
* ^X-Spam-Status: Yes
$SPAM
Which just says, "Run spamassassin -P reinjecting its output into the
processing stream.  If the resulting mail has a header which begins
"X-Spam-Status: Yes", toss it into the folder indicated by the variable
$SPAM.
SpamAssassin also adds other headers as well, which give you more detail
about how its tests fared.  I'd like to see spambayes operate in at least
this way: do its thing then return a message to stdout with a modified set
of headers which further processing downstream can key on.
Skip