from localhost (localhost [127.0.0.1])
	by phobos.labs.netnoteinc.com (Postfix) with ESMTP id 5184A440FB
	for <jm@localhost>; Mon, 29 Jul 2002 06:25:14 -0400 (EDT)
from phobos [127.0.0.1]
	by localhost with IMAP (fetchmail-5.9.0)
	for jm@localhost (single-drop); Mon, 29 Jul 2002 11:25:14 +0100 (IST)
from lugh.tuatha.org (root@lugh.tuatha.org [194.125.145.45]) by
    dogma.slashnull.org (8.11.6/8.11.6) with ESMTP id g6S2jsi13115 for
    <jm-ilug@jmason.org>; Sun, 28 Jul 2002 03:45:54 +0100
from lugh (root@localhost [127.0.0.1]) by lugh.tuatha.org
    (8.9.3/8.9.3) with ESMTP id DAA10880; Sun, 28 Jul 2002 03:44:08 +0100
from escargot.esatclear.ie (escargot.esatclear.ie
    [194.145.128.30]) by lugh.tuatha.org (8.9.3/8.9.3) with ESMTP id DAA10838
    for <ilug@linux.ie>; Sun, 28 Jul 2002 03:43:54 +0100
from esatclear.ie (h-airlock084.esatclear.ie [194.165.161.84])
    by escargot.esatclear.ie (8.9.3/8.9.3) with ESMTP id DAA29020 for
    <ilug@linux.ie>; Sun, 28 Jul 2002 03:43:52 +0100
From ilug-admin@linux.ie  Mon Jul 29 11:28:09 2002

[Guido, on the classifier pickle on SF]
> I downloaded and played with it a bit, but had no time to do anything
> systematic.
Cool!
> It correctly recognized a spam that slipped through SA.
Ditto.
> But it also identified as spam everything in my inbox that had any
> MIME structure or HTML parts, and several messages in my saved 'zope
> geeks' list that happened to be using MIME and/or HTML.
Do you know why?  The strangest implied claim there is that it hates MIME
independent of HTML.  For example, the spamprob of 'content-type:text/plain'
in that pickle is under 0.21.  'content-type:multipart/alternative' gets
0.93, but that's not a killer clue, and one bit of good content will more
than cancel it out.
WRT hating HTML, possibilities include:
1. It really had to do with something other than MIME/HTML.
2. These are pure HTML (not multipart/alternative with a text/plain part),
so that the tags aren't getting stripped.  The pickled classifier
despises all hints of HTML due to its c.l.py heritage.
3. These are multipart/alternative with a text/plain part, but the
latter doesn't contain the same text as the text/html part (for
example, as Anthony reported, perhaps the text/plain part just
says something like "This is an HMTL message.").
If it's #2, it would be easy to add an optional bool argument to tokenize()
meaning "even if it is pure HTML, strip the tags anyway".  In fact, I'd like
to do that and default it to True.  The extreme hatred of HTML on tech lists
strikes me as, umm, extreme <wink>.
> So I guess I'll have to retrain it (yes, you told me so :-).
That would be a different experiment.  I'm certainly curious to see whether
Jeremy's much-worse-than-mine error rates are typical or aberrant.