[Guido]
> Perhaps more useful would be if Tim could check in the pickle(s?)
> generated by one of his training runs, so that others can see how
> Tim's training data performs against their own corpora.
I did that yesterday, but seems like nobody bit.  Just in case <wink>, I
uploaded a new version just now.  Since MINCOUNT went away, UNKNOWN_SPAMPROB
is much less likely, and there's almost nothing that can be pruned away (so
the file is about 5x larger now).
http://sf.net/project/showfiles.php?group_id=61702
> This could also be the starting point for a self-contained distribution
> (you've got to start with *something*, and training with python-list data
> seems just as good as anything else).
The only way to know anything here is for someone to try it.

First test results using tokenizer.Tokenizer.tokenize_headers()
Second test results using mboxtest.MyTokenizer.tokenize_headers().
This uses all headers except Received, Data, and X-From_.

Before we get too far down this road, what do people think of creating a
spambayes package containing classifier and tokenizer?  This is just to
minimize clutter in site-packages.
Skip

[Jeremy Hylton]
> Here's clarification of why I did:
That's pretty much what I had guessed.  Thanks!  One more to try:
> First test results using tokenizer.Tokenizer.tokenize_headers()
> unmodified.
> ...
> Second test results using mboxtest.MyTokenizer.tokenize_headers().
> This uses all headers except Received, Data, and X-From_.
> ...
Try the latter again, but call the base tokenize_headers() too.

Your message of "Sat, 07 Sep 2002 22:55:12 EDT."
             <LNBBLJKPBEHFEDALKOLCOEOBBCAB.tim.one@comcast.net>

> > But it also identified as spam everything in my inbox that had any
> > MIME structure or HTML parts, and several messages in my saved 'zope
> > geeks' list that happened to be using MIME and/or HTML.
> Do you know why?  The strangest implied claim there is that it hates MIME
> independent of HTML.  For example, the spamprob of 'content-type:text/plain'
> in that pickle is under 0.21.  'content-type:multipart/alternative' gets
> 0.93, but that's not a killer clue, and one bit of good content will more
I reran the experiment (with the new SpamHam1.pik, but it doesn't seem
to make a difference).  Here are the clues for the two spams in my
inbox (in hammie.py's output format, which sorts the clues by
probability; the first two numbers are the message number and overall
0.85; '>content-type:text/plain': 0.85; 'please': 0.85; 'capital':
0.99; 'content-type:multipart/related': 0.99; 'economy': 0.99;
This has 6 content-types as spam clues, only one of which is related
to HTML, despite there being an HTML alternative (and 12 other spam
clues, vs. only 6 ham clues).  This was an announcement of a public
event by our building owners, with a text part that was the same as
the HTML (AFAICT).  Its language may be spammish, but the content-type
clues didn't help.  (BTW, it makes me wonder about the wisdom of
keeping punctuation -- 'economy' and 'economy"' to me don't seem to
'from:adam': 0.01; 'from:email addr:panix': 0.01; 'poked': 0.01;
"i've": 0.06; 'wednesday': 0.07; 'content-disposition:inline':
Here is the full message (Received: headers stripped), with apologies
I've been looking into hotels.  I poked around expedia for availability
from March 26 to 29 (4 nights, wednesday thorugh saturday).
I've also started contacting hotels for group rates; some of the group
rates are no better than the regular rates, and they require signing a
contract with a minimum number of rooms sold (with someone financially
responsible for unbooked rooms).  Most hotels are less than responsive...
Radission - Barcelo Hotel (DuPont Circle)
State Plaza hotel (Foggy Bottom; near GWU)
Hilton Silver Spring (Near Metro, in suburban MD)
Conn Ave, between DuPont Circle/Woodley Park Metro stations
Econo Lodge Alexandria (Near Metro, in suburban VA)
This is a hand picked list; I ignored anything over $125/night, even
though there are some really well situated hotels nearby at higher rates.
Also, I'm not sure how much these prices reflect an expedia-only
discount.  I can't vouch for any of these hotels, either.
I also found out that the down season for DC Hotels are mid-june through
mid-september, and mid-november through mid-january.
This one has no MIME structure nor HTML!  It even has a
Content-disposition which is counted as a non-spam clue.  It got
f-p'ed because of the many hospitality-related and money-related
terms.  I'm surprised $125/night and similar aren't clues too.  (And
again, several spam clues are duplicated with different variations:
'hotel', 'hotels', 'hotels.', 'subject:Hotel', 'rates,', 'rates.'.
> WRT hating HTML, possibilities include:
> 1. It really had to do with something other than MIME/HTML.
> 2. These are pure HTML (not multipart/alternative with a text/plain part),
>    so that the tags aren't getting stripped.  The pickled classifier
>    despises all hints of HTML due to its c.l.py heritage.
> 3. These are multipart/alternative with a text/plain part, but the
>    latter doesn't contain the same text as the text/html part (for
>    example, as Anthony reported, perhaps the text/plain part just
>    says something like "This is an HMTL message.").
> If it's #2, it would be easy to add an optional bool argument to tokenize()
> meaning "even if it is pure HTML, strip the tags anyway".  In fact, I'd like
> to do that and default it to True.  The extreme hatred of HTML on tech lists
I also looked in more detail at some f-p's in my geeks traffic.  The
first one's a doozie (that's the term, right? :-).  It has lots of
HTML clues that are apparently ignored.  It was a multipart/mixed with
two parts: a brief text/plain part containing one or two sentences, a
and some employer-generated spammish boilerplate; the second part was
the HTML taken directly from the above URL.  Clues:
'duct': 0.01; 'email addr:digicool': 0.01; 'email name:paul':
'framework': 0.01; 'from:email addr:digicool': 0.01; 'from:email
'http>1:products': 0.01; 'http>1:query': 0.01; 'http>1:search':
'wrote': 0.01; 'x-mailer:mozilla 4.74 [en] (windows nt 5.0; u)':
'align=left': 0.99; 'align=middle': 0.99; 'align=right': 0.99;
The second f-p had the same structure (and sender :-); the third f-p
had the same structure and a different sender.  Ditto the fifth, sixth.  (Not posting clues for
The fourth was different: plaintext with one very short sentence and a
300 1.00 S 'from:email addr:digicool': 0.01; 'http1:news': 0.24;
'from:email addr:com>': 0.32; 'from:tres': 0.50; 'http>1:1114digi':
0.50; 'proto:http': 0.50; 'subject:Geeks': 0.50; 'x-mailer:mozilla
'http>1:htm': 0.90; 'subject:Software': 0.96; 'http>1:business':
I scanned a bunch more until I got bored, and most of them were either
of the first form (brief text with URL followed by quoted HTML from
website) or the second (brief text with one or more URLs).
It's up to you to decide what to call this, but I think these are none
of your #1, #2 or #3 (they're close to #3, but all are multipart/mixed
> > So I guess I'll have to retrain it (yes, you told me so :-).
> That would be a different experiment.  I'm certainly curious to see whether
> Jeremy's much-worse-than-mine error rates are typical or aberrant.
It's possible that the corpus you've trained on is more homogeneous