>> If and when we package this, perhaps we should use Barry's trick ...
Greg> It's not a *trick*!  It just requires this
Greg>   package_dir = {'spambayes': '.'}
Greg> in the setup script.
That has the nasty side effect of placing all .py files in the package.
What about obvious executable scripts (like timtest or hammie)?  How can I
keep them out of the package?
Skip

> That has the nasty side effect of placing all .py files in the
> package.  What about obvious executable scripts (like timtest or
> hammie)?  How can I keep them out of the package?
Why would we care about installing a few extra files, as long as
they're inside a package?
--Guido van Rossum (home page: http://www.python.org/~guido/)

Because I get mail through several different email addresses, I frequently
get duplicates (or triplicates or more-plicates) of various spam messages.
In saving spam for later analysis I haven't always been careful to avoid
saving such duplicates.
I wrote a script some time ago to try an minimize the duplicates I see by
calculating a loose checksum, but I still have some duplicates.  Should I
delete the duplicates before training or not?  Would people be interested in
the script?  I'd be happy to extricate it from my local modules and check it
into CVS.
Skip

>>>>> "TP" == Tim Peters <tim.one@comcast.net> writes:
>> First test results using tokenizer.Tokenizer.tokenize_headers()
>> unmodified.  ...  Second test results using
>> mboxtest.MyTokenizer.tokenize_headers().  This uses all headers
>> except Received, Data, and X-From_.  ...
TP> Try the latter again, but call the base tokenize_headers() too.
Sorry.  I haven't found the time to try any more test runs.  Perhaps
later today.
Jeremy

Guido> Why would we care about installing a few extra files, as long as
Guido> they're inside a package?
I guess you needn't worry about that.  It just doesn't seem "clean" to me.
S

>> I wrote a script some time ago to try an minimize the duplicates I
>> see by calculating a loose checksum, but I still have some
>> duplicates.  Should I delete the duplicates before training or not?
Tim> People just can't stop thinking <wink>.  The classifier should work
Tim> best when trained on a wholly random spattering of real life.  If
Tim> real life contains duplicates, then that's what the classifier
Tim> should see.
A bit more detail.  I get destined for many addresses: skip@pobox.com,
skip@calendar.com, concerts@musi-cal.com, webmaster@mojam.com, etc.  I
originally wrote (a slightly different version of) the loosecksum.py script
I'm about to check in to avoid manually scanning all those presumed spams
which are really identical.  Once a message was identified as spam, what I
refer to as a loose checksum was computed to try and avoid saving the same
spam multiple times for later review.
>> Would people be interested in the script?  I'd be happy to extricate
>> it from my local modules and check it into CVS.
Tim> Sure!  I think it's relevant, but maybe for another purpose.  Paul
Tim> Svensson is thinking harder about real people <wink> than the rest
Tim> of us, and he may be able to get use out of approaches that
Tim> identify closely related spam.  For example, some amount of spam is
Tim> going to end up in the ham training data in real life use, and any
Tim> sort of similarity score to a piece of known spam may be an aid in
Tim> finding and purging it.
I'll check it in.  Let me know if you find it useful.
Skip