from n6.grp.scd.yahoo.com (n6.grp.scd.yahoo.com [66.218.66.90])
    by dogma.slashnull.org (8.11.6/8.11.6) with SMTP id g7MDcOZ08504 for
    <zzzz@spamassassin.taint.org>; Thu, 22 Aug 2002 14:38:25 +0100

No, hits=-21.5 required=5.0
	tests=AWL,KNOWN_MAILING_LIST,QUOTED_EMAIL_TEXT,REFERENCES,
	      T_OUTLOOK_REPLY,T_QUOTE_TWICE_1
	version=2.50-cvs

----- Original Message -----
From: "Gary Lawrence Murphy" <garym@canada.com>
>
> >>>>> "w" == wpin  <lslifriend> writes:
>
>     w> Those who want to leave for the Netherlands to carry on any
>     w> kind of education,including for PHD, must be careful
>     w> . Especially the university of Groningen(RUG) should be
>     w> avoided. This university was once a good one but now it has
>     w> lost its reputation. ... Studying is a good investment in time
>     w> and money. So invest in the right place and time. You are
>     w> warned.
Dear gawd - 'lost its reputation' - how horrible for those Northern
Europeans. What could possibly be worse than that?

[Skip Montanaro]
> Because I get mail through several different email addresses, I
> frequently get duplicates (or triplicates or more-plicates) of
> various spam messages.  In saving spam for later analysis I haven't
> always been careful to avoid saving such duplicates.
>
> I wrote a script some time ago to try an minimize the duplicates I see
> by calculating a loose checksum, but I still have some duplicates.
> Should I delete the duplicates before training or not?
People just can't stop thinking <wink>.  The classifier should work best
when trained on a wholly random spattering of real life.  If real life
contains duplicates, then that's what the classifier should see.
> Would people be interested in the script?  I'd be happy to extricate
> it from my local modules and check it into CVS.
Sure!  I think it's relevant, but maybe for another purpose.  Paul Svensson
is thinking harder about real people <wink> than the rest of us, and he may
be able to get use out of approaches that identify closely related spam.
For example, some amount of spam is going to end up in the ham training data
in real life use, and any sort of similarity score to a piece of known spam
may be an aid in finding and purging it.