[Spambayes] deleting "duplicate" spam before training?  good idea
	orbad?

On 09 September 2002, Tim Peters said:
> > Would people be interested in the script?  I'd be happy to extricate
> > it from my local modules and check it into CVS.
>
> Sure!  I think it's relevant, but maybe for another purpose.  Paul Svensson
> is thinking harder about real people <wink> than the rest of us, and he may
> be able to get use out of approaches that identify closely related spam.
> For example, some amount of spam is going to end up in the ham training data
> in real life use, and any sort of similarity score to a piece of known spam
> may be an aid in finding and purging it.
OTOH, look into DCC (Distributed Checksum Clearinghouse,
http://www.rhyolite.com/anti-spam/dcc/), which uses fuzzy checksums.
It's quite likely that DCC's checksumming scheme is better than
something any of us would throw together for personal use (no offense,
Skip!).  But I have no personal experience of it.
Greg
--
Greg Ward <gward@python.net>                         http://www.gerg.ca/
If it can't be expressed in figures, it is not science--it is opinion.

[Spambayes] python.org email harvesting ready to roll

> We've not only reduced the f-p and f-n rates in my test runs, we've also
> made the score distributions substantially sharper.  This is bad news for
> Greg, because the non-existent "middle ground" is becoming even less
Well, I've finally got around to pulling down the SF code. Starting
with it, and absolutely zero local modifications, I see the following:
My next (current) task is to complete the corpus I've got - it's currently
got ~ 9000 ham, 7800 spam, and about 9200 currently unsorted. I'm tossing
up using either hammie or spamassassin to do the initial sort  - previously
I've used various forms of 'grep' for keywords and a little gui thing to
pop a message up and let me say 'spam/ham', but that's just getting too, too
I can't make it available en masse, but I will look at finding some of
the more 'interesting' uglies. One thing I've seen (consider this
'anecdotal' for now) is that the 'skip' tokens end up in a _lot_ of the