On 09 September 2002, Tim Peters said:
> > Would people be interested in the script?  I'd be happy to extricate
> > it from my local modules and check it into CVS.
>
> Sure!  I think it's relevant, but maybe for another purpose.  Paul Svensson
> is thinking harder about real people <wink> than the rest of us, and he may
> be able to get use out of approaches that identify closely related spam.
> For example, some amount of spam is going to end up in the ham training data
> in real life use, and any sort of similarity score to a piece of known spam
> may be an aid in finding and purging it.
OTOH, look into DCC (Distributed Checksum Clearinghouse,
http://www.rhyolite.com/anti-spam/dcc/), which uses fuzzy checksums.
It's quite likely that DCC's checksumming scheme is better than
something any of us would throw together for personal use (no offense,
Skip!).  But I have no personal experience of it.
Greg
--
Greg Ward <gward@python.net>                         http://www.gerg.ca/
If it can't be expressed in figures, it is not science--it is opinion.

We've not only reduced the f-p and f-n rates in my test runs, we've also
made the score distributions substantially sharper.  This is bad news for
Greg, because the non-existent "middle ground" is becoming even less
As usual for me, this is an aggregate of 20 runs, each both training and
predicting on 4000 c.l.py ham + ~2750 BruceG spam.
Only 25 ham scores out of 80,000 are above 0.025 now (and, yes, the
"Nigerian scam"-quoting msg is still counted as ham -- I haven't taken
anything out of the ham corpus since remving the "If AOL were a car" spam),
the f-p rate wouldn't have changed at all if the spamprob cutoff were
dropped from 0.90 to 0.675, dropping the cutoff to 0.40 would have added
only 2 false positives, and dropping it to 0.15 would have added only

After my latest cvs up, timtest fails with
Traceback (most recent call last):
File "/home/skip/src/spambayes/timtest.py", line 294, in ?
drive(nsets)
File "/home/skip/src/spambayes/timtest.py", line 264, in drive
d = Driver()
File "/home/skip/src/spambayes/timtest.py", line 152, in __init__
self.global_ham_hist = Hist(options.nbuckets)
AttributeError: 'OptionsClass' object has no attribute 'nbuckets'
I'm running it as
timtest -n5 > Data/timtest.out
from my ~/Mail directory (not from my ~/src/spambayes directory).  If I
create a symlink to ~/src/spambayes/bayes.ini it works once again, but
shouldn't there be an nbuckets attribute with a default value already?
Skip