> I believe the SpamAssassin maintainers have a scheme whereby the corpus
> of non-spam is distributed, ie. several people have bodies of non-spam
> that they use for collectively evolving the SA score set.  If that
> sounds vague, it matches my level of understanding.
See if you can get a hold of that so we can do a level-playing-field
competition. :-)
--Guido van Rossum (home page: http://www.python.org/~guido/)

>>>>> "TP" == Tim Peters <tim.one@comcast.net> writes:
>> Any thought to wrapping up your spam and ham test sets for
>> inclusion w/ the spambayes project?
TP> I gave it all the thought it deserved <wink>.  It would be
TP> wonderful to get several people cranking on the same test
TP> data, and I'm all in favor of that.  OTOH, my Data/ subtree
TP> currently has more than 35,000 files slobbering over 134
TP> million bytes -- even if I had a place to put that much stuff,
TP> I'm not sure my ISP would let me email it in one msg <wink>.
Check it into the spambayes project.  SF's disks are cheap <wink>.
-Barry

>>>>> "GW" == Greg Ward <gward@python.net> writes:
GW> If you (and Guido, Barry, et. al.) prefer, I could change that
GW> last statement to "folder = None", so the mail won't be saved
GW> at all.
I don't care if the mail is foldered on python.org, but personal
messages regardless of who they're for, shouldn't be part of the
public spambayes repository unless specifically approved by both the
recipient and sender.
Note also that we are much more liberal about python.org/zope.org
mailing list traffic than most folks.  Read list-managers for any
length of time and you'll find that there are a lot of people who
assert strict copyright over their collections, are very protective of
their traffic, and got really pissed when gmane just started
gatewaying their messages without asking.
Which might be an appropriate for their lists, but not for ours (don't
think I'm suggesting we do the same -- I /like/ our laissez-faire
approach).
But for personal email, we should be more careful.
-Barry

Quite independently from testing and tuning the algorithm, I'd like to
think about deployment.
Eventually, individuals and postmasters should be able to download a
spambayes software distribution, answer a few configuration questions
about their mail setup, training and false positives, and install it
as a filter.
A more modest initial goal might be the production of a tool that can
easily be used by individuals (since we're more likely to find
individuals willing to risk this than postmasters).
There are many ways to do this.  Some ideas:
- A program that acts both as a pop client and a pop server.  You
configure it by telling it about your real pop servers.  You then
point your mail reader to the pop server at localhost.  When it
receives a connection, it connects to the remote pop servers, reads
your mail, and gives you only the non-spam.  To train it, you'd only
need to send it the false negatives somehow; it can assume that
anything is ham that you don't say is spam within 48 hours.
- A server with a custom protocol that you send a copy of a message
and that answers "spam" or "ham".  Then you have a little program
that is invoked e.g. by procmail that talks to the server.  (The
server exists so that it doesn't have to load the pickle with the
scoring database for each message.  I don't know how big that pickle
would be, maybe loading it each time is fine.  Or maybe
marshalling.)
- Your idea here.
Takers?  How is ESR's bogofilter packaged?  SpamAssassin?  The Perl
Bayes filter advertised on slashdot?
--Guido van Rossum (home page: http://www.python.org/~guido/)