[Spambayes] All Cap or Cap Word Subjects

Just curious if subject line capitalization can be used as an indicator.
Either the percentage of characters that are caps..
Or, percentage starting with a capital letter (if number of words > xx)

[Brad Clements]
> Just curious if subject line capitalization can be used as an indicator.
>
> Either the percentage of characters that are caps..
>
> Or, percentage starting with a capital letter (if number of words > xx)
Supply a mod to tokenizer.py and I'll test it (eventually <wink>).  Note
that the tokenizer already *preserves* case in subject-line words, because
experiment showed that this was better than folding case away in this
specific context (but experiment also showed-- against my
expectations --that preserving case everywhere didn't make a significant
difference to either error rate -- the subject line is a special case for
this).

> If you've still got the summary files, please cvs up and try running cmp.py
> again -- in the process of generalizing cmp.py, you managed to make it skip
Woops.  I didn't have the summary files so I regenerated them using a
slightly different set of data.  Here are the results of enabling the
total unique fp went from 17 to 14 won    -17.65%
total unique fn went from 50 to 38 won    -24.00%
My test set is different than Tim's in that all the email was received
by the same account.  Also, my set contains email sent to me, not to
mailing lists (I use a different addresses for mailing lists).  If
people cook up more ideas I will be happy to test them.

[Tim]
> ...
> On my box the current system scores about 50 msgs per second (starting
> in memory, of course).
That was a guess.  Bothering to get a clock out, it was more like 80 per
second.  See?  A 60% speedup without changing a thing <wink>.

> Woops.  I didn't have the summary files so I regenerated them using a
> slightly different set of data.  Here are the results of enabling the
>     total unique fp went from 17 to 14 won    -17.65%
>     total unique fn went from 50 to 38 won    -24.00%
> My test set is different than Tim's in that all the email was received
> by the same account.  Also, my set contains email sent to me, not to
> mailing lists (I use a different addresses for mailing lists).
Enabling the Received headers works even better for me <wink>; here's the
f-n section from a quick run on 500-element subsets:
total unique fn went from 38 to 15 won    -60.53%
A huge improvement, but for wrong reasons ... except not entirely!  The most
powerful discriminator in the whole database on one training set became:
That's got nothing to do with BruceG, right?
was also a strong spam indicator across all training sets.  I'm jealous.
> If people cook up more ideas I will be happy to test them.
Neil, are using your own tokenizer now, or the tokenizer.Tokenizer.tokenize
generator?  Whichever, someone who's not afraid of their headers should try
adding mboxtest.MyTokenizer.tokenize_headers into the mix, once in lieu of
tokenizer.Tokenizer.tokenize_headers(), and again in addition to it.  Jeremy