[Neale Pickett]
> I hacked up something to turn WordInfo into a tuple before pickling,
That's what WordInfo.__getstate__ does.
> and then turn the tuple back into WordInfo right after unpickling.
Likewise for WordInfo.__setstate__.
> Without this hack, my database was 21549056 bytes.  After, it's 9945088
bytes.
> That's a 50% savings, not a bad optimization.
I'm not sure what you're doing, but suspect you're storing individual
WordInfo pickles.  If so, most of the administrative pickle bloat is due to
that, and doesn't happen if you pickle an entire classifier instance
directly.
> So my question is, would it be too painful to ditch WordInfo in favor of
> a straight out tuple?  (Or list if you'd rather, although making it a
> tuple has the nice side-effect of forcing you to play nice with my
> DBDict class).
>
> I hope doing this sort of optimization isn't too far distant from the
> goal of this project, even though README.txt says it is :)
>
> Diff attached.  I'm not comfortable checking this in,
I think it's healthy that you're uncomfortable checking things in with
> +            # XXX: kludge kludge kludge.
comments <wink>.
> since I don't really like how it works (I'd rather just get rid of
WordInfo).
> But I guess it proves the point :)
I'm not interested in optimizing anything yet, and get many benefits from
the *ease* of working with utterly vanilla Python instance objects.  Lots of
code all over picks these apart for display and analysis purposes.  Very few
people have tried this code yet, and there are still many questions about it
(see, e.g., Jeremy's writeup of his disappointing first-time experiences
today).  Let's keep it as easy as possible to modify for now.  If you're
desparate to save memory, write a subclass?
Other people are free to vote in other directions, of course <wink>.

[Barry]
> Here's an interesting thing to test: discriminate words differently if
> they are on a line that starts with `>' or, to catch styles like
> above, that the first occurance on a line of < or > is > (to eliminate
> html).
Give me a mod to timtoken.py that does this, and I'll be happy to test it.
> Then again, it may not be worth trying to un-false-positive that
> Nigerian scam quote.
If there's any sanity in the world, even the original poster would be glad
to have his kneejerk response blocked <wink>.  OTOH, you know there are a
great many msgs on c.l.py (all over Usenet) that do nothing except quote a
previous post and add a one-line comment.  Remove the quoted sections from
those, and there may be no content left to judge except for the headers.  So
I can see this nudging the stats in either direction.  The only way to find
out for sure is for you to write some code <wink>.