Justin Mason <yyyy@example.com>,
	SpamAssassin Developers <SpamAssassin-devel@lists.sourceforge.net>
No, hits=-3.7 required=5.0
	tests=AWL,EMAIL_ATTRIBUTION,FORGED_RCVD_TRAIL,HABEAS_SWE,
	      IN_REP_TO,T_NONSENSE_FROM_30_40,USER_AGENT_PINE
	version=2.50-cvs

Daniel Quinlan wrote:
DQ> jm@jmason.org (Justin Mason) writes:
DQ>
DQ> > But I think rsync is key; it's very efficient bandwidth-wise, and it
DQ> > allows you to select which ones you want just as well as wget would.
DQ> > (Bandwidth is a much bigger issue for me and some other of the europeans
DQ> > involved, than it would be for you guys ;)  IMO it's by far the most
DQ> > efficient and scriptable way to do this stuff, these days.
DQ>
DQ> It must be horrible being a dial-up user in 2002.  If we can *automate*
DQ> rsync submission with good *authentication*, it would be fine with me.
DQ> The current rsync method is not sufficiently authenticated, though.
I'll take a look at it.  I'm sure there's got to be some reasonly easy way to
have rsync carry itself over ssh and secure access using ssh keys.
DQ> > I suggest we use the hughes-family.org server for rsyncing, as we do for
DQ> > the corpus_submit stuff -- in a separate subdirectory, though! -- then
DQ> > I'll take a look at getting a nightly hit-frequencies-collation system
DQ> > going.
Ok, I'll take a look at it.
C

[Skip]
> Hmmm.   How about you create empty Data/Ham/Set[12345], stuff all your
> files into a Data/Ham/reservoir folder, then run the rebal.py script to
> randomly parcel messages out to the various real directories?
I'm afraid rebal is quadratic-time in the # of msgs it shuffles around --
since it was only intended to move a few files around, it's dead simple.
An easy thing is to start the same way:  move all the files into a single
directory.  Then do random.shuffle() on an os.listdir() of that directory.
Then it's trivial to split the result into N slices, and move the files into
N other directories accordingly.
> I suspect you can pull the same stunt for your Data/Spam stuff.
Yup!