On Wed, 5 Jan 2011 09:11:26 -0500, BC wrote: >first one on my list would always get ALL the work. I realize it doesn't >make a difference as long as the work gets done, but for me personally, I >would rather see all the servers doing approximately the same amount of >work.
Actually, I suppose what's on the other end could give you a perfectly rational reason to worry about it. Since all the workers are on the same VM system, other users are affected equally no matter how their work is distributed, but if they're talking to different mail servers then using one of them all the time puts an extra load on its mail server and could slow down mail for that server's other users. >I believe I can do that with the idea to extend the sort key using the >random stage, and I plan to try that. Yep, that's the way to go for random distribution. Of course, if some of the servers run just a *little* faster than some others they'll still be unfairly loaded. Another approach you might consider if you need to avoid that is distributing files round-robin among workers whose loads are below either some fixed threshold or some average of the load at the moment. Query once, save the list of available servers, then give one file to each before making another query. ¬R "I love Blip just because it's the absolute opposite of fun" http://users.bestweb.net/~notr/travelog/19990710.html --Kibo
