> On Wed, 2006-03-15 at 11:10 -0500, James Ostheimer > wrote: > > One thing that really did help (the nulled > instances were not being > > collected at all before this) was removing any > stored references to the > > crawler threads. I was keeping a reference to > each running thread in a > > controller class to compute statistics on how well > I was doing (download > > speed). When I removed the reference so that each > thread was completely > > dereferenced (on its own) the memory started going > up much slower.
Keeping references to your threads should not be a problem unto itself, unless you are not using a thread pool. If you see the memory use slow down significantly when you stop doing that, then there are other issues with how you have designed the units of work, or the statistics reporting. I have also design/built a very large distributed crawler. It can be very painful. If you need some more advice, just ask. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
