Hi Josh, If you can have a look while gdb the atoms' content, you might have a better clue where the leak came from. The string itself pretty much explains where it was used for.
If your crawler frequently hit non-exist domain (dns lookup failure), have a look at earlier thread http://www.mail-archive.com/[email protected]/msg00505.html Ming On Thu, Oct 15, 2009 at 6:13 AM, Josh Triplett <[email protected]>wrote: > Hello, > > We use polipo as part of a web crawler, and it greatly simplifies our > code to not have to deal with caching ourselves; thank you for polipo. > > We seem to have run into an issue where polipo's memory usage grows > without bound. After running for a while, polipo will grow to 100M or > more. We tried configuring polipo to use less memory, but this did not > help. > > Investigating, I found the following entries in polipo's log: > > Discarded all objects, 1 + 3 objects left (4 chunks and 307966 atoms used). > Discarded all objects, 1 + 3 objects left (5 chunks and 313599 atoms used). > Discarded all objects, 1 + 3 objects left (5 chunks and 319717 atoms used). > Discarded all objects, 0 + 3 objects left (4 chunks and 325469 atoms used). > Discarded all objects, 0 + 3 objects left (4 chunks and 331110 atoms used). > Discarded all objects, 0 + 3 objects left (4 chunks and 336715 atoms used). > Discarded all objects, 1 + 3 objects left (6 chunks and 342191 atoms used). > Discarded all objects, 1 + 3 objects left (5 chunks and 347533 atoms used). > Discarded all objects, 1 + 3 objects left (5 chunks and 351679 atoms used). > > This suggests that the problem may lie with polipo's use of atoms, the > number of which continues to grow. > > We'd like to try to resolve this issue. Please let us know if we can > provide any further information. Also happy to test any potential > fixes. > > - Josh Triplett > > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart your > developing skills, take BlackBerry mobile applications to market and stay > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > http://p.sf.net/sfu/devconference > _______________________________________________ > Polipo-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/polipo-users > ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Polipo-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/polipo-users
