[nodejs] Re: Web scraping and Memory leaking issue

2012-08-13 Thread ec.developer
After upgrading from v0.8.0 to v0.8.6 - app seems to work without any memory leaks. After checking over 4 thousands of pages, it uses around 45mb - 60mb. On Monday, July 2, 2012 4:08:13 PM UTC+3, ec.developer wrote: Hi all, I've created a small app, which searches for Not Found [404]

[nodejs] Re: Web scraping and Memory leaking issue

2012-07-25 Thread ec.developer
Aflter a longer test, got these results: Mem: 795 MB Requests running: 11 Grabbed: 98050 Links queue: 1160553 Links grabbed and links queue are still stored in mongodb. On Monday, July 2, 2012 4:08:13 PM UTC+3, ec.developer wrote: Hi all, I've created a small app, which searches for Not

Re: [nodejs] Re: Web scraping and Memory leaking issue

2012-07-25 Thread Matt
Can you try switching to cheerio instead of jsdom? I found that jsdom consumed way too much ram and was slow. On Wed, Jul 25, 2012 at 8:57 AM, ec.developer ec.develo...@gmail.comwrote: Aflter a longer test, got these results: Mem: 795 MB Requests running: 11 Grabbed: 98050 Links queue:

Re: [nodejs] Re: Web scraping and Memory leaking issue

2012-07-25 Thread ec.developer
Have you checked the latest code? - http://pastebin.com/GyefDREM I'm using cheerio instead of jsdom. On Wednesday, July 25, 2012 5:45:52 PM UTC+3, Matt Sergeant wrote: Can you try switching to cheerio instead of jsdom? I found that jsdom consumed way too much ram and was slow. On Wed, Jul

Re: [nodejs] Re: Web scraping and Memory leaking issue

2012-07-25 Thread ec.developer
Thanks, will try phantomjs. But I'd like though to got the answer - where is the memory leak. I'd appreciate if smbd could point me to the right direction. Thanks again. On Wednesday, July 25, 2012 7:14:55 PM UTC+3, Davis Ford wrote: I know this is a node.js list, but I've been doing a lot