I have to scrape thousands of different websites, as fast as possible.
On a single node process I was able to fetch 10 urls per second.
Though if I fork the task to 10 worker processes, I can reach 64 reqs/
sec.
Why is so?
Why I am limited to 10 reqs/sec on a single process and have to spawn
On Fri, Jun 1, 2012 at 2:00 PM, Mick mickodan...@gmail.com wrote:
I have to scrape thousands of different websites, as fast as possible.
On a single node process I was able to fetch 10 urls per second.
Though if I fork the task to 10 worker processes, I can reach 64 reqs/
sec.
Why is so?
