Thanks pablo for your reply, you are right, I did't fully explain what I did because at first, I thought it is something that needs to be set in scrapy, but I just realized it is a server problem.
On Tue, Jan 7, 2014 at 1:45 PM, Ali Bozorgkhan <[email protected]>wrote: > I just found the solution: > > > http://stackoverflow.com/questions/19995855/increase-max-open-files-for-ubuntu-upstart-initctl > > The problem is as scrapyd starts as a service, it will ignore any > configuration in '/etc/security/limits.conf'. I solved the problem by > adding this line to '/etc/init/scrapyd.conf': > > limit nofile 524288 524288 > > > > > On Tue, Jan 7, 2014 at 1:33 PM, Pablo Hoffman <[email protected]>wrote: > >> How did you increased the ulimit?. Are you sure it was increased properly? >> >> >> On Tue, Jan 7, 2014 at 3:48 PM, Ali Bozorgkhan >> <[email protected]>wrote: >> >>> Hi, >>> >>> I am running 25 jobs on a single machine, each with 50 concurrent >>> requests. After a 10 minutes or so, I get this error in my proxy middleware: >>> >>> Couldn't bind: 24: Too many open files. >>> >>> I tried increasing the ulimit on the system but no luck. Is there anyway >>> I could fix it? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "scrapy-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/scrapy-users. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "scrapy-users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/scrapy-users/rArexq4tI7I/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/scrapy-users. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/groups/opt_out.
