No problem Ali, thanks for posting follow up with your solution. I think you should also be able to adjust the ulimit by adding the following line to /etc/default/scrapyd (which has less chances of being overwritten by a scrapyd package upgrade):
ulimit -n 524288 Pablo. On Tue, Jan 7, 2014 at 7:46 PM, Ali Bozorgkhan <[email protected]>wrote: > Thanks pablo for your reply, you are right, I did't fully explain what I > did because at first, I thought it is something that needs to be set in > scrapy, but I just realized it is a server problem. > > > On Tue, Jan 7, 2014 at 1:45 PM, Ali Bozorgkhan <[email protected]>wrote: > >> I just found the solution: >> >> >> http://stackoverflow.com/questions/19995855/increase-max-open-files-for-ubuntu-upstart-initctl >> >> The problem is as scrapyd starts as a service, it will ignore any >> configuration in '/etc/security/limits.conf'. I solved the problem by >> adding this line to '/etc/init/scrapyd.conf': >> >> limit nofile 524288 524288 >> >> >> >> >> On Tue, Jan 7, 2014 at 1:33 PM, Pablo Hoffman <[email protected]>wrote: >> >>> How did you increased the ulimit?. Are you sure it was increased >>> properly? >>> >>> >>> On Tue, Jan 7, 2014 at 3:48 PM, Ali Bozorgkhan >>> <[email protected]>wrote: >>> >>>> Hi, >>>> >>>> I am running 25 jobs on a single machine, each with 50 concurrent >>>> requests. After a 10 minutes or so, I get this error in my proxy >>>> middleware: >>>> >>>> Couldn't bind: 24: Too many open files. >>>> >>>> I tried increasing the ulimit on the system but no luck. Is there >>>> anyway I could fix it? >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "scrapy-users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/scrapy-users. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "scrapy-users" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/scrapy-users/rArexq4tI7I/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/scrapy-users. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/groups/opt_out.
