Looks like your spiders don't finished. That's might be cause the job is greater than you planned or there's some error in the logic of your code blocking your spider to finish.
There's too little information to help you. Could you tell us what you're trying to scrape and how? On Tuesday, December 31, 2013 1:08:22 PM UTC-2, Pablo Hoffman wrote: > > The maximum number of running jobs is given by the max_proc and > max_proc_per_cpu config settings: > http://scrapyd.readthedocs.org/en/latest/config.html#max-proc > http://scrapyd.readthedocs.org/en/latest/config.html#max-proc-per-cpu > > After that number is reached, new jobs will stay in pending queue, and > they will start once the running jobs have finished. > > You can increase those settings in order to run more in parallel. Is that > what you were asking? > > > On Sun, Sep 22, 2013 at 7:48 AM, mourad mourafiq > <[email protected]<javascript:> > > wrote: > >> Hy guys, >> >> I am new user of scrapy, I am managed to write some spiders, some custom >> extensions and I am using scrapy with django and celery. >> >> My problem is with scrapyd, when I schedule jobs via the API, scrapyd >> manage to run 16 jobs (which is normal, default settings), but this jobs >> are always running, which means that all new scheduled jobs stay pending >> waiting for this running jobs to end. >> >> Is this a normal behaviour? How can change their state to finished >> without killing scrapyd? >> >> thank you very much for your help, >> Mourad >> >> -- >> You received this message because you are subscribed to the Google Groups >> "scrapy-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected]<javascript:> >> . >> Visit this group at http://groups.google.com/group/scrapy-users. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/groups/opt_out.
