You can build something like a URL callback. When you request the scrapyd to start crawling, pass an url callback to system that your spider will use to update your system about the spider's state.
On Monday, December 16, 2013 3:23:24 AM UTC-2, Gheorghe Chirica wrote: > > Hi. > > I want to hook some custom logic to scrapyd spider state events ( > Running,Pending,Finished). I know that there is an API method for that, > but I don't want to call the API each time I need to find the spider state. > > I need to update DB automatically, when spider running via scrapyd, > changes it's status. > > I was wandering how can I achieve this with less overhead? From the code > seems to me that I need to create a custom launcher and poller which will > handle this states. > > Is this a good approach ? > > Another idea is to not touch scrapyd code at all and work with scrapy > spider signals: > - spider_opened - running > - spider_closed - finished > - ?? - pending (here we can't use spider_idle for > pending) > > > > Thx. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/groups/opt_out.
