One of the project ideas for GSoC 2014 is meant to address this topic: https://github.com/scrapy/scrapy/wiki/GSoC-2014-Ideas#http-api-for-scrapy-spiders
It was one of the ideas that got closest to getting accepted, but it wasn't in the end. The idea is still pretty solid and useful, so maybe someone (you?) can pick it up and work on it, regardless of GSoC. If someone is already actively working on it, please chime in! On Wed, Mar 26, 2014 at 12:10 PM, Richard Simpson <[email protected]>wrote: > I've had a search around SO and the web in general, and haven't been able > to find a solution to my issue; so I thought I would raise a question here. > > I'm currently playing around with Scrapy, and am successfully retrieving > the data I need from my target website. > > The creation of the spider is part of a larger project, and so I need to > work out a way of getting the results from the spider into my website. > > I don't want to access the database with the spider, because there is a > variety of logic that would need to be applied to the data before it was > stored, and I wouldn't want to have to repeat that logic elsewhere. > > What I'm thinking is perhaps a webservice, that receives a JSON object as > a request; and returns a JSON object with the result. > > I'm fairly new to Scrapy and Python in a websetting, but I have read the > Scrapy docs. Whilst there is a webservice for Scrapy, that seems to be more > to do with the monitoring of spiders; and not returning results. > > Any help and thoughts would be greatly appreciated. > > Thanks, > > > > (Originally posted on StackOverflow: http://stackoverflow.com/questions/ > 22653821/accessing-scrapy-via-a-webservice?noredirect=1#comment34522584_ > 22653821) > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
