Hi, My name is Shaumik and I am a 4th year undergrad at IIT Roorkee. I am interested in developing an HTTP API for Scrapy Spiders[1].
I have extensive experience in working with Python and Django in the past. I have used BeautifulSoup and Mechanize to do some scraping in my free time (some of the scraping scripts can be found here[2]). I have also been a part of GSoC 2013 working on the ATutor project of Inclusive Design Institute. I have a few doubts regarding the project. Firstly, is it necessary to go ahead with Twisted? Since the project mentions adding interactivity, I think developing a project in Django with some JavaScript can go a long way in adding the interactivity. In fact, what I have in mind is something similar to Kimono[3]- something that recently got a lot of popularity on Hacker News [4]. Secondly, I hope that for this project, it is not necessary to solve a bug within scrapy but instead, have extensive knowledge of the usage of scrapy. Would it be sufficient to push some code to GitHub demonstrating the use of Scrapy on some website to get hold of some data that I may want? Looking forward to a reply. [1] https://github.com/scrapy/scrapy/wiki/GSoC-2014-Ideas#wiki-http-api-for-scrapy-spiders [2] https://github.com/sdaityari/web-scraping [3] http://kimonolabs.com [4] https://news.ycombinator.com/item?id=7066479 -- Shaumik Daityari, Integrated M. Tech. (4th year), Geological Technology, Department of Earth Sciences, Indian Institute of Technology Roorkee, Uttarakhand -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
