Hi Shaumik, Nice to meet you!
суббота, 8 марта 2014 г., 18:59:43 UTC+6 пользователь Shaumik Daityari написал: > > Hi, > > My name is Shaumik and I am a 4th year undergrad at IIT Roorkee. I am > interested in developing an HTTP API for Scrapy Spiders[1]. > > I have extensive experience in working with Python and Django in the past. > I have used BeautifulSoup and Mechanize to do some scraping in my free time > (some of the scraping scripts can be found here[2]). I have also been a > part of GSoC 2013 working on the ATutor project of Inclusive Design > Institute. > > I have a few doubts regarding the project. Firstly, is it necessary to go > ahead with Twisted? Since the project mentions adding interactivity, I > think developing a project in Django with some JavaScript can go a long way > in adding the interactivity. In fact, what I have in mind is something > similar to Kimono[3]- something that recently got a lot of popularity on > Hacker News [4]. > Scrapy already uses Twisted, and HTTP API doesn't need Javascript. Adding interactivity is an interesting project, and it is a good idea, but I don't think this is what "HTTP API" project about. "IPython IDE for Scrapy" looks a bit more relevant - it assumes some interactivity. If you are interested in "smart" extraction techniques check https://github.com/scrapy/scrapely library. We are building a web-based GUI for scrapely at Scrapinghub and are going to open-source it soon; Shane knows more about it - it is in the same vein as Kimono. > > Secondly, I hope that for this project, it is not necessary to solve a bug > within scrapy but instead, have extensive knowledge of the usage of scrapy. > Would it be sufficient to push some code to GitHub demonstrating the use of > Scrapy on some website to get hold of some data that I may want? > No, demonstrated knowledge of Scrapy is not enough unfortunately, and a contribution to Scrapy or to a related project is required. Such contribution shows that a student can communicate well and work in an open-source environment, collaborating with other developers. Also, it is told a contribution is a requirement here: https://wiki.python.org/moin/SummerOfCode/ApplicationTemplate2014 > > Looking forward to a reply. > > [1] > https://github.com/scrapy/scrapy/wiki/GSoC-2014-Ideas#wiki-http-api-for-scrapy-spiders > [2] https://github.com/sdaityari/web-scraping > [3] http://kimonolabs.com > [4] https://news.ycombinator.com/item?id=7066479 > > -- > Shaumik Daityari, > Integrated M. Tech. (4th year), > Geological Technology, > Department of Earth Sciences, > Indian Institute of Technology Roorkee, > Uttarakhand > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
