Christoph, You can scale up scrapy using different approaches. It depends on many factors: using scrapyd, using celery, etc...
I will suggest you find what suits you best in terms of storage, network and computational requirements because there's big performance differences. To me, one of the best fits for storage is using HBase, or even better Hypertable. Cheers, Jordi 2014-05-24 1:00 GMT+10:00 christoph <[email protected]>: > Hi, > > I wonder what is a good choice for an environment for a scalable scrapy > project similar to scrapinghub? > Starting with a single vserver/root-server for crawling and data storing > with the possibility to add additional servers when I need more scraping > power or database space. > According to a blog entry ( > http://blog.scrapinghub.com/2013/07/26/introducing-dash/), scrapinghub is > using Cloudera CDH (run on which OS?) and they store their data in HBase. > So this is a good choice? > > Is there any information how to setup scrapy in a CDH environment and > saving data into HBase? > > Thank you, > Christoph > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
