Asheesh, that looks awesome! Thank you for sharing. How do other people feel about the availability/discoverability of 3rd party addons to scrapy? I suspect there is a lot of neat things like Asheesh's project, but I don't know where to find them (other than github search, but that's more effort :P ).
On Wed, Jan 28, 2015 at 11:33 PM, Asheesh Laroia <[email protected]> wrote: > Hi all, > > In https://github.com/openhatch/oh-bugimporters (see also docs at > https://oh-bugimporters.readthedocs.org/en/latest/ ) we do something > similar with Scrapy. Feel free to use our example as inspiration on how to > do it, or how *not* to do it! > > I'm excited that you're interested in doing something like this. I hope > other framework maintainers might be able to chime in. I'm "just" another > user of Scrapy like yourself. > > On Wed, Jan 28, 2015 at 11:00 PM, user12345 <[email protected]> > wrote: > >> Hi, >> >> Thanks for the interest. I'm looking for specifically for replies from >> project maintainers/contributors to see whether or this is feasible in the >> first place - don't want to waste anyone's time, including mine. If it is, >> and I can do it on my own, I'd love to tackle it with some design guidance, >> to make sure I'm working with the framework instead of against it. >> >> >> On Wednesday, January 28, 2015 at 6:15:16 PM UTC-8, user12345 wrote: >>> >>> I'm working on a scrapy project where a "rabbit client" and "crawl >>> worker" work together to consume scrape requests from a queue. These >>> requests have more configuration than a start_url - it could be something >>> like url and a set of xpaths, or a domain-specific configuration, like >>> site-specific product ID (from which we programmatically build the url) and >>> optional identifiers like color, style, and size to further specify the >>> item one wants to scrape. >>> >>> I'm wondering if it would be desirable to have built-in support for more >>> specific "crawl configurations" like this within the framework? If that's >>> the case, I'd be more than happy to have a design discussion and hash out >>> the details. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "scrapy-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/scrapy-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
