On Sat, Jun 20, 2015 at 12:24 PM Vasco <[email protected]> wrote:
> Hi Julia, > > Congrats to you and the other contributors for reaching this milestone! > The release notes show some very interesting changes! Particularly, does > per spider settings mean we can have different pipelines for different > spiders within the same project? For example, I currently have a lot of > different projects that differ only in one or two pipelines, but they share > a lot of pipelines too (which I now define in a separate package that I > make available to all projects). If I understand things correctly, with 1.0 > I could put all spiders in one project and specify different pipeline paths > for each spider. If my understanding is correct, is this something you > would typically suggest users to do in 1.0? > > You can definitely redefine the spider pipelines with *custom_settings *and keep all your spiders in a single project. Actually, per-spider settings were introduced to avoid creating projects for simple spiders, since *custom_settings* lets you forget about creating a *settings.py* file to hold all your redefined settings. It surely opens up more code organization possibilities. There are still advantages to separate spiders into different projects, for example, keeping thousands of unrelated spiders with nothing in common in a single project is probably not a good a idea, so I think users should listen to suggestions for their particular use-cases. > I also noticed that the scrapyd package on pypi hasn't been updated in > almost two years. Is using scrapyd to manage scrapy spider versions/runs > considered current best practice? > > That was a mistake on our side, we tagged a new version not so long ago and forgot to deploy it to PyPI :( Thanks for the heads up, I've just deployed it: https://github.com/scrapy/scrapyd/releases/tag/1.1.0 Scrapyd is mostly community driven so its development deeply relies on contributions, but it's still our recommended method to deploy spiders: http://scrapy.readthedocs.org/en/1.0/topics/deploy.html > Congrats again! Best, > Vasco > > Thanks for the support! -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
