working like charm thanks alot. On Thursday, 25 November 2010 21:02:17 UTC+5:30, Pablo Hoffman wrote: > > Not for the moment. > > But there are some nice alternatives for achieving that functionality. For > example, you can choose a spider attribute to define which pipelines will > be > enabled for each spider, and then check that attribute in your pipelines. > > Here's how your spiders would look: > > class SomeSpider(CrawlSpider): > pipelines = ['first'] > > class AnotherSpider(CrawlSpider): > pipelines = ['first', 'second'] > > And your pipelines: > > class FirstPipeline(object): > � def process_item(self, item, spider): > if 'first' not in getattr(spider, 'pipelines', []): > return item > > # ... pipeline code here ... > > > class SecondPipeline(object): > � def process_item(self, item, spider): > if 'second' not in getattr(spider, 'pipelines', []): > return item > > # ... pipeline code here ... > > > Btw, this code can be easily made more performant by using sets instead of > lines for the pipelines attribute, and by caching the pipelines per spider. > > Pablo. > > On Thu, Nov 25, 2010 at 07:14:12AM -0800, vitsin wrote: > > hi, > > are you planning may be to add support for custom pipeline per spider? > > 10x, > > --vs > > > > On Nov 25, 9:28�am, Pablo Hoffman <[email protected]> wrote: > > > Hi vitsin, > > > > > > You can't override settings like this in your spiders like your code > does: > > > > > > � � class FirstSpider(CrawlSpider): > > > � � � � settings.overrides['ITEM_PIPELINES'] = ... > > > > > > And you can't customize the item pipelines per spider. > > > > > > What you could do is check the spider in the process_item() of your > pipeline, > > > and ignore certain ones. For example: > > > > > > � � def process_item(self, item, spider): > > > � � � � if spider.name not in ['myspider1', 'myspider2', 'myspider3']: > > > � � � � � � return item > > > > > > Hope this helps, > > > Pablo. > > > > > > On Wed, Nov 24, 2010 at 08:19:43PM -0800, vitsin wrote: > > > > hi, > > > > how supposed to be defined two pipelines if each one of them doing > > > > completely different SQL queries? > > > > Two classes in the same pipelines.py file? > > > > I've tried: > > > > > > > # Both, ScanFirst and ScanSecond, are SQLAlchemy mappings to exsiting > > > > DB tables. > > > > from .tables.scanfirst import ScanFirst > > > > from .tables.scansecond import ScanSecond > > > > from test.items import FirstItem > > > > from test.items import SecondItem > > > > > > > class FirstPipeline(object): > > > > � � def process_item(self, item, spider): > > > > � � � � scan_res = ScanFirst( ... ) > > > > > > > class SecondPipeline(object): > > > > � � def process_item(self, item, spider): > > > > � � � � scan_res = ScanSecond( ... ) > > > > > > > When later on, spider with SecondPipeline activated from: > > > > class FirstSpider(CrawlSpider): > > > > � � settings.overrides['ITEM_PIPELINES'] = > > > > ['test.pipelines.SecondPipeline'] > > > > � � ... (rest of spider code) > > > > > > > than I see that code for FirstPipeline activated, why? > > > > > > > 10x, > > > > --vs > > > > > > > -- > > > > You received this message because you are subscribed to the Google > Groups "scrapy-users" group. > > > > To post to this group, send email to [email protected] > <javascript:>. > > > > To unsubscribe from this group, send email to > [email protected] <javascript:>. > > > > For more options, visit this group athttp:// > groups.google.com/group/scrapy-users?hl=en. > > > > -- > > You received this message because you are subscribed to the Google > Groups "scrapy-users" group. > > To post to this group, send email to [email protected] > <javascript:>. > > To unsubscribe from this group, send email to > [email protected] <javascript:>. > > For more options, visit this group at > http://groups.google.com/group/scrapy-users?hl=en. > >
-- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
