It will not work as you expect. You have to get rid of duplicate urls by your own code, maybe the "in" operation can help you. The RFPDupeFilter only works in visited urls.
在 2016年5月3日星期二 UTC+8上午4:04:34，Antoine Brunel写道： > > Hello, > > I found out that Scrapy's duplicate url filter RFPDupeFilter is disabled > for urls set in start_urls. > How can I enable it? > > Thanks! > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
