Hey guys, I've combined a library from two current libraries: scrapy-redis and scrapy-rabbitmq: https://github.com/rdcprojects/scrapy-mq-redis
I found that scrapy-rabbitmq incorrectly uses rabbitmq for duplicate filtering which it cannot do. So, I added the dupefilter from scrapy-redis to it and made some other fixes. One of the fixes is sending acknowledgement to rabbitmq when the Request is processed so that RabbitMQ can mark the message as finished and remove it from queue. I plan to add one more feature which lets you add more urls while the spiders are running. Please check it out and let me know the feedback. -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
