I'm just getting started with Scrapy and trying to figure out if we could use it for a project at work...
We have a large site and have a lot of content in Akamai for failover. Our problem however is the failover content gets stale over time. I was thinking I could spider our site, pull down a page, and do a quick comparison with the same page in failover. Same size? Same content? Etc. If yes - continue to next file, if not - refresh the failover content. Scrapy seems like a good starting point and between extensions, signals and middleware it seems like I could create the workflow I'm after unless I'm missing something. Anyone have any feedback or opinions of if this would work or not :) Thanks! Jim -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
