I'd recommend using PhantomJS or another headless full browser, if you're interacting with JS. Scrapy has something along these lines (Splash? maybe), but I never had much luck with it.
Generally when I'm scraping a js heavy site, i'll do everything I can to not need a full browser. But when I do, I use PhantomJS with some custom glue code (not using scrapy, in other words). I just never found the scrapy/js compatibility factor to be too high. Part of that is inherent to the issue -- adding a full stack, albeit headless, browser, increases system complexity, which always frustrated me when it didn't work correctly. On Tue, Mar 15, 2016 at 3:30 PM, Sean <[email protected]> wrote: > I need to scrape a website, where landing page A redirects to page B using > Javascript, and then redirects back to page A--all while using cookies. Is > scrapy a bad option? I've already done most of what I needed in Selenium, > except that: > > 1) downloading images in Selenium is difficult (many StackOverflow pages > confirm this) > 2) Selenium and scrapy don't really interact > <http://stackoverflow.com/questions/28420078/scrapy-with-selenium-for-a-webpage-requiring-authentication> > 3) Mechanize doesn't handle javascript well either > <http://stackoverflow.com/questions/802225/how-do-i-use-mechanize-to-process-javascript> > > Does anyone have sample snippets of how to do use scrapy in such a > scenario? Should I use mechanize instead? > > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
