Hello Vaibhav Jain, this looks like a Splash-specific question. You may find more help through their gitter channel at https://gitter.im/scrapinghub/splash
BR, /Paul. On Mon, Dec 12, 2016 at 8:59 AM, Vaibhav Jain <[email protected]> wrote: > I am trying to scrape a few dynamic websites using Splash for Scrapy in > python. However, I see that Splash fails to wait for the complete page to > load in certain cases. A brute force way to tackle this problem was to add > a large wait time (eg. 5 seconds in the below snippet). However, this is > extremely inefficient and still fails to load certain data (sometimes it > take longer than 5 seconds to load the content). Is there some sort of a > wait-for-element condition that can be put through these requests? > > yield SplashRequest( > url, > self.parse, > args={'wait': 5}, > 'User-Agent':"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 > (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36", > }) > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
