I am trying to scrape a few dynamic websites using Splash for Scrapy in
python. However, I see that Splash fails to wait for the complete page to
load in certain cases. A brute force way to tackle this problem was to add
a large wait time (eg. 5 seconds in the below snippet). However, this is
extremely inefficient and still fails to load certain data (sometimes it
take longer than 5 seconds to load the content). Is there some sort of a
wait-for-element condition that can be put through these requests?
yield SplashRequest(
url,
self.parse,
args={'wait': 5},
'User-Agent':"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36",
})
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
