I am trying to scrape data from this page:
https://angel.co/armando-biondi
When the page loads JS loads the Investment portion of the page. I am using
Scrapy-Splash, but it does not seem to wait before the JS has loaded the
page with the investment part.
Following is my start request sample. To circumvent the problem I had to
use Selenium.
def start_requests(self):
url = 'https://angel.co/armando-biondi'
yield scrapy.Request(url, self.parse, meta={
'splash': {
'args': {
'wait' : 10.0,
},
'endpoint': 'render.html',
}
})
Enter code here...
Complete code is here: Code-gist
<https://gist.github.com/shafaq/3da8982169efd6d06322a1a90ac0176a>
Thanks.
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
