Hello,
I'm trying to crawl a website.com/places.asp?id=100
*When i'm on the browser* i can get the <table> where the information
exists, the XPath '//p[@class="txt"]/table/...' with a firebug extension
(FirePath)
But when i'm selecting with *scrapy shell or crawling the spider*, the*
<table>* after the first 'p[@class="txt"]' *simply doesn't exist, like if
it was not created yet.*
class MySpider(BaseSpider):
name = 'xpto'
allowed_domains = ['website.com']
start_urls = [
'http://www.website.com/places.asp?id=100'
]
def parse(self, response):
sel = Selector(response)
places = sel.xpath(
'//p[@class="txt"]/table//td[@class="txtm"]/a/@href').extract()
for place in places:
print place
I've thought that was created by AJAX method, but it doesn't have any one.
Then i tried to get the HTML page with:
def parse(self, response):
open('test.html', 'wb').write(response.body)
And the table exists!
How can i get it to Selector?
Maybe a ASP thing ?
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
