Already posted this on stackoverflow, without an answer. I think it's a very relevant question though.
The story : all wanted fields but one get scraped perfectly. The content of the missing field simply doesn't show up in the Scrapy response (as checked in the scrapy shell), while it does show up when i use my browser (actually any browser) to visit the page. In the scrapy response, the expected tags are there, but not the text between the tags. There's no JavaScript involved, but it is a variable that is provided by the server (it's the current number of visits to that particular page). No iframe involved either. Already set the user agent (in the settings-file) to match my browser. Already set the download delay (in the settings-file) to 5. - The page : http://www.fincaraiz.com.co/apartamento-en-venta/bogota/salitre-det-1337688.aspx - Xpath to the wanted element : //*[@id="numAdvertVisits"] What could be the cause of this mystery ? -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
