Stephane,
What steps did you take to determine there is no JS involved? I loaded the
page w/o javascript, and while that area of the page had the stub content
("Visitas"), the actual data was written there with an ajax request.
You can still load that data using scrapy, it'll just take an additional
request. The server returns the number of visits in XML, via the script at
http://www.fincaraiz.com.co/WebServices/Statistics.asmx/GetAdvertVisits?idAdvert=1337688&idASource=40&idType=1001
(try loading that script and you'll see the # of visits for the page you
provided in the original email).
There is another ajax request that returns "True" for that page, but I'm
not sure what the data's actual meaning is. Still, it may be useful:
http://www.fincaraiz.com.co/WebServices/Statistics.asmx/DetailAdvert?idAdvert=1337688&idType=1001&idASource=40&strCookie=13/11/2014:19-05419&idSession=10hx5wsfbqybyxsywezx0n1r&idOrigin=44
(I cross-posted this answer to your SO question. If you don't mind, please
send me some sweet sweet SO karma by accepting the answer.)
Thanks,
Travis
On Thu, Nov 13, 2014 at 9:56 AM, Stephane Leonard <[email protected]>
wrote:
> Already posted this on stackoverflow, without an answer. I think it's a
> very relevant question though.
>
> The story : all wanted fields but one get scraped perfectly. The content
> of the missing field simply doesn't show up in the Scrapy response (as
> checked in the scrapy shell), while it does show up when i use my browser
> (actually any browser) to visit the page. In the scrapy response, the
> expected tags are there, but not the text between the tags.
>
> There's no JavaScript involved, but it is a variable that is provided by
> the server (it's the current number of visits to that particular page). No
> iframe involved either.
>
> Already set the user agent (in the settings-file) to match my browser.
> Already set the download delay (in the settings-file) to 5.
>
> -
>
> The page :
>
> http://www.fincaraiz.com.co/apartamento-en-venta/bogota/salitre-det-1337688.aspx
> -
>
> Xpath to the wanted element : //*[@id="numAdvertVisits"]
>
> What could be the cause of this mystery ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "scrapy-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/scrapy-users.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
