The target web server replies with an empty body when the user agent is the scrapy's default. I'm able to see the content by changing the user agent:
$ scrapy shell http://www.stvalentinshop.com/all/faguo-linden-camel.html--set USER_AGENT=googlebot Regards Rolando On Mon, Mar 31, 2014 at 8:20 AM, Andreas Bloch <[email protected]> wrote: > I'm receiving empty response bodies on some sites: > > In [1]: fetch('http://www.stvalentinshop.com/all/faguo-linden-camel.html') > > In [2]: type(response) > Out[2]: scrapy.http.response.html.HtmlResponse > > In [3]: response.body > Out[3]: '' > > Doing a simple urlopen with urllib works... > > In [9]: response = urllib2.urlopen(' > http://www.stvalentinshop.com/all/faguo-linden-camel.html').read() > In [10]: response (works) > > What it wrong here? > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
