Enrico, your problem is likely caused by the shell (bash?). "&" has a special meaning (see http://hacktux.com/bash/ampersand). To run curl (or any other console utility) on URLs which have & in them add *single* quotes around the URL, like 'http://example.com?foo=bar&egg=spam'.
воскресенье, 30 ноября 2014 г., 5:45:19 UTC+5 пользователь Enrico Jr Tuvera написал: > > I had a similar issue when dealing with curl and the -d option, it was > truncating everything past the '&' character in query strings. Maybe > something similar is happening with shell? > > Maybe try percent-encoding the URL before passing it to scrapy shell (i.e > http://www.google.com/%3Fgws_rd%3Dssl%23q%3D%2522german%2Bbeer%2Bnear%2522%26start%3D0 > > )? I'm not aware that it treats '#' as a special character but its worth a > shot. > > On Saturday, November 8, 2014 2:13:06 AM UTC+8, Jo Chasinga wrote: >> >> In scrapy shell when I tried using fetch on a Google' search result page: >> >> $ scrapy shell " >> http://www.google.com/?gws_rd=ssl#q=%22german+beer+near%22&start=0"; >> >> I got a response without the query string after '#' [s] request <GET >> http://www.google.com/?gws_rd=ssl#q=jeffrey+m+liebmann> [s] response >> <200 http://www.google.com/?gws_rd=ssl> >> >> Is this issue belongs to scrapy or Google? Tried pasting the whole URL + >> query string and Google led me to the results just fine. >> > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
