To be short, I want to be able to do this:
payload = {"project": settings['BOT_NAME'],
"spider": crawler_name,
"start_urls": ["http://www.foo.com"]}
response = requests.post("http://192.168.1.41:6800/schedule.json";,
data=payload)
And direct my spider to crawl specific url, but the seemingly right code
above will end up with the following error:
File "/usr/lib/pymodules/python2.7/scrapy/spider.py", line 53, in
make_requests_from_url
return Request(url, dont_filter=True)
File "/usr/lib/pymodules/python2.7/scrapy/http/request/__init__.py", line 26,
in __init__
self._set_url(url)
File "/usr/lib/pymodules/python2.7/scrapy/http/request/__init__.py", line 61,
in _set_url
raise ValueError('Missing scheme in request url: %s' % self._url)
exceptions.ValueError: Missing scheme in request url: h
Meaning that somehow, "http://www.foo.com"; didn't get passed as
request.url, instead, the first letter of our url gets assigned ( self._url
= 'h' ), I can't figure out how to correctly pass my url to the scrapyd API.
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
