All sorted, user error...
I had mismatching allowed domains and start urls....
El dimecres, 19 agost de 2015 10:08:59 UTC+1, Isaac Perez va escriure:
>
> No luck, same behaviour.
> It goes over all the start url, doesn't follow anything....
>
> El dimarts, 18 agost de 2015 21:00:35 UTC+1, Isaac Perez va escriure:
>>
>> I havent, I thought allow=() would do the same, but going to try straight
>> away and let you know, thanks!
>>
>> El dimarts, 18 agost de 2015 17:34:57 UTC+1, Travis Leleu va escriure:
>>>
>>> Did you try setting allow=('.*',) in your Rule definition?
>>>
>>> On Tue, Aug 18, 2015 at 8:32 AM, Isaac Perez <[email protected]>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm having some problems with the CrawlSpiders I'm creating, not one of
>>>> them follows the links more than one level deep when the rule has the
>>>> callback defined.
>>>> So far I sort of managed to get it working with specifying the links
>>>> for the categories in the start_urls.
>>>> But this spider:
>>>> http://pastebin.com/rRLAP1X1
>>>>
>>>> Is not even parsing the links for the start links.
>>>> So it only crawls the start pages and that's it.
>>>>
>>>> Not sure what I'm doing wrong, as I understand that if you have
>>>> Follow=True in the rule, it will crawl, scrap and follow.
>>>> But this doesn't seem to be happening for any of my spiders.
>>>> For the pasted spider, what is it missing that doesn't even follow the
>>>> links from the start_urls?
>>>>
>>>> Any ideas?
>>>>
>>>> Thanks,
>>>> Isaac
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "scrapy-users" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at http://groups.google.com/group/scrapy-users.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
