Hi guys,
I have 2 start_urls , each of them have 100 pages with the pattern "resumes/
url1/page-\d+"
I am only interested in first 30 pages of each start_url. In other words, I
want to crawl "resumes/url1/page-\d+" where *\d+ <= 30*
Is there an option I can specify under "rules" ?
OK to crawl: resumes/something/page-20/
NOT OK to crawl: resumes/something/page-31/
start_urls = [
"url1,"url2"
]
rules = [
Rule(SgmlLinkExtractor(allow=("resumes/\w+/page-\d+",),
restrict_xpaths=('//a[@title="Next"]')),
callback="parse_items", follow=True),
]
def parse_items(self, response):
........
Thanks,
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/groups/opt_out.
