You can add a line that logs the page like
self.log ('page: %s' % response.url, level=log.INFO)
and be sure in your settings file to be outputting your log to a file (I
append date/time to the log file, too).
Alternatively,
You can include a 'url' field in your item definition, then when harvesting
item data:
item['url'] = response.url
That way, you can do whatever you like with that data in the pipeline.
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/groups/opt_out.
