Here is my code:
import scrapy
from tutorial.items import DmozItem
class DmozSpider(scrapy.Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/";,
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/";
]
def parse(self, response):
for sel in response.xpath('//ul/li'):
item = DmozItem()
item['title'] = sel.xpath('a/text()').extract()
item['link'] = sel.xpath('a/@href').extract()
item['desc'] = sel.xpath('text()').extract()
yield item
Here is the code from the tutorial:
import scrapy
from tutorial.items import DmozItem
class DmozSpider(scrapy.Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/";,
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/";
]
def parse(self, response):
for sel in response.xpath('//ul/li'):
item = DmozItem()
item['title'] = sel.xpath('a/text()').extract()
item['link'] = sel.xpath('a/@href').extract()
item['desc'] = sel.xpath('text()').extract()
yield item
I can't see any difference here, but the result shown in the tutorial is:
[scrapy] DEBUG: Scraped from <200
http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
{'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full
text, ASCII format. Asks for feedback. [author website, Gnosis Software,
Inc.\n],
'link': [u'http://gnosis.cx/TPiP/'],
'title': [u'Text Processing in Python']}
[scrapy] DEBUG: Scraped from <200
http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
{'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192,
has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and
SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n'],
'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'],
'title': [u'XML Processing with Python']}
But my result looks like this:
2015-08-08 13:14:55 [scrapy] DEBUG: Scraped from <200
http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
{'desc': [u'\r\n\t\r\n ',
u' \r\n\t\t\t\r\n - By David
Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for
feedback. [author website, Gnosis Software,
Inc.]\r\n
\r\n ',
u'\r\n '],
'link': [u'http://gnosis.cx/TPiP/'],
'title': [u'Text Processing in Python']}
2015-08-08 13:14:55 [scrapy] DEBUG: Scraped from <200
http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
{'desc': [u'\r\n\t\r\n ',
u' \r\n\t\t\t\r\n - By Sean
McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to
build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open
source XML processing library. [Prentice Hall
PTR]\r\n \r\n
',
u'\r\n '],
'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'],
'title': [u'XML Processing with Python']}
Actually, my result is *worse* than this. I just gave you a snippet to
match what is in the tutorial. But actually, *I have the whole dmoz page
with LOTS and LOTS of newlines, whitespace, and so on. *
The tutorial does not say anything about running strip() or something like
it, so how did they get this result and I got what I got? Further, the
tutorial says:
After inspecting the page source, you’ll find that the web site’s
> information is inside a <ul> element, in fact the *second* <ul> element.
>
When I look at the source, the information is in the *fourth* <ul> element.
Maybe I can't count, maybe the writers of the tutorial can't count, or
maybe the page has changed, but I can't see how the change from 2nd to 4th
alone would account for all this whitespace.
I tried indexing to see if that would narrow the result:
for sel in response.xpath('//ul[4]/li'):
but that and [3] got me no data. [2] got me the same data as no index
reference at all.
So if someone can help me understand why I got all this whitespace, \t, \n,
and \r, and how to eliminate them, I would be very happy.
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
