I am trying my first crawl
i launch my scrap with this command:
nohup scrapy crawl prueba -o prueba.csv -t csv -s LOG_FILE=salida.out -s
JOBDIR=work -L DEBUG &
and i have configure CsvExportPipeline.py like in manual example, but when
spider has scraped 34 FEEDS finish.
Why? i are surfing in internet and everybody said memory problem but i
don't found any about memory in log.
Log level is in DEBUG but i cannot known reason why only read 34 items
Final log is this:
2014-12-17 17:02:32+0100 [prueba] INFO: Closing spider (finished)
2014-12-17 17:02:32+0100 [prueba] INFO: Stored csv feed (34 items)
in: prueba.csv
2014-12-17 17:02:32+0100 [prueba] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 14603,
'downloader/request_count': 35,
'downloader/request_method_count/GET': 35,
'downloader/response_bytes': 551613,
'downloader/response_count': 35,
'downloader/response_status_count/200': 35,
'dupefilter/filtered': 363,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2014, 12, 17, 16, 2, 32, 392134),
'item_scraped_count': 34,
'log_count/DEBUG': 72,
'log_count/ERROR': 1,
'log_count/INFO': 48,
'request_depth_max': 5,
'response_received_count': 35,
'scheduler/dequeued': 35,
'scheduler/dequeued/disk': 35,
'scheduler/enqueued': 35,
'scheduler/enqueued/disk': 35,
'start_time': datetime.datetime(2014, 12, 17, 15, 21, 55, 218630)}
2014-12-17 17:02:32+0100 [bodegas] INFO: Spider closed (finished)
Can anybody help me?
Regards
Roberto
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
