Hey guys,
I found that i can freely to use scrapy to run my spiders, but when i use
scrapyd to run my spiders, it didnt work. the job just hang up, and there
is no log, just a sign that spider has started
here are the logs that i can only get from the scrapyd, which are not that
helpful.
2014-09-04 07:39:05+0000 [scrapy] INFO: Scrapy 0.24.2 started (bot: scrape)
2014-09-04 07:39:05+0000 [scrapy] INFO: Optional features available: ssl,
http11
2014-09-04 07:39:05+0000 [scrapy] INFO: Overridden settings:
{'NEWSPIDER_MODULE': 'scrape.spiders', 'FEED_URI':
'/var/lib/scrapyd/items/scrape/dddd/60ef7092340611e487a852540008c927.jl',
'SPIDER_MODULES': ['scrape.spiders'], 'BOT_NAME': 'scrape', 'LOG_FILE':
'/var/log/scrapyd/scrape/dddd/60ef7092340611e487a852540008c927.log',
'DOWNLOAD_DELAY': 2}
2014-09-04 07:39:16+0000 [scrapy] INFO: Enabled extensions: FeedExporter,
LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2014-09-04 07:39:50+0000 [scrapy] INFO: Enabled downloader middlewares:
HttpAuthMiddleware, DownloadTimeoutMiddleware, RotateUserAgentMiddleware,
RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware,
HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware,
ChunkedTransferMiddleware, DownloaderStats
2014-09-04 07:39:51+0000 [scrapy] INFO: Enabled spider middlewares:
HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware,
UrlLengthMiddleware, DepthMiddleware
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
