Anyone?
On Monday, April 6, 2015 at 3:45:27 PM UTC+3, crawler wrote:
>
> Hi. I use scrapyd and got a problem with jobs closing by itself
> using crawler.engine.close_spider method. They stuck into 'running' state.
>
> class MysqlPipeline(object):
>
> def __init__(self, crawler):
> dispatcher.connect(self.spider_opened, signals.spider_opened)
> dispatcher.connect(self.spider_closed, signals.spider_closed)
> self.crawler = crawler
>
> @classmethod
> def from_crawler(cls, crawler):
> return cls(crawler)
>
> def spider_opened(self, spider):
> if spider.mode is None:
> self.crawler.engine.close_spider(spider, 'Run mode not
> defined')
> return
>
>
>
> Scrapyd job log
>
> 2015-04-06 00:30:18+0300 [scrapy] INFO: Scrapy 0.24.5 started (bot:
> scrapybot)
> 2015-04-06 00:30:18+0300 [scrapy] INFO: Optional features available: ssl,
> http11, django
> 2015-04-06 00:30:18+0300 [scrapy] INFO: Overridden settings: {
> 'NEWSPIDER_MODULE': 'ololo.spiders', 'FEED_URI':
> '/var/lib/scrapyd/items/ololo/myspider/eaccc874dbda11e4ab04000c29ca58d6.jl'
> , 'CONCURRENT_REQUESTS_PER_DOMAIN': 50, 'CONCURRENT_REQUESTS': 800,
> 'RANDOMIZE_DOWNLOAD_DELAY': False, 'SPIDER_MODULES': ['ololo.spiders'],
> 'RETRY_TIMES': 100, 'DOWNLOAD_TIMEOUT': 600, 'CONCURRENT_ITEMS': 200,
> 'COOKIES_ENABLED': False, 'USER_AGENT': 'Mozilla/5.0 (compatible; Yahoo!
> Slurp; http://help.yahoo.com/help/us/ysearch/slurp)', 'DEFAULT_ITEM_CLASS'
> : 'ololo.items.ololoPricesItem', 'LOG_FILE':
> '/var/log/scrapyd/ololo/myspider/eaccc874dbda11e4ab04000c29ca58d6.log'}
> 2015-04-06 00:30:18+0300 [scrapy] INFO: Enabled extensions: FeedExporter,
> LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
> 2015-04-06 00:30:18+0300 [scrapy] INFO: Enabled downloader middlewares:
> HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware,
> RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware,
> HttpCompressionMiddleware, RedirectMiddleware, TorMiddleware,
> ChunkedTransferMiddleware, DownloaderStats
> 2015-04-06 00:30:18+0300 [scrapy] INFO: Enabled spider middlewares:
> HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware,
> UrlLengthMiddleware, DepthMiddleware
> 2015-04-06 00:30:18+0300 [scrapy] INFO: Enabled item pipelines:
> ProductPipeline, MysqlPipeline
> 2015-04-06 00:30:18+0300 [myspider] INFO: Spider opened
> 2015-04-06 00:30:18+0300 [myspider] INFO: Crawled 0 pages (at 0 pages/min
> ), scraped 0 items (at 0 items/min)
> 2015-04-06 00:30:18+0300 [myspider] INFO: Closing spider (Run mode not
> defined)
> 2015-04-06 00:30:18+0300 [myspider] INFO: Dumping Scrapy stats:
> {'finish_reason': 'Bot is already running',
> 'finish_time': datetime.datetime(2015, 4, 5, 21, 30, 18, 499333),
> 'log_count/INFO': 7,
> 'start_time': datetime.datetime(2015, 4,
> ...
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
