I had that same kind of freeze too, try curl the page that you want from
command line before scheduling a spider run, you're probably banned and
that's what's causing mine to freeze.
在 2014年9月4日星期四UTC+8下午4时44分28秒，dasher写道：
>
> Hey guys,
>
> I found that i can freely to use scrapy to run my spiders, but when i use
> scrapyd to run my spiders, it didnt work. the job just hang up, and there
> is no log, just a sign that spider has started
>
> here are the logs that i can only get from the scrapyd, which are not
> that helpful.
>
> 2014-09-04 07:39:05+0000 [scrapy] INFO: Scrapy 0.24.2 started (bot: scrape)
> 2014-09-04 07:39:05+0000 [scrapy] INFO: Optional features available: ssl,
> http11
> 2014-09-04 07:39:05+0000 [scrapy] INFO: Overridden settings:
> {'NEWSPIDER_MODULE': 'scrape.spiders', 'FEED_URI':
> '/var/lib/scrapyd/items/scrape/dddd/60ef7092340611e487a852540008c927.jl',
> 'SPIDER_MODULES': ['scrape.spiders'], 'BOT_NAME': 'scrape', 'LOG_FILE':
> '/var/log/scrapyd/scrape/dddd/60ef7092340611e487a852540008c927.log',
> 'DOWNLOAD_DELAY': 2}
> 2014-09-04 07:39:16+0000 [scrapy] INFO: Enabled extensions: FeedExporter,
> LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
> 2014-09-04 07:39:50+0000 [scrapy] INFO: Enabled downloader middlewares:
> HttpAuthMiddleware, DownloadTimeoutMiddleware, RotateUserAgentMiddleware,
> RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware,
> HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware,
> ChunkedTransferMiddleware, DownloaderStats
> 2014-09-04 07:39:51+0000 [scrapy] INFO: Enabled spider middlewares:
> HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware,
> UrlLengthMiddleware, DepthMiddleware
>
>
>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
