I'm not explaining my problem good enough, sorry. What I'm saying is that DOWNLAOD_DELAY should be a random number according to the docs:
This setting is also affected by the RANDOMIZE_DOWNLOAD_DELAY > <http://doc.scrapy.org/en/latest/topics/settings.html#std:setting-RANDOMIZE_DOWNLOAD_DELAY> > setting > (which is enabled by default). By default, Scrapy doesn’t wait a fixed > amount of time between requests, but uses a random interval between 0.5 > and 1.5 * DOWNLOAD_DELAY > <http://doc.scrapy.org/en/latest/topics/settings.html#std:setting-DOWNLOAD_DELAY> > . But in the output the delay is always 6 secs. 在 2014年7月31日星期四UTC+8上午7时00分57秒，Rolando Espinoza La fuente写道： > > From the docs: > > The AutoThrottle extension honours the standard Scrapy settings for > concurrency and delay. This means that it will never set a download > delay > lower than :setting:`DOWNLOAD_DELAY` or a concurrency higher than > :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` > (or :setting:`CONCURRENT_REQUESTS_PER_IP`, depending on which one you > use). > > Regards, > Rolando > > > On Wed, Jul 30, 2014 at 3:58 PM, tim feirg <[email protected] > <javascript:>> wrote: > >> I'm currently using scrapy 0.25 under an ubuntu server, I've add these >> lines to my settings.py: >> >> CONCURRENT_REQUESTS_PER_DOMAIN = 3 >> RETRY_TIMES = 10 >> DOWNLOAD_DELAY = 6 >> AUTOTHROTTLE_ENABLED = True >> AUTOTHROTTLE_DEBUG = True >> RANDOMIZE_DOWNLOAD_DELAY = True >> >> But the output (after several days of crawling) shows that the >> DOWNLOAD_DELAY is always 6 seconds, this isn't deadly but it adds >> uncertainty to this great framework, can somebody tells me what's wrong? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "scrapy-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/scrapy-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
