hi Tim Feird, The fixed delay is logged by autothrottle extension, but the actual delay is randomized while processing the download queue at s.c.donwloader.Downloader#132 <https://github.com/scrapy/scrapy/blob/8fece4b0b8eb8772a07673b4166cdcdb5c017eb8/scrapy/core/downloader/__init__.py#L132>. So, in summary, even if the message says the delays is 6s the actual delay is in 3s-9s range.
thanks Daniel El jueves, 31 de julio de 2014 04:58:49 UTC-3, tim feirg escribió: > > I'm not explaining my problem good enough, sorry. > What I'm saying is that DOWNLAOD_DELAY should be a random number according > to the docs: > > This setting is also affected by the RANDOMIZE_DOWNLOAD_DELAY >> <http://doc.scrapy.org/en/latest/topics/settings.html#std:setting-RANDOMIZE_DOWNLOAD_DELAY> >> setting >> (which is enabled by default). By default, Scrapy doesn’t wait a fixed >> amount of time between requests, but uses a random interval between 0.5 >> and 1.5 * DOWNLOAD_DELAY >> <http://doc.scrapy.org/en/latest/topics/settings.html#std:setting-DOWNLOAD_DELAY> >> . > > But in the output the delay is always 6 secs. > > > 在 2014年7月31日星期四UTC+8上午7时00分57秒，Rolando Espinoza La fuente写道： >> >> From the docs: >> >> The AutoThrottle extension honours the standard Scrapy settings for >> concurrency and delay. This means that it will never set a download >> delay >> lower than :setting:`DOWNLOAD_DELAY` or a concurrency higher than >> :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` >> (or :setting:`CONCURRENT_REQUESTS_PER_IP`, depending on which one you >> use). >> >> Regards, >> Rolando >> >> >> On Wed, Jul 30, 2014 at 3:58 PM, tim feirg <[email protected]> wrote: >> >>> I'm currently using scrapy 0.25 under an ubuntu server, I've add these >>> lines to my settings.py: >>> >>> CONCURRENT_REQUESTS_PER_DOMAIN = 3 >>> RETRY_TIMES = 10 >>> DOWNLOAD_DELAY = 6 >>> AUTOTHROTTLE_ENABLED = True >>> AUTOTHROTTLE_DEBUG = True >>> RANDOMIZE_DOWNLOAD_DELAY = True >>> >>> But the output (after several days of crawling) shows that the >>> DOWNLOAD_DELAY is always 6 seconds, this isn't deadly but it adds >>> uncertainty to this great framework, can somebody tells me what's wrong? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "scrapy-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/scrapy-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
