<https://lh3.googleusercontent.com/-jLsQw6_Lu2Q/WE6-M0brFwI/AAAAAAAAALQ/voj3Uwl5hj0wctabVRTCvgeVMVfLO0dUwCLcB/s1600/proj_structure.jpg>
This is the structure of my project. Nothing is happens when i add wrong
middleware. I change string in settings.py to
DOWNLOADER_MIDDLEWARES = {
'cr003.middlewares166.TorMiddleware': 740,
}
and my spider works normally
my settings.py:
> # -*- coding: utf-8 -*-
>
> BOT_NAME = 'cr003'
> SPIDER_MODULES = ['cr003.spiders']
> NEWSPIDER_MODULE = 'cr003.spiders'
>
> # Obey robots.txt rules
> ROBOTSTXT_OBEY = True
>
> DOWNLOADER_MIDDLEWARES = {
> 'cr003.middlewares.TorMiddleware': 740,
> }
my middlewares.py:
#!/usr/bin/python
> from scrapy.conf import settings
> class TorMiddleware(object):
> def process_request(self, request, spider):
> request.meta["proxy"] = "http://localhost:8080";
понедельник, 12 декабря 2016 г., 12:08:47 UTC+3 пользователь Paul Tremberth
написал:
>
> have you tried adding a non-existent downloader middleware in your
> settings.py and checked if it fails?
> maybe you're adding it to the wrong settings.py?
> otherwise, can you share your settings.py?
>
> On Monday, December 5, 2016 at 12:23:13 PM UTC+1, Ramzay Ak wrote:
>>
>> crawler, i created *my_project/middlewares.py which contain your
>> *TorMiddleware
>> class and add *DOWNLOADER_MIDDLEWARES in my settings.py. But when i
>> start my scrapy i do not see **my_project.middlewares.TorMiddleware in
>> list of *Enabled downloader middlewares.
>>
>> 2016-12-04 22:13:59 [scrapy] INFO: Enabled downloader middlewares:
>> ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
>> 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
>> 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
>> 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
>> 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
>> 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
>> 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
>> 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
>> 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
>> 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
>> 'scrapy.downloadermiddlewares.stats.DownloaderStats']
>>
>>
>> понедельник, 18 июля 2011 г., 18:13:45 UTC+3 пользователь crawler написал:
>>>
>>> *This works:*
>>>
>>> class TorMiddleware(object):
>>> def process_request(self, request, spider):
>>> request.meta["proxy"] = "http://localhost:8118";
>>>
>>>
>>> *DOWNLOADER_MIDDLEWARES = { 'my_project.middlewares.TorMiddleware':
>>> 740,} *
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
