When you use os.system("scrapy ...") you are starting a new python process
which doesn't inherit the sys.path modification. And when you do
os.system("export ...") you are not modifying the current process
environment variables, that's done through os.environ but, usually, a sub
process will not inherit those variables.You should do something like this: $ export SCRAPY_SETTINGS_MODULE=tutorial.settings $ export PYTHONPATH=/home/user/tutorial/ $ scrapy crawl dmoz If you need to start the scrapy process from python then you have set up the environment variables for the sub process, see [1] or [2]. [1] http://docs.python.org/2/library/os.html#os.putenv [2] http://docs.python.org/2/library/subprocess.html#subprocess.Popen On Sat, Feb 1, 2014 at 1:03 AM, Jaiprakash Singh <[email protected]>wrote: > hey > > really really thank to reply me, i did some of experiment on your > advice, i made very basic project as shown in > http://doc.scrapy.org/en/0.20/intro/tutorial.html, then i did under > following , yes i can import the setting but unable to run the crawler > > ================================= > tree -d tutorial > > tutorial/ > `-- tutorial > `-- spiders > > > > at top most directory i.e at first tutorial > > > step 1 > > >>> import os, sys > >>> os.system("export SCRAPY_SETTINGS_MODULE=tutorial.settings") > >>> sys.path.append("/home/user/tutorial/") > >>> import tutorial.settings > > >>> os.system("scrapy crawl dmoz") > > error :-ImportError: No module named tutorial.settings > > > >>> os.system("scrapy runspider dmoz_spider.py") > > error :- ImportError: No module named tutorial.settings > > > > =============================================== > > next step:- > > on shell :- $export SCRAPY_SETTINGS_MODULE=tutorial.settings > > on interpreter > > >>> import os, sys > >>> sys.path.append("/home/user/tutorial/") > >>> import tutorial.settings > > >>> os.system("scrapy crawl dmoz") > error :- ImportError: No module named tutorial.settings > > >>> os.system("scrapy runspider dmoz_spider.py") > error :- ImportError: No module named tutorial.settings > > =================================================== > > next step:- > > added > SCRAPY_SETTINGS_MODULE=tutorial.settings it in tutorial.settings.py > > run same process > then same problem occur > > > please advice me what should i do now? > > > On Fri, Jan 31, 2014 at 8:02 PM, Rolando Espinoza La Fuente < > [email protected]> wrote: > >> You have set SCRAPY_SETTINGS_MODULE in the shell before running scrapy >> command. If you are using linux/osx, and assuming the project2 package is >> in the PYTHONPATH, try this >> >> export SCRAPY_SETTINGS_MODULE=project2.settings >> scrapy crawl yourspider >> >> This will work given that you can import project2.settings in python, you >> can verify with this: >> >> python >> >>> import project2.settings >> >> >> >> >> On Fri, Jan 31, 2014 at 9:37 AM, Jaiprakash Singh < >> [email protected]> wrote: >> >>> please help me i am really stuck here , >>> my code is ruining properly when i am running it it its own directory >>> i.e where my code is present , but i am unable to run it properly from >>> other directory >>> >>> >>> code3forlinkparsing >>> └── code3forlinkparsing >>> └── spiders >>> ├── brand_info_by_date >>> │ └── hand-bags31012014 >>> ├── brands_htmls >>> │ └── hand-bags31012014 >>> ├── code1_brandcollection >>> │ └── code1_brandcollection >>> │ └── spiders >>> ├── code2_scrolling >>> │ └── code2_scrolling >>> │ └── spiders >>> │ └── dumpsss >>> ├── item_details_csv 31012014 >>> ├── project2 >>> │ └── project2 >>> │ └── spiders >>> └── project3 >>> └── project3 >>> └── spiders >>> >>> want to run >>> code3forlinkparsing.spiders.project2/project2/spiders/mycode.py from >>> topmost directory >>> >>> i tried >>> >>> scrapy runspider >>> code3forlinkparsing.spiders.project2/project2/spiders/mycode.py >>> >>> but it skips its own setting i.e setting of >>> project2.project2.settings.py >>> >>> >>> >>> >>> under setting i am using >>> >>> BOT_NAME = 'project2' >>> >>> SCRAPY_SETTINGS_MODULE = "project2.spiders" >>> >>> SPIDER_MODULES = ['project2.spiders'] >>> NEWSPIDER_MODULE = 'project2.spiders' >>> >>> # Crawl responsibly by identifying yourself (and your website) on the >>> user-agent >>> #USER_AGENT = 'project2 (+http://www.yourdomain.com)' >>> >>> >>> DOWNLOADER_MIDDLEWARES = { >>> 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': >>> 110, >>> 'project2.proxymiddle.ProxyMiddleware': 100, >>> } >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "scrapy-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/scrapy-users. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "scrapy-users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/scrapy-users/COW-sqof5H0/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/scrapy-users. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/groups/opt_out.
