I use this bit of code to run a spider programmatically.
def run_scraper(path_to_project, **kwargs):
""" run a spider defined in a scrapy project """
os.chdir(path_to_project)
settings = get_project_settings()
process = CrawlerProcess(settings)
process.crawl(spider, **kwargs) # TODO: add kwargs here for options
process.start()
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
