Hi All,
I'm trying to figure out how to run a spider directly from django..
I've done alot of searching around but no matter how i try i keep getting
File
"/usr/local/lib/python2.7/site-packages/twisted/internet/base.py", line
1152, in _handleSignals
signal.signal(signal.SIGINT, self.sigInt)
exceptions.ValueError: signal only works in main thread
Does anyone have any idea how i can achieve this?? Please see the code i'm
using to launch scrapy within django below.
===============
class CrawlerWorker(multiprocessing.Process):
def __init__(self, spider, result_queue):
multiprocessing.Process.__init__(self)
self.result_queue = result_queue
self.crawler = CrawlerProcess(settings)
if not hasattr(project, 'crawler'):
self.crawler.install()
self.crawler.configure()
self.items = []
self.spider = spider
dispatcher.connect(self._item_passed, signals.item_passed)
def _item_passed(self, item):
self.items.append(item)
def run(self):
self.crawler.crawl(self.spider)
self.crawler.start()
self.crawler.stop()
self.result_queue.put(self.items)
result_queue = Queue()
crawler = CrawlerWorker(MyCrawler(), result_queue)
crawler.start()
for item in result_queue.get():
print item
========
Thanks
Steve
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
