Hi all, Has anybody tried to run scrapy on a cluster? If yes, I would appreciate hearing about the general approach that was taken (multiple spiders? single spider? how to distribute urls across nodes?...etc). I would also be interested in hearing about any experience running a different scraper on a cluster, maybe scrapy is not the best one.
Thank you! Maria -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/running-scrapy-or-any-other-scraper-on-the-cluster-tp9286.html Sent from the Apache Spark User List mailing list archive at Nabble.com.