Re: mahout spark-itemsimilarity does not work on EMR 4.3
Another way to get Mahout item-similarity based recommender is to use the Universal Recommender here: https://github.com/actionml/template-scala-parallel-universal-recommendation/tree/v0.3.0 It includes an event input pipeline, periodic Mahout+Spark based model generation and a realtime serving component. It can be set up to use EMR for the Spark jobs. BTW by removing the Redis part recs can be made on realtime data and queries returned in realtime. PM me if you want to know more. On Feb 24, 2016, at 12:25 AM, Zhun Shen wrote: My use case: I use mahout to do my recommender, most of our application build on Spark, I use Spark to prepare the dataset and then I want use mahout on Spark to calculate items similarity, then Spark will be use to format the output and save the result to Redis. After that, frontend can get the recommend data from Redis. You can see, I just want to do ETL job, recommend training and deploy jobs all in Spark stack in AWS EMR. By the way, when will next release of EMR ? > On Feb 24, 2016, at 11:24 AM, Jonathan Kelly wrote: > > The next release of EMR will include Mahout 0.11.1 so that Mahout on Spark > works with Spark 1.6. Sorry for any inconvenience until then. > > By the way, I'm interested to know what your use case is for running Mahout > on Spark, so please feel free to PM me if you are able to share any details. > > Thank you, > Jonathan > On Tue, Feb 23, 2016 at 6:06 PM Andrew Palumbo wrote: > >> Please update to Mahout 0.11.1 for spark versions > 1.3. >> >> Original message >> From: Zhun Shen >> Date: 02/23/2016 8:57 PM (GMT-05:00) >> To: user@mahout.apache.org >> Subject: mahout spark-itemsimilarity does not work on EMR 4.3 >> >> Hi, >> mahout version: 0.11.0 >> EMR version: 4.3 >> spark version: 1.6.0 >> >> I try to run mahout spark-itemsimilarity on AWS EMR, but it told me that >> “MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. >> Cannot find Spark classpath. Is 'SPARK_HOME' set?”, Is it a bug for EMR or >> I use mahout spark in a wrong way ? >>
Re: mahout spark-itemsimilarity does not work on EMR 4.3
My use case: I use mahout to do my recommender, most of our application build on Spark, I use Spark to prepare the dataset and then I want use mahout on Spark to calculate items similarity, then Spark will be use to format the output and save the result to Redis. After that, frontend can get the recommend data from Redis. You can see, I just want to do ETL job, recommend training and deploy jobs all in Spark stack in AWS EMR. By the way, when will next release of EMR ? > On Feb 24, 2016, at 11:24 AM, Jonathan Kelly wrote: > > The next release of EMR will include Mahout 0.11.1 so that Mahout on Spark > works with Spark 1.6. Sorry for any inconvenience until then. > > By the way, I'm interested to know what your use case is for running Mahout > on Spark, so please feel free to PM me if you are able to share any details. > > Thank you, > Jonathan > On Tue, Feb 23, 2016 at 6:06 PM Andrew Palumbo wrote: > >> Please update to Mahout 0.11.1 for spark versions > 1.3. >> >> Original message >> From: Zhun Shen >> Date: 02/23/2016 8:57 PM (GMT-05:00) >> To: user@mahout.apache.org >> Subject: mahout spark-itemsimilarity does not work on EMR 4.3 >> >> Hi, >> mahout version: 0.11.0 >> EMR version: 4.3 >> spark version: 1.6.0 >> >> I try to run mahout spark-itemsimilarity on AWS EMR, but it told me that >> “MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. >> Cannot find Spark classpath. Is 'SPARK_HOME' set?”, Is it a bug for EMR or >> I use mahout spark in a wrong way ? >>
Re: mahout spark-itemsimilarity does not work on EMR 4.3
The next release of EMR will include Mahout 0.11.1 so that Mahout on Spark works with Spark 1.6. Sorry for any inconvenience until then. By the way, I'm interested to know what your use case is for running Mahout on Spark, so please feel free to PM me if you are able to share any details. Thank you, Jonathan On Tue, Feb 23, 2016 at 6:06 PM Andrew Palumbo wrote: > Please update to Mahout 0.11.1 for spark versions > 1.3. > > Original message > From: Zhun Shen > Date: 02/23/2016 8:57 PM (GMT-05:00) > To: user@mahout.apache.org > Subject: mahout spark-itemsimilarity does not work on EMR 4.3 > > Hi, > mahout version: 0.11.0 > EMR version: 4.3 > spark version: 1.6.0 > > I try to run mahout spark-itemsimilarity on AWS EMR, but it told me that > “MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. > Cannot find Spark classpath. Is 'SPARK_HOME' set?”, Is it a bug for EMR or > I use mahout spark in a wrong way ? >
RE: mahout spark-itemsimilarity does not work on EMR 4.3
Please update to Mahout 0.11.1 for spark versions > 1.3. Original message From: Zhun Shen Date: 02/23/2016 8:57 PM (GMT-05:00) To: user@mahout.apache.org Subject: mahout spark-itemsimilarity does not work on EMR 4.3 Hi, mahout version: 0.11.0 EMR version: 4.3 spark version: 1.6.0 I try to run mahout spark-itemsimilarity on AWS EMR, but it told me that “MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. Cannot find Spark classpath. Is 'SPARK_HOME' set?”, Is it a bug for EMR or I use mahout spark in a wrong way ?
