Re: scikit learn on EMR PySpark

2016-03-01 Thread Jonathan Kelly
Hi, Myles, We do not install scikit-learn or spark-sklearn on EMR clusters by default, but you may install them yourself by just doing "sudo pip install scikit-learn spark-sklearn" (either by ssh'ing to the master instance and running this manually, or by running it as an EMR Step). ~ Jonathan

scikit learn on EMR PySpark

2016-03-01 Thread Gartland, Myles
New to Spark and MLlib. Coming from sickit learn. I am launching my Spark 1.6 instance through AWS EMR and pyspark. All the examples using Mllib work fine. But I have seen a couple examples where you can combine scikit learn packages and syntax with mllib. Like in this example-