I was able to run simple examples as well. Which version of Spark? Did you use the most recent commit or from branch-1.0?
Some background: I tried to build both on Amazon EC2, but the master kept disconnecting from the client and executors failed after connecting. So I tried to just use one machine with a lot of ram. I can set up a cluster on released 0.9.1, but I need the sparse vector representations as my data is very sparse. Any way I can access a version of 1.0 that doesn't have to be compiled and is proven to work on EC2? My code: *import* numpy *from* numpy *import* array, dot, shape *from* pyspark *import* SparkContext *from* math *import* exp, log from pyspark.mllib.classification import NaiveBayes from pyspark.mllib.linalg import SparseVector from pyspark.mllib.regression import LabeledPoint, LinearModel def isSpace(line): if line.isspace() or not line.strip(): return True sizeOfDict = 2357815 def parsePoint(line): values = line.split('\t') feat = values[1].split(' ') features = {} for f in feat: f = f.split(':') if len(f) > 1: features[f[0]] = f[1] return LabeledPoint(float(values[0]), SparseVector(sizeOfDict, features)) data = sc.textFile(".../data.txt", 6) empty = data.filter(lambda x: not isSpace(x)) // I had an extra new line between each line points = empty.map(parsePoint) model = NaiveBayes.train(points) On Thu, Apr 24, 2014 at 6:55 PM, Xiangrui Meng <men...@gmail.com> wrote: > I tried locally with the example described in the latest guide: > http://54.82.157.211:4000/mllib-naive-bayes.html , and it worked fine. > Do you mind sharing the code you used? -Xiangrui > > On Thu, Apr 24, 2014 at 1:57 PM, John King <usedforprinting...@gmail.com> > wrote: > > Yes, I got it running for large RDD (~7 million lines) and mapping. Just > > received this error when trying to classify. > > > > > > On Thu, Apr 24, 2014 at 4:32 PM, Xiangrui Meng <men...@gmail.com> wrote: > >> > >> Is your Spark cluster running? Try to start with generating simple > >> RDDs and counting. -Xiangrui > >> > >> On Thu, Apr 24, 2014 at 11:38 AM, John King > >> <usedforprinting...@gmail.com> wrote: > >> > I receive this error: > >> > > >> > Traceback (most recent call last): > >> > > >> > File "<stdin>", line 1, in <module> > >> > > >> > File > >> > "/home/ubuntu/spark-1.0.0-rc2/python/pyspark/mllib/classification.py", > >> > line > >> > 178, in train > >> > > >> > ans = sc._jvm.PythonMLLibAPI().trainNaiveBayes(dataBytes._jrdd, > >> > lambda_) > >> > > >> > File > >> > > >> > > "/home/ubuntu/spark-1.0.0-rc2/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", > >> > line 535, in __call__ > >> > > >> > File > >> > > >> > > "/home/ubuntu/spark-1.0.0-rc2/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", > >> > line 368, in send_command > >> > > >> > File > >> > > >> > > "/home/ubuntu/spark-1.0.0-rc2/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", > >> > line 361, in send_command > >> > > >> > File > >> > > >> > > "/home/ubuntu/spark-1.0.0-rc2/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", > >> > line 317, in _get_connection > >> > > >> > File > >> > > >> > > "/home/ubuntu/spark-1.0.0-rc2/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", > >> > line 324, in _create_connection > >> > > >> > File > >> > > >> > > "/home/ubuntu/spark-1.0.0-rc2/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", > >> > line 431, in start > >> > > >> > py4j.protocol.Py4JNetworkError: An error occurred while trying to > >> > connect to > >> > the Java server > > > > >