Hi,
When I execute the Spark ML Logisitc Regression example in pyspark I run
into an OutOfMemory exception. I'm wondering if any of you experienced the
same or has a hint about how to fix this.
The interesting bit is that I only get the exception when I try to write
the result DataFrame into a file. If I only "print" any of the results, it
all works fine.
My Setup:
Spark 1.5.0-SNAPSHOT built for Hadoop 2.6.0 (I'm working with the latest
nightly build)
Build flags: -Psparkr -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn
-DzincPort=3034
I'm using the default resource setup
15/09/07 08:49:04 INFO yarn.YarnAllocator: Will request 2 executor
containers, each with 1 cores and 1408 MB memory including 384 MB overhead
15/09/07 08:49:04 INFO yarn.YarnAllocator: Container request (host: Any,
capability: <memory:1408, vCores:1>)
15/09/07 08:49:04 INFO yarn.YarnAllocator: Container request (host: Any,
capability: <memory:1408, vCores:1>)
The script I'm executing:
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
conf = SparkConf().setAppName("pysparktest")
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
from pyspark.mllib.regression import LabeledPoint
from pyspark.mllib.linalg import Vector, Vectors
training = sc.parallelize((
LabeledPoint(1.0, Vectors.dense(0.0, 1.1, 0.1)),
LabeledPoint(0.0, Vectors.dense(2.0, 1.0, -1.0)),
LabeledPoint(0.0, Vectors.dense(2.0, 1.3, 1.0)),
LabeledPoint(1.0, Vectors.dense(0.0, 1.2, -0.5))))
training_df = training.toDF()
from pyspark.ml.classification import LogisticRegression
reg = LogisticRegression()
reg.setMaxIter(10).setRegParam(0.01)
model = reg.fit(training.toDF())
test = sc.parallelize((
LabeledPoint(1.0, Vectors.dense(-1.0, 1.5, 1.3)),
LabeledPoint(0.0, Vectors.dense(3.0, 2.0, -0.1)),
LabeledPoint(1.0, Vectors.dense(0.0, 2.2, -1.5))))
out_df = model.transform(test.toDF())
out_df.write.parquet("/tmp/logparquet")
And the command:
spark-submit --master yarn --deploy-mode cluster spark-ml.py
Thanks,
z
