The data is in LIBSVM format. So this line won't work:
values = [float(s) for s in line.split(' ')]
Please use the util function in MLUtils to load it as an RDD of LabeledPoint.
http://spark.apache.org/docs/latest/mllib-data-types.html#labeled-point
from pyspark.mllib.util import MLUtils
examples = MLUtils.loadLibSVMFile(sc, data/mllib/sample_libsvm_data.txt)
-Xiangrui
On Sun, Nov 23, 2014 at 11:38 AM, Venkat, Ankam
ankam.ven...@centurylink.com wrote:
Can you please suggest sample data for running the logistic_regression.py?
I am trying to use a sample data file at
https://github.com/apache/spark/blob/master/data/mllib/sample_linear_regression_data.txt
I am running this on CDH5.2 Quickstart VM.
[cloudera@quickstart mllib]$ spark-submit logistic_regression.py lr.txt 3
But, getting below error.
14/11/23 11:23:55 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times;
aborting job
14/11/23 11:23:55 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
have all completed, from pool
14/11/23 11:23:55 INFO TaskSchedulerImpl: Cancelling stage 0
14/11/23 11:23:55 INFO DAGScheduler: Failed to run runJob at
PythonRDD.scala:296
Traceback (most recent call last):
File /usr/lib/spark/examples/lib/mllib/logistic_regression.py, line 50,
in module
model = LogisticRegressionWithSGD.train(points, iterations)
File /usr/lib/spark/python/pyspark/mllib/classification.py, line 110, in
train
initialWeights)
File /usr/lib/spark/python/pyspark/mllib/_common.py, line 430, in
_regression_train_wrapper
initial_weights = _get_initial_weights(initial_weights, data)
File /usr/lib/spark/python/pyspark/mllib/_common.py, line 415, in
_get_initial_weights
initial_weights = _convert_vector(data.first().features)
File /usr/lib/spark/python/pyspark/rdd.py, line 1127, in first
rs = self.take(1)
File /usr/lib/spark/python/pyspark/rdd.py, line 1109, in take
res = self.context.runJob(self, takeUpToNumLeft, p, True)
File /usr/lib/spark/python/pyspark/context.py, line 770, in runJob
it = self._jvm.PythonRDD.runJob(self._jsc.sc(), mappedRDD._jrdd,
javaPartitions, allowLocal)
File
/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py, line
538, in __call__
File /usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py,
line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling
z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0
(TID 3, 192.168.139.145): org.apache.spark.api.python.PythonException:
Traceback (most recent call last):
File /usr/lib/spark/python/pyspark/worker.py, line 79, in main
serializer.dump_stream(func(split_index, iterator), outfile)
File /usr/lib/spark/python/pyspark/serializers.py, line 196, in
dump_stream
self.serializer.dump_stream(self._batched(iterator), stream)
File /usr/lib/spark/python/pyspark/serializers.py, line 127, in
dump_stream
for obj in iterator:
File /usr/lib/spark/python/pyspark/serializers.py, line 185, in _batched
for item in iterator:
File /usr/lib/spark/python/pyspark/rdd.py, line 1105, in takeUpToNumLeft
yield next(iterator)
File /usr/lib/spark/examples/lib/mllib/logistic_regression.py, line 37,
in parsePoint
values = [float(s) for s in line.split(' ')]
ValueError: invalid literal for float(): 1:0.4551273600657362
Regards,
Venkat
This communication is the property of CenturyLink and may contain
confidential or privileged information. Unauthorized use of this
communication is strictly prohibited and may be unlawful. If you have
received this communication in error, please immediately notify the sender
by reply e-mail and destroy all copies of the communication and any
attachments.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
