Hi,
On Fri, Oct 31, 2014 at 4:31 PM, lieyan lie...@yahoo.com wrote:
The code are here: LogReg.scala
http://apache-spark-user-list.1001560.n3.nabble.com/file/n17803/LogReg.scala
Then I click the Run button of the IDEA, and I get the following error
message
errlog.txt
http://apache-spark-user-list.1001560.n3.nabble.com/file/n17803/errlog.txt
.
But when I export the jar file, and use *spark-submit --class
net.yanl.spark.LogReg log_reg.jar 15*. The program works finely.
I have not used the spark built-in cluster manager and I don't know how
application jar distribution is done in it. However, it seems to me that
when you use spark-submit, then spark-submit takes care of distributing
your jar file properly to all the cluster nodes, that's why it works fine.
When you run it from your IDE, it seems not to do that, that's why some
classes are not there on all cluster nodes and you run
into ClassNotFoundExceptions. If you change the master to local[3]
instead of spark://master.local:7077 and run it from IDEA, does it work?
Tobias