Re: different behaviour of the same code
Hi, On Fri, Oct 31, 2014 at 4:31 PM, lieyan lie...@yahoo.com wrote: The code are here: LogReg.scala http://apache-spark-user-list.1001560.n3.nabble.com/file/n17803/LogReg.scala Then I click the Run button of the IDEA, and I get the following error message errlog.txt http://apache-spark-user-list.1001560.n3.nabble.com/file/n17803/errlog.txt . But when I export the jar file, and use *spark-submit --class net.yanl.spark.LogReg log_reg.jar 15*. The program works finely. I have not used the spark built-in cluster manager and I don't know how application jar distribution is done in it. However, it seems to me that when you use spark-submit, then spark-submit takes care of distributing your jar file properly to all the cluster nodes, that's why it works fine. When you run it from your IDE, it seems not to do that, that's why some classes are not there on all cluster nodes and you run into ClassNotFoundExceptions. If you change the master to local[3] instead of spark://master.local:7077 and run it from IDEA, does it work? Tobias
Re: different behaviour of the same code
You are right. You pointed out the very cause of my problem. Thanks. I have to specify the path to my jar file. The solution can be found in an earlier post. http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-with-simple-Spark-job-on-cluster-td932.html -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/different-behaviour-of-the-same-code-tp17803p18021.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
different behaviour of the same code
I am trying to write some sample code under IntelliJ IDEA. I start with a non-sbt scala project. In order that the program compile, I add *spark-assembly-1.1.0-hadoop2.4.0.jar* in the *spark/lib* directory as one external library of the IDEA project. http://apache-spark-user-list.1001560.n3.nabble.com/file/n17803/proj.jpg The code are here: LogReg.scala http://apache-spark-user-list.1001560.n3.nabble.com/file/n17803/LogReg.scala Then I click the Run button of the IDEA, and I get the following error message errlog.txt http://apache-spark-user-list.1001560.n3.nabble.com/file/n17803/errlog.txt . But when I export the jar file, and use *spark-submit --class net.yanl.spark.LogReg log_reg.jar 15*. The program works finely. This is somehow annoying. Can anyone resolve this issue? You may need the following file to reproduce the error. out5_training.log/out5_testing.log http://apache-spark-user-list.1001560.n3.nabble.com/file/n17803/small01.log -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/different-behaviour-of-the-same-code-tp17803.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
