Re: with SparkStreeaming spark-submit, don't see output after ssc.start()
This problem turned out to be a cockpit error. I had the same class name defined in a couple different files, and didn't realize SBT was compiling them all together, and then executing the wrong one. Mea culpa. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/with-SparkStreeaming-spark-submit-don-t-see-output-after-ssc-start-tp17989p18224.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
with SparkStreeaming spark-submit, don't see output after ssc.start()
I have a Spark Streaming program that works fine if I execute it via sbt runMain com.cray.examples.spark.streaming.cyber.StatefulDhcpServerHisto -f /Users/spr/Documents/.../tmp/ -t 10 but if I start it via $S/bin/spark-submit --master local[12] --class StatefulNewDhcpServers target/scala-2.10/newd*jar -f /Users/spr/Documents/.../tmp/ -t 10 (where $S points to the base of the Spark installation), it prints the output of print statements before the ssc.start() but nothing after that. I might well have screwed up something, but I'm getting no output anywhere AFAICT. I have set spark.eventLog.enabled to True in my spark-defaults.conf file. The Spark History Server at localhost:18080 says no completed applications found. There must be some log output somewhere. Any ideas? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/with-SparkStreeaming-spark-submit-don-t-see-output-after-ssc-start-tp17989.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: with SparkStreeaming spark-submit, don't see output after ssc.start()
From: Tobias Pfeiffer t...@preferred.jpmailto:t...@preferred.jp Am I right that you are actually executing two different classes here? Yes, I realized after I posted that I was calling 2 different classes, though they are in the same JAR. I went back and tried it again with the same class in both cases, and it failed the same way. I thought perhaps having 2 classes in a JAR was an issue, but commenting out one of the classes did not seem to make a difference.