Sorry <scope>compile</scope> makes it work locally. But, the cluster still seems to have issues with <scope>provided</scope>. Basically it does not seem to process any records, no data is shown in any of the tabs of the Streaming UI except the Streaming tab. Executors, Storage, Stages etc show empty RDDs.
On Tue, Nov 17, 2015 at 7:19 PM, swetha kasireddy <swethakasire...@gmail.com > wrote: > Hi TD, > > Basically, I see two issues. With <scope>provided</scope> the job does > not start localy. It does start in Cluster but seems no data is getting > processed. > > Thanks, > Swetha > > On Tue, Nov 17, 2015 at 7:04 PM, Tim Barthram <tim.barth...@iag.com.au> > wrote: > >> If you are running a local context, could it be that you should use: >> >> >> >> <scope>provided</scope> >> >> >> >> ? >> >> >> >> Thanks, >> >> Tim >> >> >> >> *From:* swetha kasireddy [mailto:swethakasire...@gmail.com] >> *Sent:* Wednesday, 18 November 2015 2:01 PM >> *To:* Tathagata Das >> *Cc:* user >> *Subject:* Re: Streaming Job gives error after changing to version 1.5.2 >> >> >> >> This error I see locally. >> >> >> >> On Tue, Nov 17, 2015 at 5:44 PM, Tathagata Das <t...@databricks.com> >> wrote: >> >> Are you running 1.5.2-compiled jar on a Spark 1.5.2 cluster? >> >> >> >> On Tue, Nov 17, 2015 at 5:34 PM, swetha <swethakasire...@gmail.com> >> wrote: >> >> >> >> Hi, >> >> I see java.lang.NoClassDefFoundError after changing the Streaming job >> version to 1.5.2. Any idea as to why this is happening? Following are my >> dependencies and the error that I get. >> >> <dependency> >> <groupId>org.apache.spark</groupId> >> <artifactId>spark-core_2.10</artifactId> >> <version>${sparkVersion}</version> >> <scope>provided</scope> >> </dependency> >> >> >> <dependency> >> <groupId>org.apache.spark</groupId> >> <artifactId>spark-streaming_2.10</artifactId> >> <version>${sparkVersion}</version> >> <scope>provided</scope> >> </dependency> >> >> >> <dependency> >> <groupId>org.apache.spark</groupId> >> <artifactId>spark-sql_2.10</artifactId> >> <version>${sparkVersion}</version> >> <scope>provided</scope> >> </dependency> >> >> >> <dependency> >> <groupId>org.apache.spark</groupId> >> <artifactId>spark-hive_2.10</artifactId> >> <version>${sparkVersion}</version> >> <scope>provided</scope> >> </dependency> >> >> >> >> <dependency> >> <groupId>org.apache.spark</groupId> >> <artifactId>spark-streaming-kafka_2.10</artifactId> >> <version>${sparkVersion}</version> >> </dependency> >> >> >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/spark/streaming/StreamingContext >> at java.lang.Class.getDeclaredMethods0(Native Method) >> at java.lang.Class.privateGetDeclaredMethods(Class.java:2693) >> at java.lang.Class.privateGetMethodRecursive(Class.java:3040) >> at java.lang.Class.getMethod0(Class.java:3010) >> at java.lang.Class.getMethod(Class.java:1776) >> at >> com.intellij.rt.execution.application.AppMain.main(AppMain.java:125) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.spark.streaming.StreamingContext >> at java.net.URLClassLoader$1.run(URLClassLoader.java:372) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:360) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Streaming-Job-gives-error-after-changing-to-version-1-5-2-tp25406.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >> >> >> >> _____________________________________________________________________ >> >> The information transmitted in this message and its attachments (if any) >> is intended >> only for the person or entity to which it is addressed. >> The message may contain confidential and/or privileged material. Any >> review, >> retransmission, dissemination or other use of, or taking of any action in >> reliance >> upon this information, by persons or entities other than the intended >> recipient is >> prohibited. >> >> If you have received this in error, please contact the sender and delete >> this e-mail >> and associated material from any computer. >> >> The intended recipient of this e-mail may only use, reproduce, disclose >> or distribute >> the information contained in this e-mail and any attached files, with the >> permission >> of the sender. >> >> This message has been scanned for viruses. >> _____________________________________________________________________ >> > >
