If I packaged the application and submit it, it works fine but I need to run it from eclipse.
Is there any problem running the application from eclipse ? On 9 November 2015 at 12:27, Tathagata Das <t...@databricks.com> wrote: > How are you submitting the spark application? > You are supposed to submit the fat-jar of the application that include the > spark-streaming-twitter dependency (and its subdeps) but not > spark-streaming and spark-core. > > On Mon, Nov 9, 2015 at 1:02 AM, أنس الليثي <dev.fano...@gmail.com> wrote: > >> I tried to remove maven and adding the dependencies manually using build >> path > configure build path > add external jars, then adding the jars >> manually but it did not work. >> >> I tried to create another project and copied the code from the first app >> but the problem still the same. >> >> I event tried to change eclipse with another version, but the same >> problem exist. >> >> :( :( :( :( >> >> On 9 November 2015 at 10:47, أنس الليثي <dev.fano...@gmail.com> wrote: >> >>> I tried both, the same exception still thrown >>> >>> On 9 November 2015 at 10:45, Sean Owen <so...@cloudera.com> wrote: >>> >>>> You included a very old version of the Twitter jar - 1.0.0. Did you >>>> mean 1.5.1? >>>> >>>> On Mon, Nov 9, 2015 at 7:36 AM, fanooos <dev.fano...@gmail.com> wrote: >>>> > This is my first Spark Stream application. The setup is as following >>>> > >>>> > 3 nodes running a spark cluster. One master node and two slaves. >>>> > >>>> > The application is a simple java application streaming from twitter >>>> and >>>> > dependencies managed by maven. >>>> > >>>> > Here is the code of the application >>>> > >>>> > public class SimpleApp { >>>> > >>>> > public static void main(String[] args) { >>>> > >>>> > SparkConf conf = new SparkConf().setAppName("Simple >>>> > Application").setMaster("spark://rethink-node01:7077"); >>>> > >>>> > JavaStreamingContext sc = new JavaStreamingContext(conf, new >>>> > Duration(1000)); >>>> > >>>> > ConfigurationBuilder cb = new ConfigurationBuilder(); >>>> > >>>> > cb.setDebugEnabled(true).setOAuthConsumerKey("ConsumerKey") >>>> > .setOAuthConsumerSecret("ConsumerSecret") >>>> > .setOAuthAccessToken("AccessToken") >>>> > .setOAuthAccessTokenSecret("TokenSecret"); >>>> > >>>> > OAuthAuthorization auth = new OAuthAuthorization(cb.build()); >>>> > >>>> > JavaDStream<Status> tweets = TwitterUtils.createStream(sc, >>>> auth); >>>> > >>>> > JavaDStream<String> statuses = tweets.map(new >>>> Function<Status, >>>> > String>() { >>>> > public String call(Status status) throws Exception { >>>> > return status.getText(); >>>> > } >>>> > }); >>>> > >>>> > statuses.print();; >>>> > >>>> > sc.start(); >>>> > >>>> > sc.awaitTermination(); >>>> > >>>> > } >>>> > >>>> > } >>>> > >>>> > >>>> > here is the pom file >>>> > >>>> > <project xmlns="http://maven.apache.org/POM/4.0.0" >>>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>>> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 >>>> > http://maven.apache.org/xsd/maven-4.0.0.xsd"> >>>> > <modelVersion>4.0.0</modelVersion> >>>> > <groupId>SparkFirstTry</groupId> >>>> > <artifactId>SparkFirstTry</artifactId> >>>> > <version>0.0.1-SNAPSHOT</version> >>>> > >>>> > <dependencies> >>>> > <dependency> >>>> > <groupId>org.apache.spark</groupId> >>>> > <artifactId>spark-core_2.10</artifactId> >>>> > <version>1.5.1</version> >>>> > <scope>provided</scope> >>>> > </dependency> >>>> > >>>> > <dependency> >>>> > <groupId>org.apache.spark</groupId> >>>> > <artifactId>spark-streaming_2.10</artifactId> >>>> > <version>1.5.1</version> >>>> > <scope>provided</scope> >>>> > </dependency> >>>> > >>>> > <dependency> >>>> > <groupId>org.twitter4j</groupId> >>>> > <artifactId>twitter4j-stream</artifactId> >>>> > <version>3.0.3</version> >>>> > </dependency> >>>> > <dependency> >>>> > <groupId>org.apache.spark</groupId> >>>> > <artifactId>spark-streaming-twitter_2.10</artifactId> >>>> > <version>1.0.0</version> >>>> > </dependency> >>>> > >>>> > </dependencies> >>>> > >>>> > <build> >>>> > <sourceDirectory>src</sourceDirectory> >>>> > <plugins> >>>> > <plugin> >>>> > <artifactId>maven-compiler-plugin</artifactId> >>>> > <version>3.3</version> >>>> > <configuration> >>>> > <source>1.8</source> >>>> > <target>1.8</target> >>>> > </configuration> >>>> > </plugin> >>>> > <plugin> >>>> > <artifactId>maven-assembly-plugin</artifactId> >>>> > <configuration> >>>> > <archive> >>>> > <manifest> >>>> > >>>> > <mainClass>com.test.sparkTest.SimpleApp</mainClass> >>>> > </manifest> >>>> > </archive> >>>> > <descriptorRefs> >>>> > >>>> <descriptorRef>jar-with-dependencies</descriptorRef> >>>> > </descriptorRefs> >>>> > </configuration> >>>> > </plugin> >>>> > >>>> > </plugins> >>>> > </build> >>>> > </project> >>>> > >>>> > >>>> > The application starts successfully but no tweets comes and this >>>> exception >>>> > is thrown >>>> > >>>> > 15/11/08 15:55:46 WARN TaskSetManager: Lost task 0.0 in stage 4.0 >>>> (TID 78, >>>> > 192.168.122.39): java.io.IOException: >>>> java.lang.ClassNotFoundException: >>>> > org.apache.spark.streaming.twitter.TwitterReceiver >>>> > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1163) >>>> > at >>>> > >>>> org.apache.spark.rdd.ParallelCollectionPartition.readObject(ParallelCollectionRDD.scala:70) >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> > at >>>> > >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> > at >>>> > >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> > at java.lang.reflect.Method.invoke(Method.java:497) >>>> > at >>>> > >>>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) >>>> > at >>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) >>>> > at >>>> > >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>>> > at >>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>>> > at >>>> > >>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) >>>> > at >>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>>> > at >>>> > >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>>> > at >>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>>> > at >>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>>> > at >>>> > >>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:72) >>>> > at >>>> > >>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:98) >>>> > at >>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:194) >>>> > at >>>> > >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> > at >>>> > >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> > at java.lang.Thread.run(Thread.java:745) >>>> > Caused by: java.lang.ClassNotFoundException: >>>> > org.apache.spark.streaming.twitter.TwitterReceiver >>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>> > at java.lang.Class.forName0(Native Method) >>>> > at java.lang.Class.forName(Class.java:348) >>>> > at >>>> > >>>> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:67) >>>> > at >>>> > >>>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613) >>>> > at >>>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518) >>>> > at >>>> > >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774) >>>> > at >>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>>> > at >>>> java.io.ObjectInputStream.readArray(ObjectInputStream.java:1707) >>>> > at >>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1345) >>>> > at >>>> > >>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) >>>> > at >>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>>> > at >>>> > >>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) >>>> > at >>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>>> > at >>>> > >>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) >>>> > at >>>> > >>>> java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:501) >>>> > at >>>> > >>>> org.apache.spark.rdd.ParallelCollectionPartition$$anonfun$readObject$1.apply$mcV$sp(ParallelCollectionRDD.scala:74) >>>> > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1160) >>>> > ... 20 more >>>> > I run the application locally from Eclipse. It is clear the problem >>>> is in >>>> > the dependencies but I can not figure out how to solve it. >>>> > >>>> > >>>> > >>>> > -- >>>> > View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-ClassNotFoundException-org-apache-spark-streaming-twitter-TwitterReceiver-tp25324.html >>>> > Sent from the Apache Spark User List mailing list archive at >>>> Nabble.com. >>>> > >>>> > --------------------------------------------------------------------- >>>> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> > For additional commands, e-mail: user-h...@spark.apache.org >>>> > >>>> >>> >>> >>> >>> -- >>> Anas Rabei >>> Senior Software Developer >>> Mubasher.info >>> anas.ra...@mubasher.info >>> >> >> >> >> -- >> Anas Rabei >> Senior Software Developer >> Mubasher.info >> anas.ra...@mubasher.info >> > > -- Anas Rabei Senior Software Developer Mubasher.info anas.ra...@mubasher.info