It works after sync, thanks for the pointers On Tue, Mar 10, 2015 at 1:22 PM, Mohit Anchlia <mohitanch...@gmail.com> wrote:
> I navigated to maven dependency and found scala library. I also found > Tuple2.class and when I click on it in eclipse I get "invalid LOC header > (bad signature)" > > java.util.zip.ZipException: invalid LOC header (bad signature) > at java.util.zip.ZipFile.read(Native Method) > > I am wondering if I should delete that file from local repo and re-sync > > On Tue, Mar 10, 2015 at 1:08 PM, Mohit Anchlia <mohitanch...@gmail.com> > wrote: > >> I ran the dependency command and see the following dependencies: >> >> I only see org.scala-lang. >> >> [INFO] org.spark.test:spak-test:jar:0.0.1-SNAPSHOT >> >> [INFO] +- org.apache.spark:spark-streaming_2.10:jar:1.2.0:compile >> >> [INFO] | +- org.eclipse.jetty:jetty-server:jar:8.1.14.v20131031:compile >> >> [INFO] | | +- >> org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016:co mpile >> >> [INFO] | | +- >> org.eclipse.jetty:jetty-continuation:jar:8.1.14.v20131031:compil e >> >> [INFO] | | \- org.eclipse.jetty:jetty-http:jar:8.1.14.v20131031:compile >> >> [INFO] | | \- org.eclipse.jetty:jetty-io:jar:8.1.14.v20131031:compile >> >> [INFO] | +- org.scala-lang:scala-library:jar:2.10.4:compile >> >> [INFO] | \- org.spark-project.spark:unused:jar:1.0.0:compile >> >> [INFO] \- org.apache.spark:spark-core_2.10:jar:1.2.1:compile >> >> [INFO] +- com.twitter:chill_2.10:jar:0.5.0:compile >> >> [INFO] | \- com.esotericsoftware.kryo:kryo:jar:2.21:compile >> >> [INFO] | +- com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:co >> mpile >> >> [INFO] | +- com.esotericsoftware.minlog:minlog:jar:1.2:compile >> >> [INFO] | \- org.objenesis:objenesis:jar:1.2:compile >> >> [INFO] +- com.twitter:chill-java:jar:0.5.0:compile >> >> [INFO] +- org.apache.hadoop:hadoop-client:jar:2.2.0:compile >> >> [INFO] | +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile >> >> [INFO] | | +- commons-cli:commons-cli:jar:1.2:compile >> >> [INFO] | | +- org.apache.commons:commons-math:jar:2.1:compile >> >> [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile >> >> [INFO] | | +- commons-io:commons-io:jar:2.1:compile >> >> [INFO] | | +- commons-logging:commons-logging:jar:1.1.1:compile >> >> [INFO] | | +- commons-lang:commons-lang:jar:2.5:compile >> >> [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile >> >> [INFO] | | | +- commons-collections:commons-collections:jar:3.2.1:compile >> >> [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile >> >> [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile >> >> [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile >> >> [INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile >> >> [INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile >> >> [INFO] | | +- org.apache.avro:avro:jar:1.7.4:compile >> >> [INFO] | | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile >> >> [INFO] | | +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile >> >> [INFO] | | \- org.apache.commons:commons-compress:jar:1.4.1:compile >> >> [INFO] | | \- org.tukaani:xz:jar:1.0:compile >> >> [INFO] | +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:compile >> >> [INFO] | | \- org.mortbay.jetty:jetty-util:jar:6.1.26:compile >> >> [INFO] | +- >> org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:compile >> >> [INFO] | | +- >> org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:co mpile >> >> [INFO] | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile >> >> [INFO] | | | | +- com.google.inject:guice:jar:3.0:compile >> >> [INFO] | | | | | +- javax.inject:javax.inject:jar:1:compile >> >> [INFO] | | | | | \- aopalliance:aopalliance:jar:1.0:compile >> >> [INFO] | | | | +- com.sun.jersey.jersey-test-framework:jersey-test-framew >> ork-grizzly2:jar:1.9:compile >> >> [INFO] | | | | | +- com.sun.jersey.jersey-test-framework:jersey-test-fra >> mework-core:jar:1.9:compile >> >> [INFO] | | | | | | +- javax.servlet:javax.servlet-api:jar:3.0.1:compile >> >> [INFO] | | | | | | \- com.sun.jersey:jersey-client:jar:1.9:compile >> >> [INFO] | | | | | \- com.sun.jersey:jersey-grizzly2:jar:1.9:compile >> >> [INFO] | | | | | +- org.glassfish.grizzly:grizzly-http:jar:2.1.2:comp ile >> >> [INFO] | | | | | | \- org.glassfish.grizzly:grizzly-framework:jar:2. >> 1.2:compile >> >> [INFO] | | | | | | \- org.glassfish.gmbal:gmbal-api-only:jar:3.0. >> 0-b023:compile >> >> [INFO] | | | | | | \- org.glassfish.external:management-api:ja >> r:3.0.0-b012:compile >> >> [INFO] | | | | | +- org.glassfish.grizzly:grizzly-http-server:jar:2.1 >> .2:compile >> >> [INFO] | | | | | | \- org.glassfish.grizzly:grizzly-rcm:jar:2.1.2:co mpile >> >> [INFO] | | | | | +- org.glassfish.grizzly:grizzly-http-servlet:jar:2. >> 1.2:compile >> >> [INFO] | | | | | \- org.glassfish:javax.servlet:jar:3.1:compile >> >> [INFO] | | | | +- com.sun.jersey:jersey-server:jar:1.9:compile >> >> [INFO] | | | | | +- asm:asm:jar:3.1:compile >> >> [INFO] | | | | | \- com.sun.jersey:jersey-core:jar:1.9:compile >> >> [INFO] | | | | +- com.sun.jersey:jersey-json:jar:1.9:compile >> >> [INFO] | | | | | +- org.codehaus.jettison:jettison:jar:1.1:compile >> >> [INFO] | | | | | | \- stax:stax-api:jar:1.0.1:compile >> >> [INFO] | | | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile >> >> [INFO] | | | | | | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile >> >> [INFO] | | | | | | \- javax.activation:activation:jar:1.1:compile >> >> [INFO] | | | | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile >> >> [INFO] | | | | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile >> >> [INFO] | | | | \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile >> >> [INFO] | | | \- >> org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:comp ile >> >> [INFO] | | \- >> org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:c ompile >> >> [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile >> >> [INFO] | +- >> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:compile >> >> [INFO] | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile >> >> [INFO] | +- >> org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:co mpile >> >> [INFO] | \- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile >> >> [INFO] +- org.apache.spark:spark-network-common_2.10:jar:1.2.1:compile >> >> [INFO] +- org.apache.spark:spark-network-shuffle_2.10:jar:1.2.1:compile >> >> [INFO] +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile >> >> [INFO] | +- commons-codec:commons-codec:jar:1.3:compile >> >> [INFO] | \- commons-httpclient:commons-httpclient:jar:3.1:compile >> >> [INFO] +- org.apache.curator:curator-recipes:jar:2.4.0:compile >> >> [INFO] | +- org.apache.curator:curator-framework:jar:2.4.0:compile >> >> [INFO] | | \- org.apache.curator:curator-client:jar:2.4.0:compile >> >> [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile >> >> [INFO] | | \- jline:jline:jar:0.9.94:compile >> >> [INFO] | \- com.google.guava:guava:jar:14.0.1:compile >> >> [INFO] +- org.eclipse.jetty:jetty-plus:jar:8.1.14.v20131031:compile >> >> [INFO] | +- >> org.eclipse.jetty.orbit:javax.transaction:jar:1.1.1.v20110521064 5:compile >> >> [INFO] | +- org.eclipse.jetty:jetty-webapp:jar:8.1.14.v20131031:compile >> >> [INFO] | | +- org.eclipse.jetty:jetty-xml:jar:8.1.14.v20131031:compile >> >> [INFO] | | \- org.eclipse.jetty:jetty-servlet:jar:8.1.14.v20131031:compile >> >> [INFO] | \- org.eclipse.jetty:jetty-jndi:jar:8.1.14.v20131031:compile >> >> [INFO] | \- org.eclipse.jetty.orbit:javax.mail.glassfish:jar:1.4.1.v20100 >> 5082020:compile >> >> [INFO] | \- org.eclipse.jetty.orbit:javax.activation:jar:1.1.0.v201105 >> 071233:compile >> >> [INFO] +- org.eclipse.jetty:jetty-security:jar:8.1.14.v20131031:compile >> >> [INFO] +- org.eclipse.jetty:jetty-util:jar:8.1.14.v20131031:compile >> >> [INFO] +- org.apache.commons:commons-lang3:jar:3.3.2:compile >> >> [INFO] +- org.apache.commons:commons-math3:jar:3.1.1:compile >> >> [INFO] +- com.google.code.findbugs:jsr305:jar:1.3.9:compile >> >> [INFO] +- org.slf4j:slf4j-api:jar:1.7.5:compile >> >> [INFO] +- org.slf4j:jul-to-slf4j:jar:1.7.5:compile >> >> [INFO] +- org.slf4j:jcl-over-slf4j:jar:1.7.5:compile >> >> [INFO] +- log4j:log4j:jar:1.2.17:compile >> >> [INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile >> >> [INFO] +- com.ning:compress-lzf:jar:1.0.0:compile >> >> [INFO] +- org.xerial.snappy:snappy-java:jar:1.1.1.6:compile >> >> [INFO] +- net.jpountz.lz4:lz4:jar:1.2.0:compile >> >> [INFO] +- org.roaringbitmap:RoaringBitmap:jar:0.4.5:compile >> >> [INFO] +- commons-net:commons-net:jar:2.2:compile >> >> [INFO] +- org.spark-project.akka:akka-remote_2.10:jar:2.3.4-spark:compile >> >> [INFO] | +- org.spark-project.akka:akka-actor_2.10:jar:2.3.4-spark:compile >> >> [INFO] | | \- com.typesafe:config:jar:1.2.1:compile >> >> [INFO] | +- io.netty:netty:jar:3.8.0.Final:compile >> >> [INFO] | +- >> org.spark-project.protobuf:protobuf-java:jar:2.5.0-spark:compile >> >> [INFO] | \- org.uncommons.maths:uncommons-maths:jar:1.2.2a:compile >> >> [INFO] +- org.spark-project.akka:akka-slf4j_2.10:jar:2.3.4-spark:compile >> >> [INFO] +- org.json4s:json4s-jackson_2.10:jar:3.2.10:compile >> >> [INFO] | +- org.json4s:json4s-core_2.10:jar:3.2.10:compile >> >> [INFO] | | +- org.json4s:json4s-ast_2.10:jar:3.2.10:compile >> >> [INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.6:compile >> >> [INFO] | | \- org.scala-lang:scalap:jar:2.10.0:compile >> >> [INFO] | | \- org.scala-lang:scala-compiler:jar:2.10.0:compile >> >> [INFO] | | \- org.scala-lang:scala-reflect:jar:2.10.0:compile >> >> [INFO] | \- com.fasterxml.jackson.core:jackson-databind:jar:2.3.1:compile >> >> [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0:comp >> ile >> >> [INFO] | \- com.fasterxml.jackson.core:jackson-core:jar:2.3.1:compile >> >> [INFO] +- org.apache.mesos:mesos:jar:shaded-protobuf:0.18.1:compile >> >> [INFO] +- io.netty:netty-all:jar:4.0.23.Final:compile >> >> [INFO] +- com.clearspring.analytics:stream:jar:2.7.0:compile >> >> [INFO] +- com.codahale.metrics:metrics-core:jar:3.0.0:compile >> >> [INFO] +- com.codahale.metrics:metrics-jvm:jar:3.0.0:compile >> >> [INFO] +- com.codahale.metrics:metrics-json:jar:3.0.0:compile >> >> [INFO] +- com.codahale.metrics:metrics-graphite:jar:3.0.0:compile >> >> [INFO] +- org.tachyonproject:tachyon-client:jar:0.5.0:compile >> >> [INFO] | \- org.tachyonproject:tachyon:jar:0.5.0:compile >> >> [INFO] +- org.spark-project:pyrolite:jar:2.0.1:compile >> >> [INFO] \- net.sf.py4j:py4j:jar:0.8.2.1:compile >> >> On Tue, Mar 10, 2015 at 11:40 AM, Sean Owen <so...@cloudera.com> wrote: >> >>> A couple points: >>> >>> You've got mismatched versions here -- 1.2.0 vs 1.2.1. You should fix >>> that but it's not your problem. >>> >>> These are also supposed to be 'provided' scope dependencies in Maven. >>> >>> You should get the Scala deps transitively and can import scala.* >>> classes. However, it would be a little bit more correct to depend >>> directly on the scala library classes, but in practice, easiest not to >>> in simple use cases. >>> >>> If you're still having trouble look at the output of "mvn >>> dependency:tree" >>> >>> On Tue, Mar 10, 2015 at 6:32 PM, Mohit Anchlia <mohitanch...@gmail.com> >>> wrote: >>> > I am using maven and my dependency looks like this, but this doesn't >>> seem to >>> > be working >>> > >>> > <dependencies> >>> > >>> > <dependency> >>> > >>> > <groupId>org.apache.spark</groupId> >>> > >>> > <artifactId>spark-streaming_2.10</artifactId> >>> > >>> > <version>1.2.0</version> >>> > >>> > </dependency> >>> > >>> > <dependency> >>> > >>> > <groupId>org.apache.spark</groupId> >>> > >>> > <artifactId>spark-core_2.10</artifactId> >>> > >>> > <version>1.2.1</version> >>> > >>> > </dependency> >>> > >>> > </dependencies> >>> > >>> > >>> > On Tue, Mar 10, 2015 at 11:06 AM, Tathagata Das <t...@databricks.com> >>> wrote: >>> >> >>> >> If you are using tools like SBT/Maven/Gradle/etc, they figure out all >>> the >>> >> recursive dependencies and includes them in the class path. I haven't >>> >> touched Eclipse in years so I am not sure off the top of my head >>> what's >>> >> going on instead. Just in case you only downloaded the >>> >> spark-streaming_2.10.jar then that is indeed insufficient and you >>> have to >>> >> download all the recursive dependencies. May be you should create a >>> Maven >>> >> project inside Eclipse? >>> >> >>> >> TD >>> >> >>> >> On Tue, Mar 10, 2015 at 11:00 AM, Mohit Anchlia < >>> mohitanch...@gmail.com> >>> >> wrote: >>> >>> >>> >>> How do I do that? I haven't used Scala before. >>> >>> >>> >>> Also, linking page doesn't mention that: >>> >>> >>> >>> >>> >>> >>> http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#linking >>> >>> >>> >>> On Tue, Mar 10, 2015 at 10:57 AM, Sean Owen <so...@cloudera.com> >>> wrote: >>> >>>> >>> >>>> It means you do not have Scala library classes in your project >>> >>>> classpath. >>> >>>> >>> >>>> On Tue, Mar 10, 2015 at 5:54 PM, Mohit Anchlia < >>> mohitanch...@gmail.com> >>> >>>> wrote: >>> >>>> > I am trying out streaming example as documented and I am using >>> spark >>> >>>> > 1.2.1 >>> >>>> > streaming from maven for Java. >>> >>>> > >>> >>>> > When I add this code I get compilation error on and eclipse is not >>> >>>> > able to >>> >>>> > recognize Tuple2. I also don't see any "import scala.Tuple2" >>> class. >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> > >>> http://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#a-quick-example >>> >>>> > >>> >>>> > >>> >>>> > private void map(JavaReceiverInputDStream<String> lines) { >>> >>>> > >>> >>>> > JavaDStream<String> words = lines.flatMap( >>> >>>> > >>> >>>> > new FlatMapFunction<String, String>() { >>> >>>> > >>> >>>> > @Override public Iterable<String> call(String x) { >>> >>>> > >>> >>>> > return Arrays.asList(x.split(" ")); >>> >>>> > >>> >>>> > } >>> >>>> > >>> >>>> > }); >>> >>>> > >>> >>>> > // Count each word in each batch >>> >>>> > >>> >>>> > JavaPairDStream<String, Integer> pairs = words.map( >>> >>>> > >>> >>>> > new PairFunction<String, String, Integer>() { >>> >>>> > >>> >>>> > @Override public Tuple2<String, Integer> call(String s) throws >>> >>>> > Exception { >>> >>>> > >>> >>>> > return new Tuple2<String, Integer>(s, 1); >>> >>>> > >>> >>>> > } >>> >>>> > >>> >>>> > }); >>> >>>> > >>> >>>> > } >>> >>> >>> >>> >>> >> >>> > >>> >> >> >
