Hi Paul: Here are the dependencies in spark 1.1.0-snapshot that are pulling in org.codehaus.jackson:jackson-core-asl 1.8 and 1.9 jar.
1.9 com.twitter:parquet-hadoop:jar:1.4.3 org.apache.avro:avro:jar:1.7.6 1.8 org.apache.spark:spark-hive_2.10:jar:1.1.0-SNAPSHOT org.apache.hadoop:hadoop-core:jar:1.0.4 org.apache.hbase:hbase:jar:0.94.6 Thanks Mans On Saturday, June 28, 2014 2:22 AM, Paul Brown <p...@mult.ifario.us> wrote: Hi, Mans -- Both of those versions of Jackson are pretty ancient. Do you know which of the Spark dependencies is pulling them in? It would be good for us (the Jackson, Woodstox, etc., folks) to see if we can get people to upgrade to more recent versions of Jackson. -- Paul — p...@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/ On Fri, Jun 27, 2014 at 12:58 PM, M Singh <mans6si...@yahoo.com> wrote: Hi: > > >I am using spark to stream data to cassandra and it works fine in local mode. >But when I execute the application in a standalone clustered env I got >exception included below (java.lang.NoClassDefFoundError: >org/codehaus/jackson/annotate/JsonClass). > > >I think this is due to the jackson-core-asl dependency conflict >(jackson-core-asl 1.8.8 has the JsonClass but 1.9.x does not). The 1.9.x >version is being pulled in by spark-sql project. I tried adding >jackson-core-asl 1.8.8 with --jars argument while submitting the application >for execution but it did not work. So I created a custom spark build >excluding sql project. With this custom spark install I was able to resolve >the issue at least on a single node cluster (separate master and worker). > >If there is an alternate way to resolve this conflicting jar issue without a >custom build (eg: configuration to use the user defined jars in the executor >class path first), please let me know. > >Also, is there a comprehensive list of configuration properties available for >spark ? > > > >Thanks > > >Mans > > >Exception trace > > > TaskSetManager: Loss was due to java.lang.NoClassDefFoundError >java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass >at >org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524) >at >org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732) >at org.codehaus.jackson.map.deser.BasicDeserializerFactory.createCollectionDeserializer(BasicDeserializerFactory.java:229) >at >org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:386) >at >org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307) >at >org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287) >at >org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136) >at >org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157) >at >org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468) >at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402) >at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1602) > >
