Re: spark-1.1.0-bin-hadoop2.4 java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
After lots of hacking I figure out how to resolve this problem. This is good
solution. It severalty cripples jackson but at least for now I am unblocked
1) turn off annotations.
mapper.configure(Feature.USE_ANNOTATIONS, false);
2) in maven set the jackson dependencies as provided.
properties
jackson.version1.9.13/jackson.version !-- 1.8.8 1.9.13 2.3.4-Final --
/properties
dependencies
dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-core-asl/artifactId
scopeprovided/scope
version${jackson.version}/version
/dependency
dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-mapper-asl/artifactId
scopeprovided/scope
version${jackson.version}/version
/dependency
Hope this helps someone else
andy
From: Andrew Davidson a...@santacruzintegration.com
Date: Wednesday, September 17, 2014 at 4:28 PM
To: user@spark.apache.org user@spark.apache.org
Subject: spark-1.1.0-bin-hadoop2.4 java.lang.NoClassDefFoundError:
org/codehaus/jackson/annotate/JsonClass
Hi I am new to spark.
I am trying to write a simple java program that process tweets that where
collected and stored in a file. I figured the simplest thing to do would be to
convert the JSON string into a java map. When I submit my jar file I keep
getting the following error
java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
For the life of me I can not figure out what the problem is. I am using maven
shade plugin and check to see that the missing class is my uber jar
Any suggestions would be greatly appreciated.
Andy
P.s. I should mention that I am running everything on my local machine.
properties
jackson.version1.8.8/jackson.version !-- 1.9.13 --
project.build.sourceEncodingUTF-8/project.build.sourceEncoding
/properties
dependencies
dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-core-asl/artifactId
version${jackson.version}/version
/dependency
dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-mapper-asl/artifactId
version${jackson.version}/version
/dependency
I am using shade to build an uber jar
$jar tvf target/myUber.jar
580 Wed Sep 17 16:17:36 PDT 2014
org/codehaus/jackson/annotate/JsonClass.class
14/09/17 16:13:24 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
at
org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserial
izationType(JacksonAnnotationIntrospector.java:524)
at
org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation
(BasicDeserializerFactory.java:732)
at
org.codehaus.jackson.map.deser.BasicDeserializerFactory.createMapDeserializer(
BasicDeserializerFactory.java:337)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(Std
DeserializerProvider.java:377)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDes
erializerProvider.java:307)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDes
erializer(StdDeserializerProvider.java:287)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(S
tdDeserializerProvider.java:136)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeseriali
zer(StdDeserializerProvider.java:157)
at
org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:
2468)
at
org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1609)
at com.santacruzintegration.spark.JSONHelper.toJsonMap(JSONHelper.java:40)
public class JSONHelper {
public static final ObjectMapper mapper = new ObjectMapper();
static {
mapper.configure(Feature.FAIL_ON_UNKNOWN_PROPERTIES, false);
//mapper.configure(Feature.USE_ANNOTATIONS, false);
}
public static MapString, String toJsonMap(String json) {
System.err.println(AEDWIP toJsonMap: + json);
try {
TypeReferenceHashMapString, String typeRef = new
TypeReferenceHashMapString, String() {
};
//return mapper.readValue(json, new
TypeReferenceHashMapString,String(){});
return mapper.readValue(json, typeRef);
//return mapper.convertValue(json, typeRef);
//HashMapString,String ret =
//new ObjectMapper().readValue(json, HashMap.class);
} catch (Exception e) {
throw new IOError(e);
}
}
Re: spark-1.1.0-bin-hadoop2.4 java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
Hello Andy,
Will our JSON support in Spark SQL help your case? If your JSON files store
one JSON object per line, you can use SQLContext.jsonFile to load it. If
you want to do pre-process these files, once you have an RDD[String] (one
JSON object per String), you can use SQLContext.jsonRDD. In both ways,
Spark SQL will infer the schema and you can query your datasets like
querying tables. The instruction about using jsonFile and jsonRDD can be
found at
https://spark.apache.org/docs/latest/sql-programming-guide.html#json-datasets
.
Thanks,
Yin
On Thu, Sep 18, 2014 at 2:36 PM, Andy Davidson
a...@santacruzintegration.com wrote:
After lots of hacking I figure out how to resolve this problem. This is
good solution. It severalty cripples jackson but at least for now I am
unblocked
1) turn off annotations.
mapper.configure(Feature.USE_ANNOTATIONS, false);
2) in maven set the jackson dependencies as provided.
properties
jackson.version1.9.13/jackson.version !-- 1.8.8 1.9.13 2.3.4-Final
--
/properties
dependencies
dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-core-asl/artifactId
scopeprovided/scope
version${jackson.version}/version
/dependency
dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-mapper-asl/artifactId
scopeprovided/scope
version${jackson.version}/version
/dependency
Hope this helps someone else
andy
From: Andrew Davidson a...@santacruzintegration.com
Date: Wednesday, September 17, 2014 at 4:28 PM
To: user@spark.apache.org user@spark.apache.org
Subject: spark-1.1.0-bin-hadoop2.4 java.lang.NoClassDefFoundError:
org/codehaus/jackson/annotate/JsonClass
Hi I am new to spark.
I am trying to write a simple java program that process tweets that where
collected and stored in a file. I figured the simplest thing to do would be
to convert the JSON string into a java map. When I submit my jar file I
keep getting the following error
java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
For the life of me I can not figure out what the problem is. I am using
maven shade plugin and check to see that the missing class is my uber jar
Any suggestions would be greatly appreciated.
Andy
P.s. I should mention that I am running everything on my local machine.
properties
jackson.version1.8.8/jackson.version !-- 1.9.13 --
project.build.sourceEncodingUTF-8/project.build.sourceEncoding
/properties
dependencies
dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-core-asl/artifactId
version${jackson.version}/version
/dependency
dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-mapper-asl/artifactId
version${jackson.version}/version
/dependency
I am using shade to build an uber jar
$jar –tvf target/myUber.jar
…
580 Wed Sep 17 16:17:36 PDT 2014
org/codehaus/jackson/annotate/JsonClass.class
…
14/09/17 16:13:24 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID
0)
java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
at
org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524)
at
org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732)
at
org.codehaus.jackson.map.deser.BasicDeserializerFactory.createMapDeserializer(BasicDeserializerFactory.java:337)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:377)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157)
at
org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468)
at
org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1609)
at com.santacruzintegration.spark.JSONHelper.toJsonMap(JSONHelper.java:40)
public class JSONHelper {
public static final ObjectMapper mapper = new ObjectMapper();
static {
mapper.configure(Feature.FAIL_ON_UNKNOWN_PROPERTIES, false);
//mapper.configure(Feature.USE_ANNOTATIONS, false);
}
…
public static MapString, String toJsonMap(String json) {
System.err.println(AEDWIP toJsonMap: + json);
try {
TypeReferenceHashMapString, String typeRef = new
TypeReferenceHashMapString, String() {
};
//return mapper.readValue(json, new
TypeReferenceHashMapString,String
spark-1.1.0-bin-hadoop2.4 java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
Hi I am new to spark.
I am trying to write a simple java program that process tweets that where
collected and stored in a file. I figured the simplest thing to do would be
to convert the JSON string into a java map. When I submit my jar file I keep
getting the following error
java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
For the life of me I can not figure out what the problem is. I am using
maven shade plugin and check to see that the missing class is my uber jar
Any suggestions would be greatly appreciated.
Andy
P.s. I should mention that I am running everything on my local machine.
properties
jackson.version1.8.8/jackson.version !-- 1.9.13 --
project.build.sourceEncodingUTF-8/project.build.sourceEncoding
/properties
dependencies
dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-core-asl/artifactId
version${jackson.version}/version
/dependency
dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-mapper-asl/artifactId
version${jackson.version}/version
/dependency
I am using shade to build an uber jar
$jar tvf target/myUber.jar
580 Wed Sep 17 16:17:36 PDT 2014
org/codehaus/jackson/annotate/JsonClass.class
14/09/17 16:13:24 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
at
org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeseri
alizationType(JacksonAnnotationIntrospector.java:524)
at
org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotati
on(BasicDeserializerFactory.java:732)
at
org.codehaus.jackson.map.deser.BasicDeserializerFactory.createMapDeserialize
r(BasicDeserializerFactory.java:337)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(S
tdDeserializerProvider.java:377)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdD
eserializerProvider.java:307)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueD
eserializer(StdDeserializerProvider.java:287)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer
(StdDeserializerProvider.java:136)
at
org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeseria
lizer(StdDeserializerProvider.java:157)
at
org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.jav
a:2468)
at
org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:240
2)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1609)
at com.santacruzintegration.spark.JSONHelper.toJsonMap(JSONHelper.java:40)
public class JSONHelper {
public static final ObjectMapper mapper = new ObjectMapper();
static {
mapper.configure(Feature.FAIL_ON_UNKNOWN_PROPERTIES, false);
//mapper.configure(Feature.USE_ANNOTATIONS, false);
}
public static MapString, String toJsonMap(String json) {
System.err.println(AEDWIP toJsonMap: + json);
try {
TypeReferenceHashMapString, String typeRef = new
TypeReferenceHashMapString, String() {
};
//return mapper.readValue(json, new
TypeReferenceHashMapString,String(){});
return mapper.readValue(json, typeRef);
//return mapper.convertValue(json, typeRef);
//HashMapString,String ret =
//new ObjectMapper().readValue(json, HashMap.class);
} catch (Exception e) {
throw new IOError(e);
}
}
