Hi,
I took a source code of Spark 1.2.0 and tried to build it together with
hadoop-openstack.jar ( To allow Spark an access to OpenStack Swift )
I used Hadoop 2.6.0.
The build was fine without problems, however in run time, while trying to
access "swift://" name space i got an exception:
java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
at
org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524)
at
org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732)
...and the long stack trace goes here
Digging into the problem i saw the following:
Jackson versions 1.9.X are not backward compatible, in particular they
removed JsonClass annotation.
Hadoop 2.6.0 uses jackson-asl version 1.9.13, while Spark has reference to
older version of jackson.
This is the main pom.xml of Spark 1.2.0 :
<dependency>
<!-- Matches the version of jackson-core-asl pulled in by avro -->
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.8.8</version>
</dependency>
Referencing 1.8.8 version, which is not compatible with Hadoop 2.6.0 .
If we change version to 1.9.13, than all will work fine and there will be
no run time exceptions while accessing Swift. The following change will
solve the problem:
<dependency>
<!-- Matches the version of jackson-core-asl pulled in by avro -->
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.13</version>
</dependency>
I am trying to resolve this somehow so people will not get into this
issue.
Is there any particular need in Spark for jackson 1.8.8 and not 1.9.13?
Can we remove 1.8.8 and put 1.9.13 for Avro?
It looks to me that all works fine when Spark build with jackson 1.9.13,
but i am not an expert and not sure what should be tested.
Thanks,
Gil Vernik.
