There is a lot of chatter about AWS and Jackson on their forums, etc. I have been using the AWS SDK and Jackson 2.8.9 for a couple of weeks without problems. Adding Spark to the mix is what changes this.
Jacob On Mon, Oct 2, 2017 at 11:14 AM, Romain Manni-Bucau <[email protected]> wrote: > Hi Jacob, > > isn't aws API only supporting jackson 2.6 and not 2.8? > > > Romain Manni-Bucau > @rmannibucau <https://twitter.com/rmannibucau> | Blog > <https://rmannibucau.metawerx.net/> | Old Blog > <http://rmannibucau.wordpress.com> | Github <https://github.com/ > rmannibucau> | > LinkedIn <https://www.linkedin.com/in/rmannibucau> > > 2017-10-02 20:13 GMT+02:00 Jacob Marble <[email protected]>: > > > Yes, I'm using spark-submit, and I'm giving it a shaded jar. > > > > What do you mean "aligning the dependencies"? > > > > Jacob > > > > On Mon, Oct 2, 2017 at 11:06 AM, Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi > > > > > > Do you start your pipeline with spark-submit ? If so you can provide > the > > > packages. You can also create a shaded jar. > > > > > > I have a similar issue in the spark 2 runner that I worked around by > > > aligning the dependencies. > > > > > > Regards > > > JB > > > > > > On Oct 2, 2017, 20:04, at 20:04, Jacob Marble <[email protected]> > > wrote: > > > >My Beam pipeline runs fine with DirectRunner and DataflowRunner, but > > > >fails > > > >with SparkRunner. That stack trace is after this message. > > > > > > > >The exception indicates that > > > >com.fasterxml.jackson.databind.ObjectMapper.enable doesn't exist. > > > >ObjectMapper.enable() didn't exist until Jackson 2.5. `mvn > > > >dependency:tree > > > >-Dverbose` shows that spark-core_2.10 (1.6.3) and beam-runners-spark > > > >(2.1.0) both request versions of Jackson before 2.5. > > > > > > > >Since I'm using a local, standalone Spark cluster for development, I > > > >have > > > >to include spark-core_2.10 version 1.6.3 in dependencies. > > > > > > > >I have added explicit dependencies to my pom.xml, so that I can be > > > >certain > > > >that the more recent version of Jackson is included in my shaded jar. > > > >`mvn > > > >clean package` confirms this: > > > > > > > >[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.8.9 in > > > >the > > > >shaded jar. > > > >[INFO] Including > > > >com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9 > > > >in the shaded jar. > > > >[INFO] Including com.fasterxml.jackson.core: > jackson-databind:jar:2.8.9 > > > >in > > > >the shaded jar. > > > >[INFO] Including > > > >com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.8.9 in > the > > > >shaded jar. > > > >[INFO] Including > > > >com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.8.9 in > the > > > >shaded jar. > > > >[INFO] Including > > > >com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.8.9 in > > > >the > > > >shaded jar. > > > > > > > >Beyond jar creation, is there anything I can do to ensure that my > > > >chosen > > > >version of a dependency is used when Spark runs my pipeline? I can't > be > > > >the > > > >first to encounter this problem. > > > > > > > >Thanks! > > > > > > > >Jacob > > > > > > > >-------- > > > > > > > >Exception in thread "main" java.lang.RuntimeException: > > > >java.lang.NoSuchMethodError: > > > >com.fasterxml.jackson.databind.ObjectMapper.enable([ > > > Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/ > > > fasterxml/jackson/databind/ObjectMapper; > > > >at > > > >org.apache.beam.runners.spark.SparkPipelineResult. > runtimeExceptionFrom( > > > SparkPipelineResult.java:55) > > > >at > > > >org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom( > > > SparkPipelineResult.java:72) > > > >at > > > >org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish( > > > SparkPipelineResult.java:99) > > > >at > > > >org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish( > > > SparkPipelineResult.java:87) > > > >at com.kochava.beam.jobs.ExampleS3.main(ExampleS3.java:46) > > > >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > >at > > > >sun.reflect.NativeMethodAccessorImpl.invoke( > > > NativeMethodAccessorImpl.java:62) > > > >at > > > >sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > DelegatingMethodAccessorImpl.java:43) > > > >at java.lang.reflect.Method.invoke(Method.java:498) > > > >at > > > >org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ > > > deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) > > > >at > > > >org.apache.spark.deploy.SparkSubmit$.doRunMain$1( > SparkSubmit.scala:181) > > > >at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) > > > >at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) > > > >at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > > >Caused by: java.lang.NoSuchMethodError: > > > >com.fasterxml.jackson.databind.ObjectMapper.enable([ > > > Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/ > > > fasterxml/jackson/databind/ObjectMapper; > > > >at > > > >com.amazonaws.partitions.PartitionsLoader.<clinit>( > > > PartitionsLoader.java:54) > > > >at > > > >com.amazonaws.regions.RegionMetadataFactory.create( > > > RegionMetadataFactory.java:30) > > > >at com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:64) > > > >at > > > >com.amazonaws.regions.RegionUtils.getRegionMetadata( > > RegionUtils.java:52) > > > >at com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:105) > > > >at > > > >com.amazonaws.client.builder.AwsClientBuilder.withRegion( > > > AwsClientBuilder.java:239) > > > >at com.kochava.beam.s3.S3Util.<init>(S3Util.java:103) > > > >at com.kochava.beam.s3.S3Util.<init>(S3Util.java:53) > > > >at com.kochava.beam.s3.S3Util$S3UtilFactory.create(S3Util.java:81) > > > >at com.kochava.beam.s3.S3Util$S3UtilFactory.create(S3Util.java:55) > > > > > >
