My Beam pipeline runs fine with DirectRunner and DataflowRunner, but fails with SparkRunner. That stack trace is after this message.
The exception indicates that com.fasterxml.jackson.databind.ObjectMapper.enable doesn't exist. ObjectMapper.enable() didn't exist until Jackson 2.5. `mvn dependency:tree -Dverbose` shows that spark-core_2.10 (1.6.3) and beam-runners-spark (2.1.0) both request versions of Jackson before 2.5. Since I'm using a local, standalone Spark cluster for development, I have to include spark-core_2.10 version 1.6.3 in dependencies. I have added explicit dependencies to my pom.xml, so that I can be certain that the more recent version of Jackson is included in my shaded jar. `mvn clean package` confirms this: [INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.8.9 in the shaded jar. [INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9 in the shaded jar. [INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.8.9 in the shaded jar. [INFO] Including com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.8.9 in the shaded jar. [INFO] Including com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.8.9 in the shaded jar. [INFO] Including com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.8.9 in the shaded jar. Beyond jar creation, is there anything I can do to ensure that my chosen version of a dependency is used when Spark runs my pipeline? I can't be the first to encounter this problem. Thanks! Jacob -------- Exception in thread "main" java.lang.RuntimeException: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper; at org.apache.beam.runners.spark.SparkPipelineResult.runtimeExceptionFrom(SparkPipelineResult.java:55) at org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:72) at org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:99) at org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:87) at com.kochava.beam.jobs.ExampleS3.main(ExampleS3.java:46) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper; at com.amazonaws.partitions.PartitionsLoader.<clinit>(PartitionsLoader.java:54) at com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30) at com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:64) at com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:52) at com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:105) at com.amazonaws.client.builder.AwsClientBuilder.withRegion(AwsClientBuilder.java:239) at com.kochava.beam.s3.S3Util.<init>(S3Util.java:103) at com.kochava.beam.s3.S3Util.<init>(S3Util.java:53) at com.kochava.beam.s3.S3Util$S3UtilFactory.create(S3Util.java:81) at com.kochava.beam.s3.S3Util$S3UtilFactory.create(S3Util.java:55)
