I gave up on running a Spark pipeline locally, tried AWS EMR/Spark instead.
Now this:
17/10/02 23:53:17 ERROR ApplicationMaster: User class threw exception:
java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at com.kochava.beam.s3.S3FileSystem.matchNewResource(S3FileSystem.java:133)
at com.kochava.beam.s3.S3FileSystem.matchNewResource(S3FileSystem.java:38)
at org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:518)
It doesn't make sense. Guava is definitely shaded; `maven package` tells me
so and the jar contains com/google/common/base/Preconditions.class. Here's
my shade config, in case someone sees something.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>shaded</shadedClassifierName>
<createDependencyReducedPom>false</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<!-- fixes SparkRunner class not found -->
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
Jacob
On Mon, Oct 2, 2017 at 11:17 AM, Jacob Marble <[email protected]> wrote:
> There is a lot of chatter about AWS and Jackson on their forums, etc. I
> have been using the AWS SDK and Jackson 2.8.9 for a couple of weeks without
> problems. Adding Spark to the mix is what changes this.
>
> Jacob
>
> On Mon, Oct 2, 2017 at 11:14 AM, Romain Manni-Bucau <[email protected]
> > wrote:
>
>> Hi Jacob,
>>
>> isn't aws API only supporting jackson 2.6 and not 2.8?
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github <
>> https://github.com/rmannibucau> |
>> LinkedIn <https://www.linkedin.com/in/rmannibucau>
>>
>> 2017-10-02 20:13 GMT+02:00 Jacob Marble <[email protected]>:
>>
>> > Yes, I'm using spark-submit, and I'm giving it a shaded jar.
>> >
>> > What do you mean "aligning the dependencies"?
>> >
>> > Jacob
>> >
>> > On Mon, Oct 2, 2017 at 11:06 AM, Jean-Baptiste Onofré <[email protected]>
>> > wrote:
>> >
>> > > Hi
>> > >
>> > > Do you start your pipeline with spark-submit ? If so you can provide
>> the
>> > > packages. You can also create a shaded jar.
>> > >
>> > > I have a similar issue in the spark 2 runner that I worked around by
>> > > aligning the dependencies.
>> > >
>> > > Regards
>> > > JB
>> > >
>> > > On Oct 2, 2017, 20:04, at 20:04, Jacob Marble <[email protected]>
>> > wrote:
>> > > >My Beam pipeline runs fine with DirectRunner and DataflowRunner, but
>> > > >fails
>> > > >with SparkRunner. That stack trace is after this message.
>> > > >
>> > > >The exception indicates that
>> > > >com.fasterxml.jackson.databind.ObjectMapper.enable doesn't exist.
>> > > >ObjectMapper.enable() didn't exist until Jackson 2.5. `mvn
>> > > >dependency:tree
>> > > >-Dverbose` shows that spark-core_2.10 (1.6.3) and beam-runners-spark
>> > > >(2.1.0) both request versions of Jackson before 2.5.
>> > > >
>> > > >Since I'm using a local, standalone Spark cluster for development, I
>> > > >have
>> > > >to include spark-core_2.10 version 1.6.3 in dependencies.
>> > > >
>> > > >I have added explicit dependencies to my pom.xml, so that I can be
>> > > >certain
>> > > >that the more recent version of Jackson is included in my shaded jar.
>> > > >`mvn
>> > > >clean package` confirms this:
>> > > >
>> > > >[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.8.9
>> in
>> > > >the
>> > > >shaded jar.
>> > > >[INFO] Including
>> > > >com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9
>> > > >in the shaded jar.
>> > > >[INFO] Including com.fasterxml.jackson.core:jac
>> kson-databind:jar:2.8.9
>> > > >in
>> > > >the shaded jar.
>> > > >[INFO] Including
>> > > >com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.8.9 in
>> the
>> > > >shaded jar.
>> > > >[INFO] Including
>> > > >com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.8.9 in
>> the
>> > > >shaded jar.
>> > > >[INFO] Including
>> > > >com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.8.9
>> in
>> > > >the
>> > > >shaded jar.
>> > > >
>> > > >Beyond jar creation, is there anything I can do to ensure that my
>> > > >chosen
>> > > >version of a dependency is used when Spark runs my pipeline? I can't
>> be
>> > > >the
>> > > >first to encounter this problem.
>> > > >
>> > > >Thanks!
>> > > >
>> > > >Jacob
>> > > >
>> > > >--------
>> > > >
>> > > >Exception in thread "main" java.lang.RuntimeException:
>> > > >java.lang.NoSuchMethodError:
>> > > >com.fasterxml.jackson.databind.ObjectMapper.enable([
>> > > Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/
>> > > fasterxml/jackson/databind/ObjectMapper;
>> > > >at
>> > > >org.apache.beam.runners.spark.SparkPipelineResult.runtimeEx
>> ceptionFrom(
>> > > SparkPipelineResult.java:55)
>> > > >at
>> > > >org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(
>> > > SparkPipelineResult.java:72)
>> > > >at
>> > > >org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(
>> > > SparkPipelineResult.java:99)
>> > > >at
>> > > >org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(
>> > > SparkPipelineResult.java:87)
>> > > >at com.kochava.beam.jobs.ExampleS3.main(ExampleS3.java:46)
>> > > >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > > >at
>> > > >sun.reflect.NativeMethodAccessorImpl.invoke(
>> > > NativeMethodAccessorImpl.java:62)
>> > > >at
>> > > >sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> > > DelegatingMethodAccessorImpl.java:43)
>> > > >at java.lang.reflect.Method.invoke(Method.java:498)
>> > > >at
>> > > >org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
>> > > deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>> > > >at
>> > > >org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmi
>> t.scala:181)
>> > > >at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.
>> scala:206)
>> > > >at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
>> > > >at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> > > >Caused by: java.lang.NoSuchMethodError:
>> > > >com.fasterxml.jackson.databind.ObjectMapper.enable([
>> > > Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/
>> > > fasterxml/jackson/databind/ObjectMapper;
>> > > >at
>> > > >com.amazonaws.partitions.PartitionsLoader.<clinit>(
>> > > PartitionsLoader.java:54)
>> > > >at
>> > > >com.amazonaws.regions.RegionMetadataFactory.create(
>> > > RegionMetadataFactory.java:30)
>> > > >at com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:64)
>> > > >at
>> > > >com.amazonaws.regions.RegionUtils.getRegionMetadata(
>> > RegionUtils.java:52)
>> > > >at com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:105)
>> > > >at
>> > > >com.amazonaws.client.builder.AwsClientBuilder.withRegion(
>> > > AwsClientBuilder.java:239)
>> > > >at com.kochava.beam.s3.S3Util.<init>(S3Util.java:103)
>> > > >at com.kochava.beam.s3.S3Util.<init>(S3Util.java:53)
>> > > >at com.kochava.beam.s3.S3Util$S3UtilFactory.create(S3Util.java:81)
>> > > >at com.kochava.beam.s3.S3Util$S3UtilFactory.create(S3Util.java:55)
>> > >
>> >
>>
>
>
