Re: Error with Avro/Kyro after upgrade to Flink 1.4
Thanks, Stephan. You're correct that there was in fact no shading issue in the official flink-dist_2.11-1.4.0.jar. We are using the jar in the flink docker image, but I mis-spoke when I said ObjectMapper appeared there unshaded. It turned out the issue was really a version conflict in our job's uber-jar file between jackson-core and jackson-databind, which I was able to resolve. Regarding the Verify Error, please post back here when you have a patch or a link to a pull request which we can track. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Error with Avro/Kyro after upgrade to Flink 1.4
I would assume that once you build Flink with correct shading and add all the required Jackson dependencies to your application jar, all works fine. On Jan 24, 2018 10:23, "Stephan Ewen" wrote: > Hi! > > I have a patch coming up for Verify Error > > Concerning the Jackson Error - did you build Flink yourself? It looks like > flink-dist was not built properly, the shading is incorrect. > The class "com.fasterxml.jackson.databind.ObjectMapper" should not be in > the jar in an unshaded manner. > > My first guess is you build Flink yourself with Maven version >= 3.3.0 > > Please see here: https://ci.apache.org/projects/flink/flink-docs- > release-1.4/start/building.html#dependency-shading > > Stephan > > > On Tue, Jan 23, 2018 at 5:21 PM, Edward wrote: > >> Thanks for the follow-up Stephan. >> >> I have been running this job from a built jar file which was submitted to >> an >> existing Flink 1.4 cluster, not from within the IDE. Interestingly, I am >> now >> getting the same error when any of the following 3 conditions are true: >> 1. I run the job on a local cluster from within my IDE >> 2. I run the job on a cluster where "classloader.resolve-order: >> parent-first" >> 3. I build the uber jar file without including flink-java, >> flink-streaming-java and flink-clients (I changed those to "provided" as >> you >> suggested, so they aren't in my jar) >> >> If any of those 3 cases are true, I get a new NoClassDefFoundError. This >> error is caused because com.fasterxml.jackson.databind.ObjectMapper is >> present in flink-dist_2.11-1.4.0.jar, but >> com.fasterxml.jackson.databind.SerializationConfig is not (only the >> shaded >> version: >> org.apache.flink.shaded.jackson2.com.fasterxml.jackson. >> databind.SerializationConfig) >> >> java.lang.NoClassDefFoundError: Could not initialize class >> com.fasterxml.jackson.databind.SerializationConfig >> at >> com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:558) >> at >> com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:474) >> at >> com.mycom.datapipeline.common.client.UmsCientFactory.getUser >> MappingServiceClient(UmsCientFactory.java:31) >> at >> com.mycom.datapipeline.flink.udf.UserLookupFunctionBase.open >> (UserLookupFunctionBase.java:78) >> at >> org.apache.flink.api.common.functions.util.FunctionUtils.ope >> nFunction(FunctionUtils.java:36) >> at >> org.apache.flink.streaming.api.operators.AbstractUdfStreamOp >> erator.open(AbstractUdfStreamOperator.java:102) >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllO >> perators(StreamTask.java:393) >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke( >> StreamTask.java:254) >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) >> at java.lang.Thread.run(Thread.java:748) >> >> I understand why this is happening in the case of the parent-first >> classloader, but I can't understand why it's happening when I exclude >> flink-java from my job's uber jar file -- in that 2nd case, I would expect >> the job's child classloader to be used, which contains both of those >> fasterxml classes. >> >> >> >> -- >> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/ >> > >
Re: Error with Avro/Kyro after upgrade to Flink 1.4
Hi! I have a patch coming up for Verify Error Concerning the Jackson Error - did you build Flink yourself? It looks like flink-dist was not built properly, the shading is incorrect. The class "com.fasterxml.jackson.databind.ObjectMapper" should not be in the jar in an unshaded manner. My first guess is you build Flink yourself with Maven version >= 3.3.0 Please see here: https://ci.apache.org/projects/flink/flink-docs-release-1.4/start/building.html#dependency-shading Stephan On Tue, Jan 23, 2018 at 5:21 PM, Edward wrote: > Thanks for the follow-up Stephan. > > I have been running this job from a built jar file which was submitted to > an > existing Flink 1.4 cluster, not from within the IDE. Interestingly, I am > now > getting the same error when any of the following 3 conditions are true: > 1. I run the job on a local cluster from within my IDE > 2. I run the job on a cluster where "classloader.resolve-order: > parent-first" > 3. I build the uber jar file without including flink-java, > flink-streaming-java and flink-clients (I changed those to "provided" as > you > suggested, so they aren't in my jar) > > If any of those 3 cases are true, I get a new NoClassDefFoundError. This > error is caused because com.fasterxml.jackson.databind.ObjectMapper is > present in flink-dist_2.11-1.4.0.jar, but > com.fasterxml.jackson.databind.SerializationConfig is not (only the shaded > version: > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind. > SerializationConfig) > > java.lang.NoClassDefFoundError: Could not initialize class > com.fasterxml.jackson.databind.SerializationConfig > at > com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:558) > at > com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:474) > at > com.mycom.datapipeline.common.client.UmsCientFactory. > getUserMappingServiceClient(UmsCientFactory.java:31) > at > com.mycom.datapipeline.flink.udf.UserLookupFunctionBase. > open(UserLookupFunctionBase.java:78) > at > org.apache.flink.api.common.functions.util.FunctionUtils. > openFunction(FunctionUtils.java:36) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open( > AbstractUdfStreamOperator.java:102) > at > org.apache.flink.streaming.runtime.tasks.StreamTask. > openAllOperators(StreamTask.java:393) > at > org.apache.flink.streaming.runtime.tasks.StreamTask. > invoke(StreamTask.java:254) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) > at java.lang.Thread.run(Thread.java:748) > > I understand why this is happening in the case of the parent-first > classloader, but I can't understand why it's happening when I exclude > flink-java from my job's uber jar file -- in that 2nd case, I would expect > the job's child classloader to be used, which contains both of those > fasterxml classes. > > > > -- > Sent from: http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/ >
Re: Error with Avro/Kyro after upgrade to Flink 1.4
Thanks for the follow-up Stephan. I have been running this job from a built jar file which was submitted to an existing Flink 1.4 cluster, not from within the IDE. Interestingly, I am now getting the same error when any of the following 3 conditions are true: 1. I run the job on a local cluster from within my IDE 2. I run the job on a cluster where "classloader.resolve-order: parent-first" 3. I build the uber jar file without including flink-java, flink-streaming-java and flink-clients (I changed those to "provided" as you suggested, so they aren't in my jar) If any of those 3 cases are true, I get a new NoClassDefFoundError. This error is caused because com.fasterxml.jackson.databind.ObjectMapper is present in flink-dist_2.11-1.4.0.jar, but com.fasterxml.jackson.databind.SerializationConfig is not (only the shaded version: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.SerializationConfig) java.lang.NoClassDefFoundError: Could not initialize class com.fasterxml.jackson.databind.SerializationConfig at com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:558) at com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:474) at com.mycom.datapipeline.common.client.UmsCientFactory.getUserMappingServiceClient(UmsCientFactory.java:31) at com.mycom.datapipeline.flink.udf.UserLookupFunctionBase.open(UserLookupFunctionBase.java:78) at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:393) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:254) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718) at java.lang.Thread.run(Thread.java:748) I understand why this is happening in the case of the parent-first classloader, but I can't understand why it's happening when I exclude flink-java from my job's uber jar file -- in that 2nd case, I would expect the job's child classloader to be used, which contains both of those fasterxml classes. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Error with Avro/Kyro after upgrade to Flink 1.4
Hi! Thanks for posting this. Ugly error. This does indeed look like either a dependency conflict, or a mix of incompatible class files. Some suggestions to diagnose this further: - Is this the setup of your POM to build the JAR file for your application, or for running it from within the IDE? - If this is the POM for building you application JAR, please set the "flink-java", "flink-streaming-java", and "flink-clients" dependencies to the scope "provided". - Can you try to set the following entry in your config "classloader.resolve-order: parent-first" Please let us know how this affects the program. Best, Stephan On Mon, Jan 22, 2018 at 2:45 PM, Edward wrote: > Also, I'm not sure if this would cause the uninitialized error, but I did > notice that in the maven dependency tree there are 2 different versions of > kyro listed as Flink dependencies: > flink-java 1.4 requires kyro 2.24, but flink-streaming-java_2.11 requires > kyro 2.21: > > [INFO] +- org.apache.flink:flink-java:jar:1.4.0:compile > [INFO] | +- org.apache.flink:flink-core:jar:1.4.0:compile > [INFO] | | +- com.esotericsoftware.kryo:kryo:jar:2.24.0:compile > > [INFO] +- org.apache.flink:flink-streaming-java_2.11:jar:1.4.0:compile > [INFO] | +- (org.apache.flink:flink-core:jar:1.4.0:compile - omitted for > duplicate) > [INFO] | +- org.apache.flink:flink-runtime_2.11:jar:1.4.0:compile > [INFO] | | +- com.twitter:chill_2.11:jar:0.7.4:compile > [INFO] | | | +- com.twitter:chill-java:jar:0.7.4:compile > [INFO] | | | | \- (com.esotericsoftware.kryo:kryo:jar:2.21:compile - > omitted for conflict with 2.24.0) > [INFO] | | | \- (com.esotericsoftware.kryo:kryo:jar:2.21:compile - > omitted for conflict with 2.24.0) > > > > -- > Sent from: http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/ >
Re: Error with Avro/Kyro after upgrade to Flink 1.4
Also, I'm not sure if this would cause the uninitialized error, but I did notice that in the maven dependency tree there are 2 different versions of kyro listed as Flink dependencies: flink-java 1.4 requires kyro 2.24, but flink-streaming-java_2.11 requires kyro 2.21: [INFO] +- org.apache.flink:flink-java:jar:1.4.0:compile [INFO] | +- org.apache.flink:flink-core:jar:1.4.0:compile [INFO] | | +- com.esotericsoftware.kryo:kryo:jar:2.24.0:compile [INFO] +- org.apache.flink:flink-streaming-java_2.11:jar:1.4.0:compile [INFO] | +- (org.apache.flink:flink-core:jar:1.4.0:compile - omitted for duplicate) [INFO] | +- org.apache.flink:flink-runtime_2.11:jar:1.4.0:compile [INFO] | | +- com.twitter:chill_2.11:jar:0.7.4:compile [INFO] | | | +- com.twitter:chill-java:jar:0.7.4:compile [INFO] | | | | \- (com.esotericsoftware.kryo:kryo:jar:2.21:compile - omitted for conflict with 2.24.0) [INFO] | | | \- (com.esotericsoftware.kryo:kryo:jar:2.21:compile - omitted for conflict with 2.24.0) -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
