Hi, We’ve seen this issue before [1]. The usual reason is that the httpcomponent dependencies weren’t properly shaded in the flink-dist jar. Having them properly shaded should solve the issue.
cc Bowen: Are you building Flink yourself? If yes, what Maven version are you using? If you’re using 3.3.x+, after the first build under flink/, make sure to go to flink-dist/ and build a second time for the dependencies to be properly shaded. Alternatively, Maven 3.0.x+ is the recommended version, as 3.3.x has dependency shading issues. If you’re not building Flink yourself, the cause could be that the Flink 1.3.0 flink-dist jar wasn’t shaded properly, may need to double check. Best, Gordon [1] https://issues.apache.org/jira/browse/FLINK-5013 On 20 June 2017 at 12:14:27 PM, Ted Yu (yuzhih...@gmail.com) wrote: I logged FLINK-6951, referencing this thread. We can continue discussion there. Thanks On Mon, Jun 19, 2017 at 9:06 PM, Bowen Li <bowen...@offerupnow.com> wrote: > Thanks, Ted! woo, this is unexpected. https://ci.apache. > org/projects/flink/flink-docs-release-1.3/setup/aws.html is out of date. > > I bet anyone using Kinesis with Flink will run into this issue. I can try > to build Flink myself and resolve this problem. But talking about a > feasible permanent solution for all flink-connector-kinesis users. Shall we > downgrade aws-java-sdk-kinesis version in flink-connector-kinesis, or shall > we upgrade httpcomponents version in Flink? > > Bowen > > > On Mon, Jun 19, 2017 at 7:02 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > Here is the dependency in the flink-connector-kinesis module: > > > > [INFO] +- com.amazonaws:aws-java-sdk-kinesis:jar:1.10.71:compile > > [INFO] | \- com.amazonaws:aws-java-sdk-core:jar:1.10.71:compile > > [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.3.6:compile > > [INFO] | +- org.apache.httpcomponents:httpcore:jar:4.3.3:compile > > > > Checking dependency tree of flink, the highest version is 4.2.x > > > > You can try building flink with dependency on 4.3.y of httpclient / > > httpcore > > > > FYI > > > > > > > > On Mon, Jun 19, 2017 at 4:52 PM, Bowen Li <bowen...@offerupnow.com> > wrote: > > > > > Hi guys, > > > I'm trying to enable Flink's checkpoint on our Flink app. I got the > > > following Apache http jar compatibility error, and cannot figure out > how > > to > > > resolve it. > > > > > > Here's the stacktrace: > > > > > > ``` > > > > > > 2017-06-19 15:07:39,828 INFO org.apache.flink.runtime. > taskmanager.Task > > > - Source: Custom Source -> (Timestamps/Watermarks, > Filter > > > -> Map, Filter -> Map, Filter -> Map) (1/1) > > > (37ab9429deda28e31fa0ed0ed1568654) switched from RUNNING to FAILED. > > > java.lang.NoSuchFieldError: INSTANCE > > > at > > > org.apache.flink.kinesis.shaded.com.amazonaws.http.conn. > > > SdkConnectionKeepAliveStrategy.getKeepAliveDuration( > > > SdkConnectionKeepAliveStrategy.java:48) > > > at > > > org.apache.http.impl.client.DefaultRequestDirector.execute( > > > DefaultRequestDirector.java:535) > > > at > > > org.apache.http.impl.client.AbstractHttpClient.execute( > > > AbstractHttpClient.java:906) > > > at > > > org.apache.http.impl.client.AbstractHttpClient.execute( > > > AbstractHttpClient.java:805) > > > at > > > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient. > > > executeOneRequest(AmazonHttpClient.java:837) > > > at > > > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient. > > > executeHelper(AmazonHttpClient.java:607) > > > at > > > org.apache.flink.kinesis.shaded.com.amazonaws.http. > > > AmazonHttpClient.doExecute(AmazonHttpClient.java:376) > > > at > > > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient. > > > executeWithTimer(AmazonHttpClient.java:338) > > > at > > > org.apache.flink.kinesis.shaded.com.amazonaws.http. > > > AmazonHttpClient.execute(AmazonHttpClient.java:287) > > > at > > > org.apache.flink.kinesis.shaded.com.amazonaws.services. > > > kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:1940) > > > at > > > org.apache.flink.kinesis.shaded.com.amazonaws.services. > > > kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1910) > > > at > > > org.apache.flink.kinesis.shaded.com.amazonaws.services. > > > kinesis.AmazonKinesisClient.describeStream( > AmazonKinesisClient.java:656) > > > at > > > org.apache.flink.streaming.connectors.kinesis.proxy. > > > KinesisProxy.describeStream(KinesisProxy.java:361) > > > at > > > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy. > > > getShardsOfStream(KinesisProxy.java:323) > > > at > > > org.apache.flink.streaming.connectors.kinesis.proxy. > > > KinesisProxy.getShardList(KinesisProxy.java:231) > > > at > > > org.apache.flink.streaming.connectors.kinesis.internals. > > > KinesisDataFetcher.discoverNewShardsToSubscribe( > > > KinesisDataFetcher.java:430) > > > at > > > org.apache.flink.streaming.connectors.kinesis. > FlinkKinesisConsumer.run( > > > FlinkKinesisConsumer.java:202) > > > at > > > org.apache.flink.streaming.api.operators.StreamSource. > > > run(StreamSource.java:87) > > > at > > > org.apache.flink.streaming.api.operators.StreamSource. > > > run(StreamSource.java:55) > > > at > > > org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run( > > > SourceStreamTask.java:95) > > > at > > > org.apache.flink.streaming.runtime.tasks.StreamTask. > > > invoke(StreamTask.java:262) > > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) > > > at java.lang.Thread.run(Thread.java:745) > > > > > > ``` > > > > > > Here's my Flink environment setup: > > > > > > - I'm using flink-connector-kinesis_2.11-1.3.0.jar that I built > from > > > 1.3.0 source code. > > > - I followed > > > https://ci.apache.org/projects/flink/flink-docs- > > > release-1.3/setup/aws.html#flink-for-hadoop-27 > > > and added all necessary dependency jars > > > - My application doesn't use Apache http/core. > > > > > > > > > Has anyone experienced the similar incompatibility issue? > > > > > > Thanks! > > > Bowen > > > > > >
