[
https://issues.apache.org/jira/browse/FLINK-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123159#comment-16123159
]
ASF GitHub Bot commented on FLINK-7366:
---------------------------------------
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4522
Hmm, I think Java serialization was removed from the Kinesis connector
starting from 1.3.
But thinking of it now, I recall that there was some migration path from
that. Let me double check ..
> Upgrade kinesis producer library, kinesis client library, and AWS SDK in
> flink-connector-kinesis
> ------------------------------------------------------------------------------------------------
>
> Key: FLINK-7366
> URL: https://issues.apache.org/jira/browse/FLINK-7366
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
> Affects Versions: 1.3.2
> Reporter: Bowen Li
> Assignee: Bowen Li
> Fix For: 1.4.0, 1.3.3
>
>
> We need to upgrade KPL and KCL to pick up the enhanced performance and
> stability for Flink to work better with Kinesis. Upgrading KPL is specially
> necessary, because the KPL version Flink uses is old, and doesn't have good
> retry and error handling logic.
> *KPL:*
> flink-connector-kinesis currently uses kinesis-producer-library 0.10.2, which
> is released in Nov 2015 by AWS. It's old. It's the fourth release, and thus
> problematic. It doesn't even have good retry logic, therefore Flink fails
> really frequently (about every 10 mins as we observed) when Flink writes too
> fast to Kinesis and receives RateLimitExceededException,
> Quotes from https://github.com/awslabs/amazon-kinesis-producer/issues/56,
> "*With the newer version of the KPL it uses the AWS C++ SDK which should
> offer additional retries.*" on Oct 2016. 0.12.5, the version we are upgrading
> to, is released in May 2017 and should have the enhanced retry logic.
> *KCL:*
> Upgrade KCL from 1.6.2 to 1.8.1
> *AWS SDK*
> from 1.10.71 to 1.11.171
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
