[
https://issues.apache.org/jira/browse/FLINK-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136166#comment-16136166
]
ASF GitHub Bot commented on FLINK-7366:
---------------------------------------
GitHub user bowenli86 reopened a pull request:
https://github.com/apache/flink/pull/4522
[FLINK-7366][kinesis connector] Upgrade kinesis producer library in
flink-connector-kinesis
## What is the purpose of the change
We need to upgrade KPL and KCL to pick up the enhanced performance and
stability for Flink to work better with Kinesis. Upgrading KPL is specially
necessary, because the KPL version Flink uses is old, and doesn't have good
retry and error handling logic.
**Upgrade KPL:**
flink-connector-kinesis currently uses kinesis-producer-library 0.10.2,
which is released in Nov 2015 by AWS. It's old. It's the fourth release, and
thus problematic. It doesn't even have good retry logic, therefore Flink fails
really frequently (about every 10 mins as we observed) when Flink writes too
fast to Kinesis and receives RateLimitExceededException,
Quotes from https://github.com/awslabs/amazon-kinesis-producer/issues/56,
"With the newer version of the KPL it uses the AWS C++ SDK which should offer
additional retries." on Oct 2016. 0.12.5, the version we are upgrading to, is
released in May 2017 and should have the enhanced retry logic.
**Upgrade KCL:**
Upgrade KCL from 1.6.2 to 1.8.1
**Upgrade AWS SDK:**
from 1.10.71 to 1.11.171
## Verifying this change
This change is already covered by existing tests
## Does this pull request potentially affect one of the following parts:
There should be any impact outside flink-connector-kinesis, because 1) KPL
and KCL is only used in flink-connector-kinesis, and 2) AWS SDK in
flink-connector-kinesis is shaded
## Documentation
- Does this pull request introduce a new feature? (no)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bowenli86/flink FLINK-7366
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4522.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4522
----
commit 7486878631e7238283eabfdd575387d32b210f91
Author: Bowen Li <[email protected]>
Date: 2017-08-10T22:09:56Z
FLINK-7366 Upgrade kinesis-producer-library in flink-connector-kinesis from
0.10.2 to 0.12.5
commit ba9fb5f1e189e31f8085bd384edbb682a69aae7f
Author: Bowen Li <[email protected]>
Date: 2017-08-10T23:46:59Z
upgrade KCL and AWS SDK
commit 88a16efcaf4c7e2addcf978a98f8ffbea4281639
Author: Bowen Li <[email protected]>
Date: 2017-08-13T05:49:26Z
revert changes to KCL
----
> Upgrade kinesis producer library in flink-connector-kinesis
> -----------------------------------------------------------
>
> Key: FLINK-7366
> URL: https://issues.apache.org/jira/browse/FLINK-7366
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
> Affects Versions: 1.3.2
> Reporter: Bowen Li
> Assignee: Bowen Li
> Fix For: 1.4.0, 1.3.3
>
>
> We need to upgrade KPL and KCL to pick up the enhanced performance and
> stability for Flink to work better with Kinesis. Upgrading KPL is specially
> necessary, because the KPL version Flink uses is old, and doesn't have good
> retry and error handling logic.
> *KPL:*
> flink-connector-kinesis currently uses kinesis-producer-library 0.10.2, which
> is released in Nov 2015 by AWS. It's old. It's the fourth release, and thus
> problematic. It doesn't even have good retry logic, therefore Flink fails
> really frequently (about every 10 mins as we observed) when Flink writes too
> fast to Kinesis and receives RateLimitExceededException,
> Quotes from https://github.com/awslabs/amazon-kinesis-producer/issues/56,
> "*With the newer version of the KPL it uses the AWS C++ SDK which should
> offer additional retries.*" on Oct 2016. 0.12.5, the version we are upgrading
> to, is released in May 2017 and should have the enhanced retry logic.
> *KCL:*
> Upgrade KCL from 1.6.2 to 1.8.1
> *AWS SDK*
> from 1.10.71 to 1.11.171
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
