[
https://issues.apache.org/jira/browse/SPARK-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15067125#comment-15067125
]
Martin Schade commented on SPARK-12453:
---------------------------------------
Make sense, thank you. Ideally it should be 1.9.37 instead of 1.9.40 though.
Both KCL 1.4.0 and KPL 0.10.1 reference 1.9.37.
https://github.com/awslabs/amazon-kinesis-producer/blob/v0.10.1/java/amazon-kinesis-producer/pom.xml
https://github.com/awslabs/amazon-kinesis-client/blob/v1.4.0/pom.xml
Both reference 1.9.37
In the latest version of KPL (v0.10.2) 1.10.34 is references and in latest KCL
1.6.1 it is version 1.10.20, not easy so sync then. So it would need to do some
testing which combination works actually.
> Spark Streaming Kinesis Example broken due to wrong AWS Java SDK version
> ------------------------------------------------------------------------
>
> Key: SPARK-12453
> URL: https://issues.apache.org/jira/browse/SPARK-12453
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.5.2
> Reporter: Martin Schade
> Priority: Critical
> Labels: easyfix
>
> The Spark Streaming Kinesis Example (kinesis-asl) is broken due to wrong AWS
> Java SDK version (1.9.16) referenced with the AWS KCL version (1.3.0).
> AWS KCL 1.3.0 references AWS Java SDK version 1.9.37.
> Using 1.9.16 in combination with 1.3.0 does fail to get data out of the
> stream.
> I tested Spark Streaming with 1.9.37 and it works fine.
> Testing a simple KCL client outside of Spark with 1.3.0 and 1.9.16 also
> fails, so it is due to the specific versions used in 1.5.2 and not a Spark
> related implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
