From the dev thread: What else could be removed in Spark 4? <https://lists.apache.org/thread/shxj7qmrtqbxqf85lrlsv6510892ktnz> > On Aug 17, 2023, at 1:44 AM, Yang Jie <yangji...@apache.org> wrote: > > I would like to know how we should handle the two Kinesis-related modules in > Spark 4.0. They have a very low frequency of code updates, and because the > corresponding tests are not continuously executed in any GitHub Actions > pipeline, so I think they significantly lack quality assurance. On top of > that, I am not certain if the test cases, which require AWS credentials in > these modules, get verified during each Spark version release.
Did we ever reach a decision about removing Kinesis in Spark 4? I was cleaning up some docs related to Kinesis and came across a reference to some Java API docs that I could not find <https://github.com/apache/spark/pull/44802#discussion_r1459337001>. And looking around I came across both this email thread and this thread on JIRA <https://issues.apache.org/jira/browse/SPARK-45720?focusedCommentId=17787227&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17787227> about potentially removing Kinesis. But as far as I can tell we haven’t made a clear decision one way or the other. Nick
