Hi Spark Team, We need to read/write the kinesis streams using spark streaming.
We checked the official documentation - https://spark.apache.org/docs/latest/streaming-kinesis-integration.html It does not mention kinesis connector. Alternative is - https://github.com/qubole/kinesis-sql which is not active now. This is now handed over here - https://github.com/roncemer/spark-sql-kinesis Also according to SPARK-18165 <https://issues.apache.org/jira/browse/SPARK-18165> , Spark officially do not have any kinesis connector We have few below questions , It would be great if you can answer 1. Does Spark provides officially any kinesis connector which have readstream/writestream and endorse any connector for production use cases ? 2. https://spark.apache.org/docs/latest/streaming-kinesis-integration.html This documentation does not mention how to write to kinesis. This method has default dynamodb as checkpoint, can we override it ? 3. We have rocksdb as a state store but when we ran an application using official https://spark.apache.org/docs/latest/streaming-kinesis-integration.html rocksdb configurations were not effective. Can you please confirm if rocksdb is not applicable in these cases? 4. rocksdb however works with qubole connector , do you have any plan to release kinesis connector? 5. Please help/recommend us for any good stable kinesis connector or some pointers around it
