[
https://issues.apache.org/jira/browse/SPARK-43036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh updated SPARK-43036:
---------------------------
Description:
Hi Spark Team,
We need to read/write the kinesis streams using spark streaming.
We checked the official documentation -
[https://spark.apache.org/docs/latest/streaming-kinesis-integration.html]
It does not mention about kinesis connector. Alternative is -
[https://github.com/qubole/kinesis-sql] which is not active now. This is now
handed over here - [https://github.com/roncemer/spark-sql-kinesis]
We have few below questions , It would be great if you can answer
# Does Spark provides officially any kinesis connector which have
readstream/writestream and endorse any connector for production use cases ?
# [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html]
This documentation does not mention about how to write to kinesis. This method
have default dynamodb as checkpoint, can we override it ?
# We have rocksdb as state store but when we ran application using official
[https://spark.apache.org/docs/latest/streaming-kinesis-integration.html]
rocksdb configurations were not effective. Can you please confirm if rocksdb is
not applicable in these cases?
# rocksdb however works with qubole connector , do you have any plan to
release kinesis connector?
# Please help/recommend us for any good stable kinesis connector or some
pointers around it
> Spark structuring streaming with kinesis and rocksdb integration
> ----------------------------------------------------------------
>
> Key: SPARK-43036
> URL: https://issues.apache.org/jira/browse/SPARK-43036
> Project: Spark
> Issue Type: Question
> Components: Structured Streaming
> Affects Versions: 3.3.1
> Reporter: Rajesh
> Priority: Blocker
>
> Hi Spark Team,
> We need to read/write the kinesis streams using spark streaming.
> We checked the official documentation -
> [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html]
> It does not mention about kinesis connector. Alternative is -
> [https://github.com/qubole/kinesis-sql] which is not active now. This is now
> handed over here - [https://github.com/roncemer/spark-sql-kinesis]
> We have few below questions , It would be great if you can answer
> # Does Spark provides officially any kinesis connector which have
> readstream/writestream and endorse any connector for production use cases ?
> # [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html]
> This documentation does not mention about how to write to kinesis. This
> method have default dynamodb as checkpoint, can we override it ?
> # We have rocksdb as state store but when we ran application using official
> [https://spark.apache.org/docs/latest/streaming-kinesis-integration.html]
> rocksdb configurations were not effective. Can you please confirm if rocksdb
> is not applicable in these cases?
> # rocksdb however works with qubole connector , do you have any plan to
> release kinesis connector?
> # Please help/recommend us for any good stable kinesis connector or some
> pointers around it
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
