[
https://issues.apache.org/jira/browse/SPARK-9781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Nekhaev updated SPARK-9781:
-
Description:
Currently the KinesisClientLibConfiguration for KCL Workers is created within
the KinesisReceiver and user is allowed to change only basic settings such as
endpoint URL, stream name, credentials, etc.
However, there is no way to tune some advanced settings, e.g. MaxRecords,
IdleTimeBetweenReads, FailoverTime, etc.
We can add these settings to the Spark configuration and parametrize
KinesisClientLibConfiguration with them in KinesisReceiver.
was:
Currently the KinesisClientLibConfiguration for KCL Workers is created withing
the KinesisReceiver and user is allowed to change only basic settings such as
endpoint URL, stream name, credentials, etc.
However, there is no way to tune some advanced settings, e.g. MaxRecords,
IdleTimeBetweenReads, FailoverTime, etc.
We can add this settings to the Spark configuration and parametrize
KinesisClientLibConfiguration with them in KinesisReceiver.
KCL Workers should be configurable from Spark configuration
---
Key: SPARK-9781
URL: https://issues.apache.org/jira/browse/SPARK-9781
Project: Spark
Issue Type: Improvement
Components: Streaming
Affects Versions: 1.4.1
Reporter: Anton Nekhaev
Labels: kinesis
Currently the KinesisClientLibConfiguration for KCL Workers is created within
the KinesisReceiver and user is allowed to change only basic settings such as
endpoint URL, stream name, credentials, etc.
However, there is no way to tune some advanced settings, e.g. MaxRecords,
IdleTimeBetweenReads, FailoverTime, etc.
We can add these settings to the Spark configuration and parametrize
KinesisClientLibConfiguration with them in KinesisReceiver.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
