[
https://issues.apache.org/jira/browse/SPARK-11613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heji Kim updated SPARK-11613:
-----------------------------
Description:
Currently the Kinesis ASL does not allow configuration of connection settings
on AmazonKinesisClient such as the socket timeout.
Currently we get intermittent 2 minute SocketTimeoutExceptions from Kinesis
which holds up the driver processing and pile up batch processing. We'd like to
set it to something closer to the batch polling time of 5 seconds. The requests
are normally subsecond so we'd rather fail fast.
KinesisBackedBlockRDD.KinesisSequenceRangeIterator uses only the credential
based API to create AmazonKinesisClient and so we don't have access to
com.amazonaws.ClientConfiguration which is the only API AWS provides for
setting connection settings.
was:
Currently Kinesis ASL does not allow configuration of connection settings on
AmazonKinesisClient such as socket timeout.
Currently we get intermittent 2 minute SocketTimeoutExceptions from Kinesis
which can hold up the entire driver processing and pile up batch processing.
We'd like to set it to something closer to the batch polling time of 5 seconds.
The requests are normally subsecond so we'd rather fail fast.
KinesisBackedBlockRDD.KinesisSequenceRangeIterator uses only the credential
based API to create AmazonKinesisClient and so we don't have access to
com.amazonaws.ClientConfiguration which is the only API AWS provides for
setting connection settings.
> Kinesis ASL should allow caller to set ClientConfiguration for socket
> timeouts and other connection settings
> ------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-11613
> URL: https://issues.apache.org/jira/browse/SPARK-11613
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.5.1
> Reporter: Heji Kim
>
> Currently the Kinesis ASL does not allow configuration of connection settings
> on AmazonKinesisClient such as the socket timeout.
> Currently we get intermittent 2 minute SocketTimeoutExceptions from Kinesis
> which holds up the driver processing and pile up batch processing. We'd like
> to set it to something closer to the batch polling time of 5 seconds. The
> requests are normally subsecond so we'd rather fail fast.
> KinesisBackedBlockRDD.KinesisSequenceRangeIterator uses only the credential
> based API to create AmazonKinesisClient and so we don't have access to
> com.amazonaws.ClientConfiguration which is the only API AWS provides for
> setting connection settings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
