Is "aws kinesis get-shard-iterator --shard-iterator-type LATEST" not enough for your usecase?
On Mon, Nov 14, 2016 at 10:23 PM, Shushant Arora <shushantaror...@gmail.com> wrote: > Thanks! > Is there a way to get the latest sequence number of all shards of a > kinesis stream? > > > > On Mon, Nov 14, 2016 at 5:43 PM, Takeshi Yamamuro <linguin....@gmail.com> > wrote: > >> Hi, >> >> The time interval can be controlled by `IdleTimeBetweenReadsInMillis` >> in KinesisClientLibConfiguration though, >> it is not configurable in the current implementation. >> >> The detail can be found in; >> https://github.com/apache/spark/blob/master/external/kinesis >> -asl/src/main/scala/org/apache/spark/streaming/kinesis >> /KinesisReceiver.scala#L152 >> >> // maropu >> >> >> On Sun, Nov 13, 2016 at 12:08 AM, Shushant Arora < >> shushantaror...@gmail.com> wrote: >> >>> *Hi * >>> >>> *is **spark.streaming.blockInterval* for kinesis input stream is >>> hardcoded to 1 sec or is it configurable ? Time interval at which receiver >>> fetched data from kinesis . >>> >>> Means stream batch interval cannot be less than >>> *spark.streaming.blockInterval >>> and this should be configrable , Also is there any minimum value for >>> streaming batch interval ?* >>> >>> *Thanks* >>> >>> >> >> >> -- >> --- >> Takeshi Yamamuro >> > > -- --- Takeshi Yamamuro
