Hi!
Thanks for reporting this.
This looks like an overlooked corner case that the Kinesis connector doesn’t
handle properly.
First, let me clarify the case and how it can be reproduced. Please let me know
if the following is correct:
1. You started a Kinesis connector source, with TRIM_HORIZON as the startup
position.
2. No records were written to the Kinesis stream at all.
3. After a period of time, you received the “Encountered an unexpected expired
iterator” warning in the logs, and the job failed with the misleading
AmazonKinesisException?
Cheers,
Gordon
On 13 December 2018 at 6:53:11 AM, Vijay Balakrishnan (bvija...@gmail.com)
wrote:
Hi,
Using FlinkKinesisConsumer in a long running Flink Streaming app consuming from
a Kinesis Stream.
Encountered the following Expired Iterator exception in getRecords():
org.apache.flink.streaming.connectors.kinesis.internals.ShardConsumer [] -
Encountered an unexpected expired iterator
The error on the console ends up being a misleading one: "Caused by:
org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.model.AmazonKinesisException:
1 validation error detected: Value 'EARLIEST_SEQUENCE_NUM' at
'startingSequenceNumber' failed to satisfy constraint: Member must satisfy
regular expression pattern: 0|([1-9]\d{0,128}) (Service: AmazonKinesis; Status
Code: 400; Error Code: ValidationException; Request ID: ..)
"
How do I increase the ClientConfiguration.clientExecutiontimeout to avoid this
issue or is this the right way to handle this issue ? I don't want the
FlinkKinesisConsumer streaming app to fail just because there might be no
records in the Kinesis Stream. I am using TRIM_HORIZON to read from the start
of the Kinesis Stream.
TIA,
