Danny Cranmer created FLINK-26890: ------------------------------------- Summary: [Kinesis][Consumer] DynamoDB consumer error consuming partitions close to retention Key: FLINK-26890 URL: https://issues.apache.org/jira/browse/FLINK-26890 Project: Flink Issue Type: Bug Components: Connectors / Kinesis Affects Versions: 1.15.0 Reporter: Danny Cranmer Fix For: 1.15.0, 1.16.0, 1.13.7, 1.14.5
*Background* The Amazon Kinesis Data Streams consumer supports consuming from Amazon DynamoDB via the [DynamoDB Streams Kinesis Adapter|https://github.com/awslabs/dynamodb-streams-kinesis-adapter]. *Problem* We have seen instances of consumer throwing {{ResouceNotFoundException}} when attempting to invoke {{GetShardIterator}}. {code} com.amazonaws.services.kinesis.model.ResourceNotFoundException: Requested resource not found: Shard does not exist {code} According to the DynamoDB team, the {{DescribeStream}} call may return shard IDs that are no longer valid, and this exception needs to be handled by the client. *Solution* Modify the DynamoDB consumer to treat {{ResourceNotFoundException}} as a shard closed signal. -- This message was sent by Atlassian Jira (v8.20.1#820001)