Abhi Gupta created FLINK-36239:
----------------------------------
Summary: DDB Streams Connector reprocessing due to DescribeStream
inconsistencies for trimmed shards
Key: FLINK-36239
URL: https://issues.apache.org/jira/browse/FLINK-36239
Project: Flink
Issue Type: Bug
Components: Connectors / DynamoDB
Reporter: Abhi Gupta
* We did describestream and processed shard-a >24 hours ago
* Now the shard has been removed since its more than 24 hours.
* We just got a describestream call for this shard.
* Describestream didn’t give this shard
* We got another describestream call. Describestream somehow sent that shard
back due to inconsistencies, we sent this out to SplitTracker. This shard was
not in finished shards, since we just deleted it some time back since it was
more than 25 hours old, so this shard would be processed duplicately.
Fix: We'll make the shard retention to be 48 hours to avoid these edge cases
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
