KKcorps opened a new pull request, #20040: URL: https://github.com/apache/pinot/pull/20040
When a stream partition ends, like a Kinesis shard that is fully read after a split or merge, Pinot never creates another consuming segment for it, and alerts on `llcPartitionConsuming` keep firing for a partition that is done. Today the controller only sees that the partition is missing from the stream metadata, so it cannot tell an ended partition from a missing one. This lets the stream say it. `StreamMetadataProvider` gets a `fetchEndOfLifePartitionGroupIds` method that defaults to none, and Kinesis reports shards that are closed and consumed to their end. When the stream confirms it, at commit or in `RealtimeSegmentValidationManager`, the controller sets `segment.realtime.endOfPartitionGroup` in the last segment's ZK metadata. Nothing is kept in memory, so a restart or leadership move needs nothing rebuilt. Every replica reads that ZK metadata on CONSUMING to ONLINE, so when the flag is set it removes `llcPartitionConsuming` instead of leaving it at 0, whether it committed, caught up or downloaded. The controller also emits a `partitionEndOfLife` gauge of 1 for these partitions, for alerts that fill missing series with 0. Streams that never report an ended partition, like Kafka, see no change. Tests: new cases in `KinesisStreamMetadataProviderTest`, `PinotLLCRealtimeSegmentManagerTest`, `RealtimeSegmentDataManagerTest` and `SegmentZKMetadataTest`, and each fails without its fix. The Prometheus export test, spotless, checkstyle and license pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
