xiangfu0 commented on PR #18687: URL: https://github.com/apache/pinot/pull/18687#issuecomment-4635605066
Yes, this is based on the real issue we saw, but the missing partition is not missing from Kafka. Kafka still has the partitions. The sparse state is in Pinot LLC metadata: for some existing Kafka partition ids, Pinot no longer has consuming/online segments or latest LLC metadata in ZK. In that state, RVM should be able to recreate consuming segments because the existing validation flow sets up partition groups that exist in stream metadata but are missing in Pinot metadata. The bug is that Kafka's no-subset metadata path used the SPI default, which derives new ids from `currentStatuses.size()`. That works for the normal Kafka partition expansion case, e.g. `8 -> 16`, but fails when Pinot's current status list is sparse for existing Kafka ids. Example: Kafka has `0..7`, Pinot metadata has `0,1,3,4,5,6,7`. The old computation returns `0,1,3,4,5,6,7,7`, so partition `2` never reaches RVM's `setupNewPartitionGroup` path. This PR does not assume Kafka partitions disappear intermittently. It asks Kafka for the actual live partition ids and gives RVM correct metadata for partitions that still exist in Kafka but are missing from Pinot metadata. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
