deepthi912 commented on PR #20106: URL: https://github.com/apache/pinot/pull/20106#issuecomment-6671651959
**1. Rebalance test** `PartialUpsertTableRebalanceIntegrationTest` rebalances real servers, but it uses OVERWRITE and only checks counts, so it would pass without this fix. Could we extend it? - Make `salary` an INCREMENT column and check its final value. - On the new server, delay loading one older segment, for example with a test upsert metadata manager like `DummyTableUpsertMetadataManager` in `UpsertTableIntegrationTest`. - Push increments during the rebalance. Check the consumer waits, then release the segment and check the total. **2. Follow-up, bigger scope** This check only runs when a consumer starts. If an older segment is removed while the consumer is running, for example by a RESET or a manual delete, its keys drop out of the upsert metadata and later updates are treated as new. Replicas can end up with different data. This needs its own design, so it's not for this PR. Could we add a TODO so the PR isn't read as covering it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
