xiangfu0 commented on PR #19118: URL: https://github.com/apache/pinot/pull/19118#issuecomment-5199447529
Reworked in `f4455912c3` to make the OFFLINE upsert definition explicit and keep `$creationTime` out of the normal query path. - `$comparisonColumn` is now the physical hidden per-record column configured in `upsertConfig.comparisonColumns`. - A normal ingestion transform can default it with `$comparisonColumn = $creationTime`. - `$creationTime` is supplied only while reading/building Pinot segments. - Legacy segments are backfilled from their source index creation time; existing physical values win and survive later rewrites. - CONCAT preserves each value, while DEDUP and ROLLUP retain the maximum without adding the column to equality/grouping keys. - The earlier TableCache/broker exposure and Refresh-specific production paths are removed. After the clean rebase onto current master, 133 focused tests and all four required module quality gates passed. The RefreshSegmentTask E2E also verifies that a rebuilt old segment keeps its old physical comparison value and cannot replace newer pushed data. This addresses both the hidden/physical-column guidance in https://github.com/apache/pinot/pull/19118#issuecomment-5170523641 and the per-record merge requirement in https://github.com/apache/pinot/pull/19118#issuecomment-5185675042. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
