Vamsi-klu opened a new pull request, #19087: URL: https://github.com/apache/pinot/pull/19087
## Why Ingestion-level aggregations (`SUM(col1) AS col2`) are a core rollup-on-write pattern. When `col1` is **not** in the schema (common — only the aggregated metric is stored), `DataTypeTransformer` never converted it. Aggregators then saw raw strings/objects, threw (`ClassCastException` / parse errors), and left the mutable segment on the **partial-row corruption path** described in #16316. ## Impact - **Correct, reliable rollup-on-write** for JSON/Avro string numerics without stuffing raw columns into the table schema. - **Fewer mid-row exceptions** during realtime consumption for aggregation tables. - Complements (does not replace) fail-soft indexing in #16316 — prefer failing/converting **before** mutation when possible. - Uses existing `SourceFieldConfig` / transformer machinery — no new schema field type. ## How - Auto-derive aggregation **source** columns that are absent from the schema and register them for type conversion in `RecordTransformerUtils` (alongside optional explicit `SourceFieldConfig`). - Keep `ValueAggregatorUtils.toDouble` as a safety net; document that pipeline conversion is preferred so bad values fail before `MutableSegmentImpl` mutates. - Tests for string sources not in schema through the transform + mutable aggregation path. ## Test plan - [x] `MutableSegmentImplIngestionAggregationTest` — string/non-schema sources for SUM/MIN/MAX-style configs. - [x] `RecordTransformerTest` — aggregation source auto type conversion order/behavior. - [ ] `./mvnw -pl pinot-segment-local -am -Dtest=MutableSegmentImplIngestionAggregationTest,RecordTransformerTest -Dsurefire.failIfNoSpecifiedTests=false test` ## Related fixes: #16317 related: #16316 ## Reviewers Suggested: Jackie-Jiang (issue design guidance) --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes — Grok Build (xAI) Generated-by: Grok Build (xAI) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
