xiangfu0 opened a new pull request, #18721:
URL: https://github.com/apache/pinot/pull/18721
## Problem
`TableConfigUtils.validateIngestionConfig` rejects any transform whose
destination column is absent from the schema:
```
The destination column 'container_obj' of the transform function must be
present in the schema or as a source column for aggregations
```
This blocks **chained / "parse-once" transforms**, where an intermediate
column is parsed/computed once and consumed by several downstream transforms
but is never itself stored. This is the key pattern for efficiently extracting
many fields from a JSON-string column on the ingestion path:
```jsonc
{"columnName": "message_obj", "transformFunction":
"jsonExtractObject(message)"} // intermediate
{"columnName": "level", "transformFunction":
"JSONPATHSTRING(message_obj, '$.level', null)"}
{"columnName": "msg_time", "transformFunction":
"JSONPATHSTRING(message_obj, '$.time', null)"}
```
Without an intermediate, `message` is re-parsed once per extracted field;
with one, it is parsed once (a ~2x ingestion-transform speedup on a JSON-heavy
log table).
## The runtime already supports this
`ExpressionTransformer` builds an evaluator for **every** transform config
regardless of schema membership, topologically sorts them by argument
dependency, and materializes each result into the `GenericRow`. Columns absent
from the schema are simply dropped before indexing (only schema columns are
indexed). So intermediate columns already work end-to-end — **only the config
validation was rejecting them.**
## Change
Relax the check to also accept a destination column that is **referenced as
the input of another transform function**. A pre-pass collects all
transform-function argument columns; the destination check then passes if the
column is in the schema, an aggregation source, **or** consumed by another
transform.
Destinations that are neither in the schema, an aggregation source, nor
consumed by another transform still fail — so an unreferenced (fat-fingered)
destination is still caught. The narrowed guarantee: the *leaf* of a transform
chain must still be a real (schema/agg) column; only a non-schema column that
genuinely feeds another transform is now allowed.
## Testing
`TableConfigUtilsTest#validateIngestionConfig` gains cases for: (a) an
intermediate column not in the schema consumed by another transform → passes;
(b) the same with consumer listed before producer (order-independent) → passes;
(c) a non-schema destination referenced by nothing → still fails. Verified
locally that a config with `message_obj`/`container_obj` intermediates now
validates, and an unreferenced non-schema destination is still rejected.
## Backward compatibility
Backward-compatible: every config that validated before still validates;
only previously-rejected chained-transform configs now pass. No config-key /
SPI / wire-format change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
