This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new a60a8fc1226 Allow intermediate transform columns not in the schema
when consumed by another transform (#18721)
a60a8fc1226 is described below
commit a60a8fc1226b7eb5dbc1778ab6ebbc8c874e4e5a
Author: Xiang Fu <[email protected]>
AuthorDate: Tue Jun 9 15:28:00 2026 -0700
Allow intermediate transform columns not in the schema when consumed by
another transform (#18721)
* Allow intermediate transform columns not in the schema when consumed by
another transform
TableConfigUtils.validateIngestionConfig rejected any transform whose
destination column was absent from the schema (or aggregation sources).
That blocks chained / "parse-once" transforms, where an intermediate
column is computed once and consumed by several downstream transforms but
is never itself stored, e.g.:
message_obj = jsonExtractObject(message) // intermediate
level = JSONPATHSTRING(message_obj, '$.level') // consumes it
msg_time = JSONPATHSTRING(message_obj, '$.time') // consumes it
The runtime already supports this: ExpressionTransformer builds an
evaluator for every transform config regardless of schema membership,
topologically sorts them by argument dependency, and materializes each
into the record; columns absent from the schema are simply dropped before
indexing. Only the config validation was rejecting it.
Relax the check to also accept a destination column that is referenced as
the input of another transform function. Destinations that are neither in
the schema, an aggregation source, nor consumed by another transform still
fail, so a fat-fingered (unreferenced) destination is still caught.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* Test multi-hop dead-chain rejection and non-schema cycle behavior
Strengthen the intermediate-transform-column coverage: a multi-hop chain
whose leaf is a non-schema column consumed by nothing still fails
(typo protection holds across the chain, not just for direct
destinations), and document that a non-schema 2-node cycle passes this
validation (consistent with schema-column cycles - cycle detection
happens later in ExpressionTransformer's topological sort).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* Skip Groovy expressions in the transform pre-pass when Groovy is disabled
The argument-collection pre-pass called getExpressionEvaluator() for every
transform, which compiles Groovy expressions even when _disableGroovy is
true - defeating the main loop's no-compile rejection guard and adding
needless validation cost for configs that are rejected anyway. Guard the
pre-pass with the same isGroovyExpression check.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
---
.../segment/local/utils/TableConfigUtils.java | 31 ++++++++++++++--
.../segment/local/utils/TableConfigUtilsTest.java | 41 ++++++++++++++++++++++
2 files changed, 69 insertions(+), 3 deletions(-)
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java
index 5d8c1919e6a..c833b7cdaa3 100644
---
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java
@@ -658,6 +658,29 @@ public final class TableConfigUtils {
// Transform configs
List<TransformConfig> transformConfigs =
ingestionConfig.getTransformConfigs();
if (transformConfigs != null) {
+ // Pre-pass: collect every column referenced as a transform-function
argument. A transform whose destination
+ // is not in the schema is still valid when another transform consumes
it as an input - i.e. it is an
+ // intermediate ("derived") column. This enables chained / parse-once
transforms, e.g.
+ // message_obj = jsonExtractObject(message) //
intermediate, not in the schema
+ // level = JSONPATHSTRING(message_obj, '$.level') // consumes
the intermediate
+ // The intermediate is materialized in the record during
transformation and dropped before indexing (only
+ // schema columns are indexed). Unreferenced non-schema destinations
still fail below (typo protection).
+ Set<String> transformInputColumns = new HashSet<>();
+ for (TransformConfig transformConfig : transformConfigs) {
+ String transformFunction = transformConfig.getTransformFunction();
+ // Skip Groovy expressions when Groovy is disabled: do not compile
them just to collect arguments (the main
+ // loop below rejects Groovy without compiling). Such a config is
rejected anyway, so these columns are not
+ // needed as valid intermediate targets.
+ if (transformFunction != null
+ && !(_disableGroovy &&
FunctionEvaluatorFactory.isGroovyExpression(transformFunction))) {
+ try {
+ transformInputColumns.addAll(
+
FunctionEvaluatorFactory.getExpressionEvaluator(transformFunction).getArguments());
+ } catch (Exception ignore) {
+ // Invalid functions are reported with a descriptive error in
the main loop below.
+ }
+ }
+ }
Set<String> transformColumns = new HashSet<>();
for (TransformConfig transformConfig : transformConfigs) {
String columnName = transformConfig.getColumnName();
@@ -669,10 +692,12 @@ public final class TableConfigUtils {
if (!transformColumns.add(columnName)) {
throw new IllegalStateException("Duplicate transform config found
for column '" + columnName + "'");
}
- Preconditions.checkState(schema.hasColumn(columnName) ||
aggregationSourceColumns.contains(columnName),
+ Preconditions.checkState(
+ schema.hasColumn(columnName) ||
aggregationSourceColumns.contains(columnName)
+ || transformInputColumns.contains(columnName),
"The destination column '" + columnName
- + "' of the transform function must be present in the schema
or as a source column for "
- + "aggregations");
+ + "' of the transform function must be present in the
schema, be consumed as the input of another "
+ + "transform function, or be a source column for
aggregations");
FunctionEvaluator expressionEvaluator;
if (_disableGroovy &&
FunctionEvaluatorFactory.isGroovyExpression(transformFunction)) {
throw new IllegalStateException(
diff --git
a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java
b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java
index 247b1f973df..0776eeda342 100644
---
a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java
+++
b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/utils/TableConfigUtilsTest.java
@@ -513,6 +513,47 @@ public class TableConfigUtilsTest {
new TransformConfig("myCol", "lower(transformedCol)")));
TableConfigUtils.validate(tableConfig, schema);
+ // intermediate column NOT in the schema but consumed as the input of
another transform - should pass.
+ // Enables chained / parse-once transforms (e.g. obj =
jsonExtractObject(col); field = JSONPATHSTRING(obj, ...)).
+ ingestionConfig.setTransformConfigs(Arrays.asList(new
TransformConfig("intermediateCol", "reverse(anotherCol)"),
+ new TransformConfig("myCol", "lower(intermediateCol)")));
+ TableConfigUtils.validate(tableConfig, schema);
+
+ // same as above but the consumer is listed BEFORE the producer - the
check is order-independent, should pass
+ ingestionConfig.setTransformConfigs(Arrays.asList(new
TransformConfig("myCol", "lower(intermediateCol)"),
+ new TransformConfig("intermediateCol", "reverse(anotherCol)")));
+ TableConfigUtils.validate(tableConfig, schema);
+
+ // destination column NOT in the schema and NOT consumed by any other
transform - should fail (typo protection)
+ ingestionConfig.setTransformConfigs(
+ Collections.singletonList(new TransformConfig("notInSchemaCol",
"reverse(anotherCol)")));
+ try {
+ TableConfigUtils.validate(tableConfig, schema);
+ fail("Should fail: destination column not in schema and not consumed by
another transform");
+ } catch (IllegalStateException e) {
+ // expected
+ }
+
+ // multi-hop chain whose LEAF is a non-schema column consumed by nothing -
still fails. Typo protection holds
+ // across a chain: 'intermediateCol' is allowed (consumed by
'danglingLeaf'), but 'danglingLeaf' itself is not in
+ // the schema and is referenced by nothing, so validation fails on it.
+ ingestionConfig.setTransformConfigs(Arrays.asList(new
TransformConfig("intermediateCol", "reverse(anotherCol)"),
+ new TransformConfig("danglingLeaf", "lower(intermediateCol)")));
+ try {
+ TableConfigUtils.validate(tableConfig, schema);
+ fail("Should fail: chain leaf 'danglingLeaf' is not in the schema and is
consumed by nothing");
+ } catch (IllegalStateException e) {
+ // expected
+ }
+
+ // a 2-node cycle among non-schema columns passes this validation (each is
referenced by the other), consistent
+ // with cycles among schema columns - TableConfigUtils does not do cycle
detection; a cycle is caught later by
+ // ExpressionTransformer's topological sort at ingestion time.
+ ingestionConfig.setTransformConfigs(Arrays.asList(new
TransformConfig("cycleA", "reverse(cycleB)"),
+ new TransformConfig("cycleB", "lower(cycleA)")));
+ TableConfigUtils.validate(tableConfig, schema);
+ ingestionConfig.setTransformConfigs(null);
+
// invalid field name in schema with matching prefix from
complexConfigType's prefixesToRename
ingestionConfig.setTransformConfigs(null);
ingestionConfig.setComplexTypeConfig(
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
