xiangfu0 commented on code in PR #19244:
URL: https://github.com/apache/pinot/pull/19244#discussion_r3774612129
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -620,7 +621,9 @@ static void validateIngestionConfig(TableConfig
tableConfig, Schema schema,
+ columnName + "'");
}
try {
- validateIngestionTransformFunctionVolatility(transformConfig,
existingTransformConfigs);
+ if (tableConfig.getTableType() == TableType.REALTIME) {
+ validateIngestionTransformFunctionVolatility(transformConfig,
existingTransformConfigs);
+ }
Review Comment:
Addressed in
[`bd446c3539`](https://github.com/apache/pinot/pull/19244/commits/bd446c3539269d93f7dc6ab6ed9281aa0ce2364f).
Row-major OFFLINE generation now detects non-immutable filter,
table-transform, and legacy schema-transform functions. During statistics
collection it writes each fully transformed output row to a temporary disk
spool, then replays those exact rows into the index creator without invoking
the transform pipeline a second time. A per-row offset sidecar bounds decoding
and validates row counts/boundaries; spool failures abort the build and both
temporary files are cleaned up.
The end-to-end regression suite covers dictionary-encoded and raw `rand()`,
`now()` on a `LONG` column, metadata/cardinality readback, and one source read
for volatile pipelines versus two for immutable pipelines. The spool suite
covers the physical type matrix, complex/null values, row framing, corrupt
offsets, and failure cleanup. The focused JDK 25 run passed 12/12 tests, and
the affected-module Spotless, Checkstyle, and license gates all pass.
I also added ingestion eligibility metadata so query-context-only functions
such as `reqId()` are rejected instead of validating and failing at runtime.
The PR remains draft while the ordering semantics, temporary-disk cost, and
rolling-upgrade requirement are reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
