zhang-arvin opened a new pull request, #68835: URL: https://github.com/apache/doris/pull/68835
### What problem does this PR solve? Fixes #68802. Reading a Paimon primary-key table whose `ARRAY<ROW>` column uses the `nested_update` aggregate function fails through the JNI scanner with a `ClassCastException` (e.g. `HeapBytesVector cannot be cast to LongColumnVector`) on splits that need merge-on-read. It only reproduces for such splits, so it is intermittent and disappears once files are compacted; `SET enable_prune_nested_column = false` masks it. Root cause: paimon's merge engine (`FieldNestedUpdateAgg`) builds its key projection against the column's **declared** element row type and reads the `nested-key` field by its original position. Doris' nested-column pruning narrows the read type to the sub-fields the query touches and pushes it down with `ReadBuilder.withReadType`; when the nested-key (the first element field) is dropped, the merge engine still projects that position out of the narrower rows and reads whatever now sits there. ### What changed - New `PaimonNestedUpdateColumns`: resolves, from the table's merged options, every column with `fields.<col>.aggregate-function = nested_update` that also declares a `fields.<col>.nested-key`. - `PaimonJniScanner.initReader` now keeps the whole element row (instead of the pruned shape) for such a column, so the merge engine's positional projection stays valid. The query still reads only the fields it asked for; the extra fields exist only to satisfy the merge engine. - Non-`nested_update` `ARRAY<ROW>` columns keep ordinary nested-column pruning. ### Tests - `PaimonReadTypeProjectionTest`: a `nested_update` column keeps its whole element row when pruned; a plain `ARRAY<ROW>` column still narrows. - `PaimonNestedUpdateColumnsTest`: option resolution (single/multi field nested-key, case-insensitivity, a `nested_update` column without a nested-key, non-primary-key tables). - Verified the new projection test fails without the fix. `mvn -pl be-java-extensions/paimon-scanner -am test`: 117 tests pass, checkstyle clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
