Gabriel39 commented on PR #68458: URL: https://github.com/apache/doris/pull/68458#issuecomment-5811397955
Could we add a real scanner/reader reproducer that fails before this change and passes afterward? The current test does not establish the schema mismatch described in the PR. In `PaimonJniScanner.initReader()`, `paimonDataTypeList` is populated from `table.rowType()`, then passed to `PaimonColumnValue.setIdx()`. This is the current table schema, not the historical file schema. For the described `DECIMAL(5,2) -> DECIMAL(6,3)` evolution, the Paimon type supplied here is therefore also `DECIMAL(6,3)`. I verified this with Paimon 1.3.1, the dependency used by this PR: write `1.20` to a Parquet-backed `DECIMAL(5,2)` table, evolve the column to `DECIMAL(6,3)`, write `1.234`, and read through `newRead().executeFilter().createReader(split)`. Comparing the original getter expression with the PR getter on the same actual records produced: | Record | SDK row implementation | Original getter | PR getter | | --- | --- | --- | --- | | Written before evolution | `CastedRow` | `1.200` | `1.200` | | Written after evolution | `ColumnarRow` | `1.234` | `1.234` | For the historical record, `CastedRow.getDecimal()` applies its internal cast mapping and does not use the precision/scale arguments supplied by this getter. The SDK has already handled the evolution at this boundary. The added unit test manually combines a Doris `DECIMAL(6,3)` with a Paimon `DECIMAL(5,2)`. Its proxy also changes the underlying unscaled value from `120` to `1234` depending on the requested scale, rather than decoding a fixed stored value. It proves which arguments are selected, but does not demonstrate that this mismatch occurs in the production scanner. If the actual failure requires a mismatch between the FE type and the reader's table schema, please reproduce that specific condition and clarify how it arises. The comment describing `dataType` as the physical historical schema should also be adjusted to match the actual reader contract. Validation scope: all 10 `PaimonColumnValueTest` tests passed, and the real SDK/Parquet probe above completed successfully. I did not run the full Doris/Spark regression suite. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
