JingsongLi opened a new pull request, #1102: URL: https://github.com/apache/paimon-rust/pull/1102
## Summary PyPaimon already allows explicit schema changes that narrow numeric types or reduce timestamp precision. Native reads previously delegated these conversions to Arrow: BIGINT 128 became NULL as TINYINT, and TIMESTAMP(3) 123ms stayed 123ms as TIMESTAMP(1). Java returns -128 and 100ms respectively. This change puts Java-compatible conversion in Rust core and makes filtering observe the converted values: - Preserve low bits when narrowing signed integers; truncate and saturate floating-point values to int/long before byte/short narrowing. - Truncate DECIMAL fractions and retain the low bits of BigDecimal.longValue before narrowing. - Apply declared TIMESTAMP and TIMESTAMP_LTZ precision, including pre-epoch floor semantics and precision changes within the same Arrow unit. - Reuse these conversions recursively through ROW, ARRAY, MAP and MULTISET evolution. - Evaluate predicates on changed or missing old-file columns after schema reconciliation, including predicate-only fields omitted from output projection. - Make manifest/statistics predicates fail open for unsafe type changes, following Java's castLiteralsWithEvolution whitelist; retain safe integer widening. Unknown must-match also protects NOT predicates. No public API or Python-binding API is added. Timestamp results outside Arrow's int64 storage remain NULL, following existing lenient read-side overflow handling. ## Java references The conversion rules follow NumericPrimitiveCastRule, DecimalToNumericPrimitiveCastRule, DecimalUtils.castToIntegral, TimestampToTimestampCastRule and DateTimeUtils.truncate. Statistics safety follows CastExecutors.castLiteralsWithEvolution and SimpleStatsEvolutions.filterUnsafeFilter. ## Validation - Numeric cast unit coverage across all signed source/target widths, floating-point saturation and DECIMAL overflow/fractions. - Timestamp precision coverage from 0 through 8, both timestamp kinds, negative values and int64 storage boundaries. - Append, PK and Data Evolution integration coverage for old/new files, converted predicates, NULLs and id-only output projection. Single-value old files reproduce and prevent the min/max pruning regression. - `cargo test -p paimon --lib arrow::evolution_cast::` — 4 passed. - `cargo test -p paimon --lib arrow::nested_evolution::tests::` — 23 passed. - `cargo test -p paimon --lib predicate_stats::` — 32 passed. - `cargo test -p paimon --lib data_file_reader::tests::` — 19 passed. - `cargo test -p paimon --test schema_evolution_cast_test` — 1 passed, exercising all three table modes. - `cargo test -p paimon --test pk_value_stats_test` — 1 passed. - `cargo +1.98.0 clippy -p paimon -p pypaimon_rust --all-targets -- -D warnings` and `cargo fmt --all --check` passed. - Built the Python extension from this branch and ran the related PyPaimon suite with Native plan/read/write/commit enabled: 259 tests and 43 subtests passed, with 426 Native plans, 431 reads, 441 writes and 139 commits exercised. The companion PyPaimon PR aligns its fallback conversions and stats-filter safety with Java and adds REST Native parity coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
