JingsongLi opened a new pull request, #10502: URL: https://github.com/apache/paimon/pull/10502
## Summary Align an existing Python capability—reading explicit numeric narrowing and timestamp-precision schema changes—with Java and Native. Numeric narrowing previously returned different results for overflowing floats and DECIMAL values; timestamp precision reductions within one Arrow unit were skipped. Old-file stats could also prune rows which matched after conversion. - Move shared schema-evolution casting into `pypaimon.casting.schema_evolution` and pass both declared Paimon types at every scalar leaf, including nested ROW/ARRAY/MAP evolution. - Apply Java float-to-int/long saturation and truncation before integer narrowing, DECIMAL integral truncation/low-bit retention, and declared TIMESTAMP/TIMESTAMP_LTZ precision truncation. Retain existing DECIMAL HALF_UP behavior. - Add `SimpleStatsEvolutions.filter_unsafe_filter`, following Java's same-type/integer-widening whitelist. Keep safe top-level AND conjuncts; retain OR only when every branch is safe. Reject out-of-range and NULL widening literals instead of comparing incompatible old-file bounds. - Use evolution-safe stats predicates in append and PK scans which exclude level-0 files. - Exercise Java-aligned results against both Python and Native over REST catalogs, including old/new files, id-only projections, NULL containers and nested leaf projections. Native-enabled runs fail if Native plan/read falls back to Python. Single-value files specifically cover BIGINT 128 → TINYINT -128 and TIMESTAMP(3) 123ms → TIMESTAMP(1) 100ms. A broader positive/negative min/max range can hide these pruning regressions. ## Dependency Depends on https://github.com/apache/paimon-rust/pull/1102 for Native numeric and timestamp conversion and safe evolved predicates. This PR is draft until that change reaches Rust main. Native CI continues to build `apache/paimon-rust@main`; no fork pin or workflow change is introduced. ## Java references NumericPrimitiveCastRule, DecimalToNumericPrimitiveCastRule, DecimalUtils.castToIntegral, TimestampToTimestampCastRule, DateTimeUtils.truncate, CastExecutors.castLiteralsWithEvolution and SimpleStatsEvolutions.filterUnsafeFilter. ## Validation Built the Native Python extension from the companion Rust branch and enabled Native plan/read/write/commit for: ``` pypaimon/tests/native_schema_evolution_cast_test.py pypaimon/tests/native_plan_resolved_schema_test.py pypaimon/tests/schema_evolution_cast_test.py pypaimon/tests/schema_evolution_read_test.py pypaimon/tests/schema_evolution_nested_read_test.py pypaimon/tests/manifest/simple_stats_evolutions_test.py pypaimon/tests/pushdown_correctness_test.py ``` - 259 tests and 43 subtests passed; 426 Native plans, 431 reads, 441 writes and 139 commits exercised. - Re-ran stats-evolution tests after adding the NULL `IN`/`NOT IN` literal guard: 13 tests and 20 subtests passed. - Flake8, `git diff --check`, and Python 3.6 syntax parsing passed for all seven changed/new Python files. - Rust companion regression tests, Clippy 1.98.0 and formatting checks also passed. All Native cases use only existing PyPaimon schema evolution capabilities in append, PK and Data Evolution modes. No public binding API is added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
