JingsongLi opened a new pull request, #1128: URL: https://github.com/apache/paimon-rust/pull/1128
## Purpose Complete the Native read paths for schema changes PyPaimon already supports, using Java's `SchemaEvolutionUtil` and scalar cast rules. The read path still used generic Arrow casts for several conversions even though the Java-compatible executors already existed in core. ## Changes - Reuse the core Java scalar executors for lexical parsing, numeric strings, and character/binary widths; reject invalid text instead of silently producing NULL. - Handle INTEGER timestamp casts, signed-long timestamp precision truncation, Java type aliases/default parameters, and equivalent Decimal256 physical storage. - Reconcile LargeList and other existing ARRAY layouts recursively while preserving historical field IDs. Convert only visible elements, excluding sliced backing values and children of NULL ROW/ARRAY/MAP containers. - Normalize selected MAP string keys and fresh/already assembled VARIANT projections into the read layout. - Use a consistent runtime Arrow read schema so Java cast-produced NULLs remain representable for logical NOT NULL fields. Persisted table constraints and canonical writer schemas remain unchanged. - Expose thin Python Arrow transport functions for schema casts/read schemas and the existing writer normalization. Read-to-write normalization validates actual visible NOT NULL values in core. ## Verification - Core library: 4,045 passed, 6 ignored. - Actual-Java scalar fixture reused through the production schema-evolution entry point: 1,562 cases in each of UTC, Asia/Shanghai and America/New_York. - Nested, decimal and predicate schema-evolution integration tests: 4 passed across append, PK and data-evolution modes. - DataFusion self-contained library tests: 368 passed. Seven existing tests requiring external pre-populated warehouse tables were excluded locally after confirming those tables were absent. - Rust 1.98.0 formatting and Clippy (`paimon`, `paimon-datafusion`, Python binding; library/tests) passed. - Paired PyPaimon Native acceptance suite: 639 passed, 2 skipped, including read-to-write constraints and NULL containers. Additional Python fallback regressions: 244 passed, 1 skipped. The Java fixture comparisons exclude outputs outside Arrow's physical representation and malformed UTF-8; their existing explicit storage/error behavior remains covered separately. No new table formats or Python-unsupported features are introduced. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
