XiaoHongbo-Hope opened a new pull request, #10633: URL: https://github.com/apache/paimon/pull/10633
### Purpose Fix pure Python VARIANT shredding silently changing scalar types (for example, `0` to `0.0` or `2.5` to `2`) and dropping explicit JSON null fields. Only write a typed scalar when its Arrow conversion can reconstruct the original Variant encoding; otherwise preserve the original residual bytes. Keep JSON null distinct from missing fields and SQL NULL. ### Tests - Variant and Variant path suites: 170 passed, 64 subtests passed (PyArrow 19.0.1, pypaimon-rust 0.4.0). - Type-sensitive multi-batch table round trips compare Python/native writers and plain/shredded Variant, with physical Parquet residual assertions. - Original eight-row probe: Python shredding improves from 5/8 to 8/8 equivalent rows; native controls remain 8/8. - Flake8 and `git diff --check` pass. The check is conservative and adds per-scalar conversion/encoding work; write throughput has not been benchmarked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
