JingsongLi opened a new pull request, #10629: URL: https://github.com/apache/paimon/pull/10629
### Purpose Keep Arrow upserts on Native when their matched payloads need conversions already supported by PyPaimon's row-ID updater. The current bridge checks every input column against the append schema before matching, so valid matched-only updates (STRING to INT, numeric coercion, decimal and nested conversions) fall back to Python. Only match and partition keys require a common schema before matching. Rust core selects surviving rows after last-write-wins, converts the actual matched update columns, and applies ordinary write validation to new rows. Python continues to transport Arrow input and commit messages; no matching or new conversion logic is added here. Depends on https://github.com/apache/paimon-rust/pull/1147. Native CI continues to build Rust main, so this PR is a draft until that core fix is merged. ### Tests Built the native extension from the companion Rust branch, then ran: ``` PYPAIMON_TEST_NATIVE_PLAN=1 PYPAIMON_TEST_NATIVE_READ=1 \ PYPAIMON_TEST_NATIVE_WRITE=1 PYPAIMON_TEST_NATIVE_COMMIT=1 \ python -m pytest -q \ pypaimon/tests/native_upsert_selected_values_test.py \ pypaimon/tests/native_upsert_layout_test.py \ pypaimon/tests/native_upsert_row_shapes_test.py \ pypaimon/tests/native_blob_row_write_test.py \ pypaimon/tests/native_blob_update_test.py \ pypaimon/tests/native_nested_update_test.py ``` Result: **495 passed, 2 skipped**. Native counters: 876 plans, 968 reads, 335 writes, 163 commits. Flake8 and `git diff --check` passed. New tests compare Python/Native batch and stream updates, scalar and nested conversions, NULL values, target fan-out, partition isolation, ignored columns, discarded duplicate values and required nested children, invalid surviving values, and unchanged append validation. Native fallbacks are made to fail explicitly. Six binary-layout rejection cases guard against introducing Binary/LargeBinary conversion through direct, nested MAP, or constructor-shaped inputs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
