peterxcli opened a new pull request, #5552:
URL: https://github.com/apache/datafusion-comet/pull/5552
## Which issue does this PR close?
Closes #5547.
This is the first atomic change in #5546, extracted from #5407. It is
intentionally independent of Variant scan admission.
## Rationale for this change
Native output currently exports each Arrow array with an `FFI_ArrowSchema`
reconstructed from `ArrayData.data_type()`. A DataType describes the physical
layout, but it does not carry the enclosing Field name, nullability, or
metadata. Logical types identified by Field metadata therefore lose their
identity at the Arrow C Data boundary.
The RecordBatch already owns the authoritative Field for each output column.
Exporting that Field fixes the shared boundary once for query output and the
native Parquet reader.
ArrowSchema names are NUL-terminated C strings. Passing the Field also makes
embedded NUL names visible to the exporter, so this change substitutes U+FFFD
only in the exported name while retaining the Field datatype, nullability, and
metadata. Spark's logical output name remains owned by its plan.
## What changes are included in this PR?
- Pass the corresponding RecordBatch Field to
`SparkArrowConvert::move_to_spark`.
- Construct `FFI_ArrowSchema` from the Field while leaving `FFI_ArrowArray`
export unchanged.
- Use the original output Field for offset-normalized arrays.
- Apply the same Field-aware export in `Native.currentColumnBatch`.
- Add a focused round-trip test for name sanitization, datatype,
nullability, and metadata.
This PR does not add a Variant protobuf type, Arrow extension mapping,
Parquet normalization, or scan admission. Those remain in the later #5546
subtasks.
## How are these changes tested?
```shell
cd native
cargo fmt --all -- --check
DYLD_LIBRARY_PATH="$JAVA_HOME/lib/server" \
cargo test -p datafusion-comet \
test_ffi_schema_preserves_field_and_sanitizes_nul_name
cd ..
git diff --check upstream/main...HEAD
```
The focused Rust test passed: 1 passed, 0 failed, 199 filtered out. Building
that test also compiled both updated native export callers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
