Weixin-Xu opened a new pull request, #13378:
URL: https://github.com/apache/gluten/pull/13378
## What changes are proposed in this pull request?
This PR fixes Arrow input handling and file write compatibility in the
Bolt backend.
- Introduce a stateful Arrow Python input writer. Empty input emits a
valid schema and end marker, while incremental writes consume one batch per
call within the same Arrow stream. Release resources when input
finishes or the task completes.
- Handle Arrow NullType columns during native batch import using a local
bridge option, preserving the shared bridge configuration.
- Honor `mapreduce.output.basename` when generating output file names.
- Use the existing Spark version shims for write, commit, and abort
exceptions, preserving Spark's error handling behavior.
Add focused regression tests for Python input streaming and Arrow batch
conversion. All changes are scoped to the Bolt backend.
## How was this patch tested?
These changes were validated as part of the integrated Bolt CI candidate
using Bolt revision `071e54764f887dfb4337260e440f2f12f6ab82f5` with the
recorded engine fixes applied.
Added regression coverage includes:
- Four Python input writer tests covering empty and multi-batch input
through both legacy and incremental APIs, including stream completion and
resource cleanup.
- Three native batch conversion tests covering NullType columns,
regular/dictionary/timestamp values, and identity preservation for existing
Bolt batches.
The integrated candidate completed the reference native C++ tests and
Spark test coverage.
After rebasing this topic onto Apache Gluten main, `range-diff` and stable
patch IDs confirmed that the patch was preserved. All seven changed files
retained their original contents and modes, and `git diff
--check` passed.
The standalone rebased branch has not been rebuilt or runtime-tested
independently.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
