jianguotian opened a new pull request, #10538:
URL: https://github.com/apache/paimon/pull/10538
### Purpose
`MosaicRecordsWriter.writeBundle()` currently writes ordinary
`VectorizedBundleRecords` one row at a time. This change converts supported
scalar columns into writer-owned Arrow batches and writes them through the
existing Mosaic writer.
- Preserve selection order, duplicate selections, null values, and the order
of interleaved row and batch writes. Empty batches remain a no-op; schemas
containing nested types continue to use the row path.
- Size conversion slices using `write.batch-size` and the soft
`write.batch-memory` budget. Limit allocations during multi-row conversion and
retry smaller slices when necessary. Oversized single rows retain the original
allocator limit.
- Discard partial conversions on failure and reject subsequent writes.
Native writes are not retried, including during close.
- Enforce `NOT NULL` constraints for the rows selected for column
conversion, and copy string byte slices directly into Arrow buffers without
materializing intermediate byte arrays.
### Tests
Validated on JDK 8 / Linux aarch64 with Mosaic 0.2.0:
```bash
mvn -B -pl paimon-arrow,paimon-mosaic -am \
-Dtest='ArrowFormatWriterTest,ArrowBatchReaderTest,Mosaic*Test' \
-DwildcardSuites=none -DfailIfNoTests=false \
-Dsurefire.failIfNoSpecifiedTests=false clean verify
```
- **193 tests passed**, with zero failures, errors, or skipped tests.
Checkstyle, Spotless, and Enforcer passed.
- Tests cover selection order and duplicates, nullability, string slices,
nested-type fallback, skewed variable-width data, allocation retries, oversized
rows, mixed row/batch writes, and failure cleanup.
- An additional native round-trip probe verified 18 scalar fields across 154
output rows with independent producer and writer allocators, no row-iterator
fallback, and no remaining Arrow allocations after close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
