jianguotian opened a new pull request, #10603:
URL: https://github.com/apache/paimon/pull/10603
### Purpose
A file index currently makes `RowDataFileWriter.writeBundle()` write every
data row individually. For a Mosaic table with a VIN bitmap, this prevents the
existing column-batch writer from being used.
Allow `VectorizedBundleRecords` to retain the data batch path. Build file
indexes through the batch's repeatable row views, then forward the same bundle
to the format writer. Index positions follow the selected output order.
### Changes
- Change only `RowDataFileWriter` in production. Keep eligibility checks and
index processing in private helpers so the main path reads as eligibility,
indexes, data batch, and sequence accounting.
- Process indexes before the format writer can consume borrowed buffers.
Reject closed writers before reading index fields and abort on index-processing
failure.
- Preserve the row-tracking, per-record statistics, and row-sidecar
fallbacks. Other bundle implementations, including `ArrowBundleRecords`, retain
their existing behavior.
- Reuse existing file-index algorithms and Mosaic column-batch writing. No
new public API, switch, dependency, or POM change.
Mosaic remains at 0.2.0.
### Tests
60 focused tests passed with zero failures, errors, or skips on JDK 8 /
Linux ARM64, with normal Maven checks enabled:
```sh
mvn -B -pl paimon-mosaic -am \
-Dtest=RowDataFileWriterTest,DataFileIndexWriterTest,RollingFileWriterTest,MosaicReaderWriterTest,MosaicRecordsWriterTest
\
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false \
-DwildcardSuites=none clean verify
```
- `RowDataFileWriterTest` covers batch dispatch, embedded/external indexes,
file-local positions after rollover, selection order and duplicates, nulls,
mixed row/batch writes, empty batches, sequence accounting, fallback, and
failure cleanup.
- `MosaicReaderWriterTest` adds one native integration test: read a real
Mosaic file, rewrite its column batches with VIN bitmap and statistics enabled,
and verify the output values and index positions. Any call to
`MosaicRecordsWriter.addElement()` fails the test, so correct output cannot
hide a row fallback.
### Performance
A local rewrite benchmark used original Geely Mosaic samples: **7 files,
66,658 rows in total, and 8,700 columns**, rewritten into one output file with
a VIN bitmap index and VIN statistics enabled.
Compared the original and updated `RowDataFileWriter` implementations on
Linux ARM64, JDK 8, and Mosaic 0.2.0. Each version's `RowDataFileWriter` (the
only changed production file) was compiled separately and run with the same
remaining classes and dependencies. Both used a write batch size of 1,024, a
128 MiB write buffer, Zstd level 1, and 100 Mosaic buckets.
Each version ran one full warmup and three measured iterations; results
below are medians. Total rewrite time includes source reads, index updates,
data writes, writer close, and metadata creation. Correctness checks run
outside the timed section.
| Metric | Before | After | Speedup |
| --- | ---: | ---: | ---: |
| Total local rewrite | 81.44 s | 13.20 s | **6.17x** |
| Write stage, including index updates | 78.28 s | 10.09 s | **7.76x** |
Total rewrite time decreased by **83.80%**. Per iteration, format-writer
dispatch changed from 66,658 `addElement()` calls to 7 `writeBundle()` calls.
All eight output data files, including warmups, had identical SHA-256
hashes. A full decoded-value checksum matched the original inputs; each
iteration also verified all VIN bitmap positions, row counts, sequence
accounting, and VIN statistics.
This measures local file rewriting; remote storage, job scheduling, and
table commits are outside the benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
