vinothchandar opened a new pull request, #666:
URL: https://github.com/apache/hudi-rs/pull/666
## Description
Bring the native write path to storage parity with the Apache Hudi Spark
writer (Hudi 1.1.x, table version 9 default; tv8 retained) and prove it
with
a Spark-in-the-loop parity harness.
Writer:
- Two-phase locking per write (LockProvider + InProcessLockProvider):
request instant under lock, complete + bookkeeping under lock.
- Skew-adjusted monotonic instant times, formatted in the table's
declared
timeline timezone (LOCAL default, matching Spark) so interleaved
writers
produce ordered timelines.
- Timeline-server-based marker files and eager rollback of failed writes
(rollback plan/metadata, orphan MDT cleanup).
- LSM timeline archival (manifest_N + _version_ + level-0 parquet) with
reader-side archival-floor and fencing unions.
- Small-file packing (UpsertPartitioner semantics) and group-aware COW
rewrites; parallel write tasks via hoodie.write.task.parallelism.
- MDT files/column_stats/partition_stats/record_index maintained per
commit;
MetadataPayloadMerger mirrors HoodieMetadataPayload combine semantics
and
is resolved from the MDT merge-strategy id.
- Deletes that empty a file group write an empty base file version;
commit
metadata stores the data schema without meta fields; batches differing
only in nullability are aligned.
- MDT records decode with the writer schema of their source container, so
mixed-writer tables (rs + Spark) read correctly.
- Incremental queries follow Hudi 1.x completion-time semantics with
exact
instant-set filtering, shared by table.read and manual reader paths.
Parity harness (make parity, HUDI_SPARK_PARITY=1, scripts/parity/):
six scenarios against a local Spark 3.5 + Hudi bundle — rs<->Spark reads
and
writes, Spark compaction/clustering/clean on rs tables followed by rs
writes,
interleaved writers, a 36-commit mixed workload with archival verified
via
snapshot/time-travel/incremental in both engines, and a CDC-enabled
table.
Diff size justified: this lands the writer feature set plus its test
matrix (six new integration suites and the parity harness).
## How are the changes test-covered
- [ ] N/A
- [ ] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
