GitHub user ravjot28 created a discussion: MoR support for the standalone Java client (new HoodieJavaReadClient)
## Motivation The Java engine (`hudi-java-client`) today ships only `HoodieJavaWriteClient` and `HoodieJavaTableServiceClient`. There is **no read client**, and MERGE_ON_READ tables can't be completed end-to-end from a plain Java application - no compaction execution and no MoR reads. Spark and Flink both have full MoR support; embedded Java users have none. I have a working implementation on my fork (full CI matrix green) and would like feedback on scope and API shape before opening PRs. ## What's implemented **1. New `HoodieJavaReadClient`** (Java engine): - Snapshot, read-optimized, time-travel, and completion-time-based incremental reads, for both MERGE_ON_READ and COPY_ON_WRITE tables. - One `HoodieFileGroupReader` per file slice (base file merged with log files), exposed as a lazily-chained `ClosableIterator` → bounded memory. - Fresh meta-client snapshot per call → safe under NBCC with inflight writers; inflight instants are invisible to snapshot/incremental reads. - Documented v1 limitations: no archived-timeline incremental ranges, `populate.meta.fields=false` unsupported for completion-time filtering, no `InternalSchema` evolution reconciliation, no CDC. **2. MoR write-side completion**: delta commit + compaction action executor wiring so Java writers run compaction end-to-end. **3. Bucket index for the Java engine**: `JavaBucketIndexPartitioner`, bulk-insert partitioner, index factory + layout-config wiring. **4. `BaseAvroPayload` decode fix** (`hudi-common`): `getRecord(schema)` now resolves by name from the true writer schema for projection schemas (fixes a pre-existing bug where merges against persisted data silently nulled data fields). Evolution-shaped schemas (renames/adds) keep the legacy positional decode the Hive/Presto MoR readers depend on; unresolvable fields with no default fail loudly with `AvroTypeException`. This touches the shared merge path for all engines → planned as a separate, first PR with risk called out. ## Testing - ~2,900 lines of new tests: MoR snapshot/incremental reads, NBCC concurrent reader/writer, payload schema-mismatch regressions, bucket-index partitioner. - Full fork CI matrix green (all engine jobs), plus local runs of the Hive MoR schema-evolution functional tests and `hudi-hadoop-mr` realtime suites. ## Proposed plan 1. File a `type:feature` issue (+ sub-issues) and open three PRs: - `fix(common): resolve payload decode by name for projection schemas` - `feat(client): add HoodieJavaReadClient with MoR support` - `feat(client): support bucket index for the Java engine` 3. Land the common fix first (the read-client tests depend on it). ## Questions for the community - Issue + PRs sufficient, or is an RFC preferred given the new public read API? - Any concerns with the iterator-based API surface (caller-owned iterators)? - Are the v1 limitations acceptable for a first iteration? Code: https://github.com/apache/hudi/compare/master...ravjot28:hudi:master GitHub link: https://github.com/apache/hudi/discussions/19798 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
