ravjot28 opened a new pull request, #20618:
URL: https://github.com/apache/hudi/pull/20618

   ### Describe the issue this Pull Request addresses
   
   Closes #20612
   
   On a merge-on-read table, `HoodieJavaWriteClient#upsertPreppedRecords` runs
   `JavaUpsertPreppedDeltaCommitActionExecutor`, which writes the updates to 
log files without moving the instant from
   requested to inflight. Only new records reach 
`JavaBulkInsertHelper#bulkInsert(..., executor, ...)`, which makes that
   transition (after the updates were written) and also updates the index and 
runs the pre-commit validators for their
   statuses, before the executor does both again for all statuses. A batch of 
updates only cannot be committed
   ("deltacommit.inflight does not exist"); a batch with new records updates 
the index twice. The Java metadata table
   writer moves its instant to inflight itself before calling this executor, so 
metadata table commits are not affected.
   
   ### Summary and Changelog
   
   Prepped upserts on Java merge-on-read tables move their instant to inflight 
before writing and update the index once.
   
   - `JavaUpsertPreppedDeltaCommitActionExecutor#execute`:
     - saves the workload profile of the prepped records to the inflight 
instant before writing, as
       `BaseJavaCommitActionExecutor#execute` does, unless the instant is 
already inflight (the metadata table writer's
       case; the same check `JavaBulkInsertHelper` makes);
     - writes new records through the `JavaBulkInsertHelper#bulkInsert` 
overload that returns their write statuses
       (same partitioner, handle factory and parallelism as before), so 
`updateIndexAndMaybeRunPreCommitValidations`
       runs once, for all statuses, at the end.
   - New `TestJavaUpsertPreppedDeltaCommitActionExecutor` (merge-on-read): a 
prepped batch of tagged updates only
     commits as a completed delta commit with the updated records readable; a 
batch of tagged updates and new records
     calls the index's `updateLocation` once (an index set through 
`hoodie.index.class` counts the calls) and commits
     all records.
   
   ### Impact
   
   No API or config change. Prepped upserts into Java merge-on-read data tables 
can be committed whatever the batch
   holds, and their inflight instant carries the workload profile. User-defined 
indexes and pre-commit validators see
   one call per write.
   
   **Backport:** please include this fix in the next 1.2.x patch release. On 
`release-1.2.1` the commit conflicts only in
   the import block of `JavaUpsertPreppedDeltaCommitActionExecutor` (1.2.1 does 
not import `AppendHandleFactory`); the
   code hunks and the test apply as they are.
   
   ### Risk Level
   
   low. The inflight transition and the single index update are what the other 
Java write paths already do; the
   metadata table's own transition is detected and kept.
   
   Tests, run on master with JDK 11 and passing checkstyle checks for main and 
test sources, with
   `mvn test -pl hudi-client/hudi-java-client -am -Dtest=...`:
   - Without the main code change: both tests of 
`TestJavaUpsertPreppedDeltaCommitActionExecutor` fail:
     `testUpdatesOnlyBatchCommits` with `IllegalArgumentException: File 
.../<instant>.deltacommit.inflight does not
     exist!` from `commit`, `testMixedBatchUpdatesTheIndexOnce` with `expected: 
<1> but was: <2>` index updates.
   - With the change: `TestJavaUpsertPreppedDeltaCommitActionExecutor` 2/2, 
`TestJavaHoodieBackedMetadata` 68/68 (the
     metadata table writer uses this executor), 
`TestHoodieJavaClientOnMergeOnReadStorage` 5/5,
     `TestHoodieJavaClientOnCopyOnWriteStorage` 39/39 pass.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to