ravjot28 opened a new pull request, #20593:
URL: https://github.com/apache/hudi/pull/20593

   ### Describe the issue this Pull Request addresses
   
   Closes #20583
   
   With the Java client on a merge-on-read table and an index that cannot index 
log files, an insert routed to a small
   file group rewrites the base file from the previous base file and leaves out 
the updates in that file group's log
   files; the snapshot then returns the older rows, with no error. Prepped 
inserts on a merge-on-read table run the
   copy-on-write executor and lose updates the same way.
   
   ### Summary and Changelog
   
   Inserts on a Java merge-on-read table are packed only into file groups whose 
rewrite is safe, as in the Spark client.
   
   - Add `JavaUpsertDeltaCommitPartitioner`, the Java counterpart of 
`SparkUpsertDeltaCommitPartitioner`; its small-file
     selection is copied from the Spark class: for an index that cannot index 
log files, only file slices without log
     files and outside pending compaction, smallest first, limited by 
`hoodie.merge.small.file.group.candidates.limit`;
     for an index that can index log files, file slices below the maximum file 
size.
   - `BaseJavaDeltaCommitActionExecutor#getUpsertPartitioner` creates that 
partitioner.
   - Add `JavaInsertPreppedDeltaCommitActionExecutor` and use it for 
`HoodieJavaMergeOnReadTable#insertPrepped`.
   - New tests in `TestHoodieJavaClientOnMergeOnReadStorage`: 
`testInsertIntoSmallFileGroupKeepsLoggedUpdates`
     (`BLOOM`), parameterized over `insert` and `insertPreppedRecords`, and
     `testInsertsWithAnIndexOnLogFilesFillTheSmallFileGroup` (`INMEMORY`: a 
second insert goes to the log-only file group
     of the first).
   
   ### Impact
   
   No API or config change. On Java merge-on-read tables the small-file 
selection becomes the one of the Spark client:
   
   - Index that cannot index log files (the default `SIMPLE`, `BLOOM`): inserts 
no longer go to a small base file whose
     file group has log files or a pending compaction, and at most 
`hoodie.merge.small.file.group.candidates.limit`
     (default 1) small file groups per partition are filled per write, smallest 
first; before, every base file below
     `hoodie.parquet.small.file.limit` was a candidate. This can create more 
new file groups even when existing groups
     have no logs.
   - Index that can index log files (`INMEMORY`, bucket): inserts are appended 
to any latest file slice outside a
     pending compaction whose estimated size is below 
`hoodie.parquet.max.file.size`, log-only file groups included;
     before, only to file groups with a base file below 
`hoodie.parquet.small.file.limit`.
   - Prepped inserts on a merge-on-read table are written by the delta commit 
executor, so with an index that can index
     log files they are appended to log files instead of written as base files.
   
   The inherited Java gate still disables small-file selection when 
`hoodie.parquet.small.file.limit` is zero.
   
   **Backport:** please include this fix in the next 1.2.x patch release. The 
commit cherry-picks cleanly onto
   `release-1.2.1`.
   
   ### Risk Level
   
   low. The selection logic is the one the Spark client uses for merge-on-read 
tables; the Java copy-on-write path is
   unchanged.
   
   Tests, run on master with JDK 11 and passing checkstyle checks for main and 
test sources (no Scala sources changed;
   scalastyle did not execute in this module selection), with `mvn test -pl 
hudi-client/hudi-java-client -am -Dtest=...`
   and the classes listed below:
   - Without the main code change: 3 of the 8 tests of 
`TestHoodieJavaClientOnMergeOnReadStorage` fail: both cases of
     `testInsertIntoSmallFileGroupKeepsLoggedUpdates` (the snapshot returns the 
first insert's commit time instead of
     the update's) and `testInsertsWithAnIndexOnLogFilesFillTheSmallFileGroup` 
(two file groups instead of one).
   - With the change: `TestHoodieJavaClientOnMergeOnReadStorage` 8/8, 
`TestHoodieJavaClientOnCopyOnWriteStorage` 39/39,
     `TestJavaCopyOnWriteActionExecutor` 8/8 pass.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to