ravjot28 opened a new pull request, #20593:
URL: https://github.com/apache/hudi/pull/20593
### Describe the issue this Pull Request addresses
Closes #20583
With the Java client on a merge-on-read table and an index that cannot index
log files, an insert routed to a small
file group rewrites the base file from the previous base file and leaves out
the updates in that file group's log
files; the snapshot then returns the older rows, with no error. Prepped
inserts on a merge-on-read table run the
copy-on-write executor and lose updates the same way.
### Summary and Changelog
Inserts on a Java merge-on-read table are packed only into file groups whose
rewrite is safe, as in the Spark client.
- Add `JavaUpsertDeltaCommitPartitioner`, the Java counterpart of
`SparkUpsertDeltaCommitPartitioner`; its small-file
selection is copied from the Spark class: for an index that cannot index
log files, only file slices without log
files and outside pending compaction, smallest first, limited by
`hoodie.merge.small.file.group.candidates.limit`;
for an index that can index log files, file slices below the maximum file
size.
- `BaseJavaDeltaCommitActionExecutor#getUpsertPartitioner` creates that
partitioner.
- Add `JavaInsertPreppedDeltaCommitActionExecutor` and use it for
`HoodieJavaMergeOnReadTable#insertPrepped`.
- New tests in `TestHoodieJavaClientOnMergeOnReadStorage`:
`testInsertIntoSmallFileGroupKeepsLoggedUpdates`
(`BLOOM`), parameterized over `insert` and `insertPreppedRecords`, and
`testInsertsWithAnIndexOnLogFilesFillTheSmallFileGroup` (`INMEMORY`: a
second insert goes to the log-only file group
of the first).
### Impact
No API or config change. On Java merge-on-read tables the small-file
selection becomes the one of the Spark client:
- Index that cannot index log files (the default `SIMPLE`, `BLOOM`): inserts
no longer go to a small base file whose
file group has log files or a pending compaction, and at most
`hoodie.merge.small.file.group.candidates.limit`
(default 1) small file groups per partition are filled per write, smallest
first; before, every base file below
`hoodie.parquet.small.file.limit` was a candidate. This can create more
new file groups even when existing groups
have no logs.
- Index that can index log files (`INMEMORY`, bucket): inserts are appended
to any latest file slice outside a
pending compaction whose estimated size is below
`hoodie.parquet.max.file.size`, log-only file groups included;
before, only to file groups with a base file below
`hoodie.parquet.small.file.limit`.
- Prepped inserts on a merge-on-read table are written by the delta commit
executor, so with an index that can index
log files they are appended to log files instead of written as base files.
The inherited Java gate still disables small-file selection when
`hoodie.parquet.small.file.limit` is zero.
**Backport:** please include this fix in the next 1.2.x patch release. The
commit cherry-picks cleanly onto
`release-1.2.1`.
### Risk Level
low. The selection logic is the one the Spark client uses for merge-on-read
tables; the Java copy-on-write path is
unchanged.
Tests, run on master with JDK 11 and passing checkstyle checks for main and
test sources (no Scala sources changed;
scalastyle did not execute in this module selection), with `mvn test -pl
hudi-client/hudi-java-client -am -Dtest=...`
and the classes listed below:
- Without the main code change: 3 of the 8 tests of
`TestHoodieJavaClientOnMergeOnReadStorage` fail: both cases of
`testInsertIntoSmallFileGroupKeepsLoggedUpdates` (the snapshot returns the
first insert's commit time instead of
the update's) and `testInsertsWithAnIndexOnLogFilesFillTheSmallFileGroup`
(two file groups instead of one).
- With the change: `TestHoodieJavaClientOnMergeOnReadStorage` 8/8,
`TestHoodieJavaClientOnCopyOnWriteStorage` 39/39,
`TestJavaCopyOnWriteActionExecutor` 8/8 pass.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]