ravjot28 opened a new issue, #20583:
URL: https://github.com/apache/hudi/issues/20583
### Bug Description
**What happened:**
With `HoodieJavaWriteClient` on a merge-on-read table and an index that
cannot index log files (`SIMPLE`, the Java
client's default, `BLOOM`, ...), inserts routed to a small file group
rewrite its base file from the previous base
file. The updates stored in that file group's log files drop out of the
snapshot. All writes report success, and the
read-optimized and snapshot views agree, so nothing signals the loss.
**What you expected:**
The snapshot keeps the latest version of every key: inserts are not packed
into a file group whose log files would be
left behind, as in the Spark client.
**Steps to reproduce:**
1. Create a merge-on-read table with the Java client and index type `BLOOM`
(`hoodie.parquet.small.file.limit` at its
default, above zero).
2. `insert` 20 records into one partition; they land in a base file below
the small-file limit.
3. `upsert` new versions of 5 of those records; they land in a log file of
the same file group.
4. `insert` 20 more records into the same partition (the same happens with
`insertPreppedRecords`).
5. Read the snapshot: 40 records, but the 5 updated keys carry the values of
step 2.
**Root cause (master d5230b69e22):**
-
`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/deltacommit/BaseJavaDeltaCommitActionExecutor.java:67-70`:
`getUpsertPartitioner` takes the copy-on-write `JavaUpsertPartitioner`,
whose `getSmallFiles`
(`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/action/commit/JavaUpsertPartitioner.java:240-262`)
picks small files from the base-file-only view, whether or not the latest
file slice has log files.
- `BaseJavaDeltaCommitActionExecutor.java:73-90`: for an index that cannot
index log files, inserts assigned to such a
small file run the copy-on-write merge handle, which reads only the
previous base file and writes a new base file;
the log files stay with the previous file slice.
- The Spark client avoids this in
`SparkUpsertDeltaCommitPartitioner#getSmallFileCandidates`
(`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/deltacommit/SparkUpsertDeltaCommitPartitioner.java:90-117`):
when the index cannot index log files, only file slices without log files
and outside pending compaction are
candidates. The Java client has no merge-on-read partitioner.
-
`hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/HoodieJavaMergeOnReadTable.java`
does not override
`insertPrepped`, so `insertPreppedRecords` on a merge-on-read table runs
the copy-on-write executor with the same
small-file selection and loss.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1, 0.14.x
**Query engine:** Java client (`HoodieJavaWriteClient`)
**Relevant configs:** `hoodie.table.type=MERGE_ON_READ`,
`hoodie.index.type=SIMPLE` (default) or `BLOOM` (or another index
that cannot index log files), `hoodie.parquet.small.file.limit` > 0 (default)
### Logs and Stack Trace
none: the writes succeed; the snapshot returns the older values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]