ravjot28 opened a new pull request, #20624:
URL: https://github.com/apache/hudi/pull/20624

   ### Describe the issue this Pull Request addresses
   
   Closes #20623
   
   In the Java client's commit path, bucket dispatch is serial: 
`BaseJavaCommitActionExecutor.execute` writes the buckets
   of a commit (each one existing file group to update, or an insert bucket 
that may roll over into several new file
   groups) one after the other on the calling thread, so a write that touches 
many file groups takes the sum of their
   write times. Spark writes the same buckets in parallel tasks.
   
   ### Summary and Changelog
   
   Java client users can write the file groups of a commit on several threads 
with the new
   `hoodie.java.write.max.parallelism` (default 1, which keeps the serial 
write).
   
   - `HoodieWriteConfig`: new advanced config 
`hoodie.java.write.max.parallelism` (since 1.3.0, default 1), getter,
     builder method, and a check that the value is positive.
   - New `JavaParallelTaskRunner` (`hudi-java-client`): applies a task to a 
list on a fixed pool of up to N daemon
     threads created for one call (`CustomizedThreadFactory`; the workers use 
the caller's context classloader). Workers
     take the next item until none is left. Taking an item and recording a stop 
(a task failure, or an interrupt of the
     caller) hold one lock, so once a stop is recorded no item is taken, while 
the items already taken run to their end.
     It returns or throws only after the pool has terminated, so no task runs 
any more. Results keep the input order. The
     first task failure recorded is thrown as is (a checked one wrapped in 
`HoodieException`), with the other distinct
     failures suppressed; if a pool thread cannot be created, that failure is 
thrown instead, with the task failures
     suppressed. The caller's interrupt flag is kept on every exit, including 
that one. With N = 1 or one item it runs on
     the calling thread.
   - `BaseJavaCommitActionExecutor.execute`: with the config above 1 and more 
than one bucket, the buckets are written
     through the runner. Each task drains its bucket's statuses, so insert 
buckets (whose files are written while the
     lazy iterator is consumed) are written on the pool thread too. With the 
default the existing loop runs unchanged.
   
   Why not `HoodieJavaEngineContext.map`: Flink's context honours its 
`parallelism` argument with a pool per call, and
   Java compaction merges file groups concurrently through list data, so this 
fan-out has precedents. But every
   Java-engine caller of `context.map` passes a value sized for Spark (the 
finalize-write and cleaner parallelisms
   default to 200, the rollback parallelism to 100, file listing up to 1,500), 
so making the Java context honour it
   would not be opt-in. And the write stage needs two guarantees `map` does not 
give: no new bucket starts after a
   failure, and no bucket task is still running when the exception leaves 
`execute`.
   
   Why not `ExecutorServiceBasedEngineContext` 
(https://github.com/apache/hudi/pull/18585): its `map` runs on one
   JVM-wide pool sized to the common pool's parallelism and ignores the 
`parallelism` argument, so a user cannot bound
   the file groups in flight per write; it submits every item up front and 
waits for all of them, so after one bucket
   fails the remaining buckets are still written before the exception arrives; 
and its workers carry the classloader
   that loaded Hudi rather than the caller's.
   
   Why a new key: `hoodie.upsert.shuffle.parallelism`, 
`hoodie.insert.shuffle.parallelism` and
   `hoodie.bloom.index.parallelism` describe Spark shuffle partitions and are 
ignored by the Java engine today; reusing
   them would give existing Java users that many threads on upgrade, with no 
opt-in. The name follows the engine-scoped
   `hoodie.spark.*` keys and the `max.parallelism` keys that bound concurrent 
work (`hoodie.clustering.max.parallelism`),
   and it is named for the write as a whole: a follow-up applies the same bound 
to the BLOOM index key check that tags
   the records of a write, so one value bounds the file groups a Java write 
works on at once.
   
   Thread safety follows the Java compaction path, which already writes file 
groups concurrently with one shared
   `HoodieTable`: every bucket has its own handle and write status; mergers are 
created per handle; the partitioner and
   the file system view are only read during the stage (the view manager getter 
is synchronized and views come from a
   concurrent map); file names stay distinct by file id under the constant Java 
write token; markers are one per file;
   the partition metafile is written through a temporary name. Lifecycle calls 
(heartbeat, commit) stay outside the
   stage. The metadata table's write config does not inherit the key, so its 
bucket-write stage keeps parallelism 1.
   
   ### Impact
   
   No breaking API change. Additive: the config 
`hoodie.java.write.max.parallelism` with
   `HoodieWriteConfig#getJavaWriteMaxParallelism` and 
`HoodieWriteConfig.Builder#withJavaWriteMaxParallelism`, and the
   public utility class `org.apache.hudi.client.common.JavaParallelTaskRunner` 
in `hudi-java-client`. With the default
   nothing changes. With a value above 1, peak memory of a write grows with the 
number of file groups in flight (each
   holds its merge map, up to `hoodie.memory.merge.max.size`, the row group it 
is building, on merge-on-read tables the
   log block it is buffering, and with a queue-based write executor its queue 
and threads); the config doc says so.
   
   When a bucket fails, the stage throws only after every bucket task it 
started has returned. Within a bucket task,
   cleanup is the serial path's: with a queue-based write executor, 
`JavaLazyInsertIterable.computeNext` calls
   `shutdownNow()` and then `awaitTermination()`, which waits up to 60 seconds 
for the producer threads and then up to 60
   seconds for the consumer thread; the iterable does not check whether they 
ended, and if `shutdownNow()` throws, the
   executors are not shut down and nothing is awaited. This PR does not change 
that code, so it does not guarantee that
   those nested threads have ended when the exception reaches the caller. The 
failure tests below cover a merge failure
   and a queue-based insert failure, where they have.
   
   Local timing, to show the effect of the setting on one machine (not a 
benchmark): a shared workstation (AMD Ryzen 5
   7600, 6 cores), the build container limited to 4 CPUs (`docker --cpus=4`), 
local NVMe disk, JDK 11 with the module's
   test JVM settings (`-Xmx4g`, parallel GC). Each run takes a fresh copy of a 
copy-on-write table of 400,000 rows in 16
   partitions (file groups of up to 6,250 rows, metadata table off) and writes 
the same 160,000 records to it with
   `upsertPreppedRecords`, then commits: 80,000 updates spread over all 
existing file groups and 80,000 new records, one
   new file group per partition, so 80 or more file groups per write. Wall time 
covers the write and the commit. The four
   values ran in random order, one warm-up each, then 7 measured runs each.
   
   | `hoodie.java.write.max.parallelism` | Wall time, median (fastest - 
slowest) | Rows per second (median) | Speed-up | Peak heap after GC (median) |
   |---|---|---|---|---|
   | 1 | 7.30 s (7.21 - 7.49) | 21,924 | 1.00 | 683 MB |
   | 2 | 4.01 s (3.96 - 4.39) | 39,930 | 1.82 | 691 MB |
   | 4 | 2.30 s (2.23 - 2.45) | 69,595 | 3.17 | 708 MB |
   | 8 | 2.78 s (2.67 - 2.99) | 57,492 | 2.62 | 744 MB |
   
   Rows per second is 160,000 divided by the median wall time; speed-up is the 
median at 1 divided by the median at each
   value. Peak heap is the largest heap in use right after a garbage collection 
during the write, read from the
   collectors' after-collection figures every 10 ms; before each write it was 
575 MB (input records and test state), so
   the write added about 108 MB at 1 thread and 169 MB at 8.
   
   What these numbers show: with 4 CPUs a write took 55 % of the serial time at 
2 threads and 32 % at 4; at 8, above
   the CPU count, it was slower than at 4, so here a value above the available 
cores did not help. The file groups here
   are small, so the heap figures do not show the cost for large file groups, 
where the per-file-group terms in the
   config doc apply. An earlier run of the same measurement on an earlier 
version of this change gave the same picture
   (medians 7.22, 3.88, 2.19 and 2.71 s).
   
   ### Risk Level
   
   low. Opt-in; with the default the existing serial loop runs. With the option 
on, the tests below show the table, its
   files, commit metadata and timeline equal to the serial write, and a failing 
file group stops the write with the
   serial exception after the other bucket tasks have finished.
   
   Tests, run on master with JDK 11 and checkstyle on for main and test sources:
   - `TestJavaParallelTaskRunner` (16): order with out-of-order completion, 
empty and single input, at most N tasks at
     once, one thread per item below N; failure: the item running when the 
failure is recorded finishes before the call
     returns and no further item starts, and a thread held just before it takes 
an item until another thread's failure
     is recorded takes none (both coordinated on the recorded stop, no sleeps); 
one primary and the other failures
     suppressed; the same exception from two tasks thrown unchanged; errors 
unwrapped; checked failures wrapped; caller
     interrupt (waits for the running items, keeps the flag); a thread that 
cannot be created (the running item finishes
     first, the caller's interrupt flag is kept, the submission failure is 
thrown), and the same after a task failure
     (the submission failure is thrown with the task failure suppressed); 
context classloader; no thread left.
   - `TestHoodieJavaClientConcurrentWrites` (30):
     - concurrency measured inside the write (a test write status holds the 
first record of each file group until 4
       file groups are writing) for insert buckets, update buckets, 
merge-on-read log appends and a queue-based write
       executor;
     - parity against the serial write, where each step runs on a table at 
parallelism 1 and on a byte copy of it at
       parallelism 4 (so both start from the same file groups) and compares 
rows, file groups by record keys, base and log
       files per file group, commit metadata counts, completed instants and the 
partition metafile. The full sequence
       (insert, upsert, `insertPreppedRecords`, delete, bulk insert, plus 
`upsertPreppedRecords` and insert overwrite on
       copy-on-write) runs for copy-on-write and merge-on-read with the 
embedded timeline server on and off, and once with
       `BOUNDED_IN_MEMORY`; a smaller copy-on-write insert and upsert is 
repeated 10 times; and a rollover inside a bucket;
     - record errors counted as in the serial write;
     - failures: a corrupt base file failing an upsert, with direct and 
timeline-server markers, while a sibling file
       group is held until the failing file group's writer has started and the 
stage's other pool threads have exited,
       which they do only after the failure is recorded: same exception as the 
serial write, the sibling's file is
       complete when the call throws, no write thread is alive, every created 
file has a marker, and rollback restores
       the table; and an insert that fails inside a queue-based write executor, 
with the same sibling, thread, marker and
       rollback checks, plus no queue thread of the write left;
     - default and single-bucket writes stay on the calling thread; the 
metadata write config keeps 1; values below 1
       are refused.
   - Without the main code change (`BaseJavaCommitActionExecutor` as on master, 
everything else kept), 9 of the 30 fail
     in every rerun: the 6 concurrency tests (the gate times out, as file 
groups are written one at a time) and the 3
     failure tests (the failing file group cannot run while the sibling is 
still being written); with it all pass.
   - The claim test fails against a variant of the runner that checks the stop 
flag and then takes the next index in
     two steps (the held thread has passed the check, so it takes an item after 
the failure is recorded); with the one
     lock it passes.
   - Existing suites with the default pass: 
`TestJavaCopyOnWriteActionExecutor`, `TestHoodieJavaClientOnCopyOnWriteStorage`,
     `TestHoodieJavaClientOnCopyOnWriteStorageForTestFormat`, 
`TestHoodieJavaClientOnMergeOnReadStorage`,
     `TestJavaHoodieBackedMetadata`, `TestHoodieWriteConfig`.
   
   ### Documentation Update
   
   The new config is documented in its `withDocumentation` text; the 
configurations page is generated from it.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to