prashantwason opened a new issue, #20673:
URL: https://github.com/apache/hudi/issues/20673

   ### Task Description
   
   **What needs to be done:**
   
   Make record index (and other metadata table partition) compaction cheaper. A 
metadata table file group compaction reads the base HFile and its log files and 
rewrites the whole file group. On large record index file groups (tens of 
millions of entries), almost all of the time goes to rewriting entries that no 
log record touches.
   
   We profiled compaction (JFR, exclusive attribution) with a standalone 
benchmark that runs the real compaction code on two production record index 
file groups: Table A has 34M entries and a 1.9 GB base file; Table B has 105M 
entries and an 8.3 GB base file.
   
   | Component | Table A | Table B |
   |---|---|---|
   | Avro decode | 25.9% | 20.8% |
   | Avro encode | 21.8% | 16.7% |
   | gzip compression | 19.4% | 21.3% |
   | Bloom filter add | 16.6% | 19.9% |
   | Waiting on storage reads | 7.3% | 10.6% |
   | HFile internals | 3.5% | 5.3% |
   | Record key extraction | 2.1% | — |
   | Merge (file group reader, record buffer, wrappers) | 1–2% | 1–2% |
   | gzip decompression | 1.5% | 2.2% |
   
   The merge itself is almost free: more than 99.9% of base entries have no log 
record for their key. Removing the bloom filter is handled in #20666, which 
measured a 5–10% saving. That saving is lower than the profile share, so treat 
the estimates below as upper bounds until each is benchmarked.
   
   Proposed improvements:
   
   **1. Copy untouched entries as raw bytes (estimate 40–50%)**
   Every base value is decoded with `GenericDatumReader` and a 
`ResolvingDecoder`, using a schema that adds the `_hoodie_*` meta fields, and 
wrapped in a `HoodieRecord`. `HoodieAvroHFileWriter.writeAvro` then re-encodes 
it with `HoodieAvroUtils.avroToBytes`. Instead, a record-index merge handle 
could:
   - copy the stored bytes for keys with no log record, using 
`HFileWriterImpl.append(String, byte[])`, since the reader already holds the 
value bytes;
   - decode only the keys that have a log record, and merge them through 
`HoodieMetadataPayload`.
   
   Constraints:
   - The output file must keep the input file's Avro schema, and merged records 
must be encoded with it.
   - Deletes are not a problem: they never reach base files, and log deletes 
only hit keys that are decoded anyway.
   - Gate it behind a config, initially for the record index partition only.
   
   **2. Cheaper Avro decode and encode for entries that must be materialized 
(estimate 10–15% without #1)**
   - Reuse the `BinaryEncoder` and output buffer in the HFile writer instead of 
creating new ones per record.
   - Decode with the file's own schema instead of resolving to a schema with 
the meta fields added.
   - Avoid the per-record cost of union resolution: about 8% on Table A goes to 
union branch lookup, which hashes a String for every union field of every 
record.
   
   #1 makes most of this unnecessary for compaction, but it still helps 
log-block writes and the keys that do get merged.
   
   **3. Cheaper HFile block compression (up to ~20%)**
   `HoodieAirliftGzipCompressor` opens a new gzip stream for every 1 MB block. 
Options to benchmark:
   - Reuse the `Deflater` across blocks, and allow a lower gzip level. Both 
keep the file format unchanged.
   - Add zstd. `CompressionCodec.ZSTD` (id 6) exists, but 
`HoodieCompressorFactory.getCompressor` supports only `NONE` and `GZIP`, so 
`hoodie.hfile.compression.algorithm=ZSTD` fails today. aircompressor, already a 
dependency, ships a pure-Java `ZstdHadoopStreams` that mirrors the gzip 
implementation; zstd-jni is the faster native option.
     - zstd should decompress much faster, which also helps metadata table 
lookups.
     - Every reader must support zstd before any writer uses it, because older 
readers cannot open such files.
     - The metadata table builds its own storage config, so it would need a way 
to set the codec.
   - Copy whole compressed blocks that no log key touches. Measure how many 
blocks a typical compaction touches first: log keys spread across the key space 
may touch most blocks.
   
   After #1 and the bloom filter removal, compression is the largest remaining 
CPU cost, at about 40% of the remaining time.
   
   **4. Overlap storage reads with CPU (estimate 5–10% of wall time)**
   The compaction task waits on base file reads for 7–11% of its time. 
Prefetching the next blocks on a separate thread, or reading in larger chunks, 
would hide most of it. This cuts wall time, not CPU.
   
   **Why this task is needed:**
   
   Record index compaction is CPU-bound, and its cost grows with the size of 
the table rather than the size of the update: a few thousand changed keys force 
a rewrite of tens of millions of entries. Combined with #20666, #1 and #3 
should make record index compaction about 2× cheaper, which cuts executor time 
for compaction and the executors needed to meet a compaction deadline by the 
same factor.
   
   ### Task Type
   
   Performance optimization
   
   ### Related Issues
   
   **Parent feature issue:**
   **Related issues:** #20666 (skip writing metadata table HFile bloom filters 
unless bloom filter lookups are enabled), #18936 
(`hoodie.hfile.bloom.filter.enabled`)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to