voonhous opened a new issue, #20627:
URL: https://github.com/apache/hudi/issues/20627

   Part of #20625. Line references are against master `db90a0da`.
   
   ### Problem
   `HudiTrinoReaderContext` extends `HoodieReaderContext` directly and inherits 
`getRecordSizeEstimator()` = `HoodieRecordSizeEstimator` and 
`getRecordSerializer()` = `DefaultSerializer` (Kryo). The estimator walks the 
whole record object graph, shared Avro schema included, on every 100th log 
record. The estimate therefore counts the schema per record, and the 1 GiB 
merge buffer spills far too early.
   
   Measured locally with `HudiMergeBufferBenchmark` (300k log records, JDK 25): 
estimate 13,304 B/record, 235,852 of 300,000 records spilled, put about 11 to 
14 us/record.
   
   ### Direction
   Override both in `HudiTrinoReaderContext`. `HoodieAvroReaderContext` already 
does this with `AvroRecordSizeEstimator` and `BufferedRecordSerializer` + 
`AvroRecordSerializer`. Preferably avoid the object-graph walk entirely, e.g. 
compute the size from the schema and value sizes.
   
   ### Exit criteria
   - `HudiMergeBufferBenchmark`: 0 records spilled at the default budget, put 
<= 1 us/record, estimate <= 2 KB/record.
   - At most one estimator walk per file group, pinned by a unit test that 
counts estimator calls on a multi-thousand-record buffer.
   - `HudiMorReadBenchmark` scan with `-Dbench.s3.latency.enabled=0`: `_rt` CPU 
<= 25% of master's.
   - Spill path still correct: a test with a tiny 
`hoodie.memory.merge.max.size` forces spilling and asserts the merged rows 
equal the in-memory case (`COMMIT_TIME_ORDERING` and `EVENT_TIME_ORDERING`, 
with deletes).
   - Existing hudi-trino tests green.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to