voonhous opened a new issue, #20629:
URL: https://github.com/apache/hudi/issues/20629

   Part of #20625. Line references are against master `db90a0da`.
   
   ### Problem
   When the dimension tables are MOR too, the build side of a join pays the 
slow per-split merge path. The date dynamic filter can then arrive just after 
`hudi.dynamic-filtering.wait-timeout` (default 1s), so the fact table splits 
are generated unpruned and timings swing between repeats.
   
   It shows locally only with a slow storage profile: `HudiMorReadBenchmark` 
dpp scenario with `-Dbench.s3.getMillis=60 -Dbench.s3.headMillis=40 
-Dbench.s3.listMillis=150`. At the default profile the filter arrives in 0.2 to 
0.4 s and pruning applies.
   
   ### Direction
   The estimator and per-split metadata items shrink the build-side latency; 
this item owns the end-to-end outcome. If those alone do not make pruning 
reliable, revisit the wait strategy for MOR, e.g. wait longer when the build 
side is selective, or wait on the partition-key filter specifically.
   
   ### Exit criteria
   - dpp scenario with the slow profile and `-Dbench.rounds=5`: `_rt` default 
reads the same fact input bytes as `_ro` default in every round, with no 
session overrides.
   - Date filter collection under 1s in that profile, or the connector 
demonstrably waits for it; a changed default is justified in the PR and 
documented in the config description.
   - No regression for non-DPP queries: scan wall time unchanged within noise.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to