QCLyu commented on issue #11605: URL: https://github.com/apache/gluten/issues/11605#issuecomment-5336984114
Sure. Happy to run a quick PoC before we invest in the full design. [#11657](https://github.com/apache/gluten/pull/11657) already covers ValueStream-side filtering after deserialize. For this PoC I’ll treat that as the baseline (valueStream.dynamicFilter.enabled=true) and focus on the incremental win from skipping shuffle blocks before decompress/deserialize. @zhztheplayer Could you let me know if this plan looks good to you? 1. Use a selective BHJ (synthetic + optionally TPC-DS Q94, since that already showed benefit with ValueStream filtering). 2. Log per-block join-key min/max (and sizes) on the writer side — no shuffle format change yet. 3. Estimate % of probe shuffle blocks/bytes that the pushed dynamic filter would fully reject. 4. If that looks promising, try a temporary reader skip hack and report decompress/deserialize time + query runtime vs ValueStream-only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
