QCLyu commented on issue #11605: URL: https://github.com/apache/gluten/issues/11605#issuecomment-5324152812
Hi there, could someone help validate this plan? Thinking of breaking down this issue into 3 PRs, which might take multiple weeks to finish. Is there any deadline or release schedule affected? Plan: PR1 Writer stats (foundation) Goal: Emit optional kStatisticsPayload before each plain payload. Compute per-column min/max/hasNull from buffers in VeloxHashShuffleWriter::evictBuffers() Serialize compact header; gate with config (default off) Unit tests: round-trip write/read of stats; dictionary path unchanged Must include: TPC-H/TPC-DS shuffle byte overhead numbers (related to why #11769 stalled) Deliverable: stats on disk, readers ignore unknown/new block type safely or only write when flag on. PR2 Reader block skip Goal: If stats say no rows can match filters → seek past block (no decompress). Parse stats in VeloxHashShuffleReaderDeserializer::resolveNextBlockType() Evaluate with Velox Filter::testInt64Range / testBytesRange / etc. Metric: numShuffleBlocksSkipped (or similar) Unit tests with synthetic colliding ranges / non-overlapping filters PR3 Wire filters end-to-end Goal: Push filters from ValueStreamDataSource into the shuffle deserializer. Propagate dynamicFilters_ down through RowVectorStream / iterator → reader Keep ValueStream row filtering as fallback for types/blocks that can’t be pruned Feature flag; default off until TPC join queries show clear I/O/CPU win -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
