LuciferYang opened a new pull request, #13549:
URL: https://github.com/apache/gluten/pull/13549

   ## What changes are proposed in this pull request?
   
   `RssPartitionWriter::doEvict` compressed the block before its `spillTime_` 
timer started and never added the payload's compression time to 
`compressTime_`, so RSS hash shuffle always reported zero compression time. 
This starts the timer at the top of `doEvict` and adds the compression time, 
the same way `sortEvict` already does. `stop()` subtracts `compressTime_` from 
`spillTime_`, so the push time still excludes compression.
   
   As a result, Uniffle's shuffle write time now includes compression, and the 
"time to split" metric of Celeborn and Uniffle no longer counts it. Celeborn's 
shuffle write time and SQL metrics have no compression entry, so there the 
compression time is now not shown anywhere, the same as for its other writers.
   
   Fixes #13543.
   
   ## How was this patch tested?
   
   Added `rssCompressTime` to `VeloxShuffleWriterTest`. It runs the hash 
shuffle writer with the RSS partition writer and a codec, and checks that 
`totalCompressTime()` is greater than 0. Before this change it was always 0.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Yes. Generated-by: Claude Opus.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to