sunchao commented on PR #7258: URL: https://github.com/apache/datafusion-comet/pull/7258#issuecomment-6481892539
@viirya, split the shared serializer work into #7310 (`2660acf816`) and removed `Utils.scala`/`UtilsSuite.scala` from this PR's diff. It includes the lazy per-iterator codec reuse and a regression that actually exercises underestimated small chunks. #7310 can land first, independently of the opt-in batching change. I refreshed both serializer baselines and both batching orders using the same Java/native artifacts apart from the `Utils` override. With #7310 applied, the one-row producer case measures 164.3–187.1 ms OFF versus 15.83–17.17 ms ON, with 8,192 → 8 streams. The complete-query means are OFF/ON/OFF = 856.4/630.4/694.4 ms and ON/OFF/ON = 769.6/764.2/535.4 ms. The initial reverse-order ON arm is effectively neutral; I removed the earlier blanket query-speedup characterization. The wide-string control also regresses, so there is no general or wide-row speedup claim. All eight fresh JVM runs returned the expected values/counts, with unchanged source and artifact hashes. Full methodology, controls and local alternate-lock limitation are in the descriptions. #7310 separately reports the shared serializer improvement with batching OFF in both arms. The cleanup from @comphead's pass is pushed in `48e1024bfc`, with the bitmap-padding rationale left for discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
