hhr293 commented on PR #12299: URL: https://github.com/apache/gluten/pull/12299#issuecomment-4767237527
> related: #12220 Thanks for cross-linking. [#12299](https://github.com/apache/gluten/pull/12299) and [#12220](https://github.com/apache/gluten/pull/12220) converged on the same lo/hi split-lane FFOR design for INT128 independently, but differ structurally: - #[12220](https://github.com/apache/gluten/pull/12220) materializes both lanes in heap std::vector<uint64_t>(nValues) and encodes them as two concatenated FFOR streams with an int64 loCompLen prefix to demarcate. - #12299(https://github.com/apache/gluten/pull/12299) uses fixed-size stack scratch (uint64_t loBuffer[kMaxValuesPerBlock], ~16KB) and emits interleaved per-block (lo_block, hi_block, ...), where each FFOR block header self-describes its length — no body-level prefix. Working set is O(block_size) constant vs O(nValues × 16B); for shuffle batches with nValues ≥ 4K the latter spills out of L1 and re-reads on the encode passes. #12220 has the stronger error-handling discipline and broader codec coverage. cc @luis4a0 @guowangy @zhulipeng -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
