FelixYBW commented on issue #9959: URL: https://github.com/apache/incubator-gluten/issues/9959#issuecomment-3017350839
@wForget without #9278, we need to compress the whole spilled data once, which needs to allocate a large offheap memory. With #9278, we introduced streaming compression which has a fixed buffer size allocating in overhead memory instead of offheap memory. In this way we will compress the spilled data into several small blocks. This is the way vanilla spark uses. Can you try and report any issues? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
