andygrove commented on PR #5909: URL: https://github.com/apache/datafusion-comet/pull/5909#issuecomment-5832262263
Closing as superseded by #5809, which landed the same fix on main: blocks are decoded message by message against a schema cache keyed on the raw schema message bytes, so a hit is one memcmp and the schema is not re-verified or re-parsed. The cache lives per thread inside `read_ipc_compressed` and `read_ipc_compressed_validated`, so `ShuffleScanExec`, the local `decodeShuffleBlock` entry point and the remote decoder handle all use it without a decoder being passed through them, which was the rest of this PR. #6098 has since capped how much the cache holds per thread. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
