andygrove commented on issue #7305: URL: https://github.com/apache/datafusion-comet/issues/7305#issuecomment-6479333637
Root cause found and reproduced without Spark. **Cause.** Arrow 18.3.0's `ZstdCompressionCodec.doCompress` allocates `compressBound(n) + 8` bytes and points zstd 8 bytes in, past the uncompressed-length prefix, but still passes `compressBound(n) + 8` as the destination capacity. zstd guarantees that it never writes past the capacity it is given, not that it never writes past `compressBound`: while it entropy-codes the literals of a block that turns out not to compress, it fills the destination up to that capacity before it falls back to a stored block. The attempt can write up to 8 bytes past the buffer. This is lead 1 above, and it is not latent. **Why it is rare.** The Arrow allocator rounds requests up to a power of two, so the overrun lands in slack unless `compressBound(n) + 8` is within 8 bytes below a power of two. For 512 that is n = 432..440, and `compressBound(437) + 8 = 509`, allocated as 512, leaves 3 bytes of slack. Without NMT the extra bytes land in allocator padding or in whatever sits next, with no error; with NMT the footer canary sits right after the buffer and the JVM aborts when the codec frees the discarded oversize output, which is also why the block has a zstd frame but no length prefix. A 437-byte buffer is consistent with the validity bitmap of a 3496-row batch, which fits a wide nullable table. **Reproducer.** A ~60 line Java program that hands Arrow's codec 437-byte protobuf-like inputs (random bytes with a scatter of repeated tag bytes) under `-XX:NativeMemoryTracking=detail` aborts in about a second with the same message, `NMT corruption: Block at ...: footer canary broken at ... (buffer overflow?)`, from `AbstractCompressionCodec.compress` -> `BufferLedger.release` -> `Unsafe.freeMemory0`. With the destination capacity corrected to `compressBound(n)`, 24000 attempts over the exposed lengths run clean. This matches the report that the job passes with `lz4`, whose codec is Comet's own and never calls zstd. Arrow fixed the capacity in apache/arrow-java#1119 (reported as apache/arrow-java#1116) for Arrow 20.0.0, which is not released. #7307 carries that fix in Comet's cache codec. The seven local shapes did not reproduce it because the exposed buffer sizes are narrow and the overrun is silent without NMT; the standalone reproducer removes both obstacles. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
