jarohen commented on issue #1194: URL: https://github.com/apache/arrow-java/issues/1194#issuecomment-6290660015
Hey folks - we hit this in XTDB too, in a slightly worse form: when the empty list's offset buffer is allocator.getEmpty(), it's NettyAllocationManager.EMPTY_BUFFER, a single ArrowBuf shared across the whole JVM. `setReaderAndWriterIndex` then sets that shared buffer's writer index to 4, and every other vector holding the empty buffer sees it. In our case that's other threads' validity buffers mid-serialisation, which then fail with `IndexOutOfBoundsException: index: 0, length: 4 (expected: range(0, 0))` or go out with the wrong length. Under 8 concurrent Flight streams, roughly 1 in 7 batches came out corrupted; with one stream it never happens, because a later vector resets the index to 0 before anything serialises it. So whatever #1229 settles on for freshly constructed or cleared vectors, it'd be worth making sure the empty-vector path never sets a non-zero index on the allocator's empty buffer: allocate the 4-byte [0] instead, or leave the index alone when capacity is 0. See xtdb/xtdb#6231 for the write-up and workaround on our side. Cheers 🙂 James -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
