zymap opened a new pull request #12123: URL: https://github.com/apache/pulsar/pull/12123
--- *Motivation* We found the BlobStoreBackedReadHandler enter an infinite loop when reading a offload ledger. We saw in the heap dump, there has two ledger 1 and 2, and we have a consumer is reading ledger 1, but the heap shows the buffer it reading is ledger 2. Then the read handler read a wrong entry id and the entry id is out of range between the firstEntryId and the lastEntryId, it will try to keep seeking to the right position, so that it enters an infinite loop. The buffer in the `BlobStoreBackedInputStreamImpl` is a wrong buffer, so the read handler can not read a correct entry from it. After investigating, the buffer used in the `BlobStoreBackedInputStreamImpl` is allocated from `PulsarByteBufAllocator.DEFAULT.buffer(bufferSize, bufferSize)`, by default, we used a `PooledByteBufAllocator.DEFAULT` to allocate memory for that buffer, a weird thing we found is the closed buffer still can read data if there has new buffer allocated and write things. This is the test code: ``` ByteBuf buf = PulsarByteBufAllocator.DEFAULT.buffer(1024, 1024); buf.writeByte(1); System.out.println(buf.readByte()); // print 1 buf.release(); //System.out.println(buf.readByte()); // will throw exception ByteBuf newBuf = PulsarByteBufAllocator.DEFAULT.buffer(1024, 1024); newBuf.writeByte(2); System.out.println(buf.readByte()); // print 2 newBuf.release(); //System.out.println(buf.readByte()); ``` So we suspect there has a race condition between the read and close operation. That cause there has a thread start reading after the read handler closed. *Modifications* - Add a state check before reading entries. - exit loop when the entryID is bigger than the lastEntryID -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org