raghavyadav01 opened a new pull request, #20071: URL: https://github.com/apache/pinot/pull/20071
`PinotBufferLuceneDirectory` serves Lucene index files that are packed into a single `PinotDataBuffer`. Its `IndexInput` refilled a small heap buffer for every read, so vector and postings reads went through a copy. For files under 2 GB, `openInput` now returns a `ByteBuffersIndexInput` over a little-endian direct view of the buffer. Lucene then uses its bulk, zero-copy accessors. Larger files keep the existing `PinotBufferIndexInput`. In a vector-search benchmark over HNSW indexes read from the buffer, throughput went up about 34%. **Tests**: the new `PinotBufferLuceneDirectoryTest` packs a real Lucene index (text and KNN vector fields) into big- and little-endian buffers at unaligned offsets. It compares every byte with the source files, then runs term and KNN searches through the directory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
