siddharthteotia opened a new issue, #9162: URL: https://github.com/apache/pinot/issues/9162
For off heap buffers (direct or mmap'd), we currently use Lbuffer library if the size is more than 2GB (which is supported for raw forward index by the v3 and v4 writer). We have seen production issues on 2 occasions where a race condition in the library (map followed by unmap and then map) can lead to SIGSEGV where the address the code is trying to access is 3GB beyond the total virtual address space mapped for Pinot segments on the host where crash was observed. Not easily reproducible but has happened twice in production. Also, the library is no longer being maintained by the original author so may be there is benefit in writing our own allocator for large buffers. TODO - add a link to the race condition issue and add more details from the production issues we observed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
