siddharthteotia commented on PR #18638: URL: https://github.com/apache/pinot/pull/18638#issuecomment-4671734661
Have we tested this in our production environment with PBNJ to exercise end to end build path successfully ? I ask because it's possible that memory footprint wise, this may be more than what we get out of row major Arrow record reader which handles one batch of vectors (columns) at a time in a row oriented fashion and keeps contsructing GenericRows. OTOH, this new reader materializes entire column's worth of data (from the file, one column at a time) but the peak memory it ends up using is materialization of all columns. Have we done any BOE calculations on the memory footprint of this ? The concern is unexpectedly making PBNJ more prone to OOM errors during build time. The current row major reader has default of 256MB (which may be enough for the usual batch based arrow vector record reader that) but with this PR, it's possible in theory to significantly exceed this default during build time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
