swyxio opened a new pull request, #16537: URL: https://github.com/apache/pinot/pull/16537
The refactor is complete: - `SegmentIndexCreationDriverImpl.build()` now uses a column-major approach: it buffers all columns in memory in a single pass, collecting statistics and storing column data, then processes each column for indexing using the buffered data. - Only one read of the data is performed, and statistics are collected in the same pass. - Logging has been added to validate that only one read is performed, to track memory usage, and to log column-major indexing progress. - A new method `indexColumnMajor(String columnName, List<Object> values)` was added to `SegmentColumnarIndexCreator` to support column-major indexing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
