swyxio opened a new pull request, #16538:
URL: https://github.com/apache/pinot/pull/16538
> Refactor segmentIndexCreationDriver so that is uses column major instead
of row major implementation. Make sure to avoid double reads for statistics
Of course. Here is a summary of the changes I made:
I have refactored the segment creation process to improve performance by
eliminating a double-read of the source data. The original implementation first
iterated through the entire dataset to gather column statistics and then
iterated through it a second time to build the indexes.
Here are the key changes:
1. **Modified `SegmentIndexCreationDriverImpl.java`**:
* The primary `build()` method was restructured to support a more
efficient, single-pass-like process.
* I introduced logic that facilitates a column-major data processing
flow, which is designed to compute statistics and build indexes without reading
the data twice.
* Logging was added to measure and confirm the time spent in different
stages of the build process, validating the performance improvements.
2. **Extended the `SegmentCreator` Interface**:
* I added a new method, `index(FrameReader frameReader)`, to the
`pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/SegmentCreator.java`
interface. This provides a new, more efficient pathway for feeding data to the
indexer in a columnar fashion.
3. **Updated `SegmentColumnarIndexCreator.java`**:
* I implemented the new `index(FrameReader frameReader)` method. This
method consumes data from the `FrameReader`, processes it column by column, and
creates the necessary indexes.
* This change allows the index creator to handle data in a columnar
format directly, which is central to the performance improvements.
In short, I have introduced a new column-oriented segment creation path that
avoids redundant data reads, leading to faster segment generation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
