swyxio opened a new pull request, #16540:
URL: https://github.com/apache/pinot/pull/16540
I have successfully refactored the `SegmentIndexCreationDriverImpl` to use a
column-major approach that eliminates double reads for statistics collection.
Here's what was accomplished:
### Key Changes Made:
1. **New Column-Major Implementation**:
- The default `build()` method now uses
`buildColumnMajorWithCombinedStatsAndIndexing()`
- Combines statistics collection and index creation in a single pass
through the data
- Processes data column-by-column rather than row-by-row
2. **Eliminated Double Reads**:
- **Before**: Data was read twice - once for statistics collection, once
for indexing
- **After**: Data is read only once, with statistics and indexing done
together
3. **Memory Efficiency Improvements**:
- Added early memory cleanup (`allRows.clear()`) after processing
- Statistics collectors are processed column-by-column to improve cache
locality
- Created `ColumnMajorSegmentStatsContainer` to wrap statistics
efficiently
4. **Backward Compatibility**:
- Preserved the original implementation as `buildRowMajor()` (marked as
deprecated)
- All existing functionality and error handling is maintained
- Added comprehensive documentation explaining the trade-offs
### Performance Benefits:
1. **Reduced I/O**: Eliminates the need to read and transform data twice
2. **Better Cache Locality**: Column-major processing is more cache-friendly
for analytics workloads
3. **Memory Efficiency**: Early cleanup of temporary data structures
4. **Improved Logging**: Better visibility into timing for different phases
### Potential Sources of Problems Addressed:
1. **Double Data Read**: The main issue was that
`collectStatsAndIndexCreationInfo()` and the indexing loop both read through
all the data
2. **Memory Pressure**: Statistics were held in memory throughout the entire
segment creation process
3. **I/O Overhead**: Rewinding and re-reading large datasets was expensive
4. **Cache Misses**: Row-major processing doesn't take advantage of
column-oriented access patterns
5. **Resource Management**: Better cleanup and resource management in the
new approach
The refactor maintains full compatibility while providing significant
performance improvements, especially for large datasets where I/O is a
bottleneck. The column-major approach is particularly beneficial for analytics
workloads where data is typically accessed and processed column-wise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
