FrankChen021 commented on code in PR #20173:
URL: https://github.com/apache/druid/pull/20173#discussion_r3881085105
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/PartialSegmentMergeTask.java:
##########
@@ -295,13 +296,17 @@ private DataSegmentsWithSchemas mergeAndPushSegments(
), "version for interval[%s]", interval),
0
);
+ try (QueryableIndex index =
toolbox.getIndexIO().loadIndex(mergedFileAndDimensionNames.lhs)) {
Review Comment:
[P2] Avoid loading the full segment to read its row count
IndexIO.loadIndex(File) defaults to lazy=false. For v9 and legacy segments
this eagerly builds the column holders, and for v10 it maps all containers and
eagerly deserializes every column before getNumRows() is called. This call only
needs the row count, so a large merged segment can add substantial heap/native
memory pressure (and potentially OOM) during publishing. Read the row count
through a format-specific metadata-only path instead: v10 has
baseProjection.getNumRows(), while v9/legacy can extract totalSize from the
__time column header. Passing lazy=true would reduce eager deserialization but
still maps the segment buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
