jihoonson opened a new issue #6338: OutOfMemory error while merging in AppenderatorImpl URL: https://github.com/apache/incubator-druid/issues/6338 `AppenderatorImpl` currently merges all persisted indexes at once. This can incur OOM error like below if there are too many indexes to be merged. ``` 2018-09-13T23:33:55,073 ERROR [task-runner-0-priority-0] io.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS. java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?] at io.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:1042) ~[druid-indexing-service-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:455) [druid-indexing-service-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:457) [druid-indexing-service-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:429) [druid-indexing-service-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?] at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?] at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?] at io.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAndClear(BatchAppenderatorDriver.java:141) ~[druid-server-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAllAndClear(BatchAppenderatorDriver.java:124) ~[druid-server-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:1004) ~[druid-indexing-service-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] ... 7 more Caused by: java.lang.OutOfMemoryError: Java heap space at it.unimi.dsi.fastutil.longs.LongArrays.grow(LongArrays.java:222) ~[fastutil-8.1.0.jar:?] at it.unimi.dsi.fastutil.longs.LongArrayList.grow(LongArrayList.java:277) ~[fastutil-8.1.0.jar:?] at it.unimi.dsi.fastutil.longs.LongArrayList.add(LongArrayList.java:296) ~[fastutil-8.1.0.jar:?] at io.druid.segment.data.IntermediateColumnarLongsSerializer.add(IntermediateColumnarLongsSerializer.java:89) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.segment.LongColumnSerializer.serialize(LongColumnSerializer.java:87) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.segment.IndexMergerV9.mergeIndexesAndWriteColumns(IndexMergerV9.java:458) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.segment.IndexMergerV9.makeIndexFiles(IndexMergerV9.java:209) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.segment.IndexMergerV9.merge(IndexMergerV9.java:837) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:710) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:688) ~[druid-processing-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:662) ~[druid-server-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$0(AppenderatorImpl.java:566) ~[druid-server-0.12.2-iap7-SNAPSHOT.jar:0.12.2-iap7-SNAPSHOT] at io.druid.segment.realtime.appenderator.AppenderatorImpl$$Lambda$117/371981722.apply(Unknown Source) ~[?:?] at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?] at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?] ... 3 more ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
