#general
@kennybastani: @kennybastani set the channel topic: Welcome! Show your support by starring our project repository on GitHub
@darek: @darek has joined the channel
@ddayan: @ddayan has joined the channel
@anirudhkrec: @anirudhkrec has joined the channel
#random
@darek: @darek has joined the channel
@ddayan: @ddayan has joined the channel
@anirudhkrec: @anirudhkrec has joined the channel
#troubleshooting
@ddayan: @ddayan has joined the channel
@anirudhkrec: @anirudhkrec has joined the channel
@ken: We’ve run into an issue with loading segments on our server, where we need more direct memory when building inverted indexes. The stack trace looks like: ```Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:694) ~[?:1.8.0_275] at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[?:1.8.0_275] at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_275] at org.apache.pinot.core.segment.memory.PinotByteBuffer.allocateDirect(PinotByteBuffer.java:38) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at org.apache.pinot.core.segment.memory.PinotDataBuffer.allocateDirect(PinotDataBuffer.java:116) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at org.apache.pinot.core.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator.createTempBuffer(OffHeapBitmapInvertedIndexCreator.java:254) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at org.apache.pinot.core.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator.seal(OffHeapBitmapInvertedIndexCreator.java:152) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at org.apache.pinot.core.segment.index.loader.invertedindex.InvertedIndexHandler.createInvertedIndexForColumn(InvertedIndexHandler.java:125) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at org.apache.pinot.core.segment.index.loader.invertedindex.InvertedIndexHandler.createInvertedIndices(InvertedIndexHandler.java:73) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at org.apache.pinot.core.segment.index.loader.SegmentPreProcessor.process(SegmentPreProcessor.java:109) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:99) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at org.apache.pinot.core.data.manager.offline.OfflineTableDataManager.addSegment(OfflineTableDataManager.java:52) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOfflineSegment(HelixInstanceDataManager.java:122) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21] at org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.addOrReplaceOfflineSegment(SegmentFetcherAndLoader.java:116) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-bb646baceafcd9b849a1ecdec7a11203c7027e21]``` We’ve bumped direct memory to 5gb, but still hitting these exceptions. We can continue increasing (or create the inverted index while building the segments), but wanted to confirm that it’s expected to need a lot of direct memory like this, thanks.
@fx19880617: I feel we should have an option to make segment build with bounded memory size. Issue tracking:
@mayanks: Since your inv index size is huge, better to build offline
@mailtobuchi: Hey folks, a bunch of segments are in BAD state in Pinot and reloading all the segments from the UI didn’t help. Trying to debug it further but didn’t see any warnings or errors in the logs. Any hints on where to check? Any usual suspects?
@mayanks: Server log should definitely have the error upon loading
#presto-pinot-connector
@ddayan: @ddayan has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
