Mike Heffner created CASSANDRA-7385:
---------------------------------------
Summary: sstableloader OutOfMemoryError: Java heap space
Key: CASSANDRA-7385
URL: https://issues.apache.org/jira/browse/CASSANDRA-7385
Project: Cassandra
Issue Type: Bug
Components: Tools
Reporter: Mike Heffner
We hit the following exception with sstableloader while attempting to bulk load
about 100GB of SSTs. We are now employing this workaround before starting an
sstableloader run:
sed -i -e 's/-Xmx256M/-Xmx8G/g' /usr/bin/sstableloader
{code}
ERROR 19:25:45,060 Error in ThreadPoolExecutor
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.io.util.FastByteArrayOutputStream.expand(FastByteArrayOutputStream.java:104)
at
org.apache.cassandra.io.util.FastByteArrayOutputStream.write(FastByteArrayOutputStream.java:235)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:199)
at
org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:412)
at
org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:407)
at
org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:59)
at
org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:46)
at
org.apache.cassandra.streaming.PendingFile$PendingFileSerializer.serialize(PendingFile.java:142)
at
org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.serialize(StreamHeader.java:67)
at
org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.serialize(StreamHeader.java:58)
at
org.apache.cassandra.net.MessagingService.constructStreamHeader(MessagingService.java:782)
at
org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:65)
at
org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "Streaming to /10.167.a.b:1" java.lang.OutOfMemoryError:
Java heap space
at
org.apache.cassandra.io.util.FastByteArrayOutputStream.expand(FastByteArrayOutputStream.java:104)
at
org.apache.cassandra.io.util.FastByteArrayOutputStream.write(FastByteArrayOutputStream.java:235)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:199)
at
org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:412)
at
org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:407)
at
org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:59)
at
org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:46)
at
org.apache.cassandra.streaming.PendingFile$PendingFileSerializer.serialize(PendingFile.java:142)
at
org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.serialize(StreamHeader.java:67)
at
org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.serialize(StreamHeader.java:58)
at
org.apache.cassandra.net.MessagingService.constructStreamHeader(MessagingService.java:782)
at
org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:65)
at
org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
exceeded
at
org.apache.cassandra.io.compress.CompressionMetadata.getChunksForSections(CompressionMetadata.java:210)
at
org.apache.cassandra.streaming.StreamOut.createPendingFiles(StreamOut.java:182)
at
org.apache.cassandra.streaming.StreamOut.transferSSTables(StreamOut.java:157)
at
org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:145)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:67)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
