[
https://issues.apache.org/jira/browse/FLUME-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401866#comment-13401866
]
Juhani Connolly commented on FLUME-1326:
----------------------------------------
We still run the vm with the standard 20M limit.
conf:
receiver1.channels.ch1.type = memory
receiver1.channels.ch1.capacity = 5000
receiver1.channels.ch1.transactionCapacity = 500
receiver1.channels.ch1.checkpointDir = /tmp/flume-check
receiver1.channels.ch1.dataDirs = /tmp/flume-data
receiver1.sources.avro.type = avro
receiver1.sources.avro.bind = <address>
receiver1.sources.avro.port = 41414
receiver1.sources.avro.channels = ch1
receiver1.sinks.hdfs-sink.type = hdfs
receiver1.sinks.hdfs-sink.namenode = hdfs://pat-nn01/
receiver1.sinks.hdfs-sink.hdfs.path = hdfs://pat-nn01/tmp/flume-load/%{host}
receiver1.sinks.hdfs-sink.hdfs.writeFormat = text
receiver1.sinks.hdfs-sink.hdfs.fileType = SequenceFile
receiver1.sinks.hdfs-sink.channel = ch1
receiver1.sinks.hdfs-sink.runner.type = polling
receiver1.sinks.hdfs-sink.runner.polling.interval = 60
receiver1.sinks.hdfs-sink.hdfs.rollSize = 0
receiver1.sinks.hdfs-sink.hdfs.rollInterval = 4000
receiver1.sinks.hdfs-sink.hdfs.filePrefix = %Y-%m-%d-%H
receiver1.sinks.hdfs-sink.hdfs.rollCount = 0
receiver1.sinks.hdfs-sink.hdfs.batchSize = 200
receiver1.sources = avro
receiver1.sinks = hdfs-sink
receiver1.channels = ch1
> OutOfMemoryError in HDFSSink
> ----------------------------
>
> Key: FLUME-1326
> URL: https://issues.apache.org/jira/browse/FLUME-1326
> Project: Flume
> Issue Type: Bug
> Affects Versions: v1.2.0
> Reporter: Juhani Connolly
> Priority: Critical
>
> We run a 3 node/1 collector test cluster pushing about 350events/sec per
> node... Not really high stress, but just something to evaluate flume with.
> Consistently our collector has been dying because of an OOMError killing the
> SinkRunner after running for about 30-40 hours(seems pretty consistent as
> we've had it 3 times now).
> Suspected cause would be a memory leak somewhere in HdfsSink. The feeder
> nodes which run AvroSink instead of HdfsSink have been up and running for
> about a week without restarts.
> flume-load/act-wap02/2012-06-26-17.1340697637324.tmp, packetSize=65557,
> chunksPerPacket=127, bytesCurBlock=29731328
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2012-06-26 17:12:56,080 (SinkRunner-PollingRunner-DefaultSinkProcessor)
> [ERROR -
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:411)]
> process failed
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOfRange(Arrays.java:3209)
> at java.lang.String.<init>(String.java:215)
> at java.lang.StringBuilder.toString(StringBuilder.java:430)
> at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:306)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:367)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:619)
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOfRange(Arrays.java:3209)
> at java.lang.String.<init>(String.java:215)
> at java.lang.StringBuilder.toString(StringBuilder.java:430)
> at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:306)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:367)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
