[
https://issues.apache.org/jira/browse/FLUME-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15778271#comment-15778271
]
ASF GitHub Bot commented on FLUME-2984:
---------------------------------------
GitHub user laxman-ch opened a pull request:
https://github.com/apache/flume/pull/102
FLUME-2984: Fix OOM in thrift source. Make transport buffer configurable.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/laxman-ch/flume FLUME-2984
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flume/pull/102.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #102
----
commit e044261f783b01c43b9721d29a1dd3d418156762
Author: laxman <laxman@ubuntu>
Date: 2016-08-31T22:39:47Z
KAFKA-399: Fix OOM in thrift source. Make transport buffer configurable
commit cd5851ee0f0d494422427b49fa9c2fe694a2b05e
Author: laxman <laxman@ubuntu>
Date: 2016-12-26T11:25:15Z
FLUME-2405: Update thrift source documentation with new configuration
commit 7ef2d084de3d02f2bd32e1e4566696c8dbfcbb59
Author: laxman <laxman@ubuntu>
Date: 2016-12-26T11:39:14Z
FLUME-2984: Fix formatting for checkstyle
----
> Default maxReadBufferBytes might cause OOM and cause thrift source exit
> -----------------------------------------------------------------------
>
> Key: FLUME-2984
> URL: https://issues.apache.org/jira/browse/FLUME-2984
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.6.0, v1.7.0
> Reporter: Laxman
> Assignee: Laxman
> Attachments: FLUME-2984.patch
>
>
> Our usecase has multi agent flow and we are achieving this with thrift source
> and thrift sink combination. When there is a surge in incoming traffic
> downstream agent having thrift source is running out of memory. On further
> investigation of thrift transport/server code and scribe source found that we
> may need to fix the thrift transport's maxReadBufferBytes. Similar issue has
> been fixed in ScribeSource as part of FLUME-2404 by making this buffer
> configurable and capping it to a 16MB if not configured.
> *thrift code (org.apache.thrift.server.AbstractNonblockingServer)*
> {code}
> public abstract static class AbstractNonblockingServerArgs<T extends
> AbstractNonblockingServer.AbstractNonblockingServerArgs<T>> extends
> AbstractServerArgs<T> {
> public long maxReadBufferBytes = 9223372036854775807L;
> public AbstractNonblockingServerArgs(TNonblockingServerTransport
> transport) {
> super(transport);
> this.transportFactory(new Factory());
> }
> }
> {code}
> *stacktrace*
> {noformat}
> 31 Aug 2016 12:52:24 ERROR [Thread-12] - run() exiting due to uncaught error
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
> at
> org.apache.thrift.server.TThreadedSelectorServer.requestInvoke(TThreadedSelectorServer.java:310)
> at
> org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:209)
> at
> org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.select(TThreadedSelectorServer.java:576)
> at
> org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run(TThreadedSelectorServer.java:536)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
