[
https://issues.apache.org/jira/browse/FLUME-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410737#comment-13410737
]
Mark Stern commented on FLUME-1224:
-----------------------------------
There is a workaround. Use a failover sink processor instead of a default sink
processor. This works even if there is only one sink.
> with jdbc channel and non-existant downstream agent, flume agent quickly
> OOME's
> -------------------------------------------------------------------------------
>
> Key: FLUME-1224
> URL: https://issues.apache.org/jira/browse/FLUME-1224
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Joe Crobak
>
> Also tested on trunk.
> With this sample configuration:
> {noformat}
> agent.sources = thriftLegacy
> agent.sinks = avroForward
> agent.channels = jdbcChannel
> #agent.channels = recMemChannel
> # channel configs
> agent.channels.jdbcChannel.type = jdbc
> agent.channels.recMemChannel.type =
> org.apache.flume.channel.recoverable.memory.RecoverableMemoryChannel
> agent.channels.recMemChannel.capacity = 5000
> agent.sources.thriftLegacy.channels = jdbcChannel
> agent.sinks.avroForward.channel = jdbcChannel
> #agent.sources.thriftLegacy.channels = recMemChannel
> #agent.sinks.avroForward.channel = recMemChannel
> agent.sources.thriftLegacy.type =
> org.apache.flume.source.thriftLegacy.ThriftLegacySource
> agent.sources.thriftLegacy.host = localhost
> agent.sources.thriftLegacy.port = 9108
> agent.sinks.avroForward.type = avro
> agent.sinks.avroForward.hostname = localhost
> agent.sinks.avroForward.port = 9109
> {noformat}
> The agent OOME's on startup. I can up it's heap, and then it just takes a bit
> longer to OOME on startup. The eventual backtrace is:
> {noformat}
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:632)
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:97)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
> at
> org.jboss.netty.channel.socket.nio.SocketSendBufferPool$Preallocation.<init>(SocketSendBufferPool.java:156)
> at
> org.jboss.netty.channel.socket.nio.SocketSendBufferPool.<init>(SocketSendBufferPool.java:43)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:84)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.<init>(NioClientSocketPipelineSink.java:74)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:135)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:105)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:116)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:120)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:109)
> at
> org.apache.flume.api.NettyAvroRpcClient.<init>(NettyAvroRpcClient.java:94)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:131)
> at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:148)
> at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:176)
> at org.apache.flume.sink.AvroSink.process(AvroSink.java:226)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:680)
> {noformat}
> which suggests that this is related to
> https://issues.jboss.org/browse/NETTY-424?_sscc=t
> During this time, lots and lots of ERRORs are logged. It doesn't seem that
> the sleep/retry logic is working properly. Here are some example log entries:
> {noformat}
> 2012-05-22 18:10:42,291 INFO sink.AvroSink: Avro sink starting
> 2012-05-22 18:10:42,377 ERROR api.NettyAvroRpcClient: RPC connection error :
> java.io.IOException: Error connecting to localhost:9109
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:250)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:199)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:148)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:116)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:120)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:109)
> at
> org.apache.flume.api.NettyAvroRpcClient.<init>(NettyAvroRpcClient.java:94)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:131)
> at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:148)
> at org.apache.flume.sink.AvroSink.start(AvroSink.java:188)
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:228)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
> ... 3 more
> 2012-05-22 18:10:42,379 ERROR sink.AvroSink: Unable to create avro client
> using hostname:localhost, port:9109, batchSize: 100. Exception follows.
> org.apache.flume.FlumeException: RPC connection error. Exception follows.
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:126)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:109)
> at
> org.apache.flume.api.NettyAvroRpcClient.<init>(NettyAvroRpcClient.java:94)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:131)
> at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:148)
> at org.apache.flume.sink.AvroSink.start(AvroSink.java:188)
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:228)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.io.IOException: Error connecting to localhost:9109
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:250)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:199)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:148)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:116)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:120)
> ... 17 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
> ... 3 more
> 2012-05-22 18:10:42,394 ERROR api.NettyAvroRpcClient: RPC connection error :
> java.io.IOException: Error connecting to localhost:9109
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:250)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:199)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:148)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:116)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:120)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:109)
> at
> org.apache.flume.api.NettyAvroRpcClient.<init>(NettyAvroRpcClient.java:94)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:131)
> at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:148)
> at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:176)
> at org.apache.flume.sink.AvroSink.process(AvroSink.java:226)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ... 1 more
> 2012-05-22 18:10:42,395 WARN impl.JdbcTransactionImpl: Marking transaction
> for rollback
> 2012-05-22 18:10:42,395 INFO impl.JdbcTransactionImpl: Attempting transaction
> roll-back
> 2012-05-22 18:10:42,395 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
> org.apache.flume.EventDeliveryException: RPC connection error. Exception
> follows.
> at org.apache.flume.sink.AvroSink.process(AvroSink.java:264)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: org.apache.flume.FlumeException: RPC connection error. Exception
> follows.
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:126)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:109)
> at
> org.apache.flume.api.NettyAvroRpcClient.<init>(NettyAvroRpcClient.java:94)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:131)
> at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:148)
> at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:176)
> at org.apache.flume.sink.AvroSink.process(AvroSink.java:226)
> ... 3 more
> Caused by: java.io.IOException: Error connecting to localhost:9109
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:250)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:199)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:148)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:116)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:120)
> ... 9 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ... 1 more
> {noformat}
> There is about 15ms between each of these log entries until the app OOMEs.
> Note that in my config, I've tried the commented out
> RecoverableMemoryChannel. When I do that, I do not experience the bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
