Joe Crobak created FLUME-1224:
---------------------------------
Summary: with jdbc channel and non-existant downstream agent,
flume agent quickly OOME's
Key: FLUME-1224
URL: https://issues.apache.org/jira/browse/FLUME-1224
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.1.0
Reporter: Joe Crobak
Also tested on trunk.
With this sample configuration:
{noformat}
agent.sources = thriftLegacy
agent.sinks = avroForward
agent.channels = jdbcChannel
#agent.channels = recMemChannel
# channel configs
agent.channels.jdbcChannel.type = jdbc
agent.channels.recMemChannel.type =
org.apache.flume.channel.recoverable.memory.RecoverableMemoryChannel
agent.channels.recMemChannel.capacity = 5000
agent.sources.thriftLegacy.channels = jdbcChannel
agent.sinks.avroForward.channel = jdbcChannel
#agent.sources.thriftLegacy.channels = recMemChannel
#agent.sinks.avroForward.channel = recMemChannel
agent.sources.thriftLegacy.type =
org.apache.flume.source.thriftLegacy.ThriftLegacySource
agent.sources.thriftLegacy.host = localhost
agent.sources.thriftLegacy.port = 9108
agent.sinks.avroForward.type = avro
agent.sinks.avroForward.hostname = localhost
agent.sinks.avroForward.port = 9109
{noformat}
The agent OOME's on startup. I can up it's heap, and then it just takes a bit
longer to OOME on startup. The eventual backtrace is:
{noformat}
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:632)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:97)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at
org.jboss.netty.channel.socket.nio.SocketSendBufferPool$Preallocation.<init>(SocketSendBufferPool.java:156)
at
org.jboss.netty.channel.socket.nio.SocketSendBufferPool.<init>(SocketSendBufferPool.java:43)
at
org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:84)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.<init>(NioClientSocketPipelineSink.java:74)
at
org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:135)
at
org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:105)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:116)
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:120)
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:109)
at
org.apache.flume.api.NettyAvroRpcClient.<init>(NettyAvroRpcClient.java:94)
at
org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:131)
at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:148)
at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:176)
at org.apache.flume.sink.AvroSink.process(AvroSink.java:226)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:680)
{noformat}
which suggests that this is related to
https://issues.jboss.org/browse/NETTY-424?_sscc=t
During this time, lots and lots of ERRORs are logged. It doesn't seem that the
sleep/retry logic is working properly. Here are some example log entries:
{noformat}
2012-05-22 18:10:42,291 INFO sink.AvroSink: Avro sink starting
2012-05-22 18:10:42,377 ERROR api.NettyAvroRpcClient: RPC connection error :
java.io.IOException: Error connecting to localhost:9109
at
org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:250)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:199)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:148)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:116)
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:120)
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:109)
at
org.apache.flume.api.NettyAvroRpcClient.<init>(NettyAvroRpcClient.java:94)
at
org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:131)
at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:148)
at org.apache.flume.sink.AvroSink.start(AvroSink.java:188)
at
org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:228)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
... 3 more
2012-05-22 18:10:42,379 ERROR sink.AvroSink: Unable to create avro client using
hostname:localhost, port:9109, batchSize: 100. Exception follows.
org.apache.flume.FlumeException: RPC connection error. Exception follows.
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:126)
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:109)
at
org.apache.flume.api.NettyAvroRpcClient.<init>(NettyAvroRpcClient.java:94)
at
org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:131)
at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:148)
at org.apache.flume.sink.AvroSink.start(AvroSink.java:188)
at
org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:228)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.io.IOException: Error connecting to localhost:9109
at
org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:250)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:199)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:148)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:116)
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:120)
... 17 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
... 3 more
2012-05-22 18:10:42,394 ERROR api.NettyAvroRpcClient: RPC connection error :
java.io.IOException: Error connecting to localhost:9109
at
org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:250)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:199)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:148)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:116)
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:120)
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:109)
at
org.apache.flume.api.NettyAvroRpcClient.<init>(NettyAvroRpcClient.java:94)
at
org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:131)
at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:148)
at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:176)
at org.apache.flume.sink.AvroSink.process(AvroSink.java:226)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
... 1 more
2012-05-22 18:10:42,395 WARN impl.JdbcTransactionImpl: Marking transaction for
rollback
2012-05-22 18:10:42,395 INFO impl.JdbcTransactionImpl: Attempting transaction
roll-back
2012-05-22 18:10:42,395 ERROR flume.SinkRunner: Unable to deliver event.
Exception follows.
org.apache.flume.EventDeliveryException: RPC connection error. Exception
follows.
at org.apache.flume.sink.AvroSink.process(AvroSink.java:264)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.flume.FlumeException: RPC connection error. Exception
follows.
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:126)
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:109)
at
org.apache.flume.api.NettyAvroRpcClient.<init>(NettyAvroRpcClient.java:94)
at
org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:131)
at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:148)
at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:176)
at org.apache.flume.sink.AvroSink.process(AvroSink.java:226)
... 3 more
Caused by: java.io.IOException: Error connecting to localhost:9109
at
org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:250)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:199)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:148)
at
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:116)
at
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:120)
... 9 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
... 1 more
{noformat}
There is about 15ms between each of these log entries until the app OOMEs. Note
that in my config, I've tried the commented out RecoverableMemoryChannel. When
I do that, I do not experience the bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
