Miklos Szurap created FLUME-3326:
------------------------------------
Summary: HDFSEventSink should log the exceptions before rolling
back
Key: FLUME-3326
URL: https://issues.apache.org/jira/browse/FLUME-3326
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Reporter: Miklos Szurap
Currently in the {{HDFSEventSink}} if the
[transaction.commit()|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L444]
fails then in the corresponding catch blocks for IOException and Throwable it
tries to roll back the transaction with {{transaction.rollback()}} and logs the
exception/throwable only after the rollback.
However the rollback also can fail (for example with
[FileChannel|https://github.com/apache/flume/blob/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java#L666]
in form of a ChannelException which is a RuntimeException) throwing the
exception one layer up and *hiding away the original cause*.
Flume should first log the error, then try the rollback.
In such cases we see just that the rollback failed. In a recent incident we
saw it with free disk space issues, however this can be more general.
{noformat}
2018-11-14 06:21:12,400 ERROR org.apache.flume.source.SyslogUDPSource: Error
writting to channel
org.apache.flume.ChannelException: Commit failed due to IO error
[channel=my-channel]
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:660)
at
org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
at
org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:272)
at
org.apache.flume.source.SyslogUDPSource$syslogHandler.messageReceived(SyslogUDPSource.java:90)
at
org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:52)
at
org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Usable space exhausted, only 467148800 bytes
remaining, required 524288026 bytes
at org.apache.flume.channel.file.Log.rollback(Log.java:744)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:658)
... 14 more
{noformat}
(line numbers in the above may differ due to different flume version)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
