[
https://issues.apache.org/jira/browse/FLUME-798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135154#comment-13135154
]
Björn Edström commented on FLUME-798:
-------------------------------------
I'm experiencing the same issue, using the cloudera package
0.9.4+25.9-1~lenny-cdh3. Below is a stack trace that is similar but not
completely identical to the original above.
2011-10-25 11:38:42,346 INFO
com.cloudera.flume.handlers.endtoend.AckListener$Empty: Empty Ack Listener
ended 20111025-113813833+0000.1129353122525368.00000046
2011-10-25 11:38:42,347 INFO
com.cloudera.flume.agent.durability.NaiveFileWALManager: File lives in
/var/lib/flume/flume-flume/agent/machine-044.d.company.net/writing/20111025-113813833+0000.1129353122525368.00000046
2011-10-25 11:38:42,347 INFO com.cloudera.flume.handlers.hdfs.SeqfileEventSink:
constructed new seqfile event sink:
file=/var/lib/flume/flume-flume/agent/machine-044.d.company.net/writing/20111025-113842347+0000.1129381636287890.00000046
2011-10-25 11:38:41,797 ERROR com.cloudera.flume.core.connector.DirectDriver:
Closing down due to exception during append calls
java.lang.InterruptedException: Blocked append interrupted by rotation event
at
com.cloudera.flume.handlers.rolling.RollSink.append(RollSink.java:209)
at
com.cloudera.flume.agent.durability.NaiveFileWALDeco.append(NaiveFileWALDeco.java:132)
at com.cloudera.flume.agent.AgentSink.append(AgentSink.java:139)
at
com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
2011-10-25 11:38:42,348 INFO com.cloudera.flume.core.connector.DirectDriver:
Connector logicalNode machine-044.d.company.net-44 exited with error: Blocked
append interrupted by rotation event
java.lang.InterruptedException: Blocked append interrupted by rotation event
at
com.cloudera.flume.handlers.rolling.RollSink.append(RollSink.java:209)
at
com.cloudera.flume.agent.durability.NaiveFileWALDeco.append(NaiveFileWALDeco.java:132)
at com.cloudera.flume.agent.AgentSink.append(AgentSink.java:139)
at
com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
2011-10-25 11:38:42,348 ERROR com.cloudera.flume.core.connector.DirectDriver:
Driver interrupted attempting to close source
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1186)
at java.lang.Thread.join(Thread.java:1239)
at
com.spotify.flume.syslog2.ServerSocketSource.close(ServerSocketSource.java:121)
at
com.cloudera.flume.core.connector.DirectDriver$PumperThread.ensureClosed(DirectDriver.java:142)
at
com.cloudera.flume.core.connector.DirectDriver$PumperThread.errorCleanup(DirectDriver.java:163)
at
com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:116)
2011-10-25 11:38:42,348 INFO com.cloudera.flume.handlers.rolling.RollSink:
closing RollSink 'ackingWal'
2011-10-25 11:38:42,349 INFO
com.cloudera.flume.agent.durability.NaiveFileWALManager: opening log file
20111025-113813833+0000.1129353122525368.00000046
2011-10-25 11:38:42,350 INFO com.cloudera.flume.agent.WALAckManager: Ack for
20111025-113813833+0000.1129353122525368.00000046 is queued to be checked
2011-10-25 11:38:42,350 INFO com.cloudera.flume.agent.durability.WALSource: end
of file NaiveFileWALManager
(dir=/var/lib/flume/flume-flume/agent/machine-044.d.company.net )
2011-10-25 11:38:42,351 INFO
com.cloudera.flume.handlers.endtoend.AckListener$Empty: Empty Ack Listener
began 20111025-113842347+0000.1129381636287890.00000046
2011-10-25 11:38:42,352 INFO com.cloudera.flume.handlers.hdfs.SeqfileEventSink:
closed
/var/lib/flume/flume-flume/agent/machine-044.d.company.net/writing/20111025-113842347+0000.1129381636287890.00000046
2011-10-25 11:38:42,352 INFO
com.cloudera.flume.handlers.endtoend.AckListener$Empty: Empty Ack Listener
ended 20111025-113842347+0000.1129381636287890.00000046
2011-10-25 11:38:42,352 INFO
com.cloudera.flume.agent.durability.NaiveFileWALManager: File lives in
/var/lib/flume/flume-flume/agent/machine-044.d.company.net/writing/20111025-113842347+0000.1129381636287890.00000046
2011-10-25 11:38:42,352 INFO
com.cloudera.flume.agent.durability.NaiveFileWALManager: NaiveFileWALManager
shutting down
2011-10-25 11:38:42,352 INFO
com.cloudera.flume.agent.durability.NaiveFileWALManager: opening log file
20111025-113842347+0000.1129381636287890.00000046
2011-10-25 11:38:42,353 INFO com.cloudera.flume.agent.WALAckManager: Ack for
20111025-113842347+0000.1129381636287890.00000046 is queued to be checked
2011-10-25 11:38:42,354 INFO com.cloudera.flume.agent.durability.WALSource: end
of file NaiveFileWALManager
(dir=/var/lib/flume/flume-flume/agent/machine-044.d.company.net )
2011-10-25 11:38:42,554 WARN
com.cloudera.flume.agent.durability.NaiveFileWALManager: Already shutting down,
but getting another shutting down notice, odd
2011-10-25 11:38:42,768 INFO
com.cloudera.flume.agent.durability.NaiveFileWALManager: NaiveFileWALManager
shutting down
2011-10-25 11:38:42,775 INFO
com.cloudera.flume.handlers.thrift.ThriftEventSink: ThriftEventSink on port
35853 closed
2011-10-25 11:38:42,852 ERROR com.cloudera.flume.core.connector.DirectDriver:
Exiting driver logicalNode machine-044.d.company.net-44 in error state
SyslogSocketSource | Agent because Blocked append interrupted by rotation event
.. And here it just locks ..
> Blocked append interrupted by rotation event
> ---------------------------------------------
>
> Key: FLUME-798
> URL: https://issues.apache.org/jira/browse/FLUME-798
> Project: Flume
> Issue Type: Bug
> Components: Node
> Affects Versions: v0.9.5
> Reporter: Cameron Gandevia
>
> Our flume collector seem's to work for a short period of time and then fails
> with the following exception. When this happens the collector does not
> reconnect and the system becomes inactive with the processes still running.
> 2011-10-14 01:49:47,386 [logicalNode collector0_log_dir-115] ERROR
> com.cloudera.flume.core.connector.DirectDriver - Closing down due to
> exception during append calls
> 2011-10-14 01:49:47,387 [logicalNode collector0_log_dir-115] INFO
> com.cloudera.flume.core.connector.DirectDriver - Connector logicalNode
> collector0_log_dir-115 exited with error: Blocked append interrupted by
> rotation event
> java.lang.InterruptedException: Blocked append interrupted by rotation event
> at
> com.cloudera.flume.handlers.rolling.RollSink.append(RollSink.java:209)
> at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at com.cloudera.flume.core.MaskDecorator.append(MaskDecorator.java:43)
> at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.append(InsistentOpenDecorator.java:169)
> at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:71)
> at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
> at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
> at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
> at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
> at
> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
> at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
> com.cloudera.flume.core.extractors.DateExtractor.append(DateExtractor.java:129)
> at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
> at
> com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88)
> at
> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:133)
> 2011-10-14 01:49:47,388 [logicalNode collector0_log_dir-115] INFO
> com.cloudera.flume.collector.CollectorSource - closed
> 2011-10-14 01:49:48,391 [logicalNode collector0_log_dir-115] INFO
> com.cloudera.flume.handlers.thrift.ThriftEventSource - Closed server on port
> 36892...
> 2011-10-14 01:49:48,391 [logicalNode collector0_log_dir-115] INFO
> com.cloudera.flume.handlers.thrift.ThriftEventSource - Queue still has 1000
> elements ...
> 2011-10-14 01:49:58,399 [logicalNode collector0_log_dir-115] WARN
> com.cloudera.flume.handlers.thrift.ThriftEventSource - Close timed out due to
> no progress. Closing despite having 1000 values still enqueued
> 2011-10-14 01:49:58,399 [logicalNode collector0_log_dir-115] INFO
> com.cloudera.flume.handlers.rolling.RollSink - closing RollSink
> 'escapedCustomDfs("hdfs://van-mang-perf-hadoop-namenode1.net:8020/rawLogs/%{dateyear}-%{datemonth}-%{dateday}/%{datehr}00","raw-%{rolltag}"
> )'
> 2011-10-14 01:49:58,400 [logicalNode collector0_log_dir-115] INFO
> com.cloudera.flume.handlers.rolling.RollSink - double close
> 'escapedCustomDfs("hdfs://van-mang-perf-hadoop-namenode1.net:8020/rawLogs/%{dateyear}-%{datemonth}-%{dateday}/%{datehr}00","raw-%{rolltag}"
> )'
> 2011-10-14 01:49:58,400 [logicalNode collector0_log_dir-115] ERROR
> com.cloudera.flume.core.connector.DirectDriver - Exiting driver logicalNode
> collector0_log_dir-115 in error state CollectorSource | RegexExtractor
> because Blocked append interrupted by rotation event
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
