Cameron Thanks for fliling the jira. https://issues.apache.org/jira/browse/FLUME-757
I've commented there. Basically, I think there was a race introduced in the new version. I've described two potential solutions in the jira. Jon. On Fri, Sep 2, 2011 at 11:41 AM, Cameron Gandevia <[email protected]>wrote: > Hi > > I am experiencing a weird issue with my Flume configuration. Occasionally > when I start my agents they start delivering messages to the collectors and > after awhile I receive the following exceptions and stop receiving > messages. > > I am also curious if anyone can comment how they monitor these kind of > errors in their flume deployments. Currently we are monitoring the flume > process which is still running. I am guessing the best option is to also > monitor the hdfs destination to ensure we are constantly receiving a stream > of events from flume? > > I am currently using the Cloudera Flume 0.9.4+25.9 build in multi master > mode. > > The collectors are configured as follows. > > Source: collectorSource(36892) > Sink: {regex("^(?:\\n)?(\\d\\d\\d\\d-\\d\\d-\\d\\d\\s\\d\\d:\\d\\d)", 1, > "date") => {exDate("date", "yyyy-MM-dd HH:mm") => > > collectorSink("hdfs://hadoop-namenode1:8020/logs/%{dateyear}-%{datemonth}-%{dateday}/%{datehr}00/%{host}", > "log-")}} > > The agents are configured as follows. > > Source: tailDir("/data1/logs", delim="\\n\\d\\d\\d\\d", delimMode="next") > Sink: agentDFOChain("flume-collector1:36892", "flume-collector2:36892") > > The following exceptions are from my collectors. > > INFO com.cloudera.flume.handlers.debug.InsistentAppendDecorator - Failed > due to unexpected runtime exception during append attempt > java.lang.NullPointerException > at > com.cloudera.flume.handlers.hdfs.CustomDfsSink.close(CustomDfsSink.java:88) > at > > com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.close(EscapedCustomDfsSink.java:132) > at > com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56) > at > > com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) > at > > com.cloudera.flume.collector.CollectorSink$RollDetectDeco.close(CollectorSink.java:171) > at > com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:331) > at > > com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) > at > > com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) > at > > com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175) > at > > com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) > at > > com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:96) > at > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > > com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110) > at > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > > com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113) > at > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > > com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62) > at > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > > com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81) > at > com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222) > at > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > > com.cloudera.flume.core.extractors.DateExtractor.append(DateExtractor.java:129) > at > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > > com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88) > at > > com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110) > ERROR com.cloudera.flume.core.connector.DirectDriver - Closing down due to > exception during append calls > java.lang.NullPointerException > at > com.cloudera.flume.handlers.hdfs.CustomDfsSink.close(CustomDfsSink.java:88) > at > > com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.close(EscapedCustomDfsSink.java:132) > at > com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56) > at > > com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) > at > > com.cloudera.flume.collector.CollectorSink$RollDetectDeco.close(CollectorSink.java:171) > at > com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:331) > at > > com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) > at > > com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) > at > > com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175) > at > > com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) > at > > com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:96) > at > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > > com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110) > at > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > > com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113) > at > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > > com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62) > at > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > > com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81) > at > com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222) > at > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > > com.cloudera.flume.core.extractors.DateExtractor.append(DateExtractor.java:129) > at > > com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) > at > > com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88) > at > > com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110) > > The following are from my agents > > ERROR com.cloudera.flume.core.connector.DirectDriver - Expected IDLE but > timed out in state ACTIVE > INFO com.cloudera.flume.agent.diskfailover.DiskFailoverDeco - Closing disk > failover log, subsink still making progre > ERROR com.cloudera.flume.agent.LogicalNode - Forcing driver to exit > uncleanly > INFO com.cloudera.flume.agent.LogicalNode - Node config successfully set > to > com.cloudera.flume.conf.FlumeConfigData@19a0feb > WARN com.cloudera.flume.handlers.debug.LazyOpenDecorator - Closing a lazy > sink that was not logically opened > INFO com.cloudera.flume.handlers.text.TailDirSource - added file > /data1/mutator/logs/one.log > INFO com.cloudera.flume.handlers.rolling.RollSink - Created RollSink: > trigger=[TimeTrigger: maxAge=10000 tagger=com. > cloudera.flume.handlers.rolling.ProcessTagger@6279d] checkPeriodMs = 250 > spec='NaiveFileFailover' > INFO com.cloudera.flume.handlers.rolling.RollSink - opening RollSink > 'NaiveFileFailover' > ERROR com.cloudera.flume.agent.diskfailover.DiskFailoverDeco - WAL drain > thread interrupted > java.lang.InterruptedException > at java.lang.Object.wait(Native Method) > at > > com.cloudera.flume.core.connector.DirectDriver.waitForAtLeastState(DirectDriver.java:308) > at > > com.cloudera.flume.agent.diskfailover.DiskFailoverDeco.ensureClosedDrainDriver(DiskFailoverDeco.java:129) > at > > com.cloudera.flume.agent.diskfailover.DiskFailoverDeco.close(DiskFailoverDeco.java:177) > at > > com.cloudera.flume.core.BackOffFailOverSink.close(BackOffFailOverSink.java:165) > at > com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56) > at > > com.cloudera.flume.agent.AgentFailChainSink.close(AgentFailChainSink.java:98) > at > com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56) > at > > com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:126) > > Thanks > -- // Jonathan Hsieh (shay) // Software Engineer, Cloudera // [email protected]
