Looks like the either the file limit is set to too low or some file descriptor leakage. Assuming former, try standard *nix tools like ulimit and lsof to find open fd and if needed increase the ulimit to a suitable value.
HTH ! On Sun, Aug 9, 2015 at 8:36 AM, George Blazer <[email protected]> wrote: > Hello there, > > My flume seems to be getting stuck in the weird state and I'm not sure what > to do about it. > > Metrics endpoint looks like Flume is relatively happy, but I'm seeing the > error below > > curl localhost:5653/metrics > {"SINK.s3-sink":{"BatchCompleteCount":"0","ConnectionFailedCount":"0","EventDrainAttemptCount":"0","ConnectionCreatedCount":"0","Type":"SINK","BatchEmptyCount":"0","ConnectionClosedCount":"0","EventDrainSuccessCount":"0","StopTime":"0","StartTime":"1439133984193","BatchUnderflowCount":"0"},"SOURCE.spooling-directory":{"OpenConnectionCount":"0","Type":"SOURCE","AppendBatchAcceptedCount":"0","AppendBatchReceivedCount":"1","EventAcceptedCount":"0","AppendReceivedCount":"0"} > > typekit@prod-polka-51145e82:/var/log/flume-ng$ ls -la > /mnt/flume/data/log-1955.meta > -rw-r--r-- 1 root root 47 Jul 26 15:57 /mnt/flume/data/log-1955.meta > > The error I see is > > 2015-08-09 15:31:24,236 (SinkRunner-PollingRunner-DefaultSinkProcessor) > [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] > Unable to deliver event. Exception follows. > java.lang.IllegalStateException: Channel closed [channel=fileChannel]. Due > to java.io.FileNotFoundException: /mnt/flume/data/log-1955.meta (Too many > open files) > at > org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:340) > at > org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122) > at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:368) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.FileNotFoundException: /mnt/flume/data/log-1955.meta (Too > many open files) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.<init>(FileInputStream.java:146) > at > org.apache.flume.channel.file.LogFileV3$SequentialReader.<init>(LogFileV3.java:318) > at > org.apache.flume.channel.file.LogFileFactory.getSequentialReader(LogFileFactory.java:165) > at > org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:264) > at org.apache.flume.channel.file.Log.doReplay(Log.java:519) > at org.apache.flume.channel.file.Log.replay(Log.java:445) > at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:290) > at > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > Is /metrics basically useless to look at the health of Flume? > > Please help. > > -George > > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
