Could you set the value for the open files to 8000, reboot to OS, inspect that it is 8000 and see whether this resolves your problem?
2016-04-20 10:20 GMT+02:00 Chris Horrocks <[email protected]>: > Are the permissions on the files the same? Does the user running the flume > agents have read permissions? > Are the files still being written to/locked open by another process? > Are there any logs being generated by the flume agent? > > -- > Chris Horrocks > > On 20 April 2016 at 08:00:14, Saurabh Sharma ([email protected]) > wrote: > >> Hi Ron, >> >> >> >> The maximum number of open files in our OS is 1024. >> >> Thanks >> >> >> >> *From:* Ronald Van De Kuil [mailto:[email protected]] >> *Sent:* 20 April 2016 12:24 >> *To:* [email protected] >> *Subject:* Re: Flume not marking log files as completed and do not >> process file further >> >> >> >> Not sure that this helps, ... have you checked your operating system >> settings for the maximum number of open files? >> >> Met vriendelijke groet, >> >> Ronald van de Kuil >> >> >> Op 20 apr. 2016 om 08:48 heeft Saurabh Sharma <[email protected]> >> het volgende geschreven: >> >> Hi, >> >> >> >> I have a scenario where we are ingesting the log files(around 80MB each) >> in flume and flume does process these files and marks as completed but >> after processing few files it does not process any files further and >> moreover it does not mark the log file as completed. >> >> >> >> We are using spooling directory source. >> >> >> >> I looked into the flume logs and found that when this scenario happens it >> shows the following line continuously in flume logs. >> >> >> >> DEBUG [conf-file-poller-0] >> (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) >> - Checking file:../conf/commerce-sense.conf for changes >> >> >> >> We have following configuration. >> >> agent.sources = spoolDir >> >> agent.channels = memoryChannel >> >> agent.sinks = sink >> >> agent.sources.spoolDir.interceptors = i1 >> >> >> >> >> >> #Channel Configuration >> >> agent.channels.memoryChannel.type = memory >> >> >> >> #Source configuration >> >> agent.sources.spoolDir.type = spooldir >> >> agent.sources.spoolDir.spoolDir = /opt/flume/spoolDir >> >> agent.sources.spoolDir.fileHeader = true >> >> agent.sources.spoolDir.basenameHeader = true >> >> agent.sources.spoolDir.deserializer = LINE >> >> agent.sources.spoolDir.inputCharset = ISO8859-1 >> >> agent.sources.spoolDir.deserializer.maxLineLength = 10000 >> >> agent.sources.spoolDir.interceptors.i1.type = >> org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder >> >> agent.sources.spoolDir.interceptors.i1.preserveExisting = true >> >> agent.sources.spoolDir.interceptors.i1.prefix = test >> >> agent.sources.spoolDir.channels = memoryChannel >> >> >> >> >> >> #Sink Configuration >> >> agent.sinks.sink.type = com. flume.sink.ExtendedKafkaSink >> >> agent.sinks.sink.topic = cdnLogsTopic >> >> agent.sinks.sink.brokerList = localhost:9092 >> >> agent.sinks.sink.batchSize = 100 >> >> agent.sinks.sink.sink.serializer = com. >> flume.serializer.ExtendedSerializer$Builder >> >> agent.sinks.sink.channel = memoryChannel >> >> >> >> Thanks, >> Saurabh >> >>
