Yes Johny. I can see some of the files with 0 size. Thanks!
Regards, Nikhil Gopishetti Sadanand. Illinois, USA. 857-253-9553. On Wed, Jul 22, 2015 at 11:55 AM, Johny Rufus <[email protected]> wrote: > Can you confirm one more thing, do you have some files in the spool > directory that are 0 bytes ? > > Thanks, > Rufus > > On Wed, Jul 22, 2015 at 9:48 AM, Nikhil Gs <[email protected]> > wrote: > >> We are deleting the files from the spool directory only after consumption >> of files. So now, if I upgrade my Flume to 1.6... Will it be solved? >> >> Regards, >> Nikhil Gopishetti Sadanand. >> Illinois, USA. >> 857-253-9553. >> >> On Wed, Jul 22, 2015 at 10:35 AM, Johny Rufus <[email protected]> >> wrote: >> >>> [image: Boxbe] <https://www.boxbe.com/overview> This message is >>> eligible for Automatic Cleanup! ([email protected]) Add cleanup rule >>> <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Ftoken%3DI1AGGNBeRixnJqSFO%252B2Azw4T5PQV5DpBzTm%252FcvBKi8os7xt4EueN0aTFQl4EnHRGPzdDVz88BqNDmQT1cyrwKbpunPRBSVULGIrYk%252FyYGv1CV8Y7f0WSZ8HWozaRjKymoKYMBGjt3%252FY%253D%26key%3DBdlCAxP8uw46NdLhIif5DMc4DAtRhSsTF2T7Hg85pEo%253D&tc_serial=22052737395&tc_rand=1998199434&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> >>> | More info >>> <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=22052737395&tc_rand=1998199434&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> >>> >>> A couple of things. >>> 1) You should not delete the file while the file is still being read >>> from. This could lead to the IlegalStateException >>> 2) As you pointed out, you could end up with this issue when 0 byte >>> files are placed in the spool directory and if the fix for FLUME-1934 is >>> not there. >>> >>> Thanks, >>> Rufus >>> >>> On Wed, Jul 22, 2015 at 8:25 AM, Nikhil Gs <[email protected]> >>> wrote: >>> >>>> Deleting the file after placed in the spooling directory. >>>> >>>> Is this because of my flume version i.e. Flume 1.5.0-cdh5.4.1 >>>> Because, I have noticed the same issue that has been resolved in Flume >>>> 1.6. >>>> https://issues.apache.org/jira/browse/FLUME-1934 >>>> >>>> Thanks, >>>> Flume User. >>>> >>>> >>>> On Wed, Jul 22, 2015 at 10:16 AM, Johny Rufus <[email protected]> >>>> wrote: >>>> >>>>> Are you renaming or deleting the file that has been placed in the >>>>> spooling directory ? >>>>> >>>>> Thanks, >>>>> Rufus >>>>> >>>>> On Wed, Jul 22, 2015 at 6:41 AM, Nikhil Gs <[email protected]> >>>>> wrote: >>>>> >>>>>> Hello Everyone, >>>>>> >>>>>> Facing a problem with flume spool. >>>>>> Below is my configuration, >>>>>> >>>>>> # Please paste flume.conf here. Example: >>>>>> >>>>>> # Sources, channels, and sinks are defined per >>>>>> # agent name, in this case 'pnm'. >>>>>> pnm.sources = SPOOL >>>>>> pnm.channels = MemChannel >>>>>> pnm.sinks = AVRO >>>>>> >>>>>> # For each source, channel, and sink, set >>>>>> # standard properties. >>>>>> pnm.sources.SPOOL.type = spooldir >>>>>> pnm.sources.SPOOL.spoolDir = >>>>>> /home/s_sdldalplhdxxxedh/pnm-poll-results >>>>>> pnm.sources.SPOOL.channels = MemChannel MemChannel2 >>>>>> pnm.sources.SPOOL.fileHeader = true >>>>>> pnm.sources.SPOOL.deletePolicy = immediate >>>>>> pnm.sources.SPOOL.consumeOrder = oldest >>>>>> pnm.sources.SPOOL.batchSize = 1 >>>>>> >>>>>> pnm.sources.SPOOL.interceptors = time >>>>>> pnm.sources.SPOOL.interceptors.time.type = >>>>>> org.apache.flume.interceptor.TimestampInterceptor$Builder >>>>>> pnm.sources.SPOOL.deserializer = >>>>>> com.suddenlink.flume.WholeFileDeserializer$Builder >>>>>> >>>>>> pnm.sinks.AVRO.type = avro >>>>>> pnm.sinks.AVRO.channel = MemChannel >>>>>> pnm.sinks.AVRO.hostname = sdldalplhdw01.suddenlink.cequel3.com >>>>>> pnm.sinks.AVRO.port = 40001 >>>>>> pnm.sinks.AVRO.batchSize = 1 >>>>>> pnm.sinks.AVRO.connect-timeout = 40000 >>>>>> >>>>>> >>>>>> # pnm.sinks.HDFS.type = hdfs >>>>>> # pnm.sinks.HDFS.channel = MemChannel2 >>>>>> # pnm.sinks.HDFS.hdfs.path = /user/flume/poll/%Y/%m/%d/%H/ >>>>>> # pnm.sinks.HDFS.hdfs.fileType = DataStream >>>>>> # pnm.sinks.HDFS.hdfs.writeFormat = Text >>>>>> # pnm.sinks.HDFS.hdfs.batchSize = 100 >>>>>> # pnm.sinks.HDFS.hdfs.rollSize = 0 >>>>>> # pnm.sinks.HDFS.hdfs.rollCount = 1000 >>>>>> # pnm.sinks.HDFS.hdfs.rollInterval = 600 >>>>>> >>>>>> # Other properties are specific to each type of >>>>>> # source, channel, or sink. In this case, we >>>>>> # specify the capacity of the memory channel. >>>>>> >>>>>> pnm.channels.MemChannel.capacity = 1000000 >>>>>> pnm.channels.MemChannel.type = memory >>>>>> >>>>>> # pnm.channels.MemChannel2.capacity = 10000 >>>>>> # pnm.channels.MemChannel2.type = memory >>>>>> >>>>>> >>>>>> *Error Log file* >>>>>> >>>>>> org.apache.flume.source.SpoolDirectorySource >>>>>> FATAL: Spool Directory source SPOOL: { spoolDir: >>>>>> /home/s_sdldalplhdxxxedh/pnm-poll-results }: Uncaught exception in >>>>>> SpoolDirectorySource thread. Restart or reconfigure Flume to continue >>>>>> processing. >>>>>> java.lang.IllegalStateException: File should not roll when commit is >>>>>> outstanding. >>>>>> at >>>>>> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:235) >>>>>> at >>>>>> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:228) >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> >>>>>> I have tried to restart the agent, but the same error is appearing. >>>>>> >>>>>> Regards, >>>>>> Nikhil >>>>>> >>>>> >>>>> >>>> >>> >>> >> >
