Okay. Thanks! Regards, Nikhil Gopishetti Sadanand. Illinois, USA. 857-253-9553.
On Wed, Jul 22, 2015 at 12:10 PM, Johny Rufus <[email protected]> wrote: > Sure, then we can confirm that FLUME-1934 is the issue, and upgrading to > flume-1.6 should take care of it. > > Thanks, > Rufus > > On Wed, Jul 22, 2015 at 10:02 AM, Nikhil Gs <[email protected]> > wrote: > >> Yes Johny. I can see some of the files with 0 size. >> >> Thanks! >> >> Regards, >> Nikhil Gopishetti Sadanand. >> Illinois, USA. >> 857-253-9553. >> >> On Wed, Jul 22, 2015 at 11:55 AM, Johny Rufus <[email protected]> >> wrote: >> >>> Can you confirm one more thing, do you have some files in the spool >>> directory that are 0 bytes ? >>> >>> Thanks, >>> Rufus >>> >>> On Wed, Jul 22, 2015 at 9:48 AM, Nikhil Gs <[email protected]> >>> wrote: >>> >>>> We are deleting the files from the spool directory only after >>>> consumption of files. So now, if I upgrade my Flume to 1.6... Will it be >>>> solved? >>>> >>>> Regards, >>>> Nikhil Gopishetti Sadanand. >>>> Illinois, USA. >>>> 857-253-9553. >>>> >>>> On Wed, Jul 22, 2015 at 10:35 AM, Johny Rufus <[email protected]> >>>> wrote: >>>> >>>>> [image: Boxbe] <https://www.boxbe.com/overview> This message is >>>>> eligible for Automatic Cleanup! ([email protected]) Add cleanup rule >>>>> <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Ftoken%3DI1AGGNBeRixnJqSFO%252B2Azw4T5PQV5DpBzTm%252FcvBKi8os7xt4EueN0aTFQl4EnHRGPzdDVz88BqNDmQT1cyrwKbpunPRBSVULGIrYk%252FyYGv1CV8Y7f0WSZ8HWozaRjKymoKYMBGjt3%252FY%253D%26key%3DBdlCAxP8uw46NdLhIif5DMc4DAtRhSsTF2T7Hg85pEo%253D&tc_serial=22052737395&tc_rand=1998199434&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> >>>>> | More info >>>>> <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=22052737395&tc_rand=1998199434&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> >>>>> >>>>> A couple of things. >>>>> 1) You should not delete the file while the file is still being read >>>>> from. This could lead to the IlegalStateException >>>>> 2) As you pointed out, you could end up with this issue when 0 byte >>>>> files are placed in the spool directory and if the fix for FLUME-1934 is >>>>> not there. >>>>> >>>>> Thanks, >>>>> Rufus >>>>> >>>>> On Wed, Jul 22, 2015 at 8:25 AM, Nikhil Gs <[email protected]> >>>>> wrote: >>>>> >>>>>> Deleting the file after placed in the spooling directory. >>>>>> >>>>>> Is this because of my flume version i.e. Flume 1.5.0-cdh5.4.1 >>>>>> Because, I have noticed the same issue that has been resolved in >>>>>> Flume 1.6. >>>>>> https://issues.apache.org/jira/browse/FLUME-1934 >>>>>> >>>>>> Thanks, >>>>>> Flume User. >>>>>> >>>>>> >>>>>> On Wed, Jul 22, 2015 at 10:16 AM, Johny Rufus <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Are you renaming or deleting the file that has been placed in the >>>>>>> spooling directory ? >>>>>>> >>>>>>> Thanks, >>>>>>> Rufus >>>>>>> >>>>>>> On Wed, Jul 22, 2015 at 6:41 AM, Nikhil Gs < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hello Everyone, >>>>>>>> >>>>>>>> Facing a problem with flume spool. >>>>>>>> Below is my configuration, >>>>>>>> >>>>>>>> # Please paste flume.conf here. Example: >>>>>>>> >>>>>>>> # Sources, channels, and sinks are defined per >>>>>>>> # agent name, in this case 'pnm'. >>>>>>>> pnm.sources = SPOOL >>>>>>>> pnm.channels = MemChannel >>>>>>>> pnm.sinks = AVRO >>>>>>>> >>>>>>>> # For each source, channel, and sink, set >>>>>>>> # standard properties. >>>>>>>> pnm.sources.SPOOL.type = spooldir >>>>>>>> pnm.sources.SPOOL.spoolDir = >>>>>>>> /home/s_sdldalplhdxxxedh/pnm-poll-results >>>>>>>> pnm.sources.SPOOL.channels = MemChannel MemChannel2 >>>>>>>> pnm.sources.SPOOL.fileHeader = true >>>>>>>> pnm.sources.SPOOL.deletePolicy = immediate >>>>>>>> pnm.sources.SPOOL.consumeOrder = oldest >>>>>>>> pnm.sources.SPOOL.batchSize = 1 >>>>>>>> >>>>>>>> pnm.sources.SPOOL.interceptors = time >>>>>>>> pnm.sources.SPOOL.interceptors.time.type = >>>>>>>> org.apache.flume.interceptor.TimestampInterceptor$Builder >>>>>>>> pnm.sources.SPOOL.deserializer = >>>>>>>> com.suddenlink.flume.WholeFileDeserializer$Builder >>>>>>>> >>>>>>>> pnm.sinks.AVRO.type = avro >>>>>>>> pnm.sinks.AVRO.channel = MemChannel >>>>>>>> pnm.sinks.AVRO.hostname = sdldalplhdw01.suddenlink.cequel3.com >>>>>>>> pnm.sinks.AVRO.port = 40001 >>>>>>>> pnm.sinks.AVRO.batchSize = 1 >>>>>>>> pnm.sinks.AVRO.connect-timeout = 40000 >>>>>>>> >>>>>>>> >>>>>>>> # pnm.sinks.HDFS.type = hdfs >>>>>>>> # pnm.sinks.HDFS.channel = MemChannel2 >>>>>>>> # pnm.sinks.HDFS.hdfs.path = /user/flume/poll/%Y/%m/%d/%H/ >>>>>>>> # pnm.sinks.HDFS.hdfs.fileType = DataStream >>>>>>>> # pnm.sinks.HDFS.hdfs.writeFormat = Text >>>>>>>> # pnm.sinks.HDFS.hdfs.batchSize = 100 >>>>>>>> # pnm.sinks.HDFS.hdfs.rollSize = 0 >>>>>>>> # pnm.sinks.HDFS.hdfs.rollCount = 1000 >>>>>>>> # pnm.sinks.HDFS.hdfs.rollInterval = 600 >>>>>>>> >>>>>>>> # Other properties are specific to each type of >>>>>>>> # source, channel, or sink. In this case, we >>>>>>>> # specify the capacity of the memory channel. >>>>>>>> >>>>>>>> pnm.channels.MemChannel.capacity = 1000000 >>>>>>>> pnm.channels.MemChannel.type = memory >>>>>>>> >>>>>>>> # pnm.channels.MemChannel2.capacity = 10000 >>>>>>>> # pnm.channels.MemChannel2.type = memory >>>>>>>> >>>>>>>> >>>>>>>> *Error Log file* >>>>>>>> >>>>>>>> org.apache.flume.source.SpoolDirectorySource >>>>>>>> FATAL: Spool Directory source SPOOL: { spoolDir: >>>>>>>> /home/s_sdldalplhdxxxedh/pnm-poll-results }: Uncaught exception in >>>>>>>> SpoolDirectorySource thread. Restart or reconfigure Flume to continue >>>>>>>> processing. >>>>>>>> java.lang.IllegalStateException: File should not roll when commit >>>>>>>> is outstanding. >>>>>>>> at >>>>>>>> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:235) >>>>>>>> at >>>>>>>> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:228) >>>>>>>> at >>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>>>>>> at >>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>>>>>> at >>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>>>>>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> >>>>>>>> I have tried to restart the agent, but the same error is appearing. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Nikhil >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> >
