Hi Nguyễn,
The spooling source only works on "done", immutable files. So they have to
be atomically moved and they cannot be modified after being placed into the
spooling directory.
Regards,
Mike
On Sun, Jan 27, 2013 at 11:14 PM, NGuyen thi Kim Tuyen <
tuyen03a...@gmail.com> wrote:
> Hi ,
>
> Please help me .
>
> I want to use Flume in the following case :
> Spooling directory source --> FileChannel --> HBase sink . But I have some
> problems with Spooling directory source :
>
> Here is my test flume.conf :
> t-game-db194.sources = test-hbase
>
> t-game-db194.sinks = sink-hbase
>
> t-game-db194.channels = hbase-channel
>
> #source spoolDir
> t-game-db194.sources.test-hbase.type = spooldir
>
> t-game-db194.sources.test-hbase.spoolDir =/var/log/testhbase
>
> t-game-db194.sources.test-hbase.fileHeader = true
>
> t-game-db194.sources.test-hbase.channels = hbase-channel
>
> #file Channel
> t-game-db194.channels.hbase-channel.type = file
>
> t-game-db194.channels.hbase-channel.checkpointDir =
> /var/log/flume-ng/checkpoint
>
> t-game-db194.channels.hbase-channel.dataDir = /var/log/flume-ng/filedata
>
>
> #sink
> t-game-db194.sinks.sink-hbase.type = logger
>
> t-game-db194.sinks.sink-hbase.channel = hbase-channel
>
> And I tested : echo "tuyen ssssssssss " >>
> "/var/log/testhbase/hbase_1.log" . The first event is OK , but the next
> events are not work . Here is flume.log
>
> 28 Jan 2013 13:16:47,424 INFO [lifecycleSupervisor-1-0]
> (org.apache.flume.source.SpoolDirectorySource.start:64) -
> SpoolDirectorySource source starting with directory:/var/log/testhbase
> 28 Jan 2013 13:16:47,732 INFO [pool-7-thread-1]
> (org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229)
> - Preparing to move file /var/log/testhbase/hbase_1.log to
> /var/log/testhbase/hbase_1.log.COMPLETED
> 28 Jan 2013 13:16:48,436 INFO
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.LoggerSink.process:70) - Event: {
> headers:{file=/var/log/testhbase/hbase_1.log} body: 74 75 79 65 6E 20 73 73
> 73 73 73 73 73 73 73 73 tuyen ssssssssss }
>
> 28 Jan 2013 13:17:08,836 INFO [pool-7-thread-1]
> (org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229)
> - Preparing to move file /var/log/testhbase/hbase_1.log to
> /var/log/testhbase/hbase_1.log.COMPLETED
> 28 Jan 2013 13:17:08,837 ERROR [pool-7-thread-1]
> (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148)
> - Uncaught exception in Runnable
> java.lang.IllegalStateException: File name has been re-used with different
> files. Spooling assumption violated for
> /var/log/testhbase/hbase_1.log.COMPLETED
> at
> org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile(SpoolingFileLineReader.java:272)
> at
> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:185)
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 28 Jan 2013 13:17:09,340 ERROR [pool-7-thread-1]
> (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148)
> - Uncaught exception in Runnable
> java.io.IOException: Stream closed
> at java.io.BufferedReader.ensureOpen(BufferedReader.java:97)
> at java.io.BufferedReader.readLine(BufferedReader.java:292)
> at java.io.BufferedReader.readLine(BufferedReader.java:362)
> at
> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:180)
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
>
> Are there more documents about Flume-ng spooling source , beside
> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source ?
>
> Could you please give me some advice ?
>
> --
> Nguyễn Thị Kim Tuyên
> Computer Science Engineering
> HCMC University Of Technology.
