Jasper D created FLUME-3423:
-------------------------------
Summary: Taildir source will throw unnecessay exception when
restart agent
Key: FLUME-3423
URL: https://issues.apache.org/jira/browse/FLUME-3423
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: 1.9.0
Reporter: Jasper D
Fix For: 1.10.0
h2. *Phenomenon*
When we use Taildir source, it will auto create an empty tail position file at
start time. But if the filegroup directory is empty, it has no file to tail
and then we restart agent, it will throw an EOFException. Although it has
little effect on functionality, it will annoy users.
h2. *Recur*
This problem can esaily recur.
We can use the example.conf from flume user guide directly.
{code:java}
//代码占位符
a1.sources = r1
a1.channels = c1
a1.sources.r1.type = TAILDIR
a1.sources.r1.channels = c1
a1.sources.r1.positionFile = /var/log/flume/taildir_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /var/log/test1/*.log
a1.sources.ri.maxBatchCount = 1000
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1{code}
and start flume with it
{code:java}
//代码占位符
bin/flume-ng agent --conf conf --conf-file conf/example.conf --name a1
-Dflume.root.logger=INFO,console {code}
then ctrl + c stop it.
Just wait a moment, and then start flume again, then it will throw an exception
like below.
{code:java}
//代码占位符
ERROR [lifecycleSupervisor-1-1]
(org.apache.flume.source.taildir.ReliableTaildirEventReader.loadPositionFile:147)
- Failed loading positionFile: /var/log/flume/taildir_position.json
java.io.EOFException: End of input at line 1 column 1
at
com.google.gson.stream.JsonReader.nextNonWhitespace(JsonReader.java:954)
~[gson-2.2.2.jar:?]
at com.google.gson.stream.JsonReader.nextValue(JsonReader.java:771)
~[gson-2.2.2.jar:?]
at com.google.gson.stream.JsonReader.peek(JsonReader.java:367)
~[gson-2.2.2.jar:?]
at com.google.gson.stream.JsonReader.expect(JsonReader.java:337)
~[gson-2.2.2.jar:?]
at com.google.gson.stream.JsonReader.beginArray(JsonReader.java:306)
~[gson-2.2.2.jar:?]
at
org.apache.flume.source.taildir.ReliableTaildirEventReader.loadPositionFile(ReliableTaildirEventReader.java:111)
[flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.flume.source.taildir.ReliableTaildirEventReader.<init>(ReliableTaildirEventReader.java:96)
[flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.flume.source.taildir.ReliableTaildirEventReader.<init>(ReliableTaildirEventReader.java:49)
[flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:355)
[flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:105)
[flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71)
[flume-ng-core-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249)
[flume-ng-core-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0-292]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0-292]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0-292]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0-292]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0-292]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0-292]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0-292] {code}
h2. Fix
We just add a condition in ReliableTaildirEventReader constructor.
The pr/patch will be submitted as as shown below.
Or let it still exist and using warning log instead of exception thrown is
better?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
