Hi,
Currently we are using a modified version of flume-0.9.3 in production.
I tried release 1.1.0. The basic command is still bin/flume-ng node, not
bin/flume-ng agent (by doc).
No document for HDFS sink, I need to read the source code to config it. HDFS
sink of 1.1.0 seems be developed with hadoop 0.20.205.0. It can rotate files by
count of events, but the last hadoop file cannot be closed.
Rotating the hadoop files by time interval does not work. It seems that problem
is here:
A mistake by timing 1000 in BucketWriter:
lastProcessTime = System.currentTimeMillis() * 1000;
So the lines in 'if' will never get executed:
if ((rollInterval > 0)
&& (rollInterval < (System.currentTimeMillis() - lastProcessTime) /
1000))
doRotate = true;
But the code has been changed aggressively in later versions, so it is no
longer a bug in the trunk.
I tested NG trunk (1.3 snapshot). The code is aggressively changed compared to
1.1.0. It seems that it is developed with hadoop 0.23.1 or 1.0.1. It can work
with hadoop 0.20.205.0 (time-interval-based rotation works perfectly, last
hadoop file closed correctly).
Will keep an eye on NG and wait for later releases.
Cheers,
Yongkun Wang (Kun)
