Hi, On Jul 20, 2012, at 12:13 PM, Wang, Yongkun | Yongkun | BDD wrote:
inline > Hi, > > Currently we are using a modified version of flume-0.9.3 in production. > > I tried release 1.1.0. The basic command is still bin/flume-ng node, not > bin/flume-ng agent (by doc). Did you build Flume from git? In 1.1.x the notation was changed into agent, node is deprecated. > No document for HDFS sink, I need to read the source code to config it. HDFS > sink of 1.1.0 seems be developed with hadoop 0.20.205.0. It can rotate files > by count of events, but the last hadoop file cannot be closed. http://archive.cloudera.com/cdh4/cdh/4/flume-ng/ > > Rotating the hadoop files by time interval does not work. It seems that > problem is here: > A mistake by timing 1000 in BucketWriter: > > lastProcessTime = System.currentTimeMillis() * 1000; > > So the lines in 'if' will never get executed: > > if ((rollInterval > 0) > > && (rollInterval < (System.currentTimeMillis() - lastProcessTime) / > 1000)) > > doRotate = true; > > But the code has been changed aggressively in later versions, so it is no > longer a bug in the trunk. > > I tested NG trunk (1.3 snapshot). The code is aggressively changed compared > to 1.1.0. It seems that it is developed with hadoop 0.23.1 or 1.0.1. It can > work with hadoop 0.20.205.0 (time-interval-based rotation works perfectly, > last hadoop file closed correctly). > Flume looks into the exported classpath and determine the installed hadoop version to use it. > Will keep an eye on NG and wait for later releases. cheers, Alex -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
