----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6377/#review10168 -----------------------------------------------------------
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java <https://reviews.apache.org/r/6377/#comment21530> I will add a patch fixing all this whitespace crap tomorrow, sorry about that. - Patrick Wendell On Aug. 11, 2012, 5:32 a.m., Patrick Wendell wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/6377/ > ----------------------------------------------------------- > > (Updated Aug. 11, 2012, 5:32 a.m.) > > > Review request for Flume. > > > Description > ------- > > This patch extends the existing avro client to support a file-based spooling > mechanism. See in-line documentation for precise details, but the basic idea > is that a user can have a spool directory where files are deposited for > ingestion into flume. Once ingested the files are clearly renamed and the > implementation guarantees at-least-once delivery semantics similar to those > achieved within flume itself, even across failures and restarts of the JVM > running the code. > > I feel vaguely uneasy about building this as part of the standlone avro > client rather than as its own source. An alternative would be to build this > as a proper source (in fact, there are some ad-hoc transaction semantics used > here which would really be a better fit for a source). Interested in hearing > feedback on that as well. The benefit of having this in the avro client is > that you don't need the flume runner scripts which are not windows compatible. > > > This addresses bug FlUME-1425. > https://issues.apache.org/jira/browse/FlUME-1425 > > > Diffs > ----- > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java > da804d7 > > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java > abbbf1c > flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java > 4a5ecae > > flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java > PRE-CREATION > flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java > PRE-CREATION > flume-ng-doc/sphinx/FlumeUserGuide.rst 45dd7cc > > Diff: https://reviews.apache.org/r/6377/diff/ > > > Testing > ------- > > Extensive unit tests and I also built and played with this using a stub flume > agent. If you look at the JIRA I have a configuration file for an agent that > will print out Avro events to the command line - that's helpful when testing > this. > > > Thanks, > > Patrick Wendell > >
