> On 2012-05-30 06:37:16, Juhani Connolly wrote: > > flume-ng-core/src/main/java/org/apache/flume/interceptor/TimestampInterceptor.java, > > line 49 > > <https://reviews.apache.org/r/5188/diff/1/?file=109768#file109768line49> > > > > You may also want to check you're not overwriting an existing timestamp > > header here. While I think it may not be a common use case, I'm sure there > > will cases where people have stuff coming from multiple sources, some that > > supply their own stamps. > > Inder Singh wrote: > That would prevent header modification at each hop in the pipeline. > Eg: > > 1. i might want timestamp at last node in the pipeline, enough though > timestamp is injected by producers > 2. i want the to sneak the headers and each hop of pipeline and timestamp > can change at each hop. > > Overall putting this check will restrict the ability of using the same > interceptor at multiple agents in the pipeline. > It might add more ambiguity wherein if we the deploy the same config > across multiple agents, then the first agent putting this header is only > applicable and everything else is a NOP. > > My take is to leave this flexibility to the users as to how they want to > configure FLUME. > > Juhani Connolly wrote: > I think the more common use case would be people wanting to insert > timestamps as data enters the timeline, though I guess this is also a > possibility. > > It may be an idea to make it configurable > > Inder Singh wrote: > Juhani, > > our use-case is to publish HDFS paths like /flume-data/YYYY/MM/DD/HH/mm/.. > path should be based out of event arrival time-stamp on agent. We may or > may not have control over "timestamp" header being injected at the producers > side. > > We can always keep certain headers reserved and solve this if header > modification isn't supported. > I like the idea of keeping it configurable.
Hey guys, in the latest patch I have added support for a configuration parameter called "preserveExisting". It defaults to false. If set to true, it will not clobber existing timestamp headers. The default setting, which I think should be intuitive, is to overwrite the existing timestamp. I also added unit tests for both cases. - Mike ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5188/#review8190 ----------------------------------------------------------- On 2012-05-31 07:40:36, Mike Percy wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/5188/ > ----------------------------------------------------------- > > (Updated 2012-05-31 07:40:36) > > > Review request for Flume. > > > Summary > ------- > > Created a timestamp interceptor that sets the current time as the "timestamp" > header of all events that flow through it. > > > This addresses bug FLUME-1215. > https://issues.apache.org/jira/browse/FLUME-1215 > > > Diffs > ----- > > flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java > abf2f75 > > flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorBuilderFactory.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/interceptor/TimestampInterceptor.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/interceptor/TestTimestampInterceptor.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/5188/diff > > > Testing > ------- > > Added unit test. > > > Thanks, > > Mike > >
