[
https://issues.apache.org/jira/browse/FLUME-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258280#comment-13258280
]
[email protected] commented on FLUME-1097:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4826/#review7092
-----------------------------------------------------------
Looks pretty good. I have some feedback which is inline.
Suggestion: Do you think you could add support for the decorators in the
sources as well (though we can do that later, if you want to leave it for now)?
Eg: ExecSource might want to add some header based on the event size etc? It
would be good to have that
flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java
<https://reviews.apache.org/r/4826/#comment15699>
It would be good to refine this factory a bit. Please take a look at the
other factory classes. Use an enum to specify the type of the decorator to
build, or if it is not a know type, use the class name specified. Take a look
at DefaultSourceFactory etc.
flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java
<https://reviews.apache.org/r/4826/#comment15698>
This should be named something like TimeStampEventDecorator or something.
We might want to add decorators that add other stuff at the sink.
- Hari
On 2012-04-20 10:33:14, Inder Singh wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4826/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-20 10:33:14)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari
Shreedharan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Overview of Changes
bq.
bq. 1.EventHeaderDecorator -> generic class having processHeader()
bq. 2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides
timestamp header at this hop.
bq. 3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on
<sink>.headerDecorator config
bq. 4.AbstractSink -> has a eventHeaderDecortaor
bq. 5.HDFSEventSink -> checks for decorator through config and calls
processHeader() for each event which modifies/add headers based on event
decorator defined through config.
bq.
bq. Sample Configuration for testing
bq. ----------------------------------
bq. agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator
bq.
bq.
bq. This addresses bug FLUME-1097.
bq. https://issues.apache.org/jira/browse/FLUME-1097
bq.
bq.
bq. Diffs
bq. -----
bq.
bq.
flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java
PRE-CREATION
bq.
flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java
PRE-CREATION
bq.
flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java
PRE-CREATION
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java
2334059
bq.
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
1b61cad
bq.
bq. Diff: https://reviews.apache.org/r/4826/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Testing Done
bq. -------------
bq. 1. Unit tests
bq. 2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix =
flume-data/%D results in publishing data with timestamp at agent's hop running
HDFSEventSink
bq.
bq.
bq. Thanks,
bq.
bq. Inder
bq.
bq.
> Support modification of event Headers through configuration
> -----------------------------------------------------------
>
> Key: FLUME-1097
> URL: https://issues.apache.org/jira/browse/FLUME-1097
> Project: Flume
> Issue Type: New Feature
> Reporter: Inder SIngh
> Assignee: Inder SIngh
> Priority: Minor
> Attachments: FLUME-1097-1.patch
>
>
> As discussed on the flume threads we want to support adding/modifying headers
> at flume agents through configuration.
> Primary use-case as of now -
> Use HDFSEventSink to publish data in paths like /flume-data/YYYY/mm/DD/HH/MM
> wherein timestamp is of agent running HDFSEventSink.
> What's not there today
> -------------------------
> 1. headers are put by clientLIB and can't be added/modified in flight by
> agents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
