[jira] [Commented] (FLUME-2140) Support diverting bad events from pipeline
[ https://issues.apache.org/jira/browse/FLUME-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131959#comment-14131959 ] Jean lagarutte commented on FLUME-2140: --- hello any news or advance in this problem ? This is a major stopper for me and i'm currently considering switching to another software. The events comes from several sources and sometimes they are malformed. But the regex interceptor can still parse them but generate some headers with special characters. The hdfs sink use theses header for the path but with characters like '\ * or :' it fails to deliver event. I need to restart flume and i lost all event frequently. The failover sink is not really a good option for me. Can't we just drop the bad event and write them to agent logfile ? > Support diverting bad events from pipeline > -- > > Key: FLUME-2140 > URL: https://issues.apache.org/jira/browse/FLUME-2140 > Project: Flume > Issue Type: New Feature > Components: Node >Reporter: Arvind Prabhakar > > A *bad event* can be any event that causes persistent sink side processing > failure due to the inherent nature of the event itself. Note that failures > that are not related to the inherent nature of the event such as network > communication failure, downstream capacity failure etc., do not make the > event a bad-event. > The presence of a bad event in a channel can cause the entire pipleline to > choke and become unusable. Flume should therefore be able to identify bad > events and provide a facility to route them out of the pipleline in order to > ensure the transport of other events continues uninterrupted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2140) Support diverting bad events from pipeline
[ https://issues.apache.org/jira/browse/FLUME-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13732732#comment-13732732 ] Arvind Prabhakar commented on FLUME-2140: - [Discussion thread|http://flume.markmail.org/thread/y3cks6hdgof3kxu6#query:+page:1+mid:rx3zm53t4dhmqskk+state:results] on this subject in the user-list for reference. > Support diverting bad events from pipeline > -- > > Key: FLUME-2140 > URL: https://issues.apache.org/jira/browse/FLUME-2140 > Project: Flume > Issue Type: New Feature > Components: Node >Reporter: Arvind Prabhakar > > A *bad event* can be any event that causes persistent sink side processing > failure due to the inherent nature of the event itself. Note that failures > that are not related to the inherent nature of the event such as network > communication failure, downstream capacity failure etc., do not make the > event a bad-event. > The presence of a bad event in a channel can cause the entire pipleline to > choke and become unusable. Flume should therefore be able to identify bad > events and provide a facility to route them out of the pipleline in order to > ensure the transport of other events continues uninterrupted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-2140) Support diverting bad events from pipeline
[ https://issues.apache.org/jira/browse/FLUME-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727912#comment-13727912 ] Arvind Prabhakar commented on FLUME-2140: - Another case - a downstream filter is buggy and causes a batch to fail repeatedly due to a malformed header or some other details. > Support diverting bad events from pipeline > -- > > Key: FLUME-2140 > URL: https://issues.apache.org/jira/browse/FLUME-2140 > Project: Flume > Issue Type: New Feature > Components: Node >Reporter: Arvind Prabhakar > > A *bad event* can be any event that causes persistent sink side processing > failure due to the inherent nature of the event itself. Note that failures > that are not related to the inherent nature of the event such as network > communication failure, downstream capacity failure etc., do not make the > event a bad-event. > The presence of a bad event in a channel can cause the entire pipleline to > choke and become unusable. Flume should therefore be able to identify bad > events and provide a facility to route them out of the pipleline in order to > ensure the transport of other events continues uninterrupted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-2140) Support diverting bad events from pipeline
[ https://issues.apache.org/jira/browse/FLUME-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13727742#comment-13727742 ] Roshan Naik commented on FLUME-2140: It appears that may be only the sink is able to know for certain (at the time to delivery) that a particular event is bad. One scenario that comes to mind here: - an event that may be bad for one sink maynot be bad for another. for instance, logger sink or rolling file sink may not have trouble delivering an event that is troublesome for elastic search sink. > Support diverting bad events from pipeline > -- > > Key: FLUME-2140 > URL: https://issues.apache.org/jira/browse/FLUME-2140 > Project: Flume > Issue Type: New Feature > Components: Node >Reporter: Arvind Prabhakar > > A *bad event* can be any event that causes persistent sink side processing > failure due to the inherent nature of the event itself. Note that failures > that are not related to the inherent nature of the event such as network > communication failure, downstream capacity failure etc., do not make the > event a bad-event. > The presence of a bad event in a channel can cause the entire pipleline to > choke and become unusable. Flume should therefore be able to identify bad > events and provide a facility to route them out of the pipleline in order to > ensure the transport of other events continues uninterrupted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
