[
https://issues.apache.org/jira/browse/FLUME-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841212#comment-16841212
]
James Grinter commented on FLUME-2580:
--------------------------------------
Agreed, the Interceptor interface isn't currently able to increase the number
of events - though they can drop them - because the interface requires
supporting taking and returning a single Event:
{noformat}
/**
* Interception of a single {@link Event}.
* @param event Event to be intercepted
* @return Original or modified event, or {@code null} if the Event
* is to be dropped (i.e. filtered out).
*/
public Event intercept(Event event);
{noformat}
(which in turn is because ChannelProcessor interface supports both
processEventBatch *and* processEvent, and that requirement continues all the
way down into the ChannelProcessor).
I too have a situation where I'd like to be able to process the inputs, across
different types of sources (Kafka, HTTP, TCP, etc.) and turn them into multiple
Events, which is how I ended up at this Jira ticket.
HTTP Source has evolved an HTTP-specific mechanism that can do this - the
HTTPSourceHandler interface. But whilst sources could support a slightly more
generic version of this not specific to processing an HttpServletRequest
object, that would impose an additional requirement on all Sources.
So ChannelProcessor would seem to be the best place to solve the problem of
Interceptors not being able to create new Events: processEvent could be
refactored to use processEventBatch (it seems to be just an optimisation for
handling single events?), and then remove the singular
Interceptor.intercept(Event) method.
> Sink side interceptors
> ----------------------
>
> Key: FLUME-2580
> URL: https://issues.apache.org/jira/browse/FLUME-2580
> Project: Flume
> Issue Type: New Feature
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Priority: Major
> Attachments: SinkSideInterceptors.pdf
>
>
> Currently, we only have source-side interceptors that help routing. But if we
> use something like Kafka Channel, having a sink-side interceptor can help us
> modify events as they come in. We could also do validation on event schemas
> and drop them before they hit the sink, rather than have an infinite loop due
> to such events.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
