I know this is not exactly what you are asking for but have you had a look at the spillable memory channel. https://flume.apache.org/FlumeUserGuide.html#spillable-memory-channel
On Sun, Oct 19, 2014 at 1:38 AM, terreyshih <[email protected]> wrote: > In other words, I would like to explicitly drop the events if the > channel/event queue if full. > > thx > > > On Oct 18, 2014, at 11:42 PM, terreyshih <[email protected]> wrote: > > > > Hi, > > > > If I have a upstream agent duplicating the event to two channels and > subsequently to two downstream sinks. If one of the downstream sinks for > some reason can not process the events fast enough and the event overflows > back over the buffer limit of the upstream agent, would it cause any issues > for the other channel since the events are duplicated ? > > > > In general, what happens if the downstream agent is not able to handle > the speed of the upstream agent sending the events ? I have seen > exceptions that says unable to send events or memory channel commit issues. > > > > However, the flume agent does not seem to handle that gracefully and the > whole agent just stops processing the events for all channels, even for the > channels that can keep up with events. > > > > What if I separate the two channels into two different agents, would > that solve the problem. > > > > Can upstream flume agent not just drop the events when one of the its > sinks or downstream agents cannot catch up ? > > > > thx > >
