I think I was not clear. I was taking about an offline tool which would help us clean the Channel with valid Flume Event, with invalid/malformed payload. What you described would work in Agent flow. Given the current scenario, Charles would be keen to push the backed up events to destination.
Here is what I was taking about offline tool -> read event from file channel -> event not corrupt -> decodePayload (user supplied) -> if valid write to file channel else drop the event As far as my reading goes, everything on file channel is stored a TransactionEvent, so this might not be as simple as it looks. In another words, it would be like a FileChannelSink with user provided function to validate payload. On Thu, Feb 5, 2015 at 1:02 AM, Hari Shreedharan <[email protected]> wrote: > It is not as easy, since the channel does not know what an event looks like > or why the transaction is being rolled back. That is something that is being > handled by the serializer and sink. We need to some how remove an event from > the channel if it the channel is broken. We might perhaps have to add a new > interface that allows the sink/serializer to tell the channel “forget”a > specific event, and that event will be dropped from the channel and > transaction. > > I don’t see how we can do it outside the sink for this reason. > > Thanks, > Hari > > > On Wed, Feb 4, 2015 at 5:32 AM, Ashish <[email protected]> wrote: >> >> Is it possible to extend File Channel Integrity tool to support >> filtering out corrupt events? Something like once we get a record and >> it's not corrupt, provide a placeholder function to validate Event >> implemented by user. I still don't have too much insight into >> FileChannel implementation. >> >> On Tue, Feb 3, 2015 at 12:49 AM, Hari Shreedharan >> <[email protected]> wrote: >> > Currently, no - there is no such tool, but this is a request that has >> > come >> > up time and again. Can you file a jira for this? If someone has time, >> > they’d >> > probably pick it up >> > >> > Thanks, >> > Hari >> > >> > >> > On Mon, Feb 2, 2015 at 10:52 AM, Charles McLaughlin >> > <[email protected]> >> > wrote: >> >> >> >> Hello, >> >> >> >> We had a situation where one of our Flume agents got stuck on a message >> >> due to unexpected format. To get things moving again, I stopped the >> >> Flume >> >> agent, moved the file backed channel data out of the way and re-started >> >> the >> >> Flume agent. I'd like to pop the bad message from the queue data on >> >> disk... >> >> are there any tools or recommended ways to do this? >> >> >> >> Thanks, >> >> Charles >> > >> > >> >> >> >> -- >> thanks >> ashish >> >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal > > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
