Re: Delete individual message from queue

2015-02-09 Thread Hari Shreedharan
Sounds good! Thanks! Thanks, Hari On Mon, Feb 9, 2015 at 6:42 PM, Ashish wrote: > I did tried it, but was lost a bit on getting an Event from > TransactionalEvent. I shall work out the details, meanwhile you can > help me out with the Event part. > thanks > ashish > On Tue, Feb 10, 2015 at 5

Re: Delete individual message from queue

2015-02-09 Thread Ashish
I did tried it, but was lost a bit on getting an Event from TransactionalEvent. I shall work out the details, meanwhile you can help me out with the Event part. thanks ashish On Tue, Feb 10, 2015 at 5:56 AM, Hari Shreedharan wrote: > Correct - that would be pretty tricky. We could indeed modify

Re: Delete individual message from queue

2015-02-09 Thread Hari Shreedharan
Correct - that would be pretty tricky. We could indeed modify the tool to take a custom function to process each event - that would work. We must specify an interface that the user must implement, say, FileChannelDataVerifier or something. We then call this on each event.  Do you want to tak

Re: Delete individual message from queue

2015-02-04 Thread Ashish
I think I was not clear. I was taking about an offline tool which would help us clean the Channel with valid Flume Event, with invalid/malformed payload. What you described would work in Agent flow. Given the current scenario, Charles would be keen to push the backed up events to destination. Here

Re: Delete individual message from queue

2015-02-04 Thread Hari Shreedharan
It is not as easy, since the channel does not know what an event looks like or why the transaction is being rolled back. That is something that is being handled by the serializer and sink. We need to some how remove an event from the channel if it the channel is broken. We might perhaps have to

Re: Delete individual message from queue

2015-02-04 Thread Ashish
Is it possible to extend File Channel Integrity tool to support filtering out corrupt events? Something like once we get a record and it's not corrupt, provide a placeholder function to validate Event implemented by user. I still don't have too much insight into FileChannel implementation. On Tue,

Re: Delete individual message from queue

2015-02-02 Thread Charles McLaughlin
Ok, thanks for the response. I created this ticket: https://issues.apache.org/jira/browse/FLUME-2613 On Mon, Feb 2, 2015 at 11:19 AM, Hari Shreedharan wrote: > Currently, no - there is no such tool, but this is a request that has come > up time and again. Can you file a jira for this? If some

Re: Delete individual message from queue

2015-02-02 Thread Hari Shreedharan
Currently, no - there is no such tool, but this is a request that has come up time and again. Can you file a jira for this? If someone has time, they’d probably pick it up Thanks, Hari On Mon, Feb 2, 2015 at 10:52 AM, Charles McLaughlin wrote: > Hello, > We had a situation where one of our

Delete individual message from queue

2015-02-02 Thread Charles McLaughlin
Hello, We had a situation where one of our Flume agents got stuck on a message due to unexpected format. To get things moving again, I stopped the Flume agent, moved the file backed channel data out of the way and re-started the Flume agent. I'd like to pop the bad message from the queue data on