You could implement a deserializer that reads compressed files. You’d have to decompress the file and return the data event by event though. Take a look at something like the BlobSerializer as an example (it is a pretty simple one that demostrates the behavior).
If you shutdown the executor which you started in the start() method, and you shut it down in the stop method, you probably will get an exception from the Executor framework complaining that the executor was shutdown and not a channel exception. If you get a channel exception it means that the channel is full or can’t accept events right now for some other reason — not that the source can’t handle it. If that is the case, the bug is in the source implementation. Do you see that behavior in a source bundled with flume? Thanks, Hari On Mon, Nov 10, 2014 at 4:01 AM, Sverre Bakke <[email protected]> wrote: > Hi, > When creating a new EventDrivenSource running as an executor, what is > the correct approach to handling shutdown gracefully? > I am writing a custom source that will poll a compressed file line by > line using BufferedReader and pushing these lines to a > ChannelProcessor using processEvent(). This as a result of Spooling > Directory not supporting compressed files. This also means that most > of the time, my Flume source will be blocking on > BufferedReader.readLine() or blocking on > ChannelProcessor.processEvent(). > If I shutdown the executor from the stop() method of my source, the > typical response from Flume will be that the ChannelProcessor will > generate a ChannelException. In what situations can I expect that the > ChannelException actually is the result of a shutdown (e.g. ctrl+c) > rather than some other issue that should be handled as a truly > exceptional situation/error? Or am I approaching graceful shutdown > completely wrong? > Is there any specific order in which the Flume sources, interceptors > and sinks are signaled to shut down? > I feel that when it comes to error handling (and shutdowns), the > developer guide and javadoc is a bit lacking unfortunately. > Regards, > Sverre
