We've written a simple sink decorator to do in-memory aggregations. Currently, we're using a roll sink to cause the aggregator decorator to be closed/reopened ever 60 seconds. Based upon the info in [1], by default the close() operation has 30 seconds to complete. We're seeing this fail in some cases due to other bottlenecks. I'm hesitant to just up the timeout, though, since long GCs or other events could cause the problem regardless of the timeout.
With all this in mind, I have two questions. 1) Rollsink and BatchingDecorator seem to share a lot of similar logic to run a background thread to flush events periodically. There seems to be a lot of subtly in these implementations to avoid deadlocks. Are either of these suitable for subclassing? (I guess BatchingDecorator is closer to what I'm looking for)... has anyone ever done this before? 2) It's possible for our sink decorator to generate more events than it receives, so I am afraid it could become behind -- are there dangers in using a threadpool to call append() from a decorator to forward events to the collector? Thanks, Joe [1] http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_buffered_sink_and_decorator_semantics
