I love this. I have a situation where I need this capability and the only way to do it is to change the configuration and restart. Being able to enable/disable a channel and sink dynamically would be great.
FWIW, I would expect that this could also be done through the existing JMX support. Ralph > On Feb 1, 2017, at 1:09 AM, Tristan Stevens <tris...@cloudera.com> wrote: > > Hi all, > I'd like to put forward a proposal that I've been considering based on > conversations with users and on observations of some threads on this > mailing list. > > My proposal is that we build into Flume a REST API that would give > administrators greater control over a running instance of Flume. Basically > I'm thinking of the following features: > - Allow browsing of status and configuration of all components (Sources, > Channels, Sinks). > - Allow starting and stopping of individual components. > - Allow deployment of new components (Sources, Channels, Sinks) into a > running agent. > - Allow modification of configuration of deployed components (Sources, > Channels, Sinks). > - Allow modification of log4j configuration of a running instance > (FLUME-3038). > > Overall long-term goal: Eliminate the need for routine administration of > Flume via command-line during a dev-cycle and increase the > supportability/administerability of Flume in general. > > In terms of benefits, my thinking is as follows: > > - Granular visibility of component statuses. > - Graceful shutdown of agent (e.g. shut down Sources, allow Channels to > drain, and then shut down Sinks) (I think there's a JIRA kicking around for > this) > - Failure scenario management: > - Enable re-pointing of Sinks (e.g. because of downstream issues) > without interrupting Sources or losing events in Channels. > - Re-configuring channels or sinks in order to improve performance. > - Add sinks to running instance in order to relieve pressure on > over-full channel. > - Improve developer experience by allowing for dynamic (re)configuration > of agent without using the command-line and without needing to restart the > whole process. > - Significantly lower the barrier to adoption for both developers and > administrators. > > There is also the possibility that we could then support third party > tooling for building interactive web UIs on top of Flume, which would > greatly improve usability for both administrators and also developers (e.g. > configurators). > > I've knocked together a bit of a design proposal which I've made available > at: > https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing > Please > add specific comments inline in the doc and general comments back to this > thread. > > My question to the group is threefold: > > 1. Is this something that we think is a) worthwhile and b) achievable? > 2. I'm happy to lead the development, but is there a committer who can > offer time to review and support? > 3. Would anyone else be interested in contributing features or testing? > > Many thanks, > Tristan
