Thanks for sending this out, Tristan. Sorry for my late response. I added a couple of comments to the doc. One missing piece is how do the configuration changes get persisted (or do they?)
Due to other commitments it would be difficult for me to help out very much. However, it may be possible for the PMC to grant access for a feature branch to use for collaboration. Mike On Wed, Feb 1, 2017 at 12:09 AM, Tristan Stevens <tris...@cloudera.com> wrote: > Hi all, > I'd like to put forward a proposal that I've been considering based on > conversations with users and on observations of some threads on this > mailing list. > > My proposal is that we build into Flume a REST API that would give > administrators greater control over a running instance of Flume. Basically > I'm thinking of the following features: > - Allow browsing of status and configuration of all components (Sources, > Channels, Sinks). > - Allow starting and stopping of individual components. > - Allow deployment of new components (Sources, Channels, Sinks) into a > running agent. > - Allow modification of configuration of deployed components (Sources, > Channels, Sinks). > - Allow modification of log4j configuration of a running instance > (FLUME-3038). > > Overall long-term goal: Eliminate the need for routine administration of > Flume via command-line during a dev-cycle and increase the > supportability/administerability of Flume in general. > > In terms of benefits, my thinking is as follows: > > - Granular visibility of component statuses. > - Graceful shutdown of agent (e.g. shut down Sources, allow Channels to > drain, and then shut down Sinks) (I think there's a JIRA kicking around for > this) > - Failure scenario management: > - Enable re-pointing of Sinks (e.g. because of downstream issues) > without interrupting Sources or losing events in Channels. > - Re-configuring channels or sinks in order to improve performance. > - Add sinks to running instance in order to relieve pressure on > over-full channel. > - Improve developer experience by allowing for dynamic (re)configuration > of agent without using the command-line and without needing to restart the > whole process. > - Significantly lower the barrier to adoption for both developers and > administrators. > > There is also the possibility that we could then support third party > tooling for building interactive web UIs on top of Flume, which would > greatly improve usability for both administrators and also developers (e.g. > configurators). > > I've knocked together a bit of a design proposal which I've made available > at: > https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7R > P1wIhA8/edit?usp=sharing > Please > add specific comments inline in the doc and general comments back to this > thread. > > My question to the group is threefold: > > 1. Is this something that we think is a) worthwhile and b) achievable? > 2. I'm happy to lead the development, but is there a committer who can > offer time to review and support? > 3. Would anyone else be interested in contributing features or testing? > > Many thanks, > Tristan >
