I like this idea and would be interested in contributing features and testing, am guessing the first step is to review the design proposal if I'm not mistaken, let me know.
________________________________ From: Tristan Stevens <tris...@cloudera.com> Sent: Wednesday, February 1, 2017 12:09 AM To: dev@flume.apache.org Subject: REST API Proposal Hi all, I'd like to put forward a proposal that I've been considering based on conversations with users and on observations of some threads on this mailing list. My proposal is that we build into Flume a REST API that would give administrators greater control over a running instance of Flume. Basically I'm thinking of the following features: - Allow browsing of status and configuration of all components (Sources, Channels, Sinks). - Allow starting and stopping of individual components. - Allow deployment of new components (Sources, Channels, Sinks) into a running agent. - Allow modification of configuration of deployed components (Sources, Channels, Sinks). - Allow modification of log4j configuration of a running instance (FLUME-3038). Overall long-term goal: Eliminate the need for routine administration of Flume via command-line during a dev-cycle and increase the supportability/administerability of Flume in general. In terms of benefits, my thinking is as follows: - Granular visibility of component statuses. - Graceful shutdown of agent (e.g. shut down Sources, allow Channels to drain, and then shut down Sinks) (I think there's a JIRA kicking around for this) - Failure scenario management: - Enable re-pointing of Sinks (e.g. because of downstream issues) without interrupting Sources or losing events in Channels. - Re-configuring channels or sinks in order to improve performance. - Add sinks to running instance in order to relieve pressure on over-full channel. - Improve developer experience by allowing for dynamic (re)configuration of agent without using the command-line and without needing to restart the whole process. - Significantly lower the barrier to adoption for both developers and administrators. There is also the possibility that we could then support third party tooling for building interactive web UIs on top of Flume, which would greatly improve usability for both administrators and also developers (e.g. configurators). I've knocked together a bit of a design proposal which I've made available at: https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing [https://lh3.googleusercontent.com/mBCEYs5pz2OADkkd_YGpFpQYlg8uwAF02FJ59dnq70DgqwoBhf4JL7H3PR_ZS-jHQ71VRg=w1200-h630-p]<https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing> flume-rest-proposal<https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing> docs.google.com Design Proposal: Flume REST API This proposal details the design rationale for a REST API onto Flume. Current Architecture As currently implemented, the ability for an administrator to interact with a running instance of Flume is very limited The overall scope of a REST API is to provide a mech... Please add specific comments inline in the doc and general comments back to this thread. My question to the group is threefold: 1. Is this something that we think is a) worthwhile and b) achievable? 2. I'm happy to lead the development, but is there a committer who can offer time to review and support? 3. Would anyone else be interested in contributing features or testing? Many thanks, Tristan
