+1 for idea, +1 for feature contribution, and +1 for design proposal reviewing.
Thanks.
BR,
Shang
On 2/1/17, 7:38 AM, "Saikat Kanjilal" <sxk1...@hotmail.com> wrote:
I like this idea and would be interested in contributing features and
testing, am guessing the first step is to review the design proposal if I'm not
mistaken, let me know.
________________________________
From: Tristan Stevens <tris...@cloudera.com>
Sent: Wednesday, February 1, 2017 12:09 AM
To: dev@flume.apache.org
Subject: REST API Proposal
Hi all,
I'd like to put forward a proposal that I've been considering based on
conversations with users and on observations of some threads on this
mailing list.
My proposal is that we build into Flume a REST API that would give
administrators greater control over a running instance of Flume. Basically
I'm thinking of the following features:
- Allow browsing of status and configuration of all components (Sources,
Channels, Sinks).
- Allow starting and stopping of individual components.
- Allow deployment of new components (Sources, Channels, Sinks) into a
running agent.
- Allow modification of configuration of deployed components (Sources,
Channels, Sinks).
- Allow modification of log4j configuration of a running instance
(FLUME-3038).
Overall long-term goal: Eliminate the need for routine administration of
Flume via command-line during a dev-cycle and increase the
supportability/administerability of Flume in general.
In terms of benefits, my thinking is as follows:
- Granular visibility of component statuses.
- Graceful shutdown of agent (e.g. shut down Sources, allow Channels to
drain, and then shut down Sinks) (I think there's a JIRA kicking around for
this)
- Failure scenario management:
- Enable re-pointing of Sinks (e.g. because of downstream issues)
without interrupting Sources or losing events in Channels.
- Re-configuring channels or sinks in order to improve performance.
- Add sinks to running instance in order to relieve pressure on
over-full channel.
- Improve developer experience by allowing for dynamic (re)configuration
of agent without using the command-line and without needing to restart the
whole process.
- Significantly lower the barrier to adoption for both developers and
administrators.
There is also the possibility that we could then support third party
tooling for building interactive web UIs on top of Flume, which would
greatly improve usability for both administrators and also developers (e.g.
configurators).
I've knocked together a bit of a design proposal which I've made available
at:
https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing
[https://lh3.googleusercontent.com/mBCEYs5pz2OADkkd_YGpFpQYlg8uwAF02FJ59dnq70DgqwoBhf4JL7H3PR_ZS-jHQ71VRg=w1200-h630-p]<https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing>
flume-rest-proposal<https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing>
docs.google.com
Design Proposal: Flume REST API This proposal details the design rationale
for a REST API onto Flume. Current Architecture As currently implemented, the
ability for an administrator to interact with a running instance of Flume is
very limited The overall scope of a REST API is to provide a mech...
Please
add specific comments inline in the doc and general comments back to this
thread.
My question to the group is threefold:
1. Is this something that we think is a) worthwhile and b) achievable?
2. I'm happy to lead the development, but is there a committer who can
offer time to review and support?
3. Would anyone else be interested in contributing features or testing?
Many thanks,
Tristan
