Gosh I hate being two hours behind you guys! :-)
I’ll go back through the thread and collect open questions, but wanted to put a
word in about Zookeeper integration.
I had been about to ask what the benefits of using ZK are, and I’ve now heard
two features:
- A logical “single place” that is visible and efficiently accessible by all
processes on all nodes.
- Supports async notification, and therefore updates without restarts.
If only the first point was there, it could be replaced by Ambari, because
Ambari manages propagating configs so they look local to all nodes. And btw,
there’s no bottlenecking, because clients don’t call the Ambari REST APIs to
*read* configs, just to *change* them. For reading established configs,
clients just go to those local files, which are managed by the ambari-agents.
But the ability to use ZK and Curator to support async config updates, as we
do, is really important. In fact, I think if we make our use of it consistent
we might offer an example to the Ambari team for a general feature they could
adopt, precisely to support config changes without restart.
Three additional considerations:
1. HBase has always used ZK for various things. I don’t know if that includes
configuration. If so, that’s already integrated with Ambari. We should look
into the details of that.
2. Can folks who’ve been here a while clarify why use of ZK is so piecemeal,
and scattered several places in the znode tree? There seems to be a vague idea
that “things that get changed” go in ZK, while other configs go in local files.
But all configs, by definition, can be changed. Is there any real reason not
to put the whole Metron configuration in ZK, with a clean and consistent
directory structure? It’s okay if Metron cannot actually consume all updates
asynchronously (like some topology configs that require a topo restart if
changed). We just document which configs do and don’t support async change.
HDFS has lots of those.
3. I’m pretty sure we can suppress the “need to restart” warning from Ambari.
We’ll need to dig in to find out how flexible this is.
I do think we should continue supporting non-Ambari use, and if we put all
configs in ZK, that gets way easier to do in a simple and consistent way.
(Propagation problem solved). More thoughts after I have ‘em :-)
Thanks,
--Matt
On 1/13/17, 8:30 AM, "Casey Stella" <ceste...@gmail.com> wrote:
I think that looks good. One last question, do we support the manual
install use-case (one where ambari does not exist, I mean)?
Casey
On Fri, Jan 13, 2017 at 11:28 AM, David Lyle <dlyle65...@gmail.com> wrote:
> That's good feedback, Jon. I think that puts us at:
>
> - Expand ambari to manage the remaining sensor-specific configs
> - Refactor the push calls to zookeeper (in ConfigurationUtils, I think)
> to push to ambari and take an Ambari user/pw and (optionally) reason
> - We shall retain current functionality wrt live configuration changes.
> Suggestion- ConfigurationUtils will push to both zookeeper and Ambari in
an
> atomic operation. (I suspect we can make ambari do this as well)
> - Refactor the middleware that Ryan submitted to have the API calls take
> an Ambari user/pw and (optionally) reason
> - Refactor the management UI to pass in an Ambari user/pw and
(optionally)
> reason
> - Refactor the Stellar Management functions CONFIG_PUT to accept an
Ambari
> user/pw and (optionally) reason
>
> -D...
>
>
> On Fri, Jan 13, 2017 at 11:17 AM, Nick Allen <n...@nickallen.org> wrote:
>
> > +1 I strongly agree with Jon's view. Requiring a restart would be a
> big
> > step backwards.
> >
> > I think the power of the platform is that the user can act on live
> > streaming data in a quick, iterative fashion. Adding enrichments,
> creating
> > triage rules, adjusting profiles are all operational activities that can
> be
> > performed at any time in response to active threats.
> >
> >
> >
> >
> > On Fri, Jan 13, 2017 at 10:59 AM, zeo...@gmail.com <zeo...@gmail.com>
> > wrote:
> >
> > > Right, good conversation to bring up for sure.
> > >
> > > Just to comment on production generally only being updated during
> > > maintenance windows - I can tell you that my plans are to make my dev,
> > > test, and prod Metron a very dynamic and frequently changing
> environment
> > > which will have coordinated but frequent modifications and I strongly
> > > prefer not having to restart anywhere that I can. Of course it will
> > > happen, but keeping it to a minimum is key.
> > >
> > > Jon
> > >
> > > On Fri, Jan 13, 2017 at 10:53 AM Nick Allen <n...@nickallen.org>
> wrote:
> > >
> > > > Makes sense, Dave. I am totally clear on the proposal. I just
> wanted
> > to
> > > > ask the stupid question to bring the conversation full circle, leave
> no
> > > > stone unturned, insert favorite idiom here.
> > > >
> > > > On Fri, Jan 13, 2017 at 10:46 AM, David Lyle <dlyle65...@gmail.com>
> > > wrote:
> > > >
> > > > > To be clear- NOBODY is suggesting replacing Zookeeper with Ambari.
> > > > >
> > > > > So, as a bit of a reset- here's what's being proposed:
> > > > >
> > > > > - Expand ambari to manage the remaining sensor-specific configs
> > > > > - Refactor the push calls to zookeeper (in ConfigurationUtils, I
> > > think)
> > > > > to push to ambari and take an Ambari user/pw and (optionally)
> > reason
> > > > > - (Ambari can push to zookeeper, but it requires a service
> restart,
> > so
> > > > for
> > > > > "live changes" you may
> > > > > want do both a rest call and zookeeper update from
> > > > ConfigurationUtils)
> > > > > WAS
> > > > > Question remains about whether ambari can do the push to
> > zookeeper
> > > > > or whetheror whether ConfigurationUtils has to push to
> zookeeper
> > as
> > > > > well as update
> > > > > ambari.
> > > > > - Refactor the middleware that Ryan submitted to have the API
> calls
> > > > take
> > > > > an Ambari user/pw and (optionally) reason
> > > > > - Refactor the management UI to pass in an Ambari user/pw and
> > > > > (optionally) reason
> > > > > - Refactor the Stellar Management functions CONFIG_PUT to accept
> an
> > > > > Ambari user/pw and (optionally) reason
> > > > >
> > > > > -D...
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Jan 13, 2017 at 10:42 AM, Ryan Merriman <
> merrim...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > The driver for using Zookeeper is that it is asynchronous and
> > accepts
> > > > > > callbacks. Ambari would need to have that capability, otherwise
> we
> > > > have
> > > > > to
> > > > > > poll which is a deal breaker in my opinion.
> > > > > >
> > > > > > On Fri, Jan 13, 2017 at 9:28 AM, Casey Stella <
> ceste...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > No, it was good to bring up, Nick. I might have it wrong re:
> > > Ambari.
> > > > > > >
> > > > > > > Casey
> > > > > > >
> > > > > > > On Fri, Jan 13, 2017 at 10:27 AM, Nick Allen <
> n...@nickallen.org
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > That makes sense. I wasn't sure based on Matt's original
> > > > > > > > suggestion/description of Ambari, whether that was something
> > that
> > > > > > Ambari
> > > > > > > > had also designed for or not.
> > > > > > > >
> > > > > > > > On Fri, Jan 13, 2017 at 10:14 AM, Casey Stella <
> > > ceste...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Polling the Ambari server via REST (or their API if they
> have
> > > > one),
> > > > > > > would
> > > > > > > > > entail all workers hitting one server and create a single
> > point
> > > > of
> > > > > > > > failure
> > > > > > > > > (the ambari server is what serves up REST). Zookeeper's
> > intent
> > > > is
> > > > > to
> > > > > > > not
> > > > > > > > > have a single point of failure like this and (one of its
> > main)
> > > > > > > use-cases
> > > > > > > > is
> > > > > > > > > to serve up configs in a distributed environment.
> > > > > > > > >
> > > > > > > > > Casey
> > > > > > > > >
> > > > > > > > > On Fri, Jan 13, 2017 at 9:55 AM, Nick Allen <
> > > n...@nickallen.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Let me ask a stupid question. What does Zookeeper do
for
> > us
> > > > that
> > > > > > > > Ambari
> > > > > > > > > > cannot? Why keep Zookeeper in the mix?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Jan 13, 2017 at 9:28 AM, David Lyle <
> > > > > dlyle65...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > In the main yes- I've made some changes:
> > > > > > > > > > >
> > > > > > > > > > > - Expand ambari to manage the remaining
> sensor-specific
> > > > > configs
> > > > > > > > > > > - Refactor the push calls to zookeeper (in
> > > > > ConfigurationUtils, I
> > > > > > > > > think)
> > > > > > > > > > > to push to ambari and take an Ambari user/pw and
> > > > > (optionally)
> > > > > > > > reason
> > > > > > > > > > > - (Ambari can push to zookeeper, but it requires a
> > service
> > > > > > > restart,
> > > > > > > > so
> > > > > > > > > > for
> > > > > > > > > > > "live changes" you may
> > > > > > > > > > > want do both a rest call and zookeeper update from
> > > > > > > > > > ConfigurationUtils)
> > > > > > > > > > > WAS
> > > > > > > > > > > Question remains about whether ambari can do the
> push
> > > to
> > > > > > > > zookeeper
> > > > > > > > > > > or whetheror whether ConfigurationUtils has to
push
> > to
> > > > > > > zookeeper
> > > > > > > > as
> > > > > > > > > > > well as update
> > > > > > > > > > > ambari.
> > > > > > > > > > > - Refactor the middleware that Ryan submitted to
have
> > the
> > > > API
> > > > > > > calls
> > > > > > > > > > take
> > > > > > > > > > > an Ambari user/pw and (optionally) reason
> > > > > > > > > > > - Refactor the management UI to pass in an Ambari
> > user/pw
> > > > and
> > > > > > > > > > > (optionally) reason
> > > > > > > > > > > - Refactor the Stellar Management functions
> CONFIG_PUT
> > to
> > > > > > accept
> > > > > > > an
> > > > > > > > > > > Ambari user/pw and (optionally) reason
> > > > > > > > > > >
> > > > > > > > > > > I think we'd need to do some detailed design around
how
> > to
> > > > > handle
> > > > > > > > what
> > > > > > > > > we
> > > > > > > > > > > expect to be dynamic configs, but the main principle
> > should
> > > > > (imo)
> > > > > > > be
> > > > > > > > to
> > > > > > > > > > > always know who and why and make sure that Ambari is
> > aware
> > > > and
> > > > > is
> > > > > > > the
> > > > > > > > > > > static backing store for Zookeeper.
> > > > > > > > > > >
> > > > > > > > > > > -D...
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jan 13, 2017 at 9:19 AM, Casey Stella <
> > > > > > ceste...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > So, basically, your proposed changes, broken into
> > > tangible
> > > > > > > gobbets
> > > > > > > > of
> > > > > > > > > > > work:
> > > > > > > > > > > >
> > > > > > > > > > > > - Expand ambari to manage the remaining
> > > sensor-specific
> > > > > > > configs
> > > > > > > > > > > > - Refactor the push calls to zookeeper (in
> > > > > > > ConfigurationUtils, I
> > > > > > > > > > > think)
> > > > > > > > > > > > to push to ambari and take a reason
> > > > > > > > > > > > - Question remains about whether ambari can do
> > the
> > > > push
> > > > > > to
> > > > > > > > > > > zookeeper
> > > > > > > > > > > > or whether ConfigurationUtils has to push to
> > > > zookeeper
> > > > > as
> > > > > > > > well
> > > > > > > > > as
> > > > > > > > > > > > update
> > > > > > > > > > > > ambari.
> > > > > > > > > > > > - Refactor the middleware that Ryan submitted to
> > have
> > > > the
> > > > > > API
> > > > > > > > > calls
> > > > > > > > > > > take
> > > > > > > > > > > > a reason
> > > > > > > > > > > > - Refactor the management UI to pass in a reason
> > > > > > > > > > > > - Refactor the Stellar Management functions
> > CONFIG_PUT
> > > > to
> > > > > > > > accept a
> > > > > > > > > > > > reason
> > > > > > > > > > > >
> > > > > > > > > > > > Just so we can evaluate it and I can ensure I
haven't
> > > > > > overlooked
> > > > > > > > some
> > > > > > > > > > > > important point. Please tell me if Ambari cannot do
> > the
> > > > > things
> > > > > > > > we're
> > > > > > > > > > > > suggesting it can do.
> > > > > > > > > > > >
> > > > > > > > > > > > Casey
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:15 AM, David Lyle <
> > > > > > > dlyle65...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > That's exactly correct, Casey. Basically, an
> > expansion
> > > of
> > > > > > what
> > > > > > > > > we're
> > > > > > > > > > > > > currently doing with global.json,
> > enrichment.properties
> > > > and
> > > > > > > > > > > > > elasticsearch.properties.
> > > > > > > > > > > > >
> > > > > > > > > > > > > -D...
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:12 AM, Casey Stella <
> > > > > > > > ceste...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I would suggest not having Ambari replace
> > > zookeeper. I
> > > > > > think
> > > > > > > > the
> > > > > > > > > > > > > proposal
> > > > > > > > > > > > > > is to have Ambari replace the editable store
> (like
> > > the
> > > > > JSON
> > > > > > > > files
> > > > > > > > > > on
> > > > > > > > > > > > > > disk). Zookeeper woudl be the source of truth
> for
> > > the
> > > > > > > running
> > > > > > > > > > > > topologies
> > > > > > > > > > > > > > and ambari would be sync'd to it.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Correct if I misspeak, dave or matt.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Casey
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:09 AM, Nick Allen <
> > > > > > > > n...@nickallen.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Ambari seems like a logical choice.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *>> It doesn’t natively integrate Zookeeper
> > storage
> > > > of
> > > > > > > > configs,
> > > > > > > > > > but
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > > is a natural place to specify copy to/from
> > > Zookeeper
> > > > > for
> > > > > > > the
> > > > > > > > > > files
> > > > > > > > > > > > > > > desired.*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > How would Ambari interact with Zookeeper in
> this
> > > > > > scenario?
> > > > > > > > > Would
> > > > > > > > > > > > > Ambari
> > > > > > > > > > > > > > > replace Zookeeper completely? Or would
> Zookeeper
> > > act
> > > > as
> > > > > > the
> > > > > > > > > > > > persistence
> > > > > > > > > > > > > > > tier under Ambari?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Jan 12, 2017 at 9:24 PM, Matt Foley <
> > > > > > > > ma...@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Mike, could you try again on the image,
> please,
> > > > > making
> > > > > > > sure
> > > > > > > > > it
> > > > > > > > > > > is a
> > > > > > > > > > > > > > > simple
> > > > > > > > > > > > > > > > format (gif, png, or jpeg)? It got munched,
> at
> > > > least
> > > > > > in
> > > > > > > my
> > > > > > > > > > > viewer.
> > > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Casey, responding to some of the questions
> you
> > > > > raised:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I’m going to make a rather strong statement:
> > We
> > > > > > already
> > > > > > > > > have a
> > > > > > > > > > > > > service
> > > > > > > > > > > > > > > > “to intermediate and handle config
> > > > update/retrieval”.
> > > > > > > > > > > > > > > > Furthermore, it:
> > > > > > > > > > > > > > > > - Correctly handles the problems of
> distributed
> > > > > > services
> > > > > > > > > > running
> > > > > > > > > > > on
> > > > > > > > > > > > > > > > multi-node clusters. (That’s a HARD
problem,
> > > > people,
> > > > > > and
> > > > > > > > we
> > > > > > > > > > > > > shouldn’t
> > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > > to reinvent the wheel.)
> > > > > > > > > > > > > > > > - Correctly handles Kerberos security.
> (That’s
> > > > kinda
> > > > > > hard
> > > > > > > > > too,
> > > > > > > > > > or
> > > > > > > > > > > > at
> > > > > > > > > > > > > > > least
> > > > > > > > > > > > > > > > a lot of work.)
> > > > > > > > > > > > > > > > - It does automatic versioning of
> > configurations,
> > > > and
> > > > > > > > allows
> > > > > > > > > > > > viewing,
> > > > > > > > > > > > > > > > comparing, and reverting historical configs
> > > > > > > > > > > > > > > > - It has a capable REST API for all those
> > things.
> > > > > > > > > > > > > > > > It doesn’t natively integrate Zookeeper
> storage
> > > of
> > > > > > > configs,
> > > > > > > > > but
> > > > > > > > > > > > there
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > natural place to specify copy to/from
> Zookeeper
> > > for
> > > > > the
> > > > > > > > files
> > > > > > > > > > > > > desired.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It is Ambari. And we should commit to it,
> > rather
> > > > > than
> > > > > > > try
> > > > > > > > to
> > > > > > > > > > > > > re-create
> > > > > > > > > > > > > > > > such features.
> > > > > > > > > > > > > > > > Because it has a good REST API, it is
> perfectly
> > > > > > feasible
> > > > > > > to
> > > > > > > > > > > > implement
> > > > > > > > > > > > > > > > Stellar functions that call it.
> > > > > > > > > > > > > > > > GUI configuration tools can also use the
> Ambari
> > > > APIs,
> > > > > > or
> > > > > > > > > better
> > > > > > > > > > > yet
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > integrated in an “Ambari View”. (Eg, see the
> > > “Yarn
> > > > > > > Capacity
> > > > > > > > > > > > Scheduler
> > > > > > > > > > > > > > > > Configuration Tool” example in the Ambari
> > > > > > documentation,
> > > > > > > > > under
> > > > > > > > > > > > “Using
> > > > > > > > > > > > > > > > Ambari Views”.)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Arguments are: Parsimony, Sufficiency, Not
> > > > > reinventing
> > > > > > > the
> > > > > > > > > > wheel,
> > > > > > > > > > > > and
> > > > > > > > > > > > > > Not
> > > > > > > > > > > > > > > > spending weeks and weeks of developer time
> over
> > > the
> > > > > > next
> > > > > > > > year
> > > > > > > > > > > > > > reinventing
> > > > > > > > > > > > > > > > the wheel while getting details wrong
> multiple
> > > > times…
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Okay, off soapbox.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Casey asked what the config update behavior
> of
> > > > Ambari
> > > > > > is,
> > > > > > > > and
> > > > > > > > > > how
> > > > > > > > > > > > it
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > interact with changes made from outside
> Ambari.
> > > > > > > > > > > > > > > > The following is from my experience working
> > with
> > > > the
> > > > > > > Ambari
> > > > > > > > > > Mpack
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > Metron. I am not otherwise an Ambari
expert,
> > so
> > > > > > tomorrow
> > > > > > > > > I’ll
> > > > > > > > > > > get
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > > reviewed by an Ambari development engineer.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Ambari-server runs on one node, and
> > Ambari-agent
> > > > runs
> > > > > > on
> > > > > > > > each
> > > > > > > > > > of
> > > > > > > > > > > > all
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > nodes.
> > > > > > > > > > > > > > > > Ambari-server has a private set of py, xml,
> and
> > > > > > template
> > > > > > > > > files,
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > > together are used both to generate the
Ambari
> > > > > > > configuration
> > > > > > > > > > GUI,
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > > defaults, and to generate configuration
files
> > (of
> > > > any
> > > > > > > > needed
> > > > > > > > > > > > > filetype)
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the various Stack components.
> > > > > > > > > > > > > > > > Ambari-server also has a database where it
> > stores
> > > > the
> > > > > > > > schema
> > > > > > > > > > > > related
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > these files, so even if you reach in and
edit
> > > > > Ambari’s
> > > > > > > > files,
> > > > > > > > > > it
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > Error
> > > > > > > > > > > > > > > > out if the set of parameters or parameter
> names
> > > > > > changes.
> > > > > > > > The
> > > > > > > > > > > > > > historical
> > > > > > > > > > > > > > > > information about configuration changes is
> also
> > > > > stored
> > > > > > in
> > > > > > > > the
> > > > > > > > > > db.
> > > > > > > > > > > > > > > > For each component (and in the case of
> Metron,
> > > for
> > > > > each
> > > > > > > > > > > topology),
> > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > is a python file which controls the logic
for
> > > these
> > > > > > > > actions,
> > > > > > > > > > > among
> > > > > > > > > > > > > > > others:
> > > > > > > > > > > > > > > > - Install
> > > > > > > > > > > > > > > > - Start / stop / restart / status
> > > > > > > > > > > > > > > > - Configure
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It is actually up to this python code (which
> we
> > > > wrote
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > > Metron
> > > > > > > > > > > > > > > > Mpack) what happens in each of these API
> calls.
> > > > But
> > > > > > the
> > > > > > > > > > current
> > > > > > > > > > > > > code,
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > I believe this is typical of Ambari-managed
> > > > > components,
> > > > > > > > > > performs
> > > > > > > > > > > a
> > > > > > > > > > > > > > > > “Configure” action whenever you press the
> > “Save”
> > > > > button
> > > > > > > > after
> > > > > > > > > > > > > changing
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > component config in Ambari, and also on each
> > > > Install
> > > > > > and
> > > > > > > > > Start
> > > > > > > > > > or
> > > > > > > > > > > > > > > Restart.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The Configure action consists of
> approximately
> > > the
> > > > > > > > following
> > > > > > > > > > > > sequence
> > > > > > > > > > > > > > > (see
> > > > > > > > > > > > > > > > disclaimer above :-)
> > > > > > > > > > > > > > > > - Recreate the generated config files, using
> > the
> > > > > > template
> > > > > > > > > files
> > > > > > > > > > > and
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > actual configuration most recently set in
> > Ambari
> > > > > > > > > > > > > > > > o Note this is also under the control of
> python
> > > > code
> > > > > > that
> > > > > > > > we
> > > > > > > > > > > wrote,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > this is the appropriate place to push to ZK
> if
> > > > > desired.
> > > > > > > > > > > > > > > > - Propagate those config files to each
> > > > Ambari-agent,
> > > > > > > with a
> > > > > > > > > > > command
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > set
> > > > > > > > > > > > > > > > them locally
> > > > > > > > > > > > > > > > - The ambari-agents on each node receive the
> > > files
> > > > > and
> > > > > > > > write
> > > > > > > > > > them
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > specified locations on local storage
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Ambari-server then whines that the updated
> > > services
> > > > > > > should
> > > > > > > > be
> > > > > > > > > > > > > > restarted,
> > > > > > > > > > > > > > > > but does not initiate that action itself
> > (unless
> > > of
> > > > > > > course
> > > > > > > > > the
> > > > > > > > > > > > > > initiating
> > > > > > > > > > > > > > > > action was a Start command from the
> > > administrator).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Make sense? It’s all quite straightforward
> in
> > > > > concept,
> > > > > > > > > there’s
> > > > > > > > > > > > just
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > awful lot of stuff wrapped around that to
> make
> > it
> > > > all
> > > > > > go
> > > > > > > > > > smoothly
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > handle the problems when it doesn’t.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There’s additional complexity in that the
> > > > > Ambari-agent
> > > > > > > also
> > > > > > > > > > > caches
> > > > > > > > > > > > > (on
> > > > > > > > > > > > > > > > each node) both the template files and
> COMPILED
> > > > forms
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > python
> > > > > > > > > > > > > > files
> > > > > > > > > > > > > > > > (.pyc) involved in transforming them. The
> pyc
> > > > files
> > > > > > > > > > incorporate
> > > > > > > > > > > > some
> > > > > > > > > > > > > > > > amount of additional info regarding
parameter
> > > > values,
> > > > > > but
> > > > > > > > I’m
> > > > > > > > > > not
> > > > > > > > > > > > > sure
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the form. I don’t think that changes the
> above
> > > in
> > > > > any
> > > > > > > > > > practical
> > > > > > > > > > > > way
> > > > > > > > > > > > > > > unless
> > > > > > > > > > > > > > > > you’re trying to cheat Ambari by reaching in
> > and
> > > > > > editing
> > > > > > > > its
> > > > > > > > > > > files
> > > > > > > > > > > > > > > > directly. In that case, you also need to
> whack
> > > the
> > > > > pyc
> > > > > > > > files
> > > > > > > > > > (on
> > > > > > > > > > > > > each
> > > > > > > > > > > > > > > > node) to force the data to be reloaded from
> > > > > > > Ambari-server.
> > > > > > > > > > Best
> > > > > > > > > > > > > > solution
> > > > > > > > > > > > > > > > is don’t cheat.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Also, there may be circumstances under which
> > the
> > > > > > > > Ambari-agent
> > > > > > > > > > > will
> > > > > > > > > > > > > > detect
> > > > > > > > > > > > > > > > changes and re-write the latest version it
> > knows
> > > of
> > > > > the
> > > > > > > > > config
> > > > > > > > > > > > files,
> > > > > > > > > > > > > > > even
> > > > > > > > > > > > > > > > without a Save or Start action at the
> > > > Ambari-server.
> > > > > > I’m
> > > > > > > > not
> > > > > > > > > > > sure
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > and need to check with Ambari developers.
It
> > may
> > > > no
> > > > > > > longer
> > > > > > > > > > > happen,
> > > > > > > > > > > > > > altho
> > > > > > > > > > > > > > > > I’m pretty sure change detection/reversion
> was
> > a
> > > > > > feature
> > > > > > > of
> > > > > > > > > > early
> > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > of Ambari.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hope this helps,
> > > > > > > > > > > > > > > > --Matt
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ==============================
> > ==================
> > > > > > > > > > > > > > > > From: Michael Miklavcic <
> > > > michael.miklav...@gmail.com
> > > > > >
> > > > > > > > > > > > > > > > Reply-To: "dev@metron.incubator.apache.org"
> > > > > > > > > > > > > > > <dev@metron.incubator.apache.
> > > > > > > > > > > > > > > > org>
> > > > > > > > > > > > > > > > Date: Thursday, January 12, 2017 at 3:59 PM
> > > > > > > > > > > > > > > > To: "dev@metron.incubator.apache.org"
> > > > > > > > > > > > <dev@metron.incubator.apache.
> > > > > > > > > > > > > org
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] Ambari Metron
> > > Configuration
> > > > > > > > Management
> > > > > > > > > > > > > > > consequences
> > > > > > > > > > > > > > > > and call to action
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Casey,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for starting this thread. I believe
> you
> > > are
> > > > > > > correct
> > > > > > > > in
> > > > > > > > > > > your
> > > > > > > > > > > > > > > > assessment of the 4 options for updating
> > configs
> > > in
> > > > > > > Metron.
> > > > > > > > > > When
> > > > > > > > > > > > > using
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > than one of these options we can get into a
> > > > > split-brain
> > > > > > > > > > > scenario. A
> > > > > > > > > > > > > > basic
> > > > > > > > > > > > > > > > example is updating the global config on
disk
> > and
> > > > > using
> > > > > > > the
> > > > > > > > > > > > > > > > zk_load_configs.sh. Later, if a user decides
> to
> > > > > restart
> > > > > > > > > Ambari,
> > > > > > > > > > > the
> > > > > > > > > > > > > > > cached
> > > > > > > > > > > > > > > > version stored by Ambari (it's in the MySQL
> or
> > > > other
> > > > > > > > database
> > > > > > > > > > > > backing
> > > > > > > > > > > > > > > > Ambari) will be written out to disk in the
> > > defined
> > > > > > config
> > > > > > > > > > > > directory,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > subsequently loaded using the
> > zk_load_configs.sh
> > > > > under
> > > > > > > the
> > > > > > > > > > hood.
> > > > > > > > > > > > Any
> > > > > > > > > > > > > > > global
> > > > > > > > > > > > > > > > configuration modified outside of Ambari
will
> > be
> > > > lost
> > > > > > at
> > > > > > > > this
> > > > > > > > > > > > point.
> > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > is obviously undesirable, but I also like
the
> > > > purpose
> > > > > > and
> > > > > > > > > > utility
> > > > > > > > > > > > > > exposed
> > > > > > > > > > > > > > > > by the multiple config management interfaces
> we
> > > > > > currently
> > > > > > > > > have
> > > > > > > > > > > > > > > available. I
> > > > > > > > > > > > > > > > also agree that a service would be best.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For reference, here's my understanding of
the
> > > > current
> > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > loading mechanisms and their deps.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > <image>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Mike
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Jan 12, 2017 at 3:08 PM, Casey
> Stella <
> > > > > > > > > > > ceste...@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > In the course of discussion on the PR for
> > > > METRON-652
> > > > > > > > > > > > > > > > <
> > > > https://github.com/apache/incubator-metron/pull/415
> > > > > >
> > > > > > > > > > something
> > > > > > > > > > > > > that I
> > > > > > > > > > > > > > > > should definitely have understood better
came
> > to
> > > > > light
> > > > > > > and
> > > > > > > > I
> > > > > > > > > > > > thought
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > it was worth bringing to the attention of
the
> > > > > community
> > > > > > > to
> > > > > > > > > get
> > > > > > > > > > > > > > > > clarification/discuss is just how we manage
> > > > configs.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Currently (assuming the management UI that
> Ryan
> > > > > > Merriman
> > > > > > > > > > > submitted)
> > > > > > > > > > > > > > > configs
> > > > > > > > > > > > > > > > are managed/adjusted via a couple of
> different
> > > > > > mechanism.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - zk_load_utils.sh: pushed and pulled
from
> > > disk
> > > > to
> > > > > > > > > zookeeper
> > > > > > > > > > > > > > > > - Stellar REPL: pushed and pulled via the
> > > > > > > > > > > CONFIG_GET/CONFIG_PUT
> > > > > > > > > > > > > > > > functions
> > > > > > > > > > > > > > > > - Ambari: initialized via the
> zk_load_utils
> > > > script
> > > > > > and
> > > > > > > > > then
> > > > > > > > > > > some
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > them
> > > > > > > > > > > > > > > > are managed directly (global config) and
> > some
> > > > > > > indirectly
> > > > > > > > > > > > > > > > (sensor-specific
> > > > > > > > > > > > > > > > configs).
> > > > > > > > > > > > > > > > - NOTE: Upon service restart, it may
or
> > may
> > > > not
> > > > > > > > > overwrite
> > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > disk or on zookeeper. *Can someone
> more
> > > > > > > > knowledgeable
> > > > > > > > > > than
> > > > > > > > > > > > me
> > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > this describe precisely the semantics
> > that
> > > we
> > > > > can
> > > > > > > > > expect
> > > > > > > > > > on
> > > > > > > > > > > > > > > > service restart
> > > > > > > > > > > > > > > > for Ambari? What gets overwritten on
> disk
> > > and
> > > > > > what
> > > > > > > > gets
> > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > in ambari?*
> > > > > > > > > > > > > > > > - The Management UI: manages some of the
> > > > configs.
> > > > > > > *RYAN:
> > > > > > > > > > Which
> > > > > > > > > > > > > > configs
> > > > > > > > > > > > > > > > do we support here and which don't we
> > support
> > > > > here?*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > As you can see, we have a mishmash of
> > mechanisms
> > > to
> > > > > > > update
> > > > > > > > > and
> > > > > > > > > > > > manage
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > configuration for Metron in zookeeper. In
> the
> > > > > > beginning
> > > > > > > > the
> > > > > > > > > > > > approach
> > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > just to edit configs on disk and push/pull
> them
> > > via
> > > > > > > > > > > zk_load_utils.
> > > > > > > > > > > > > > > Configs
> > > > > > > > > > > > > > > > could be historically managed using source
> > > control,
> > > > > > etc.
> > > > > > > > As
> > > > > > > > > we
> > > > > > > > > > > got
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > and more components managing the configs, we
> > > > haven't
> > > > > > > taken
> > > > > > > > > care
> > > > > > > > > > > > that
> > > > > > > > > > > > > > they
> > > > > > > > > > > > > > > > they all work with each other in an expected
> > way
> > > (I
> > > > > > > believe
> > > > > > > > > > these
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > > true..correct me if I'm wrong):
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - If configs are modified in the
> management
> > UI
> > > > or
> > > > > > the
> > > > > > > > > > Stellar
> > > > > > > > > > > > REPL
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > someone forgets to pull the configs from
> > > > zookeeper
> > > > > > to
> > > > > > > > > disk,
> > > > > > > > > > > > before
> > > > > > > > > > > > > > > they
> > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > a push via zk_load_utils, they will
> clobber
> > > the
> > > > > > > configs
> > > > > > > > in
> > > > > > > > > > > > > zookeeper
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > old configs.
> > > > > > > > > > > > > > > > - If the global config is changed on disk
> > and
> > > > the
> > > > > > > ambari
> > > > > > > > > > > service
> > > > > > > > > > > > > > > > restarts, it'll get reset with the
> original
> > > > global
> > > > > > > > config.
> > > > > > > > > > > > > > > > - *Ryan, in the management UI, if someone
> > > > changes
> > > > > > the
> > > > > > > > > > > zookeeper
> > > > > > > > > > > > > > > configs
> > > > > > > > > > > > > > > > from outside, are those configs reflected
> > > > > > immediately
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > UI?*
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It seems to me that we have a couple of
> options
> > > > here:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - A service to intermediate and handle
> > config
> > > > > > > > > > update/retrieval
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > tracking historical changes so these
> > different
> > > > > > > > mechanisms
> > > > > > > > > > can
> > > > > > > > > > > > use
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > common
> > > > > > > > > > > > > > > > component for config management/tracking
> and
> > > > > > refactor
> > > > > > > > the
> > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > mechanisms to use that service
> > > > > > > > > > > > > > > > - Standardize on exactly one component to
> > > manage
> > > > > the
> > > > > > > > > configs
> > > > > > > > > > > and
> > > > > > > > > > > > > > > regress
> > > > > > > > > > > > > > > > the others (that's a verb, right? nicer
> > than
> > > > > > > delete.)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I happen to like the service approach,
> myself,
> > > but
> > > > I
> > > > > > > wanted
> > > > > > > > > to
> > > > > > > > > > > put
> > > > > > > > > > > > it
> > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > for discussion and hopefully someone will
> > > volunteer
> > > > > to
> > > > > > > > design
> > > > > > > > > > > such
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > To frame the debate, I want us to keep in
> mind
> > a
> > > > > couple
> > > > > > > of
> > > > > > > > > > things
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > or may not be relevant to the discussion:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - We will eventually be moving to support
> > > > kerberos
> > > > > > so
> > > > > > > > > there
> > > > > > > > > > > > should
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > least be a path to use kerberos for any
> > > solution
> > > > > IMO
> > > > > > > > > > > > > > > > - There is value in each of the different
> > > > > mechanisms
> > > > > > > in
> > > > > > > > > > place
> > > > > > > > > > > > now.
> > > > > > > > > > > > > > If
> > > > > > > > > > > > > > > > there weren't, then they wouldn't have
> been
> > > > > created.
> > > > > > > > > Before
> > > > > > > > > > > we
> > > > > > > > > > > > > try
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > this a "there can be only one" argument,
> I'd
> > > > like
> > > > > to
> > > > > > > > hear
> > > > > > > > > > very
> > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > arguments.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Finally, I'd appreciate if some people might
> > > answer
> > > > > the
> > > > > > > > > > > questions I
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > bold there. Hopefully this discussion, if
> > > nothing
> > > > > else
> > > > > > > > > > happens,
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > > result in fodder for proper documentation of
> > the
> > > > ins
> > > > > > and
> > > > > > > > outs
> > > > > > > > > > of
> > > > > > > > > > > > each
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the components bulleted above.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Casey
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > Nick Allen <n...@nickallen.org>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Nick Allen <n...@nickallen.org>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Nick Allen <n...@nickallen.org>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Nick Allen <n...@nickallen.org>
> > > >
> > > --
> > >
> > > Jon
> > >
> > > Sent from my mobile device
> > >
> >
> >
> >
> > --
> > Nick Allen <n...@nickallen.org>
> >
>
