Great discussion you have here! I have a newbie question, what is the case for installing Metron manually (without Ambari)? The only one I can think of right now is to integrate Metron with other non-ambari toolset/framework. What are the others?
- Dima On 01/13/2017 10:03 PM, Matt Foley wrote: > Thanks Casey. And I’m happy to see so much agreement on directions in this > thread. > > I looked up HBase and its use of ZK. Basically, it uses ZK for coordination, > but not configuration. > https://blog.cloudera.com/blog/2013/10/what-are-hbase-znodes/ says “HBase has > a design policy of using ZooKeeper only for transient data (that is, for > coordination and state communication). Thus if the HBase’s ZooKeeper data is > removed, only the transient operations are affected — data can continue to be > written and read to/from HBase.” > > However, config management for distributed apps is specifically one of the > uses contemplated for Zookeeper, and its persistent store guarantees > recoverability. So the only constraint in using ZK is that ZK be up and > running, which is already required for other aspects of Metron. > > Casey brings up a good point below that Metron’s idea of “configuration” > needs to go beyond the simple text or list values of typical Hadoop > configuration parameters. Json provides rich data structures that are > nevertheless easily editable, and that’s good for Metron. However, this in > itself doesn’t require using ZK for parameter storage, since the Hadoop > ecosystem and Ambari do not impose constraints on the type of files used to > store configs. > > So the main reasons for using ZK in addition to Ambari are: > - Supports asynchronous change notification for subscribed watchers, which is > (fairly) easy to use in Metron via Curator framework, thus enabling config > update without restarting processes, for selected config parameters. > - Easy propagation of changes to distributed services, when Ambari is not in > use (manual install case) > > > On 1/13/17, 11:16 AM, "Casey Stella" <[email protected]> wrote: > > Ok, I'll try to give the historical context for the parser and enrichment > and indexing topologies since I was the one who made a lot of the > mistakes. > ;) > > In the beginning there was only configuration changes at topology start > and > this was done via those flux properties. As the configs got more > complicated and more multi-dimensional, it became evident that properties > files weren't sufficient to contain them, so we started contemplating > externalizing the configs in JSON files. During this investigation, we > contemplated using zookeeper because it was one of the usecases for that > tool, apache curator is awesome and it would enable us, later on, to > support runtime changes without topology restart. So, pieces of the > config > that were the most important to change ended up getting implemented into > zookeeper. We did not go back and consider deeply the remaining configs > in > the flux properties for potential retrofit into zookeeper. We should > probably do eventually for consistency sake. That being said, I don't > know > of any addition to the flux properties for enrichment for a looong time, > so > new configs end up in zookeeper. > > I can't speak about the rationale for some of the profiler's flux > properties; I'll defer to Nick to cover that if there are questions. > > On Fri, Jan 13, 2017 at 11:58 AM, Matt Foley <[email protected]> wrote: > > > Gosh I hate being two hours behind you guys! :-) > > > > I’ll go back through the thread and collect open questions, but wanted > to > > put a word in about Zookeeper integration. > > I had been about to ask what the benefits of using ZK are, and I’ve now > > heard two features: > > - A logical “single place” that is visible and efficiently accessible by > > all processes on all nodes. > > - Supports async notification, and therefore updates without restarts. > > > > If only the first point was there, it could be replaced by Ambari, > because > > Ambari manages propagating configs so they look local to all nodes. And > > btw, there’s no bottlenecking, because clients don’t call the Ambari > REST > > APIs to *read* configs, just to *change* them. For reading established > > configs, clients just go to those local files, which are managed by the > > ambari-agents. > > > > But the ability to use ZK and Curator to support async config updates, > as > > we do, is really important. In fact, I think if we make our use of it > > consistent we might offer an example to the Ambari team for a general > > feature they could adopt, precisely to support config changes without > > restart. > > > > Three additional considerations: > > > > 1. HBase has always used ZK for various things. I don’t know if that > > includes configuration. If so, that’s already integrated with Ambari. > We > > should look into the details of that. > > > > 2. Can folks who’ve been here a while clarify why use of ZK is so > > piecemeal, and scattered several places in the znode tree? There seems > to > > be a vague idea that “things that get changed” go in ZK, while other > > configs go in local files. But all configs, by definition, can be > > changed. Is there any real reason not to put the whole Metron > > configuration in ZK, with a clean and consistent directory structure? > It’s > > okay if Metron cannot actually consume all updates asynchronously (like > > some topology configs that require a topo restart if changed). We just > > document which configs do and don’t support async change. HDFS has > lots of > > those. > > > > 3. I’m pretty sure we can suppress the “need to restart” warning from > > Ambari. We’ll need to dig in to find out how flexible this is. > > > > I do think we should continue supporting non-Ambari use, and if we put > all > > configs in ZK, that gets way easier to do in a simple and consistent > way. > > (Propagation problem solved). More thoughts after I have ‘em :-) > > > > Thanks, > > --Matt > > > > > > On 1/13/17, 8:30 AM, "Casey Stella" <[email protected]> wrote: > > > > I think that looks good. One last question, do we support the > manual > > install use-case (one where ambari does not exist, I mean)? > > > > Casey > > > > On Fri, Jan 13, 2017 at 11:28 AM, David Lyle <[email protected]> > > wrote: > > > > > That's good feedback, Jon. I think that puts us at: > > > > > > - Expand ambari to manage the remaining sensor-specific configs > > > - Refactor the push calls to zookeeper (in ConfigurationUtils, I > > think) > > > to push to ambari and take an Ambari user/pw and (optionally) > > reason > > > - We shall retain current functionality wrt live configuration > > changes. > > > Suggestion- ConfigurationUtils will push to both zookeeper and > > Ambari in an > > > atomic operation. (I suspect we can make ambari do this as well) > > > - Refactor the middleware that Ryan submitted to have the API > calls > > take > > > an Ambari user/pw and (optionally) reason > > > - Refactor the management UI to pass in an Ambari user/pw and > > (optionally) > > > reason > > > - Refactor the Stellar Management functions CONFIG_PUT to accept > an > > Ambari > > > user/pw and (optionally) reason > > > > > > -D... > > > > > > > > > On Fri, Jan 13, 2017 at 11:17 AM, Nick Allen <[email protected]> > > wrote: > > > > > > > +1 I strongly agree with Jon's view. Requiring a restart > would > > be a > > > big > > > > step backwards. > > > > > > > > I think the power of the platform is that the user can act on > live > > > > streaming data in a quick, iterative fashion. Adding > enrichments, > > > creating > > > > triage rules, adjusting profiles are all operational activities > > that can > > > be > > > > performed at any time in response to active threats. > > > > > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 10:59 AM, [email protected] < > > [email protected]> > > > > wrote: > > > > > > > > > Right, good conversation to bring up for sure. > > > > > > > > > > Just to comment on production generally only being updated > during > > > > > maintenance windows - I can tell you that my plans are to make > > my dev, > > > > > test, and prod Metron a very dynamic and frequently changing > > > environment > > > > > which will have coordinated but frequent modifications and I > > strongly > > > > > prefer not having to restart anywhere that I can. Of course > it > > will > > > > > happen, but keeping it to a minimum is key. > > > > > > > > > > Jon > > > > > > > > > > On Fri, Jan 13, 2017 at 10:53 AM Nick Allen > <[email protected]> > > > wrote: > > > > > > > > > > > Makes sense, Dave. I am totally clear on the proposal. I > just > > > wanted > > > > to > > > > > > ask the stupid question to bring the conversation full > circle, > > leave > > > no > > > > > > stone unturned, insert favorite idiom here. > > > > > > > > > > > > On Fri, Jan 13, 2017 at 10:46 AM, David Lyle < > > [email protected]> > > > > > wrote: > > > > > > > > > > > > > To be clear- NOBODY is suggesting replacing Zookeeper with > > Ambari. > > > > > > > > > > > > > > So, as a bit of a reset- here's what's being proposed: > > > > > > > > > > > > > > - Expand ambari to manage the remaining sensor-specific > > configs > > > > > > > - Refactor the push calls to zookeeper (in > > ConfigurationUtils, I > > > > > think) > > > > > > > to push to ambari and take an Ambari user/pw and > > (optionally) > > > > reason > > > > > > > - (Ambari can push to zookeeper, but it requires a > service > > > restart, > > > > so > > > > > > for > > > > > > > "live changes" you may > > > > > > > want do both a rest call and zookeeper update from > > > > > > ConfigurationUtils) > > > > > > > WAS > > > > > > > Question remains about whether ambari can do the push > to > > > > zookeeper > > > > > > > or whetheror whether ConfigurationUtils has to push to > > > zookeeper > > > > as > > > > > > > well as update > > > > > > > ambari. > > > > > > > - Refactor the middleware that Ryan submitted to have > the > > API > > > calls > > > > > > take > > > > > > > an Ambari user/pw and (optionally) reason > > > > > > > - Refactor the management UI to pass in an Ambari > user/pw > > and > > > > > > > (optionally) reason > > > > > > > - Refactor the Stellar Management functions CONFIG_PUT > to > > accept > > > an > > > > > > > Ambari user/pw and (optionally) reason > > > > > > > > > > > > > > -D... > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 10:42 AM, Ryan Merriman < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > The driver for using Zookeeper is that it is > asynchronous > > and > > > > accepts > > > > > > > > callbacks. Ambari would need to have that capability, > > otherwise > > > we > > > > > > have > > > > > > > to > > > > > > > > poll which is a deal breaker in my opinion. > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:28 AM, Casey Stella < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > No, it was good to bring up, Nick. I might have it > > wrong re: > > > > > Ambari. > > > > > > > > > > > > > > > > > > Casey > > > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 10:27 AM, Nick Allen < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > That makes sense. I wasn't sure based on Matt's > > original > > > > > > > > > > suggestion/description of Ambari, whether that was > > something > > > > that > > > > > > > > Ambari > > > > > > > > > > had also designed for or not. > > > > > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 10:14 AM, Casey Stella < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Polling the Ambari server via REST (or their API > if > > they > > > have > > > > > > one), > > > > > > > > > would > > > > > > > > > > > entail all workers hitting one server and create a > > single > > > > point > > > > > > of > > > > > > > > > > failure > > > > > > > > > > > (the ambari server is what serves up REST). > > Zookeeper's > > > > intent > > > > > > is > > > > > > > to > > > > > > > > > not > > > > > > > > > > > have a single point of failure like this and (one > of > > its > > > > main) > > > > > > > > > use-cases > > > > > > > > > > is > > > > > > > > > > > to serve up configs in a distributed environment. > > > > > > > > > > > > > > > > > > > > > > Casey > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:55 AM, Nick Allen < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Let me ask a stupid question. What does > Zookeeper > > do for > > > > us > > > > > > that > > > > > > > > > > Ambari > > > > > > > > > > > > cannot? Why keep Zookeeper in the mix? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:28 AM, David Lyle < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > In the main yes- I've made some changes: > > > > > > > > > > > > > > > > > > > > > > > > > > - Expand ambari to manage the remaining > > > sensor-specific > > > > > > > configs > > > > > > > > > > > > > - Refactor the push calls to zookeeper (in > > > > > > > ConfigurationUtils, I > > > > > > > > > > > think) > > > > > > > > > > > > > to push to ambari and take an Ambari > user/pw > > and > > > > > > > (optionally) > > > > > > > > > > reason > > > > > > > > > > > > > - (Ambari can push to zookeeper, but it > > requires a > > > > service > > > > > > > > > restart, > > > > > > > > > > so > > > > > > > > > > > > for > > > > > > > > > > > > > "live changes" you may > > > > > > > > > > > > > want do both a rest call and zookeeper > > update from > > > > > > > > > > > > ConfigurationUtils) > > > > > > > > > > > > > WAS > > > > > > > > > > > > > Question remains about whether ambari can > do > > the > > > push > > > > > to > > > > > > > > > > zookeeper > > > > > > > > > > > > > or whetheror whether ConfigurationUtils > has > > to push > > > > to > > > > > > > > > zookeeper > > > > > > > > > > as > > > > > > > > > > > > > well as update > > > > > > > > > > > > > ambari. > > > > > > > > > > > > > - Refactor the middleware that Ryan > submitted > > to have > > > > the > > > > > > API > > > > > > > > > calls > > > > > > > > > > > > take > > > > > > > > > > > > > an Ambari user/pw and (optionally) reason > > > > > > > > > > > > > - Refactor the management UI to pass in an > > Ambari > > > > user/pw > > > > > > and > > > > > > > > > > > > > (optionally) reason > > > > > > > > > > > > > - Refactor the Stellar Management functions > > > CONFIG_PUT > > > > to > > > > > > > > accept > > > > > > > > > an > > > > > > > > > > > > > Ambari user/pw and (optionally) reason > > > > > > > > > > > > > > > > > > > > > > > > > > I think we'd need to do some detailed design > > around how > > > > to > > > > > > > handle > > > > > > > > > > what > > > > > > > > > > > we > > > > > > > > > > > > > expect to be dynamic configs, but the main > > principle > > > > should > > > > > > > (imo) > > > > > > > > > be > > > > > > > > > > to > > > > > > > > > > > > > always know who and why and make sure that > > Ambari is > > > > aware > > > > > > and > > > > > > > is > > > > > > > > > the > > > > > > > > > > > > > static backing store for Zookeeper. > > > > > > > > > > > > > > > > > > > > > > > > > > -D... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:19 AM, Casey Stella > < > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > So, basically, your proposed changes, broken > > into > > > > > tangible > > > > > > > > > gobbets > > > > > > > > > > of > > > > > > > > > > > > > work: > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Expand ambari to manage the remaining > > > > > sensor-specific > > > > > > > > > configs > > > > > > > > > > > > > > - Refactor the push calls to zookeeper > (in > > > > > > > > > ConfigurationUtils, I > > > > > > > > > > > > > think) > > > > > > > > > > > > > > to push to ambari and take a reason > > > > > > > > > > > > > > - Question remains about whether > ambari > > can do > > > > the > > > > > > push > > > > > > > > to > > > > > > > > > > > > > zookeeper > > > > > > > > > > > > > > or whether ConfigurationUtils has to > > push to > > > > > > zookeeper > > > > > > > as > > > > > > > > > > well > > > > > > > > > > > as > > > > > > > > > > > > > > update > > > > > > > > > > > > > > ambari. > > > > > > > > > > > > > > - Refactor the middleware that Ryan > > submitted to > > > > have > > > > > > the > > > > > > > > API > > > > > > > > > > > calls > > > > > > > > > > > > > take > > > > > > > > > > > > > > a reason > > > > > > > > > > > > > > - Refactor the management UI to pass in a > > reason > > > > > > > > > > > > > > - Refactor the Stellar Management > functions > > > > CONFIG_PUT > > > > > > to > > > > > > > > > > accept a > > > > > > > > > > > > > > reason > > > > > > > > > > > > > > > > > > > > > > > > > > > > Just so we can evaluate it and I can ensure > I > > haven't > > > > > > > > overlooked > > > > > > > > > > some > > > > > > > > > > > > > > important point. Please tell me if Ambari > > cannot do > > > > the > > > > > > > things > > > > > > > > > > we're > > > > > > > > > > > > > > suggesting it can do. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Casey > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:15 AM, David Lyle > < > > > > > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That's exactly correct, Casey. Basically, > an > > > > expansion > > > > > of > > > > > > > > what > > > > > > > > > > > we're > > > > > > > > > > > > > > > currently doing with global.json, > > > > enrichment.properties > > > > > > and > > > > > > > > > > > > > > > elasticsearch.properties. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -D... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:12 AM, Casey > > Stella < > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would suggest not having Ambari > replace > > > > > zookeeper. I > > > > > > > > think > > > > > > > > > > the > > > > > > > > > > > > > > > proposal > > > > > > > > > > > > > > > > is to have Ambari replace the editable > > store > > > (like > > > > > the > > > > > > > JSON > > > > > > > > > > files > > > > > > > > > > > > on > > > > > > > > > > > > > > > > disk). Zookeeper woudl be the source of > > truth > > > for > > > > > the > > > > > > > > > running > > > > > > > > > > > > > > topologies > > > > > > > > > > > > > > > > and ambari would be sync'd to it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Correct if I misspeak, dave or matt. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Casey > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 13, 2017 at 9:09 AM, Nick > > Allen < > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ambari seems like a logical choice. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *>> It doesn’t natively integrate > > Zookeeper > > > > storage > > > > > > of > > > > > > > > > > configs, > > > > > > > > > > > > but > > > > > > > > > > > > > > > there > > > > > > > > > > > > > > > > > is a natural place to specify copy > > to/from > > > > > Zookeeper > > > > > > > for > > > > > > > > > the > > > > > > > > > > > > files > > > > > > > > > > > > > > > > > desired.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > How would Ambari interact with > Zookeeper > > in > > > this > > > > > > > > scenario? > > > > > > > > > > > Would > > > > > > > > > > > > > > > Ambari > > > > > > > > > > > > > > > > > replace Zookeeper completely? Or would > > > Zookeeper > > > > > act > > > > > > as > > > > > > > > the > > > > > > > > > > > > > > persistence > > > > > > > > > > > > > > > > > tier under Ambari? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 12, 2017 at 9:24 PM, Matt > > Foley < > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Mike, could you try again on the > image, > > > please, > > > > > > > making > > > > > > > > > sure > > > > > > > > > > > it > > > > > > > > > > > > > is a > > > > > > > > > > > > > > > > > simple > > > > > > > > > > > > > > > > > > format (gif, png, or jpeg)? It got > > munched, > > > at > > > > > > least > > > > > > > > in > > > > > > > > > my > > > > > > > > > > > > > viewer. > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Casey, responding to some of the > > questions > > > you > > > > > > > raised: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’m going to make a rather strong > > statement: > > > > We > > > > > > > > already > > > > > > > > > > > have a > > > > > > > > > > > > > > > service > > > > > > > > > > > > > > > > > > “to intermediate and handle config > > > > > > update/retrieval”. > > > > > > > > > > > > > > > > > > Furthermore, it: > > > > > > > > > > > > > > > > > > - Correctly handles the problems of > > > distributed > > > > > > > > services > > > > > > > > > > > > running > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > multi-node clusters. (That’s a HARD > > problem, > > > > > > people, > > > > > > > > and > > > > > > > > > > we > > > > > > > > > > > > > > > shouldn’t > > > > > > > > > > > > > > > > > try > > > > > > > > > > > > > > > > > > to reinvent the wheel.) > > > > > > > > > > > > > > > > > > - Correctly handles Kerberos > security. > > > (That’s > > > > > > kinda > > > > > > > > hard > > > > > > > > > > > too, > > > > > > > > > > > > or > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > least > > > > > > > > > > > > > > > > > > a lot of work.) > > > > > > > > > > > > > > > > > > - It does automatic versioning of > > > > configurations, > > > > > > and > > > > > > > > > > allows > > > > > > > > > > > > > > viewing, > > > > > > > > > > > > > > > > > > comparing, and reverting historical > > configs > > > > > > > > > > > > > > > > > > - It has a capable REST API for all > > those > > > > things. > > > > > > > > > > > > > > > > > > It doesn’t natively integrate > Zookeeper > > > storage > > > > > of > > > > > > > > > configs, > > > > > > > > > > > but > > > > > > > > > > > > > > there > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > natural place to specify copy > to/from > > > Zookeeper > > > > > for > > > > > > > the > > > > > > > > > > files > > > > > > > > > > > > > > > desired. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It is Ambari. And we should commit > to > > it, > > > > rather > > > > > > > than > > > > > > > > > try > > > > > > > > > > to > > > > > > > > > > > > > > > re-create > > > > > > > > > > > > > > > > > > such features. > > > > > > > > > > > > > > > > > > Because it has a good REST API, it > is > > > perfectly > > > > > > > > feasible > > > > > > > > > to > > > > > > > > > > > > > > implement > > > > > > > > > > > > > > > > > > Stellar functions that call it. > > > > > > > > > > > > > > > > > > GUI configuration tools can also use > > the > > > Ambari > > > > > > APIs, > > > > > > > > or > > > > > > > > > > > better > > > > > > > > > > > > > yet > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > integrated in an “Ambari View”. (Eg, > > see the > > > > > “Yarn > > > > > > > > > Capacity > > > > > > > > > > > > > > Scheduler > > > > > > > > > > > > > > > > > > Configuration Tool” example in the > > Ambari > > > > > > > > documentation, > > > > > > > > > > > under > > > > > > > > > > > > > > “Using > > > > > > > > > > > > > > > > > > Ambari Views”.) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Arguments are: Parsimony, > Sufficiency, > > Not > > > > > > > reinventing > > > > > > > > > the > > > > > > > > > > > > wheel, > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > Not > > > > > > > > > > > > > > > > > > spending weeks and weeks of > developer > > time > > > over > > > > > the > > > > > > > > next > > > > > > > > > > year > > > > > > > > > > > > > > > > reinventing > > > > > > > > > > > > > > > > > > the wheel while getting details > wrong > > > multiple > > > > > > times… > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Okay, off soapbox. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Casey asked what the config update > > behavior > > > of > > > > > > Ambari > > > > > > > > is, > > > > > > > > > > and > > > > > > > > > > > > how > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > interact with changes made from > outside > > > Ambari. > > > > > > > > > > > > > > > > > > The following is from my experience > > working > > > > with > > > > > > the > > > > > > > > > Ambari > > > > > > > > > > > > Mpack > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > Metron. I am not otherwise an > Ambari > > expert, > > > > so > > > > > > > > tomorrow > > > > > > > > > > > I’ll > > > > > > > > > > > > > get > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > reviewed by an Ambari development > > engineer. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ambari-server runs on one node, and > > > > Ambari-agent > > > > > > runs > > > > > > > > on > > > > > > > > > > each > > > > > > > > > > > > of > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > nodes. > > > > > > > > > > > > > > > > > > Ambari-server has a private set of > py, > > xml, > > > and > > > > > > > > template > > > > > > > > > > > files, > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > together are used both to generate > the > > Ambari > > > > > > > > > configuration > > > > > > > > > > > > GUI, > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > defaults, and to generate > > configuration files > > > > (of > > > > > > any > > > > > > > > > > needed > > > > > > > > > > > > > > > filetype) > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > the various Stack components. > > > > > > > > > > > > > > > > > > Ambari-server also has a database > > where it > > > > stores > > > > > > the > > > > > > > > > > schema > > > > > > > > > > > > > > related > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > these files, so even if you reach in > > and edit > > > > > > > Ambari’s > > > > > > > > > > files, > > > > > > > > > > > > it > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > Error > > > > > > > > > > > > > > > > > > out if the set of parameters or > > parameter > > > names > > > > > > > > changes. > > > > > > > > > > The > > > > > > > > > > > > > > > > historical > > > > > > > > > > > > > > > > > > information about configuration > > changes is > > > also > > > > > > > stored > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > db. > > > > > > > > > > > > > > > > > > For each component (and in the case > of > > > Metron, > > > > > for > > > > > > > each > > > > > > > > > > > > > topology), > > > > > > > > > > > > > > > > there > > > > > > > > > > > > > > > > > > is a python file which controls the > > logic for > > > > > these > > > > > > > > > > actions, > > > > > > > > > > > > > among > > > > > > > > > > > > > > > > > others: > > > > > > > > > > > > > > > > > > - Install > > > > > > > > > > > > > > > > > > - Start / stop / restart / status > > > > > > > > > > > > > > > > > > - Configure > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It is actually up to this python > code > > (which > > > we > > > > > > wrote > > > > > > > > for > > > > > > > > > > the > > > > > > > > > > > > > > Metron > > > > > > > > > > > > > > > > > > Mpack) what happens in each of these > > API > > > calls. > > > > > > But > > > > > > > > the > > > > > > > > > > > > current > > > > > > > > > > > > > > > code, > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > I believe this is typical of > > Ambari-managed > > > > > > > components, > > > > > > > > > > > > performs > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > “Configure” action whenever you > press > > the > > > > “Save” > > > > > > > button > > > > > > > > > > after > > > > > > > > > > > > > > > changing > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > component config in Ambari, and also > > on each > > > > > > Install > > > > > > > > and > > > > > > > > > > > Start > > > > > > > > > > > > or > > > > > > > > > > > > > > > > > Restart. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The Configure action consists of > > > approximately > > > > > the > > > > > > > > > > following > > > > > > > > > > > > > > sequence > > > > > > > > > > > > > > > > > (see > > > > > > > > > > > > > > > > > > disclaimer above :-) > > > > > > > > > > > > > > > > > > - Recreate the generated config > files, > > using > > > > the > > > > > > > > template > > > > > > > > > > > files > > > > > > > > > > > > > and > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > actual configuration most recently > set > > in > > > > Ambari > > > > > > > > > > > > > > > > > > o Note this is also under the > control > > of > > > python > > > > > > code > > > > > > > > that > > > > > > > > > > we > > > > > > > > > > > > > wrote, > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > this is the appropriate place to > push > > to ZK > > > if > > > > > > > desired. > > > > > > > > > > > > > > > > > > - Propagate those config files to > each > > > > > > Ambari-agent, > > > > > > > > > with a > > > > > > > > > > > > > command > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > set > > > > > > > > > > > > > > > > > > them locally > > > > > > > > > > > > > > > > > > - The ambari-agents on each node > > receive the > > > > > files > > > > > > > and > > > > > > > > > > write > > > > > > > > > > > > them > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > specified locations on local storage > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ambari-server then whines that the > > updated > > > > > services > > > > > > > > > should > > > > > > > > > > be > > > > > > > > > > > > > > > > restarted, > > > > > > > > > > > > > > > > > > but does not initiate that action > > itself > > > > (unless > > > > > of > > > > > > > > > course > > > > > > > > > > > the > > > > > > > > > > > > > > > > initiating > > > > > > > > > > > > > > > > > > action was a Start command from the > > > > > administrator). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Make sense? It’s all quite > > straightforward > > > in > > > > > > > concept, > > > > > > > > > > > there’s > > > > > > > > > > > > > > just > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > awful lot of stuff wrapped around > that > > to > > > make > > > > it > > > > > > all > > > > > > > > go > > > > > > > > > > > > smoothly > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > handle the problems when it doesn’t. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There’s additional complexity in > that > > the > > > > > > > Ambari-agent > > > > > > > > > also > > > > > > > > > > > > > caches > > > > > > > > > > > > > > > (on > > > > > > > > > > > > > > > > > > each node) both the template files > and > > > COMPILED > > > > > > forms > > > > > > > > of > > > > > > > > > > the > > > > > > > > > > > > > python > > > > > > > > > > > > > > > > files > > > > > > > > > > > > > > > > > > (.pyc) involved in transforming > them. > > The > > > pyc > > > > > > files > > > > > > > > > > > > incorporate > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > > > amount of additional info regarding > > parameter > > > > > > values, > > > > > > > > but > > > > > > > > > > I’m > > > > > > > > > > > > not > > > > > > > > > > > > > > > sure > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > the form. I don’t think that > changes > > the > > > above > > > > > in > > > > > > > any > > > > > > > > > > > > practical > > > > > > > > > > > > > > way > > > > > > > > > > > > > > > > > unless > > > > > > > > > > > > > > > > > > you’re trying to cheat Ambari by > > reaching in > > > > and > > > > > > > > editing > > > > > > > > > > its > > > > > > > > > > > > > files > > > > > > > > > > > > > > > > > > directly. In that case, you also > need > > to > > > whack > > > > > the > > > > > > > pyc > > > > > > > > > > files > > > > > > > > > > > > (on > > > > > > > > > > > > > > > each > > > > > > > > > > > > > > > > > > node) to force the data to be > reloaded > > from > > > > > > > > > Ambari-server. > > > > > > > > > > > > Best > > > > > > > > > > > > > > > > solution > > > > > > > > > > > > > > > > > > is don’t cheat. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Also, there may be circumstances > under > > which > > > > the > > > > > > > > > > Ambari-agent > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > detect > > > > > > > > > > > > > > > > > > changes and re-write the latest > > version it > > > > knows > > > > > of > > > > > > > the > > > > > > > > > > > config > > > > > > > > > > > > > > files, > > > > > > > > > > > > > > > > > even > > > > > > > > > > > > > > > > > > without a Save or Start action at > the > > > > > > Ambari-server. > > > > > > > > I’m > > > > > > > > > > not > > > > > > > > > > > > > sure > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > > > and need to check with Ambari > > developers. It > > > > may > > > > > > no > > > > > > > > > longer > > > > > > > > > > > > > happen, > > > > > > > > > > > > > > > > altho > > > > > > > > > > > > > > > > > > I’m pretty sure change > > detection/reversion > > > was > > > > a > > > > > > > > feature > > > > > > > > > of > > > > > > > > > > > > early > > > > > > > > > > > > > > > > > versions > > > > > > > > > > > > > > > > > > of Ambari. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hope this helps, > > > > > > > > > > > > > > > > > > --Matt > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ============================== > > > > ================== > > > > > > > > > > > > > > > > > > From: Michael Miklavcic < > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > Reply-To: " > > [email protected]" > > > > > > > > > > > > > > > > > <[email protected]. > > > > > > > > > > > > > > > > > > org> > > > > > > > > > > > > > > > > > > Date: Thursday, January 12, 2017 at > > 3:59 PM > > > > > > > > > > > > > > > > > > To: > "[email protected]" > > > > > > > > > > > > > > <[email protected]. > > > > > > > > > > > > > > > org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] Ambari Metron > > > > > Configuration > > > > > > > > > > Management > > > > > > > > > > > > > > > > > consequences > > > > > > > > > > > > > > > > > > and call to action > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Casey, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for starting this thread. I > > believe > > > you > > > > > are > > > > > > > > > correct > > > > > > > > > > in > > > > > > > > > > > > > your > > > > > > > > > > > > > > > > > > assessment of the 4 options for > > updating > > > > configs > > > > > in > > > > > > > > > Metron. > > > > > > > > > > > > When > > > > > > > > > > > > > > > using > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > than one of these options we can get > > into a > > > > > > > split-brain > > > > > > > > > > > > > scenario. A > > > > > > > > > > > > > > > > basic > > > > > > > > > > > > > > > > > > example is updating the global > config > > on disk > > > > and > > > > > > > using > > > > > > > > > the > > > > > > > > > > > > > > > > > > zk_load_configs.sh. Later, if a user > > decides > > > to > > > > > > > restart > > > > > > > > > > > Ambari, > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > cached > > > > > > > > > > > > > > > > > > version stored by Ambari (it's in > the > > MySQL > > > or > > > > > > other > > > > > > > > > > database > > > > > > > > > > > > > > backing > > > > > > > > > > > > > > > > > > Ambari) will be written out to disk > in > > the > > > > > defined > > > > > > > > config > > > > > > > > > > > > > > directory, > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > subsequently loaded using the > > > > zk_load_configs.sh > > > > > > > under > > > > > > > > > the > > > > > > > > > > > > hood. > > > > > > > > > > > > > > Any > > > > > > > > > > > > > > > > > global > > > > > > > > > > > > > > > > > > configuration modified outside of > > Ambari will > > > > be > > > > > > lost > > > > > > > > at > > > > > > > > > > this > > > > > > > > > > > > > > point. > > > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > > > is obviously undesirable, but I also > > like the > > > > > > purpose > > > > > > > > and > > > > > > > > > > > > utility > > > > > > > > > > > > > > > > exposed > > > > > > > > > > > > > > > > > > by the multiple config management > > interfaces > > > we > > > > > > > > currently > > > > > > > > > > > have > > > > > > > > > > > > > > > > > available. I > > > > > > > > > > > > > > > > > > also agree that a service would be > > best. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For reference, here's my > understanding > > of the > > > > > > current > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > > > > > loading mechanisms and their deps. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <image> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 12, 2017 at 3:08 PM, > Casey > > > Stella < > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the course of discussion on the > PR > > for > > > > > > METRON-652 > > > > > > > > > > > > > > > > > > < > > > > > > https://github.com/apache/incubator-metron/pull/415 > > > > > > > > > > > > > > > > > > > > something > > > > > > > > > > > > > > > that I > > > > > > > > > > > > > > > > > > should definitely have understood > > better came > > > > to > > > > > > > light > > > > > > > > > and > > > > > > > > > > I > > > > > > > > > > > > > > thought > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > it was worth bringing to the > attention > > of the > > > > > > > community > > > > > > > > > to > > > > > > > > > > > get > > > > > > > > > > > > > > > > > > clarification/discuss is just how we > > manage > > > > > > configs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Currently (assuming the management > UI > > that > > > Ryan > > > > > > > > Merriman > > > > > > > > > > > > > submitted) > > > > > > > > > > > > > > > > > configs > > > > > > > > > > > > > > > > > > are managed/adjusted via a couple of > > > different > > > > > > > > mechanism. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - zk_load_utils.sh: pushed and > > pulled from > > > > > disk > > > > > > to > > > > > > > > > > > zookeeper > > > > > > > > > > > > > > > > > > - Stellar REPL: pushed and pulled > > via the > > > > > > > > > > > > > CONFIG_GET/CONFIG_PUT > > > > > > > > > > > > > > > > > > functions > > > > > > > > > > > > > > > > > > - Ambari: initialized via the > > > zk_load_utils > > > > > > script > > > > > > > > and > > > > > > > > > > > then > > > > > > > > > > > > > some > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > them > > > > > > > > > > > > > > > > > > are managed directly (global > > config) and > > > > some > > > > > > > > > indirectly > > > > > > > > > > > > > > > > > > (sensor-specific > > > > > > > > > > > > > > > > > > configs). > > > > > > > > > > > > > > > > > > - NOTE: Upon service restart, > it > > may or > > > > may > > > > > > not > > > > > > > > > > > overwrite > > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > disk or on zookeeper. *Can > > someone > > > more > > > > > > > > > > knowledgeable > > > > > > > > > > > > than > > > > > > > > > > > > > > me > > > > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > this describe precisely the > > semantics > > > > that > > > > > we > > > > > > > can > > > > > > > > > > > expect > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > service restart > > > > > > > > > > > > > > > > > > for Ambari? What gets > > overwritten on > > > disk > > > > > and > > > > > > > > what > > > > > > > > > > gets > > > > > > > > > > > > > > updated > > > > > > > > > > > > > > > > > > in ambari?* > > > > > > > > > > > > > > > > > > - The Management UI: manages some > > of the > > > > > > configs. > > > > > > > > > *RYAN: > > > > > > > > > > > > Which > > > > > > > > > > > > > > > > configs > > > > > > > > > > > > > > > > > > do we support here and which > don't > > we > > > > support > > > > > > > here?* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As you can see, we have a mishmash > of > > > > mechanisms > > > > > to > > > > > > > > > update > > > > > > > > > > > and > > > > > > > > > > > > > > manage > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > configuration for Metron in > > zookeeper. In > > > the > > > > > > > > beginning > > > > > > > > > > the > > > > > > > > > > > > > > approach > > > > > > > > > > > > > > > > was > > > > > > > > > > > > > > > > > > just to edit configs on disk and > > push/pull > > > them > > > > > via > > > > > > > > > > > > > zk_load_utils. > > > > > > > > > > > > > > > > > Configs > > > > > > > > > > > > > > > > > > could be historically managed using > > source > > > > > control, > > > > > > > > etc. > > > > > > > > > > As > > > > > > > > > > > we > > > > > > > > > > > > > got > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > and more components managing the > > configs, we > > > > > > haven't > > > > > > > > > taken > > > > > > > > > > > care > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > they > > > > > > > > > > > > > > > > > > they all work with each other in an > > expected > > > > way > > > > > (I > > > > > > > > > believe > > > > > > > > > > > > these > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > true..correct me if I'm wrong): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - If configs are modified in the > > > management > > > > UI > > > > > > or > > > > > > > > the > > > > > > > > > > > > Stellar > > > > > > > > > > > > > > REPL > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > someone forgets to pull the > configs > > from > > > > > > zookeeper > > > > > > > > to > > > > > > > > > > > disk, > > > > > > > > > > > > > > before > > > > > > > > > > > > > > > > > they > > > > > > > > > > > > > > > > > > do > > > > > > > > > > > > > > > > > > a push via zk_load_utils, they > will > > > clobber > > > > > the > > > > > > > > > configs > > > > > > > > > > in > > > > > > > > > > > > > > > zookeeper > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > old configs. > > > > > > > > > > > > > > > > > > - If the global config is changed > > on disk > > > > and > > > > > > the > > > > > > > > > ambari > > > > > > > > > > > > > service > > > > > > > > > > > > > > > > > > restarts, it'll get reset with > the > > > original > > > > > > global > > > > > > > > > > config. > > > > > > > > > > > > > > > > > > - *Ryan, in the management UI, if > > someone > > > > > > changes > > > > > > > > the > > > > > > > > > > > > > zookeeper > > > > > > > > > > > > > > > > > configs > > > > > > > > > > > > > > > > > > from outside, are those configs > > reflected > > > > > > > > immediately > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > > UI?* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me that we have a > couple of > > > options > > > > > > here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - A service to intermediate and > > handle > > > > config > > > > > > > > > > > > update/retrieval > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > tracking historical changes so > these > > > > different > > > > > > > > > > mechanisms > > > > > > > > > > > > can > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > common > > > > > > > > > > > > > > > > > > component for config > > management/tracking > > > and > > > > > > > > refactor > > > > > > > > > > the > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > > > > mechanisms to use that service > > > > > > > > > > > > > > > > > > - Standardize on exactly one > > component to > > > > > manage > > > > > > > the > > > > > > > > > > > configs > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > regress > > > > > > > > > > > > > > > > > > the others (that's a verb, right? > > nicer > > > > than > > > > > > > > > delete.) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I happen to like the service > approach, > > > myself, > > > > > but > > > > > > I > > > > > > > > > wanted > > > > > > > > > > > to > > > > > > > > > > > > > put > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > up > > > > > > > > > > > > > > > > > > for discussion and hopefully someone > > will > > > > > volunteer > > > > > > > to > > > > > > > > > > design > > > > > > > > > > > > > such > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > thing. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To frame the debate, I want us to > keep > > in > > > mind > > > > a > > > > > > > couple > > > > > > > > > of > > > > > > > > > > > > things > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > may > > > > > > > > > > > > > > > > > > or may not be relevant to the > > discussion: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - We will eventually be moving to > > support > > > > > > kerberos > > > > > > > > so > > > > > > > > > > > there > > > > > > > > > > > > > > should > > > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > least be a path to use kerberos > for > > any > > > > > solution > > > > > > > IMO > > > > > > > > > > > > > > > > > > - There is value in each of the > > different > > > > > > > mechanisms > > > > > > > > > in > > > > > > > > > > > > place > > > > > > > > > > > > > > now. > > > > > > > > > > > > > > > > If > > > > > > > > > > > > > > > > > > there weren't, then they wouldn't > > have > > > been > > > > > > > created. > > > > > > > > > > > Before > > > > > > > > > > > > > we > > > > > > > > > > > > > > > try > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > make > > > > > > > > > > > > > > > > > > this a "there can be only one" > > argument, > > > I'd > > > > > > like > > > > > > > to > > > > > > > > > > hear > > > > > > > > > > > > very > > > > > > > > > > > > > > > good > > > > > > > > > > > > > > > > > > arguments. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Finally, I'd appreciate if some > people > > might > > > > > answer > > > > > > > the > > > > > > > > > > > > > questions I > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > bold there. Hopefully this > > discussion, if > > > > > nothing > > > > > > > else > > > > > > > > > > > > happens, > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > result in fodder for proper > > documentation of > > > > the > > > > > > ins > > > > > > > > and > > > > > > > > > > outs > > > > > > > > > > > > of > > > > > > > > > > > > > > each > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > the components bulleted above. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Casey > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > Nick Allen <[email protected]> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Nick Allen <[email protected]> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Nick Allen <[email protected]> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Nick Allen <[email protected]> > > > > > > > > > > > -- > > > > > > > > > > Jon > > > > > > > > > > Sent from my mobile device > > > > > > > > > > > > > > > > > > > > > -- > > > > Nick Allen <[email protected]> > > > > > > > > > > > > > > > > > > > > > > > >
