Concerning the externalized provider configuration, the idea is to have sets of "named" (unique ID'd) config, which can be referenced by any Knox topology. So, you could have a single provider configuration that is applied (by reference) to all your cluster topologies, if that made sense for your deployment. This will allow a change to a single config (externalized provider config) to affect all the clusters topologies that reference it; rather than what is required today, which is to make the same change in every affected cluster topology config. Does that answer your question?
I'll defer the HA question because I'm not yet familiar enough with it. Thanks for the feedback, Phil On Thu, Aug 31, 2017 at 3:42 AM, Krishna Pandey <krish.pande...@gmail.com> wrote: > > > > As a result, I'm wondering if Knox should support provider configuration > > references in topology.xml, rather than having to duplicate it across > > descriptors. > > > +1 to Phil's remark above. Adding to that as Knox provides multi-cluster > support, can we have external provider config cluster-wise? Or we can > define multiple external provider config in single "global" provider config > file and refer them with some key? > > Also, when services are deployed in HA, we specify HaProvider and multiple > Service URLs in the topology. Will it make sense to use Load balancer IP > address / External URL for those services in topology instead of multiple > service URLs and using HaProvider? > > Thanks, > Krishna Pandey > > On Wed, Aug 30, 2017 at 10:05 PM, larry mccay <lmc...@apache.org> wrote: > > > Terrific, Phil! > > > > > > On Wed, Aug 30, 2017 at 11:03 AM, Philip Zampino <pzamp...@gmail.com> > > wrote: > > > > > After giving this KIP some thought, I would like to work on it. I've > > added > > > more detail to the wiki, and I'll create a JIRA for it. > > > > > > On Fri, Aug 25, 2017 at 10:03 AM, larry mccay <lmc...@apache.org> > wrote: > > > > > > > Wonderful details, Phil! > > > > > > > > > > > > On Fri, Aug 25, 2017 at 9:40 AM, Philip Zampino <pzamp...@gmail.com> > > > > wrote: > > > > > > > > > Thanks! I've added the Ambari API details to the wiki. > > > > > > > > > > On Fri, Aug 25, 2017 at 8:27 AM, larry mccay <lmc...@apache.org> > > > wrote: > > > > > > > > > > > HI Phil - > > > > > > > > > > > > Thank you for digging into this topic! > > > > > > > > > > > > I've added you as a contributor to the wiki and you should be > able > > to > > > > > edit > > > > > > the KIP now. > > > > > > > > > > > > thanks, > > > > > > > > > > > > --larry > > > > > > > > > > > > On Thu, Aug 24, 2017 at 3:45 PM, Philip Zampino < > > pzamp...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > I've put together a quick python POC targeting Ambari as the > > > > discovery > > > > > > > source, just to prove that we can indeed get the necessary > > details > > > > via > > > > > > > Ambari's REST API. > > > > > > > It generates a proper topology.xml descriptor based on a simple > > > > > > descriptor > > > > > > > (I chose YAML for this POC), which has a reference to what I've > > > > called > > > > > a > > > > > > > policy descriptor (the <gateway/> portion of the topology > > > > descriptor). > > > > > > > > > > > > > > I am currently unable to update the KIP, but I can share the > REST > > > > APIs > > > > > > I've > > > > > > > employed if there is interest. > > > > > > > > > > > > > > As a result, I'm wondering if Knox should support provider > > > > > configuration > > > > > > > references in topology.xml, rather than having to duplicate it > > > across > > > > > > > descriptors. > > > > > > > So, instead of <gateway><provider>...</gateway> in each > > > > topology.xml, > > > > > > have > > > > > > > a single element that points to an external provider config > > (e.g., > > > > > > > <policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider- > > > > > > > config.xml</policy-ref>). > > > > > > > I've already externalized it for input to the POC, but I'm > still > > > > > copying > > > > > > > the contents into the resulting topology descriptor; I think it > > > would > > > > > be > > > > > > > better to copy only the reference. > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > - Phil > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 1:55 PM, larry mccay < > lmc...@apache.org> > > > > > wrote: > > > > > > > > > > > > > > > Good to hear, Phil. > > > > > > > > > > > > > > > > Yes, I was looking to go back and add some of the API > specifics > > > and > > > > > > > > investigation details for at least Ambari and ZK. > > > > > > > > If others make sense to add such as Consul, etcd, etc that > > would > > > be > > > > > > good > > > > > > > as > > > > > > > > well and it would help to tease out some of what may be > needed > > > for > > > > > the > > > > > > > > abstraction API and config. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino < > > > > pzamp...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > This sounds like a good improvement from a usability > > > perspective. > > > > > > > > > > > > > > > > > > I agree that Knox must continue its support for direct > > topology > > > > > (XML) > > > > > > > > > definitions, but for those deployments where there is a > > service > > > > > > > registry > > > > > > > > > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of > > the > > > > > > > topology, > > > > > > > > > the simplified config coupled with the service discovery > > could > > > > > reduce > > > > > > > > > topology (especially ServiceConnectivity) errors. > > > > > > > > > > > > > > > > > > I also agree that consolidating provider configurations > into > > > > named > > > > > > > sets, > > > > > > > > > which can be referenced from topology configurations, will > > be a > > > > > good > > > > > > > > > change, especially if there are commonly grouped providers > > > > employed > > > > > > by > > > > > > > > > multiple topologies. > > > > > > > > > > > > > > > > > > Concerning the TODO sections, is your intention is that > they > > > > > contain > > > > > > > the > > > > > > > > > respective API facilities that will provide the information > > > > needed > > > > > to > > > > > > > > > populate a topology definition? > > > > > > > > > > > > > > > > > > - Phil > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay < > > > lmc...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > All - > > > > > > > > > > > > > > > > > > > > I have published an initial stab at a proposal for > > > simplifying > > > > > > > topology > > > > > > > > > > management [1] - as I briefly mentioned on the 0.14.0 > > > planning > > > > > > > thread. > > > > > > > > > > > > > > > > > > > > There are a couple TODO sections that need some > > investigation > > > > for > > > > > > > > Ambari > > > > > > > > > > API and Zookeeper Registry API as discovery plugins. > > > > > > > > > > > > > > > > > > > > Please give the KIP a read and let me know if it makes > > sense > > > or > > > > > if > > > > > > > you > > > > > > > > > > would like to take on specific pieces of this work. If > you > > > > would > > > > > > like > > > > > > > > to > > > > > > > > > > add other options or ideas, etc. > > > > > > > > > > > > > > > > > > > > I think that this will have a huge impact as usability > > > > > improvements > > > > > > > for > > > > > > > > > > management of the gateway! > > > > > > > > > > > > > > > > > > > > thanks, > > > > > > > > > > > > > > > > > > > > --larry > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+ > > > > > > > > > > Service+Discovery+and+Topology+Generation > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
