And it seems to be working in 1.7. Thanks everyone for your help - now into more detailed service definition.
Cos On Tue, Dec 09, 2014 at 08:45PM, Konstantin Boudnik wrote: > Ah! Very timely remark indeed! Will spin a new one in the morning and have it > a go. Thanks! > > On Tue, Dec 09, 2014 at 07:25PM, Yusaku Sako wrote: > > Hi Konstantin, > > > > Custom services added via the API won't automatically show up in the UI if > > you are using Ambari 1.6.1 or below. > > Please try with Ambari 1.7.0. > > > > Thanks, > > Yusaku > > > > > > On Tue, Dec 9, 2014 at 6:42 PM, Konstantin Boudnik <c...@apache.org> wrote: > > > > > Thanks Nate - that does the trick. With the command-line upgrade + a curl > > > call > > > to add my new service now I can see it among > > > > > > http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2/services > > > > > > The issues I still see are: > > > - the service doesn't appear in "Add Service" wizard > > > - even after getting added by POST'ing > > > {"ServiceInfo":{"service_name":"NSNN"}} info, I don't see the service > > > in > > > the dashboard. It might be caused by some inconsistencies in my service > > > definition as I can use the same procedure to add Tez service, and > > > being > > > able to see it in the dashboard. > > > > > > Will dig a bit more into this tomorrow and report back if I hit anything > > > new. > > > > > > Thank you all guys for your help - quite appreciated! > > > Cos > > > > > > On Mon, Dec 08, 2014 at 05:11PM, Nate Cole wrote: > > > > Cos, > > > > > > > > There is an ambari server command that should allow you to switch the > > > version. Here are the steps I took to change the cluster version: > > > > > > > > 1. Install a cluster as normal (I used cluster HDP-2.1 as the stack). > > > > 2. In /var/lib/ambari-server/resources/stacks/HDP, copy 2.1 to > > > 2.1.TestStack > > > > 3. Shutdown ambari. > > > > 4. ran: ambari-server upgradestack HDP-2.1.TestStack > > > > 5. Start ambari. > > > > 6. On the cluster admin page, verified at the top of the page that the > > > cluster says "HDP-2.1.TestStack" > > > > > > > > Hope that helps, > > > > Nate > > > > > > > > On Dec 8, 2014, at 4:24 PM, Konstantin Boudnik <c...@apache.org> wrote: > > > > > > > > > On Mon, Dec 08, 2014 at 04:14PM, John Speidel wrote: > > > > >> Cos, > > > > >> > > > > >> I don't know why this is occurring. I have not actually ever changed > > > the > > > > >> stack of a running cluster, my response was a result of my > > > understanding > > > > >> and talking to others that have done this. > > > > >> > > > > >> Hopefully somebody that has actually done this can comment. > > > > >> In the meantime, I will see if I can find somebody that may may know > > > why > > > > >> this isn't working for you. > > > > > > > > > > Thank you very much - really appreciate your help! In the interest of > > > full > > > > > disclosure: I am using 1.6.1 server, not sure if this is of any > > > significance. > > > > > > > > > > Cos > > > > > > > > > >> On Fri, Dec 5, 2014 at 8:38 PM, Konstantin Boudnik <c...@apache.org> > > > wrote: > > > > >> > > > > >>> release Thanks for the info John. I think I am getting there, but > > > > >>> not > > > > >>> completely.... > > > > >>> > > > > >>> So, here's what I've done: > > > > >>> - added service, bounced Ambari server > > > > >>> > > > > >>> - curl to > > > http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/stacks/HDP/ > > > > >>> says that I have new stack > > > > >>> { > > > > >>> "href" : " > > > > >>> > > > http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/stacks/HDP/versions/2.1.WANdisco > > > > >>> ", > > > > >>> "Versions" : { > > > > >>> "stack_name" : "HDP", > > > > >>> "stack_version" : "2.1.WANdisco" > > > > >>> } > > > > >>> } > > > > >>> > > > > >>> - ran update > > > > >>> curl -u admin:admin -i -X PUT -H 'X-Requested-By: ambari' -d > > > > >>> '{"Clusters":{"version":"HDP-2.1.WANdisco"}}' > > > > >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2 > > > > >>> > > > > >>> The result was: > > > > >>> HTTP/1.1 200 OK > > > > >>> Set-Cookie: AMBARISESSIONID=vueucwdh8p6mkmcygj803vi9;Path=/ > > > > >>> Expires: Thu, 01 Jan 1970 00:00:00 GMT > > > > >>> Content-Type: text/plain > > > > >>> Content-Length: 0 > > > > >>> Server: Jetty(7.6.7.v20120910) > > > > >>> > > > > >>> However, running > > > > >>> curl -u admin:admin > > > > >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/ > > > > >>> > > > > >>> Still gives me this.... > > > > >>> { > > > > >>> "href" : " > > > http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/ > > > > >>> ", > > > > >>> "items" : [ > > > > >>> { > > > > >>> "href" : " > > > > >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2";, > > > > >>> "Clusters" : { > > > > >>> "cluster_name" : "DC-2", > > > > >>> "version" : "HDP-2.1" > > > > >>> } > > > > >>> } > > > > >>> ] > > > > >>> > > > > >>> Any ideas/hints? Thank you very much for your help! > > > > >>> Cos > > > > >>> > > > > >>> On Thu, Dec 04, 2014 at 08:08PM, John Speidel wrote: > > > > >>>> Cos, > > > > >>>> > > > > >>>> Yes, if you are able to push a new stack to the ambari server hosts > > > > >>>> filesystem, you can use a REST api call to update the cluster to be > > > > >>>> associated with the new stack. > > > > >>>> > > > > >>>> So, the steps would would look like this: > > > > >>>> - install cluster with existing stack > > > > >>>> - create new stack with additional services/components > > > > >>>> - write the new stack to the filesystem of the ambari server host > > > > >>>> - bounce ambari server > > > > >>>> -- after the server restart, the hash of the stacks directory is > > > > >>>> recalculated which will result in the new updated stack to be > > > pushed to > > > > >>> the > > > > >>>> agents (not immediate, the check of the stack dir hash occurs every > > > n(not > > > > >>>> sure of exact value) heartbeats from the agents) > > > > >>>> - do an update (PUT) on the cluster resource changing the value of > > > the > > > > >>>> Clusters/version property to point to the new stack > > > > >>>> > > > > >>>> When these steps are completed the new service/components should be > > > > >>>> recognized by ambari. > > > > >>>> > > > > >>>> -John > > > > >>>> > > > > >>>> > > > > >>>> On Thu, Dec 4, 2014 at 6:07 PM, Konstantin Boudnik > > > > >>>> <c...@apache.org> > > > > >>> wrote: > > > > >>>> > > > > >>>>> Hi John. > > > > >>>>> > > > > >>>>> Thanks for more info. I'd like to pry more for a little bit of > > > > >>> information > > > > >>>>> (in-lined): > > > > >>>>> > > > > >>>>> On Thu, Dec 04, 2014 at 04:19PM, John Speidel wrote: > > > > >>>>>> Hi Cos, > > > > >>>>>> As mentioned above, it isn't currently possible to augment an > > > > >>> existing > > > > >>>>>> stack at runtime with new services/components. You could > > > > >>>>>> possibly > > > > >>> change > > > > >>>>>> the stack associated with a running cluster to a new stack which > > > > >>> contains > > > > >>>>>> the required services/components, but this new stack definition > > > would > > > > >>>>> need > > > > >>>>>> to be available to ambari so it really isn't very dynamic. We > > > > >>> understand > > > > >>>>> > > > > >>>>> Let me describe this scenario as I understood this. Let's say I > > > have > > > > >>> two > > > > >>>>> stacks: > > > > >>>>> - StackA (with usual set of components), currently installed on a > > > > >>> cluster > > > > >>>>> - StackB == StackA + ComponentX > > > > >>>>> So, if I add StackB definition to the Ambari server and restart > > > > >>>>> it, > > > > >>> will I > > > > >>>>> be > > > > >>>>> able to switch from StackA to StackB _without_ resintalling the > > > > >>> cluster, > > > > >>>>> and > > > > >>>>> then add/configure ComponentX's service? > > > > >>>>> > > > > >>>>> If my depiction is correct then it might be an acceptable > > > workaround > > > > >>> in the > > > > >>>>> absence of dynamic extensibility. > > > > >>>>> > > > > >>>>>> that going forward that the stack framework will need to be more > > > > >>>>> extensible > > > > >>>>>> and dynamic as additional stack definitions are created. Some of > > > the > > > > >>>>>> underlying work to make stack processing more flexible has > > > > >>>>>> already > > > > >>> been > > > > >>>>>> done in anticipation of new requirements in this area. > > > > >>>>> > > > > >>>>> Shivaji Dutta, in an offline email exchange, pointed me to > > > AMBARI-7175 > > > > >>> and > > > > >>>>> AMBARI-7201, which is seemingly going into the right direction, > > > > >>>>> but > > > > >>> still > > > > >>>>> would require a stack re-definition, if I read it right. > > > > >>>>> > > > > >>>>>> This is actually the second time that this scenario has came up > > > for > > > > >>> me > > > > >>>>>> today so it would be nice to get a Jira filed for an enhancement > > > > >>> where > > > > >>>>> the > > > > >>>>>> community can start to discuss the associated use cases. > > > > >>>>> > > > > >>>>> I will start a JIRA and put together the set of requirements. > > > > >>>>> > > > > >>>>> Regards, > > > > >>>>> Cos > > > > >>>>> > > > > >>>>>> -John > > > > >>>>>> > > > > >>>>>> On Tue, Dec 2, 2014 at 2:00 PM, Konstantin Boudnik < > > > c...@apache.org> > > > > >>>>> wrote: > > > > >>>>>> > > > > >>>>>>> Thanks for chiming in, Erin! Do you know if there's any plan to > > > > >>> have > > > > >>>>>>> dynamic > > > > >>>>>>> extensibility for services? For instance, Cloudera Manager has > > > this > > > > >>>>>>> concept of > > > > >>>>>>> Custom Services. And while it was done as an added late fix for > > > > >>>>> parcels, it > > > > >>>>>>> provides a way of adding new software components into an > > > > >>>>>>> existing > > > > >>>>> Hadoop > > > > >>>>>>> cluster. Which seems to be a pretty handy concept, considering > > > the > > > > >>>>> liquid > > > > >>>>>>> state of the whole Hadoop ecosystem. > > > > >>>>>>> > > > > >>>>>>> It seems that this functionality, if desired, is going to be > > > brand > > > > >>> new > > > > >>>>> and > > > > >>>>>>> as > > > > >>>>>>> such require not just development but also re-consideration at > > > the > > > > >>>>> design > > > > >>>>>>> level. Am I right? Appreciate the thoughts! > > > > >>>>>>> > > > > >>>>>>> The best part of active-active masters paradigm is that a client > > > > >>> can > > > > >>>>> work > > > > >>>>>>> with > > > > >>>>>>> either of them without a concern of which one is ahead or behind > > > > >>> of the > > > > >>>>>>> rest > > > > >>>>>>> as they are equal. Essentially this solves some major issues > > > > >>>>>>> that > > > > >>>>> pester > > > > >>>>>>> active-standby solutions. > > > > >>>>>>> > > > > >>>>>>> And yes, we do install the services to the cluster's nodes as > > > > >>>>>>> you > > > > >>>>>>> suggested in > > > > >>>>>>> the later part of your email - after all they are just Linux > > > > >>> daemons > > > > >>>>>>> (although > > > > >>>>>>> Ambari tries to hide the fact for whatever reason that is). > > > However > > > > >>>>> this > > > > >>>>>>> presents the core of the very issue: Ambari isn't aware about > > > > >>> services > > > > >>>>>>> installed outside of its realm, and it doesn't provide any > > > > >>> monitoring > > > > >>>>> or > > > > >>>>>>> life-cycle management for those. And because of that I am trying > > > to > > > > >>>>> figure > > > > >>>>>>> out > > > > >>>>>>> what can be done in this regards to have this dynamism. > > > > >>>>>>> > > > > >>>>>>> With regards, > > > > >>>>>>> Cos > > > > >>>>>>> > > > > >>>>>>> On Tue, Dec 02, 2014 at 08:41AM, Erin Boyd wrote: > > > > >>>>>>>> Hi Cos, > > > > >>>>>>>> Currently, adding services dynamically is not supported in > > > > >>> Ambari. > > > > >>>>> It is > > > > >>>>>>>> generally done, as you mentioned through the creation of a > > > > >>>>>>>> stack > > > > >>>>> with the > > > > >>>>>>>> services definition defined within in. I also don't believe we > > > > >>>>> currently > > > > >>>>>>>> support multiple masters on the same cluster. How would one > > > > >>>>>>> deferientiate > > > > >>>>>>>> which master it should be using in such an environment? > > > > >>>>>>>> > > > > >>>>>>>> Of course services can be installed independent of Ambari on > > > > >>>>>>>> the > > > > >>>>> hosts > > > > >>>>>>>> inside the cluster...but I don't think that is what you are > > > > >>> getting > > > > >>>>> at. > > > > >>>>>>> Are > > > > >>>>>>>> you expecting the services to have a presence on the UI though > > > > >>>>> installed > > > > >>>>>>>> outside of Ambari? > > > > >>>>>>>> > > > > >>>>>>>> Erin > > > > >>>>>>>> > > > > >>>>>>>> ----- Original Message ----- > > > > >>>>>>>> From: "Konstantin Boudnik" <c...@apache.org> > > > > >>>>>>>> To: dev@ambari.apache.org > > > > >>>>>>>> Sent: Monday, December 1, 2014 6:42:30 PM > > > > >>>>>>>> Subject: About extensibility of Ambari stacks (not inheritance) > > > > >>>>>>>> > > > > >>>>>>>> Guys, > > > > >>>>>>>> > > > > >>>>>>>> I am looking into possible stack extensibility properties of > > > > >>> Ambari > > > > >>>>> (not > > > > >>>>>>> to be > > > > >>>>>>>> confused with inheritance), but haven't been able to derive any > > > > >>> final > > > > >>>>>>>> conclusions just yet. Hence, I'd appreciate the input from the > > > > >>> people > > > > >>>>>>> behind > > > > >>>>>>>> the system. > > > > >>>>>>>> > > > > >>>>>>>> I have a few questions about current state of the Ambari > > > (version > > > > >>>>> 1.6.1 > > > > >>>>>>> and > > > > >>>>>>>> coming 1.7, and possibly later?) with regards to ability to > > > > >>> expand an > > > > >>>>>>> existing > > > > >>>>>>>> stack definitions with a 3rd party services and do it in the > > > > >>> runtime, > > > > >>>>>>> rather > > > > >>>>>>>> than only during the installation. We need to be able to run a > > > > >>>>> multiple > > > > >>>>>>>> instances of our master service in the cluster, which isn't > > > > >>> typical > > > > >>>>> for > > > > >>>>>>>> "normal" Hadoop concept where only one master can exist for any > > > > >>>>> giving > > > > >>>>>>>> service. > > > > >>>>>>>> > > > > >>>>>>>> Our use case is to be able to amend an exiting cluster setup > > > > >>> (HDP, > > > > >>>>>>> Bigtop, > > > > >>>>>>>> etc.) with a new service running on top of HDFS; but not to > > > > >>>>> reinstall the > > > > >>>>>>>> whole stack. The reason we have the use case is that oftentimes > > > > >>> our > > > > >>>>>>> software > > > > >>>>>>>> is being added to an existing 3rd party environment, as an > > > > >>>>>>>> added > > > > >>>>> bonus, > > > > >>>>>>> not > > > > >>>>>>>> available during the initial planning and setup of the cluster. > > > > >>>>>>>> > > > > >>>>>>>> The way I understand Ambari's stack inheritance is that a scion > > > > >>> stack > > > > >>>>>>> will > > > > >>>>>>>> be a brand new entity, e.g. I won't be able to cherry-pick > > > > >>> services > > > > >>>>> from > > > > >>>>>>> it > > > > >>>>>>>> and add them to the already installed parent stack's cluster, > > > > >>> right? > > > > >>>>>>>> > > > > >>>>>>>> So far from I see it doesn't seem possible without introducing > > > > >>>>>>>> a > > > > >>>>>>> brand-new > > > > >>>>>>>> version of a stack e.g. 'stack inheritance'. Which, > > > > >>> unfortunately, > > > > >>>>> won't > > > > >>>>>>> work > > > > >>>>>>>> as per my explanation of the use above case. I guess another > > > > >>>>>>>> way > > > > >>> to > > > > >>>>> look > > > > >>>>>>> at it > > > > >>>>>>>> is this: would it be possible to add a component (or a service) > > > > >>> that > > > > >>>>> will > > > > >>>>>>>> override an existing component or a service, but without a need > > > > >>> to > > > > >>>>>>> reinstall > > > > >>>>>>>> the rest of the stack? > > > > >>>>>>>> > > > > >>>>>>>> Thanks in advance for any info/ideas. > > > > >>>>>>>> Cos > > > > >>>>>>> > > > > >>>>>> > > > > >>>>>> -- > > > > >>>>>> CONFIDENTIALITY NOTICE > > > > >>>>>> NOTICE: This message is intended for the use of the individual or > > > > >>> entity > > > > >>>>> to > > > > >>>>>> which it is addressed and may contain information that is > > > > >>> confidential, > > > > >>>>>> privileged and exempt from disclosure under applicable law. If > > > > >>>>>> the > > > > >>> reader > > > > >>>>>> of this message is not the intended recipient, you are hereby > > > > >>> notified > > > > >>>>> that > > > > >>>>>> any printing, copying, dissemination, distribution, disclosure or > > > > >>>>>> forwarding of this communication is strictly prohibited. If you > > > have > > > > >>>>>> received this communication in error, please contact the sender > > > > >>>>> immediately > > > > >>>>>> and delete it from your system. Thank You. > > > > >>>>> > > > > >>>>> > > > > >>>> > > > > >>>> -- > > > > >>>> CONFIDENTIALITY NOTICE > > > > >>>> NOTICE: This message is intended for the use of the individual or > > > entity > > > > >>> to > > > > >>>> which it is addressed and may contain information that is > > > confidential, > > > > >>>> privileged and exempt from disclosure under applicable law. If the > > > reader > > > > >>>> of this message is not the intended recipient, you are hereby > > > notified > > > > >>> that > > > > >>>> any printing, copying, dissemination, distribution, disclosure or > > > > >>>> forwarding of this communication is strictly prohibited. If you > > > > >>>> have > > > > >>>> received this communication in error, please contact the sender > > > > >>> immediately > > > > >>>> and delete it from your system. Thank You. > > > > >>> > > > > >> > > > > >> -- > > > > >> CONFIDENTIALITY NOTICE > > > > >> NOTICE: This message is intended for the use of the individual or > > > entity to > > > > >> which it is addressed and may contain information that is > > > confidential, > > > > >> privileged and exempt from disclosure under applicable law. If the > > > reader > > > > >> of this message is not the intended recipient, you are hereby > > > notified that > > > > >> any printing, copying, dissemination, distribution, disclosure or > > > > >> forwarding of this communication is strictly prohibited. If you have > > > > >> received this communication in error, please contact the sender > > > immediately > > > > >> and delete it from your system. Thank You. > > > > > > > > > > > > -- > > > > CONFIDENTIALITY NOTICE > > > > NOTICE: This message is intended for the use of the individual or entity > > > to > > > > which it is addressed and may contain information that is confidential, > > > > privileged and exempt from disclosure under applicable law. If the > > > > reader > > > > of this message is not the intended recipient, you are hereby notified > > > that > > > > any printing, copying, dissemination, distribution, disclosure or > > > > forwarding of this communication is strictly prohibited. If you have > > > > received this communication in error, please contact the sender > > > immediately > > > > and delete it from your system. Thank You. > > > > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender immediately > > and delete it from your system. Thank You.
