Shouldn't it also show component host:port in ZK when they are being exported in registry ? I thought that was the idea of registry ...
On Tue, Jan 12, 2016 at 10:25 PM, Gour Saha <gs...@hortonworks.com> wrote: > The only host port that is in zk is that of the AM. > > -Gour > > > On Jan 12, 2016, at 6:21 PM, "Manoj Samel" <manojsamelt...@gmail.com> > wrote: > > > > Agreed. But I am not seeing even the host/port that is supposed to be in > ZK > > > > Thanks, > > > > Manoj > > > >> On Tue, Jan 12, 2016 at 5:52 PM, Gour Saha <gs...@hortonworks.com> > wrote: > >> > >> In case you are expecting the data to be in zk, then you should know > that > >> Slider stores a very small amount of metadata (primarily endpoints and > >> resource paths to AM) using which you can retrieve the actual data. Most > >> of the information is stored in AM runtime memory and not persisted > >> anywhere. In case of AM failure all the AM state is rebuilt. > >> > >> -Gour > >> > >>> On 1/12/16, 5:24 PM, "Manoj Samel" <manojsamelt...@gmail.com> wrote: > >>> > >>> Hi Gour, > >>> > >>> As you asked, the urls and their outputs > >>> > >>> http://host36:1025/ws/v1/slider/publisher/exports > >>> > >>> > {"exports":{"container_log_dirs":{"description":"container_log_dirs","upda > >>> ted":1452647463881,"updatedTime":"Wed > >>> Jan 13 01:11:03 UTC > >>> > 2016","entries":{},"empty":true},"container_work_dirs":{"description":"con > >>> tainer_work_dirs","updated":1452647463881,"updatedTime":"Wed > >>> Jan 13 01:11:03 UTC 2016","entries":{},"empty":true}}} > >>> > >>> http://host36:1025/ws/v1/slider/publisher > >>> > >>> {"uris":{"slider":"http://host36:1025/ws/v1/slider/publisher/slider > >> ","expo > >>> rts":"http://host36:1025/ws/v1/slider/publisher/exports"}} > >>> > >>> http://host36:1025/ws/v1/slider/publisher/slider > >>> > >>> {"configurations":{"hdfs-site":{"description":"HDFS site > >>> > settings","updated":0,"entries":{},"empty":true},"yarn-site":{"description > >>> ":"YARN > >>> site > >>> > settings","updated":0,"entries":{},"empty":true},"complete-config":{"descr > >>> iption":"Complete > >>> slider application > >>> > settings","updated":0,"entries":{},"empty":true},"core-site":{"description > >>> ":"Core > >>> site > >>> > settings","updated":0,"entries":{},"empty":true},"componentinstancedata":{ > >>> > "description":"ComponentInstanceData","updated":0,"entries":{},"empty":tru > >>> e}}} > >>> > >>> Thanks, > >>> > >>> Manoj > >>> > >>> > >>>> On Tue, Jan 12, 2016 at 3:03 PM, Gour Saha <gs...@hortonworks.com> > wrote: > >>>> > >>>> When you do the below get, you will see an api entry for > >>>> "classpath:org.apache.slider.publisher.exports" > >>>> > >>>> get /registry/users/foo/services/org-apache-slider/<appname> > >>>> > >>>> Do a curl on the uri value. The uri will be in the format of > >>>> http://host:port/ws/v1/slider/publisher/exports. Do you find the info > >>>> you > >>>> are looking for there? > >>>> > >>>> Also do a curl on the path ws/v1/slider/publisher and its sub-paths. > >>>> What > >>>> do you see? > >>>> > >>>> -Gour > >>>> > >>>> From: Manoj Samel <manojsamelt...@gmail.com<mailto: > >>>> manojsamelt...@gmail.com>> > >>>> Reply-To: "dev@slider.incubator.apache.org<mailto: > >>>> dev@slider.incubator.apache.org>" > >>>> <dev@slider.incubator.apache.org<mailto: > >>>> dev@slider.incubator.apache.org>> > >>>> Date: Tuesday, January 12, 2016 at 11:24 AM > >>>> To: "dev@slider.incubator.apache.org<mailto: > >>>> dev@slider.incubator.apache.org>" > >>>> <dev@slider.incubator.apache.org<mailto: > >>>> dev@slider.incubator.apache.org>> > >>>> Subject: Re: Component level host:port export seen in AM but not in ZK > >>>> > >>>> Hi Josh, > >>>> > >>>> I have created a minimal use case using the jmemcached to reproduce > the > >>>> problem. The only change from OOB jmemcached is I am passing > additional > >>>> ports (not relevant for jmem but relevant for my ultimate use case), > >>>> one of > >>>> which I try to export. > >>>> > >>>> I have attached the 3 config files as well as a output file containing > >>>> the > >>>> slider-am URL output (which contains the component host:port) and ZK > >>>> output > >>>> (which does not contains component host:port). > >>>> > >>>> Thanks for looking into the issue & let me know if you need additional > >>>> info > >>>> > >>>> Manoj > >>>> > >>>> On Sun, Jan 10, 2016 at 11:26 AM, Josh Elser > >>>> <josh.el...@gmail.com<mailto: > >>>> josh.el...@gmail.com>> wrote: > >>>> Hey Manoj, > >>>> > >>>> I'm not sure at a glance, but it might be a bug in Slider's use of the > >>>> registry? I'm not quite sure, I'd have to dig into the source to be > >>>> sure. > >>>> > >>>> Do you have an example that I could run which exhibits this? > >>>> > >>>> > >>>> Manoj Samel wrote: > >>>> Any thoughts ? > >>>> > >>>> Thanks, > >>>> > >>>> > >>>> > >>>> On Thu, Jan 7, 2016 at 10:25 AM, Manoj Samel<manojsamelt...@gmail.com > >>>> <mailto:manojsamelt...@gmail.com>> > >>>> wrote: > >>>> > >>>> Hi, > >>>> > >>>> Slider 0.80 with secured cluster. Registry is enabled *** without *** > >>>> security > >>>> > >>>> Export is configured as follows > >>>> > >>>> 1. No application level export > >>>> 2. Component level port allocated per container. In appConfig.json, > *** > >>>> "global" *** section > >>>> "site.global.user1.http_port": > >>>> "${user1.ALLOCATED_PORT}{PER_CONTAINER}", > >>>> 3. Each component exports its own host/port (slider allocated). > >>>> metainfo.xml > >>>> <components> > >>>> <component> > >>>> <name>user1</name> > >>>> <category>MASTER</category> > >>>> <componentExports> > >>>> <componentExport> > >>>> <name>my_info</name> > >>>> <value>${THIS_HOST}:${user1.http_port}</value> > >>>> </componentExport> > >>>> </componentExports> > >>>> <commandScript> > >>>> <script>scripts/memcached.py</script> > >>>> <scriptType>PYTHON</scriptType> > >>>> </commandScript> > >>>> </component> > >>>> > >>>> At runtime, in > >>>> > http://am-host:1025/ws/v1/slider/publisher/slider/componentinstancedata > >> , > >>>> the host:port info can be seen as expected > >>>> > >>>> > >>>> > {"description":"ComponentInstanceData","updated":0,"entries":{"container_ > >>>> 1452045276360_0005_01_000004.my_info":"host1:33786", > >>>> .... > >>>> > >>>> However, when I look at ZK data, > >>>> > >>>> get > >>>> > >>>> > /registry/users/foo/services/org-apache-slider/<appname>/components/conta > >>>> iner-1452045276360-0005-01-000004, > >>>> I don't see the host:port info. It only shows > >>>> > >>>> { > >>>> "type" : "JSONServiceRecord", > >>>> "description" : "user1", > >>>> "external" : [ ], > >>>> "internal" : [ ], > >>>> "yarn:id" : "container-1452045276360-0005-01-000004", > >>>> "yarn:persistence" : "container" > >>>> } > >>>> > >>>> Any thought what I may be missing ? > >>>> > >>>> Related question - looking @ > >> > https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/regist > >>>> ry/yarn-registry.html, > >>>> the spec and all examples of export list "protocol" > >>>> > >>>> e.g. REST, rmi etc. Its not clear if and how is that protocol to be > >>>> specified in the component export section ? Can anyone give a example > ? > >>>> > >>>> Thanks, > >>>> > >>>> Manoj > >> > >> >
