At some point, it is expected that the component host:port info ( i.e. component discovery) is available in registry - Right ?
On Wed, Jan 13, 2016 at 12:55 PM, Gour Saha <gs...@hortonworks.com> wrote: > It is actually the other way round. The AM does not know about the > components. The components lookup the registry to find the AM host:port > and then heartbeat into it. In one of the heartbeat cycle it sends all its > config info to AM. This design works well for AM fault tolerance as well. > > -Gour > > On 1/13/16, 12:40 PM, "Josh Elser" <josh.el...@gmail.com> wrote: > > >Yeah, I'm with you Manoj. It seems wrong to not see the exports for > >jmemcached. > > > >Gour -- am I missing something too? > > > >Manoj Samel wrote: > >> Shouldn't it also show component host:port in ZK when they are being > >> exported in registry ? I thought that was the idea of registry ... > >> > >> On Tue, Jan 12, 2016 at 10:25 PM, Gour Saha<gs...@hortonworks.com> > >>wrote: > >> > >>> The only host port that is in zk is that of the AM. > >>> > >>> -Gour > >>> > >>>> On Jan 12, 2016, at 6:21 PM, "Manoj Samel"<manojsamelt...@gmail.com> > >>> wrote: > >>>> Agreed. But I am not seeing even the host/port that is supposed to be > >>>>in > >>> ZK > >>>> Thanks, > >>>> > >>>> Manoj > >>>> > >>>>> On Tue, Jan 12, 2016 at 5:52 PM, Gour Saha<gs...@hortonworks.com> > >>> wrote: > >>>>> In case you are expecting the data to be in zk, then you should know > >>> that > >>>>> Slider stores a very small amount of metadata (primarily endpoints > >>>>>and > >>>>> resource paths to AM) using which you can retrieve the actual data. > >>>>>Most > >>>>> of the information is stored in AM runtime memory and not persisted > >>>>> anywhere. In case of AM failure all the AM state is rebuilt. > >>>>> > >>>>> -Gour > >>>>> > >>>>>> On 1/12/16, 5:24 PM, "Manoj Samel"<manojsamelt...@gmail.com> > wrote: > >>>>>> > >>>>>> Hi Gour, > >>>>>> > >>>>>> As you asked, the urls and their outputs > >>>>>> > >>>>>> http://host36:1025/ws/v1/slider/publisher/exports > >>>>>> > >>>>>> > >>> > >>>{"exports":{"container_log_dirs":{"description":"container_log_dirs","up > >>>da > >>>>>> ted":1452647463881,"updatedTime":"Wed > >>>>>> Jan 13 01:11:03 UTC > >>>>>> > >>> > >>>2016","entries":{},"empty":true},"container_work_dirs":{"description":"c > >>>on > >>>>>> tainer_work_dirs","updated":1452647463881,"updatedTime":"Wed > >>>>>> Jan 13 01:11:03 UTC 2016","entries":{},"empty":true}}} > >>>>>> > >>>>>> http://host36:1025/ws/v1/slider/publisher > >>>>>> > >>>>>> {"uris":{"slider":"http://host36:1025/ws/v1/slider/publisher/slider > >>>>> ","expo > >>>>>> rts":"http://host36:1025/ws/v1/slider/publisher/exports"}} > >>>>>> > >>>>>> http://host36:1025/ws/v1/slider/publisher/slider > >>>>>> > >>>>>> {"configurations":{"hdfs-site":{"description":"HDFS site > >>>>>> > >>> > >>>settings","updated":0,"entries":{},"empty":true},"yarn-site":{"descripti > >>>on > >>>>>> ":"YARN > >>>>>> site > >>>>>> > >>> > >>>settings","updated":0,"entries":{},"empty":true},"complete-config":{"des > >>>cr > >>>>>> iption":"Complete > >>>>>> slider application > >>>>>> > >>> > >>>settings","updated":0,"entries":{},"empty":true},"core-site":{"descripti > >>>on > >>>>>> ":"Core > >>>>>> site > >>>>>> > >>> > >>>settings","updated":0,"entries":{},"empty":true},"componentinstancedata" > >>>:{ > >>> > >>>"description":"ComponentInstanceData","updated":0,"entries":{},"empty":t > >>>ru > >>>>>> e}}} > >>>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> Manoj > >>>>>> > >>>>>> > >>>>>>> On Tue, Jan 12, 2016 at 3:03 PM, Gour Saha<gs...@hortonworks.com> > >>> wrote: > >>>>>>> When you do the below get, you will see an api entry for > >>>>>>> "classpath:org.apache.slider.publisher.exports" > >>>>>>> > >>>>>>> get /registry/users/foo/services/org-apache-slider/<appname> > >>>>>>> > >>>>>>> Do a curl on the uri value. The uri will be in the format of > >>>>>>> http://host:port/ws/v1/slider/publisher/exports. Do you find the > >>>>>>>info > >>>>>>> you > >>>>>>> are looking for there? > >>>>>>> > >>>>>>> Also do a curl on the path ws/v1/slider/publisher and its > >>>>>>>sub-paths. > >>>>>>> What > >>>>>>> do you see? > >>>>>>> > >>>>>>> -Gour > >>>>>>> > >>>>>>> From: Manoj Samel<manojsamelt...@gmail.com<mailto: > >>>>>>> manojsamelt...@gmail.com>> > >>>>>>> Reply-To: "dev@slider.incubator.apache.org<mailto: > >>>>>>> dev@slider.incubator.apache.org>" > >>>>>>> <dev@slider.incubator.apache.org<mailto: > >>>>>>> dev@slider.incubator.apache.org>> > >>>>>>> Date: Tuesday, January 12, 2016 at 11:24 AM > >>>>>>> To: "dev@slider.incubator.apache.org<mailto: > >>>>>>> dev@slider.incubator.apache.org>" > >>>>>>> <dev@slider.incubator.apache.org<mailto: > >>>>>>> dev@slider.incubator.apache.org>> > >>>>>>> Subject: Re: Component level host:port export seen in AM but not > >>>>>>>in ZK > >>>>>>> > >>>>>>> Hi Josh, > >>>>>>> > >>>>>>> I have created a minimal use case using the jmemcached to reproduce > >>> the > >>>>>>> problem. The only change from OOB jmemcached is I am passing > >>> additional > >>>>>>> ports (not relevant for jmem but relevant for my ultimate use > >>>>>>>case), > >>>>>>> one of > >>>>>>> which I try to export. > >>>>>>> > >>>>>>> I have attached the 3 config files as well as a output file > >>>>>>>containing > >>>>>>> the > >>>>>>> slider-am URL output (which contains the component host:port) and > >>>>>>>ZK > >>>>>>> output > >>>>>>> (which does not contains component host:port). > >>>>>>> > >>>>>>> Thanks for looking into the issue& let me know if you need > >>>>>>>additional > >>>>>>> info > >>>>>>> > >>>>>>> Manoj > >>>>>>> > >>>>>>> On Sun, Jan 10, 2016 at 11:26 AM, Josh Elser > >>>>>>> <josh.el...@gmail.com<mailto: > >>>>>>> josh.el...@gmail.com>> wrote: > >>>>>>> Hey Manoj, > >>>>>>> > >>>>>>> I'm not sure at a glance, but it might be a bug in Slider's use of > >>>>>>>the > >>>>>>> registry? I'm not quite sure, I'd have to dig into the source to be > >>>>>>> sure. > >>>>>>> > >>>>>>> Do you have an example that I could run which exhibits this? > >>>>>>> > >>>>>>> > >>>>>>> Manoj Samel wrote: > >>>>>>> Any thoughts ? > >>>>>>> > >>>>>>> Thanks, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Jan 7, 2016 at 10:25 AM, Manoj > >>>>>>>Samel<manojsamelt...@gmail.com > >>>>>>> <mailto:manojsamelt...@gmail.com>> > >>>>>>> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> Slider 0.80 with secured cluster. Registry is enabled *** without > >>>>>>>*** > >>>>>>> security > >>>>>>> > >>>>>>> Export is configured as follows > >>>>>>> > >>>>>>> 1. No application level export > >>>>>>> 2. Component level port allocated per container. In appConfig.json, > >>> *** > >>>>>>> "global" *** section > >>>>>>> "site.global.user1.http_port": > >>>>>>> "${user1.ALLOCATED_PORT}{PER_CONTAINER}", > >>>>>>> 3. Each component exports its own host/port (slider allocated). > >>>>>>> metainfo.xml > >>>>>>> <components> > >>>>>>> <component> > >>>>>>> <name>user1</name> > >>>>>>> <category>MASTER</category> > >>>>>>> <componentExports> > >>>>>>> <componentExport> > >>>>>>> <name>my_info</name> > >>>>>>> > >>>>>>><value>${THIS_HOST}:${user1.http_port}</value> > >>>>>>> </componentExport> > >>>>>>> </componentExports> > >>>>>>> <commandScript> > >>>>>>> <script>scripts/memcached.py</script> > >>>>>>> <scriptType>PYTHON</scriptType> > >>>>>>> </commandScript> > >>>>>>> </component> > >>>>>>> > >>>>>>> At runtime, in > >>>>>>> > >>> > http://am-host:1025/ws/v1/slider/publisher/slider/componentinstancedata > >>>>> , > >>>>>>> the host:port info can be seen as expected > >>>>>>> > >>>>>>> > >>>>>>> > >>> > >>>{"description":"ComponentInstanceData","updated":0,"entries":{"container > >>>_ > >>>>>>> 1452045276360_0005_01_000004.my_info":"host1:33786", > >>>>>>> .... > >>>>>>> > >>>>>>> However, when I look at ZK data, > >>>>>>> > >>>>>>> get > >>>>>>> > >>>>>>> > >>> > >>>/registry/users/foo/services/org-apache-slider/<appname>/components/cont > >>>a > >>>>>>> iner-1452045276360-0005-01-000004, > >>>>>>> I don't see the host:port info. It only shows > >>>>>>> > >>>>>>> { > >>>>>>> "type" : "JSONServiceRecord", > >>>>>>> "description" : "user1", > >>>>>>> "external" : [ ], > >>>>>>> "internal" : [ ], > >>>>>>> "yarn:id" : "container-1452045276360-0005-01-000004", > >>>>>>> "yarn:persistence" : "container" > >>>>>>> } > >>>>>>> > >>>>>>> Any thought what I may be missing ? > >>>>>>> > >>>>>>> Related question - looking @ > >>> > >>> > https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/regis > >>>t > >>>>>>> ry/yarn-registry.html, > >>>>>>> the spec and all examples of export list "protocol" > >>>>>>> > >>>>>>> e.g. REST, rmi etc. Its not clear if and how is that protocol to be > >>>>>>> specified in the component export section ? Can anyone give a > >>>>>>>example > >>> ? > >>>>>>> Thanks, > >>>>>>> > >>>>>>> Manoj > >>>>> > >> > > > >
