It is actually the other way round. The AM does not know about the components. The components lookup the registry to find the AM host:port and then heartbeat into it. In one of the heartbeat cycle it sends all its config info to AM. This design works well for AM fault tolerance as well.
-Gour On 1/13/16, 12:40 PM, "Josh Elser" <josh.el...@gmail.com> wrote: >Yeah, I'm with you Manoj. It seems wrong to not see the exports for >jmemcached. > >Gour -- am I missing something too? > >Manoj Samel wrote: >> Shouldn't it also show component host:port in ZK when they are being >> exported in registry ? I thought that was the idea of registry ... >> >> On Tue, Jan 12, 2016 at 10:25 PM, Gour Saha<gs...@hortonworks.com> >>wrote: >> >>> The only host port that is in zk is that of the AM. >>> >>> -Gour >>> >>>> On Jan 12, 2016, at 6:21 PM, "Manoj Samel"<manojsamelt...@gmail.com> >>> wrote: >>>> Agreed. But I am not seeing even the host/port that is supposed to be >>>>in >>> ZK >>>> Thanks, >>>> >>>> Manoj >>>> >>>>> On Tue, Jan 12, 2016 at 5:52 PM, Gour Saha<gs...@hortonworks.com> >>> wrote: >>>>> In case you are expecting the data to be in zk, then you should know >>> that >>>>> Slider stores a very small amount of metadata (primarily endpoints >>>>>and >>>>> resource paths to AM) using which you can retrieve the actual data. >>>>>Most >>>>> of the information is stored in AM runtime memory and not persisted >>>>> anywhere. In case of AM failure all the AM state is rebuilt. >>>>> >>>>> -Gour >>>>> >>>>>> On 1/12/16, 5:24 PM, "Manoj Samel"<manojsamelt...@gmail.com> wrote: >>>>>> >>>>>> Hi Gour, >>>>>> >>>>>> As you asked, the urls and their outputs >>>>>> >>>>>> http://host36:1025/ws/v1/slider/publisher/exports >>>>>> >>>>>> >>> >>>{"exports":{"container_log_dirs":{"description":"container_log_dirs","up >>>da >>>>>> ted":1452647463881,"updatedTime":"Wed >>>>>> Jan 13 01:11:03 UTC >>>>>> >>> >>>2016","entries":{},"empty":true},"container_work_dirs":{"description":"c >>>on >>>>>> tainer_work_dirs","updated":1452647463881,"updatedTime":"Wed >>>>>> Jan 13 01:11:03 UTC 2016","entries":{},"empty":true}}} >>>>>> >>>>>> http://host36:1025/ws/v1/slider/publisher >>>>>> >>>>>> {"uris":{"slider":"http://host36:1025/ws/v1/slider/publisher/slider >>>>> ","expo >>>>>> rts":"http://host36:1025/ws/v1/slider/publisher/exports"}} >>>>>> >>>>>> http://host36:1025/ws/v1/slider/publisher/slider >>>>>> >>>>>> {"configurations":{"hdfs-site":{"description":"HDFS site >>>>>> >>> >>>settings","updated":0,"entries":{},"empty":true},"yarn-site":{"descripti >>>on >>>>>> ":"YARN >>>>>> site >>>>>> >>> >>>settings","updated":0,"entries":{},"empty":true},"complete-config":{"des >>>cr >>>>>> iption":"Complete >>>>>> slider application >>>>>> >>> >>>settings","updated":0,"entries":{},"empty":true},"core-site":{"descripti >>>on >>>>>> ":"Core >>>>>> site >>>>>> >>> >>>settings","updated":0,"entries":{},"empty":true},"componentinstancedata" >>>:{ >>> >>>"description":"ComponentInstanceData","updated":0,"entries":{},"empty":t >>>ru >>>>>> e}}} >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Manoj >>>>>> >>>>>> >>>>>>> On Tue, Jan 12, 2016 at 3:03 PM, Gour Saha<gs...@hortonworks.com> >>> wrote: >>>>>>> When you do the below get, you will see an api entry for >>>>>>> "classpath:org.apache.slider.publisher.exports" >>>>>>> >>>>>>> get /registry/users/foo/services/org-apache-slider/<appname> >>>>>>> >>>>>>> Do a curl on the uri value. The uri will be in the format of >>>>>>> http://host:port/ws/v1/slider/publisher/exports. Do you find the >>>>>>>info >>>>>>> you >>>>>>> are looking for there? >>>>>>> >>>>>>> Also do a curl on the path ws/v1/slider/publisher and its >>>>>>>sub-paths. >>>>>>> What >>>>>>> do you see? >>>>>>> >>>>>>> -Gour >>>>>>> >>>>>>> From: Manoj Samel<manojsamelt...@gmail.com<mailto: >>>>>>> manojsamelt...@gmail.com>> >>>>>>> Reply-To: "dev@slider.incubator.apache.org<mailto: >>>>>>> dev@slider.incubator.apache.org>" >>>>>>> <dev@slider.incubator.apache.org<mailto: >>>>>>> dev@slider.incubator.apache.org>> >>>>>>> Date: Tuesday, January 12, 2016 at 11:24 AM >>>>>>> To: "dev@slider.incubator.apache.org<mailto: >>>>>>> dev@slider.incubator.apache.org>" >>>>>>> <dev@slider.incubator.apache.org<mailto: >>>>>>> dev@slider.incubator.apache.org>> >>>>>>> Subject: Re: Component level host:port export seen in AM but not >>>>>>>in ZK >>>>>>> >>>>>>> Hi Josh, >>>>>>> >>>>>>> I have created a minimal use case using the jmemcached to reproduce >>> the >>>>>>> problem. The only change from OOB jmemcached is I am passing >>> additional >>>>>>> ports (not relevant for jmem but relevant for my ultimate use >>>>>>>case), >>>>>>> one of >>>>>>> which I try to export. >>>>>>> >>>>>>> I have attached the 3 config files as well as a output file >>>>>>>containing >>>>>>> the >>>>>>> slider-am URL output (which contains the component host:port) and >>>>>>>ZK >>>>>>> output >>>>>>> (which does not contains component host:port). >>>>>>> >>>>>>> Thanks for looking into the issue& let me know if you need >>>>>>>additional >>>>>>> info >>>>>>> >>>>>>> Manoj >>>>>>> >>>>>>> On Sun, Jan 10, 2016 at 11:26 AM, Josh Elser >>>>>>> <josh.el...@gmail.com<mailto: >>>>>>> josh.el...@gmail.com>> wrote: >>>>>>> Hey Manoj, >>>>>>> >>>>>>> I'm not sure at a glance, but it might be a bug in Slider's use of >>>>>>>the >>>>>>> registry? I'm not quite sure, I'd have to dig into the source to be >>>>>>> sure. >>>>>>> >>>>>>> Do you have an example that I could run which exhibits this? >>>>>>> >>>>>>> >>>>>>> Manoj Samel wrote: >>>>>>> Any thoughts ? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Jan 7, 2016 at 10:25 AM, Manoj >>>>>>>Samel<manojsamelt...@gmail.com >>>>>>> <mailto:manojsamelt...@gmail.com>> >>>>>>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Slider 0.80 with secured cluster. Registry is enabled *** without >>>>>>>*** >>>>>>> security >>>>>>> >>>>>>> Export is configured as follows >>>>>>> >>>>>>> 1. No application level export >>>>>>> 2. Component level port allocated per container. In appConfig.json, >>> *** >>>>>>> "global" *** section >>>>>>> "site.global.user1.http_port": >>>>>>> "${user1.ALLOCATED_PORT}{PER_CONTAINER}", >>>>>>> 3. Each component exports its own host/port (slider allocated). >>>>>>> metainfo.xml >>>>>>> <components> >>>>>>> <component> >>>>>>> <name>user1</name> >>>>>>> <category>MASTER</category> >>>>>>> <componentExports> >>>>>>> <componentExport> >>>>>>> <name>my_info</name> >>>>>>> >>>>>>><value>${THIS_HOST}:${user1.http_port}</value> >>>>>>> </componentExport> >>>>>>> </componentExports> >>>>>>> <commandScript> >>>>>>> <script>scripts/memcached.py</script> >>>>>>> <scriptType>PYTHON</scriptType> >>>>>>> </commandScript> >>>>>>> </component> >>>>>>> >>>>>>> At runtime, in >>>>>>> >>> http://am-host:1025/ws/v1/slider/publisher/slider/componentinstancedata >>>>> , >>>>>>> the host:port info can be seen as expected >>>>>>> >>>>>>> >>>>>>> >>> >>>{"description":"ComponentInstanceData","updated":0,"entries":{"container >>>_ >>>>>>> 1452045276360_0005_01_000004.my_info":"host1:33786", >>>>>>> .... >>>>>>> >>>>>>> However, when I look at ZK data, >>>>>>> >>>>>>> get >>>>>>> >>>>>>> >>> >>>/registry/users/foo/services/org-apache-slider/<appname>/components/cont >>>a >>>>>>> iner-1452045276360-0005-01-000004, >>>>>>> I don't see the host:port info. It only shows >>>>>>> >>>>>>> { >>>>>>> "type" : "JSONServiceRecord", >>>>>>> "description" : "user1", >>>>>>> "external" : [ ], >>>>>>> "internal" : [ ], >>>>>>> "yarn:id" : "container-1452045276360-0005-01-000004", >>>>>>> "yarn:persistence" : "container" >>>>>>> } >>>>>>> >>>>>>> Any thought what I may be missing ? >>>>>>> >>>>>>> Related question - looking @ >>> >>>https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/regis >>>t >>>>>>> ry/yarn-registry.html, >>>>>>> the spec and all examples of export list "protocol" >>>>>>> >>>>>>> e.g. REST, rmi etc. Its not clear if and how is that protocol to be >>>>>>> specified in the component export section ? Can anyone give a >>>>>>>example >>> ? >>>>>>> Thanks, >>>>>>> >>>>>>> Manoj >>>>> >> >
