Any thoughts ? I feel this is a bug - should I file Jira to track ? Thanks,
On Wed, Jan 13, 2016 at 1:35 PM, Manoj Samel <manojsamelt...@gmail.com> wrote: > At some point, it is expected that the component host:port info ( i.e. > component discovery) is available in registry - Right ? > > On Wed, Jan 13, 2016 at 12:55 PM, Gour Saha <gs...@hortonworks.com> wrote: > >> It is actually the other way round. The AM does not know about the >> components. The components lookup the registry to find the AM host:port >> and then heartbeat into it. In one of the heartbeat cycle it sends all its >> config info to AM. This design works well for AM fault tolerance as well. >> >> -Gour >> >> On 1/13/16, 12:40 PM, "Josh Elser" <josh.el...@gmail.com> wrote: >> >> >Yeah, I'm with you Manoj. It seems wrong to not see the exports for >> >jmemcached. >> > >> >Gour -- am I missing something too? >> > >> >Manoj Samel wrote: >> >> Shouldn't it also show component host:port in ZK when they are being >> >> exported in registry ? I thought that was the idea of registry ... >> >> >> >> On Tue, Jan 12, 2016 at 10:25 PM, Gour Saha<gs...@hortonworks.com> >> >>wrote: >> >> >> >>> The only host port that is in zk is that of the AM. >> >>> >> >>> -Gour >> >>> >> >>>> On Jan 12, 2016, at 6:21 PM, "Manoj Samel"<manojsamelt...@gmail.com> >> >>> wrote: >> >>>> Agreed. But I am not seeing even the host/port that is supposed to be >> >>>>in >> >>> ZK >> >>>> Thanks, >> >>>> >> >>>> Manoj >> >>>> >> >>>>> On Tue, Jan 12, 2016 at 5:52 PM, Gour Saha<gs...@hortonworks.com> >> >>> wrote: >> >>>>> In case you are expecting the data to be in zk, then you should know >> >>> that >> >>>>> Slider stores a very small amount of metadata (primarily endpoints >> >>>>>and >> >>>>> resource paths to AM) using which you can retrieve the actual data. >> >>>>>Most >> >>>>> of the information is stored in AM runtime memory and not persisted >> >>>>> anywhere. In case of AM failure all the AM state is rebuilt. >> >>>>> >> >>>>> -Gour >> >>>>> >> >>>>>> On 1/12/16, 5:24 PM, "Manoj Samel"<manojsamelt...@gmail.com> >> wrote: >> >>>>>> >> >>>>>> Hi Gour, >> >>>>>> >> >>>>>> As you asked, the urls and their outputs >> >>>>>> >> >>>>>> http://host36:1025/ws/v1/slider/publisher/exports >> >>>>>> >> >>>>>> >> >>> >> >> >>>{"exports":{"container_log_dirs":{"description":"container_log_dirs","up >> >>>da >> >>>>>> ted":1452647463881,"updatedTime":"Wed >> >>>>>> Jan 13 01:11:03 UTC >> >>>>>> >> >>> >> >> >>>2016","entries":{},"empty":true},"container_work_dirs":{"description":"c >> >>>on >> >>>>>> tainer_work_dirs","updated":1452647463881,"updatedTime":"Wed >> >>>>>> Jan 13 01:11:03 UTC 2016","entries":{},"empty":true}}} >> >>>>>> >> >>>>>> http://host36:1025/ws/v1/slider/publisher >> >>>>>> >> >>>>>> {"uris":{"slider":" >> http://host36:1025/ws/v1/slider/publisher/slider >> >>>>> ","expo >> >>>>>> rts":"http://host36:1025/ws/v1/slider/publisher/exports"}} >> >>>>>> >> >>>>>> http://host36:1025/ws/v1/slider/publisher/slider >> >>>>>> >> >>>>>> {"configurations":{"hdfs-site":{"description":"HDFS site >> >>>>>> >> >>> >> >> >>>settings","updated":0,"entries":{},"empty":true},"yarn-site":{"descripti >> >>>on >> >>>>>> ":"YARN >> >>>>>> site >> >>>>>> >> >>> >> >> >>>settings","updated":0,"entries":{},"empty":true},"complete-config":{"des >> >>>cr >> >>>>>> iption":"Complete >> >>>>>> slider application >> >>>>>> >> >>> >> >> >>>settings","updated":0,"entries":{},"empty":true},"core-site":{"descripti >> >>>on >> >>>>>> ":"Core >> >>>>>> site >> >>>>>> >> >>> >> >> >>>settings","updated":0,"entries":{},"empty":true},"componentinstancedata" >> >>>:{ >> >>> >> >> >>>"description":"ComponentInstanceData","updated":0,"entries":{},"empty":t >> >>>ru >> >>>>>> e}}} >> >>>>>> >> >>>>>> Thanks, >> >>>>>> >> >>>>>> Manoj >> >>>>>> >> >>>>>> >> >>>>>>> On Tue, Jan 12, 2016 at 3:03 PM, Gour Saha<gs...@hortonworks.com> >> >>> wrote: >> >>>>>>> When you do the below get, you will see an api entry for >> >>>>>>> "classpath:org.apache.slider.publisher.exports" >> >>>>>>> >> >>>>>>> get /registry/users/foo/services/org-apache-slider/<appname> >> >>>>>>> >> >>>>>>> Do a curl on the uri value. The uri will be in the format of >> >>>>>>> http://host:port/ws/v1/slider/publisher/exports. Do you find the >> >>>>>>>info >> >>>>>>> you >> >>>>>>> are looking for there? >> >>>>>>> >> >>>>>>> Also do a curl on the path ws/v1/slider/publisher and its >> >>>>>>>sub-paths. >> >>>>>>> What >> >>>>>>> do you see? >> >>>>>>> >> >>>>>>> -Gour >> >>>>>>> >> >>>>>>> From: Manoj Samel<manojsamelt...@gmail.com<mailto: >> >>>>>>> manojsamelt...@gmail.com>> >> >>>>>>> Reply-To: "dev@slider.incubator.apache.org<mailto: >> >>>>>>> dev@slider.incubator.apache.org>" >> >>>>>>> <dev@slider.incubator.apache.org<mailto: >> >>>>>>> dev@slider.incubator.apache.org>> >> >>>>>>> Date: Tuesday, January 12, 2016 at 11:24 AM >> >>>>>>> To: "dev@slider.incubator.apache.org<mailto: >> >>>>>>> dev@slider.incubator.apache.org>" >> >>>>>>> <dev@slider.incubator.apache.org<mailto: >> >>>>>>> dev@slider.incubator.apache.org>> >> >>>>>>> Subject: Re: Component level host:port export seen in AM but not >> >>>>>>>in ZK >> >>>>>>> >> >>>>>>> Hi Josh, >> >>>>>>> >> >>>>>>> I have created a minimal use case using the jmemcached to >> reproduce >> >>> the >> >>>>>>> problem. The only change from OOB jmemcached is I am passing >> >>> additional >> >>>>>>> ports (not relevant for jmem but relevant for my ultimate use >> >>>>>>>case), >> >>>>>>> one of >> >>>>>>> which I try to export. >> >>>>>>> >> >>>>>>> I have attached the 3 config files as well as a output file >> >>>>>>>containing >> >>>>>>> the >> >>>>>>> slider-am URL output (which contains the component host:port) and >> >>>>>>>ZK >> >>>>>>> output >> >>>>>>> (which does not contains component host:port). >> >>>>>>> >> >>>>>>> Thanks for looking into the issue& let me know if you need >> >>>>>>>additional >> >>>>>>> info >> >>>>>>> >> >>>>>>> Manoj >> >>>>>>> >> >>>>>>> On Sun, Jan 10, 2016 at 11:26 AM, Josh Elser >> >>>>>>> <josh.el...@gmail.com<mailto: >> >>>>>>> josh.el...@gmail.com>> wrote: >> >>>>>>> Hey Manoj, >> >>>>>>> >> >>>>>>> I'm not sure at a glance, but it might be a bug in Slider's use of >> >>>>>>>the >> >>>>>>> registry? I'm not quite sure, I'd have to dig into the source to >> be >> >>>>>>> sure. >> >>>>>>> >> >>>>>>> Do you have an example that I could run which exhibits this? >> >>>>>>> >> >>>>>>> >> >>>>>>> Manoj Samel wrote: >> >>>>>>> Any thoughts ? >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> On Thu, Jan 7, 2016 at 10:25 AM, Manoj >> >>>>>>>Samel<manojsamelt...@gmail.com >> >>>>>>> <mailto:manojsamelt...@gmail.com>> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>> Hi, >> >>>>>>> >> >>>>>>> Slider 0.80 with secured cluster. Registry is enabled *** without >> >>>>>>>*** >> >>>>>>> security >> >>>>>>> >> >>>>>>> Export is configured as follows >> >>>>>>> >> >>>>>>> 1. No application level export >> >>>>>>> 2. Component level port allocated per container. In >> appConfig.json, >> >>> *** >> >>>>>>> "global" *** section >> >>>>>>> "site.global.user1.http_port": >> >>>>>>> "${user1.ALLOCATED_PORT}{PER_CONTAINER}", >> >>>>>>> 3. Each component exports its own host/port (slider allocated). >> >>>>>>> metainfo.xml >> >>>>>>> <components> >> >>>>>>> <component> >> >>>>>>> <name>user1</name> >> >>>>>>> <category>MASTER</category> >> >>>>>>> <componentExports> >> >>>>>>> <componentExport> >> >>>>>>> <name>my_info</name> >> >>>>>>> >> >>>>>>><value>${THIS_HOST}:${user1.http_port}</value> >> >>>>>>> </componentExport> >> >>>>>>> </componentExports> >> >>>>>>> <commandScript> >> >>>>>>> <script>scripts/memcached.py</script> >> >>>>>>> <scriptType>PYTHON</scriptType> >> >>>>>>> </commandScript> >> >>>>>>> </component> >> >>>>>>> >> >>>>>>> At runtime, in >> >>>>>>> >> >>> >> http://am-host:1025/ws/v1/slider/publisher/slider/componentinstancedata >> >>>>> , >> >>>>>>> the host:port info can be seen as expected >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>> >> >> >>>{"description":"ComponentInstanceData","updated":0,"entries":{"container >> >>>_ >> >>>>>>> 1452045276360_0005_01_000004.my_info":"host1:33786", >> >>>>>>> .... >> >>>>>>> >> >>>>>>> However, when I look at ZK data, >> >>>>>>> >> >>>>>>> get >> >>>>>>> >> >>>>>>> >> >>> >> >> >>>/registry/users/foo/services/org-apache-slider/<appname>/components/cont >> >>>a >> >>>>>>> iner-1452045276360-0005-01-000004, >> >>>>>>> I don't see the host:port info. It only shows >> >>>>>>> >> >>>>>>> { >> >>>>>>> "type" : "JSONServiceRecord", >> >>>>>>> "description" : "user1", >> >>>>>>> "external" : [ ], >> >>>>>>> "internal" : [ ], >> >>>>>>> "yarn:id" : "container-1452045276360-0005-01-000004", >> >>>>>>> "yarn:persistence" : "container" >> >>>>>>> } >> >>>>>>> >> >>>>>>> Any thought what I may be missing ? >> >>>>>>> >> >>>>>>> Related question - looking @ >> >>> >> >>> >> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/regis >> >>>t >> >>>>>>> ry/yarn-registry.html, >> >>>>>>> the spec and all examples of export list "protocol" >> >>>>>>> >> >>>>>>> e.g. REST, rmi etc. Its not clear if and how is that protocol to >> be >> >>>>>>> specified in the component export section ? Can anyone give a >> >>>>>>>example >> >>> ? >> >>>>>>> Thanks, >> >>>>>>> >> >>>>>>> Manoj >> >>>>> >> >> >> > >> >> >
