The only host port that is in zk is that of the AM. -Gour
> On Jan 12, 2016, at 6:21 PM, "Manoj Samel" <[email protected]> wrote: > > Agreed. But I am not seeing even the host/port that is supposed to be in ZK > > Thanks, > > Manoj > >> On Tue, Jan 12, 2016 at 5:52 PM, Gour Saha <[email protected]> wrote: >> >> In case you are expecting the data to be in zk, then you should know that >> Slider stores a very small amount of metadata (primarily endpoints and >> resource paths to AM) using which you can retrieve the actual data. Most >> of the information is stored in AM runtime memory and not persisted >> anywhere. In case of AM failure all the AM state is rebuilt. >> >> -Gour >> >>> On 1/12/16, 5:24 PM, "Manoj Samel" <[email protected]> wrote: >>> >>> Hi Gour, >>> >>> As you asked, the urls and their outputs >>> >>> http://host36:1025/ws/v1/slider/publisher/exports >>> >>> {"exports":{"container_log_dirs":{"description":"container_log_dirs","upda >>> ted":1452647463881,"updatedTime":"Wed >>> Jan 13 01:11:03 UTC >>> 2016","entries":{},"empty":true},"container_work_dirs":{"description":"con >>> tainer_work_dirs","updated":1452647463881,"updatedTime":"Wed >>> Jan 13 01:11:03 UTC 2016","entries":{},"empty":true}}} >>> >>> http://host36:1025/ws/v1/slider/publisher >>> >>> {"uris":{"slider":"http://host36:1025/ws/v1/slider/publisher/slider >> ","expo >>> rts":"http://host36:1025/ws/v1/slider/publisher/exports"}} >>> >>> http://host36:1025/ws/v1/slider/publisher/slider >>> >>> {"configurations":{"hdfs-site":{"description":"HDFS site >>> settings","updated":0,"entries":{},"empty":true},"yarn-site":{"description >>> ":"YARN >>> site >>> settings","updated":0,"entries":{},"empty":true},"complete-config":{"descr >>> iption":"Complete >>> slider application >>> settings","updated":0,"entries":{},"empty":true},"core-site":{"description >>> ":"Core >>> site >>> settings","updated":0,"entries":{},"empty":true},"componentinstancedata":{ >>> "description":"ComponentInstanceData","updated":0,"entries":{},"empty":tru >>> e}}} >>> >>> Thanks, >>> >>> Manoj >>> >>> >>>> On Tue, Jan 12, 2016 at 3:03 PM, Gour Saha <[email protected]> wrote: >>>> >>>> When you do the below get, you will see an api entry for >>>> "classpath:org.apache.slider.publisher.exports" >>>> >>>> get /registry/users/foo/services/org-apache-slider/<appname> >>>> >>>> Do a curl on the uri value. The uri will be in the format of >>>> http://host:port/ws/v1/slider/publisher/exports. Do you find the info >>>> you >>>> are looking for there? >>>> >>>> Also do a curl on the path ws/v1/slider/publisher and its sub-paths. >>>> What >>>> do you see? >>>> >>>> -Gour >>>> >>>> From: Manoj Samel <[email protected]<mailto: >>>> [email protected]>> >>>> Reply-To: "[email protected]<mailto: >>>> [email protected]>" >>>> <[email protected]<mailto: >>>> [email protected]>> >>>> Date: Tuesday, January 12, 2016 at 11:24 AM >>>> To: "[email protected]<mailto: >>>> [email protected]>" >>>> <[email protected]<mailto: >>>> [email protected]>> >>>> Subject: Re: Component level host:port export seen in AM but not in ZK >>>> >>>> Hi Josh, >>>> >>>> I have created a minimal use case using the jmemcached to reproduce the >>>> problem. The only change from OOB jmemcached is I am passing additional >>>> ports (not relevant for jmem but relevant for my ultimate use case), >>>> one of >>>> which I try to export. >>>> >>>> I have attached the 3 config files as well as a output file containing >>>> the >>>> slider-am URL output (which contains the component host:port) and ZK >>>> output >>>> (which does not contains component host:port). >>>> >>>> Thanks for looking into the issue & let me know if you need additional >>>> info >>>> >>>> Manoj >>>> >>>> On Sun, Jan 10, 2016 at 11:26 AM, Josh Elser >>>> <[email protected]<mailto: >>>> [email protected]>> wrote: >>>> Hey Manoj, >>>> >>>> I'm not sure at a glance, but it might be a bug in Slider's use of the >>>> registry? I'm not quite sure, I'd have to dig into the source to be >>>> sure. >>>> >>>> Do you have an example that I could run which exhibits this? >>>> >>>> >>>> Manoj Samel wrote: >>>> Any thoughts ? >>>> >>>> Thanks, >>>> >>>> >>>> >>>> On Thu, Jan 7, 2016 at 10:25 AM, Manoj Samel<[email protected] >>>> <mailto:[email protected]>> >>>> wrote: >>>> >>>> Hi, >>>> >>>> Slider 0.80 with secured cluster. Registry is enabled *** without *** >>>> security >>>> >>>> Export is configured as follows >>>> >>>> 1. No application level export >>>> 2. Component level port allocated per container. In appConfig.json, *** >>>> "global" *** section >>>> "site.global.user1.http_port": >>>> "${user1.ALLOCATED_PORT}{PER_CONTAINER}", >>>> 3. Each component exports its own host/port (slider allocated). >>>> metainfo.xml >>>> <components> >>>> <component> >>>> <name>user1</name> >>>> <category>MASTER</category> >>>> <componentExports> >>>> <componentExport> >>>> <name>my_info</name> >>>> <value>${THIS_HOST}:${user1.http_port}</value> >>>> </componentExport> >>>> </componentExports> >>>> <commandScript> >>>> <script>scripts/memcached.py</script> >>>> <scriptType>PYTHON</scriptType> >>>> </commandScript> >>>> </component> >>>> >>>> At runtime, in >>>> http://am-host:1025/ws/v1/slider/publisher/slider/componentinstancedata >> , >>>> the host:port info can be seen as expected >>>> >>>> >>>> {"description":"ComponentInstanceData","updated":0,"entries":{"container_ >>>> 1452045276360_0005_01_000004.my_info":"host1:33786", >>>> .... >>>> >>>> However, when I look at ZK data, >>>> >>>> get >>>> >>>> /registry/users/foo/services/org-apache-slider/<appname>/components/conta >>>> iner-1452045276360-0005-01-000004, >>>> I don't see the host:port info. It only shows >>>> >>>> { >>>> "type" : "JSONServiceRecord", >>>> "description" : "user1", >>>> "external" : [ ], >>>> "internal" : [ ], >>>> "yarn:id" : "container-1452045276360-0005-01-000004", >>>> "yarn:persistence" : "container" >>>> } >>>> >>>> Any thought what I may be missing ? >>>> >>>> Related question - looking @ >> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/regist >>>> ry/yarn-registry.html, >>>> the spec and all examples of export list "protocol" >>>> >>>> e.g. REST, rmi etc. Its not clear if and how is that protocol to be >>>> specified in the component export section ? Can anyone give a example ? >>>> >>>> Thanks, >>>> >>>> Manoj >> >>
