I have filed https://issues.apache.org/jira/browse/SLIDER-1064
Thanks, Manoj On Fri, Jan 15, 2016 at 9:48 AM, Manoj Samel <manojsamelt...@gmail.com> wrote: > Any thoughts ? I feel this is a bug - should I file Jira to track ? > > Thanks, > > On Wed, Jan 13, 2016 at 1:35 PM, Manoj Samel <manojsamelt...@gmail.com> > wrote: > >> At some point, it is expected that the component host:port info ( i.e. >> component discovery) is available in registry - Right ? >> >> On Wed, Jan 13, 2016 at 12:55 PM, Gour Saha <gs...@hortonworks.com> >> wrote: >> >>> It is actually the other way round. The AM does not know about the >>> components. The components lookup the registry to find the AM host:port >>> and then heartbeat into it. In one of the heartbeat cycle it sends all >>> its >>> config info to AM. This design works well for AM fault tolerance as well. >>> >>> -Gour >>> >>> On 1/13/16, 12:40 PM, "Josh Elser" <josh.el...@gmail.com> wrote: >>> >>> >Yeah, I'm with you Manoj. It seems wrong to not see the exports for >>> >jmemcached. >>> > >>> >Gour -- am I missing something too? >>> > >>> >Manoj Samel wrote: >>> >> Shouldn't it also show component host:port in ZK when they are being >>> >> exported in registry ? I thought that was the idea of registry ... >>> >> >>> >> On Tue, Jan 12, 2016 at 10:25 PM, Gour Saha<gs...@hortonworks.com> >>> >>wrote: >>> >> >>> >>> The only host port that is in zk is that of the AM. >>> >>> >>> >>> -Gour >>> >>> >>> >>>> On Jan 12, 2016, at 6:21 PM, "Manoj Samel"<manojsamelt...@gmail.com >>> > >>> >>> wrote: >>> >>>> Agreed. But I am not seeing even the host/port that is supposed to >>> be >>> >>>>in >>> >>> ZK >>> >>>> Thanks, >>> >>>> >>> >>>> Manoj >>> >>>> >>> >>>>> On Tue, Jan 12, 2016 at 5:52 PM, Gour Saha<gs...@hortonworks.com> >>> >>> wrote: >>> >>>>> In case you are expecting the data to be in zk, then you should >>> know >>> >>> that >>> >>>>> Slider stores a very small amount of metadata (primarily endpoints >>> >>>>>and >>> >>>>> resource paths to AM) using which you can retrieve the actual data. >>> >>>>>Most >>> >>>>> of the information is stored in AM runtime memory and not persisted >>> >>>>> anywhere. In case of AM failure all the AM state is rebuilt. >>> >>>>> >>> >>>>> -Gour >>> >>>>> >>> >>>>>> On 1/12/16, 5:24 PM, "Manoj Samel"<manojsamelt...@gmail.com> >>> wrote: >>> >>>>>> >>> >>>>>> Hi Gour, >>> >>>>>> >>> >>>>>> As you asked, the urls and their outputs >>> >>>>>> >>> >>>>>> http://host36:1025/ws/v1/slider/publisher/exports >>> >>>>>> >>> >>>>>> >>> >>> >>> >>> >>>{"exports":{"container_log_dirs":{"description":"container_log_dirs","up >>> >>>da >>> >>>>>> ted":1452647463881,"updatedTime":"Wed >>> >>>>>> Jan 13 01:11:03 UTC >>> >>>>>> >>> >>> >>> >>> >>>2016","entries":{},"empty":true},"container_work_dirs":{"description":"c >>> >>>on >>> >>>>>> tainer_work_dirs","updated":1452647463881,"updatedTime":"Wed >>> >>>>>> Jan 13 01:11:03 UTC 2016","entries":{},"empty":true}}} >>> >>>>>> >>> >>>>>> http://host36:1025/ws/v1/slider/publisher >>> >>>>>> >>> >>>>>> {"uris":{"slider":" >>> http://host36:1025/ws/v1/slider/publisher/slider >>> >>>>> ","expo >>> >>>>>> rts":"http://host36:1025/ws/v1/slider/publisher/exports"}} >>> >>>>>> >>> >>>>>> http://host36:1025/ws/v1/slider/publisher/slider >>> >>>>>> >>> >>>>>> {"configurations":{"hdfs-site":{"description":"HDFS site >>> >>>>>> >>> >>> >>> >>> >>>settings","updated":0,"entries":{},"empty":true},"yarn-site":{"descripti >>> >>>on >>> >>>>>> ":"YARN >>> >>>>>> site >>> >>>>>> >>> >>> >>> >>> >>>settings","updated":0,"entries":{},"empty":true},"complete-config":{"des >>> >>>cr >>> >>>>>> iption":"Complete >>> >>>>>> slider application >>> >>>>>> >>> >>> >>> >>> >>>settings","updated":0,"entries":{},"empty":true},"core-site":{"descripti >>> >>>on >>> >>>>>> ":"Core >>> >>>>>> site >>> >>>>>> >>> >>> >>> >>> >>>settings","updated":0,"entries":{},"empty":true},"componentinstancedata" >>> >>>:{ >>> >>> >>> >>> >>>"description":"ComponentInstanceData","updated":0,"entries":{},"empty":t >>> >>>ru >>> >>>>>> e}}} >>> >>>>>> >>> >>>>>> Thanks, >>> >>>>>> >>> >>>>>> Manoj >>> >>>>>> >>> >>>>>> >>> >>>>>>> On Tue, Jan 12, 2016 at 3:03 PM, Gour Saha<gs...@hortonworks.com >>> > >>> >>> wrote: >>> >>>>>>> When you do the below get, you will see an api entry for >>> >>>>>>> "classpath:org.apache.slider.publisher.exports" >>> >>>>>>> >>> >>>>>>> get /registry/users/foo/services/org-apache-slider/<appname> >>> >>>>>>> >>> >>>>>>> Do a curl on the uri value. The uri will be in the format of >>> >>>>>>> http://host:port/ws/v1/slider/publisher/exports. Do you find the >>> >>>>>>>info >>> >>>>>>> you >>> >>>>>>> are looking for there? >>> >>>>>>> >>> >>>>>>> Also do a curl on the path ws/v1/slider/publisher and its >>> >>>>>>>sub-paths. >>> >>>>>>> What >>> >>>>>>> do you see? >>> >>>>>>> >>> >>>>>>> -Gour >>> >>>>>>> >>> >>>>>>> From: Manoj Samel<manojsamelt...@gmail.com<mailto: >>> >>>>>>> manojsamelt...@gmail.com>> >>> >>>>>>> Reply-To: "dev@slider.incubator.apache.org<mailto: >>> >>>>>>> dev@slider.incubator.apache.org>" >>> >>>>>>> <dev@slider.incubator.apache.org<mailto: >>> >>>>>>> dev@slider.incubator.apache.org>> >>> >>>>>>> Date: Tuesday, January 12, 2016 at 11:24 AM >>> >>>>>>> To: "dev@slider.incubator.apache.org<mailto: >>> >>>>>>> dev@slider.incubator.apache.org>" >>> >>>>>>> <dev@slider.incubator.apache.org<mailto: >>> >>>>>>> dev@slider.incubator.apache.org>> >>> >>>>>>> Subject: Re: Component level host:port export seen in AM but not >>> >>>>>>>in ZK >>> >>>>>>> >>> >>>>>>> Hi Josh, >>> >>>>>>> >>> >>>>>>> I have created a minimal use case using the jmemcached to >>> reproduce >>> >>> the >>> >>>>>>> problem. The only change from OOB jmemcached is I am passing >>> >>> additional >>> >>>>>>> ports (not relevant for jmem but relevant for my ultimate use >>> >>>>>>>case), >>> >>>>>>> one of >>> >>>>>>> which I try to export. >>> >>>>>>> >>> >>>>>>> I have attached the 3 config files as well as a output file >>> >>>>>>>containing >>> >>>>>>> the >>> >>>>>>> slider-am URL output (which contains the component host:port) and >>> >>>>>>>ZK >>> >>>>>>> output >>> >>>>>>> (which does not contains component host:port). >>> >>>>>>> >>> >>>>>>> Thanks for looking into the issue& let me know if you need >>> >>>>>>>additional >>> >>>>>>> info >>> >>>>>>> >>> >>>>>>> Manoj >>> >>>>>>> >>> >>>>>>> On Sun, Jan 10, 2016 at 11:26 AM, Josh Elser >>> >>>>>>> <josh.el...@gmail.com<mailto: >>> >>>>>>> josh.el...@gmail.com>> wrote: >>> >>>>>>> Hey Manoj, >>> >>>>>>> >>> >>>>>>> I'm not sure at a glance, but it might be a bug in Slider's use >>> of >>> >>>>>>>the >>> >>>>>>> registry? I'm not quite sure, I'd have to dig into the source to >>> be >>> >>>>>>> sure. >>> >>>>>>> >>> >>>>>>> Do you have an example that I could run which exhibits this? >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> Manoj Samel wrote: >>> >>>>>>> Any thoughts ? >>> >>>>>>> >>> >>>>>>> Thanks, >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> On Thu, Jan 7, 2016 at 10:25 AM, Manoj >>> >>>>>>>Samel<manojsamelt...@gmail.com >>> >>>>>>> <mailto:manojsamelt...@gmail.com>> >>> >>>>>>> wrote: >>> >>>>>>> >>> >>>>>>> Hi, >>> >>>>>>> >>> >>>>>>> Slider 0.80 with secured cluster. Registry is enabled *** without >>> >>>>>>>*** >>> >>>>>>> security >>> >>>>>>> >>> >>>>>>> Export is configured as follows >>> >>>>>>> >>> >>>>>>> 1. No application level export >>> >>>>>>> 2. Component level port allocated per container. In >>> appConfig.json, >>> >>> *** >>> >>>>>>> "global" *** section >>> >>>>>>> "site.global.user1.http_port": >>> >>>>>>> "${user1.ALLOCATED_PORT}{PER_CONTAINER}", >>> >>>>>>> 3. Each component exports its own host/port (slider allocated). >>> >>>>>>> metainfo.xml >>> >>>>>>> <components> >>> >>>>>>> <component> >>> >>>>>>> <name>user1</name> >>> >>>>>>> <category>MASTER</category> >>> >>>>>>> <componentExports> >>> >>>>>>> <componentExport> >>> >>>>>>> <name>my_info</name> >>> >>>>>>> >>> >>>>>>><value>${THIS_HOST}:${user1.http_port}</value> >>> >>>>>>> </componentExport> >>> >>>>>>> </componentExports> >>> >>>>>>> <commandScript> >>> >>>>>>> <script>scripts/memcached.py</script> >>> >>>>>>> <scriptType>PYTHON</scriptType> >>> >>>>>>> </commandScript> >>> >>>>>>> </component> >>> >>>>>>> >>> >>>>>>> At runtime, in >>> >>>>>>> >>> >>> >>> http://am-host:1025/ws/v1/slider/publisher/slider/componentinstancedata >>> >>>>> , >>> >>>>>>> the host:port info can be seen as expected >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>> >>> >>> >>>{"description":"ComponentInstanceData","updated":0,"entries":{"container >>> >>>_ >>> >>>>>>> 1452045276360_0005_01_000004.my_info":"host1:33786", >>> >>>>>>> .... >>> >>>>>>> >>> >>>>>>> However, when I look at ZK data, >>> >>>>>>> >>> >>>>>>> get >>> >>>>>>> >>> >>>>>>> >>> >>> >>> >>> >>>/registry/users/foo/services/org-apache-slider/<appname>/components/cont >>> >>>a >>> >>>>>>> iner-1452045276360-0005-01-000004, >>> >>>>>>> I don't see the host:port info. It only shows >>> >>>>>>> >>> >>>>>>> { >>> >>>>>>> "type" : "JSONServiceRecord", >>> >>>>>>> "description" : "user1", >>> >>>>>>> "external" : [ ], >>> >>>>>>> "internal" : [ ], >>> >>>>>>> "yarn:id" : "container-1452045276360-0005-01-000004", >>> >>>>>>> "yarn:persistence" : "container" >>> >>>>>>> } >>> >>>>>>> >>> >>>>>>> Any thought what I may be missing ? >>> >>>>>>> >>> >>>>>>> Related question - looking @ >>> >>> >>> >>> >>> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/regis >>> >>>t >>> >>>>>>> ry/yarn-registry.html, >>> >>>>>>> the spec and all examples of export list "protocol" >>> >>>>>>> >>> >>>>>>> e.g. REST, rmi etc. Its not clear if and how is that protocol to >>> be >>> >>>>>>> specified in the component export section ? Can anyone give a >>> >>>>>>>example >>> >>> ? >>> >>>>>>> Thanks, >>> >>>>>>> >>> >>>>>>> Manoj >>> >>>>> >>> >> >>> > >>> >>> >> >
