Thanks for filing the jira. I think that¹s the best way to capture it. -Gour
On 1/18/16, 11:48 AM, "Manoj Samel" <manojsamelt...@gmail.com> wrote: >I have filed https://issues.apache.org/jira/browse/SLIDER-1064 > >Thanks, > >Manoj > >On Fri, Jan 15, 2016 at 9:48 AM, Manoj Samel <manojsamelt...@gmail.com> >wrote: > >> Any thoughts ? I feel this is a bug - should I file Jira to track ? >> >> Thanks, >> >> On Wed, Jan 13, 2016 at 1:35 PM, Manoj Samel <manojsamelt...@gmail.com> >> wrote: >> >>> At some point, it is expected that the component host:port info ( i.e. >>> component discovery) is available in registry - Right ? >>> >>> On Wed, Jan 13, 2016 at 12:55 PM, Gour Saha <gs...@hortonworks.com> >>> wrote: >>> >>>> It is actually the other way round. The AM does not know about the >>>> components. The components lookup the registry to find the AM >>>>host:port >>>> and then heartbeat into it. In one of the heartbeat cycle it sends all >>>> its >>>> config info to AM. This design works well for AM fault tolerance as >>>>well. >>>> >>>> -Gour >>>> >>>> On 1/13/16, 12:40 PM, "Josh Elser" <josh.el...@gmail.com> wrote: >>>> >>>> >Yeah, I'm with you Manoj. It seems wrong to not see the exports for >>>> >jmemcached. >>>> > >>>> >Gour -- am I missing something too? >>>> > >>>> >Manoj Samel wrote: >>>> >> Shouldn't it also show component host:port in ZK when they are >>>>being >>>> >> exported in registry ? I thought that was the idea of registry ... >>>> >> >>>> >> On Tue, Jan 12, 2016 at 10:25 PM, Gour Saha<gs...@hortonworks.com> >>>> >>wrote: >>>> >> >>>> >>> The only host port that is in zk is that of the AM. >>>> >>> >>>> >>> -Gour >>>> >>> >>>> >>>> On Jan 12, 2016, at 6:21 PM, "Manoj >>>>Samel"<manojsamelt...@gmail.com >>>> > >>>> >>> wrote: >>>> >>>> Agreed. But I am not seeing even the host/port that is supposed >>>>to >>>> be >>>> >>>>in >>>> >>> ZK >>>> >>>> Thanks, >>>> >>>> >>>> >>>> Manoj >>>> >>>> >>>> >>>>> On Tue, Jan 12, 2016 at 5:52 PM, Gour >>>>Saha<gs...@hortonworks.com> >>>> >>> wrote: >>>> >>>>> In case you are expecting the data to be in zk, then you should >>>> know >>>> >>> that >>>> >>>>> Slider stores a very small amount of metadata (primarily >>>>endpoints >>>> >>>>>and >>>> >>>>> resource paths to AM) using which you can retrieve the actual >>>>data. >>>> >>>>>Most >>>> >>>>> of the information is stored in AM runtime memory and not >>>>persisted >>>> >>>>> anywhere. In case of AM failure all the AM state is rebuilt. >>>> >>>>> >>>> >>>>> -Gour >>>> >>>>> >>>> >>>>>> On 1/12/16, 5:24 PM, "Manoj Samel"<manojsamelt...@gmail.com> >>>> wrote: >>>> >>>>>> >>>> >>>>>> Hi Gour, >>>> >>>>>> >>>> >>>>>> As you asked, the urls and their outputs >>>> >>>>>> >>>> >>>>>> http://host36:1025/ws/v1/slider/publisher/exports >>>> >>>>>> >>>> >>>>>> >>>> >>> >>>> >>>> >>>>>>>{"exports":{"container_log_dirs":{"description":"container_log_dirs" >>>>>>>,"up >>>> >>>da >>>> >>>>>> ted":1452647463881,"updatedTime":"Wed >>>> >>>>>> Jan 13 01:11:03 UTC >>>> >>>>>> >>>> >>> >>>> >>>> >>>>>>>2016","entries":{},"empty":true},"container_work_dirs":{"description >>>>>>>":"c >>>> >>>on >>>> >>>>>> tainer_work_dirs","updated":1452647463881,"updatedTime":"Wed >>>> >>>>>> Jan 13 01:11:03 UTC 2016","entries":{},"empty":true}}} >>>> >>>>>> >>>> >>>>>> http://host36:1025/ws/v1/slider/publisher >>>> >>>>>> >>>> >>>>>> {"uris":{"slider":" >>>> http://host36:1025/ws/v1/slider/publisher/slider >>>> >>>>> ","expo >>>> >>>>>> rts":"http://host36:1025/ws/v1/slider/publisher/exports"}} >>>> >>>>>> >>>> >>>>>> http://host36:1025/ws/v1/slider/publisher/slider >>>> >>>>>> >>>> >>>>>> {"configurations":{"hdfs-site":{"description":"HDFS site >>>> >>>>>> >>>> >>> >>>> >>>> >>>>>>>settings","updated":0,"entries":{},"empty":true},"yarn-site":{"descr >>>>>>>ipti >>>> >>>on >>>> >>>>>> ":"YARN >>>> >>>>>> site >>>> >>>>>> >>>> >>> >>>> >>>> >>>>>>>settings","updated":0,"entries":{},"empty":true},"complete-config":{ >>>>>>>"des >>>> >>>cr >>>> >>>>>> iption":"Complete >>>> >>>>>> slider application >>>> >>>>>> >>>> >>> >>>> >>>> >>>>>>>settings","updated":0,"entries":{},"empty":true},"core-site":{"descr >>>>>>>ipti >>>> >>>on >>>> >>>>>> ":"Core >>>> >>>>>> site >>>> >>>>>> >>>> >>> >>>> >>>> >>>>>>>settings","updated":0,"entries":{},"empty":true},"componentinstanced >>>>>>>ata" >>>> >>>:{ >>>> >>> >>>> >>>> >>>>>>>"description":"ComponentInstanceData","updated":0,"entries":{},"empt >>>>>>>y":t >>>> >>>ru >>>> >>>>>> e}}} >>>> >>>>>> >>>> >>>>>> Thanks, >>>> >>>>>> >>>> >>>>>> Manoj >>>> >>>>>> >>>> >>>>>> >>>> >>>>>>> On Tue, Jan 12, 2016 at 3:03 PM, Gour >>>>Saha<gs...@hortonworks.com >>>> > >>>> >>> wrote: >>>> >>>>>>> When you do the below get, you will see an api entry for >>>> >>>>>>> "classpath:org.apache.slider.publisher.exports" >>>> >>>>>>> >>>> >>>>>>> get /registry/users/foo/services/org-apache-slider/<appname> >>>> >>>>>>> >>>> >>>>>>> Do a curl on the uri value. The uri will be in the format of >>>> >>>>>>> http://host:port/ws/v1/slider/publisher/exports. Do you find >>>>the >>>> >>>>>>>info >>>> >>>>>>> you >>>> >>>>>>> are looking for there? >>>> >>>>>>> >>>> >>>>>>> Also do a curl on the path ws/v1/slider/publisher and its >>>> >>>>>>>sub-paths. >>>> >>>>>>> What >>>> >>>>>>> do you see? >>>> >>>>>>> >>>> >>>>>>> -Gour >>>> >>>>>>> >>>> >>>>>>> From: Manoj Samel<manojsamelt...@gmail.com<mailto: >>>> >>>>>>> manojsamelt...@gmail.com>> >>>> >>>>>>> Reply-To: "dev@slider.incubator.apache.org<mailto: >>>> >>>>>>> dev@slider.incubator.apache.org>" >>>> >>>>>>> <dev@slider.incubator.apache.org<mailto: >>>> >>>>>>> dev@slider.incubator.apache.org>> >>>> >>>>>>> Date: Tuesday, January 12, 2016 at 11:24 AM >>>> >>>>>>> To: "dev@slider.incubator.apache.org<mailto: >>>> >>>>>>> dev@slider.incubator.apache.org>" >>>> >>>>>>> <dev@slider.incubator.apache.org<mailto: >>>> >>>>>>> dev@slider.incubator.apache.org>> >>>> >>>>>>> Subject: Re: Component level host:port export seen in AM but >>>>not >>>> >>>>>>>in ZK >>>> >>>>>>> >>>> >>>>>>> Hi Josh, >>>> >>>>>>> >>>> >>>>>>> I have created a minimal use case using the jmemcached to >>>> reproduce >>>> >>> the >>>> >>>>>>> problem. The only change from OOB jmemcached is I am passing >>>> >>> additional >>>> >>>>>>> ports (not relevant for jmem but relevant for my ultimate use >>>> >>>>>>>case), >>>> >>>>>>> one of >>>> >>>>>>> which I try to export. >>>> >>>>>>> >>>> >>>>>>> I have attached the 3 config files as well as a output file >>>> >>>>>>>containing >>>> >>>>>>> the >>>> >>>>>>> slider-am URL output (which contains the component host:port) >>>>and >>>> >>>>>>>ZK >>>> >>>>>>> output >>>> >>>>>>> (which does not contains component host:port). >>>> >>>>>>> >>>> >>>>>>> Thanks for looking into the issue& let me know if you need >>>> >>>>>>>additional >>>> >>>>>>> info >>>> >>>>>>> >>>> >>>>>>> Manoj >>>> >>>>>>> >>>> >>>>>>> On Sun, Jan 10, 2016 at 11:26 AM, Josh Elser >>>> >>>>>>> <josh.el...@gmail.com<mailto: >>>> >>>>>>> josh.el...@gmail.com>> wrote: >>>> >>>>>>> Hey Manoj, >>>> >>>>>>> >>>> >>>>>>> I'm not sure at a glance, but it might be a bug in Slider's >>>>use >>>> of >>>> >>>>>>>the >>>> >>>>>>> registry? I'm not quite sure, I'd have to dig into the source >>>>to >>>> be >>>> >>>>>>> sure. >>>> >>>>>>> >>>> >>>>>>> Do you have an example that I could run which exhibits this? >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> Manoj Samel wrote: >>>> >>>>>>> Any thoughts ? >>>> >>>>>>> >>>> >>>>>>> Thanks, >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> On Thu, Jan 7, 2016 at 10:25 AM, Manoj >>>> >>>>>>>Samel<manojsamelt...@gmail.com >>>> >>>>>>> <mailto:manojsamelt...@gmail.com>> >>>> >>>>>>> wrote: >>>> >>>>>>> >>>> >>>>>>> Hi, >>>> >>>>>>> >>>> >>>>>>> Slider 0.80 with secured cluster. Registry is enabled *** >>>>without >>>> >>>>>>>*** >>>> >>>>>>> security >>>> >>>>>>> >>>> >>>>>>> Export is configured as follows >>>> >>>>>>> >>>> >>>>>>> 1. No application level export >>>> >>>>>>> 2. Component level port allocated per container. In >>>> appConfig.json, >>>> >>> *** >>>> >>>>>>> "global" *** section >>>> >>>>>>> "site.global.user1.http_port": >>>> >>>>>>> "${user1.ALLOCATED_PORT}{PER_CONTAINER}", >>>> >>>>>>> 3. Each component exports its own host/port (slider >>>>allocated). >>>> >>>>>>> metainfo.xml >>>> >>>>>>> <components> >>>> >>>>>>> <component> >>>> >>>>>>> <name>user1</name> >>>> >>>>>>> <category>MASTER</category> >>>> >>>>>>> <componentExports> >>>> >>>>>>> <componentExport> >>>> >>>>>>> <name>my_info</name> >>>> >>>>>>> >>>> >>>>>>><value>${THIS_HOST}:${user1.http_port}</value> >>>> >>>>>>> </componentExport> >>>> >>>>>>> </componentExports> >>>> >>>>>>> <commandScript> >>>> >>>>>>> <script>scripts/memcached.py</script> >>>> >>>>>>> <scriptType>PYTHON</scriptType> >>>> >>>>>>> </commandScript> >>>> >>>>>>> </component> >>>> >>>>>>> >>>> >>>>>>> At runtime, in >>>> >>>>>>> >>>> >>> >>>> >>>>http://am-host:1025/ws/v1/slider/publisher/slider/componentinstancedata >>>> >>>>> , >>>> >>>>>>> the host:port info can be seen as expected >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>> >>>> >>>> >>>>>>>{"description":"ComponentInstanceData","updated":0,"entries":{"conta >>>>>>>iner >>>> >>>_ >>>> >>>>>>> 1452045276360_0005_01_000004.my_info":"host1:33786", >>>> >>>>>>> .... >>>> >>>>>>> >>>> >>>>>>> However, when I look at ZK data, >>>> >>>>>>> >>>> >>>>>>> get >>>> >>>>>>> >>>> >>>>>>> >>>> >>> >>>> >>>> >>>>>>>/registry/users/foo/services/org-apache-slider/<appname>/components/ >>>>>>>cont >>>> >>>a >>>> >>>>>>> iner-1452045276360-0005-01-000004, >>>> >>>>>>> I don't see the host:port info. It only shows >>>> >>>>>>> >>>> >>>>>>> { >>>> >>>>>>> "type" : "JSONServiceRecord", >>>> >>>>>>> "description" : "user1", >>>> >>>>>>> "external" : [ ], >>>> >>>>>>> "internal" : [ ], >>>> >>>>>>> "yarn:id" : "container-1452045276360-0005-01-000004", >>>> >>>>>>> "yarn:persistence" : "container" >>>> >>>>>>> } >>>> >>>>>>> >>>> >>>>>>> Any thought what I may be missing ? >>>> >>>>>>> >>>> >>>>>>> Related question - looking @ >>>> >>> >>>> >>> >>>> >>>>https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/regi >>>>s >>>> >>>t >>>> >>>>>>> ry/yarn-registry.html, >>>> >>>>>>> the spec and all examples of export list "protocol" >>>> >>>>>>> >>>> >>>>>>> e.g. REST, rmi etc. Its not clear if and how is that protocol >>>>to >>>> be >>>> >>>>>>> specified in the component export section ? Can anyone give a >>>> >>>>>>>example >>>> >>> ? >>>> >>>>>>> Thanks, >>>> >>>>>>> >>>> >>>>>>> Manoj >>>> >>>>> >>>> >> >>>> > >>>> >>>> >>> >>
