No problem. Please do take a look & comment when you get a chance though.
Thanks, Manoj On Mon, Jan 18, 2016 at 12:54 PM, Gour Saha <gs...@hortonworks.com> wrote: > Thanks for filing the jira. I think that¹s the best way to capture it. > > -Gour > > On 1/18/16, 11:48 AM, "Manoj Samel" <manojsamelt...@gmail.com> wrote: > > >I have filed https://issues.apache.org/jira/browse/SLIDER-1064 > > > >Thanks, > > > >Manoj > > > >On Fri, Jan 15, 2016 at 9:48 AM, Manoj Samel <manojsamelt...@gmail.com> > >wrote: > > > >> Any thoughts ? I feel this is a bug - should I file Jira to track ? > >> > >> Thanks, > >> > >> On Wed, Jan 13, 2016 at 1:35 PM, Manoj Samel <manojsamelt...@gmail.com> > >> wrote: > >> > >>> At some point, it is expected that the component host:port info ( i.e. > >>> component discovery) is available in registry - Right ? > >>> > >>> On Wed, Jan 13, 2016 at 12:55 PM, Gour Saha <gs...@hortonworks.com> > >>> wrote: > >>> > >>>> It is actually the other way round. The AM does not know about the > >>>> components. The components lookup the registry to find the AM > >>>>host:port > >>>> and then heartbeat into it. In one of the heartbeat cycle it sends all > >>>> its > >>>> config info to AM. This design works well for AM fault tolerance as > >>>>well. > >>>> > >>>> -Gour > >>>> > >>>> On 1/13/16, 12:40 PM, "Josh Elser" <josh.el...@gmail.com> wrote: > >>>> > >>>> >Yeah, I'm with you Manoj. It seems wrong to not see the exports for > >>>> >jmemcached. > >>>> > > >>>> >Gour -- am I missing something too? > >>>> > > >>>> >Manoj Samel wrote: > >>>> >> Shouldn't it also show component host:port in ZK when they are > >>>>being > >>>> >> exported in registry ? I thought that was the idea of registry ... > >>>> >> > >>>> >> On Tue, Jan 12, 2016 at 10:25 PM, Gour Saha<gs...@hortonworks.com> > >>>> >>wrote: > >>>> >> > >>>> >>> The only host port that is in zk is that of the AM. > >>>> >>> > >>>> >>> -Gour > >>>> >>> > >>>> >>>> On Jan 12, 2016, at 6:21 PM, "Manoj > >>>>Samel"<manojsamelt...@gmail.com > >>>> > > >>>> >>> wrote: > >>>> >>>> Agreed. But I am not seeing even the host/port that is supposed > >>>>to > >>>> be > >>>> >>>>in > >>>> >>> ZK > >>>> >>>> Thanks, > >>>> >>>> > >>>> >>>> Manoj > >>>> >>>> > >>>> >>>>> On Tue, Jan 12, 2016 at 5:52 PM, Gour > >>>>Saha<gs...@hortonworks.com> > >>>> >>> wrote: > >>>> >>>>> In case you are expecting the data to be in zk, then you should > >>>> know > >>>> >>> that > >>>> >>>>> Slider stores a very small amount of metadata (primarily > >>>>endpoints > >>>> >>>>>and > >>>> >>>>> resource paths to AM) using which you can retrieve the actual > >>>>data. > >>>> >>>>>Most > >>>> >>>>> of the information is stored in AM runtime memory and not > >>>>persisted > >>>> >>>>> anywhere. In case of AM failure all the AM state is rebuilt. > >>>> >>>>> > >>>> >>>>> -Gour > >>>> >>>>> > >>>> >>>>>> On 1/12/16, 5:24 PM, "Manoj Samel"<manojsamelt...@gmail.com> > >>>> wrote: > >>>> >>>>>> > >>>> >>>>>> Hi Gour, > >>>> >>>>>> > >>>> >>>>>> As you asked, the urls and their outputs > >>>> >>>>>> > >>>> >>>>>> http://host36:1025/ws/v1/slider/publisher/exports > >>>> >>>>>> > >>>> >>>>>> > >>>> >>> > >>>> > >>>> > >>>>>>>{"exports":{"container_log_dirs":{"description":"container_log_dirs" > >>>>>>>,"up > >>>> >>>da > >>>> >>>>>> ted":1452647463881,"updatedTime":"Wed > >>>> >>>>>> Jan 13 01:11:03 UTC > >>>> >>>>>> > >>>> >>> > >>>> > >>>> > >>>>>>>2016","entries":{},"empty":true},"container_work_dirs":{"description > >>>>>>>":"c > >>>> >>>on > >>>> >>>>>> tainer_work_dirs","updated":1452647463881,"updatedTime":"Wed > >>>> >>>>>> Jan 13 01:11:03 UTC 2016","entries":{},"empty":true}}} > >>>> >>>>>> > >>>> >>>>>> http://host36:1025/ws/v1/slider/publisher > >>>> >>>>>> > >>>> >>>>>> {"uris":{"slider":" > >>>> http://host36:1025/ws/v1/slider/publisher/slider > >>>> >>>>> ","expo > >>>> >>>>>> rts":"http://host36:1025/ws/v1/slider/publisher/exports"}} > >>>> >>>>>> > >>>> >>>>>> http://host36:1025/ws/v1/slider/publisher/slider > >>>> >>>>>> > >>>> >>>>>> {"configurations":{"hdfs-site":{"description":"HDFS site > >>>> >>>>>> > >>>> >>> > >>>> > >>>> > >>>>>>>settings","updated":0,"entries":{},"empty":true},"yarn-site":{"descr > >>>>>>>ipti > >>>> >>>on > >>>> >>>>>> ":"YARN > >>>> >>>>>> site > >>>> >>>>>> > >>>> >>> > >>>> > >>>> > >>>>>>>settings","updated":0,"entries":{},"empty":true},"complete-config":{ > >>>>>>>"des > >>>> >>>cr > >>>> >>>>>> iption":"Complete > >>>> >>>>>> slider application > >>>> >>>>>> > >>>> >>> > >>>> > >>>> > >>>>>>>settings","updated":0,"entries":{},"empty":true},"core-site":{"descr > >>>>>>>ipti > >>>> >>>on > >>>> >>>>>> ":"Core > >>>> >>>>>> site > >>>> >>>>>> > >>>> >>> > >>>> > >>>> > >>>>>>>settings","updated":0,"entries":{},"empty":true},"componentinstanced > >>>>>>>ata" > >>>> >>>:{ > >>>> >>> > >>>> > >>>> > >>>>>>>"description":"ComponentInstanceData","updated":0,"entries":{},"empt > >>>>>>>y":t > >>>> >>>ru > >>>> >>>>>> e}}} > >>>> >>>>>> > >>>> >>>>>> Thanks, > >>>> >>>>>> > >>>> >>>>>> Manoj > >>>> >>>>>> > >>>> >>>>>> > >>>> >>>>>>> On Tue, Jan 12, 2016 at 3:03 PM, Gour > >>>>Saha<gs...@hortonworks.com > >>>> > > >>>> >>> wrote: > >>>> >>>>>>> When you do the below get, you will see an api entry for > >>>> >>>>>>> "classpath:org.apache.slider.publisher.exports" > >>>> >>>>>>> > >>>> >>>>>>> get /registry/users/foo/services/org-apache-slider/<appname> > >>>> >>>>>>> > >>>> >>>>>>> Do a curl on the uri value. The uri will be in the format of > >>>> >>>>>>> http://host:port/ws/v1/slider/publisher/exports. Do you find > >>>>the > >>>> >>>>>>>info > >>>> >>>>>>> you > >>>> >>>>>>> are looking for there? > >>>> >>>>>>> > >>>> >>>>>>> Also do a curl on the path ws/v1/slider/publisher and its > >>>> >>>>>>>sub-paths. > >>>> >>>>>>> What > >>>> >>>>>>> do you see? > >>>> >>>>>>> > >>>> >>>>>>> -Gour > >>>> >>>>>>> > >>>> >>>>>>> From: Manoj Samel<manojsamelt...@gmail.com<mailto: > >>>> >>>>>>> manojsamelt...@gmail.com>> > >>>> >>>>>>> Reply-To: "dev@slider.incubator.apache.org<mailto: > >>>> >>>>>>> dev@slider.incubator.apache.org>" > >>>> >>>>>>> <dev@slider.incubator.apache.org<mailto: > >>>> >>>>>>> dev@slider.incubator.apache.org>> > >>>> >>>>>>> Date: Tuesday, January 12, 2016 at 11:24 AM > >>>> >>>>>>> To: "dev@slider.incubator.apache.org<mailto: > >>>> >>>>>>> dev@slider.incubator.apache.org>" > >>>> >>>>>>> <dev@slider.incubator.apache.org<mailto: > >>>> >>>>>>> dev@slider.incubator.apache.org>> > >>>> >>>>>>> Subject: Re: Component level host:port export seen in AM but > >>>>not > >>>> >>>>>>>in ZK > >>>> >>>>>>> > >>>> >>>>>>> Hi Josh, > >>>> >>>>>>> > >>>> >>>>>>> I have created a minimal use case using the jmemcached to > >>>> reproduce > >>>> >>> the > >>>> >>>>>>> problem. The only change from OOB jmemcached is I am passing > >>>> >>> additional > >>>> >>>>>>> ports (not relevant for jmem but relevant for my ultimate use > >>>> >>>>>>>case), > >>>> >>>>>>> one of > >>>> >>>>>>> which I try to export. > >>>> >>>>>>> > >>>> >>>>>>> I have attached the 3 config files as well as a output file > >>>> >>>>>>>containing > >>>> >>>>>>> the > >>>> >>>>>>> slider-am URL output (which contains the component host:port) > >>>>and > >>>> >>>>>>>ZK > >>>> >>>>>>> output > >>>> >>>>>>> (which does not contains component host:port). > >>>> >>>>>>> > >>>> >>>>>>> Thanks for looking into the issue& let me know if you need > >>>> >>>>>>>additional > >>>> >>>>>>> info > >>>> >>>>>>> > >>>> >>>>>>> Manoj > >>>> >>>>>>> > >>>> >>>>>>> On Sun, Jan 10, 2016 at 11:26 AM, Josh Elser > >>>> >>>>>>> <josh.el...@gmail.com<mailto: > >>>> >>>>>>> josh.el...@gmail.com>> wrote: > >>>> >>>>>>> Hey Manoj, > >>>> >>>>>>> > >>>> >>>>>>> I'm not sure at a glance, but it might be a bug in Slider's > >>>>use > >>>> of > >>>> >>>>>>>the > >>>> >>>>>>> registry? I'm not quite sure, I'd have to dig into the source > >>>>to > >>>> be > >>>> >>>>>>> sure. > >>>> >>>>>>> > >>>> >>>>>>> Do you have an example that I could run which exhibits this? > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> Manoj Samel wrote: > >>>> >>>>>>> Any thoughts ? > >>>> >>>>>>> > >>>> >>>>>>> Thanks, > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> On Thu, Jan 7, 2016 at 10:25 AM, Manoj > >>>> >>>>>>>Samel<manojsamelt...@gmail.com > >>>> >>>>>>> <mailto:manojsamelt...@gmail.com>> > >>>> >>>>>>> wrote: > >>>> >>>>>>> > >>>> >>>>>>> Hi, > >>>> >>>>>>> > >>>> >>>>>>> Slider 0.80 with secured cluster. Registry is enabled *** > >>>>without > >>>> >>>>>>>*** > >>>> >>>>>>> security > >>>> >>>>>>> > >>>> >>>>>>> Export is configured as follows > >>>> >>>>>>> > >>>> >>>>>>> 1. No application level export > >>>> >>>>>>> 2. Component level port allocated per container. In > >>>> appConfig.json, > >>>> >>> *** > >>>> >>>>>>> "global" *** section > >>>> >>>>>>> "site.global.user1.http_port": > >>>> >>>>>>> "${user1.ALLOCATED_PORT}{PER_CONTAINER}", > >>>> >>>>>>> 3. Each component exports its own host/port (slider > >>>>allocated). > >>>> >>>>>>> metainfo.xml > >>>> >>>>>>> <components> > >>>> >>>>>>> <component> > >>>> >>>>>>> <name>user1</name> > >>>> >>>>>>> <category>MASTER</category> > >>>> >>>>>>> <componentExports> > >>>> >>>>>>> <componentExport> > >>>> >>>>>>> <name>my_info</name> > >>>> >>>>>>> > >>>> >>>>>>><value>${THIS_HOST}:${user1.http_port}</value> > >>>> >>>>>>> </componentExport> > >>>> >>>>>>> </componentExports> > >>>> >>>>>>> <commandScript> > >>>> >>>>>>> <script>scripts/memcached.py</script> > >>>> >>>>>>> <scriptType>PYTHON</scriptType> > >>>> >>>>>>> </commandScript> > >>>> >>>>>>> </component> > >>>> >>>>>>> > >>>> >>>>>>> At runtime, in > >>>> >>>>>>> > >>>> >>> > >>>> > >>>> > http://am-host:1025/ws/v1/slider/publisher/slider/componentinstancedata > >>>> >>>>> , > >>>> >>>>>>> the host:port info can be seen as expected > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>> > >>>> > >>>> > >>>>>>>{"description":"ComponentInstanceData","updated":0,"entries":{"conta > >>>>>>>iner > >>>> >>>_ > >>>> >>>>>>> 1452045276360_0005_01_000004.my_info":"host1:33786", > >>>> >>>>>>> .... > >>>> >>>>>>> > >>>> >>>>>>> However, when I look at ZK data, > >>>> >>>>>>> > >>>> >>>>>>> get > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>> > >>>> > >>>> > >>>>>>>/registry/users/foo/services/org-apache-slider/<appname>/components/ > >>>>>>>cont > >>>> >>>a > >>>> >>>>>>> iner-1452045276360-0005-01-000004, > >>>> >>>>>>> I don't see the host:port info. It only shows > >>>> >>>>>>> > >>>> >>>>>>> { > >>>> >>>>>>> "type" : "JSONServiceRecord", > >>>> >>>>>>> "description" : "user1", > >>>> >>>>>>> "external" : [ ], > >>>> >>>>>>> "internal" : [ ], > >>>> >>>>>>> "yarn:id" : "container-1452045276360-0005-01-000004", > >>>> >>>>>>> "yarn:persistence" : "container" > >>>> >>>>>>> } > >>>> >>>>>>> > >>>> >>>>>>> Any thought what I may be missing ? > >>>> >>>>>>> > >>>> >>>>>>> Related question - looking @ > >>>> >>> > >>>> >>> > >>>> > >>>> > https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/regi > >>>>s > >>>> >>>t > >>>> >>>>>>> ry/yarn-registry.html, > >>>> >>>>>>> the spec and all examples of export list "protocol" > >>>> >>>>>>> > >>>> >>>>>>> e.g. REST, rmi etc. Its not clear if and how is that protocol > >>>>to > >>>> be > >>>> >>>>>>> specified in the component export section ? Can anyone give a > >>>> >>>>>>>example > >>>> >>> ? > >>>> >>>>>>> Thanks, > >>>> >>>>>>> > >>>> >>>>>>> Manoj > >>>> >>>>> > >>>> >> > >>>> > > >>>> > >>>> > >>> > >> > >
