The only host port that is in zk is that of the AM.
-Gour
On Jan 12, 2016, at 6:21 PM, "Manoj Samel"<[email protected]>
wrote:
Agreed. But I am not seeing even the host/port that is supposed to be in
ZK
Thanks,
Manoj
On Tue, Jan 12, 2016 at 5:52 PM, Gour Saha<[email protected]>
wrote:
In case you are expecting the data to be in zk, then you should know
that
Slider stores a very small amount of metadata (primarily endpoints and
resource paths to AM) using which you can retrieve the actual data. Most
of the information is stored in AM runtime memory and not persisted
anywhere. In case of AM failure all the AM state is rebuilt.
-Gour
On 1/12/16, 5:24 PM, "Manoj Samel"<[email protected]> wrote:
Hi Gour,
As you asked, the urls and their outputs
http://host36:1025/ws/v1/slider/publisher/exports
{"exports":{"container_log_dirs":{"description":"container_log_dirs","upda
ted":1452647463881,"updatedTime":"Wed
Jan 13 01:11:03 UTC
2016","entries":{},"empty":true},"container_work_dirs":{"description":"con
tainer_work_dirs","updated":1452647463881,"updatedTime":"Wed
Jan 13 01:11:03 UTC 2016","entries":{},"empty":true}}}
http://host36:1025/ws/v1/slider/publisher
{"uris":{"slider":"http://host36:1025/ws/v1/slider/publisher/slider
","expo
rts":"http://host36:1025/ws/v1/slider/publisher/exports"}}
http://host36:1025/ws/v1/slider/publisher/slider
{"configurations":{"hdfs-site":{"description":"HDFS site
settings","updated":0,"entries":{},"empty":true},"yarn-site":{"description
":"YARN
site
settings","updated":0,"entries":{},"empty":true},"complete-config":{"descr
iption":"Complete
slider application
settings","updated":0,"entries":{},"empty":true},"core-site":{"description
":"Core
site
settings","updated":0,"entries":{},"empty":true},"componentinstancedata":{
"description":"ComponentInstanceData","updated":0,"entries":{},"empty":tru
e}}}
Thanks,
Manoj
On Tue, Jan 12, 2016 at 3:03 PM, Gour Saha<[email protected]>
wrote:
When you do the below get, you will see an api entry for
"classpath:org.apache.slider.publisher.exports"
get /registry/users/foo/services/org-apache-slider/<appname>
Do a curl on the uri value. The uri will be in the format of
http://host:port/ws/v1/slider/publisher/exports. Do you find the info
you
are looking for there?
Also do a curl on the path ws/v1/slider/publisher and its sub-paths.
What
do you see?
-Gour
From: Manoj Samel<[email protected]<mailto:
[email protected]>>
Reply-To: "[email protected]<mailto:
[email protected]>"
<[email protected]<mailto:
[email protected]>>
Date: Tuesday, January 12, 2016 at 11:24 AM
To: "[email protected]<mailto:
[email protected]>"
<[email protected]<mailto:
[email protected]>>
Subject: Re: Component level host:port export seen in AM but not in ZK
Hi Josh,
I have created a minimal use case using the jmemcached to reproduce
the
problem. The only change from OOB jmemcached is I am passing
additional
ports (not relevant for jmem but relevant for my ultimate use case),
one of
which I try to export.
I have attached the 3 config files as well as a output file containing
the
slider-am URL output (which contains the component host:port) and ZK
output
(which does not contains component host:port).
Thanks for looking into the issue& let me know if you need additional
info
Manoj
On Sun, Jan 10, 2016 at 11:26 AM, Josh Elser
<[email protected]<mailto:
[email protected]>> wrote:
Hey Manoj,
I'm not sure at a glance, but it might be a bug in Slider's use of the
registry? I'm not quite sure, I'd have to dig into the source to be
sure.
Do you have an example that I could run which exhibits this?
Manoj Samel wrote:
Any thoughts ?
Thanks,
On Thu, Jan 7, 2016 at 10:25 AM, Manoj Samel<[email protected]
<mailto:[email protected]>>
wrote:
Hi,
Slider 0.80 with secured cluster. Registry is enabled *** without ***
security
Export is configured as follows
1. No application level export
2. Component level port allocated per container. In appConfig.json,
***
"global" *** section
"site.global.user1.http_port":
"${user1.ALLOCATED_PORT}{PER_CONTAINER}",
3. Each component exports its own host/port (slider allocated).
metainfo.xml
<components>
<component>
<name>user1</name>
<category>MASTER</category>
<componentExports>
<componentExport>
<name>my_info</name>
<value>${THIS_HOST}:${user1.http_port}</value>
</componentExport>
</componentExports>
<commandScript>
<script>scripts/memcached.py</script>
<scriptType>PYTHON</scriptType>
</commandScript>
</component>
At runtime, in
http://am-host:1025/ws/v1/slider/publisher/slider/componentinstancedata
,
the host:port info can be seen as expected
{"description":"ComponentInstanceData","updated":0,"entries":{"container_
1452045276360_0005_01_000004.my_info":"host1:33786",
....
However, when I look at ZK data,
get
/registry/users/foo/services/org-apache-slider/<appname>/components/conta
iner-1452045276360-0005-01-000004,
I don't see the host:port info. It only shows
{
"type" : "JSONServiceRecord",
"description" : "user1",
"external" : [ ],
"internal" : [ ],
"yarn:id" : "container-1452045276360-0005-01-000004",
"yarn:persistence" : "container"
}
Any thought what I may be missing ?
Related question - looking @
https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/regist
ry/yarn-registry.html,
the spec and all examples of export list "protocol"
e.g. REST, rmi etc. Its not clear if and how is that protocol to be
specified in the component export section ? Can anyone give a example
?
Thanks,
Manoj
