Hello All:
First, thanks to Billie Rinaldi for the previous help, however I am still
trying to get the memcached example to work in my slider environment and I
could use some guidance as to whether there is some configuration error or an
actual slider issue.
Informally it looks like the application master container is launched but the
container that is supposed to launch memcached is failing at registration time,
in particular, after adding some logging, I wee that the Registration Response
is as follows:
INFO 2017-04-27 21:58:40,722 Controller.py:140 - Registration response =
'<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1"/>\n<title>Error 415 Unsupported Media
Type</title>\n</head>\n<body><h2>HTTP ERROR 415</h2>\n<p>Problem accessing
/ws/v1/slider/agents/container_1493330089109_0001_01_000002___MEMCACHED/register.
Reason:\n<pre> Unsupported Media Type</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>
\n<br/> \n<br/>
\n<br/>
\n<br/> \n<br/>
\n<br/>
\n<br/> \n<br/>
\n<br/>
\n<br/> \n<br/>
\n<br/>
\n<br/>
\n<br/> \n<br/>
\n<br/>
\n<br/> \n<br/>
\n<br/>
\n\n</body>\n</html>\n'
Here is a more detailed description of what I did and my configuration. I run
the following commands in a single node docker instance (effectively on a
logical edge node):
To install the package I run the command:
/usr/lib/slider/bin/slider install-package --package
/tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.1.zip
--name jmemcached --debug --replacepkg
To launch the package I run the command:
/usr/lib/slider/bin/slider create jmemcached --template
/tmp/mybuild/incubator-slider/app-packages/memcached/appConfig.json --resources
/tmp/mybuild/incubator-slider/app-packages/memcached/resources-default.json
--manager localhost:8032 --debug
Where the configuration files read:
[root@quickstart mybuild]# cat
/tmp/mybuild/incubator-slider/app-packages/memcached/appConfig.json
{
"schema": "http://example.org/specification/v2.0.0";,
"metadata": {
},
"global": {
"agent.conf": "artifacts/agent/conf/agent.ini",
"application.def": ".slider/package/jmemcached/jmemcached-1.0.1.zip",
"java_home": "/usr/lib/jvm/java/",
"site.global.additional_cp": "",
"site.global.xmx_val": "256m",
"site.global.xms_val": "128m",
"site.global.memory_val": "200M",
"site.global.listen_port": "11211"
},
"components": {
"slider-appmaster": {
"jvm.heapsize": "256M"
}
}
}
[root@quickstart mybuild]# cat
/tmp/mybuild/incubator-slider/app-packages/memcached/resources-default.json
{
"schema" : "http://example.org/specification/v2.0.0";,
"metadata" : {
},
"global" : {
},
"components": {
"slider-appmaster": {
},
"MEMCACHED": {
"yarn.role.priority": "1",
"yarn.component.instances": "1",
"yarn.memory": "256"
}
}
}
[root@quickstart mybuild]# cat
./incubator-slider/app-packages/memcached/metainfo.xml
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<metainfo>
<schemaVersion>2.0</schemaVersion>
<application>
<name>MEMCACHED</name>
<comment>Memcache is a network accessible key/value storage system, often
used as a distributed cache.</comment>
<!-- Changed version to see if I can get a hand built fat jar to work,
where I control dependencies
<version>1.0.0</version>
-->
<version>1.0.1-SNAPSHOT-jar-with-dependencies</version>
<exportedConfigs>None</exportedConfigs>
<exportGroups>
<exportGroup>
<name>Servers</name>
<exports>
<export>
<name>host_port</name>
<value>${MEMCACHED_HOST}:${site.global.listen_port}</value>
</export>
</exports>
</exportGroup>
</exportGroups>
<components>
<component>
<name>MEMCACHED</name>
<category>MASTER</category>
<compExports>Servers-host_port</compExports>
<commandScript>
<script>scripts/memcached.py</script>
<scriptType>PYTHON</scriptType>
</commandScript>
</component>
</components>
<osSpecifics>
<osSpecific>
<osType>any</osType>
<packages>
<package>
<type>tarball</type>
<name>files/jmemcached-1.0.1.tar</name>
</package>
</packages>
</osSpecific>
</osSpecifics>
</application>
</metainfo>
I did some code diving and enhance the logging, based on earlier observations:
In Controller.py I added some logging, in particular look at the Registration
Response and to determine if the registerUrl was well formed.
self.registerUrl = self.server_url + SLIDER_PATH_AGENTS + self.label +
SLIDER_REL_PATH_REGISTER
#Some code omitted
while not self.isRegistered:
try:
data = json.dumps(self.register.build(
self.componentActualState,
self.componentExpectedState,
self.actionQueue.customServiceOrchestrator.allocated_ports,
self.actionQueue.customServiceOrchestrator.log_folders,
self.tags,
id))
logger.info("Registering with the server at " + self.registerUrl +
" with data " + pprint.pformat(data))
response = self.sendRequest(self.registerUrl, data)
logger.info("Registration response = " + pprint.pformat(response))
logger.info("Dumping dir(self) = " + pprint.pformat(dir(self)))
logger.info("Dumping vars(self) = " + pprint.pformat(vars(self)))
logger.info("Dumping vars(self.config" + pprint.pformat(vars(self.config)))
regResp = json.loads(response)
The logs indicated several HTTP 415 error responses, that seemed to prevent the
launch.
INFO 2017-04-27 21:58:40,606 Controller.py:138 - Registering with the server at
https://quickstart.cloudera:42131/ws/v1/slider/agents/container_1493330089109_0001_01_000002___MEMCACHED/register
with data '{"actualState": 0, "logFolders": {}, "agentVersion": "1",
"allocatedPorts": {}, "timestamp": 1493330320605, "expectedState": 0, "tags":
"", "responseId": -1, "publicHostname": "quickstart.cloudera", "label":
"container_1493330089109_0001_01_000002___MEMCACHED"}'
[ Lines removed to reduce log noise]
INFO 2017-04-27 21:58:40,722 Controller.py:140 - Registration response =
'<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1"/>\n<title>Error 415 Unsupported Media
Type</title>\n</head>\n<body><h2>HTTP ERROR 415</h2>\n<p>Problem accessing
/ws/v1/slider/agents/container_1493330089109_0001_01_000002___MEMCACHED/register.
Reason:\n<pre> Unsupported Media Type</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>
\n<br/> \n<br/>
\n<br/>
\n<br/> \n<br/>
\n<br/>
\n<br/> \n<br/>
\n<br/>
\n<br/> \n<br/>
\n<br/>
\n<br/>
\n<br/> \n<br/>
\n<br/>
\n<br/> \n<br/>
\n<br/>
\n\n</body>\n</html>\n'
[Lines removed here to reduce log noise]
INFO 2017-04-27 21:58:40,724 Controller.py:143 - Dumping
vars(self.config{'label': 'container_1493330089109_0001_01_000002___MEMCACHED',
'logroot':
'/var/log/hadoop-yarn/containers/application_1493330089109_0001/container_1493330089109_0001_01_000002',
'workroot':
'/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/root/appcache/application_1493330089109_0001/container_1493330089109_0001_01_000002'}
INFO 2017-04-27 21:58:40,725 Controller.py:185 - Unable to connect to:
https://quickstart.cloudera:42131/ws/v1/slider/agents/container_1493330089109_0001_01_000002___MEMCACHED/register
Traceback (most recent call last):
File
"/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/root/appcache/application_1493330089109_0001/filecache/21/slider-agent.tar.gz/slider-agent/agent/Controller.py",
line 144, in registerWithServer
regResp = json.loads(response)
File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
I did some code diving and enhance the logging, where I put strings in a python
list and dump the information once the logging output is configured (hence the
multiple logged messages at line 265).
i.e, main.py has:
def main():
parser = OptionParser()
parser.add_option("-v", "--verbose", dest="verbose", help="verbose log
output", default=False)
parser.add_option("-l", "--label", dest="label", help="label of the agent",
default=None)
parser.add_option("--zk-quorum", dest=Constants.ZK_QUORUM, help="Zookeeper
Quorum", default=None)
parser.add_option("--zk-reg-path", dest=Constants.ZK_REG_PATH,
help="Zookeeper Registry Path", default=None)
parser.add_option("--debug", dest="debug", help="Agent debug hint",
default="")
(options, args) = parser.parse_args()
preLogConfigStrs = ["after parse_args, args = " + pprint.pformat(args),
"after parse_args, options = " + pprint.pformat(options),
"after parse_args, vars(options) = " + pprint.pformat(vars(options)),
"after parse_args, sys.argv = " + pprint.pformat(sys.argv)]
#some code omitted
logger.info("Using AGENT_WORK_ROOT = " + options.root_folder)
logger.info("Using AGENT_LOG_ROOT = " + options.log_folder)
logger.info("logFile = " + repr(logFile))
logger.info("Dumping data to log acquired before logging configuration was
complete")
for str in preLogConfigStrs:
logger.info(str)
Further inspection of the failed containers launching jmemcached, shows that
the command line arguments have the
INFO 2017-04-27 21:58:40,285 main.py:97 - Newloglevel=logging.DEBUG
INFO 2017-04-27 21:58:40,285 main.py:260 - Using AGENT_WORK_ROOT =
/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/root/appcache/application_1493330089109_0001/container_1493330089109_0001_01_000002
INFO 2017-04-27 21:58:40,285 main.py:261 - Using AGENT_LOG_ROOT =
/var/log/hadoop-yarn/containers/application_1493330089109_0001/container_1493330089109_0001_01_000002
INFO 2017-04-27 21:58:40,285 main.py:262 - logFile =
'/var/log/hadoop-yarn/containers/application_1493330089109_0001/container_1493330089109_0001_01_000002/./slider-agent.log'
INFO 2017-04-27 21:58:40,285 main.py:263 - Dumping data to log acquired before
logging configuration was complete
INFO 2017-04-27 21:58:40,285 main.py:265 - after parse_args, args = []
INFO 2017-04-27 21:58:40,285 main.py:265 - after parse_args, options = <Values
at 0x2bde2d8: {'debug': '', 'zk_quorum': 'localhost:2181', 'zk_reg_path':
'/registry/users/root/services/org-apache-slider/jmemcached', 'verbose': False,
'label': 'container_1493330089109_0001_01_000002___MEMCACHED'}>
INFO 2017-04-27 21:58:40,285 main.py:265 - after parse_args, vars(options) =
{'debug': '',
'label': 'container_1493330089109_0001_01_000002___MEMCACHED',
'verbose': False,
'zk_quorum': 'localhost:2181',
'zk_reg_path': '/registry/users/root/services/org-apache-slider/jmemcached'}
INFO 2017-04-27 21:58:40,285 main.py:265 - after parse_args, sys.argv =
['./infra/agent/slider-agent/agent/main.py',
'--label',
'container_1493330089109_0001_01_000002___MEMCACHED',
'--zk-quorum',
'localhost:2181',
'--zk-reg-path',
'/registry/users/root/services/org-apache-slider/jmemcached']
I'm still trying to get to the bottom of this, any ideas of what should be done
next?
With best regards:
Bill
