I tried spinning up a memcached app and did not encounter this issue. I
don't really see anything wrong with your configuration. We do not
recommend trying to use a custom agent.ini, so I'd remove that. Also you
don't have to specify --manager localhost:8032 in the Slider create command
when Slider is reading that info from the yarn-site.xml. I don't expect
those changes to affect the issue you're seeing, though. What version of
Python are you using? What does the AM log say?
On Thu, Apr 27, 2017 at 5:47 PM, Foolish Ewe <foolish...@hotmail.com> wrote:
> Hello All:
>
>
> First, thanks to Billie Rinaldi for the previous help, however I am still
> trying to get the memcached example to work in my slider environment and I
> could use some guidance as to whether there is some configuration error or
> an actual slider issue.
>
> Informally it looks like the application master container is launched but
> the container that is supposed to launch memcached is failing at
> registration time, in particular, after adding some logging, I wee that the
> Registration Response is as follows:
>
> INFO 2017-04-27 21:58:40,722 Controller.py:140 - Registration response =
> '<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;
> charset=ISO-8859-1"/>\n<title>Error 415 Unsupported Media
> Type</title>\n</head>\n<body><h2>HTTP ERROR 415</h2>\n<p>Problem
> accessing
> /ws/v1/slider/agents/container_1493330089109_0001_01_000002___MEMCACHED/register.
> Reason:\n<pre> Unsupported Media Type</pre></p><hr /><i><small>Powered
> by Jetty://</small></i><br/>
> \n<br/> \n<br/>
> \n<br/>
> \n<br/>
> \n<br/> \n<br/>
> \n<br/>
> \n<br/>
> \n<br/> \n<br/>
> \n<br/>
> \n<br/>
> \n<br/> \n<br/>
> \n<br/>
> \n<br/>
> \n<br/> \n<br/>
> \n<br/>
> \n\n</body>\n</html>\n'
>
>
>
>
> Here is a more detailed description of what I did and my configuration. I
> run the following commands in a single node docker instance (effectively on
> a logical edge node):
>
> To install the package I run the command:
>
> /usr/lib/slider/bin/slider install-package --package
> /tmp/mybuild/incubator-slider/app-packages/memcached/jmemcached-1.0.1.zip
> --name jmemcached --debug --replacepkg
>
> To launch the package I run the command:
>
> /usr/lib/slider/bin/slider create jmemcached --template
> /tmp/mybuild/incubator-slider/app-packages/memcached/appConfig.json
> --resources
> /tmp/mybuild/incubator-slider/app-packages/memcached/resources-default.json
> --manager localhost:8032 --debug
>
> Where the configuration files read:
>
> [root@quickstart mybuild]# cat /tmp/mybuild/incubator-slider/
> app-packages/memcached/appConfig.json
>
> {
>
> "schema": "http://example.org/specification/v2.0.0";,
>
> "metadata": {
>
> },
>
> "global": {
>
> "agent.conf": "artifacts/agent/conf/agent.ini",
>
> "application.def": ".slider/package/jmemcached/jmemcached-1.0.1.zip",
>
> "java_home": "/usr/lib/jvm/java/",
>
> "site.global.additional_cp": "",
>
> "site.global.xmx_val": "256m",
>
> "site.global.xms_val": "128m",
>
> "site.global.memory_val": "200M",
>
> "site.global.listen_port": "11211"
>
> },
>
> "components": {
>
> "slider-appmaster": {
>
> "jvm.heapsize": "256M"
>
> }
>
> }
>
> }
>
> [root@quickstart mybuild]# cat /tmp/mybuild/incubator-slider/
> app-packages/memcached/resources-default.json
>
> {
>
> "schema" : "http://example.org/specification/v2.0.0";,
>
> "metadata" : {
>
> },
>
> "global" : {
>
> },
>
> "components": {
>
> "slider-appmaster": {
>
> },
>
> "MEMCACHED": {
>
> "yarn.role.priority": "1",
>
> "yarn.component.instances": "1",
>
> "yarn.memory": "256"
>
> }
>
> }
>
> }
>
> [root@quickstart mybuild]# cat ./incubator-slider/app-package
> s/memcached/metainfo.xml
>
> <?xml version="1.0"?>
>
> <!--
>
> Licensed to the Apache Software Foundation (ASF) under one or more
>
> contributor license agreements. See the NOTICE file distributed with
>
> this work for additional information regarding copyright ownership.
>
> The ASF licenses this file to You under the Apache License, Version 2.0
>
> (the "License"); you may not use this file except in compliance with
>
> the License. You may obtain a copy of the License at
>
>
> http://www.apache.org/licenses/LICENSE-2.0
>
>
> Unless required by applicable law or agreed to in writing, software
>
> distributed under the License is distributed on an "AS IS" BASIS,
>
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>
> See the License for the specific language governing permissions and
>
> limitations under the License.
>
> -->
>
>
> <metainfo>
>
> <schemaVersion>2.0</schemaVersion>
>
> <application>
>
> <name>MEMCACHED</name>
>
> <comment>Memcache is a network accessible key/value storage system,
> often used as a distributed cache.</comment>
>
> <!-- Changed version to see if I can get a hand built fat jar to work,
> where I control dependencies
>
> <version>1.0.0</version>
>
> -->
>
> <version>1.0.1-SNAPSHOT-jar-with-dependencies</version>
>
> <exportedConfigs>None</exportedConfigs>
>
> <exportGroups>
>
> <exportGroup>
>
> <name>Servers</name>
>
> <exports>
>
> <export>
>
> <name>host_port</name>
>
> <value>${MEMCACHED_HOST}:${site.global.listen_port}</value>
>
> </export>
>
> </exports>
>
> </exportGroup>
>
> </exportGroups>
>
>
> <components>
>
> <component>
>
> <name>MEMCACHED</name>
>
> <category>MASTER</category>
>
> <compExports>Servers-host_port</compExports>
>
> <commandScript>
>
> <script>scripts/memcached.py</script>
>
> <scriptType>PYTHON</scriptType>
>
> </commandScript>
>
> </component>
>
> </components>
>
>
> <osSpecifics>
>
> <osSpecific>
>
> <osType>any</osType>
>
> <packages>
>
> <package>
>
> <type>tarball</type>
>
> <name>files/jmemcached-1.0.1.tar</name>
>
> </package>
>
> </packages>
>
> </osSpecific>
>
> </osSpecifics>
>
>
> </application>
>
> </metainfo>
>
> I did some code diving and enhance the logging, based on earlier
> observations:
> In Controller.py I added some logging, in particular look at the
> Registration Response and to determine if the registerUrl was well formed.
>
> self.registerUrl = self.server_url + SLIDER_PATH_AGENTS + self.label +
> SLIDER_REL_PATH_REGISTER
>
> #Some code omitted
>
> while not self.isRegistered:
> try:
> data = json.dumps(self.register.build(
> self.componentActualState,
> self.componentExpectedState,
> self.actionQueue.customServiceOrchestrator.allocated_ports,
> self.actionQueue.customServiceOrchestrator.log_folders,
> self.tags,
> id))
> logger.info("Registering with the server at " + self.registerUrl +
> " with data " + pprint.pformat(data))
> response = self.sendRequest(self.registerUrl, data)
> logger.info("Registration response = " + pprint.pformat(response))
> logger.info("Dumping dir(self) = " + pprint.pformat(dir(self)))
> logger.info("Dumping vars(self) = " + pprint.pformat(vars(self)))
> logger.info("Dumping vars(self.config" +
> pprint.pformat(vars(self.config)))
> regResp = json.loads(response)
>
> The logs indicated several HTTP 415 error responses, that seemed to
> prevent the launch.
>
> INFO 2017-04-27 21:58:40,606 Controller.py:138 - Registering with the
> server at https://quickstart.cloudera:42131/ws/v1/slider/agents/contai
> ner_1493330089109_0001_01_000002___MEMCACHED/register with data
> '{"actualState": 0, "logFolders": {}, "agentVersion": "1",
> "allocatedPorts": {}, "timestamp": 1493330320605, "expectedState": 0,
> "tags": "", "responseId": -1, "publicHostname": "quickstart.cloudera",
> "label": "container_1493330089109_0001_01_000002___MEMCACHED"}'
>
>
> [ Lines removed to reduce log noise]
>
> INFO 2017-04-27 21:58:40,722 Controller.py:140 - Registration response =
> '<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;
> charset=ISO-8859-1"/>\n<title>Error 415 Unsupported Media
> Type</title>\n</head>\n<body><h2>HTTP ERROR 415</h2>\n<p>Problem
> accessing
> /ws/v1/slider/agents/container_1493330089109_0001_01_000002___MEMCACHED/register.
> Reason:\n<pre> Unsupported Media Type</pre></p><hr /><i><small>Powered
> by Jetty://</small></i><br/>
> \n<br/> \n<br/>
> \n<br/>
> \n<br/>
> \n<br/> \n<br/>
> \n<br/>
> \n<br/>
> \n<br/> \n<br/>
> \n<br/>
> \n<br/>
> \n<br/> \n<br/>
> \n<br/>
> \n<br/>
> \n<br/> \n<br/>
> \n<br/>
> \n\n</body>\n</html>\n'
> [Lines removed here to reduce log noise]
> INFO 2017-04-27 21:58:40,724 Controller.py:143 - Dumping
> vars(self.config{'label': 'container_1493330089109_0001_
> 01_000002___MEMCACHED',
> 'logroot': '/var/log/hadoop-yarn/containers/application_1493330089109_
> 0001/container_1493330089109_0001_01_000002',
> 'workroot': '/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/root
> /appcache/application_1493330089109_0001/container_149333008
> 9109_0001_01_000002'}
> INFO 2017-04-27 21:58:40,725 Controller.py:185 - Unable to connect to:
> https://quickstart.cloudera:42131/ws/v1/slider/agents/contai
> ner_1493330089109_0001_01_000002___MEMCACHED/register
> Traceback (most recent call last):
> File "/var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/root
> /appcache/application_1493330089109_0001/filecache/21/
> slider-agent.tar.gz/slider-agent/agent/Controller.py", line 144, in
> registerWithServer
> regResp = json.loads(response)
> File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
> return _default_decoder.decode(s)
> File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
> File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
> raise ValueError("No JSON object could be decoded")
> ValueError: No JSON object could be decoded
>
>
> I did some code diving and enhance the logging, where I put strings in a
> python list and dump the information once the logging output is configured
> (hence the multiple logged messages at line 265).
> i.e, main.py has:
>
> def main():
> parser = OptionParser()
> parser.add_option("-v", "--verbose", dest="verbose", help="verbose log
> output", default=False)
> parser.add_option("-l", "--label", dest="label", help="label of the
> agent", default=None)
> parser.add_option("--zk-quorum", dest=Constants.ZK_QUORUM,
> help="Zookeeper Quorum", default=None)
> parser.add_option("--zk-reg-path", dest=Constants.ZK_REG_PATH,
> help="Zookeeper Registry Path", default=None)
> parser.add_option("--debug", dest="debug", help="Agent debug hint",
> default="")
> (options, args) = parser.parse_args()
> preLogConfigStrs = ["after parse_args, args = " + pprint.pformat(args),
> "after parse_args, options = " + pprint.pformat(options),
> "after parse_args, vars(options) = " + pprint.pformat(vars(options)),
> "after parse_args, sys.argv = " + pprint.pformat(sys.argv)]
>
> #some code omitted
>
> logger.info("Using AGENT_WORK_ROOT = " + options.root_folder)
> logger.info("Using AGENT_LOG_ROOT = " + options.log_folder)
> logger.info("logFile = " + repr(logFile))
> logger.info("Dumping data to log acquired before logging configuration
> was complete")
> for str in preLogConfigStrs:
> logger.info(str)
>
> Further inspection of the failed containers launching jmemcached, shows
> that the command line arguments have the
>
> INFO 2017-04-27 21:58:40,285 main.py:97 - Newloglevel=logging.DEBUG
> INFO 2017-04-27 21:58:40,285 main.py:260 - Using AGENT_WORK_ROOT =
> /var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/root/
> appcache/application_1493330089109_0001/container_1493330089
> 109_0001_01_000002
> INFO 2017-04-27 21:58:40,285 main.py:261 - Using AGENT_LOG_ROOT =
> /var/log/hadoop-yarn/containers/application_1493330089109_
> 0001/container_1493330089109_0001_01_000002
> INFO 2017-04-27 21:58:40,285 main.py:262 - logFile =
> '/var/log/hadoop-yarn/containers/application_1493330089109_
> 0001/container_1493330089109_0001_01_000002/./slider-agent.log'
> INFO 2017-04-27 21:58:40,285 main.py:263 - Dumping data to log acquired
> before logging configuration was complete
> INFO 2017-04-27 21:58:40,285 main.py:265 - after parse_args, args = []
> INFO 2017-04-27 21:58:40,285 main.py:265 - after parse_args, options =
> <Values at 0x2bde2d8: {'debug': '', 'zk_quorum': 'localhost:2181',
> 'zk_reg_path': '/registry/users/root/services
> /org-apache-slider/jmemcached', 'verbose': False, 'label':
> 'container_1493330089109_0001_01_000002___MEMCACHED'}>
> INFO 2017-04-27 21:58:40,285 main.py:265 - after parse_args, vars(options)
> = {'debug': '',
> 'label': 'container_1493330089109_0001_01_000002___MEMCACHED',
> 'verbose': False,
> 'zk_quorum': 'localhost:2181',
> 'zk_reg_path': '/registry/users/root/services
> /org-apache-slider/jmemcached'}
> INFO 2017-04-27 21:58:40,285 main.py:265 - after parse_args, sys.argv =
> ['./infra/agent/slider-agent/agent/main.py',
> '--label',
> 'container_1493330089109_0001_01_000002___MEMCACHED',
> '--zk-quorum',
> 'localhost:2181',
> '--zk-reg-path',
> '/registry/users/root/services/org-apache-slider/jmemcached']
>
>
> I'm still trying to get to the bottom of this, any ideas of what should be
> done next?
>
>
> With best regards:
>
>
> Bill
>
