Dmitry Lysnichenko created AMBARI-6748:
------------------------------------------
Summary: Nimbus start failed after deployment
Key: AMBARI-6748
URL: https://issues.apache.org/jira/browse/AMBARI-6748
Project: Ambari
Issue Type: Bug
Components: agent
Affects Versions: 1.7.0
Reporter: Dmitry Lysnichenko
Assignee: Dmitry Lysnichenko
Fix For: 1.7.0
Deployed HDP-2.1, start all services failed. Nimbus cannot start.
error log:
{code}
stderr:
2014-08-04 18:12:28,049 - Error while executing command 'start':
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 122, in execute
method(env)
File
"/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/STORM/package/scripts/nimbus.py",
line 43, in start
service("nimbus", action="start")
File
"/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/STORM/package/scripts/service.py",
line 64, in service
try_sleep=10
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
line 148, in __init__
self.env.run()
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 149, in run
self.run_action(resource, action)
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 115, in run_action
provider_action()
File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 241, in action_run
raise ex
Fail: Execution of 'pgrep -f "^java.+backtype.storm.daemon.nimbus$" && pgrep -f
"^java.+backtype.storm.daemon.nimbus$" > /var/run/storm/nimbus.pid' returned 1.
stdout:
2014-08-04 18:11:37,197 - Execute['mkdir -p /tmp/HDP-artifacts/; curl -kf
-x "" --retry 10
http://ambsmoke6-4-1407193726-1.cs1cloud.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip
-o /tmp/HDP-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'environment': ...,
'not_if': 'test -e /tmp/HDP-artifacts//UnlimitedJCEPolicyJDK7.zip',
'ignore_failures': True, 'path': ['/bin', '/usr/bin/']}
2014-08-04 18:11:37,211 - Skipping Execute['mkdir -p /tmp/HDP-artifacts/;
curl -kf -x "" --retry 10
http://ambsmoke6-4-1407193726-1.cs1cloud.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip
-o /tmp/HDP-artifacts//UnlimitedJCEPolicyJDK7.zip'] due to not_if
2014-08-04 18:11:37,212 - Execute['rm -f local_policy.jar; rm -f
US_export_policy.jar; unzip -o -j -q
/tmp/HDP-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'path': ['/bin/',
'/usr/bin'], 'only_if': 'test -e /usr/jdk64/jdk1.7.0_45/jre/lib/security &&
test -f /tmp/HDP-artifacts//UnlimitedJCEPolicyJDK7.zip', 'cwd':
'/usr/jdk64/jdk1.7.0_45/jre/lib/security'}
2014-08-04 18:11:37,390 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root',
'group': 'root', 'recursive': True}
2014-08-04 18:11:37,391 - Link['/etc/hadoop/conf'] {'not_if': 'ls
/etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'}
2014-08-04 18:11:37,404 - Skipping Link['/etc/hadoop/conf'] due to not_if
2014-08-04 18:11:37,419 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content':
InlineTemplate(...), 'owner': 'root'}
2014-08-04 18:11:37,419 - XmlConfig['core-site.xml'] {'owner': 'hdfs', 'group':
'hadoop', 'conf_dir': '/etc/hadoop/conf', 'configuration_attributes': ...,
'configurations': ...}
2014-08-04 18:11:37,429 - Generating config: /etc/hadoop/conf/core-site.xml
2014-08-04 18:11:37,430 - File['/etc/hadoop/conf/core-site.xml'] {'owner':
'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None,
'encoding': 'UTF-8'}
2014-08-04 18:11:37,431 - Writing File['/etc/hadoop/conf/core-site.xml']
because contents don't match
2014-08-04 18:11:37,443 - Execute['/bin/echo 0 > /selinux/enforce'] {'only_if':
'test -f /selinux/enforce'}
2014-08-04 18:11:37,456 - Skipping Execute['/bin/echo 0 > /selinux/enforce']
due to only_if
2014-08-04 18:11:37,457 - Execute['mkdir -p
/usr/lib/hadoop/lib/native/Linux-i386-32; ln -sf /usr/lib/libsnappy.so
/usr/lib/hadoop/lib/native/Linux-i386-32/libsnappy.so'] {}
2014-08-04 18:11:37,488 - Execute['mkdir -p
/usr/lib/hadoop/lib/native/Linux-amd64-64; ln -sf /usr/lib64/libsnappy.so
/usr/lib/hadoop/lib/native/Linux-amd64-64/libsnappy.so'] {}
2014-08-04 18:11:37,500 - Directory['/grid/0/log/hadoop'] {'owner': 'root',
'group': 'root', 'recursive': True}
2014-08-04 18:11:37,501 - Directory['/var/run/hadoop'] {'owner': 'root',
'group': 'root', 'recursive': True}
2014-08-04 18:11:37,502 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs',
'recursive': True}
2014-08-04 18:11:37,506 - File['/etc/hadoop/conf/commons-logging.properties']
{'content': Template('commons-logging.properties.j2'), 'owner': 'root'}
2014-08-04 18:11:37,508 - File['/etc/hadoop/conf/health_check'] {'content':
Template('health_check-v2.j2'), 'owner': 'root'}
2014-08-04 18:11:37,509 - File['/etc/hadoop/conf/log4j.properties'] {'content':
'...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2014-08-04 18:11:37,516 - File['/etc/hadoop/conf/hadoop-metrics2.properties']
{'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2014-08-04 18:11:37,517 - File['/etc/hadoop/conf/task-log4j.properties']
{'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2014-08-04 18:11:37,517 - File['/etc/hadoop/conf/configuration.xsl'] {'owner':
'hdfs', 'group': 'hadoop'}
2014-08-04 18:11:37,705 - Directory['/var/log/storm'] {'owner': 'storm',
'group': 'hadoop', 'recursive': True}
2014-08-04 18:11:37,707 - Directory['/var/run/storm'] {'owner': 'storm',
'group': 'hadoop', 'recursive': True}
2014-08-04 18:11:37,707 - Directory['/grid/0/hadoop/storm'] {'owner': 'storm',
'group': 'hadoop', 'recursive': True}
2014-08-04 18:11:37,707 - Directory['/etc/storm/conf'] {'owner': 'storm',
'group': 'hadoop', 'recursive': True}
2014-08-04 18:11:37,714 - File['/etc/storm/conf/config.yaml'] {'owner':
'storm', 'content': Template('config.yaml.j2'), 'group': 'hadoop'}
2014-08-04 18:11:37,719 - File['/etc/storm/conf/storm.yaml'] {'owner': 'storm',
'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None}
2014-08-04 18:11:37,721 - Writing File['/etc/storm/conf/storm.yaml'] because
contents don't match
2014-08-04 18:11:37,722 - File['/etc/storm/conf/storm-env.sh'] {'content':
'\n#!/bin/bash\n\n# Set Storm specific environment variables here.\n\n# The
java implementation to use.\nexport JAVA_HOME={{java_home}}\n\n# export
STORM_CONF_DIR=""', 'owner': 'storm'}
2014-08-04 18:11:37,722 - TemplateConfig['/etc/storm/conf/storm_jaas.conf']
{'owner': 'storm'}
2014-08-04 18:11:37,724 - File['/etc/storm/conf/storm_jaas.conf'] {'content':
Template('storm_jaas.conf.j2'), 'owner': 'storm', 'group': None, 'mode': None}
2014-08-04 18:11:37,725 - Execute['env JAVA_HOME=/usr/jdk64/jdk1.7.0_45
PATH=$PATH:/usr/jdk64/jdk1.7.0_45/bin /usr/bin/storm nimbus >
/var/log/storm/nimbus.out 2>&1'] {'wait_for_finish': False, 'not_if': 'ls
/var/run/storm/nimbus.pid >/dev/null 2>&1 && ps `cat /var/run/storm/nimbus.pid`
>/dev/null 2>&1', 'user': 'storm'}
2014-08-04 18:11:37,752 - Execute['pgrep -f
"^java.+backtype.storm.daemon.nimbus$" && pgrep -f
"^java.+backtype.storm.daemon.nimbus$" > /var/run/storm/nimbus.pid']
{'logoutput': True, 'tries': 6, 'user': 'storm', 'try_sleep': 10}
2014-08-04 18:11:37,790 - Retrying after 10 seconds. Reason: Execution of
'pgrep -f "^java.+backtype.storm.daemon.nimbus$" && pgrep -f
"^java.+backtype.storm.daemon.nimbus$" > /var/run/storm/nimbus.pid' returned 1.
2014-08-04 18:11:47,827 - Retrying after 10 seconds. Reason: Execution of
'pgrep -f "^java.+backtype.storm.daemon.nimbus$" && pgrep -f
"^java.+backtype.storm.daemon.nimbus$" > /var/run/storm/nimbus.pid' returned 1.
2014-08-04 18:11:57,881 - Retrying after 10 seconds. Reason: Execution of
'pgrep -f "^java.+backtype.storm.daemon.nimbus$" && pgrep -f
"^java.+backtype.storm.daemon.nimbus$" > /var/run/storm/nimbus.pid' returned 1.
2014-08-04 18:12:07,946 - Retrying after 10 seconds. Reason: Execution of
'pgrep -f "^java.+backtype.storm.daemon.nimbus$" && pgrep -f
"^java.+backtype.storm.daemon.nimbus$" > /var/run/storm/nimbus.pid' returned 1.
2014-08-04 18:12:17,989 - Retrying after 10 seconds. Reason: Execution of
'pgrep -f "^java.+backtype.storm.daemon.nimbus$" && pgrep -f
"^java.+backtype.storm.daemon.nimbus$" > /var/run/storm/nimbus.pid' returned 1.
2014-08-04 18:12:28,049 - Error while executing command 'start':
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 122, in execute
method(env)
File
"/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/STORM/package/scripts/nimbus.py",
line 43, in start
service("nimbus", action="start")
File
"/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/STORM/package/scripts/service.py",
line 64, in service
try_sleep=10
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
line 148, in __init__
self.env.run()
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 149, in run
self.run_action(resource, action)
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 115, in run_action
provider_action()
File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 241, in action_run
raise ex
Fail: Execution of 'pgrep -f "^java.+backtype.storm.daemon.nimbus$" && pgrep -f
"^java.+backtype.storm.daemon.nimbus$" > /var/run/storm/nimbus.pid' returned 1.
{code}
Nimbus.out
{code}
Traceback (most recent call last):
File "/usr/lib/storm/bin/storm.py", line 463, in <module>
main()
File "/usr/lib/storm/bin/storm.py", line 460, in main
(COMMANDS.get(COMMAND, unknown_command))(*ARGS)
File "/usr/lib/storm/bin/storm.py", line 276, in nimbus
jvmopts = parse_args(confvalue("nimbus.childopts", cppaths)) + [
File "/usr/lib/storm/bin/storm.py", line 87, in confvalue
p = sub.Popen(command, stdout=sub.PIPE)
File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
