Public bug reported:
vi /var/log/nova-all.log
<180>Jul 27 10:15:33 node-1 nova-compute Auditing locally available compute
resources
<179>Jul 27 10:15:33 node-1 nova-compute Error during
ComputeManager.update_available_resource: [Errno 24] Too many open files
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
Traceback (most recent call last):
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py", line
198, in run_periodic_tasks
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
task(self, context)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5963, in
update_available_resource
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
rt.update_available_resource(context)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 313,
in update_available_resource
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
resources = self.driver.get_available_resource(self.nodename)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4939, in
get_available_resource
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
stats = self.get_host_stats(refresh=True)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5809, in
get_host_stats
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
return self.host_state.get_host_stats(refresh=refresh)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6383, in
get_host_stats
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
self.update_status()
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6406, in
update_status
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
disk_info_dict = self.driver._get_local_gb_info()
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4552, in
_get_local_gb_info
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
info = LibvirtDriver._get_rbd_driver().get_pool_info()
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/rbd_utils.py", line 273, in
get_pool_info
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
with RADOSClient(self) as client:
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/rbd_utils.py", line 86, in
__init__
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
self.cluster, self.ioctx = driver._connect_to_rados(pool)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/rbd_utils.py", line 108, in
_connect_to_rados
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
conffile=self.ceph_conf)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/dist-packages/rados.py", line 198, in __init__
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
librados_path = find_library('rados')
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/ctypes/util.py", line 224, in find_library
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task File
"/usr/lib/python2.7/ctypes/util.py", line 213, in _findSoname_ldconfig
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task f =
os.popen('/sbin/ldconfig -p 2>/dev/null')
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
OSError: [Errno 24] Too many open files
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
Current limit setting:
root@node-1:/tmp# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 386140
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 386140
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
root@node-1:/tmp# ulimit -n
1024
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1478630
Title:
nova-compute was forced down due to "[Errno 24] too many open files"
Status in OpenStack Compute (nova):
New
Bug description:
vi /var/log/nova-all.log
<180>Jul 27 10:15:33 node-1 nova-compute Auditing locally available compute
resources
<179>Jul 27 10:15:33 node-1 nova-compute Error during
ComputeManager.update_available_resource: [Errno 24] Too many open files
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
Traceback (most recent call last):
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py",
line 198, in run_periodic_tasks
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
task(self, context)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5963, in
update_available_resource
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
rt.update_available_resource(context)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line
313, in update_available_resource
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
resources = self.driver.get_available_resource(self.nodename)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4939,
in get_available_resource
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
stats = self.get_host_stats(refresh=True)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5809,
in get_host_stats
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
return self.host_state.get_host_stats(refresh=refresh)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6383,
in get_host_stats
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
self.update_status()
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6406,
in update_status
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
disk_info_dict = self.driver._get_local_gb_info()
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4552,
in _get_local_gb_info
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
info = LibvirtDriver._get_rbd_driver().get_pool_info()
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/rbd_utils.py", line
273, in get_pool_info
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
with RADOSClient(self) as client:
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/rbd_utils.py", line
86, in __init__
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
self.cluster, self.ioctx = driver._connect_to_rados(pool)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/rbd_utils.py", line
108, in _connect_to_rados
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
conffile=self.ceph_conf)
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/rados.py", line 198, in __init__
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
librados_path = find_library('rados')
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/ctypes/util.py", line 224, in find_library
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/ctypes/util.py", line 213, in _findSoname_ldconfig
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task f
= os.popen('/sbin/ldconfig -p 2>/dev/null')
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
OSError: [Errno 24] Too many open files
2015-07-27 10:15:33.401 12422 TRACE nova.openstack.common.periodic_task
Current limit setting:
root@node-1:/tmp# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 386140
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 386140
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
root@node-1:/tmp# ulimit -n
1024
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1478630/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
