[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]
** Changed in: nova Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1493783 Title: nova-compute ceph "too many open files" Status in OpenStack Compute (nova): Expired Bug description: We’ve deployed OpenStack with Ceph as a storage (rbd). The instances are being boot using cinder volumes through Ceph too. Sometimes the following ERROR appears at the nova-compute.log: [Errno 24] Too many open files If we get the list of open files from any nova-compute process, we’ll see the following: root@qk-3:~# ps -auxwww | grep nova-compute nova 137430 2.4 0.0 2395468 106888 ? Ssl Sep07 65:54 /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf root@qk-3:~# lsof -p 137430 | wc -l 5406 Almost all open files is: nova-comp 137430 nova 5232r FIFO 0,8 0t0 948057084 pipe nova-comp 137430 nova 5233u unix 0xffff881ad3192300 0t0 948012913 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok nova-comp 137430 nova 5234r FIFO 0,8 0t0 948057088 pipe nova-comp 137430 nova 5235u unix 0xffff881ad3192680 0t0 948057085 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok nova-comp 137430 nova 5236r FIFO 0,8 0t0 948073475 pipe nova-comp 137430 nova 5237u unix 0xffff881ad3192d80 0t0 948073473 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok nova-comp 137430 nova 5238r FIFO 0,8 0t0 948073522 pipe nova-comp 137430 nova 5239u unix 0xffff881ad3195080 0t0 948073476 /var/run/ceph/guests/ceph-client.cinder.137430.35361600.asok nova-comp 137430 nova 5240r FIFO 0,8 0t0 948073526 pipe nova-comp 137430 nova 5241u unix 0xffff881ad3197700 0t0 948073523 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok nova-comp 137430 nova 5242r FIFO 0,8 0t0 948073532 pipe nova-comp 137430 nova 5243u unix 0xffff881ad3196c80 0t0 948073527 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok nova-comp 137430 nova 5244r FIFO 0,8 0t0 948073535 pipe nova-comp 137430 nova 5245u unix 0xffff881ad3196580 0t0 948073533 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok nova-comp 137430 nova 5246r FIFO 0,8 0t0 948163602 pipe nova-comp 137430 nova 5247u unix 0xffff881ad3195b00 0t0 948073536 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok nova-comp 137430 nova 5248r FIFO 0,8 0t0 948160472 pipe nova-comp 137430 nova 5249u unix 0xffff88301f2ec600 0t0 948163603 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok nova-comp 137430 nova 5250r FIFO 0,8 0t0 948160475 pipe nova-comp 137430 nova 5251u unix 0xffff881ad3190000 0t0 948160473 /var/run/ceph/guests/ceph-client.cinder.137430.35361600.asok nova-comp 137430 nova 5252r FIFO 0,8 0t0 948160490 pipe nova-comp 137430 nova 5253u unix 0xffff881ad3190380 0t0 948160476 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok nova-comp 137430 nova 5254r FIFO 0,8 0t0 948160493 pipe nova-comp 137430 nova 5255u unix 0xffff881ad3195780 0t0 948160491 /var/run/ceph/guests/ceph-client.cinder.137430.35244752.asok nova-comp 137430 nova 5256r FIFO 0,8 0t0 948160498 pipe nova-comp 137430 nova 5257u unix 0xffff881fafe18000 0t0 948160494 /var/run/ceph/guests/ceph-client.cinder.137430.35260960.asok nova-comp 137430 nova 5258r FIFO 0,8 0t0 948261835 pipe nova-comp 137430 nova 5259u unix 0xffff881fafe18e00 0t0 948160499 /var/run/ceph/guests/ceph-client.cinder.137430.35666576.asok nova-comp 137430 nova 5260r FIFO 0,8 0t0 948265362 pipe nova-comp 137430 nova 5261u unix 0xffff881fafe19180 0t0 948261836 /var/run/ceph/guests/ceph-client.cinder.137430.35361600.asok nova-comp 137430 nova 5262r FIFO 0,8 0t0 948265500 pipe nova-comp 137430 nova 5263u unix 0xffff881fafe19500 0t0 948265363 /var/run/ceph/guests/ceph-client.cinder.137430.35318896.asok And: root@qk-3:~# ls -la /var/run/ceph/guests/ceph-client.cinder.137430.33776880.asok ls: cannot access /var/run/ceph/guests/ceph-client.cinder.137430.33776880.asok: No such file or directory As you can realize, all of these files were already closed. If we restart nova-compute process, the number of open file will be 140-150, but within a few days it grows up to 4000-5000-10000 files and we get the ”too many open files" Error again. Below is the ceph.conf from nova-compute node: [global] fsid = 91ef7ba7-463b-4410-8128-82217052d2dd mon_initial_members = ceph-1, ceph-2, ns mon_host = 192.168.2.4,192.168.2.5,192.168.2.6 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true osd_pool_default_size = 3 public_network = 192.168.2.0/24 cluster_network = 192.168.1.0/24 [client] rbd_cache = true rbd_cache_writethrough_until_flush = false rbd_cache_size = 2 GiB rbd_cache_max_dirty = 2 GiB rbd_cache_target_dirty = 256 MiB rbd_cache_max_dirty_age = 1.0 admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok log file = /var/log/qemu/qemu-guest-$pid.log rbd concurrent management ops = 20 Part of nova-compute: disk_cachemodes="network=writeback" inject_password = false inject_key = false inject_partition = -2 hw_disk_discard = unmap root@qk-3:~# dpkg -l | grep nova ii nova-common 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - common files ii nova-compute 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node base ii nova-compute-kvm 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM) ii nova-compute-libvirt 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support ii python-nova 1:2015.1.0-0ubuntu1~cloud0 all OpenStack Compute Python libraries ii python-novaclient 1:2.22.0-0ubuntu1~cloud0 all client library for OpenStack Compute API To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1493783/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp