[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.] ** Changed in: libvirt (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
FYI - my upstream change to raise the limits got accepted. But that doesn't help in your case were you were lower than the limit set by the service. So no change to this bug. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
Thanks Chris, until then I'll continue bringing slightly increased limits upstream based on what we learned here. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
Christian, The customer that I was working with has changed their environment, so I cannot go back to them. I will have to deploy openstack in our test lab to verify the default settings. It will take some time for me to get this checked out. I will reply back when I get it running. Thanks. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
Submitted upstream to increase the limit in regard to what we found (4 per guest). Also we had similar discussions for disks on s390x, so doing those in one shot. Reference: https://www.redhat.com/archives/libvir- list/2017-October/msg00735.html Further on I'm waiting for some data/insight on why the original system has a lower limit than what the service configures. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
That is very interesting Chris. As it should not be on 1024/4096 at all. The limit was not set back in libvirt 2.5 and before, so older versions had a higher limit - that much is true. I checked and it has Artful: Max open files8192 8192 files Pre-Artful: Max open files1048576 1048576 files But both limits are way above yours. Your case has ~4 files per guest, but with the right limit that should be enough to close to 2k guests. I'd suggest two things: 1. I'll discuss upstream to raise the 8192 to 16384 as it seems Openstack (as a common setup) has 4 files per guest. I'll do that and link the Mailing List entry here 2. We need to find why you have a limit applied even lower than what the service file configures. @Chris - can you have a look on that system why that could be so low - any extra limits applied somewhere? Does the virlogd service file have the 8192 by default - if it has 1024/4096 what changed it? If it has 8192, can you see on the system where the 1024/4096 might come from? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
This is the data that led me to believe the default setting was low. I gathered this from a customer: ubuntu@x:~$ virsh list | wc -l 212 ubuntu@x:~$ sudo lsof -p $(pgrep virtlogd) | wc -l 893 ubuntu@x:~$ sudo grep "open files" /proc/$(pgrep virtlogd)/limits Max open files 1024 4096 files This is with 212 VMs and when the lsof output gets over 1024, it throws errors. Thanks, Chris -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
At 1022 I exceed the limit of my basic bridge, but that is fine I guess. At this point I use ~4000 files and things are working. virsh list | tail; sudo ls -laF /proc/2905/fd | wc -l; 1035 x-test1014 running 1036 x-test1015 running 1037 x-test1016 running 1038 x-test1017 running 1039 x-test1018 running 1040 x-test1019 running 1041 x-test1020 running 1042 x-test1021 running 1043 x-test1022 running 4102 So I'm really looking forward to see the details of your case and why it causes this (in regard to the virtlogd limit). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
Yeah I can reach the thousands (which already is kind of dense considering that a real case will not drive the nonsense 64MB guests I use for testing). So what do we need now - TL;DR: 1. eventually we want to get the change accepted upstream 2. to get those we need some good data why 8192 is not sufficient 3. We need to understand how you exceed these numbers with just ~250 guests @Chris - Please: 1. check the FDs you virtlogd has attached (get its pid and check /proc//fd 2. report how many get added per guest 3. report which ones get added per guest 4. share the generated guest XML (to hopefully show us what is the reason for so many logs) ** Changed in: libvirt (Ubuntu) Status: Triaged => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
IIRC Openstack adds two in most cases, like: Twice the same path btw, but also same alias. I checked and that still means only 2 files per guest. The default guest log like: /var/log/libvirt/qemu/x-test1.log And the file redirected console like: /tmp/consoleIDX2.log So under these conditions the assumption of ~4096 should work. In detail I found that under this config there also are 2 pipes along the two logs. So a single guest looks like: l-wx-- 1 root root 64 Sep 25 20:48 22 -> /var/log/libvirt/qemu/x-test3.log lr-x-- 1 root root 64 Sep 25 20:48 23 -> pipe:[5014140] l-wx-- 1 root root 64 Sep 25 20:48 24 -> /tmp/consoleIDX3.log lr-x-- 1 root root 64 Sep 25 20:48 25 -> pipe:[5014157] With 5 guests I had 20 new FDs on the virtlogd service. That said the limit should still be enough for 1024 guests. Testing against those numbers now ... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
We have at least to think if infinity is the right value before changing it (and if we change we want to push it upstream, so consideration thought are useful in any case). Obviously it removes it being a blocker for many guests, but OTOH it is meant to avoid overloads of the system. The current limit (default) is 8192 (read from /proc//limits of the process. It starts with a base of about 15 and adds 1 per guest. That would make me expect being a limit short of 8k guests which for a single host is high enough that I'd agree to an admin having to opt in changing a conffile. But you reported that your cases break around 250 which should be working IMHO. The comments in the upstream .service file already kind of support my theory that the number should be sufficient. # Need to have at least one file open per guest (eg QEMU # stdio log), but might be more (eg serial console logs) # libvirtd.service written to expect 4096 guests, so if we # guess at 2 log files per guest here (stdio + 1 serial): LimitNOFILE=8192 But that also means that if you have many serials by default, then you'll exceed it much faster. Never the less to break around 250 you'd need 32 serials per guest which seems a bit too much right? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
Some of you knew that I was on PTO a few days, but for everyone sorry for the delay. Yet OTOH this wasn't rocket science enough for anybody else to pick up in the meantime. I started on it today, so expect updates soon ... Thanks Chris already for the direct pointer to the workaround/fix in your case. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
@freyes Yes definitely! ** Changed in: charm-nova-compute Status: New => Invalid ** Changed in: libvirt (Ubuntu) Status: New => Triaged ** Changed in: libvirt (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1720887] Re: Default settings for virtlogd results in "too many open files" errors
I wonder if this is something we should fix in the libvirt package. ** Tags added: sts ** Also affects: libvirt (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1720887 Title: Default settings for virtlogd results in "too many open files" errors To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1720887/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
