Re: open files limit was: Re: Enabling Apache HTTP/2 => Too many open files error
On Mar 2, 2023, at 03:35, Roberto Ragusa wrote: > > On 2/27/23 13:44, Richard W.M. Jones wrote: > >> It turns out that the default open file limit (1024!) is too low. To >> change this and fix the problem: >> # systemctl edit httpd > This low limit can be an issue for many processes. > I don't understand why it is still so low in modern machines. > I've raised it a lot, many years ago, by editing > /etc/security/limits.conf, but I now see I get again > 1024. My guess is systemd is deciding by itself, and not > respecting my settings. /etc /security/limits.conf is only read by pam_limits.so in the PAM stack. Systemd processes don’t automatically use pam (although if you use ‘su’ or login somehow). You’d change it with the LimitsNOfile systemd directive. I think there’s a default setting in /etc/systemd/system.conf you can override the default. (Double-check man systemd.resource-control) -- Jonathan Billings ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: open files limit was: Re: Enabling Apache HTTP/2 => Too many open files error
On Thu, Mar 2, 2023 at 2:35 AM Roberto Ragusa wrote: > > On 2/27/23 13:44, Richard W.M. Jones wrote: > > > It turns out that the default open file limit (1024!) is too low. To > > change this and fix the problem: > > > ># systemctl edit httpd > This low limit can be an issue for many processes. > I don't understand why it is still so low in modern machines. > I've raised it a lot, many years ago, by editing > /etc/security/limits.conf, but I now see I get again > 1024. My guess is systemd is deciding by itself, and not > respecting my settings. > > Regards. > Systemd is started really early and seems to use the defaults. Anything started from systemd needs to explicitly override in the startup script. Looking at the initramfs I don't see limits.conf in there so when the switchroot happens and systemd gets started it would have the default limits. ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
open files limit was: Re: Enabling Apache HTTP/2 => Too many open files error
On 2/27/23 13:44, Richard W.M. Jones wrote: It turns out that the default open file limit (1024!) is too low. To change this and fix the problem: # systemctl edit httpd This low limit can be an issue for many processes. I don't understand why it is still so low in modern machines. I've raised it a lot, many years ago, by editing /etc/security/limits.conf, but I now see I get again 1024. My guess is systemd is deciding by itself, and not respecting my settings. Regards. -- Roberto Ragusamail at robertoragusa.it ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Enabling Apache HTTP/2 => Too many open files error
On Mon, Feb 27, 2023 at 11:42:49AM -0500, Jeffrey Walton wrote: > Hi Rich, > > > [although it's way more > > complicated than it needs to be, why isn't HTTP/2 the default out of > > the box?] > > HTTP/2 is insecure out-of-the-box. Remember CRIME and BREACH? The > protocol requires compression, and compression is a known attack > vector. From the abstract of RFC 7450: > >This specification describes an optimized expression of the semantics >of the Hypertext Transfer Protocol (HTTP), referred to as HTTP >version 2 (HTTP/2). HTTP/2 enables a more efficient use of network >resources and a reduced perception of latency by introducing header >field compression and allowing multiple concurrent exchanges on the >same connection. It also introduces unsolicited push of >representations from servers to clients. > > I am also not sure the push functionality is well understood in a > security context. > > So it is probably a good idea to make HTTP/2 optional, until an > organization has an opportunity to weigh the risks versus reward. Good points, thanks. Rich. > Jeff > > On Mon, Feb 27, 2023 at 7:44 AM Richard W.M. Jones wrote: > > > > I fixed this now, but I could find virtually no documentation about it > > online, so I'm writing this email to document what surely must be a > > common problem ... > > > > I wanted to enable HTTP/2 support in Apache on Fedora 38. > > > > I followed the documentation here which worked [although it's way more > > complicated than it needs to be, why isn't HTTP/2 the default out of > > the box?] > > > > https://httpd.apache.org/docs/2.4/howto/http2.html > > > > Anyway the problem I had was that the server worked fine provided > > there were not too many clients (and by "too many" I mean a simple > > load test with 4-16 clients failed). Apache randomly threw 403 > > Forbidden errors, but with less load it gave a normal (2xx) response. > > > > The first problem is the error is misleading: > > > > [Wed Feb 22 13:24:52.013780 2023] [core:error] [pid 3047850:tid 3047899] > > (24)Too many open files: [remote 192.168.0.139:53738] AH00132: file > > permissions deny server access: /var/www/html/[filename] > > > > If you concentrate on the second part "file permissions deny server > > access" -- as I did -- then you'll be looking at file permissions, > > SELinux, restorecon, ausearch etc. That's a red herring, there is no > > permissions problem. > > > > The real error is the first part "Too many open files". > > > > It turns out that the default open file limit (1024!) is too low. To > > change this and fix the problem: > > > > # systemctl edit httpd > > > > This creates an "override" file to which you should add (or you could > > just create this file directly): > > > > # cat /etc/systemd/system/httpd.service.d/override.conf > > [Service] > > LimitNOFILE=65536 > > > > and then restart Apache for the change to take effect. > > > > Why on earth Apache needs > 1024 open files to serve a dozen clients > > is not clear at all. > > > ___ > users mailing list -- users@lists.fedoraproject.org > To unsubscribe send an email to users-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Enabling Apache HTTP/2 => Too many open files error
Hi Rich, > [although it's way more > complicated than it needs to be, why isn't HTTP/2 the default out of > the box?] HTTP/2 is insecure out-of-the-box. Remember CRIME and BREACH? The protocol requires compression, and compression is a known attack vector. From the abstract of RFC 7450: This specification describes an optimized expression of the semantics of the Hypertext Transfer Protocol (HTTP), referred to as HTTP version 2 (HTTP/2). HTTP/2 enables a more efficient use of network resources and a reduced perception of latency by introducing header field compression and allowing multiple concurrent exchanges on the same connection. It also introduces unsolicited push of representations from servers to clients. I am also not sure the push functionality is well understood in a security context. So it is probably a good idea to make HTTP/2 optional, until an organization has an opportunity to weigh the risks versus reward. Jeff On Mon, Feb 27, 2023 at 7:44 AM Richard W.M. Jones wrote: > > I fixed this now, but I could find virtually no documentation about it > online, so I'm writing this email to document what surely must be a > common problem ... > > I wanted to enable HTTP/2 support in Apache on Fedora 38. > > I followed the documentation here which worked [although it's way more > complicated than it needs to be, why isn't HTTP/2 the default out of > the box?] > > https://httpd.apache.org/docs/2.4/howto/http2.html > > Anyway the problem I had was that the server worked fine provided > there were not too many clients (and by "too many" I mean a simple > load test with 4-16 clients failed). Apache randomly threw 403 > Forbidden errors, but with less load it gave a normal (2xx) response. > > The first problem is the error is misleading: > > [Wed Feb 22 13:24:52.013780 2023] [core:error] [pid 3047850:tid 3047899] > (24)Too many open files: [remote 192.168.0.139:53738] AH00132: file > permissions deny server access: /var/www/html/[filename] > > If you concentrate on the second part "file permissions deny server > access" -- as I did -- then you'll be looking at file permissions, > SELinux, restorecon, ausearch etc. That's a red herring, there is no > permissions problem. > > The real error is the first part "Too many open files". > > It turns out that the default open file limit (1024!) is too low. To > change this and fix the problem: > > # systemctl edit httpd > > This creates an "override" file to which you should add (or you could > just create this file directly): > > # cat /etc/systemd/system/httpd.service.d/override.conf > [Service] > LimitNOFILE=65536 > > and then restart Apache for the change to take effect. > > Why on earth Apache needs > 1024 open files to serve a dozen clients > is not clear at all. > ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Enabling Apache HTTP/2 => Too many open files error
Do a lsof -p and it will list out the files. It may be a file handle leak. I have seen leaks from failure to close a file when a process is done. There can be leaks if a process uses an anonymous memory allocation trick that relies on file handles, and there are probably others. The lsof output will give you an idea if it is a leak (usually it will be the same files being opened again and again without closing). If it is a leak and the rate is high enough you may not be able to set the open files high enough for it to work for long. On Mon, Feb 27, 2023 at 6:45 AM Richard W.M. Jones wrote: > > I fixed this now, but I could find virtually no documentation about it > online, so I'm writing this email to document what surely must be a > common problem ... > > I wanted to enable HTTP/2 support in Apache on Fedora 38. > > I followed the documentation here which worked [although it's way more > complicated than it needs to be, why isn't HTTP/2 the default out of > the box?] > > https://httpd.apache.org/docs/2.4/howto/http2.html > > Anyway the problem I had was that the server worked fine provided > there were not too many clients (and by "too many" I mean a simple > load test with 4-16 clients failed). Apache randomly threw 403 > Forbidden errors, but with less load it gave a normal (2xx) response. > > The first problem is the error is misleading: > > [Wed Feb 22 13:24:52.013780 2023] [core:error] [pid 3047850:tid 3047899] > (24)Too many open files: [remote 192.168.0.139:53738] AH00132: file > permissions deny server access: /var/www/html/[filename] > > If you concentrate on the second part "file permissions deny server > access" -- as I did -- then you'll be looking at file permissions, > SELinux, restorecon, ausearch etc. That's a red herring, there is no > permissions problem. > > The real error is the first part "Too many open files". > > It turns out that the default open file limit (1024!) is too low. To > change this and fix the problem: > > # systemctl edit httpd > > This creates an "override" file to which you should add (or you could > just create this file directly): > > # cat /etc/systemd/system/httpd.service.d/override.conf > [Service] > LimitNOFILE=65536 > > and then restart Apache for the change to take effect. > > Why on earth Apache needs > 1024 open files to serve a dozen clients > is not clear at all. > > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > virt-top is 'top' for virtual machines. Tiny program with many > powerful monitoring features, net stats, disk stats, logging, etc. > http://people.redhat.com/~rjones/virt-top > ___ > users mailing list -- users@lists.fedoraproject.org > To unsubscribe send an email to users-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Enabling Apache HTTP/2 => Too many open files error
I fixed this now, but I could find virtually no documentation about it online, so I'm writing this email to document what surely must be a common problem ... I wanted to enable HTTP/2 support in Apache on Fedora 38. I followed the documentation here which worked [although it's way more complicated than it needs to be, why isn't HTTP/2 the default out of the box?] https://httpd.apache.org/docs/2.4/howto/http2.html Anyway the problem I had was that the server worked fine provided there were not too many clients (and by "too many" I mean a simple load test with 4-16 clients failed). Apache randomly threw 403 Forbidden errors, but with less load it gave a normal (2xx) response. The first problem is the error is misleading: [Wed Feb 22 13:24:52.013780 2023] [core:error] [pid 3047850:tid 3047899] (24)Too many open files: [remote 192.168.0.139:53738] AH00132: file permissions deny server access: /var/www/html/[filename] If you concentrate on the second part "file permissions deny server access" -- as I did -- then you'll be looking at file permissions, SELinux, restorecon, ausearch etc. That's a red herring, there is no permissions problem. The real error is the first part "Too many open files". It turns out that the default open file limit (1024!) is too low. To change this and fix the problem: # systemctl edit httpd This creates an "override" file to which you should add (or you could just create this file directly): # cat /etc/systemd/system/httpd.service.d/override.conf [Service] LimitNOFILE=65536 and then restart Apache for the change to take effect. Why on earth Apache needs > 1024 open files to serve a dozen clients is not clear at all. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
