RE: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support
-Original Message- From: William A. Rowe Jr. [mailto:wmr...@gmail.com] Sent: Donnerstag, 6. März 2014 06:58 To: dev@httpd.apache.org Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support If you want to truly re-architect the MPM, by all means, propose it as another MPM module. If it isn't adopted here, please don't hesitate to offer it to interested users as separate source (although I hope we find a way to adopt it.) The idea of different MPM's was that they were swappable. MPM foo isn't MPM bar. E.g., worker, prefork, event each have their own tree. Likewise, there is nothing stopping us from having 2, or 3 MPM's on Windows, and there is nothing stopping us from stating that there is a prerequisite on a particular MPM of Linux 3.1 kernels or Windows 2008+. +1 to a new MPM on trunk. This gives it more time to settle and to stabilize without disrupting current stuff. And if it is fast and stable it will certainly cause the 'older' MPM to drop in userbase :-). IMHO this would even open a path to 2.4.x provided that we do not need any other non backportable changes to do this. Regards Rüdiger
Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support
On 06 Mar 2014, at 10:15 AM, Plüm, Rüdiger, Vodafone Group ruediger.pl...@vodafone.com wrote: +1 to a new MPM on trunk. This gives it more time to settle and to stabilize without disrupting current stuff. And if it is fast and stable it will certainly cause the 'older' MPM to drop in userbase :-). IMHO this would even open a path to 2.4.x provided that we do not need any other non backportable changes to do this. +1. Regards, Graham --
Proxy Timeout problem with proxy_balancer
Hi, i have actual an problem with my proxy-timeout configuration. (Apache 2.2.26, but 2.4 seems to have the same problem). In my apache configuration we have defined a global ProxyTimeout to 30 seconds. But one of our JBoss-backends have some slow pages, and we want to raise the timeout to 2 minutes. (without change of the default timeout). But the timeout is not used, for the backend. The timeout change for an other backend works fine. Imho, the imported difference is the backend type. balancer doesn't work, http work. Example: ... ProxyTimeout 30 ProxyPass /content balancer://jboss timeout=120 # timeout after 30 secs ProxyPass /mon http://monitor timeout=10# timeout after 10 secs ProxyPass /other http://other # timeout after 30 secs ... In the documentation, the timeout parameter seems to be redefined for balancer. If this correct? Documentation bug? Software bug? My configuration buggy? Know anyone a simple solution? We have many backends, so the workaround to change the default ProxyTimeout and set the timeout per backend is the last way. Greats 'n Thanks, Sascha
Changing rpm spec file to install httpd in a different directory
Hi, I have a situation where I have to install httpd of v 2.X in a centos linux machine where already older version of httpd is running there. For some reasons, I couldn't make use of single upgraded httpd. I have to push this install in production servers via rpm and yum. So I have compiled rpm from source using spec file without much issues in it. Now the problem is I have to change the install path else it will end with conflict files. I know it is not an easy one line change in rpm spec file to do this as we have files in various folders as below, /etc/httpd has conf directory etc... /usr/lib/httpd has modules /usr/sbin/has httpd binary executables /etc/rc.d/init.d/ has init files /var/log/httpd has log files /var/run/httpd.pid As most of the above files are already existing, I have changed the SPEC file so as it will install files in a new directory 'http2' like this, /etc/httpd2/ has conf, logs, modules etc... /usr/lib/httpd2/ has modules /usr/sbin/httpd2/ has binary executables (NOTE I have to add new folder 'httpd2' in this sbin directory for new executables) /etc/rc.d/init.d/httpd2 /var/log/httpd2 has log files /var/run/httpd2.pid I have attached 2 files 1. httpd_original.spec (original spec file for version 2.2.24) 2. httpd_modified.spec (has my changes mentioned above) Though the compiled rpm after this change installed without any conflict issue, I am bit worried about much changes in SPEC file. Also you can see I have to create a directory 'httpd2' under /usr/sbin/ to hold the executables. Another thing here is init scripts still expects binary file path as normal. For example, in /etc/logrotate.d/httpd2 script, it still points to /var/log/httpd/*log and /var/run/httpd.pid. Also in /etc/init/httpd2, it still checks /usr/sbin/httpd and regular pid file and so on.. Is there any another better way of doing this without complicating like above. ? Thanks in advance for your help on this. Prabu httpd_modified.spec Description: Binary data httpd_original.spec Description: Binary data
Re: Proxy Timeout problem with proxy_balancer
On Thu, Mar 6, 2014 at 2:17 AM, InuSasha i...@inusasha.de wrote: i have actual an problem with my proxy-timeout configuration. (Apache 2.2.26, but 2.4 seems to have the same problem). Try us...@httpd.apache.org
Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support
On Wed, Mar 5, 2014 at 6:38 PM, Lu, Yingqi yingqi...@intel.com wrote:
1. If I understand correctly (please correct me if not), do you suggest
duplicating the listen socks inside the child process with SO_REUSEPROT
enabled? Yes, I agree this would be a cleaner implementation and I actually
tried that before. However, I encountered the connection reset error
since the number of the child process is changing. I googled online and
found it actually being discussed here at http://lwn.net/Articles/542629/.
Actually I found that article, but expected the defect was solved since
then...
This looks like a thorn in the side of MPMs in general,
but couldn't find any pointer to a fix, do you know if there is some
progress on this in the latest linux kernel?
For testing purpose (until then?), you could also configure MPM prefork to
not create/terminate children processes once started (using the same value
for StartServers and ServerLimit, still MaxRequetsPerChild 0).
It could be interesting to see how SO_REUSEPORT scales in these optimal
conditions (no lock, full OS round-robin on all listeners).
For this you would have to use your former patch (duplicating listeners in
each child process), and do nothing in SAFE_ACCEPT when HAVE_SO_REUSEPORT.
Also, SO_REUSEPORT exists on (and even comes from) FreeBSD if I am not
mistaken, but it seems that there is no round-robin garantee for it, is
there? Could this patch also take advantage of BSD's SO_REUSEPORT
implementation?
2. Then, I decided to do the socket duplication in the parent process.
The goal of this change is to extend the CPU thread scalability with the
big thread count system. Therefore, I just very simply defined
number_of_listen_buckets=total_number_active_thread/8, and each listen
bucket has a dedicated listener. I do not want to over duplicate the
socket; otherwise, it would create too many child processes at the
beginning. One listen bucket should have at least one child process to
start with. However, this is only my understanding and it may not be
correct and complete. If you have other ideas, please share with us.
Feedbacks and comments are very welcome here J
The listeners buckets make sense with SO_REUSEPORT given the defect, I
hope this is temporary.
3. I am struggling with myself as well on if we should put with and
without SO_REUSEPORT into two different patches. The only reason I put them
together is because they both use the concept of listen buckets. If you
think it would make more sense to separate them into two patches, I can
certainly do that. Also, I am a little bit confused about your comments On
the other hand, each child is dedicated, won't one have to multiply the
configured ServerLimit by the number of Listen to achieve the same (maximum
theoretical) scalability with regard to all the listeners?. Can you please
explain a little bit more on this? Really appreciate.
Sorry to have not been clear enough (nay at all).
I'm referring to the following code.
In prefork.c::make_child(), each child is assigned a listener like this
(before fork()ing) :
child_listen = mpm_listen[bucket[slot]];
and then each child will use child_listen as listeners list.
The duplicated listeners array (mpm_listen) is built by the following (new)
function :
/* This function is added for the patch. This function duplicates
* open_listeners, alloc_listener() and re-call make_sock() for the
* duplicated listeners. In this function, the newly created sockets
* will bind and listen*/
AP_DECLARE(apr_status_t) ap_post_config_listeners(server_rec *s, apr_pool_t
*p,
int num_buckets) {
mpm_listen = apr_palloc(p, sizeof(ap_listen_rec*) * num_buckets);
int i;
ap_listen_rec *lr;
/* duplicate from alloc_listener() for the additional listen record*/
lr = ap_listeners;
for (i = 0; i num_buckets; i++) {
#ifdef HAVE_SO_REUSEPORT
ap_listen_rec *templr;
ap_listen_rec *last = NULL;
while (lr) {
templr = ap_duplicate_listener(p, lr);
ap_apply_accept_filter(p, templr, s);
if (last == NULL) {
mpm_listen[i] = last = templr;
}
else {
last-next = templr;
last = templr;
}
lr = lr-next;
}
lr = ap_listeners;
#else
mpm_listen[i] = ap_duplicate_listener(p, lr);
lr = (lr-next) ? lr-next : ap_listeners;
#endif
}
return APR_SUCCESS;
}
Since ap_duplicate_listener() will duplicate a single (unlinked) listener,
my understanding is that :
- with SO_REUSEPORT: each child will use all the listeners (the whole list
is duplicated per bucket),
- without SO_REUSEPORT: each child will use a single listener (one per
bucket, although muliple children will use the same listener at the same
time should num_buckets num_listeners).
That's what I mean by each child is dedicated (without
Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support
++1. On Mar 6, 2014, at 3:15 AM, Plüm, Rüdiger, Vodafone Group ruediger.pl...@vodafone.com wrote: -Original Message- From: William A. Rowe Jr. [mailto:wmr...@gmail.com] Sent: Donnerstag, 6. März 2014 06:58 To: dev@httpd.apache.org Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support If you want to truly re-architect the MPM, by all means, propose it as another MPM module. If it isn't adopted here, please don't hesitate to offer it to interested users as separate source (although I hope we find a way to adopt it.) The idea of different MPM's was that they were swappable. MPM foo isn't MPM bar. E.g., worker, prefork, event each have their own tree. Likewise, there is nothing stopping us from having 2, or 3 MPM's on Windows, and there is nothing stopping us from stating that there is a prerequisite on a particular MPM of Linux 3.1 kernels or Windows 2008+. +1 to a new MPM on trunk. This gives it more time to settle and to stabilize without disrupting current stuff. And if it is fast and stable it will certainly cause the 'older' MPM to drop in userbase :-). IMHO this would even open a path to 2.4.x provided that we do not need any other non backportable changes to do this. Regards Rüdiger
Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support
On Wed, Mar 5, 2014 at 11:38 AM, Lu, Yingqi yingqi...@intel.com wrote: 1. If I understand correctly (please correct me if not), do you suggest duplicating the listen socks inside the child process with SO_REUSEPROT enabled? Yes, I agree this would be a cleaner implementation and I actually tried that before. However, I encountered the connection reset error since the number of the child process is changing. I googled online and found it actually being discussed here at http://lwn.net/Articles/542629/. You actually need to refer to Unix Network Programming, Vol 1 by W Richard Stevens to have a context to ask the question. Apologies if that comes across as harsh, but either Stevens is wrong, or the kernel developers are wrong. It is unlikely that both are wrong and very likely that both are not right. Sometimes we just have to send kernel bugs upstream.
Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support
I'm not sure (yet) a new MPM is needed, or rather, multiple new MPMs are needed. The bucketized listeners is applyable to all (*nix only?) MPMs, that would lead to as much forks... Couldn't new directives be created instead (ServerBucketsNum, Listen ip:port ratio, ...), defaulting to the current behaviour? On Thu, Mar 6, 2014 at 2:49 PM, Jim Jagielski j...@jagunet.com wrote: ++1. On Mar 6, 2014, at 3:15 AM, Plüm, Rüdiger, Vodafone Group ruediger.pl...@vodafone.com wrote: -Original Message- From: William A. Rowe Jr. [mailto:wmr...@gmail.com] Sent: Donnerstag, 6. März 2014 06:58 To: dev@httpd.apache.org Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support If you want to truly re-architect the MPM, by all means, propose it as another MPM module. If it isn't adopted here, please don't hesitate to offer it to interested users as separate source (although I hope we find a way to adopt it.) The idea of different MPM's was that they were swappable. MPM foo isn't MPM bar. E.g., worker, prefork, event each have their own tree. Likewise, there is nothing stopping us from having 2, or 3 MPM's on Windows, and there is nothing stopping us from stating that there is a prerequisite on a particular MPM of Linux 3.1 kernels or Windows 2008+. +1 to a new MPM on trunk. This gives it more time to settle and to stabilize without disrupting current stuff. And if it is fast and stable it will certainly cause the 'older' MPM to drop in userbase :-). IMHO this would even open a path to 2.4.x provided that we do not need any other non backportable changes to do this. Regards Rüdiger
Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support
Yann, what you might wish to consider is that each individual MPM may be compiled alongside the others. If you do a feature select, you are left with one of the other. If it is designed to cohabitate, then it may share sources under the os/ branch, but can still exist as a separate loadable MPM. Any new MPM idea with a proper ICLA/CCLA/Code Grant, such that our users can make the decision for themselves, I expect will be swiftly tested, benchmarked and perhaps adopted. If it is to exist as a replacement of an MPM in the stable 2.4 tree, I expect it will meet with stiff resistance. Let's work out an appropriate way to adopt the code? Respectfully, Bill On Thu, Mar 6, 2014 at 8:26 AM, Yann Ylavic ylavic@gmail.com wrote: I'm not sure (yet) a new MPM is needed, or rather, multiple new MPMs are needed. The bucketized listeners is applyable to all (*nix only?) MPMs, that would lead to as much forks... Couldn't new directives be created instead (ServerBucketsNum, Listen ip:port ratio, ...), defaulting to the current behaviour? On Thu, Mar 6, 2014 at 2:49 PM, Jim Jagielski j...@jagunet.com wrote: ++1. On Mar 6, 2014, at 3:15 AM, Plüm, Rüdiger, Vodafone Group ruediger.pl...@vodafone.com wrote: -Original Message- From: William A. Rowe Jr. [mailto:wmr...@gmail.com] Sent: Donnerstag, 6. März 2014 06:58 To: dev@httpd.apache.org Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support If you want to truly re-architect the MPM, by all means, propose it as another MPM module. If it isn't adopted here, please don't hesitate to offer it to interested users as separate source (although I hope we find a way to adopt it.) The idea of different MPM's was that they were swappable. MPM foo isn't MPM bar. E.g., worker, prefork, event each have their own tree. Likewise, there is nothing stopping us from having 2, or 3 MPM's on Windows, and there is nothing stopping us from stating that there is a prerequisite on a particular MPM of Linux 3.1 kernels or Windows 2008+. +1 to a new MPM on trunk. This gives it more time to settle and to stabilize without disrupting current stuff. And if it is fast and stable it will certainly cause the 'older' MPM to drop in userbase :-). IMHO this would even open a path to 2.4.x provided that we do not need any other non backportable changes to do this. Regards Rüdiger
Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support
Hi Bill, I was just worried about forking mpm_prefork into mpm_prefork_buckets, and so on with worker/event/..., most of the code would have been the same. But I can't disagree with you, factorizing the existing MPMs shared codes (it seems there are quite some) and future ones into a common interface is indeed the way to go. +1 for as much MPM as needed provided minimal code is duplicated. Best regards, Yann. On Thu, Mar 6, 2014 at 5:49 PM, William A. Rowe Jr. wmr...@gmail.comwrote: Yann, what you might wish to consider is that each individual MPM may be compiled alongside the others. If you do a feature select, you are left with one of the other. If it is designed to cohabitate, then it may share sources under the os/ branch, but can still exist as a separate loadable MPM. Any new MPM idea with a proper ICLA/CCLA/Code Grant, such that our users can make the decision for themselves, I expect will be swiftly tested, benchmarked and perhaps adopted. If it is to exist as a replacement of an MPM in the stable 2.4 tree, I expect it will meet with stiff resistance. Let's work out an appropriate way to adopt the code? Respectfully, Bill On Thu, Mar 6, 2014 at 8:26 AM, Yann Ylavic ylavic@gmail.com wrote: I'm not sure (yet) a new MPM is needed, or rather, multiple new MPMs are needed. The bucketized listeners is applyable to all (*nix only?) MPMs, that would lead to as much forks... Couldn't new directives be created instead (ServerBucketsNum, Listen ip:port ratio, ...), defaulting to the current behaviour? On Thu, Mar 6, 2014 at 2:49 PM, Jim Jagielski j...@jagunet.com wrote: ++1. On Mar 6, 2014, at 3:15 AM, Plüm, Rüdiger, Vodafone Group ruediger.pl...@vodafone.com wrote: -Original Message- From: William A. Rowe Jr. [mailto:wmr...@gmail.com] Sent: Donnerstag, 6. März 2014 06:58 To: dev@httpd.apache.org Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support If you want to truly re-architect the MPM, by all means, propose it as another MPM module. If it isn't adopted here, please don't hesitate to offer it to interested users as separate source (although I hope we find a way to adopt it.) The idea of different MPM's was that they were swappable. MPM foo isn't MPM bar. E.g., worker, prefork, event each have their own tree. Likewise, there is nothing stopping us from having 2, or 3 MPM's on Windows, and there is nothing stopping us from stating that there is a prerequisite on a particular MPM of Linux 3.1 kernels or Windows 2008+. +1 to a new MPM on trunk. This gives it more time to settle and to stabilize without disrupting current stuff. And if it is fast and stable it will certainly cause the 'older' MPM to drop in userbase :-). IMHO this would even open a path to 2.4.x provided that we do not need any other non backportable changes to do this. Regards Rüdiger
RE: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT support
Hi Yann,
Yes, without SO_REUSEPORT, child only accepts connections from a single
listening socket only. In order to address the situation of in-balanced traffic
among different sockets/listen statements, the patch makes each bucket does its
own idler server maintenance. For example, if we have two listen statements
defined, one is very busy and the other is almost idle. The patch creates two
buckets, each listens to 1 IP:port. The busy bucket would end up with lots of
children and idle bucket would only maintain minimum number of children which
is equal to 1/2 of the min idle servers defined in the httpd.conf file.
Thanks,
Yingqi
From: Yann Ylavic [mailto:ylavic@gmail.com]
Sent: Thursday, March 06, 2014 5:49 AM
To: httpd
Subject: Re: [PATCH ASF bugzilla# 55897]prefork_mpm patch with SO_REUSEPORT
support
On Wed, Mar 5, 2014 at 6:38 PM, Lu, Yingqi
yingqi...@intel.commailto:yingqi...@intel.com wrote:
1. If I understand correctly (please correct me if not), do you suggest
duplicating the listen socks inside the child process with SO_REUSEPROT
enabled? Yes, I agree this would be a cleaner implementation and I actually
tried that before. However, I encountered the connection reset error since
the number of the child process is changing. I googled online and found it
actually being discussed here at http://lwn.net/Articles/542629/.
Actually I found that article, but expected the defect was solved since
then...
This looks like a thorn in the side of MPMs in general,
but couldn't find any pointer to a fix, do you know if there is some progress
on this in the latest linux kernel?
For testing purpose (until then?), you could also configure MPM prefork to not
create/terminate children processes once started (using the same value for
StartServers and ServerLimit, still MaxRequetsPerChild 0).
It could be interesting to see how SO_REUSEPORT scales in these optimal
conditions (no lock, full OS round-robin on all listeners).
For this you would have to use your former patch (duplicating listeners in each
child process), and do nothing in SAFE_ACCEPT when HAVE_SO_REUSEPORT.
Also, SO_REUSEPORT exists on (and even comes from) FreeBSD if I am not
mistaken, but it seems that there is no round-robin garantee for it, is there?
Could this patch also take advantage of BSD's SO_REUSEPORT implementation?
2. Then, I decided to do the socket duplication in the parent process. The goal
of this change is to extend the CPU thread scalability with the big thread
count system. Therefore, I just very simply defined
number_of_listen_buckets=total_number_active_thread/8, and each listen bucket
has a dedicated listener. I do not want to over duplicate the socket;
otherwise, it would create too many child processes at the beginning. One
listen bucket should have at least one child process to start with. However,
this is only my understanding and it may not be correct and complete. If you
have other ideas, please share with us. Feedbacks and comments are very welcome
here :)
The listeners buckets make sense with SO_REUSEPORT given the defect, I hope
this is temporary.
3. I am struggling with myself as well on if we should put with and without
SO_REUSEPORT into two different patches. The only reason I put them together is
because they both use the concept of listen buckets. If you think it would make
more sense to separate them into two patches, I can certainly do that. Also, I
am a little bit confused about your comments On the other hand, each child is
dedicated, won't one have to multiply the configured ServerLimit by the number
of Listen to achieve the same (maximum theoretical) scalability with regard to
all the listeners?. Can you please explain a little bit more on this? Really
appreciate.
Sorry to have not been clear enough (nay at all).
I'm referring to the following code.
In prefork.c::make_child(), each child is assigned a listener like this (before
fork()ing) :
child_listen = mpm_listen[bucket[slot]];
and then each child will use child_listen as listeners list.
The duplicated listeners array (mpm_listen) is built by the following (new)
function :
/* This function is added for the patch. This function duplicates
* open_listeners, alloc_listener() and re-call make_sock() for the
* duplicated listeners. In this function, the newly created sockets
* will bind and listen*/
AP_DECLARE(apr_status_t) ap_post_config_listeners(server_rec *s, apr_pool_t *p,
int num_buckets) {
mpm_listen = apr_palloc(p, sizeof(ap_listen_rec*) * num_buckets);
int i;
ap_listen_rec *lr;
/* duplicate from alloc_listener() for the additional listen record*/
lr = ap_listeners;
for (i = 0; i num_buckets; i++) {
#ifdef HAVE_SO_REUSEPORT
ap_listen_rec *templr;
ap_listen_rec *last = NULL;
while (lr) {
templr = ap_duplicate_listener(p, lr);
ap_apply_accept_filter(p, templr, s);
