Re: mod_remoteip and mod_http2 combined

[Sander Hoentjen]

On 03/11/2017 07:57 PM, Daniel Ruggeri wrote:
> Thanks, all, for the patience as I finally got back to this.
>
> On 2/24/2017 11:05 AM, Sander Hoentjen wrote:
>> On 02/20/2017 07:48 PM, William A Rowe Jr wrote:
>>> On Sat, Feb 18, 2017 at 4:25 PM, Daniel Ruggeri  wrote:
 On 2017-02-15 09:07 (-0600), William A Rowe Jr  wrote:
> On Wed, Feb 15, 2017 at 9:02 AM, Sander Hoentjen  
> wrote:
>> mod_remote ip has:
>> /* mod_proxy creates outgoing connections - we don't want those */
>> if (!remoteip_is_server_port(c->local_addr->port)) {
>> return DECLINED;
>> }
>> I am guessing something similar is needed for h2 connections?
> I suspect that the mod_remoteip logic is wrong, that it should be guarding
> against any subordinate connections and examining only explicitly 
> configured
> ports / origin IPs. the PROXY protocol is not part of the HTTP protocol 
> and
> incompatible with it, so the trust list logic isn't directly compatible 
> (this is
> clearly explained in the PROXY pseudo-RFC.)
>
 Hi, Bill. That is what the module is doing. The original authors wrote it 
 to have a list of virtual hosts it is explicitly enabled for and 
 explicitly disabled for. I added a third list for optional vhosts. In the 
 pre_connection hook, it checks to see if the connection's local_addr 
 (which should normally be the server's IP) is explicitly configured to 
 enable PROXY handling. It then checks to see if the local port is a server 
 port.

 Looking at the logs shared, 192.168.122.249:84 is the server IP:Port combo 
 and is also the local IP:Port from mod_h2. If h2 sets the master of this 
 connection, then we could skip the whole ordeal with this patch:

 Index: modules/metadata/mod_remoteip.c
 ===
 --- modules/metadata/mod_remoteip.c (revision 1781701)
 +++ modules/metadata/mod_remoteip.c (working copy)
 @@ -862,6 +862,10 @@
  remoteip_conn_config_t *conn_conf;
  int optional;

 +if (c->master != NULL) {
 +return DECLINED;
 +}
 +
  conf = ap_get_module_config(ap_server_conf->module_config,
  &remoteip_module);

 .. but I don't know if that potentially means we are looking at the wrong 
 connection.
>> First I'll say that with the "Optional" mode it worked, just not with On
>> I just tried this patch and as far as I have tested this seems to work
>> fine in On mode, as well as in Optional. I do see some other issue, but
>> that is probably in my own code, I'll try to track that down later.
> This is good news and about what I was expecting to happen. I will add
> this to the commit I've got coming that incorporates a lot of Ruediger's
> feedback.
>
>>> That should be close, but need to ensure c->master is initialized for
>>> http as well
>>> where there is no master/subordinate.
>> I am not sure what this means, how should I test this?
> Hi, Bill - also hoping for a bit more input. Since PROXY protocol is not
> tied to any particular layer 7 protocol, I don't think we'd have to
> verify it is initialized for HTTP - just that there is no master at all.
> At least, that's my understanding so I appreciate any corrections.
Here are my changes by the way:
https://github.com/AntagonistHQ/httpd/commit/2d208793b4494e73289477c231c79be9e0030a2b
> Sure, to clarify, the Optional use case came from a member on one of our
> cousin projects (Tomcat) Chris Schultz as well as my own use cases. It
> is useful for internally accessing the site from behind the
> loadbalancer. When there is a publicly addressed upstream loadbalancer
> (Amazon ELB or just HAProxy itself) talking to RFC1918 addressed or
> non-routeable backend httpd servers, it becomes impossible to enable
> internal communication on the RFC1918 space to the backend instances.
> If the goal is to monitor or probe the site and (httpd proxy) backends
> internally for health, this *can* be done by duplicating the virtual
> hosts. Depending on the complexity of the virtual hosts, what resources
> those virtual hosts have (proxies and whatnot) and their general size,
> this could result in a fairly unmanageable httpd configuration having a
> vhost that requires PROXY header and a second one on a different port or
> IP that does not.
> It gets even more complicated when you are aiming to do management
> tasks. If you have balancers configured at the vhost as intended, you
> can only manage those balancers from the vhost they live in. Further,
> you may want to view server statistics, check info about the ldap cache,
> etc but permit access to those things only from a trusted network in
> addition to the user credentials protecting it.
> So for those examples, aside from creating an internal HAProxy to
> provide the header, it's not really possible

Apache 2.4.26 AH: scoreboard is full, not at MaxRequestWorkers.Increase,ServerLimit

[Stefan Priebe - Profihost AG]

Hi Yann,

no sure whether this is due to your mpm event patch.

>From time to time i see the following error mesages in my logs and the
only chance to fix it is to restart apache.

[Thu Mar 16 01:00:35.445184 2017] [mpm_event:error] [pid 27485:tid
140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
ServerLimit.
[Thu Mar 16 01:00:36.446178 2017] [mpm_event:error] [pid 27485:tid
140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
ServerLimit.
[Thu Mar 16 01:00:37.447181 2017] [mpm_event:error] [pid 27485:tid
140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
ServerLimit.
[Thu Mar 16 01:00:38.448177 2017] [mpm_event:error] [pid 27485:tid
140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
ServerLimit.
[Thu Mar 16 01:00:39.449185 2017] [mpm_event:error] [pid 27485:tid
140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
ServerLimit.
[Thu Mar 16 01:00:40.450184 2017] [mpm_event:error] [pid 27485:tid
140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
ServerLimit.

Settings:
ServerLimit   50
ThreadLimit   200
ThreadsPerChild200
MinSpareThreads200
MaxSpareThreads400
MaxClients 1
MaxRequestsPerChild1

MaxRequestWorkers isn't set at all.

Greets,
Stefan

Re: Apache 2.4.26 AH: scoreboard is full, not at MaxRequestWorkers.Increase,ServerLimit

[Daniel]

Hello,

imho maxsparethreads is too low as well as maxrequestsperchild, which will
make your server reclaim threads too often, this often causes the
scoreboard full warning (although it is not an error, just a little
degradation from my experience testing).

Can you try to adjust maxsparethreads to something above the average load
you usually have? Say you have a typical use of 1000 threads, rise it to
1500 or 2000, or even 5000.

Also if you can set maxrequestsperchild to a very high number  (10 million)
or to 0, unless you suspect a mem leak.

2017-03-16 12:14 GMT+01:00 Stefan Priebe - Profihost AG <
s.pri...@profihost.ag>:

> Hi Yann,
>
> no sure whether this is due to your mpm event patch.
>
> From time to time i see the following error mesages in my logs and the
> only chance to fix it is to restart apache.
>
> [Thu Mar 16 01:00:35.445184 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:36.446178 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:37.447181 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:38.448177 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:39.449185 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:40.450184 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
> ServerLimit.
>
> Settings:
> ServerLimit   50
> ThreadLimit   200
> ThreadsPerChild200
> MinSpareThreads200
> MaxSpareThreads400
> MaxClients 1
> MaxRequestsPerChild1
>
> MaxRequestWorkers isn't set at all.
>
> Greets,
> Stefan
>



-- 
*Daniel Ferradal*
IT Specialist

email dferradal at gmail.com
linkedin es.linkedin.com/in/danielferradal

Re: Apache 2.4.26 AH: scoreboard is full, not at MaxRequestWorkers.Increase,ServerLimit

[Luca Toscano]

Hi Stefan,

2017-03-16 12:14 GMT+01:00 Stefan Priebe - Profihost AG <
s.pri...@profihost.ag>:

> Hi Yann,
>
> no sure whether this is due to your mpm event patch.
>
> From time to time i see the following error mesages in my logs and the
> only chance to fix it is to restart apache.
>
> [Thu Mar 16 01:00:35.445184 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:36.446178 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:37.447181 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:38.448177 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:39.449185 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:40.450184 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at MaxRequestWorkers.Increase
> ServerLimit.
>
> Settings:
> ServerLimit   50
> ThreadLimit   200
> ThreadsPerChild200
> MinSpareThreads200
> MaxSpareThreads400
> MaxClients 1
> MaxRequestsPerChild1
>
> MaxRequestWorkers isn't set at all.
>

I believe that MaxClients (its old name) takes the place of
MaxRequestWorkers, but it is set as the default value (ServerLimit *
ThreadsPerChild). From 2.4.25 onwards mpm-event offers a new functionality
to handle Gracefully terminating processes (
https://httpd.apache.org/docs/current/mod/event.html#how-it-works), but it
must be tuned raising the value of ServerLimit (more details in the docs).

This is only a speculation from my side, to have a better idea of what's
happening it would be great to see how the Scoreboard looks like in server
status, and if the error status happens during specific events like
graceful reload for log rotation.

Hope that helps! As always, thanks a lot for your tests :)

Luca

Re: Apache 2.4.26 AH: scoreboard is full, not at MaxRequestWorkers.Increase,ServerLimit

[Stefan Priebe - Profihost AG]

Hi,

Am 16.03.2017 um 12:26 schrieb Luca Toscano:
> Hi Stefan,
> 
> 2017-03-16 12:14 GMT+01:00 Stefan Priebe - Profihost AG
> mailto:s.pri...@profihost.ag>>:
> 
> Hi Yann,
> 
> no sure whether this is due to your mpm event patch.
> 
> From time to time i see the following error mesages in my logs and the
> only chance to fix it is to restart apache.
> 
> [Thu Mar 16 01:00:35.445184 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at
> MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:36.446178 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at
> MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:37.447181 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at
> MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:38.448177 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at
> MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:39.449185 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at
> MaxRequestWorkers.Increase
> ServerLimit.
> [Thu Mar 16 01:00:40.450184 2017] [mpm_event:error] [pid 27485:tid
> 140212799559552] AH: scoreboard is full, not at
> MaxRequestWorkers.Increase
> ServerLimit.
> 
> Settings:
> ServerLimit   50
> ThreadLimit   200
> ThreadsPerChild200
> MinSpareThreads200
> MaxSpareThreads400
> MaxClients 1
> MaxRequestsPerChild1
> 
> MaxRequestWorkers isn't set at all.
> 
> 
> I believe that MaxClients (its old name) takes the place of
> MaxRequestWorkers, but it is set as the default value (ServerLimit *
> ThreadsPerChild). From 2.4.25 onwards mpm-event offers a new
> functionality to handle Gracefully terminating processes
> (https://httpd.apache.org/docs/current/mod/event.html#how-it-works), but
> it must be tuned raising the value of ServerLimit (more details in the
> docs).

Hui didn't know this. So i should remove ALL settings and only set:
ThreadsPerChild
ServerLimit
MaxRequestWorkers
AsyncRequestWorkerFactor

? is this true? I'm missing some examples also considering MinSpare and
ThreadLimit or are they no longer needed?

> This is only a speculation from my side, to have a better idea of what's
> happening it would be great to see how the Scoreboard looks like in
> server status, and if the error status happens during specific events
> like graceful reload for log rotation. 

All of them happened after a reload - but I'll recheck. I'm pretty sure
that the /server-status page was no longer responding. Is there any
other way to get the status of of httpd while it does no longer serve pages?

> Hope that helps! As always, thanks a lot for your tests :)

Greets,
Stefan

> Luca 

Re: Apache 2.4.26 AH: scoreboard is full, not at MaxRequestWorkers.Increase,ServerLimit

[Luca Toscano]

2017-03-16 15:24 GMT+01:00 Stefan Priebe - Profihost AG <
s.pri...@profihost.ag>:

> Hi,
>
> Am 16.03.2017 um 12:26 schrieb Luca Toscano:
> > Hi Stefan,
> >
> > 2017-03-16 12:14 GMT+01:00 Stefan Priebe - Profihost AG
> > mailto:s.pri...@profihost.ag>>:
> >
> > Hi Yann,
> >
> > no sure whether this is due to your mpm event patch.
> >
> > From time to time i see the following error mesages in my logs and
> the
> > only chance to fix it is to restart apache.
> >
> > [Thu Mar 16 01:00:35.445184 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:36.446178 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:37.447181 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:38.448177 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:39.449185 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:40.450184 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> >
> > Settings:
> > ServerLimit   50
> > ThreadLimit   200
> > ThreadsPerChild200
> > MinSpareThreads200
> > MaxSpareThreads400
> > MaxClients 1
> > MaxRequestsPerChild1
> >
> > MaxRequestWorkers isn't set at all.
> >
> >
> > I believe that MaxClients (its old name) takes the place of
> > MaxRequestWorkers, but it is set as the default value (ServerLimit *
> > ThreadsPerChild). From 2.4.25 onwards mpm-event offers a new
> > functionality to handle Gracefully terminating processes
> > (https://httpd.apache.org/docs/current/mod/event.html#how-it-works), but
> > it must be tuned raising the value of ServerLimit (more details in the
> > docs).
>
> Hui didn't know this. So i should remove ALL settings and only set:
> ThreadsPerChild
> ServerLimit
> MaxRequestWorkers
> AsyncRequestWorkerFactor
>
> ? is this true? I'm missing some examples also considering MinSpare and
> ThreadLimit or are they no longer needed?
>

I would simply replace MaxClients with MaxRequestWorkers keeping the rest
of your config (that it is still valid, all the settings that you mentioned
are still used by mpm-event), without touching  AsyncRequestWorkerFactor
(unless you want to play with it but the default is generally good). About
your specific max scoreboard issue, I would:

1) try to raise ServerLimit to allow more space for slots occupied by
processes in G state (graceful termination), as indicated in the docs.
2) Follow Daniel's suggestion about Max/Min spare workers.



> > This is only a speculation from my side, to have a better idea of what's
> > happening it would be great to see how the Scoreboard looks like in
> > server status, and if the error status happens during specific events
> > like graceful reload for log rotation.
>
> All of them happened after a reload - but I'll recheck. I'm pretty sure
> that the /server-status page was no longer responding. Is there any
> other way to get the status of of httpd while it does no longer serve
> pages?


I am not aware of any other way sadly, but hopefully you will not need it
with the new settings :)

Do you have long running http connections that can keep httpd processes in
the G state after reload? Usually this is the main problem, and very easy
to test.

Let us know how it goes!

Luca

Re: Apache 2.4.26 AH: scoreboard is full, not at MaxRequestWorkers.Increase,ServerLimit

[Stefan Priebe - Profihost AG]

Hi,

Am 16.03.2017 um 18:05 schrieb Luca Toscano:
> 
> 
> 2017-03-16 15:24 GMT+01:00 Stefan Priebe - Profihost AG
> mailto:s.pri...@profihost.ag>>:
> 
> Hi,
> 
> Am 16.03.2017 um 12:26 schrieb Luca Toscano:
> > Hi Stefan,
> >
> > 2017-03-16 12:14 GMT+01:00 Stefan Priebe - Profihost AG
> > mailto:s.pri...@profihost.ag>
> >>:
> >
> > Hi Yann,
> >
> > no sure whether this is due to your mpm event patch.
> >
> > From time to time i see the following error mesages in my logs
> and the
> > only chance to fix it is to restart apache.
> >
> > [Thu Mar 16 01:00:35.445184 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:36.446178 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:37.447181 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:38.448177 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:39.449185 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> > [Thu Mar 16 01:00:40.450184 2017] [mpm_event:error] [pid 27485:tid
> > 140212799559552] AH: scoreboard is full, not at
> > MaxRequestWorkers.Increase
> > ServerLimit.
> >
> > Settings:
> > ServerLimit   50
> > ThreadLimit   200
> > ThreadsPerChild200
> > MinSpareThreads200
> > MaxSpareThreads400
> > MaxClients 1
> > MaxRequestsPerChild1
> >
> > MaxRequestWorkers isn't set at all.
> >
> >
> > I believe that MaxClients (its old name) takes the place of
> > MaxRequestWorkers, but it is set as the default value (ServerLimit *
> > ThreadsPerChild). From 2.4.25 onwards mpm-event offers a new
> > functionality to handle Gracefully terminating processes
> > (https://httpd.apache.org/docs/current/mod/event.html#how-it-works
> ), but
> > it must be tuned raising the value of ServerLimit (more details in the
> > docs).
> 
> Hui didn't know this. So i should remove ALL settings and only set:
> ThreadsPerChild
> ServerLimit
> MaxRequestWorkers
> AsyncRequestWorkerFactor
> 
> ? is this true? I'm missing some examples also considering MinSpare and
> ThreadLimit or are they no longer needed?
> 
> 
> I would simply replace MaxClients with MaxRequestWorkers keeping the
> rest of your config (that it is still valid, all the settings that you
> mentioned are still used by mpm-event), without touching
>  AsyncRequestWorkerFactor (unless you want to play with it but the
> default is generally good). About your specific max scoreboard issue, I
> would:
> 
> 1) try to raise ServerLimit to allow more space for slots occupied by
> processes in G state (graceful termination), as indicated in the docs.
> 2) Follow Daniel's suggestion about Max/Min spare workers.

Will try to fix this. Need to implement this in our env. I'll most
probably need until Monday.


> > This is only a speculation from my side, to have a better idea of what's
> > happening it would be great to see how the Scoreboard looks like in
> > server status, and if the error status happens during specific events
> > like graceful reload for log rotation.
> 
> All of them happened after a reload - but I'll recheck. I'm pretty sure
> that the /server-status page was no longer responding. Is there any
> other way to get the status of of httpd while it does no longer
> serve pages?
> 
> 
> I am not aware of any other way sadly, but hopefully you will not need
> it with the new settings :)
> 
> Do you have long running http connections that can keep httpd processes
> in the G state after reload? Usually this is the main problem, and very
> easy to test.
> 
> Let us know how it goes!
> 
> Luca 
>