Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[Yann Ylavic]

On Thu, Apr 14, 2016 at 10:58 PM, olli hauer  wrote:
>
> OK, I've attached the output from a 2.4.18 scoreboard some sec. after
>  ab -k -n 10 -c 100 -f TLS1.2 $host/$url
>
> the 2.4.18 scoreboard looks similar to 2.4.20 with your patch

Thank you very much olli.

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[olli hauer]

On 2016-04-14 22:43, Yann Ylavic wrote:
> On Thu, Apr 14, 2016 at 10:28 PM, olli hauer  wrote:
>>
>> I've done some tests with 2.4.19 there is maybe an interesting detail.
>> With 2.4.19 the last request is empty for an idle worker, with 2.4.20 not 
>> (shows the client, proto, Vhost and request)
> 
> Sorry for the confusion, 2.4.19 (not released) contained the change
> already, the relevant version to compare with is 2.4.18...
> 

OK, I've attached the output from a 2.4.18 scoreboard some sec. after
 ab -k -n 10 -c 100 -f TLS1.2 $host/$url

the 2.4.18 scoreboard looks similar to 2.4.20 with your patch

-- 
olli
Server Version: Apache/2.4.18 (FreeBSD) OpenSSL/1.0.2g SVN/1.8.15 
mod_wsgi/4.4.22 Python/2.7.11
Server MPM: event
Server Built: Apr 14 2016 22:44:09

Current Time: Thursday, 14-Apr-2016 22:45:55 CEST
Restart Time: Thursday, 14-Apr-2016 22:45:33 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 21 seconds
Server load: 0.69 0.70 0.77
Total accesses: 0 - Total Traffic: 0 kB
CPU Usage: u0 s0 cu0 cs0
0 requests/sec - 0 B/second -
1 requests currently being processed, 74 idle workers

PID Connections Threads Async connections
total   accepting   busyidlewriting keep-alive  closing
97140   0   yes 0   25  0   0   0
97141   0   yes 1   24  0   0   0
97142   0   yes 0   25  0   0   0
Sum 0   1   74  0   0   0





Current Time: Thursday, 14-Apr-2016 22:47:16 CEST
Restart Time: Thursday, 14-Apr-2016 22:45:33 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 1 minute 43 seconds
Server load: 0.70 0.71 0.77
Total accesses: 99957 - Total Traffic: 21.7 MB
CPU Usage: u22.9766 s8.60156 cu0 cs0 - 30.7% CPU load
970 requests/sec - 216.0 kB/second - 227 B/request
1 requests currently being processed, 124 idle workers

PID Connections Threads Async connections
total   accepting   busyidlewriting keep-alive  closing
97140   0   yes 0   25  0   0   0
97141   1   yes 1   24  0   0   0
97142   0   yes 0   25  0   0   0
97192   0   yes 0   25  0   0   0
97212   0   yes 0   25  0   0   0
Sum 1   1   124 0   0   0

_R__
_...






Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv PID Acc M   CPU SS  Req ConnChild   Slot
Client  VHost   Request
0-0 97140   0/1377/1377 _   10.77   39  0   0.0 0.29
0.29$client 
0-0 97140   0/1556/1556 _   10.73   39  0   0.0 0.33
0.33$client $host:443   GET /server-status/ HTTP/1.0
0-0 97140   0/1355/1355 _   10.73   39  0   0.0 0.29
0.29$client $host:443   GET /server-status/ HTTP/1.0
0-0 97140   0/1385/1385 _   10.77   39  0   0.0 0.29
0.29$client 
0-0 97140   0/1326/1326 _   10.74   39  0   0.0 0.28
0.28$client $host:443   GET /server-status/ HTTP/1.0
0-0 97140   0/1402/1402 _   10.76   39  0   0.0 0.30
0.30$client $host:443   GET /server-status/ HTTP/1.0
0-0 97140   0/1381/1381 _   10.77   39  0   0.0 0.29
0.29$client $host:443   GET /server-status/ HTTP/1.0
0-0 97140   0/1467/1467 _   10.77   39  0   0.0 0.31
0.31$client $host:443   GET /server-status/ HTTP/1.0
0-0 97140   0/1486/1486 _   10.73   39  0   0.0 0.32
0.32$client $host:443   GET /server-status/ HTTP/1.0
0-0 97140   0/1420/1420 _   10.76   39  0   0.0 0.30
0.30$client $host:443   GET /server-status/ HTTP/1.0
0-0 97140   0/1377/1377 _   10.77   39  0   0.0 0.29
0.29$client 
0-0 97140   0/1289/1289 _   10.74   39  0   0.0 0.27
0.27$client $host:443   GET /server-status/ HTTP/1.0
0-0 97140   0/1436/1436 _   10.73   39  0   0.0 0.31
0.31$client $host:443   GET /server-status/ HTTP/1.0
0-0 97140   0/1352/1352 _   10.72   

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[Yann Ylavic]

On Thu, Apr 14, 2016 at 10:28 PM, olli hauer  wrote:
>
> I've done some tests with 2.4.19 there is maybe an interesting detail.
> With 2.4.19 the last request is empty for an idle worker, with 2.4.20 not 
> (shows the client, proto, Vhost and request)

Sorry for the confusion, 2.4.19 (not released) contained the change
already, the relevant version to compare with is 2.4.18...

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[olli hauer]

On 2016-04-14 22:14, Yann Ylavic wrote:
> On Thu, Apr 14, 2016 at 10:05 PM, olli hauer  wrote:
>> On 2016-04-14 21:48, Yann Ylavic wrote:
>>> On Thu, Apr 14, 2016 at 9:40 PM, olli hauer  wrote:
 I've done a quick test with
  $ ab -n 1 -c 100 $host/$url

 During the test the count of idle worker are incrementing and decrementing 
 but it ab has finished the requests the count of idle workers stays on the 
 last highest count ...
>>>
>>> This looks normal to me, did the bahaviour changed w.r.t. 2.4.19?
>>>
>>> When the stress is running, the number of idlers (workers that do
>>> nothing) fluctuates, according to the number of connections/requests
>>> and children created to sustain the load.
>>>
>>> When the load stops, the number of idlers increases, and won't
>>> decrease unless MaxSpareThreads is reached (i.e. children are stopped,
>>> each releasing ThreadsPerChild idlers).
>>>
>>
>> Ah, OK this will explain the numbers.
>> Unluckily I haven't looked to scoreboard with 2.4.19, but give me some 
>> minutes ...
> 
> Could you also confirm with 2.4.20 + my patch that the Client, VHost
> and Request colomns are always filled with latest values (not empty)
> for idle workers (used at least once)?
> 
> This requires "ExtendedStatus on" (or default) for the corresponding
> table to be displayed by mod_status.
> 
> Thanks for testing!
> 

Yes, I can confirm this, see the log in the answer to Rainer.

-- 
olli

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[olli hauer]

On 2016-04-14 22:05, Rainer Jung wrote:
> Am 14.04.2016 um 22:05 schrieb olli hauer:
>> On 2016-04-14 21:48, Yann Ylavic wrote:
>>> On Thu, Apr 14, 2016 at 9:40 PM, olli hauer  wrote:
 I've done a quick test with
   $ ab -n 1 -c 100 $host/$url

 During the test the count of idle worker are incrementing and decrementing 
 but it ab has finished the requests the count of idle workers stays on the 
 last highest count ...
>>>
>>> This looks normal to me, did the bahaviour changed w.r.t. 2.4.19?
>>>
>>> When the stress is running, the number of idlers (workers that do
>>> nothing) fluctuates, according to the number of connections/requests
>>> and children created to sustain the load.
>>>
>>> When the load stops, the number of idlers increases, and won't
>>> decrease unless MaxSpareThreads is reached (i.e. children are stopped,
>>> each releasing ThreadsPerChild idlers).
>>>
>>
>> Ah, OK this will explain the numbers.
>> Unluckily I haven't looked to scoreboard with 2.4.19, but give me some 
>> minutes ...
> 
> Not that important for the current discussion, but please note that "ab" does 
> not use Keep-Alive connections by default. If you want to use HTTP 
> Keep-Alive, add "-k".
> 
> Without it you create many new connections per second (depending on 
> throughput this can easily be more than 1000 connections per second) which 
> are only used very shortly (one request) and after shutting down they might 
> end up in TIME_WAIT and stay there e.g. for a minute depending on OS details 
> (you can check via "netstat"). Once you get more than a few thousand 
> connections in TIME_WAIT, TCP/IP could get slower. Thus "ab -k" will often 
> result in more stable behavior.
> 

Hi Rainer,

Thanks for the hint!
(indeed without -k I can simulate the svnstat tool that pulled my svn servers 
down)

I've done some tests with 2.4.19 there is maybe an interesting detail.
With 2.4.19 the last request is empty for an idle worker, with 2.4.20 not 
(shows the client, proto, Vhost and request)

I've attached the results as .txt file.

-- 
olli

httpd some seconds after start:
-
Server MPM: event
Server Built: Apr 14 2016 22:02:14

Current Time: Thursday, 14-Apr-2016 22:03:24 CEST
Restart Time: Thursday, 14-Apr-2016 22:03:08 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 16 seconds
Server load: 1.21 0.69 0.66
Total accesses: 0 - Total Traffic: 0 kB
CPU Usage: u0 s0 cu0 cs0
0 requests/sec - 0 B/second -
1 requests currently being processed, 74 idle workers

PID Connections Threads Async connections
total   accepting   busyidlewriting keep-alive  closing
56486   0   yes 0   25  0   0   0
56487   0   yes 0   25  0   0   0
56488   0   yes 1   24  0   0   0
Sum 0   1   74  0   0   0



during ab test
-
Current Time: Thursday, 14-Apr-2016 22:04:14 CEST
Restart Time: Thursday, 14-Apr-2016 22:03:08 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 1 minute 6 seconds
Server load: 12.23 3.14 1.53
Total accesses: 7868 - Total Traffic: 1.8 MB
CPU Usage: u86.8047 s1.15625 cu0 cs0 - 133% CPU load
119 requests/sec - 28.2 kB/second - 242 B/request
78 requests currently being processed, 97 idle workers

PID Connections Threads Async connections
total   accepting   busyidlewriting keep-alive  closing
56486   4   yes 1   24  0   0   0
56487   7   yes 7   18  0   0   0
56488   10  yes 8   17  0   0   1
56506   10  yes 3   22  0   0   7
56560   25  yes 24  1   0   0   1
56561   5   yes 14  11  0   0   0
56562   8   yes 21  4   0   0   0
Sum 69  78  97  0   0   9



some time after ab finished
-
Current Time: Thursday, 14-Apr-2016 22:06:59 CEST
Restart Time: Thursday, 14-Apr-2016 22:03:08 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 3 minutes 51 seconds
Server load: 1.17 2.09 1.39
Total accesses: 10011 - Total Traffic: 2.5 MB
CPU Usage: u109.742 s1.46094 cu0 cs0 - 48.1% CPU load
43.3 requests/sec - 10.9 kB/second - 257 B/request
1 requests currently being processed, 174 idle workers

PID Connections Threads Async connections
total   accepting   busyidlewriting keep-alive  closing
56486   0   yes 0   25  0   0   0
56487   0   yes 1   24  0   0   0
56488   0   yes 0   25  0   0   0
56506   0   yes 0   25  0   0   0
56560   0   yes 0   25  0   0   0
56561   0   yes 0   

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[Yann Ylavic]

On Thu, Apr 14, 2016 at 10:05 PM, olli hauer  wrote:
> On 2016-04-14 21:48, Yann Ylavic wrote:
>> On Thu, Apr 14, 2016 at 9:40 PM, olli hauer  wrote:
>>> I've done a quick test with
>>>  $ ab -n 1 -c 100 $host/$url
>>>
>>> During the test the count of idle worker are incrementing and decrementing 
>>> but it ab has finished the requests the count of idle workers stays on the 
>>> last highest count ...
>>
>> This looks normal to me, did the bahaviour changed w.r.t. 2.4.19?
>>
>> When the stress is running, the number of idlers (workers that do
>> nothing) fluctuates, according to the number of connections/requests
>> and children created to sustain the load.
>>
>> When the load stops, the number of idlers increases, and won't
>> decrease unless MaxSpareThreads is reached (i.e. children are stopped,
>> each releasing ThreadsPerChild idlers).
>>
>
> Ah, OK this will explain the numbers.
> Unluckily I haven't looked to scoreboard with 2.4.19, but give me some 
> minutes ...

Could you also confirm with 2.4.20 + my patch that the Client, VHost
and Request colomns are always filled with latest values (not empty)
for idle workers (used at least once)?

This requires "ExtendedStatus on" (or default) for the corresponding
table to be displayed by mod_status.

Thanks for testing!

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[Rainer Jung]

Am 14.04.2016 um 22:05 schrieb olli hauer:

On 2016-04-14 21:48, Yann Ylavic wrote:

On Thu, Apr 14, 2016 at 9:40 PM, olli hauer  wrote:

I've done a quick test with
  $ ab -n 1 -c 100 $host/$url

During the test the count of idle worker are incrementing and decrementing but 
it ab has finished the requests the count of idle workers stays on the last 
highest count ...


This looks normal to me, did the bahaviour changed w.r.t. 2.4.19?

When the stress is running, the number of idlers (workers that do
nothing) fluctuates, according to the number of connections/requests
and children created to sustain the load.

When the load stops, the number of idlers increases, and won't
decrease unless MaxSpareThreads is reached (i.e. children are stopped,
each releasing ThreadsPerChild idlers).



Ah, OK this will explain the numbers.
Unluckily I haven't looked to scoreboard with 2.4.19, but give me some minutes 
...


Not that important for the current discussion, but please note that "ab" 
does not use Keep-Alive connections by default. If you want to use HTTP 
Keep-Alive, add "-k".


Without it you create many new connections per second (depending on 
throughput this can easily be more than 1000 connections per second) 
which are only used very shortly (one request) and after shutting down 
they might end up in TIME_WAIT and stay there e.g. for a minute 
depending on OS details (you can check via "netstat"). Once you get more 
than a few thousand connections in TIME_WAIT, TCP/IP could get slower. 
Thus "ab -k" will often result in more stable behavior.


Regards,

Rainer

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[olli hauer]

On 2016-04-14 21:48, Yann Ylavic wrote:
> On Thu, Apr 14, 2016 at 9:40 PM, olli hauer  wrote:
>> I've done a quick test with
>>  $ ab -n 1 -c 100 $host/$url
>>
>> During the test the count of idle worker are incrementing and decrementing 
>> but it ab has finished the requests the count of idle workers stays on the 
>> last highest count ...
> 
> This looks normal to me, did the bahaviour changed w.r.t. 2.4.19?
> 
> When the stress is running, the number of idlers (workers that do
> nothing) fluctuates, according to the number of connections/requests
> and children created to sustain the load.
> 
> When the load stops, the number of idlers increases, and won't
> decrease unless MaxSpareThreads is reached (i.e. children are stopped,
> each releasing ThreadsPerChild idlers).
> 

Ah, OK this will explain the numbers.
Unluckily I haven't looked to scoreboard with 2.4.19, but give me some minutes 
...

-- 
olli

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[olli hauer]

On 2016-04-14 21:47, Eric Covener wrote:
> On Thu, Apr 14, 2016 at 3:40 PM, olli hauer  wrote:
>> During the test the count of idle worker are incrementing and decrementing 
>> but it ab has finished the requests the count of idle workers stays on the 
>> last highest count ...
> 
> What was your MaxSpareThreads during the test?
> 

The system is running with the default settings:


StartServers 3
MinSpareThreads 75
MaxSpareThreads250
ThreadsPerChild 25
MaxRequestWorkers  400
MaxConnectionsPerChild   0

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[Yann Ylavic]

On Thu, Apr 14, 2016 at 9:40 PM, olli hauer  wrote:
> I've done a quick test with
>  $ ab -n 1 -c 100 $host/$url
>
> During the test the count of idle worker are incrementing and decrementing 
> but it ab has finished the requests the count of idle workers stays on the 
> last highest count ...

This looks normal to me, did the bahaviour changed w.r.t. 2.4.19?

When the stress is running, the number of idlers (workers that do
nothing) fluctuates, according to the number of connections/requests
and children created to sustain the load.

When the load stops, the number of idlers increases, and won't
decrease unless MaxSpareThreads is reached (i.e. children are stopped,
each releasing ThreadsPerChild idlers).

Regards,
Yann.

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[Eric Covener]

On Thu, Apr 14, 2016 at 3:40 PM, olli hauer  wrote:
> During the test the count of idle worker are incrementing and decrementing 
> but it ab has finished the requests the count of idle workers stays on the 
> last highest count ...

What was your MaxSpareThreads during the test?

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[olli hauer]

On 2016-04-14 20:36, Yann Ylavic wrote:
> On Thu, Apr 14, 2016 at 7:10 PM, olli hauer  wrote:
>> I got this morning a request from a FreeBSD user to back-port r1739008, and 
>> also already the feedback that scoreboard is usable again with the patch.
>> 
>> Since I cannot find a reference in branches/2.4.x/STATUS for r1739008 I told 
>> the user I will ask additional upstream devs if they think it is OK to 
>> include the patch until a new httpd release will be shipped.
> 
> r1739008+r1739146+r1739151 proposed for backport (full patch is [1]). Let's 
> wait for votes...
> 
> Regards, Yann.
> 
> [1] 
> http://home.apache.org/~ylavic/patches/httpd-2.4.x-scoreboard_preserve.patch
> 

Hi Yann,

thanks for the link!

I've done a quick test with
 $ ab -n 1 -c 100 $host/$url

During the test the count of idle worker are incrementing and decrementing but 
it ab has finished the requests the count of idle workers stays on the last 
highest count ...


httpd some seconds after start:
-
Server Version: Apache/2.4.20 (FreeBSD) OpenSSL/1.0.2g SVN/1.8.15 
mod_wsgi/4.4.22 Python/2.7.11
Server MPM: event
Server Built: Apr 14 2016 21:09:43

Current Time: Thursday, 14-Apr-2016 21:22:13 CEST
Restart Time: Thursday, 14-Apr-2016 21:22:09 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 4 seconds
Server load: 0.29 1.36 1.11
Total accesses: 0 - Total Traffic: 0 kB
CPU Usage: u0 s0 cu0 cs0
0 requests/sec - 0 B/second -
1 requests currently being processed, 74 idle workers

PID Connections Threads Async connections
total   accepting   busyidlewriting keep-alive  closing
31501   0   yes 0   25  0   0   0
31502   0   yes 1   24  0   0   0
31503   0   yes 0   25  0   0   0
Sum 0   1   74  0   0   0


after some manual requests:
-
Current Time: Thursday, 14-Apr-2016 21:22:48 CEST
Restart Time: Thursday, 14-Apr-2016 21:22:09 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 39 seconds
Server load: 0.28 1.24 1.08
Total accesses: 5 - Total Traffic: 18 kB
CPU Usage: u.0234375 s0 cu0 cs0 - .0601% CPU load
.128 requests/sec - 472 B/second - 3686 B/request
1 requests currently being processed, 99 idle workers

PID Connections Threads Async connections
total   accepting   busyidlewriting keep-alive  closing
31501   0   yes 0   25  0   0   0
31502   1   yes 1   24  0   0   0
31503   0   yes 0   25  0   0   0
31505   0   yes 0   25  0   0   0
Sum 1   1   99  0   0   0


give some load with ab -n 1 -c 100
-
Current Time: Thursday, 14-Apr-2016 21:23:29 CEST
Restart Time: Thursday, 14-Apr-2016 21:22:09 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 1 minute 20 seconds
Server load: 1.49 1.38 1.13
Total accesses: 3315 - Total Traffic: 923 kB
CPU Usage: u36.7969 s.570313 cu0 cs0 - 46.7% CPU load
41.4 requests/sec - 11.5 kB/second - 285 B/request
86 requests currently being processed, 89 idle workers

PID Connections Threads Async connections
total   accepting   busyidlewriting keep-alive  closing
31501   3   yes 1   24  0   0   0
31502   2   yes 0   25  0   0   1
31503   25  yes 25  0   0   0   1
31505   25  yes 21  4   0   0   1
31579   2   yes 23  2   0   0   0
31580   3   yes 14  11  0   0   0
31581   3   yes 2   23  0   0   1
Sum 63  86  89  0   0   4


During the test the idle workers are going up and down but stay on the highest 
value after the test is finished.


ab requests finished, wait some time ...
--
Current Time: Thursday, 14-Apr-2016 21:29:12 CEST
Restart Time: Thursday, 14-Apr-2016 21:22:09 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 7 minutes 3 seconds
Server load: 0.20 0.71 0.92
Total accesses: 10024 - Total Traffic: 2.9 MB
CPU Usage: u109.984 s1.4375 cu0 cs0 - 26.3% CPU load
23.7 requests/sec - 7.0 kB/second - 302 B/request
1 requests currently being processed, 174 idle workers

PID Connections Threads Async connections
total   accepting   busyidlewriting keep-alive  closing
31501   0   yes 0   25  0   0   0
31502   0   yes 0   25  0   0   0
31503   0   yes 0   25  0   0   0
31505   0   yes 0   25  0   0   0
31579   0   yes 0   25  0   

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[Yann Ylavic]

On Thu, Apr 14, 2016 at 7:10 PM, olli hauer  wrote:
> I got this morning a request from a FreeBSD user to back-port r1739008, and
> also already the feedback that scoreboard is usable again with the patch.
>
> Since I cannot find a reference in branches/2.4.x/STATUS for r1739008 I told 
> the
> user I will ask additional upstream devs if they think it is OK to include
> the patch until a new httpd release will be shipped.

r1739008+r1739146+r1739151 proposed for backport (full patch is [1]).
Let's wait for votes...

Regards,
Yann.

[1] http://home.apache.org/~ylavic/patches/httpd-2.4.x-scoreboard_preserve.patch

Re: New segfault with 2.4.20 with mod_perl

[William A Rowe Jr]

The defect appears to be in t/protocol/TestProtocol/pseudo_http.pm...

First, the handler is registered using

  PerlProcessConnectionHandler TestProtocol::pseudo_http

so its activities are outside of the request handling phase.

Note that this logic has been broken, for a long time;

   2.4.1>
  
  Order Deny,Allow
  Allow from @servername@
  
  

Where @servername@ is a hostname, this module defect was
identified in version 2.4.20 when we began using the per-req
hostname in comparison (based on r->useragent_addr, which
is obviously is null during part of the read_request phase).

But this module using mod_access_compat during the connection
phase has been broken for much longer, since Allow from {ip-addr}
would already have failed since 2.4.1 was released, due to the
same null r->useragent_addr.

Effectively, mod_access_compat.c never supported per-connection
IP addresses since it was added.  The fact that it supported
per-connection hostname comparison was a quirk, and that the
pseudo_http tests only looked at hostname and not ip comparisons
was an oversight.

But the module will fail in other manners if attempting to use
http request_rec processing since that record is never fleshed
out with the proper read/post_read request hook phases.

My thought is to simply decouple access_compat from this
module test... opinions?

See also; https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820824;msg=5


On Thu, Apr 14, 2016 at 11:55 AM, William A Rowe Jr 
wrote:

> We can be more vigilant about unexpectedly null values, however...
>
> how are you running request processing in the connection callback
> of mod_perl?  That makes no sense, and probably signals a deeper
> logic error.
>
> The access checker is configured per-dir, so until the request rec
> is completely initialized during read_request, this doesn't make
> much sense to me (full backtrace .. including frames #6-#10, for
> those who are curious...)
>
> Either the callback list registered for modperl_callback_connection,
> or the Perl_runops_standard, or the Perl_pp_entersub invoking the
> run_access_checker hook seem the most suspect here.
>
> #0  apr_getnameinfo (hostname=hostname@entry=0x7fd4461ee368, sockaddr=0x0, 
> flags=flags@entry=0)
> at /tmp/buildd/apr-1.5.2/network_io/unix/sockaddr.c:663
> #1  0x55feaf0f513a in ap_get_useragent_host (r=r@entry=0x7fd4461ee0a0, 
> type=type@entry=3,
> str_is_ip=str_is_ip@entry=0x7fd44740c9c4) at core.c:990
> #2  0x7fd4519d7212 in find_allowdeny (r=r@entry=0x7fd4461ee0a0, 
> method=method@entry=0, a=,
> a=) at mod_access_compat.c:279
> #3  0x7fd4519d74b2 in check_dir_access (r=0x7fd4461ee0a0) at 
> mod_access_compat.c:332
> #4  0x55feaf0f8f30 in ap_run_access_checker (r=r@entry=0x7fd4461ee0a0) at 
> request.c:87
> #5  0x7fd448a6f7dd in XS_Apache2__RequestRec_run_access_checker 
> (my_perl=0x55feb2964a20, cv=)
> at HookRun.c:235
> #6  0x7fd44f5f7e6a in Perl_pp_entersub () from 
> /usr/lib/x86_64-linux-gnu/libperl.so.5.22
> #7  0x7fd44f5f0ca6 in Perl_runops_standard () from 
> /usr/lib/x86_64-linux-gnu/libperl.so.5.22
> #8  0x7fd44f575f06 in Perl_call_sv () from 
> /usr/lib/x86_64-linux-gnu/libperl.so.5.22
> #9  0x7fd44f91ec28 in modperl_callback 
> (my_perl=my_perl@entry=0x55feb2964a20, handler=0x7fd4461f2750,
> p=p@entry=0x7fd4461f2028, r=r@entry=0x0, s=s@entry=0x7fd453ddc628, 
> args=0x55feb3beebd0)
> at modperl_callback.c:100
> #10 0x7fd44f91f576 in modperl_callback_run_handlers (idx=0, 
> type=type@entry=1, r=r@entry=0x0,
> c=, s=0x7fd453ddc628, pconf=pconf@entry=0x0, plog=0x0, 
> ptemp=0x0, run_mode=MP_HOOK_RUN_FIRST)
> at modperl_callback.c:236
> #11 0x7fd44f91fd4f in modperl_callback_connection (idx=, 
> c=,
> run_mode=) at modperl_callback.c:359
> #12 0x55feaf10cdf0 in ap_run_process_connection 
> (c=c@entry=0x7fd4461f22b8) at connection.c:42
> #13 0x55feaf10d340 in ap_process_connection (c=c@entry=0x7fd4461f22b8, 
> csd=csd@entry=0x7fd4461f20a0)
> at connection.c:226
> #14 0x7fd4523f3e6b in process_socket (bucket_alloc=0x7fd4461f0028, 
> my_thread_num=1, my_child_num=0,
> sock=0x7fd4461f20a0, p=0x7fd4461f2028, thd=0x7fd453eb27a0) at worker.c:631
> #15 worker_thread (thd=0x7fd453eb27a0, dummy=) at worker.c:990
> #16 0x7fd453418454 in start_thread (arg=0x7fd44740d700) at 
> pthread_create.c:334
> #17 0x7fd453155ecd in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
>
>
> Before we chase down a potential non-defect in httpd, any thoughts
> on the underlying modperl or script logic?
>
>
> On Thu, Apr 14, 2016 at 1:44 AM, Takashi Sato  wrote:
>
>> r->useragent_addr is assigned on ap_read_request (http_core.c),
>> called from ap_process_http_(async_)connection
>> called from process_connection hook (APR_HOOK_REALLY_LAST).
>>
>> The SEGV occured on process_connection hook, maybe 

Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/ssl_engine_kernel.c server/con

[olli hauer]

On 2016-04-13 22:04, Yann Ylavic wrote:
> On Thu, Feb 25, 2016 at 11:27 AM,   wrote:
>> Author: icing
>> Date: Thu Feb 25 10:27:27 2016
>> New Revision: 1732275
>>
>> URL: http://svn.apache.org/viewvc?rev=1732275=rev
>> Log:
>> merging pre_close_connection hook, prep_lingering_close and 
>> ap_update_child() additions from trunk
>>
>> Modified:
> []
>> httpd/httpd/branches/2.4.x/server/scoreboard.c
> []
...
> 
> This seems to have changed mod_status' output.
> Prior to this change the previous values were displayed until the
> worker was reused for another connection/request, now it is reset as
> soon as the worker is idle (i.e. each connection/request end).
> 
> I prefer the new behaviour, but there are some concerns on users@...
> 

I got this morning a request from a FreeBSD user to back-port r1739008, and
also already the feedback that scoreboard is usable again with the patch.

Since I cannot find a reference in branches/2.4.x/STATUS for r1739008 I told the
user I will ask additional upstream devs if they think it is OK to include
the patch until a new httpd release will be shipped.


Do you see any potential issues if r1739008 will be back-ported, or is there
additional work for a better patch in work?

-- 
olli

Re: New segfault with 2.4.20 with mod_perl

[William A Rowe Jr]

We can be more vigilant about unexpectedly null values, however...

how are you running request processing in the connection callback
of mod_perl?  That makes no sense, and probably signals a deeper
logic error.

The access checker is configured per-dir, so until the request rec
is completely initialized during read_request, this doesn't make
much sense to me (full backtrace .. including frames #6-#10, for
those who are curious...)

Either the callback list registered for modperl_callback_connection,
or the Perl_runops_standard, or the Perl_pp_entersub invoking the
run_access_checker hook seem the most suspect here.

#0  apr_getnameinfo (hostname=hostname@entry=0x7fd4461ee368,
sockaddr=0x0, flags=flags@entry=0)
at /tmp/buildd/apr-1.5.2/network_io/unix/sockaddr.c:663
#1  0x55feaf0f513a in ap_get_useragent_host
(r=r@entry=0x7fd4461ee0a0, type=type@entry=3,
str_is_ip=str_is_ip@entry=0x7fd44740c9c4) at core.c:990
#2  0x7fd4519d7212 in find_allowdeny (r=r@entry=0x7fd4461ee0a0,
method=method@entry=0, a=,
a=) at mod_access_compat.c:279
#3  0x7fd4519d74b2 in check_dir_access (r=0x7fd4461ee0a0) at
mod_access_compat.c:332
#4  0x55feaf0f8f30 in ap_run_access_checker
(r=r@entry=0x7fd4461ee0a0) at request.c:87
#5  0x7fd448a6f7dd in XS_Apache2__RequestRec_run_access_checker
(my_perl=0x55feb2964a20, cv=)
at HookRun.c:235
#6  0x7fd44f5f7e6a in Perl_pp_entersub () from
/usr/lib/x86_64-linux-gnu/libperl.so.5.22
#7  0x7fd44f5f0ca6 in Perl_runops_standard () from
/usr/lib/x86_64-linux-gnu/libperl.so.5.22
#8  0x7fd44f575f06 in Perl_call_sv () from
/usr/lib/x86_64-linux-gnu/libperl.so.5.22
#9  0x7fd44f91ec28 in modperl_callback
(my_perl=my_perl@entry=0x55feb2964a20, handler=0x7fd4461f2750,
p=p@entry=0x7fd4461f2028, r=r@entry=0x0, s=s@entry=0x7fd453ddc628,
args=0x55feb3beebd0)
at modperl_callback.c:100
#10 0x7fd44f91f576 in modperl_callback_run_handlers (idx=0,
type=type@entry=1, r=r@entry=0x0,
c=, s=0x7fd453ddc628, pconf=pconf@entry=0x0,
plog=0x0, ptemp=0x0, run_mode=MP_HOOK_RUN_FIRST)
at modperl_callback.c:236
#11 0x7fd44f91fd4f in modperl_callback_connection (idx=, c=,
run_mode=) at modperl_callback.c:359
#12 0x55feaf10cdf0 in ap_run_process_connection
(c=c@entry=0x7fd4461f22b8) at connection.c:42
#13 0x55feaf10d340 in ap_process_connection
(c=c@entry=0x7fd4461f22b8, csd=csd@entry=0x7fd4461f20a0)
at connection.c:226
#14 0x7fd4523f3e6b in process_socket (bucket_alloc=0x7fd4461f0028,
my_thread_num=1, my_child_num=0,
sock=0x7fd4461f20a0, p=0x7fd4461f2028, thd=0x7fd453eb27a0) at worker.c:631
#15 worker_thread (thd=0x7fd453eb27a0, dummy=) at worker.c:990
#16 0x7fd453418454 in start_thread (arg=0x7fd44740d700) at
pthread_create.c:334
#17 0x7fd453155ecd in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109


Before we chase down a potential non-defect in httpd, any thoughts
on the underlying modperl or script logic?


On Thu, Apr 14, 2016 at 1:44 AM, Takashi Sato  wrote:

> r->useragent_addr is assigned on ap_read_request (http_core.c),
> called from ap_process_http_(async_)connection
> called from process_connection hook (APR_HOOK_REALLY_LAST).
>
> The SEGV occured on process_connection hook, maybe before
> ap_process_http_(async_)connection,
>
> #11 0x7fd44f91fd4f in modperl_callback_connection (idx= out>, c=,
> run_mode=) at modperl_callback.c:359
> #12 0x55feaf10cdf0 in ap_run_process_connection
> (c=c@entry=0x7fd4461f22b8) at connection.c:42
> #13 0x55feaf10d340 in ap_process_connection
> (c=c@entry=0x7fd4461f22b8, csd=csd@entry=0x7fd4461f20a0)
> at connection.c:226
>
> so r->useragent_addr had not been assigned any value.
>

Re: Unintended Side Effects (Was: Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/

[Stefan Eissing]

> Am 14.04.2016 um 15:09 schrieb Jim Jagielski :
> 
> We should always ensure that when we fix something, or
> when we add something, we don't change the current behavior
> of httpd in an unintended way. This rev is a great example of
> that. The clearing of those scoreboard slots is completely
> unrelated to the actual change itself.

Yes, I meant well, but should have known better or at least
asked for confirmation on the list. I have no excuse.

Scoreboard is one of the areas where unwanted changes can slip
through more easily than in other places, as there are no test
cases covering this text output.

Since its functionality is vital for a lot of users, maybe
we should invest into having a formatted, parseable variant
of its output. That we could use in tests.

> Of course, the "blame" is shared because even if the patch
> does change something, it is up to all of us to review it,
> not just by running and testing the code, but actually
> *inspecting* the patch for what it does.
> 
> The very fact that this change made it all the way
> thru the backport process to end up in 2.4.x is a reminder
> that we all need to stay vigilant.

Anything that is not covered by test code will break sooner
or later unless the original, sole author becomes immortal.

That is not meant as an excuse, just an observation.

-Stefan

Unintended Side Effects (Was: Re: svn commit: r1732275 - in /httpd/httpd/branches/2.4.x: ./ include/ap_mmn.h include/http_connection.h include/scoreboard.h modules/generators/mod_status.c modules/ssl/

[Jim Jagielski]

We should always ensure that when we fix something, or
when we add something, we don't change the current behavior
of httpd in an unintended way. This rev is a great example of
that. The clearing of those scoreboard slots is completely
unrelated to the actual change itself.

Of course, the "blame" is shared because even if the patch
does change something, it is up to all of us to review it,
not just by running and testing the code, but actually
*inspecting* the patch for what it does.

The very fact that this change made it all the way
thru the backport process to end up in 2.4.x is a reminder
that we all need to stay vigilant.

Re: Allow SSLProxy* config in context?

[Rainer Jung]

Am 14.04.2016 um 02:57 schrieb Daniel Ruggeri:


On 4/13/2016 2:22 PM, Rainer Jung wrote:


We could pass the worker name from mod_proxy to mod_ssl via a
connection note, similar to currently already passing the SNI name via
the connection note proxy-request-hostname.


+1 on the connection note idea, but see below about having to inform the
callback function about too much information

On 4/13/2016 10:04 AM, Graham Leggett wrote:

What I had in mind was a syntax where the certs were named, for example:

SSLProxyCertificate foo /path/to/foo.cert

Followed somewhere else by:


   SSLProxyEnable foo



On one hand, this is an attractive idea because we have already
conditioned users/administrators to think about naming things via config
as you can do with authn providers and aliases.

On another hand, there are two gotchas I see. Giving something a name in
the configuration does not necessarily mean anything to the file or
certificate store because they aren't married in the same way that authn
providers and their aliases are. That is, if a different team/individual
manages the certificate store than the server configuration, the two
would have to keep in sync with additions or removals. This can be a
PITA if you have a short certificate lifecycle and have to renew often.
PLUS, there are cases where there may be many certificates in the file.
Combine that with the previous point and you could have a nightmare.

Perhaps in addition to or instead of aliases via config, something more
"dynamic" could be used where a DN, CN or some other attribute about the
cert that was parsed during startup becomes the representation?


...
   #As today
SSLProxyMachineCertificateFile /path/to/file

   #Plus new stuff
   SSLProxyMachineCertificateCN myIdentity
   #or
   SSLProxyMachineCertificateDN /C=US/ST=Missouri/L=St.
Louis/O=BitNebula/CN=DanielRuggeri.bitnebula.com



This could be a really clean solution because the proxy can just pass
this name or DN to the ssl module in a note and the ssl module could
make the selection of certs from that data.

Stashing this (completely untested and thrown together) code around line
1778 in modules/ssl/ssl_engine_kernel.c could be all that's needed in
mod_ssl
 char *preferred_name;
..
 if (preferred_name) {
 for (i = 0; i < sk_X509_INFO_num(certs); i++) {
 X509_NAME *name;
 info = sk_X509_INFO_value(certs, i);
 name = X509_get_subject_name(inf->x509);

 for(j = 0; j < X509_NAME_entry_count(name); j++) {
 const char *this_name = //something
 if (strcmp(preferred_name, this_name) == 0) {
 modssl_proxy_info_log(c, info, APLOGNO(02279)
   "found acceptable cert");

 modssl_set_cert_info(info, x509, pkey);
 return TRUE;
 }
 }
 }
 ap_log_error(APLOG_MARK, APLOG_WARNING, 0, s, APLOGNO(00)
 SSLPROXY_CERT_CB_LOG_FMT
 "server configuration requested certificate name %s"
 "but it was not found in the certificate store",
preferred_name);
 }



I haven't cracked open the proxy code to see what it would take to
collapse this down to a per proxy/worker/etc, but it doesn't seem like
terrible endeavor.


Yeah that's looks very feasible and would allow the actual use case. I 
was wondering whether more SSLProxy* directives would benefit from a per 
backend config possibility. And as a variation of Graham's suggestion, I 
was thinking about (example)


http://mycluster>
SSLProxyMachineCertificateFile /path/to/file
... proxy settings ...


In the vhost config we would not then have an array or hash of ssl proxy 
ocnfig setting. The key to the hash would be "http://mycluster; in the 
example above. And mod_proxy would forward "http://mycluster; to mod_ssl 
via connection notes to allow mod_ssl to select the right proxy ssl config.


So there would be no additional name "foo", instead we would reuse the 
name of the worker from the surrounding proxy block. The selection of 
the certificate would still happen based on the CA, but it would be 
selected from the right store (the one associated with "http://mycluster;).


I see your point about having to administer more ssl files outside of 
the config, ie. client cert files or directories per backend. I would 
hope that those would be manageable by choosing some naming convention 
for the file names.


Your idea to allow selecting a client cert based on CN or DN sounds 
attractive to me as well. But since it wouldn't help with other per 
backend settings (like different Verify settings) we might even think 
about combining both. As long as your only problem would be client 
certs, you can select via CN or DN and keep the config vhost global, but 
once you need more adjustments per backend, you would start to configure 
SSLProxy* 

Re: Allow SSLProxy* config in context?

[Yann Ylavic]

On Wed, Apr 13, 2016 at 7:49 PM, Rainer Jung  wrote:
> Am 13.04.2016 um 17:04 schrieb Graham Leggett:
>>
>> The catch is that mod_ssl forces us to declare SSL certs and keys server
>> wide, not per directory, loaded and parsed at startup. We however want to
>> specify certs per directory.
>
> Per directory or better in some new way per proxy backend (or proxy worker,
> proxy balancer).

IIUC, the  block is a per_dir context already, which can/could
accept any directive provided their ap_check_cmd_context() allows it
(we may need to declare a new PROXY_CONF).

So how about making per_dir SSLProxy* directives, restricted to server
and  context?
This would let the loading (and validation) work like currently,
mod_ssl could still do its standalone post_config stuff (server AND
per_dir wise).

At runtime, proxy_walk() would still do the merging (there may be same
SSLProxy* in both  and  which need merging, but
that would be a single one since we restrict those directives to
server and  context).

Finally, if ssl_proxy_enable[_ex]() is given r->per_dir_config, it
could initialize the backend connection with the right context.

Wouldn't that work without so many changes?

Re: New segfault with 2.4.20 with mod_perl

[Takashi Sato]

r->useragent_addr is assigned on ap_read_request (http_core.c),
called from ap_process_http_(async_)connection
called from process_connection hook (APR_HOOK_REALLY_LAST).

The SEGV occured on process_connection hook, maybe before
ap_process_http_(async_)connection,

#11 0x7fd44f91fd4f in modperl_callback_connection (idx=, c=,
run_mode=) at modperl_callback.c:359
#12 0x55feaf10cdf0 in ap_run_process_connection
(c=c@entry=0x7fd4461f22b8) at connection.c:42
#13 0x55feaf10d340 in ap_process_connection
(c=c@entry=0x7fd4461f22b8, csd=csd@entry=0x7fd4461f20a0)
at connection.c:226

so r->useragent_addr had not been assigned any value.