Hello,
My apologizes, my post of yesterday included HTML tables from server-status
output.(cut/paste).
Here is a more digest version.
Our httpd Apache server keeps hanging every day or many times a day
depending on the activity.
Apache is the frontend of our Subversion 1.6.5 server.
We have about a 100 client developers accessing SVN thru Apache and a few
Hudson continuous integration servers accessing SVN as well and running
many automatic SVN accesses 24/7.
SVN is working well because when the problem happens, I can isssue local
SVN commands outside of httpd without any problem.
Our Apache is listening to 2 different ports 20000 and 20001.
Port 20000 is the standard SVN port and port 20001 is used by the SVN PHP
Admin tools. (svnadmin and websvn)
When the httpd Apache hangs it is always on port 20000.
All SVN requests are stuck for some reasons.
When the httpd hangs on port 20000, if we issue an http request using
svnadmin on port 20001 then the server is unblocked for a short moment
sometime few minutes after hanging again.
Then if we re-issue a request on port 20001 it will again release port
20000 requests for a while.
The only clean solution is to restart the httpd server.
Then it will eventualy run for a few hours or a day or two and then hang
again.
We turned keepalive to OFF to test if we were hiting socket release issues
but the problem just happened with KEEPALIVE set to OFF.
Our server is a large solaris machine with plenty of memory and swap.
Solaris : SunOS MyMachine 5.10 Generic_147440-24 sun4v sparc sun4v
Apache: Server Version: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.7d
DAV/2 SVN/1.6.5 PHP/5.3.1
------------------------------------------------------------------------
Here is the extract from our httpd.conf file
ServerRoot "/opt/xxxxxx/xxx/xxxxxxxxxxxxxxxxxxxxxxx"
LockFile accept.lock
PidFile apache2.pid
Listen 0.0.0.0:20001
LockFile accept.lock
PidFile apache2.pid
Timeout 300
KeepAlive Off # was On before
MaxKeepAliveRequests 1536
KeepAliveTimeout 2
<IfModule mpm_prefork_module>
StartServers 32
MinSpareServers 48
MaxSpareServers 64
ServerLimit 512
MaxClients 96
MaxRequestsPerChild 1536
</IfModule>
[...]
<IfModule alias_module>
ScriptAlias /cgi-bin/ "/opt/blablabla/cgi-bin/"
</IfModule>
<Directory "/opt/blablabla/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
# Main server Admin SVN
ServerName scm.xyz.com
DocumentRoot "/opt/blablabla/htdocs"
<Directory />
Options FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
</Directory>
<Directory "/opt/blablabla/htdocs">
Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>
# PHP5 :
Include conf/extra/httpd-php.conf
# Repository SVN
Listen 0.0.0.0:20000
<VirtualHost *:20000>
Include conf/extra/httpd-svn.conf
</VirtualHost>
------------------------------------------------------------------------
After the apache is hung here is the status-server request:
Apache Server Status for scm.xyz.com
Server Version: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.7d DAV/2
SVN/1.6.5 PHP/5.3.1
Server Built: Jan 13 2010 15:49:24
Current Time: Friday, 16-Aug-2013 15:21:52 CEST
Restart Time: Friday, 16-Aug-2013 03:10:02 CEST
Parent Server Generation: 0
Server uptime: 12 hours 11 minutes 50 seconds
Total accesses: 228561 - Total Traffic: 369.5 MB
CPU Usage: u637.01 s183.88 cu1.4 cs0 - 1.87% CPU load
5.21 requests/sec - 8.6 kB/second - 1694 B/request
8 requests currently being processed, 55 idle workers
_______________________________WWW_W___W______WWR______________.
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
Srv PID Acc M CPU SS Req Conn Child Slot Client
VHost Request
0-0 20113 0/751/3794 _ 7.48 606 6 0.0 0.49 2.95 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
1-0 27522 0/640/3674 _ 12.28 627 7 0.0 0.49 3.04 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1
2-0 20194 0/735/3776 _ 7.80 627 6 0.0 0.42 4.36 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
3-0 27523 0/647/3685 _ 44.08 442 162540 0.0 20.14 21.82
xx.yyy.yy.108 myserver.a.b.c REPORT /my.path/!svn/blablabla/default
HTTP/1.1
4-0 3076 0/1391/3425 W 14.89 0 0 0.0 0.90 2.42 xx.yyy.yy.108
myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.
5-0 10815 0/397/3735 _ 4.42 627 18 0.0 0.24 3.12 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/bblablabla/trunk H
6-0 27524 0/643/3682 _ 13.14 606 33 0.0 1.68 3.58 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/vcc/default HTTP/1.1
7-0 28908 0/633/3666 _ 7.06 627 9 0.0 0.42 5.62 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/vcc/default HTTP/1.1
8-0 27525 0/640/3673 _ 13.35 627 9 0.0 4.97 7.44 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
9-0 27527 0/639/3675 _ 27.84 627 11 0.0 2.62 5.38 xx.yyy.yy.114
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1
10-0 27528 0/638/3677 _ 8.09 606 3 0.0 0.60 7.14 xx.yyy.yy.108
myserver.a.b.c OPTIONS /my.path/blablabla HTTP/1.1
11-0 16355 0/301/3469 _ 3.08 627 6 0.0 0.18 4.83 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla HTTP/1.1
12-0 2761 0/989/3282 _ 10.41 627 6 0.0 0.67 2.72 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
13-0 29771 0/632/3667 _ 6.51 627 16 0.0 0.38 3.40 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk HT
14-0 29928 0/586/3621 _ 8.50 627 5 0.0 0.49 10.14 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla HTTP/1.1
15-0 29772 0/629/3666 _ 6.77 606 8 0.0 0.37 3.15 xx.yyy.yy.108
myserver.a.b.c REPORT /my.path/!svn/blablabla HTTP/1.1
16-0 27529 0/642/3682 _ 6.56 627 6 0.0 0.36 3.67 xx.yyy.yy.114
myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.1
17-0 27530 0/631/3672 _ 11.19 606 6 0.0 0.43 3.80 xx.yyy.yy.108
myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.1
18-0 27827 0/641/3678 _ 6.79 615 5 0.0 0.38 2.20 xx.yyy.yy.140
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
19-0 28230 0/640/3098 _ 7.22 627 6 0.0 0.39 1.72 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
20-0 28302 0/639/3679 _ 7.11 606 6 0.0 0.48 4.59 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
21-0 28231 0/639/3675 W 8.22 0 0 0.0 0.60 4.22 xx.yyy.yy.5
myserver.a.b.c GET /my.path/ HTTP/1.1
22-0 3077 0/1373/2899 _ 26.72 627 17 0.0 3.16 10.28
xx.yyy.yy.108 myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk
23-0 20679 0/235/3408 _ 2.49 606 15 0.0 0.13 2.20 xx.yyy.yy.108
myserver.a.b.c REPORT /my.path/!svn/blablabla/default HTTP/1.1
24-0 3078 0/1386/2905 _ 16.83 627 14 0.0 2.16 3.08 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablablat/trunk
25-0 28297 0/616/2131 _ 7.94 627 11 0.0 0.59 11.05 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
26-0 28299 0/634/3674 _ 8.10 627 15 0.0 0.53 4.19 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk
27-0 28300 0/641/2157 _ 6.58 606 5 0.0 0.34 1.12 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla HTTP/1.1
28-0 28303 0/642/2159 _ 6.82 615 5 0.0 0.43 1.61 xx.yyy.yy.140
myserver.a.b.c PROPFIND /my.path HTTP/1.1
29-0 28304 0/639/3673 _ 7.29 615 10 0.0 0.50 7.08 xx.yyy.yy.140
myserver.a.b.c PROPFIND /my.path/!svn/blablabla HTTP/1.1
30-0 28305 0/633/2149 W 7.62 0 0 0.0 0.44 3.47 xx.yyy.yy.119
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
31-0 29929 0/585/3619 W 6.31 0 0 0.0 0.41 4.00 xx.yyy.yy.108
myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.1
32-0 29930 0/592/3627 W 7.45 0 0 0.0 0.55 2.51 xx.yyy.yy.108
myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.
33-0 28910 0/635/3673 W 8.87 0 0 0.0 0.38 2.44 xx.yyy.yy.108
myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.1
34-0 28696 0/641/3676 _ 6.57 606 98 0.0 0.38 10.64 xx.yyy.yy.108
myserver.a.b.c REPORT /my.path/!svn/blablabla/default HTTP/1.1
35-0 6557 0/481/3517 W 4.98 0 0 0.0 0.33 8.89 xx.yyy.yy.108
myserver.a.b.c OPTIONS /my.path/blablabla/trunk H
36-0 28700 0/633/3671 _ 7.61 627 6 0.0 0.38 2.22 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
37-0 29931 0/588/3621 _ 6.22 627 14 0.0 0.36 13.57 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/testsSecurity
38-0 28697 0/626/3661 _ 9.76 606 71 0.0 0.86 3.63 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
39-0 29766 0/628/3662 W 6.74 0 0 0.0 0.42 4.28 xx.yyy.yy.108
myserver.a.b.c OPTIONS /my.path/trunk/ HTTP/1.1
40-0 29932 0/586/3622 _ 7.25 606 4 0.0 0.40 2.32 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
41-0 13813 0/366/3616 _ 3.58 606 6 0.0 0.22 5.97 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
42-0 9 0/570/3606 _ 5.83 627 13 0.0 0.37 21.85 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla HTTP/1.1
43-0 29770 0/634/3668 _ 7.88 627 11 0.0 0.51 4.62 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
44-0 29916 0/614/3651 _ 8.82 627 16 0.0 2.77 4.50 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1
45-0 29905 0/631/3670 _ 6.87 627 8 0.0 0.37 6.40 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
46-0 29906 0/623/3658 W 7.33 0 0 0.0 0.42 5.68 xx.yyy.yy.108
myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1
47-0 29907 0/627/2494 W 7.11 0 0 0.0 0.51 1.57 xx.yyy.yy.243
myserver.a.b.c GET /server-status HTTP/1.1
48-0 29917 0/615/3648 W 6.73 0 0 0.0 1.35 4.79 xx.yyy.yy.108
myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1
49-0 29918 0/607/3640 _ 9.02 606 6 0.0 0.79 3.13 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk HTTP/1.1
50-0 29919 0/614/3648 _ 6.30 627 17 0.0 0.35 2.08 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/trunk HTTP/
51-0 29933 0/587/3620 W 8.42 0 0 0.0 0.41 2.94 xx.yyy.yy.5
myserver.a.b.c GET /my.path/trunk/ HTTP/1.1
52-0 29934 0/583/3618 _ 7.54 606 6 0.0 0.47 3.41 xx.yyy.yy.108
myserver.a.b.c PROPFIND /blablabla/trunk HTTP/1.1
53-0 29935 0/593/3630 _ 7.32 615 5 0.0 0.64 3.50 xx.yyy.yy.140
myserver.a.b.c PROPFIND blablabla HTTP/1.1
54-0 6566 0/467/3504 W 5.79 0 0 0.0 0.28 2.77 xx.yyy.yy.140
myserver.a.b.c PROPFIND /blablabla HTTP/1.1
55-0 6567 0/473/3508 _ 4.58 606 6 0.0 0.27 2.09 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
56-0 27600 0/132/1651 _ 1.71 627 9 0.0 0.14 7.04 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
57-0 27616 0/63/1602 _ 29.34 615 5 0.0 19.80 22.77 xx.yyy.yy.140
myserver.a.b.c PROPFIND /my.path HTTP/1.1
58-0 27617 0/118/1288 _ 1.65 627 6 0.0 0.08 0.86 xx.yyy.yy.114
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1.1
59-0 27618 0/101/1277 W 0.96 0 0 0.0 0.07 3.19 xx.yyy.yy.108
myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.1
60-0 27619 0/101/3137 _ 1.22 627 5 0.0 0.08 2.15 xx.yyy.yy.114
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
61-0 27621 0/101/3133 _ 1.01 615 5 0.0 0.06 1.77 xx.yyy.yy.140
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
62-0 27623 0/99/3133 _ 1.44 627 18 0.0 0.09 1.81 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla
63-0 - 0/0/1513 . 19.93 10227 11 0.0 0.00 1.34 xx.yyy.yy.108
myserver.a.b.c OPTIONS /my.path/blablabla/trunk HTTP/1.1
64-0 - 0/0/1513 . 16.68 10235 6 0.0 0.00 1.06 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
65-0 - 0/0/1511 . 51.59 10167 9 0.0 0.00 20.72 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/4389 HTTP/1.1
66-0 - 0/0/1520 . 17.51 4947 7 0.0 0.00 1.19 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/blablabla/trunk HTTP/1
67-0 - 0/0/1520 . 17.32 4998 7 0.0 0.00 1.32 xx.yyy.yy.30
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
68-0 - 0/0/1520 . 29.50 4995 10 0.0 0.00 2.92 xx.yyy.yy.30
myserver.a.b.c REPORT /my.path/!svn/blablabla/branches HTTP/1.1
69-0 - 0/0/784 . 8.89 12044 0 0.0 0.00 0.50 127.0.0.1
myserver.a.b.c OPTIONS * HTTP/1.0
70-0 - 0/0/1524 . 17.60 4849 46 0.0 0.00 2.01 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
71-0 - 0/0/1520 . 28.70 4850 18 0.0 0.00 2.42 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/
72-0 - 0/0/1520 . 15.69 5015 7 0.0 0.00 0.89 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
73-0 - 0/0/1520 . 31.96 4955 19 0.0 0.00 8.35 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla
74-0 - 0/0/1521 . 18.82 4998 4 0.0 0.00 0.95 xx.yyy.yy.30
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
75-0 - 0/0/1520 . 16.71 5015 8 0.0 0.00 0.96 xx.yyy.yy.108
myserver.a.b.c PROPFIND /my.path/!svn/blablabla/default HTTP/1.1
------------------------------------------------------------------------
Any help appreciated.
/Nourrédine
