RE: [users@httpd] How to troubleshoot/fix DAV errors during SVN checkout
> -Original Message-
> From: Yann Ylavic
> Sent: Thursday, 14 January 2021 14:24
> To: users@httpd.apache.org
> Subject: Re: [users@httpd] How to troubleshoot/fix DAV errors during SVN
> checkout
>
> On Thu, Jan 14, 2021 at 2:14 PM Yann Ylavic wrote:
> >
> > You could use a "ProxyPass ... ttl=[timeout]" on the proxy with
> > [timeout] < KeepAliveTimeout on the backend (say the KeepAliveTimeout
> > on the backend's vhost is 5, try ttl=3 in the proxy's ProxyPass
> > directive).
> > By doing this, you avoid the case where the proxy is reusing a
> > connection that is being closed by the backend at the same time
> > (because its KeepAliveTimeout expires), the more connections the
> > greater the risk of this happening..
> >
> > But this is unlikely to fix the errors raised by the backend, because
> > those seem to happen on well established connections.
> > I may be missing something though, so this first step could help already..
>
> As a first first-step and before fine tuning like described above though, I'd
> suggest completely disabling keepalive to see if it helps.
> So either "ProxyPass ... disablereuse=on" on the proxy or "KeepAlive off" on
> the
> backend.
>
> Regards;
> Yann.
Sorry for the very late reply.
I'm unable to reproduce this often enough on our test server so I need to plan
my tests in production carefully.
Adding ttl=3 with the default KeepAliveTimeout on the backend did not help.
I increased ttl to7 while increasing the KeepAliveTimeout on the backend to 10
but that too did not help.
Neither did disablereuse.
I then added the SVNInMemoryCacheSize section that we had on the old Debian
server but that too does not resolve the problem.
Large checkouts (many thousands of files, 6+ GB total) fail consistently.
The checkout failed 3 times on my laptop (SVN clean followed by SVN up each
time).
Checkouts in Jenkins which perform a clean SVN checkout every time fail every
time.
What I noticed today is that even though the Jenkins SVN agent is reporting
hundreds of files being checked out (lines starting with 'A ' there is no
activity logged in the web server logs. Neither on the proxy server nor the
backend server.
For a checkout of 27655 files, 2.12 GB only the following lines are logged
(192.168.14.195 is the client on which the Jenkins agent runs and performs the
checkout):
Logs on the backend:
$ sudo tail -f /var/log/httpd/{access,error}_log|grep .14.195
192.168.14.195 - - [23/Mar/2021:14:45:22 +0100] "OPTIONS /customer/trunk
HTTP/1.1" 401 381 "-" "SVN/1.10.0 SVNKit/1.10.1 (http://svnkit.com/) r10777"
192.168.14.195 - build [23/Mar/2021:14:45:22 +0100] "OPTIONS /customer/trunk
HTTP/1.1" 200 186 "-" "SVN/1.10.0 SVNKit/1.10.1 (http://svnkit.com/) r10777"
192.168.14.195 - build [23/Mar/2021:14:45:22 +0100] "PROPFIND /customer/trunk
HTTP/1.1" 207 682 "-" "SVN/1.10.0 SVNKit/1.10.1 (http://svnkit.com/) r10777"
192.168.14.195 - build [23/Mar/2021:14:45:22 +0100] "PROPFIND
/customer/!svn/vcc/default HTTP/1.1" 207 396 "-" "SVN/1.10.0 SVNKit/1.10.1
(http://svnkit.com/) r10777"
192.168.14.195 - build [23/Mar/2021:14:45:22 +0100] "PROPFIND
/customer/!svn/bln/273 HTTP/1.1" 207 451 "-" "SVN/1.10.0 SVNKit/1.10.1
(http://svnkit.com/) r10777"
192.168.14.195 - build [23/Mar/2021:14:45:22 +0100] "PROPFIND /customer/trunk
HTTP/1.1" 207 682 "-" "SVN/1.10.0 SVNKit/1.10.1 (http://svnkit.com/) r10777"
192.168.14.195 - build [23/Mar/2021:14:45:22 +0100] "PROPFIND
/customer/!svn/vcc/default HTTP/1.1" 207 410 "-" "SVN/1.10.0 SVNKit/1.10.1
(http://svnkit.com/) r10777"
192.168.14.195 - build [23/Mar/2021:14:45:22 +0100] "PROPFIND
/customer/!svn/bc/273/trunk HTTP/1.1" 207 328 "-" "SVN/1.10.0 SVNKit/1.10.1
(http://svnkit.com/) r10777"
192.168.14.195 - build [23/Mar/2021:14:45:22 +0100] "PROPFIND
/customer/!svn/bc/273/trunk HTTP/1.1" 207 328 "-" "SVN/1.10.0 SVNKit/1.10.1
(http://svnkit.com/) r10777"
192.168.14.195 - build [23/Mar/2021:14:45:25 +0100] "PROPFIND /customer/trunk
HTTP/1.1" 207 682 "-" "SVN/1.10.0 SVNKit/1.10.1 (http://svnkit.com/) r10777"
192.168.14.195 - build [23/Mar/2021:14:45:25 +0100] "PROPFIND
/customer/!svn/vcc/default HTTP/1.1" 207 410 "-" "SVN/1.10.0 SVNKit/1.10.1
(http://svnkit.com/) r10777"
192.168.14.195 - build [23/Mar/2021:14:45:25 +0100] "REPORT
/customer/!svn/bc/273/trunk HTTP/1.1" 200 124 "-" "SVN/1.10.0 SVNKit/1.10.1
(http://svnkit.com/) r10777"
192.168.14.195 - build [23/Mar/2021:14:45:25 +0100] "PROPFIND /customer/trunk
H
RE: [users@httpd] How to troubleshoot/fix DAV errors during SVN checkout
Yann, Thanks for the feedback. I'll implement the recommendations and will monitor the situation. Regards Bram -Original Message- From: Yann Ylavic Sent: Thursday, 14 January 2021 14:24 To: users@httpd.apache.org Subject: Re: [users@httpd] How to troubleshoot/fix DAV errors during SVN checkout On Thu, Jan 14, 2021 at 2:14 PM Yann Ylavic wrote: > > You could use a "ProxyPass ... ttl=[timeout]" on the proxy with > [timeout] < KeepAliveTimeout on the backend (say the KeepAliveTimeout > on the backend's vhost is 5, try ttl=3 in the proxy's ProxyPass > directive). > By doing this, you avoid the case where the proxy is reusing a > connection that is being closed by the backend at the same time > (because its KeepAliveTimeout expires), the more connections the > greater the risk of this happening.. > > But this is unlikely to fix the errors raised by the backend, because > those seem to happen on well established connections. > I may be missing something though, so this first step could help already.. As a first first-step and before fine tuning like described above though, I'd suggest completely disabling keepalive to see if it helps. So either "ProxyPass ... disablereuse=on" on the proxy or "KeepAlive off" on the backend. Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] How to troubleshoot/fix DAV errors during SVN checkout
On Thu, Jan 14, 2021 at 2:14 PM Yann Ylavic wrote: > > You could use a "ProxyPass ... ttl=[timeout]" on the proxy with > [timeout] < KeepAliveTimeout on the backend (say the KeepAliveTimeout > on the backend's vhost is 5, try ttl=3 in the proxy's ProxyPass > directive). > By doing this, you avoid the case where the proxy is reusing a > connection that is being closed by the backend at the same time > (because its KeepAliveTimeout expires), the more connections the > greater the risk of this happening.. > > But this is unlikely to fix the errors raised by the backend, because > those seem to happen on well established connections. > I may be missing something though, so this first step could help already.. As a first first-step and before fine tuning like described above though, I'd suggest completely disabling keepalive to see if it helps. So either "ProxyPass ... disablereuse=on" on the proxy or "KeepAlive off" on the backend. Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] How to troubleshoot/fix DAV errors during SVN checkout
Hi, On Thu, Jan 14, 2021 at 11:15 AM Bram Mertens wrote: > > Our setup: > Subversion 1.10 on RHEL8 served by httpd 2.4 > Reverse proxy httpd 2.4 on RHEL8 [snip backend errors] > Am I correct to assume that it is the client (or something between the client > and the proxy server) that is breaking the connection? Or is this a problem > on either of the apache servers? The errors on the backend suggest that the connection with the client was aborted (downstream). Do determine whether it happens before the reverse proxy or between the proxy and the backend you'd have to capture traffic there (tcpdump/wireshark to look for tcp connections reset) or use "LogLevel trace1" on the proxy (that's the level where client network errors are logged there). > > In the error_log of the proxy server I see errors like: > > [Thu Jan 14 06:42:01.647633 2021] [proxy_http:error] [pid 17204:tid > 140412413015808] (104)Connection reset by peer: [client 192.168.x.y:38940] > AH01102: error reading status line from remote server :443 This may be the symptom of a KeepAliveTimeout mismatch between the proxy and the backend, which can be addressed by using a slightly lower value on the proxy. You could use a "ProxyPass ... ttl=[timeout]" on the proxy with [timeout] < KeepAliveTimeout on the backend (say the KeepAliveTimeout on the backend's vhost is 5, try ttl=3 in the proxy's ProxyPass directive). By doing this, you avoid the case where the proxy is reusing a connection that is being closed by the backend at the same time (because its KeepAliveTimeout expires), the more connections the greater the risk of this happening.. But this is unlikely to fix the errors raised by the backend, because those seem to happen on well established connections. I may be missing something though, so this first step could help already.. Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
[users@httpd] How to troubleshoot/fix DAV errors during SVN checkout
Hi, Our setup: Subversion 1.10 on RHEL8 served by httpd 2.4 Reverse proxy httpd 2.4 on RHEL8 We're seeing one or more failures/day during SVN checkout/updates. There appear to be some variations on the error: These are the logs on the SVN (backend) server First type: [dav:error] Provider encountered an error while streaming a REPORT response. [500, #0] [dav:error] A failure occurred while driving the update report editor [500, #104] [dav:error] Error writing base64 data: Connection reset by peer [500, #104] Second type: [dav:error] Provider encountered an error while streaming a REPORT response. [500, #0] [dav:error] A failure occurred while driving the update report editor [500, #32] [dav:error] Error writing base64 data: Broken pipe [500, #32] Third: [dav:error] Provider encountered an error while streaming a REPORT response. [400, #0] [dav:error] Broken pipe [400, #32] Fourth: [dav:error] Provider encountered an error while streaming a REPORT response. [400, #0] [dav:error] Connection reset by peer [400, #104] Clients are both TortoiseSVN on Windows and Jenkins SVNkit on Windows and Linux. Am I correct to assume that it is the client (or something between the client and the proxy server) that is breaking the connection? Or is this a problem on either of the apache servers? In the error_log of the proxy server I see errors like: [Thu Jan 14 06:42:01.647633 2021] [proxy_http:error] [pid 17204:tid 140412413015808] (104)Connection reset by peer: [client 192.168.x.y:38940] AH01102: error reading status line from remote server :443 For this error the following SO post (https://stackoverflow.com/a/64365796/1313616) suggests to add ` SetEnv proxy-initial-not-pooled 1` but https://httpd.apache.org/docs/2.4/mod/mod_proxy_http.html warns that this may downgrade performance. But the timings of these errors do not match the dav:error error messages. Clocks on both servers are synchronized via NTP. How can I fix/troubleshoot this further? I cannot reliably reproduce this on the dev server using a single large checkout. I have found that when a lot of checkouts are triggered simultaneously the error happened. Whether that was because of the high load or simply because more checkouts increase the chance of running into the problem is not clear to me. Thanks in advance Bram
