RE: How to troubleshoot/resolve "connection was forcibly closed by the remote host"
Hi Pavel, Yes, I have added that option but the problem persists. Note that we also have the problem using the TortoiseSVN client. Regards Bram From: Pavel Lyalyakin Sent: Monday, 27 September 2021 15:38 To: Bram Mertens Cc: users@subversion.apache.org Subject: Re: How to troubleshoot/resolve "connection was forcibly closed by the remote host" Hello Bram, Did you enable Spooling in SVNKit? See https://stackoverflow.com/a/57789602/761095 On Mon, Sep 27, 2021 at 2:15 PM Bram Mertens mailto:bram.mert...@anubex.com>> wrote: Hi, Our users (mostly on Windows using TortoiseSVN) and our Jenkins CI builds report errors during checkout and commit of our SVN repositories a couple of times a day. In TortoiseSVN the error message is usually "Error running context: An existing connection was forcibly closed by the remote host.". In Jenkins the stack trace usually shows: ERROR: Failed to check out org.tmatesoft.svn.core.SVNException: svn: E175002: Connection reset svn: E175002: REPORT request failed on '//!svn/vcc/default' ... Caused by: javax.net.ssl.SSLException: Connection reset The problem occurs more frequently since we migrated the SVN repositories from a Debian Linux 8 server with subversion 1.9.5-1+deb9u1 and apache 2.4.25-3+deb9u3 to a RHEL 8 server with subversion-1.10.2-4.module+el8.3.0+9886+ac338b6d and apache 2.4.37-30.module+el8.3.0+7001+0766b9e7. Another main difference is that the old Debian server was located inside our DMZ. The new server is in our LAN and an apache reverse proxy server (also RHEL8 with apache 2.4.37-30.module+el8.3.0+7001+0766b9e7) is in the DMZ proxying requests to the SVN server. In the apache error_log of the proxy server we see occasionally (far fewer than the amount of failures) [proxy_http:error] [pid 716257:tid 140295500982016] (103)Software caused connection abort: [client 192.168.14.209:56678<http://192.168.14.209:56678>] AH01102: error reading status line from remote server :443 In the apache error_log of the SVN server we see other errors but also less than the amount of failures reported by users: [dav:error] [pid 943370:tid 140318913550080] [client :0] Provider encountered an error while streaming a REPORT response. [400, #0] [dav:error] [pid 943370:tid 140318913550080] [client :0] Connection reset by peer [400, #104] [dav:error] [pid 943370:tid 140318863193856] [client :0] Provider encountered an error while streaming a REPORT response. [400, #0] [dav:error] [pid 943370:tid 140318863193856] [client :0] Broken pipe [400, #32] and [dav:error] [pid 935399:tid 140319082698496] [client :0] Provider encountered an error while streaming a REPORT response. [500, #0] [dav:error] [pid 935399:tid 140319082698496] (32)Broken pipe: [client :0] Error flushing brigade. [500, #175002] And [dav:error] [pid 935400:tid 140318141814528] [client :0] Provider encountered an error while streaming a REPORT response. [500, #0] [dav:error] [pid 935400:tid 140318141814528] [client :0] A failure occurred while driving the update report editor [500, #32] [dav:error] [pid 935400:tid 140318141814528] [client :0] Broken pipe [500, #32] And sometimes: [dav:error] [pid 21083:tid 140486543132416] [client :0] Provider encountered an error while streaming a REPORT response. [500, #0] [dav:error] [pid 21083:tid 140486543132416] [client :0] A failure occurred while driving the update report editor [500, #104] [dav:error] [pid 21083:tid 140486543132416] [client :0] Error writing base64 data: Connection reset by peer [500, #104] We see the problem mostly with some of our large repositories although we have also seen it with much smaller repos. I *suspect* that the larger amount of queries needed to checkout a larger repository just means that it is statistically more likely to experience this issue. We also appear to see the problem more often on Windows clients than on Linux but the problem does occur on both OS's. Our Windows clients (servers and laptops) are considerably slower to check out the same repository. Part of this is probably caused by Anti virus software I have already increased the apache timeouts. I have set up a proxy server in our LAN and I'm running some tests to see if I can reproduce the problem bypassing the Firewall but because there is no fixed procedure to reproduce the problem I cannot yet draw conclusions from this. How can I find out what is causing these connection issues? Thanks in advance Bram -- With best regards, Pavel Lyalyakin VisualSVN Team
Re: How to troubleshoot/resolve "connection was forcibly closed by the remote host"
Hello Bram, Did you enable Spooling in SVNKit? See https://stackoverflow.com/a/57789602/761095 On Mon, Sep 27, 2021 at 2:15 PM Bram Mertens wrote: > Hi, > > Our users (mostly on Windows using TortoiseSVN) and our Jenkins CI builds > report errors during checkout and commit of our SVN repositories a couple > of times a day. > > In TortoiseSVN the error message is usually "Error running context: An > existing connection was forcibly closed by the remote host.". > > In Jenkins the stack trace usually shows: > ERROR: Failed to check out > org.tmatesoft.svn.core.SVNException: svn: E175002: Connection reset > svn: E175002: REPORT request failed on '//!svn/vcc/default' > ... > Caused by: javax.net.ssl.SSLException: Connection reset > > The problem occurs more frequently since we migrated the SVN repositories > from a Debian Linux 8 server with subversion 1.9.5-1+deb9u1 and apache > 2.4.25-3+deb9u3 to a RHEL 8 server with > subversion-1.10.2-4.module+el8.3.0+9886+ac338b6d and apache > 2.4.37-30.module+el8.3.0+7001+0766b9e7. > > Another main difference is that the old Debian server was located inside > our DMZ. The new server is in our LAN and an apache reverse proxy server > (also RHEL8 with apache 2.4.37-30.module+el8.3.0+7001+0766b9e7) is in the > DMZ proxying requests to the SVN server. > > In the apache error_log of the proxy server we see occasionally (far fewer > than the amount of failures) > [proxy_http:error] [pid 716257:tid 140295500982016] (103)Software caused > connection abort: [client 192.168.14.209:56678] AH01102: error reading > status line from remote server :443 > > In the apache error_log of the SVN server we see other errors but also > less than the amount of failures reported by users: > [dav:error] [pid 943370:tid 140318913550080] [client :0] Provider > encountered an error while streaming a REPORT response. [400, #0] > [dav:error] [pid 943370:tid 140318913550080] [client :0] Connection > reset by peer [400, #104] > [dav:error] [pid 943370:tid 140318863193856] [client :0] Provider > encountered an error while streaming a REPORT response. [400, #0] > [dav:error] [pid 943370:tid 140318863193856] [client :0] Broken pipe > [400, #32] > and > [dav:error] [pid 935399:tid 140319082698496] [client :0] Provider > encountered an error while streaming a REPORT response. [500, #0] > [dav:error] [pid 935399:tid 140319082698496] (32)Broken pipe: [client > :0] Error flushing brigade. [500, #175002] > And > [dav:error] [pid 935400:tid 140318141814528] [client :0] Provider > encountered an error while streaming a REPORT response. [500, #0] > [dav:error] [pid 935400:tid 140318141814528] [client :0] A failure > occurred while driving the update report editor [500, #32] > [dav:error] [pid 935400:tid 140318141814528] [client :0] Broken pipe > [500, #32] > And sometimes: > [dav:error] [pid 21083:tid 140486543132416] [client :0] Provider > encountered an error while streaming a REPORT response. [500, #0] > [dav:error] [pid 21083:tid 140486543132416] [client :0] A failure > occurred while driving the update report editor [500, #104] > [dav:error] [pid 21083:tid 140486543132416] [client :0] Error writing > base64 data: Connection reset by peer [500, #104] > > We see the problem mostly with some of our large repositories although we > have also seen it with much smaller repos. I *suspect* that the larger > amount of queries needed to checkout a larger repository just means that it > is statistically more likely to experience this issue. > > We also appear to see the problem more often on Windows clients than on > Linux but the problem does occur on both OS's. Our Windows clients (servers > and laptops) are considerably slower to check out the same repository. Part > of this is probably caused by Anti virus software > > I have already increased the apache timeouts. > > I have set up a proxy server in our LAN and I'm running some tests to see > if I can reproduce the problem bypassing the Firewall but because there is > no fixed procedure to reproduce the problem I cannot yet draw conclusions > from this. > > How can I find out what is causing these connection issues? > > Thanks in advance > > Bram > -- With best regards, Pavel Lyalyakin VisualSVN Team
How to troubleshoot/resolve "connection was forcibly closed by the remote host"
Hi, Our users (mostly on Windows using TortoiseSVN) and our Jenkins CI builds report errors during checkout and commit of our SVN repositories a couple of times a day. In TortoiseSVN the error message is usually "Error running context: An existing connection was forcibly closed by the remote host.". In Jenkins the stack trace usually shows: ERROR: Failed to check out org.tmatesoft.svn.core.SVNException: svn: E175002: Connection reset svn: E175002: REPORT request failed on '//!svn/vcc/default' ... Caused by: javax.net.ssl.SSLException: Connection reset The problem occurs more frequently since we migrated the SVN repositories from a Debian Linux 8 server with subversion 1.9.5-1+deb9u1 and apache 2.4.25-3+deb9u3 to a RHEL 8 server with subversion-1.10.2-4.module+el8.3.0+9886+ac338b6d and apache 2.4.37-30.module+el8.3.0+7001+0766b9e7. Another main difference is that the old Debian server was located inside our DMZ. The new server is in our LAN and an apache reverse proxy server (also RHEL8 with apache 2.4.37-30.module+el8.3.0+7001+0766b9e7) is in the DMZ proxying requests to the SVN server. In the apache error_log of the proxy server we see occasionally (far fewer than the amount of failures) [proxy_http:error] [pid 716257:tid 140295500982016] (103)Software caused connection abort: [client 192.168.14.209:56678] AH01102: error reading status line from remote server :443 In the apache error_log of the SVN server we see other errors but also less than the amount of failures reported by users: [dav:error] [pid 943370:tid 140318913550080] [client :0] Provider encountered an error while streaming a REPORT response. [400, #0] [dav:error] [pid 943370:tid 140318913550080] [client :0] Connection reset by peer [400, #104] [dav:error] [pid 943370:tid 140318863193856] [client :0] Provider encountered an error while streaming a REPORT response. [400, #0] [dav:error] [pid 943370:tid 140318863193856] [client :0] Broken pipe [400, #32] and [dav:error] [pid 935399:tid 140319082698496] [client :0] Provider encountered an error while streaming a REPORT response. [500, #0] [dav:error] [pid 935399:tid 140319082698496] (32)Broken pipe: [client :0] Error flushing brigade. [500, #175002] And [dav:error] [pid 935400:tid 140318141814528] [client :0] Provider encountered an error while streaming a REPORT response. [500, #0] [dav:error] [pid 935400:tid 140318141814528] [client :0] A failure occurred while driving the update report editor [500, #32] [dav:error] [pid 935400:tid 140318141814528] [client :0] Broken pipe [500, #32] And sometimes: [dav:error] [pid 21083:tid 140486543132416] [client :0] Provider encountered an error while streaming a REPORT response. [500, #0] [dav:error] [pid 21083:tid 140486543132416] [client :0] A failure occurred while driving the update report editor [500, #104] [dav:error] [pid 21083:tid 140486543132416] [client :0] Error writing base64 data: Connection reset by peer [500, #104] We see the problem mostly with some of our large repositories although we have also seen it with much smaller repos. I *suspect* that the larger amount of queries needed to checkout a larger repository just means that it is statistically more likely to experience this issue. We also appear to see the problem more often on Windows clients than on Linux but the problem does occur on both OS's. Our Windows clients (servers and laptops) are considerably slower to check out the same repository. Part of this is probably caused by Anti virus software I have already increased the apache timeouts. I have set up a proxy server in our LAN and I'm running some tests to see if I can reproduce the problem bypassing the Firewall but because there is no fixed procedure to reproduce the problem I cannot yet draw conclusions from this. How can I find out what is causing these connection issues? Thanks in advance Bram
Re: Connection was forcibly closed
Den fre 12 mars 2021 kl 16:25 skrev Renzo Rosales : > The SSH tunnel is set to use local port 8080 redirect to port 80 on the > remote Fedora system on port 80. > > After running the svn up over the SSH tunnel, most items are updated but 3 > external items fail with the error: "svn: warning: W730054: Error running > context: An existing connection was forcibly closed by the remote host." > How are the externals defined (ie, what is the content of svn:externals property)? Maybe they are setup in such a way that they don't go through the SSH tunnel? Then after running "svn up -r BASE" over the VPN I receive "svn: E730054: > Error running context: An existing connection was forcibly closed by the > remote host." > > Running "svn ls -r BASE --depth=empty" gives me an empty response. > AFAIK, depth=empty only operate on the current dir, and if the problem occurs in a subdirectory, then it maybe you don't run into the error. > Paths: > Doesn't work: > http://server/root/folderA/folder1/folderi/folderone > > Works: > http://server/root/folderA/folder1/folderi/folderone/foldertwo Kind regards, Daniel Sahlberg > > > > > -Original Message- > From: Daniel Shahaf > Sent: Friday, March 5, 2021 11:52 AM > To: Renzo Rosales > Cc: users@subversion.apache.org > Subject: Re: Connection was forcibly closed > > Renzo Rosales wrote on Fri, Mar 05, 2021 at 14:53:02 +: > > We have a few remote users who are unable to run "svn up" to an internal > server in specific paths in a repository but can access others. The error > is "svn: E730054: Error running context: An existing connection was > forcibly closed by the remote host." The server is in the US and some of > the users are in Russia, they communicate with the server over a VPN not > using NAT. If they use Putty to create a SSH port forward to the server, > they can run svn up without error. > > In the Putty case, what URL scheme do they use over the port forward? > > > The rule that allows traffic to traverse the VPN does not have any > network filtering in place. I know this server has an OS release and > software dated from 2011 and 2012 (details below), the httpd access logs > don't show any issues (HTTP code 200 and 207), the error log is bare and > does not show a relevant entry that shows why a sync was blocked or not > permitted. > > > > What are some suggestions on where I can look to better troubleshoot the > issue? > > Run `svn up` over SSH port forwarding then `svn up -r BASE` over the VPN. > That should be a no-op update. Does it fail the same way? What about, > say, `svn ls -r BASE --depth=empty`? (That's a network > operation) > > Check not just the httpd error log but also the system one, in case it's a > packet filter or firewall that killed the connection (notwithstanding the > rule you've reviewed). > > Also, naturally, what's the common thread to the paths that fail, and to > the paths that don't. > > > Would a packet capture help me identify what is going on? If so, what > should I look for in the capture? > > Well, there might be a clue in there. Look for whether it was a FIN or a > RST, and what happened immediately before it — e.g., silence (suggesting > some sort of timeout), or client→server data, or server→client data. > > Cheers, > > Daniel > Email as a communication tool is quite similar to a POST CARD. Short, > public and not certified in any way. As such, we cannot assure you that the > information you receive from us by Email is accurate, complete in every > respect or even FROM whom you believe it to be from. In addition, as > senders of Email we cannot be certain that it is "You" (the "you" to whom > this Email was addressed) that is reading this. Therefore, we offer NO > ASSURANCE about the relevance or validity of information contained herein > or the confidentiality and privacy of that information. We encourage you to > TALK to us directly if you have any questions or comments. >
RE: Connection was forcibly closed
And these are the last few of a tcpdump run from the server side, 10.0.0.44 is the client in Russia and 192.168.2.41 is the server. Does this mean the client is sending the reset and closing the connection? 10:39:26.621041 IP 10.0.0.44.63556 > 192.168.2.41.80: Flags [.], ack 649, win 254, length 0 10:39:26.621047 IP 10.0.0.44.63557 > 192.168.2.41.80: Flags [S], seq 982372545, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0 10:39:26.621631 IP 192.168.2.41.80 > 10.0.0.44.63557: Flags [S.], seq 3498329006, ack 982372546, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 10:39:26.780947 IP 10.0.0.44.63557 > 192.168.2.41.80: Flags [.], ack 1, win 256, length 0 10:39:26.782188 IP 10.0.0.44.63557 > 192.168.2.41.80: Flags [R.], seq 1, ack 1, win 0, length 0 -Original Message- From: Renzo Rosales Sent: Friday, March 12, 2021 10:25 AM To: Daniel Shahaf Cc: users@subversion.apache.org Subject: RE: Connection was forcibly closed The SSH tunnel is set to use local port 8080 redirect to port 80 on the remote Fedora system on port 80. After running the svn up over the SSH tunnel, most items are updated but 3 external items fail with the error: "svn: warning: W730054: Error running context: An existing connection was forcibly closed by the remote host." Then after running "svn up -r BASE" over the VPN I receive "svn: E730054: Error running context: An existing connection was forcibly closed by the remote host." Running "svn ls -r BASE --depth=empty" gives me an empty response. Paths: Doesn't work: http://server/root/folderA/folder1/folderi/folderone Works: http://server/root/folderA/folder1/folderi/folderone/foldertwo -Original Message- From: Daniel Shahaf Sent: Friday, March 5, 2021 11:52 AM To: Renzo Rosales Cc: users@subversion.apache.org Subject: Re: Connection was forcibly closed Renzo Rosales wrote on Fri, Mar 05, 2021 at 14:53:02 +: > We have a few remote users who are unable to run "svn up" to an internal > server in specific paths in a repository but can access others. The error is > "svn: E730054: Error running context: An existing connection was forcibly > closed by the remote host." The server is in the US and some of the users are > in Russia, they communicate with the server over a VPN not using NAT. If they > use Putty to create a SSH port forward to the server, they can run svn up > without error. In the Putty case, what URL scheme do they use over the port forward? > The rule that allows traffic to traverse the VPN does not have any network > filtering in place. I know this server has an OS release and software dated > from 2011 and 2012 (details below), the httpd access logs don't show any > issues (HTTP code 200 and 207), the error log is bare and does not show a > relevant entry that shows why a sync was blocked or not permitted. > > What are some suggestions on where I can look to better troubleshoot the > issue? Run `svn up` over SSH port forwarding then `svn up -r BASE` over the VPN. That should be a no-op update. Does it fail the same way? What about, say, `svn ls -r BASE --depth=empty`? (That's a network operation) Check not just the httpd error log but also the system one, in case it's a packet filter or firewall that killed the connection (notwithstanding the rule you've reviewed). Also, naturally, what's the common thread to the paths that fail, and to the paths that don't. > Would a packet capture help me identify what is going on? If so, what should > I look for in the capture? Well, there might be a clue in there. Look for whether it was a FIN or a RST, and what happened immediately before it — e.g., silence (suggesting some sort of timeout), or client→server data, or server→client data. Cheers, Daniel Email as a communication tool is quite similar to a POST CARD. Short, public and not certified in any way. As such, we cannot assure you that the information you receive from us by Email is accurate, complete in every respect or even FROM whom you believe it to be from. In addition, as senders of Email we cannot be certain that it is "You" (the "you" to whom this Email was addressed) that is reading this. Therefore, we offer NO ASSURANCE about the relevance or validity of information contained herein or the confidentiality and privacy of that information. We encourage you to TALK to us directly if you have any questions or comments.
RE: Connection was forcibly closed
The SSH tunnel is set to use local port 8080 redirect to port 80 on the remote Fedora system on port 80. After running the svn up over the SSH tunnel, most items are updated but 3 external items fail with the error: "svn: warning: W730054: Error running context: An existing connection was forcibly closed by the remote host." Then after running "svn up -r BASE" over the VPN I receive "svn: E730054: Error running context: An existing connection was forcibly closed by the remote host." Running "svn ls -r BASE --depth=empty" gives me an empty response. Paths: Doesn't work: http://server/root/folderA/folder1/folderi/folderone Works: http://server/root/folderA/folder1/folderi/folderone/foldertwo -Original Message- From: Daniel Shahaf Sent: Friday, March 5, 2021 11:52 AM To: Renzo Rosales Cc: users@subversion.apache.org Subject: Re: Connection was forcibly closed Renzo Rosales wrote on Fri, Mar 05, 2021 at 14:53:02 +: > We have a few remote users who are unable to run "svn up" to an internal > server in specific paths in a repository but can access others. The error is > "svn: E730054: Error running context: An existing connection was forcibly > closed by the remote host." The server is in the US and some of the users are > in Russia, they communicate with the server over a VPN not using NAT. If they > use Putty to create a SSH port forward to the server, they can run svn up > without error. In the Putty case, what URL scheme do they use over the port forward? > The rule that allows traffic to traverse the VPN does not have any network > filtering in place. I know this server has an OS release and software dated > from 2011 and 2012 (details below), the httpd access logs don't show any > issues (HTTP code 200 and 207), the error log is bare and does not show a > relevant entry that shows why a sync was blocked or not permitted. > > What are some suggestions on where I can look to better troubleshoot the > issue? Run `svn up` over SSH port forwarding then `svn up -r BASE` over the VPN. That should be a no-op update. Does it fail the same way? What about, say, `svn ls -r BASE --depth=empty`? (That's a network operation) Check not just the httpd error log but also the system one, in case it's a packet filter or firewall that killed the connection (notwithstanding the rule you've reviewed). Also, naturally, what's the common thread to the paths that fail, and to the paths that don't. > Would a packet capture help me identify what is going on? If so, what should > I look for in the capture? Well, there might be a clue in there. Look for whether it was a FIN or a RST, and what happened immediately before it — e.g., silence (suggesting some sort of timeout), or client→server data, or server→client data. Cheers, Daniel Email as a communication tool is quite similar to a POST CARD. Short, public and not certified in any way. As such, we cannot assure you that the information you receive from us by Email is accurate, complete in every respect or even FROM whom you believe it to be from. In addition, as senders of Email we cannot be certain that it is "You" (the "you" to whom this Email was addressed) that is reading this. Therefore, we offer NO ASSURANCE about the relevance or validity of information contained herein or the confidentiality and privacy of that information. We encourage you to TALK to us directly if you have any questions or comments.
Re: Connection was forcibly closed
On Fri, Mar 5, 2021 at 9:53 AM Renzo Rosales wrote: > > We have a few remote users who are unable to run "svn up" to an internal > server in specific paths in a repository but can access others. The error is > "svn: E730054: Error running context: An existing connection was forcibly > closed by the remote host." The server is in the US and some of the users are > in Russia, they communicate with the server over a VPN not using NAT. If they > use Putty to create a SSH port forward to the server, they can run svn up > without error. > > > > The rule that allows traffic to traverse the VPN does not have any network > filtering in place. I know this server has an OS release and software dated > from 2011 and 2012 (details below), the httpd access logs don't show any > issues (HTTP code 200 and 207), the error log is bare and does not show a > relevant entry that shows why a sync was blocked or not permitted. > > > > What are some suggestions on where I can look to better troubleshoot the > issue? Would a packet capture help me identify what is going on? If so, what > should I look for in the capture? Fedora 15? That is 10 years old. Don't even waste time trying to tune this, you need to update your base OS for any exposed server, and your exposed httpd and subversion. Don't waste your time tuning your obsolete versions until you've done the basic server updates. If you still have the problem after that, you should be able to get help tuning a version that is still supported. That said, HTTP and HTTPS connections are often "helped" by proxies that are not tuned for really bulky transmissions such as uploading oversized Subversion commits, especially because many proxy owners want to *block* their clients from uploading bulky content and actively hinder long connections. If you're faced with such proxy difficulties, use svn+SSH rather than HTTPS. Nico Kadel-Garcia
Re: Connection was forcibly closed
Renzo Rosales wrote on Fri, Mar 05, 2021 at 14:53:02 +: > We have a few remote users who are unable to run "svn up" to an internal > server in specific paths in a repository but can access others. The error is > "svn: E730054: Error running context: An existing connection was forcibly > closed by the remote host." The server is in the US and some of the users are > in Russia, they communicate with the server over a VPN not using NAT. If they > use Putty to create a SSH port forward to the server, they can run svn up > without error. In the Putty case, what URL scheme do they use over the port forward? > The rule that allows traffic to traverse the VPN does not have any network > filtering in place. I know this server has an OS release and software dated > from 2011 and 2012 (details below), the httpd access logs don't show any > issues (HTTP code 200 and 207), the error log is bare and does not show a > relevant entry that shows why a sync was blocked or not permitted. > > What are some suggestions on where I can look to better troubleshoot the > issue? Run `svn up` over SSH port forwarding then `svn up -r BASE` over the VPN. That should be a no-op update. Does it fail the same way? What about, say, `svn ls -r BASE --depth=empty`? (That's a network operation) Check not just the httpd error log but also the system one, in case it's a packet filter or firewall that killed the connection (notwithstanding the rule you've reviewed). Also, naturally, what's the common thread to the paths that fail, and to the paths that don't. > Would a packet capture help me identify what is going on? If so, what should > I look for in the capture? Well, there might be a clue in there. Look for whether it was a FIN or a RST, and what happened immediately before it — e.g., silence (suggesting some sort of timeout), or client→server data, or server→client data. Cheers, Daniel
Connection was forcibly closed
We have a few remote users who are unable to run "svn up" to an internal server in specific paths in a repository but can access others. The error is "svn: E730054: Error running context: An existing connection was forcibly closed by the remote host." The server is in the US and some of the users are in Russia, they communicate with the server over a VPN not using NAT. If they use Putty to create a SSH port forward to the server, they can run svn up without error. The rule that allows traffic to traverse the VPN does not have any network filtering in place. I know this server has an OS release and software dated from 2011 and 2012 (details below), the httpd access logs don't show any issues (HTTP code 200 and 207), the error log is bare and does not show a relevant entry that shows why a sync was blocked or not permitted. What are some suggestions on where I can look to better troubleshoot the issue? Would a packet capture help me identify what is going on? If so, what should I look for in the capture? Fedora 15 (Locklock) subversion 1.6.18 Apache 2.2.22 Renzo Rosales Email as a communication tool is quite similar to a POST CARD. Short, public and not certified in any way. As such, we cannot assure you that the information you receive from us by Email is accurate, complete in every respect or even FROM whom you believe it to be from. In addition, as senders of Email we cannot be certain that it is "You" (the "you" to whom this Email was addressed) that is reading this. Therefore, we offer NO ASSURANCE about the relevance or validity of information contained herein or the confidentiality and privacy of that information. We encourage you to TALK to us directly if you have any questions or comments.
RE: E730054: Error running context: An existing connection was forcibly closed by the remote host.
Sorry, my bad - it’s actually RHEL 6.5 (Santiago) DPM From: Mark Phippard [mailto:markp...@gmail.com] Sent: Wednesday, 18 January, 2017 05:55 To: Maloney, Daniel <daniel.malo...@bms.com> Cc: users@subversion.apache.org Subject: Re: E730054: Error running context: An existing connection was forcibly closed by the remote host. On Tue, Jan 17, 2017 at 12:40 PM, Maloney, Daniel <daniel.malo...@bms.com<mailto:daniel.malo...@bms.com>> wrote: I've got a Collabnet Subversion Edge instance that I've been running for years with no issues. Got a complaint from a user about a month ago that she couldn't checkout a repo. I verified the issue, error text in TortoiseSVN client window was "E730054: Error running context: An existing connection was forcibly closed by the remote host." I've tried other repos, and they all fail with this same error. Same thing happens with TortoiseSVN or the command line client. All repos were working fine and everyone's permissions were correct before the issue cropped up, and everyone can still browse the repos on ViewVC. The same error is happening on multiple repos and even happens on new test repos. The error seems to crop up at random places in the checkout process - sometimes a few files into the checkout, sometimes dozens. It never stops at the same file, or even a similar file in terms of size. It seems pretty random. And again, it’s happening on multiple repos. I've tried restarting Subversion Edge, rebooting the server, upgrading the Tortoise client, and I went so far as to upgrade Subversion Edge on the server to 5.1.4-4353.41. I can't see anything useful in the logs from the times I try to check out and get errors. Haven't found much useful when I Google this, so I thought I'd ask for help from the forum. OS is RHEL 4.4.7-4, SVN Edge is providing Apache 2.4.23 and SVN 1.8.17 Is this accurate in terms of the OS? RHEL 4? I am somewhat surprised SVN Edge would run on that OS as we have never tested it. Not even 8 years ago when SVN Edge first came out and RHEL 4 was only "old". -- Thanks Mark Phippard http://markphip.blogspot.com/ This message (including any attachments) may contain confidential, proprietary, privileged and/or private information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution or other use of this message or any attachments by an individual or entity other than the intended recipient is prohibited.
Re: E730054: Error running context: An existing connection was forcibly closed by the remote host.
On Tue, Jan 17, 2017 at 12:40 PM, Maloney, Daniel <daniel.malo...@bms.com> wrote: > I've got a Collabnet Subversion Edge instance that I've been running for > years with no issues. Got a complaint from a user about a month ago that > she couldn't checkout a repo. I verified the issue, error text in > TortoiseSVN client window was "E730054: Error running context: An existing > connection was forcibly closed by the remote host." I've tried other repos, > and they all fail with this same error. Same thing happens with TortoiseSVN > or the command line client. All repos were working fine and everyone's > permissions were correct before the issue cropped up, and everyone can > still browse the repos on ViewVC. The same error is happening on multiple > repos and even happens on new test repos. The error seems to crop up at > random places in the checkout process - sometimes a few files into the > checkout, sometimes dozens. It never stops at the same file, or even a > similar file in terms of size. It seems pretty random. And again, it’s > happening on multiple repos. > > I've tried restarting Subversion Edge, rebooting the server, upgrading the > Tortoise client, and I went so far as to upgrade Subversion Edge on the > server to 5.1.4-4353.41. I can't see anything useful in the logs from the > times I try to check out and get errors. Haven't found much useful when I > Google this, so I thought I'd ask for help from the forum. > > > > OS is RHEL 4.4.7-4, SVN Edge is providing Apache 2.4.23 and SVN 1.8.17 > > > Is this accurate in terms of the OS? RHEL 4? I am somewhat surprised SVN Edge would run on that OS as we have never tested it. Not even 8 years ago when SVN Edge first came out and RHEL 4 was only "old". -- Thanks Mark Phippard http://markphip.blogspot.com/
Re: E730054: Error running context: An existing connection was forcibly closed by the remote host.
On 17.01.2017 18:40, Maloney, Daniel wrote: > I've got a Collabnet Subversion Edge instance that I've been running for > years with no issues. Got a complaint from a user about a month ago that she > couldn't checkout a repo. I verified the issue, error text in TortoiseSVN > client window was "E730054: Error running context: An existing connection was > forcibly closed by the remote host." I've tried other repos, and they all > fail with this same error. Same thing happens with TortoiseSVN or the command > line client. All repos were working fine and everyone's permissions were > correct before the issue cropped up, and everyone can still browse the repos > on ViewVC. The same error is happening on multiple repos and even happens on > new test repos. The error seems to crop up at random places in the checkout > process - sometimes a few files into the checkout, sometimes dozens. It never > stops at the same file, or even a similar file in terms of size. It seems > pretty random. And again, it's happening on multiple repos. This is suspiciously similar to the issue described in this thread: https://lists.apache.org/thread.html/2dd0f5dcd44e66515b70c6cbc07caec21afb67d77085af348a237e2e@1388873289@%3Cusers.subversion.apache.org%3E -- Brane
E730054: Error running context: An existing connection was forcibly closed by the remote host.
I've got a Collabnet Subversion Edge instance that I've been running for years with no issues. Got a complaint from a user about a month ago that she couldn't checkout a repo. I verified the issue, error text in TortoiseSVN client window was "E730054: Error running context: An existing connection was forcibly closed by the remote host." I've tried other repos, and they all fail with this same error. Same thing happens with TortoiseSVN or the command line client. All repos were working fine and everyone's permissions were correct before the issue cropped up, and everyone can still browse the repos on ViewVC. The same error is happening on multiple repos and even happens on new test repos. The error seems to crop up at random places in the checkout process - sometimes a few files into the checkout, sometimes dozens. It never stops at the same file, or even a similar file in terms of size. It seems pretty random. And again, it's happening on multiple repos. I've tried restarting Subversion Edge, rebooting the server, upgrading the Tortoise client, and I went so far as to upgrade Subversion Edge on the server to 5.1.4-4353.41. I can't see anything useful in the logs from the times I try to check out and get errors. Haven't found much useful when I Google this, so I thought I'd ask for help from the forum. OS is RHEL 4.4.7-4, SVN Edge is providing Apache 2.4.23 and SVN 1.8.17 Any help appreciated. Dan Maloney This message (including any attachments) may contain confidential, proprietary, privileged and/or private information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution or other use of this message or any attachments by an individual or entity other than the intended recipient is prohibited.
Re: Error An existing connection was forcibly closed by the remote host with F5 content switch or working copy on shared drive
On Fri, Mar 12, 2010 at 9:34 AM, Justin Johnson jus...@honesthacker.comwrote: Hi, I'm trying to understand why the following error occurs. svn: REPORT request failed on '/svn/reponame/!svn/vcc/default' svn: REPORT of '/svn/reponame/!svn/vcc/default': Could not read response body: An existing connection was forcibly closed by the remote host. ( http://HOSTNAME http://hostname/) command exit code: 1 I've seen this error in a couple of scenarios: 1) when performing a checkout on a Windows box with the working copy stored on a drive mapped to a NAS share 2) when performing a checkout on a Windows box and the server is an F5 content switch that just redirects traffic to the Subversion server The first scenario is of less concern to me, but I mention it anyway since I think it is the same problem. For the second scenario, I worked with someone on our networking team to understand the problem. What he discovered and how he resolved it with our F5 content switch can be found below. The server is running Solaris 10, Subversion 1.6.6, Apache 2.2.11, and repositories are served via HTTP. The client is running Windows XP SP3 and Subversion 1.6.7 (error occurs with TortoiseSVN as well), but the error also occurs on Windows Server 2003. I haven't tested any other Windows client OSes and haven't seen the error on UNIX, but suspect the underlying problem may exist there and the OS handles it more gracefully. Here is the explanation by my networking contact. The problem that is presenting is that the client's receive buffer is filling up and staying full for a long period of time. When this occurs, he advertises a tcp window size of 0 in packets he sends to the destination F5. This also happens when he goes directly against a server. The server seems to tolerate it while the F5 does not. Last year, I took traces of the traffic against the server by the client directly, and through the F5, and saw that the server was seeing different MTU and options from the F5. I modified the standard TCP profile on the F5 to have it proxy the TCP options the client offered so the server would get them. I also set it to proxy the MTU setting the client offered. This seemed to have fixed the problem at that time. But your current testing failed. Upon closer inspection, I determined that the F5 was resetting the connections, not the server as I had previously thought. This time, I turned off those two options from last year and increased the Maximum Segment Retransmissions from the default of 8 to 16. This controls the number of times the F5 resends a packet after it gets no response. This also controls the zero window probes he sends to see if the client can receive data yet. TCP uses a back-off algorithm and increases the time between retries. With 8 attempts, the total retry time is just over a minute. I suspect retries of 16 will cause it to retry for 5 or 10 minutes. I would really like to get this in front of SVN developers, because something is getting hosed on the client that causes him to stop pulling off the receive buffer. If the zero window lasted 10 seconds or so, it would not be a problem. But for him to in effect go offline for over a minute is, I believe, a bug. We can just assume that the reason the error does not occur when you hit the server directly is that the Sun box handles the zero window issue differently, or it might just retry more than 8 times by default. Might be a question for the UNIX team as to the retry count. If we get some time, we could do some packet captures and find out for certain. Yesterday and today, I did a few other things that *did not *help. I increased the TCP receive buffers on the client side sessions, then on the server side sessions, then both. I then turned off all of the tcp options in the F5 default TCP profile. So, in summary, my problem is currently resolved by increasing the Maximum Segment Retransmissions from the default of 8 to 16 on the F5. However, as I mentioned above I've seen this problem when connecting directly to the Subversion server and storing the working copy on a network drive. Does anyone have any ideas? Is this something that can be fixed in the Subversion code itself? Thanks. Justin No responses? This seems like something more for the dev list, but I wanted to follow protocol and wait for a response from the users list first. Thanks. Justin
RE: Error An existing connection was forcibly closed by the remote host with F5 content switch or working copy on shared drive
Hi, Subversion uses the neon (or serf) library for connecting with webdav repositories. It doesn't change any of the tcp settings itself, nor does it handle the tcp connections. (Neither of those has specific MTU handling or anything like that as far as I can tell. They just use the self-tuning support implemented in the operating system) For the server side everything is handled by the Apache httpd process and our mod_dav_svn just receives the pre-parsed requests; so no tcp handling there in the Subversion layer either. So I don't think the Subversion project can really fix this for you (as part of Subversion). But you might find the developers of the other projects on our development list. (Neon, Serf, Neon and Apache Httpd have development lists themselves too) Bert From: Justin Johnson [mailto:jus...@honesthacker.com] Sent: woensdag 17 maart 2010 12:47 To: users@subversion.apache.org Subject: Re: Error An existing connection was forcibly closed by the remote host with F5 content switch or working copy on shared drive On Fri, Mar 12, 2010 at 9:34 AM, Justin Johnson jus...@honesthacker.com wrote: Hi, I'm trying to understand why the following error occurs. svn: REPORT request failed on '/svn/reponame/!svn/vcc/default' svn: REPORT of '/svn/reponame/!svn/vcc/default': Could not read response body: An existing connection was forcibly closed by the remote host. (http://HOSTNAME http://hostname/ ) command exit code: 1 I've seen this error in a couple of scenarios: 1) when performing a checkout on a Windows box with the working copy stored on a drive mapped to a NAS share 2) when performing a checkout on a Windows box and the server is an F5 content switch that just redirects traffic to the Subversion server The first scenario is of less concern to me, but I mention it anyway since I think it is the same problem. For the second scenario, I worked with someone on our networking team to understand the problem. What he discovered and how he resolved it with our F5 content switch can be found below. The server is running Solaris 10, Subversion 1.6.6, Apache 2.2.11, and repositories are served via HTTP. The client is running Windows XP SP3 and Subversion 1.6.7 (error occurs with TortoiseSVN as well), but the error also occurs on Windows Server 2003. I haven't tested any other Windows client OSes and haven't seen the error on UNIX, but suspect the underlying problem may exist there and the OS handles it more gracefully. Here is the explanation by my networking contact. The problem that is presenting is that the client's receive buffer is filling up and staying full for a long period of time. When this occurs, he advertises a tcp window size of 0 in packets he sends to the destination F5. This also happens when he goes directly against a server. The server seems to tolerate it while the F5 does not. Last year, I took traces of the traffic against the server by the client directly, and through the F5, and saw that the server was seeing different MTU and options from the F5. I modified the standard TCP profile on the F5 to have it proxy the TCP options the client offered so the server would get them. I also set it to proxy the MTU setting the client offered. This seemed to have fixed the problem at that time. But your current testing failed. Upon closer inspection, I determined that the F5 was resetting the connections, not the server as I had previously thought. This time, I turned off those two options from last year and increased the Maximum Segment Retransmissions from the default of 8 to 16. This controls the number of times the F5 resends a packet after it gets no response. This also controls the zero window probes he sends to see if the client can receive data yet. TCP uses a back-off algorithm and increases the time between retries. With 8 attempts, the total retry time is just over a minute. I suspect retries of 16 will cause it to retry for 5 or 10 minutes. I would really like to get this in front of SVN developers, because something is getting hosed on the client that causes him to stop pulling off the receive buffer. If the zero window lasted 10 seconds or so, it would not be a problem. But for him to in effect go offline for over a minute is, I believe, a bug. We can just assume that the reason the error does not occur when you hit the server directly is that the Sun box handles the zero window issue differently, or it might just retry more than 8 times by default. Might be a question for the UNIX team as to the retry count. If we get some time, we could do some packet captures and find out for certain. Yesterday and today, I did a few other things that did not help. I increased the TCP receive buffers on the client side sessions, then on the server side sessions, then both. I then turned off all of the tcp options in the F5 default
