Errors while checking out large directory
I've found several references to this problem via Google & the list archives, but never any answers. When checking out from *some* clients (one in particular), I get the following errors in my Apache error log on a regular basis: [Fri Jan 15 12:33:57 2010] [error] [client IP] Provider encountered an error while streaming a REPORT response. [500, #0] [Fri Jan 15 12:33:57 2010] [error] [client IP] A failure occurred while driving the update report editor [500, #190004] My SVN server is CollabNet's distribution of 1.5.2 running on Windows 2003 with the default configuration of Apache that's included. We have the most difficulty when performing a build using AnthillOS on another Windows 2003 server. HOWEVER, I have three servers running identical versions of AnthillOS, checking out from the same SVN server. Only one of them generates these errors. I have multiple copies of my project being built on each server, but they are done sequentially. Sometimes one or two of the builds will work OK; eventually it'll get to the point where all of them fail. I'm left with a partial checkout of the WC; if I run svn update manually while logged into the server, it completes successfully. If I restart Apache, it seems to run OK...for a while. I have a checkout running right now and I'm showing httpd using about 33MB of memory. This is slowing our development processes significantly, as we're forced to re-attempt each build multiple times. Our change management processes are designed around doing the builds (which also involve tagging) through AnthillOS, so I can't just push it through manually without additional administrative overhead. Has anyone found any resolution to this? I've seen references to this happening as recently as SVN 1.6.1. Do I have to schedule regular restarts of the service? Is there a configuration option I can change?
Re: Errors while checking out large directory
On Fri, Jan 15, 2010 at 15:08, Andy Levy wrote: > I've found several references to this problem via Google & the list > archives, but never any answers. > > When checking out from *some* clients (one in particular), I get the > following errors in my Apache error log on a regular basis: > > [Fri Jan 15 12:33:57 2010] [error] [client IP] Provider encountered an > error while streaming a REPORT response. [500, #0] > [Fri Jan 15 12:33:57 2010] [error] [client IP] A failure occurred > while driving the update report editor [500, #190004] > > My SVN server is CollabNet's distribution of 1.5.2 running on Windows > 2003 with the default configuration of Apache that's included. > > We have the most difficulty when performing a build using AnthillOS on > another Windows 2003 server. HOWEVER, I have three servers running > identical versions of AnthillOS, checking out from the same SVN > server. Only one of them generates these errors. > > I have multiple copies of my project being built on each server, but > they are done sequentially. Sometimes one or two of the builds will > work OK; eventually it'll get to the point where all of them fail. I'm > left with a partial checkout of the WC; if I run svn update manually > while logged into the server, it completes successfully. > > If I restart Apache, it seems to run OK...for a while. I have a > checkout running right now and I'm showing httpd using about 33MB of > memory. > > This is slowing our development processes significantly, as we're > forced to re-attempt each build multiple times. Our change management > processes are designed around doing the builds (which also involve > tagging) through AnthillOS, so I can't just push it through manually > without additional administrative overhead. > > Has anyone found any resolution to this? I've seen references to this > happening as recently as SVN 1.6.1. Do I have to schedule regular > restarts of the service? Is there a configuration option I can change? > Sorry, replying to myself as I omitted what may be important information. The checkout always fails while working on a directory holding (at the moment) 956 items (PDF files), totaling 663MB.
Re: Errors while checking out large directory
On Fri, Jan 15, 2010 at 3:08 PM, Andy Levy wrote: > I've found several references to this problem via Google & the list > archives, but never any answers. > > When checking out from *some* clients (one in particular), I get the > following errors in my Apache error log on a regular basis: > > [Fri Jan 15 12:33:57 2010] [error] [client IP] Provider encountered an > error while streaming a REPORT response. [500, #0] > [Fri Jan 15 12:33:57 2010] [error] [client IP] A failure occurred > while driving the update report editor [500, #190004] > > My SVN server is CollabNet's distribution of 1.5.2 running on Windows > 2003 with the default configuration of Apache that's included. This is usually an issue with the client where it is not responding fast enough and the server thinks the connection has been dropped. For example, the client can spend so long writing the working copy that the server thinks it dropped. http://subversion.tigris.org/faq.html#secure-connection-truncated There is a timeout in the httpd.conf you can set to make the server wait longer. I do not recall what it is off the top of my head. I think if you changed the client to use Serf instead of Neon you would also avoid this problem. Neon does this all as one giant HTTP request, where as Serf breaks it up into lots of smaller GET requests. So I do not think Serf would likely timeout. You should be able to just edit the %AppData%\Subversion\servers file on the client to tell it to use Serf over Neon (assuming your binaries support it). [global] http-library = serf If Anthill uses SVNKit then I am not sure what you can do other than increase the timeout on the server. -- Thanks Mark Phippard http://markphip.blogspot.com/
Re: Errors while checking out large directory
On Fri, Jan 15, 2010 at 15:16, Mark Phippard wrote: > On Fri, Jan 15, 2010 at 3:08 PM, Andy Levy wrote: >> I've found several references to this problem via Google & the list >> archives, but never any answers. >> >> When checking out from *some* clients (one in particular), I get the >> following errors in my Apache error log on a regular basis: >> >> [Fri Jan 15 12:33:57 2010] [error] [client IP] Provider encountered an >> error while streaming a REPORT response. [500, #0] >> [Fri Jan 15 12:33:57 2010] [error] [client IP] A failure occurred >> while driving the update report editor [500, #190004] >> >> My SVN server is CollabNet's distribution of 1.5.2 running on Windows >> 2003 with the default configuration of Apache that's included. > > This is usually an issue with the client where it is not responding > fast enough and the server thinks the connection has been dropped. > For example, the client can spend so long writing the working copy > that the server thinks it dropped. > > http://subversion.tigris.org/faq.html#secure-connection-truncated > > There is a timeout in the httpd.conf you can set to make the server > wait longer. I do not recall what it is off the top of my head. > > I think if you changed the client to use Serf instead of Neon you > would also avoid this problem. Neon does this all as one giant HTTP > request, where as Serf breaks it up into lots of smaller GET requests. > So I do not think Serf would likely timeout. > > You should be able to just edit the %AppData%\Subversion\servers file > on the client to tell it to use Serf over Neon (assuming your binaries > support it). > > [global] > http-library = serf > > If Anthill uses SVNKit then I am not sure what you can do other than > increase the timeout on the server. I'll experiment some with the client config this morning, thanks. I'm not presently capturing User-Agent in my httpd logs - if I was, would that help me identify whether a client is using Serf or Neon?
Re: Errors while checking out large directory
On Mon, Jan 18, 2010 at 8:44 AM, Andy Levy wrote: >> You should be able to just edit the %AppData%\Subversion\servers file >> on the client to tell it to use Serf over Neon (assuming your binaries >> support it). >> >> [global] >> http-library = serf >> >> If Anthill uses SVNKit then I am not sure what you can do other than >> increase the timeout on the server. > > I'll experiment some with the client config this morning, thanks. > > I'm not presently capturing User-Agent in my httpd logs - if I was, > would that help me identify whether a client is using Serf or Neon? Neon is the default so it is pretty unlikely someone would be using Serf. I do not think the User-Agent identifies which library is used. It does however, tell you if SVNKit is being used. The request pattern in the logs between Serf and Neon is very different and that would be the main giveaway. An update with Neon is mostly a single large REPORT request. With Serf it is a small REPORT request followed by a GET for each resource you need to update. -- Thanks Mark Phippard http://markphip.blogspot.com/
Re: Errors while checking out large directory
On Mon, Jan 18, 2010 at 08:55, Mark Phippard wrote: > On Mon, Jan 18, 2010 at 8:44 AM, Andy Levy wrote: >>> You should be able to just edit the %AppData%\Subversion\servers file >>> on the client to tell it to use Serf over Neon (assuming your binaries >>> support it). >>> >>> [global] >>> http-library = serf >>> >>> If Anthill uses SVNKit then I am not sure what you can do other than >>> increase the timeout on the server. >> >> I'll experiment some with the client config this morning, thanks. >> >> I'm not presently capturing User-Agent in my httpd logs - if I was, >> would that help me identify whether a client is using Serf or Neon? > > Neon is the default so it is pretty unlikely someone would be using > Serf. I do not think the User-Agent identifies which library is used. > It does however, tell you if SVNKit is being used. > > The request pattern in the logs between Serf and Neon is very > different and that would be the main giveaway. An update with Neon is > mostly a single large REPORT request. With Serf it is a small REPORT > request followed by a GET for each resource you need to update. Looks like Anthill can't use serf, my logs show the same pattern between yesterday & today. I know that the client config is being picked up from the right place because use-commit-times is being honored. Here's what I'm seeing: 172.30.5.40 - - [17/Jan/2010:20:56:02 -0500] "PROPFIND PATH HTTP/1.1" 207 736 172.30.5.40 - - [17/Jan/2010:20:56:02 -0500] "REPORT /Repos/Code/!svn/vcc/default HTTP/1.1" 200 883378845 883MB response being streamed back to the client, I guess the default 5 minute timeout isn't quite enough sometimes. The way I have things configured, Anthill is checking out a copy of the project while Tomcat is finishing the deployment of the previous build into its temp/working directory. Maybe these overlapped I/O-intensive tasks are slowing each other down? Often the first build succeeds, and subsequent ones fail. I'll have to try doing them independently later today.
Re: Errors while checking out large directory
On Mon, Jan 18, 2010 at 09:27, Andy Levy wrote: > On Mon, Jan 18, 2010 at 08:55, Mark Phippard wrote: >> On Mon, Jan 18, 2010 at 8:44 AM, Andy Levy wrote: You should be able to just edit the %AppData%\Subversion\servers file on the client to tell it to use Serf over Neon (assuming your binaries support it). [global] http-library = serf If Anthill uses SVNKit then I am not sure what you can do other than increase the timeout on the server. >>> >>> I'll experiment some with the client config this morning, thanks. >>> >>> I'm not presently capturing User-Agent in my httpd logs - if I was, >>> would that help me identify whether a client is using Serf or Neon? >> >> Neon is the default so it is pretty unlikely someone would be using >> Serf. I do not think the User-Agent identifies which library is used. >> It does however, tell you if SVNKit is being used. >> >> The request pattern in the logs between Serf and Neon is very >> different and that would be the main giveaway. An update with Neon is >> mostly a single large REPORT request. With Serf it is a small REPORT >> request followed by a GET for each resource you need to update. > > Looks like Anthill can't use serf, my logs show the same pattern > between yesterday & today. I know that the client config is being > picked up from the right place because use-commit-times is being > honored. > > Here's what I'm seeing: > > 172.30.5.40 - - [17/Jan/2010:20:56:02 -0500] "PROPFIND PATH HTTP/1.1" 207 736 > 172.30.5.40 - - [17/Jan/2010:20:56:02 -0500] "REPORT > /Repos/Code/!svn/vcc/default HTTP/1.1" 200 883378845 > > 883MB response being streamed back to the client, I guess the default > 5 minute timeout isn't quite enough sometimes. > > The way I have things configured, Anthill is checking out a copy of > the project while Tomcat is finishing the deployment of the previous > build into its temp/working directory. Maybe these overlapped > I/O-intensive tasks are slowing each other down? Often the first build > succeeds, and subsequent ones fail. I'll have to try doing them > independently later today. It was definitely the I/O doing it. When doing the builds of each project separately (not chaining them one after the other automatically), everything runs properly. It's slowing the checkout down enough that Apache times out. Thanks for the pointer on neon vs. serf & the access log pattern. I just wish I had a more automatic way to do the builds without a major reconfiguration of that server.
