Re: Subversion 1.8 httpd.exe taking 100% CPU
On Wed, Dec 11, 2013 at 12:28 AM, Ben Reser b...@reser.org wrote: On 8/20/13 3:15 AM, Ivan Zhakov wrote: But from what I see process is stuck in libaprutil-1.dll!7489fc50(), given the last message in debug log is auth_ldap authenticate: using URL it most likely problem with communication with LDAP server (ldap protocol implemented in libapr-util). It could be misconfiguration or bug in libapr-util. Note there was another similar thread on this here: https://mail-archives.apache.org/mod_mbox/subversion-users/201311.mbox/%3CCEB51A88.2B46A%25davek%40gamehouse.com%3E I came to the conclusion that there was an LDAP problem as well. Things that would be interesting to know is if the httpd version changed along with the Subversion version or if httpd was left alone and only Subversion was updated. I would be happy to answer any questions on the included version of Apache components in SVN Edge. That said, you can also just check the Apache error log from the SVN Edge web UI and the line when the server is started gives a breakdown of the versions. The current SVN Edge release is 4.0.4 and it includes SVN 1.8.5 and httpd 2.4.7: https://ctf.open.collab.net/sf/wiki/do/viewPage/projects.svnedge/wiki/OpenSourceComponents Based on Valentijn's request from this summer, we have also started capturing the PDB files from the build process and I can supply them to anyone that needs them. Just email me. Valentijn changed his server to Linux before we got a chance to provide them to him. -- Thanks Mark Phippard http://markphip.blogspot.com/
RE: Update-Only Checkout Enhancement
Yes, I understand the export function. I want functionality for release management into test and production environments. For these environments I have a few requirements: Files in these environments will NEVER be edited For new releases I will need to perform an update to revision, which will add, update and delete needed files I want as small of a .svn directory as possible I'm fairly certain that this functionality is currently not present, but I think there are many installations where this would be valuable. I'd like to create an enhancement request, but according to the website it was recommended that I post to this mailing list first. -Original Message- From: Cooke, Mark [mailto:mark.co...@siemens.com] Sent: Wednesday, December 11, 2013 1:44 AM To: Mark Kneisler; users@subversion.apache.org Subject: RE: Update-Only Checkout Enhancement -Original Message- From: Mark Kneisler [mailto:mark.kneis...@tceq.texas.gov] Sent: 11 December 2013 01:45 I have several environments where I'd like to use a SVN checkout, but where I'd never ever want to make changes to the files or perform a commit. For these environments, I'd only want to perform an update or an update to revision. I'm hoping that this type of checkout would require far less space for the .svn directory. AFAIK there is no such option for a checkout, however, you can get the source without the .svn folder using the export command:- http://svnbook.red-bean.com/nightly/en/svn.ref.svn.c.export.html ...you can specify the revision etc. Note, however, that it will not remove unwanted files like an update can. ~ mark c P.s. I think Thorsten just beat me to it...
Re: Update-Only Checkout Enhancement
On Wed, Dec 11, 2013 at 10:46 AM, Mark Kneisler mark.kneis...@tceq.texas.gov wrote: Yes, I understand the export function. I want functionality for release management into test and production environments. For these environments I have a few requirements: Files in these environments will NEVER be edited For new releases I will need to perform an update to revision, which will add, update and delete needed files I want as small of a .svn directory as possible I'm fairly certain that this functionality is currently not present, but I think there are many installations where this would be valuable. I'd like to create an enhancement request, but according to the website it was recommended that I post to this mailing list first. Pristine files do not exist solely so you can edit files. It is convenient they exist so that you can diff your changes or revert your changes without needing to contact the server, or even be online, but obviously those commands could be made to use the server for that and simply require a connection when no pristine is available. The main reason the pristine files exist is so that the client and server can exchange deltas with each other and minimize network traffic. This is just as true for update as it is for commit. One of the original mantra's for the 1.0 release was disk is cheap, network is expensive. There is an existing request in the system for making these files optional: http://subversion.tigris.org/issues/show_bug.cgi?id=525 There is also an existing request for storing the pristine files in compressed format: http://subversion.tigris.org/issues/show_bug.cgi?id=908 -- Thanks Mark Phippard http://markphip.blogspot.com/
Re: Update-Only Checkout Enhancement
Mark Kneisler mark.kneis...@tceq.texas.gov writes: Yes, I understand the export function. I want functionality for release management into test and production environments. For these environments I have a few requirements: Files in these environments will NEVER be edited For new releases I will need to perform an update to revision, which will add, update and delete needed files I want as small of a .svn directory as possible I'm fairly certain that this functionality is currently not present, but I think there are many installations where this would be valuable. I'd like to create an enhancement request, but according to the website it was recommended that I post to this mailing list first. You probably want the optional text base feature: http://subversion.tigris.org/issues/show_bug.cgi?id=525 -- Philip Martin | Subversion Committer WANdisco // *Non-Stop Data*
RE: Update-Only Checkout Enhancement
I think making the pristine files optional would work for me. Here's an idea. Instead of having pristine copies of all files, how about adding to the pristine directory only when a file is changed? From: Mark Phippard [mailto:markp...@gmail.com] Sent: Wednesday, December 11, 2013 9:58 AM To: Mark Kneisler Cc: Cooke, Mark; users@subversion.apache.org Subject: Re: Update-Only Checkout Enhancement On Wed, Dec 11, 2013 at 10:46 AM, Mark Kneisler mark.kneis...@tceq.texas.govmailto:mark.kneis...@tceq.texas.gov wrote: Yes, I understand the export function. I want functionality for release management into test and production environments. For these environments I have a few requirements: Files in these environments will NEVER be edited For new releases I will need to perform an update to revision, which will add, update and delete needed files I want as small of a .svn directory as possible I'm fairly certain that this functionality is currently not present, but I think there are many installations where this would be valuable. I'd like to create an enhancement request, but according to the website it was recommended that I post to this mailing list first. Pristine files do not exist solely so you can edit files. It is convenient they exist so that you can diff your changes or revert your changes without needing to contact the server, or even be online, but obviously those commands could be made to use the server for that and simply require a connection when no pristine is available. The main reason the pristine files exist is so that the client and server can exchange deltas with each other and minimize network traffic. This is just as true for update as it is for commit. One of the original mantra's for the 1.0 release was disk is cheap, network is expensive. There is an existing request in the system for making these files optional: http://subversion.tigris.org/issues/show_bug.cgi?id=525 There is also an existing request for storing the pristine files in compressed format: http://subversion.tigris.org/issues/show_bug.cgi?id=908 -- Thanks Mark Phippard http://markphip.blogspot.com/
Re: Update-Only Checkout Enhancement
On 12/11/13 7:46 AM, Mark Kneisler wrote: Yes, I understand the export function. I want functionality for release management into test and production environments. For these environments I have a few requirements: Files in these environments will NEVER be edited For new releases I will need to perform an update to revision, which will add, update and delete needed files I want as small of a .svn directory as possible I'm fairly certain that this functionality is currently not present, but I think there are many installations where this would be valuable. I'd like to create an enhancement request, but according to the website it was recommended that I post to this mailing list first. Just to set some expectations here. The working copy code is highly complicated and assumes that pristine files are available throughout it. Making this change is not simple. Which is the reason it hasn't already been done.
RE: Update-Only Checkout Enhancement
Yes, I understand the export function. I want functionality for release management into test and production environments. For these environments I have a few requirements: Files in these environments will NEVER be edited For new releases I will need to perform an update to revision, which will add, update and delete needed files I want as small of a .svn directory as possible Why? Disk is cheap. Much cheaper than the time it would take to modify svn to work the way you are requesting. Heck, I just bought a 1TB SSD for ~$500. The spinning version was about $120. I would recommend you write a script that does an export rather than using the update feature. Although, this would probably mean the task would take longer, since you don't have the benefit of the .svn pristines to know what changed files need to be requested from the server. But, it sounds like you care more about disk space than network traffic and/or time. Of course, that is your decision to make. I'm fairly certain that this functionality is currently not present, but I think there are many installations where this would be valuable. I'd like to create an enhancement request, but according to the website it was recommended that I post to this mailing list first. -Original Message- From: Cooke, Mark [mailto:mark.co...@siemens.com] Sent: Wednesday, December 11, 2013 1:44 AM To: Mark Kneisler; users@subversion.apache.org Subject: RE: Update-Only Checkout Enhancement -Original Message- From: Mark Kneisler [mailto:mark.kneis...@tceq.texas.gov] Sent: 11 December 2013 01:45 I have several environments where I'd like to use a SVN checkout, but where I'd never ever want to make changes to the files or perform a commit. For these environments, I'd only want to perform an update or an update to revision. I'm hoping that this type of checkout would require far less space for the .svn directory. AFAIK there is no such option for a checkout, however, you can get the source without the .svn folder using the export command:- http://svnbook.red-bean.com/nightly/en/svn.ref.svn.c.export.html ...you can specify the revision etc. Note, however, that it will not remove unwanted files like an update can. ~ mark c P.s. I think Thorsten just beat me to it...
Re: Update-Only Checkout Enhancement
On Wed, Dec 11, 2013 at 11:16 AM, Bob Archer bob.arc...@amsi.com wrote: Yes, I understand the export function. I want functionality for release management into test and production environments. For these environments I have a few requirements: Files in these environments will NEVER be edited For new releases I will need to perform an update to revision, which will add, update and delete needed files I want as small of a .svn directory as possible Why? Disk is cheap. Much cheaper than the time it would take to modify svn to work the way you are requesting. Heck, I just bought a 1TB SSD for ~$500. The spinning version was about $120. Sometimes it is a good idea to distribute work out to clients and sometimes you really want the client to just be a lightweight client and make the server do the work of serving. I've always liked the minimal amount of data that CVS needs on the clients and sometimes wished that svn could match that. I would recommend you write a script that does an export rather than using the update feature. Although, this would probably mean the task would take longer, since you don't have the benefit of the .svn pristines to know what changed files need to be requested from the server. But, it sounds like you care more about disk space than network traffic and/or time. Of course, that is your decision to make. Within reasonable limits it doesn't cost anything more to send more network traffic. But the cost of client disks scales up by the number of clients. Sometimes you can get by mounting a network disk into all the clients, but then performance suffers, especially with windows clients. -- Les Mikesell lesmikes...@gmail.com
Re: Update-Only Checkout Enhancement
On 12/11/13 9:47 AM, Les Mikesell wrote: Within reasonable limits it doesn't cost anything more to send more network traffic. But the cost of client disks scales up by the number of clients. Sometimes you can get by mounting a network disk into all the clients, but then performance suffers, especially with windows clients. Network traffic has scaling costs just like storage space. If we'd made the decision to not store pristines and you had to go to the server for pristine copies then the discussion here would be reversed. Someone would be asking why we don't just store pristines and pointing out how disk space is cheap compared to the cost of converting their entire network to have more capacity. Can't make everyone happy all the time.
Re: Update-Only Checkout Enhancement
On 11.12.2013 17:21, Mark Kneisler wrote: I think making the pristine files optional would work for me. Here’s an idea. Instead of having pristine copies of all files, how about adding to the pristine directory only when a file is changed? You know, that's a great idea! I wonder why we never thought of it ourselves? :) -- Branko Čibej | Director of Subversion WANdisco // Non-Stop Data e. br...@wandisco.com
RE: Update-Only Checkout Enhancement
On 11.12.2013 17:21, Mark Kneisler wrote: I think making the pristine files optional would work for me. Here’s an idea. Instead of having pristine copies of all files, how about adding to the pristine directory only when a file is changed? You know, that's a great idea! I wonder why we never thought of it ourselves? :) Wouldn't that mean that you need to have some daemon service (or file watcher or something) running to determine if a file is modified? Also, it would mean you would need a constant connection to the server to use a subversion working copy. BOb
Re: Update-Only Checkout Enhancement
On 11.12.2013 20:19, Bob Archer wrote: On 11.12.2013 17:21, Mark Kneisler wrote: I think making the pristine files optional would work for me. Here’s an idea. Instead of having pristine copies of all files, how about adding to the pristine directory only when a file is changed? You know, that's a great idea! I wonder why we never thought of it ourselves? :) Wouldn't that mean that you need to have some daemon service (or file watcher or something) running to determine if a file is modified? Yes. Also, it would mean you would need a constant connection to the server to use a subversion working copy. Not necessarily; we don't need a pristine copy to check if a file is modified, or if 's out of date WRT the repository. But the former problem (requiring a daemon) is already a non-starter. -- Brane -- Branko Čibej | Director of Subversion WANdisco // Non-Stop Data e. br...@wandisco.com
RE: Update-Only Checkout Enhancement
On 11.12.2013 20:19, Bob Archer wrote: On 11.12.2013 17:21, Mark Kneisler wrote: I think making the pristine files optional would work for me. Here’s an idea. Instead of having pristine copies of all files, how about adding to the pristine directory only when a file is changed? You know, that's a great idea! I wonder why we never thought of it ourselves? :) Wouldn't that mean that you need to have some daemon service (or file watcher or something) running to determine if a file is modified? Yes. Also, it would mean you would need a constant connection to the server to use a subversion working copy. Not necessarily; we don't need a pristine copy to check if a file is modified, or if 's out of date WRT the repository. But the former problem (requiring a daemon) is already a non-starter. Right, but if a file is modified you would need to contact the repository to get the pristine because you are going to get an event after the file is modified. There may be some transactional file systems that allow you to get an event before the modification is committed to the file system so you can access the original copy, but I think they are few and far between. (So I guess your original response of great idea was sarcastic and I didn't see it.) BOb
Re: Update-Only Checkout Enhancement
On Wed, 11 Dec 2013, Les Mikesell wrote: On Wed, Dec 11, 2013 at 11:16 AM, Bob Archer bob.arc...@amsi.com wrote: Yes, I understand the export function. I want functionality for release management into test and production environments. For these environments I have a few requirements: Files in these environments will NEVER be edited For new releases I will need to perform an update to revision, which will add, update and delete needed files I want as small of a .svn directory as possible Why? Disk is cheap. Much cheaper than the time it would take to modify svn to work the way you are requesting. Heck, I just bought a 1TB SSD for ~$500. The spinning version was about $120. Sometimes it is a good idea to distribute work out to clients and sometimes you really want the client to just be a lightweight client and make the server do the work of serving. I've always liked the minimal amount of data that CVS needs on the clients and sometimes wished that svn could match that. When FreeBSD switched from CVS to Subversion, there were a couple of projects attempting to replace our cvsup/csup commands for simple, history-less, source checkouts. One of these projects was svnup, http://jcm.dsl.visi.com/freebsd/svnup/ Not sure of the present status.
Re: Update-Only Checkout Enhancement
On Wed, Dec 11, 2013 at 12:24 PM, Ben Reser b...@reser.org wrote: On 12/11/13 9:47 AM, Les Mikesell wrote: Within reasonable limits it doesn't cost anything more to send more network traffic. But the cost of client disks scales up by the number of clients. Sometimes you can get by mounting a network disk into all the clients, but then performance suffers, especially with windows clients. Network traffic has scaling costs just like storage space. Not exactly. Network traffic is generally bursty. Clients rarely spend 100% of their time checking out files, so a very large number could share a local network even if they always deleted their workspaces and checked out fresh copies. But when storing the pristine copies, they can't share anything - even if you map a shared network volume you don't want to share the workspaces. If we'd made the decision to not store pristines and you had to go to the server for pristine copies then the discussion here would be reversed. Someone would be asking why we don't just store pristines and pointing out how disk space is cheap compared to the cost of converting their entire network to have more capacity. Sure - if you aren't local or the server is overloaded it is nice to have the pristine copies. Can't make everyone happy all the time. Well, that's why programs have options - all situations are not the same... -- Les Mikesell lesmikes...@gmail.com
Re: Update-Only Checkout Enhancement
On Wed, Dec 11, 2013 at 3:05 PM, Bob Archer bob.arc...@amsi.com wrote: Wouldn't that mean that you need to have some daemon service (or file watcher or something) running to determine if a file is modified? Yes. Why would you need that in real-time instead of only when an svn operation is done (possibly never...). Also, it would mean you would need a constant connection to the server to use a subversion working copy. That's hardly a problem these days, Not necessarily; we don't need a pristine copy to check if a file is modified, or if 's out of date WRT the repository. But the former problem (requiring a daemon) is already a non-starter. Right, but if a file is modified you would need to contact the repository to get the pristine because you are going to get an event after the file is modified. There may be some transactional file systems that allow you to get an event before the modification is committed to the file system so you can access the original copy, but I think they are few and far between. Assuming you have metadata to know the revision of the file, what possible scenario can there be that you could not get a copy of that revision back from the server if you happen to need it? Isn't that why you put it there in the first place? Or if you were headed this route, wouldn't it be better to send the new copy to the server and let it do the server work? Unless your pending operation is a revert, in which case you would want that copy from the server. -- Les Mikesell lesmikes...@gmail.com
Re: Update-Only Checkout Enhancement
On 12/11/13 5:10 PM, Les Mikesell wrote: Not exactly. Network traffic is generally bursty. Clients rarely spend 100% of their time checking out files, so a very large number could share a local network even if they always deleted their workspaces and checked out fresh copies. But when storing the pristine copies, they can't share anything - even if you map a shared network volume you don't want to share the workspaces. Key phrase there local network. When Subversion was written the vast majority of the users were using it over a WAN. This lead to the assumption that disk space was cheaper than improving the network. That's not necessarily reflective of the use today, but it shaped the assumption that we would always have pristines in the code that was written. Well, that's why programs have options - all situations are not the same... Absolutely, the answer here isn't a one size fits all. Nobody is objecting to the idea of allowing this. The problem is that the code is not designed to allow this and it's a ton of work to change that. I can think of a good 10 other things that would require the same level of effort that would improve things for a lot more people Once you no longer need a pristine there are a lot of potential scenarios that require different behaviors. So it's not even a simple matter of just changing the working copy library to support pristines not existing. It becomes thinking about how to deal with these scenarios (not that all of them need to be implemented immediately, but you probably want to not pigeon hole yourself into an implementation that doesn't support them). Some files don't delta very well and essentially change every bit when the file changes, so those files update and commit are better off just transmitting full text. Some files do delta fine but are just large, if you change the file you may want to run a diff and update/commit would benefit from deltas. If you have a file that deltas well should you download the full text from the server and produce a delta locally or just send the full text to the server? The answer to this depends on the network. If you're on a asymmetric connection where upload or download or different whichever method that sends the largest data over the fastest direction is better (e.g. a typical DSL or Cable internet connection where download is faster it may be far better to download full text and upload the delta you calculate). If you're on a symmetric network you just do whichever takes the least work (directly sending/retrieving full text). Due to laptops that came move between networks you also can't just configure the file on the server side (i.e. set a property). What about compressing the file. If it compresses well then you could have your cake and eat it too... You save disk space and you still have the pristine. The user may want us to retain the pristine data we retrieved to execute a command or they may just want us to destroy it immediately to limit disk space. For that matter the pristine store as of 1.7 doesn't even remove files. So if you're updating a working copy it will just continue to grow. As of 1.8 we use this to allow faster updates when you already have a copy of a version of a file in the pristine store. So we need some sort of garbage collector for the pristine store even for the setup that has it: http://subversion.tigris.org/issues/show_bug.cgi?id=4071 So all in all it's not a matter of throwing a simple option in and being done with it. I realize that your particular use case is probably one of the simpler ones to deal with. But it's by far the least common use case that we hear complaints about with respect to the pristine store. Please realize that considerable thought has gone into this. It's just not done yet.
Re: Update-Only Checkout Enhancement
On Wed, Dec 11, 2013 at 8:26 PM, Ben Reser b...@reser.org wrote: Absolutely, the answer here isn't a one size fits all. Nobody is objecting to the idea of allowing this. The problem is that the code is not designed to allow this and it's a ton of work to change that. I can think of a good 10 other things that would require the same level of effort that would improve things for a lot more people Once you no longer need a pristine there are a lot of potential scenarios that require different behaviors. So it's not even a simple matter of just changing the working copy library to support pristines not existing. It becomes thinking about how to deal with these scenarios (not that all of them need to be implemented immediately, but you probably want to not pigeon hole yourself into an implementation that doesn't support them). I guess I don't understand why it couldn't be as simple as having the library get a pristine copy on demand if some operation needs it. Isn't there already a recovery procedure for a missing pristine file? And then make saving it optional. As a case in point, consider the accumulation of cruft on a typical jenkins build server where a large set of projects are built and rarely removed - you have to allow much more disk space to each build slave to accommodate the pristine files that don't have a whole lot of use. -- Les Mikesell lesmikes...@gmail.com
Re: Update-Only Checkout Enhancement
On Dec 11, 2013, at 19:19, Les Mikesell wrote: Also, it would mean you would need a constant connection to the server to use a subversion working copy. That's hardly a problem these days, You apparently don’t try to work at the kinds of coffee shops I go to, where 50 college students are all watching youtube videos and making the network unbearable and Subversion’s ability to do some things offline very valuable to me.
Re: Update-Only Checkout Enhancement
On 12/11/13 7:16 PM, Les Mikesell wrote: I guess I don't understand why it couldn't be as simple as having the library get a pristine copy on demand if some operation needs it. The original wc code had almost no abstraction around the pristines (as far as I can recall, it's been a long time since I looked at it). WC-NG has more abstraction around them than I recall (I just went and looked). There is a function that retrieves a stream for a pristine, but it doesn't seem to be used everywhere. There are an awful lot of places where it seems we're just passing around the filename (many of which appear to me to be passing through several different layers of the libraries). Someone needs to untangle this mess and get every use of the pristine using a definitive abstraction that can then be made to handle missing pristines. That's not a trivial task. Isn't there already a recovery procedure for a missing pristine file? No there is not. You delete a pristine the working copy is busted.
Re: Update-Only Checkout Enhancement
On Wed, Dec 11, 2013 at 11:01 PM, Ryan Schmidt subversion-20...@ryandesign.com wrote: On Dec 11, 2013, at 19:19, Les Mikesell wrote: Also, it would mean you would need a constant connection to the server to use a subversion working copy. That's hardly a problem these days, You apparently don’t try to work at the kinds of coffee shops I go to, where 50 college students are all watching youtube videos and making the network unbearable and Subversion’s ability to do some things offline very valuable to me. Some things... But not the things you really need to complete any amount of actual work - like updates and commits. -- Les Mikesell lesmikes...@gmail.com
