High availability
Hi, I'm running a rather large Subversion installation (1000 repos, 1 TB total storage, 1000 users). I'm looking for advice how to improve our availability. We're currently quite good, but I'm a bit worried about e.g. hardware failure. Performance is not an issue for now (machine load 2, at about 20 svn-requests per second). How are you out there doing it? I have a few ideas, but I'd like to hear the opinions of others and get maybe some pointers for more research. My ideas: * use svnsync replication. Drawback: failure needs manual intervention, hook scripts need to be transferred manually. * use an active-passive cluster with e.g. heartbeat. That would be possible, the data reside on a SAN anyway. * use an active-active cluster with two separate machines sharing the storage via a cluster fs (GFS? GPFS?) with a HA load balancer in front. Probably the sexiest solution ;-) * I already discarded the idea of using active-active with NFS since I can remember the reports of strange failures... Does anyone here know how other large/high profile sites (e.g. the Apache foundation) are ensuring availability? I couldn't find any hints at the website... Cheers, Ulli -- Ullrich Jans, Specialist, IT-A Phone: +49 9131 7701-6627, mailto:ullrich.j...@elektrobit.com Fax: +49 9131 7701-6333, www.elektrobit.com Elektrobit Automotive GmbH, Am Wolfsmantel 46, 91058 Erlangen, Germany Managing Directors: Alexander Kocher, Gregor Zink Register Court Fürth HRB 4886 Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: High availability
Hi Nico, -Original Message- From: Nico Kadel-Garcia [mailto:nka...@gmail.com] Sent: Thursday, October 04, 2012 10:30 AM To: Jans Ullrich Cc: users@subversion.apache.org Subject: Re: High availability Go call WanDisco. They have *precisely* this sort of high availability setup in their commercial offerings, with consensus based selection of a Subversion master among a set of synchronized distinct Subversion servers, and I strongly suspect it's a lot more economical to buy their product than spend your time implementing it from scratch. That's part of the problem: at 1000+ users, their licensing costs tend to become rather high. (At least, last time I checked - currently, I can't seem to find a price list.) I'll call them anyway, but doing my job properly will require me to at least look at the cost of alternatives. Cheers, Ulli Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: svn copy vs svn add in pre-commit
Hi, -Original Message- From: Nico Kadel-Garcia [mailto:nka...@gmail.com] Why do you want to do this? To assure that tags have been part of a QA release process? No - for that, we don't need to check if it's a copy. We mostly want to avoid the case with someone copying in the shell, then adding the stuff in a tag (or branch) - if you have a several GB trunk, this adds up pretty quickly... Cheers, Ulli Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: svn copy vs svn add in pre-commit
Hi, -Original Message- From: Johan Corveleyn [mailto:jcor...@gmail.com] try 'svnlook changed --copy-info -t $TXN $REPOS' The --copy-info should show things like (from trunk/:rXXX). That's exactly what I was looking for. :-) How could I have overlooked this!? Many thanks, I'll go and feel stupid now. Cheers, Ulli Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: Unexpected behaviour with SVNPath/SVNParentPath mixture
Hi, -Original Message- From: Cooke, Mark [mailto:mark.co...@siemens.com] Sent: Friday, March 25, 2011 12:38 PM To: Jans Ullrich; subversion-20...@ryandesign.com Cc: users@subversion.apache.org Subject: RE: Unexpected behaviour with SVNPath/SVNParentPath mixture The question is, why not? According to the Apache docs, it should work, so it looks like a problem in subversion. Can this be considered a bug? Should I file an issue? ...because when you use the SVNParentPath directive, apache has no further involvement in any child paths, all paths that start with the parent root are passed to the DAV handler... You can get round this by specifiying separate handlers for the separate paths using multiple SVNPath blocks but again, anything below those paths is handled by DAV and not apache. This is different from nesting access restrictions to artifacts that are all handled by apache (i.e. uri Locations or local Directorys so those docs you mention do not apply in this scenario. Thanks for the explanation. I think I get it now, just the effect with the path component suddenly doubling still puzzles me a bit. I still suspect a bug or misfeature there, since I'd have thought it should either work not at all or work as expected. But since it looks like we have a workaround, I guess I can live with that. We'd like to provide our users with the ability to create repositories themselves, then possibly promote select repositories to a different permission set. Restricting ourselves to only using SVNPath would be inconvenient... ;-) You could consider using one (set of) parent path(s) for restricted repos and another (set) for less restricted ones? I'll have to check that, but I suspect it will be hard because a lot of our build architecture is already using these paths. The modified permissions are the new part. But I think the idea is good, maybe we can have a completely different location for the SVNPath configs (like /svn/parentpathtest_ext/...) then we should have no conflict, should we? Thanks for the idea and the explanation above! Cheers, Ulli Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: Update and Status in large working copy causes Windows to hang
Hi Stefan, -Original Message- From: Stefan Sperling [mailto:s...@elego.de] Sent: Monday, November 08, 2010 6:16 PM To: Jans Ullrich Cc: jus...@honesthacker.com; andy.l...@gmail.com; markp...@gmail.com; users@subversion.apache.org Subject: Re: Update and Status in large working copy causes Windows to hang Have you tried to reproduce the issue with command line clients provided by other vendors? See http://subversion.apache.org/packages.html#windows Hadn't tried until now. Please also try with version 1.6.13, if possible. I downloaded the most recent (1.6.13 at the time) command line client from Collabnet, tested it, same problem: insufficient resources. We did some more tests, ran the update while running procmon, didn't fail. (WTF?!) After running procmon, the issue didn't reappear on the machine we ran the tests on. We're now investigating what running procmon changed on the machine. (Nothing was installed, just the exe from sysinternals was run) Cheers, Ulli Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: Update and Status in large working copy causes Windows to hang
Hi, -Original Message- From: Johan Corveleyn [mailto:jcor...@gmail.com] Sent: Friday, September 10, 2010 1:24 PM Some quick questions from the hip: - What kind of disks are the working copies located on for machine A and B? Locally or remotely? The disks are in all cases local hard drives, no sharing. - For A and B: are they 32-bit or 64-bit? Both machines are 64 Bit machines, running XP 32 Bit - How big is the working copy we're talking about? Can you give an idea of the number of directories, the number of files, and the total disk space (excluding .svn dirs)? The working copy is about 2.0 GB (according to Windows taking up 2.54 GB on disk), 158515 files, 33471 folders (excluding .svn folders). I know this is large, but since it's working on one machine and not the other, it's still puzzling. Cheers, Ulli -- Ullrich Jans, Specialist, IT-A Phone: +49 9131 7701-6627, mailto:ullrich.j...@elektrobit.com Fax: +49 9131 7701-6333, www.elektrobit.com Elektrobit Automotive GmbH, Am Wolfsmantel 46, 91058 Erlangen, Germany Managing Directors: Otto Fößel, Jarkko Sairanen Register Court Fürth HRB 4886 Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: subversion tunables
-Original Message- From: Campbell Allan [mailto:campbell.al...@sword-ciboodle.com] Sent: Monday, July 12, 2010 2:09 PM To: users@subversion.apache.org Cc: west alto Subject: Re: subversion tunables On Thursday 08 Jul 2010, west alto wrote: Hi Gurus, Any advise on initial tuning values for apache MinSpareServers, MaxSpareServers, and StartServers, tcp tunings, ulimits, sysctl I'm running svn 1.6 over apache2 pre-fork. System load goes high as much as 10 during heavy usage. How does mod_dav_svn works on checkout? is every file = 1 thread? What would be your recommended MaxRequestsPerChild? I'm serving 91 repositories and 2 of them are at least access every 10 mins with 4 concurrent users. Thanks, West I was kind of hoping to see if anyone else replied to this. I've not personally tuned apache like this as I don't believe it to be a bottleneck in my environment. Pointing a browser at apache's /server-status page will tell you how many servers are actively processing requests and if you use something like munin then that will tell you more historical information and make it much easier to spot if any changes have positive effects. Creating threads is pretty expensive so I'd expect (but don't take this as correct) that mod_dav_svn uses the apache request processing thread to perform all operations. It might be there is an extra thread created per request so that there is one for comms and one for the actual subversion request. Definately not one per file though. IO wait and authentication checks seem to be the biggest problem in my install. To reduce the number of authentication checks I disabled per directory auth checks with the 'SVNPathAuthz off' directive. This made a big difference to checkouts, our AD authentication servers were temporarily blacklisting the subversion server as they were thinking it was DOS'ing them when I was performing a test checkout! I also have a patch for mod_auth_pam to cache some authentication details and also allow setting the name it provides to pam. The production installation is running on top of apache 2.0 which seems to have a minor memory leak or is over eager with caching data. To help reduce the impact of that there is a cron entry to gracefully restart the server 3 times a day. This doesn't seem to be an issue on apache 2.2 though as I have an almost identical setup which doesn't suffer from that problem. Hi, just my $0.02: * Check what the server is doing most of the time: ** is it spending its time in IO wait? ** is it spending its time in CPU? ** is it spending its time (god forbid!) paging? I think the authentication is not a problem, that would be waiting for network, which doesn't add to the load number. Also, Apache 2.2 caches some authentication credentials. * Authorization* is something else - SVNPathAuthz can be very expensive, CPU-wise, so it is a point to disable it. * Data compression can also be a pig, so don't use Deflate in the Apache configuration. * IO wait can't usually be cured well by software, unless it's paging or swapping. You could add more memory and/or faster disks. * If the machine is swapping a bit, decrease the Maxclients setting, but be careful of setting it not too low. We've seen long waits for clients while the machine was doing almost nothing, the reason was a too low Maxclients setting, causing clients sitting idle hogging connections while other clients were waiting for a connection to become available. * If you're running in a VMWare VM, check if there are enough CPUs available often enough to handle the requests. (I read somewhere that ESX only gives a VM access to the processors if it can reserve the number of CPUs assigned to the VM, which means, on a Host with 8 cores and a bunch of VMs, a VM that has 4 cores assigned to it can only get CPU time when at leaset 4 CPUs are available and not doing anything for other VMs. Using 2 (or even just one) core for that VM might actually result in the VM becoming faster!) Try running with this: Maxclients 100 Minspareservers 10 Maxrequestsperchild 500 # this can be much higher, but we reduced it when we saw a memory leak that caused apache bloat (historically) Startservers 10 We didn't tune tcp or other system parameters much. We are serving about 900 repositories (most of them inactive) to about 1000 users, server is a IBM HS20 with 8 cores, 8GB of memory and 700 GB Fibre Channel disks (SAN, 4 GBit uplink). Load on the machine usually 1. Best regards, Ullrich Jans Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: svn move directory will create a new revision for every file inside this directory?
Hi, -Original Message- From: Leonardo Azize Martins [mailto:laz...@gmail.com] Sent: Friday, June 11, 2010 2:16 PM To: Bob Archer Cc: users@subversion.apache.org Subject: Re: svn move directory will create a new revision for every file inside this directory? Hi Bob, You have already answered my question. It does not create a new revision for files inside directory that was moved. Actually, when you check out a fresh working copy, you should see that all files are at the same revision (4, in this case). So, if I want to get FILE_A in REV 2 from path /DIR_B/DIR_A/FILE_A, I must use PEGREV 4 I'm not quite getting what you're trying to do. You should be able to tell svn that you want this file, as it was in rev. 2 - it should automatically follow the trail back and hand you the correct state. Best regards, Ullrich Jans -- Ullrich Jans, Specialist, IT-A Phone: +49 9131 7701-6627, mailto:ullrich.j...@elektrobit.com Fax: +49 9131 7701-6333, www.elektrobit.com Elektrobit Automotive GmbH, Am Wolfsmantel 46, 91058 Erlangen, Germany Managing Directors: Otto Fößel, Jarkko Sairanen Register Court Fürth HRB 4886 Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: How to speed up subversion
Hi, -Original Message- From: Jeremy Conlin [mailto:jlcon...@gmail.com] Sent: Wednesday, April 21, 2010 5:48 PM To: users@subversion.apache.org Subject: How to speed up subversion I have a working copy/respository with many files that are several hundred MB each. Whenever I try to check the status of my working copy or do a commit, it can take a long time (~1 min) before I get a response. I don't have any externals and the number of total files in my repository is ~100. Has anyone else experienced this or knows how to speed things up? I'm running svn 1.6.5 on Mac OSX 10.6.2. I guess the speed of your hard disk is the issue. Commit and status both check if the files on the local hard drive have changed since the last update. They do this by comparing the originals in the .svn dirs to the files you are working on. When you have a lot of large files, that can take quite a long time (100 MB times 100 files are 10 GB that have to be read, twice!) I hope this points you in the right direction... Cheers, Ulli Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: svnadmin create not honoring sticky bit
Hi, users-return-1912-ullrich.jans=elektrobit@subversion.apach e.org wrote: Subject: RE: svnadmin create not honoring sticky bit Stefan Sperling wrote: On Tue, Mar 30, 2010 at 02:16:50PM +0200, ullrich.j...@elektrobit.com wrote: ls -l test1/db/rep-cache.db test2/db/rep-cache.db -rw-r-+ 1 root www 4096 2010-03-30 14:07 test1/db/rep-cache.db -rw-r-+ 1 username users 4096 2010-03-30 14:07 test2/db/rep-cache.db See http://subversion.tigris.org/issues/show_bug.cgi?id=3437 I know about that one. (That's why I mentioned it in my original mail) What's new (to me) is the permissions on *all* the files in db (just one example from db to keep the email short): drwxrws---+ 3 root www 4096 2010-03-30 14:07 test1/db/revs drwxrwx---+ 3 username users 4096 2010-03-30 14:07 test2/db/revs Looking at the main directory shows that the sticky bit on db got dropped: ls -l test2 total 48 drwxrws---+ 2 username www 4096 2010-03-30 14:07 conf drwxrwx---+ 6 username www 4096 2010-03-30 14:07 db ^here -r--r-+ 1 username www2 2010-03-30 14:07 format drwxrws---+ 2 username www 4096 2010-03-30 14:07 hooks drwxrws---+ 2 username www 4096 2010-03-30 14:07 locks -rw-rw+ 1 username www 229 2010-03-30 14:07 README.txt Occurred on svn 1.6.9 (package from the Suse Buildservice): subversion-1.6.9-4.3 subversion-perl-1.6.9-4.3 subversion-tools-1.6.9-4.3 subversion-server-1.6.9-4.3 Any ideas on that one? Should I open an issue in the tracker? Cheers, Ulli Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
svnadmin create not honoring sticky bit
Hi, I encountered an unexpected behaviour during a svnadmin create as a normal user. We have a setup where a normal user can create repositories below an SvnParentPath structure. The directories are setgroupid www, with ACLs allowing the user write permissions. When I create these directories as root, the permissions are passed down properly, everything works, except for that misbehaviour with sqlite and rep-cache.db. When I create a repository as a normal user (with the proper permissions), the sticky bit doesn't get passed down to the db directory, so all files and directories in there end up owned by the user's primary group, with all traces of www removed, thus not readable. I created a test case: cd /tmp mkdir test chgrp www test chmod 2770 test setfacl -m u:username:rwx test setfacl -m d:u:username:rwx test cd test svnadmin create test1 su - username -c cd /tmp/test; svnadmin create test2 Result: ls -l total 16 drwxrws---+ 6 root www 4096 2010-03-30 14:07 test1 drwxrws---+ 6 username www 4096 2010-03-30 14:07 test2 ls -ld test1/db test2/db drwxrws---+ 6 root www 4096 2010-03-30 14:07 test1/db drwxrwx---+ 6 username www 4096 2010-03-30 14:07 test2/db ls -l test1/db/rep-cache.db test2/db/rep-cache.db -rw-r-+ 1 root www 4096 2010-03-30 14:07 test1/db/rep-cache.db -rw-r-+ 1 username users 4096 2010-03-30 14:07 test2/db/rep-cache.db ls -ld test1/db/revs test2/db/revs drwxrws---+ 3 root www 4096 2010-03-30 14:07 test1/db/revs drwxrwx---+ 3 username users 4096 2010-03-30 14:07 test2/db/revs Am I doing something wrong or am I too stupid to see the obvious? Is this possibly a bug? Best regards Ullrich Jans -- Ullrich Jans, Specialist, IT-A Phone: +49 9131 7701-6627, mailto:ullrich.j...@elektrobit.com Fax: +49 9131 7701-6333, www.elektrobit.com Elektrobit Automotive GmbH, Am Wolfsmantel 46, 91058 Erlangen, Germany Managing Directors: Otto Fößel, Jarkko Sairanen Register Court Fürth HRB 4886 Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: svnadmin create not honoring sticky bit
Stefan Sperling wrote: On Tue, Mar 30, 2010 at 02:16:50PM +0200, ullrich.j...@elektrobit.com wrote: ls -l test1/db/rep-cache.db test2/db/rep-cache.db -rw-r-+ 1 root www 4096 2010-03-30 14:07 test1/db/rep-cache.db -rw-r-+ 1 username users 4096 2010-03-30 14:07 test2/db/rep-cache.db See http://subversion.tigris.org/issues/show_bug.cgi?id=3437 I know about that one. (That's why I mentioned it in my original mail) What's new (to me) is the permissions on *all* the files in db (just one example from db to keep the email short): drwxrws---+ 3 root www 4096 2010-03-30 14:07 test1/db/revs drwxrwx---+ 3 username users 4096 2010-03-30 14:07 test2/db/revs Looking at the main directory shows that the sticky bit on db got dropped: ls -l test2 total 48 drwxrws---+ 2 username www 4096 2010-03-30 14:07 conf drwxrwx---+ 6 username www 4096 2010-03-30 14:07 db ^here -r--r-+ 1 username www2 2010-03-30 14:07 format drwxrws---+ 2 username www 4096 2010-03-30 14:07 hooks drwxrws---+ 2 username www 4096 2010-03-30 14:07 locks -rw-rw+ 1 username www 229 2010-03-30 14:07 README.txt Occurred on svn 1.6.9 (package from the Suse Buildservice): subversion-1.6.9-4.3 subversion-perl-1.6.9-4.3 subversion-tools-1.6.9-4.3 subversion-server-1.6.9-4.3 Cheers, Ulli -- Ullrich Jans, Specialist, IT-A Phone: +49 9131 7701-6627, mailto:ullrich.j...@elektrobit.com Fax: +49 9131 7701-6333, www.elektrobit.com Elektrobit Automotive GmbH, Am Wolfsmantel 46, 91058 Erlangen, Germany Managing Directors: Otto Fößel, Jarkko Sairanen Register Court Fürth HRB 4886 Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: Subversion queries hanging, timing out
Hi, (sorry for the format - OutBarf again..) From: Dave Purrington [mailto:dave.purring...@gmail.com] Sent: Monday, January 11, 2010 8:25 PM To: users@subversion.apache.org Subject: Subversion queries hanging, timing out Hello, Lately we have been experiencing intermittent timeouts with our Subversion operations. It does not happen initially, but after a while it starts happening.Restarting Apache alleviates the problem, but it comes back after a time. As you can imagine, this wreaks havoc. [...] depending on how many users you have, you might want to check the max_clients setting in Apache - I was bitten by this recently. The default is 150 (AFAIK) and that's too small. Increase it somewhat (like 500), reload Apache. This might help - in my case, it did. I think the reason behind that is clients keeping the connection to the server alive and the server running out of slots. Cheers, Ulli Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
RE: Copy data from one repos to another and history should not be losed
Ryan Schmidt wrote: On Jan 5, 2010, at 06:02, Chetan Chatwani wrote: We have two repositories names as : bmrepos and bmdevrepos. We want to copy/add data from bmrepos to bmdevrepos but history should not losed. Note: bpmrepos already have some contents. Yes, you can do this with svnadmin dump and svnadmin load. But watch out: the dates in the revisions will not be linear afterwards, so searching for revisions by date will not work any more. (AFAIK) Cheers, Ulli -- Ullrich Jans, Application Support, IM Phone: +49 9131 7701-6627, mailto:ullrich.j...@elektrobit.com Fax: +49 9131 7701-6333, www.elektrobit.com Elektrobit Automotive GmbH, Am Wolfsmantel 46, 91058 Erlangen, Germany Managing Directors: Otto Fößel, Jarkko Sairanen Register Court Fürth HRB 4886 Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
