Re: upgrading server
On Tue, Jul 25, 2017 at 9:00 PM, Mark Phippard wrote: > On Tue, Jul 25, 2017 at 2:00 PM, Andy So wrote: >> >> We have an old subversion version 1.4.3 (r23084) running on Solaris. >> >> We would like to upgrade to use new hardware on Linux based OS (CentOS >> 6.9), possibly version 1.8.x or 1.9.x >> >> >> >> Our plan is to installed and configure the latest SVN on CentOS 6.9. Then >> go through dump and load of the repository as described in various online >> post and documentations. The repository is quite large…guessing the size to >> be in the order of 20-40GB >> >> >> >> Before we start undertaking such tasks >> >> 1. Does anyone know if there are there any problem/gotcha in >> migrating the repository? >> >> 2. Does anyone know how long it would take to export the repository >> of this size? This will give us an estimate how long to schedule down time >> and cut off time. >> >> >> >> Thanks for any insight. > > > > Dump and load is a good idea because it lets the repository be rewritten > using the newer code and repository format which will give you a smaller > repository that will run a little faster. That said you do not HAVE to > dump/load. You have other options: > > 1. Just move the repository folder to the new server. Perhaps using tar and > then moving the archive. > > 2. Instead of using dump/load, use svnsync. This gives all the benefits of > the dump/load but allows you to shrink your downtime to almost nothing. > Just svnsync the repository to the new server. This will probably take a > long time, but it does not matter since the original server can be running > while it happens. At the time of your choosing, do a final svnsync, and > then shutdown the old server and use the new one. > > 3. Do option 1 now and then do a dump/load or svnsync at some future time > that is more convenient for downtime. It will probably run faster since it > is on new and better hardware too. > > There are some gotchas no matter what: > > 1. Does the new server have a new hostname or do you intend to update DNS > to point to new server? If you are not doing the latter, then all of your > existing working copies and scripts have to be modified for the new server. > Also any use of svn:externals property has to be modified. > > 2. With an old repository there is a good chance you will run into bugs in > your data that cause svnsync or load to fail. There are workaround for > different failures but be prepared to run into them and account for finding > the solutions. SVN 1.8 and 1.9 have added various options to let you > workaround some of the known bugs. A common problem is having svn: > properties that are not UTF-8 encoded or do not have LF line endings. > svnsync has this option to work around the encoding problem: > > --source-prop-encoding ARG : convert translatable properties from encoding > ARG > to UTF-8. If not specified, then properties are > presumed to be encoded in UTF-8. > > And it automatically fixes the LF problems. > > svnadmin load does not have any options to fix the problems, but you can add > the --bypass-prop-validation option to ignore them and just carry the > problems into your new repository. Indeed, dump/load is a bit more limited than svnsync regarding normalization. Also, be careful: both dump/load and svnsync do not migrate your hook scripts, config files and locks (server side locks of the svn:needs-lock type). I've just extended the FAQ entry about dump/load with some of the known caveats, and explained a procedure to do dump/load with minimal downtime. See: http://subversion.apache.org/faq.html#dumpload -- Johan
Re: Questions on migrating SVN (and Trac) to a Google Compute Engine instance
On Sat, Jul 29, 2017 at 1:16 AM, James H. H. Lampert wrote: > On 7/27/17, 11:15 PM, Ryan Schmidt wrote: >> >> Sounds plausible. An empty pre-revprop-change hook script would allow >> any revprop change, which you may not want. It's probably possible to >> write a more-specific script that would allow only the changes >> svnsync needs and disallow others. > > . . . >> >> svnsync is probably best, since it allows you to easily incrementally >> mirror a live read/write repository to another server. It can be slow >> but once it's done it makes it very quick to switch from the old >> server to the new one with minimal downtime. Some of the other >> methods require you to make the source repository read-only or take >> it offline for the duration of the migration, which could take hours >> or days depending on how large your repository is. > > . . . > and Arwin and Nico said similar things. > > Thanks, Ryan, Arwin, and Nico. > > It took a bit of futzing around, but as I type this, I'm replicating a > repository (the smallest and currently least-active one). > > It took me a while to realize that the hooks with .tmpl extensions were > templates, not live hooks, and I was right on the verge of asking for help > when I looked at the instructions again, and realized that instead of > setting up the required null hook, I'd overwritten a template (DUH!) Indeed, you only need to copy the files under $REPOS/hooks which are not ending in .tmpl :-). And for the custom hook scripts you have: maybe review them and compare the comments section at the beginning with the new corresonding template to see if there are new features (perhaps copy over the new comment block to your custom hook script). Two other things which are also not transferred by svnsync (nor by dump/load): - config files (under $REPOS/conf), for instance authz - locks (server-side locks, of the svn:needs-lock type): these should be copied from $OLDREPOS/db/locks to $NEWREPOS/db/locks FWIW, I've documented some of these things in an extended answer in our FAQ, about dump/load (some of these things also apply to svnsync): http://subversion.apache.org/faq.html#dumpload For the record, you can also perform an incemental dump/load with minimal downtime (I explained this procedure in the FAQ). But svnsync is still a lot easier, because of the automatic normalization of properties (which is not done automatically by 'svnadmin load' at the moment, which will error out on non-LF line endings in svn:log messages and things like that, possibly giving a lot of headaches ... see FAQ). -- Johan
RE: Checksum mismatch bug in 1.8.18
This specific error message was added in the last release, so yes it is new in the last versions. The last 1.9.x version also added it and I'm surprised that you see the error on 1.8 and not 1.9 (or vice versa). It tries to tell you that you have two files with an identical SHA-1 hash, but different contents. A case that we accidentally didn't catch before, that might have caused some data loss under extreme circumstances. This scenario is pretty unlikely to trigger, unless you specifically try to do that. I'll try to verify what exactly is in your dump file to see if you really hit this scenario, or that we have a bug in our code that tries to catch this scenario. Assuming you properly verified with 1.9.6 (and not 1.9.5 or earlier) I assume that this is a bug in 1.8.18, as other scenarios are really extremely unlikely. I'm not sure I'll have time to look into this before Monday though, so perhaps one of the other developers beats me to it. Bert From: David Engel [mailto:den...@magnitude.com] Sent: zaterdag 29 juli 2017 00:01 To: users@subversion.apache.org Subject: Checksum mismatch bug in 1.8.18 Hi, I think I'm running into a bug in svnadmin in the latest 1.8 release (1.8.18). It is not present in 1.8.17 nor 1.9.5 nor 1.9.6. I'm guessing it's related to the recent changes around strict repsharing. I'm not subscribed to the mailing list, so please CC me on any responses. Here are the relevant details: OS: Windows 10 and Windows 2012 R2 Subversion release: 1.8.18 Both SlikSVN release: svnadmin, version 1.8.18-SlikSvn-1.8.18-X64 (SlikSvn/1.8.18) X64 compiled Jul 17 2017, 13:03:37 on x86_64-microsoft-windows6.2.9200 And Collabnet release: svnadmin, version 1.8.18 (r1800620) compiled Jul 7 2017, 05:51:59 on x86_64-microsoft-windows6.2.9200 Performing an svnadmin load results in the following error: ... <<< Started new transaction, based on original revision 650 * editing path : Dev/Common/Utils/External/StampVerData/MetaBuilderVersion.inf ... done. * editing path : Dev/Common/Utils/External/StampVerData/MetaBuilderVersionDBG.inf ... done. * editing path : Dev/Common/Utils/External/StampVerData/OracleVersion.inf ...svnadmin: E16: SHA1 of reps '-1 0 45 132 a6ea37d29094deece56250ebe167ce46 6a2d6bc4c7e58c0d2627a1b0dd5b23e542aec4d1 650-i2/_8' and '-1 136 45 132 a6ea37d29094deece56250ebe167ce46 6a2d6bc4c7e58c0d2627a1b0dd5b23e542aec4d1 650-i2/_8' matches (6a2d6bc4c7e58c0d2627a1b0dd5b23e542aec4d1) but contents differ svnadmin: E160004: Filesystem is corrupt svnadmin: E200014: Checksum mismatch while reading representation: expected: a6ea37d29094deece56250ebe167ce46 actual: 5f696f5d0755f3bcb5e927bdfba4bfa8 In order to reproduce the error, you can use the attached VersionStamps3 repository dump file. It was taken with the same version of svnadmin from a multi-GB repository that's been around for over a decade and pruned (svndumpfilter) to a small set of files and revisions in one folder while still encountering the error. I first encountered the error another place very quickly in an incremental load on top of a repo that was loaded from 1.8.14, but that one was on a revision north of 120,000+. This is way easier to reproduce. I tried filtering the dump further and dropping empty revisions, but the error did not occur in those cases. The error also did not occur using 1.8.17, 1.9.5., and 1.9.6. It also did not occur in 1.8.18 if I set enable-rep-sharing = false before performing the load. Repro.bat script: @echo off :defineCommands rem You might need to adjust these lines to point to your rem compiled-from-source Subversion binaries, if using those: for %%i in (svn.exe) do set SVN="%%~$PATH:i" for %%i in (svnadmin.exe) do set SVNADMIN="%%~$PATH:i" :defineUrls rem Only supported access method: file:// . If http:// or svn://, then rem you'll have to configure it yourself first. set URL=file:///%CD%/repos set URL=%URL:\=/% echo Base url for repo: %URL% :cleanAllDirsAndCreateRepo if exist repos rmdir /s /q repos if exist import-me rmdir /s /q import-me if exist wc rmdir /s /q wc %SVNADMIN% create repos pause :prepareGreekTree echo Making a Tree for import... mkdir import-me\Dev\Common\Utils\External echo Importing it... cd import-me %SVN% import -q -m "Initial import." %URL% cd .. rem This is where your reproduction recipe goes. svnadmin --version svnadmin load repos < VersionStamps3 goto :eof Thanks, David
Re: Questions on migrating SVN (and Trac) to a Google Compute Engine instance
Johan Corveleyn wrote on Sat, 29 Jul 2017 11:39 +0200: > Two other things which are also not transferred by svnsync (nor by dump/load): > - locks (server-side locks, of the svn:needs-lock type): these should > be copied from $OLDREPOS/db/locks to $NEWREPOS/db/locks It would be best to do this under 'svnadmin freeze'. Currently, the failure mode of a non-atomic copy is harmless, but we don't promise it'll always be this way.