Hi Dana From: Dana Epp Sent: Saturday, 17 August 2013 4:47 AM We have an old SVN database corruption we need help with. (SVN 1.1 with BDB 4.2) Does anyone know of someone who consults on this sort of thing?
We've been dealing with SVN 1.2.3 for a long time, and have developed a process for dealing with corrupted databases. Below is a copy from our (internal) Wiki page. I hope it helps you and others. Regards, Geoff Recovering Corrupted Repositories Simple Corruption (Log File) Very occasionally, a file is committed to a repository which causes the repository to be corrupted. This usually manifests itself as access to the repository freezing. See the extensive notes in Mantis Issue 7585 . Here are the steps required to recover: Warn everybody that SubVersion is about to go down! Remember that some repositories may not be corrupt. Locate a drive with sufficient spare space for the affected repositories Log onto the SubVersion server. This requires administrative access, which is limited to a very small number of people. Stop the Apache service Backup the affected repositories Remove the log files which don't have the db version as 0x0A in the 16th byte of the offending log file. For example, log.0000000002030 had a db version of 0x0b. Run "db_recover" from within the db directory of the repository. This requires the utility to be in the path (C:\Program Files\Sleepycat Software\Berkeley DB 4.3.29\bin) Run "svnadmin recover .\" from within the repo directory (i.e. one above the db dir) Re-start the Apache service Inform everybody that SubVersion is back up. Note Sometimes db_recover does not work. In this case, the "-c" flag helps for "critical errors" Serious Corruption (Database) From http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=1617554 "Nicholas Walker" <NWalker at dialogue-marketing dot com> Full name "Nicholas Walker" <NWalker at dialogue-marketing dot com> Date 2009-04-09 13:31:59 PDT Message I went through a week of trying to resolve various issues with an old out of control corrupted v 1.4 Berkeley Db subversion repository. These steps will not work for an fsfs repository. I put together a process for how I resolved my issues. I hope it helps somebody else out. Note: This process does not solve corruption, it just puts the repository into a state where svnadmin dump will not fail. The steps below will result in specific histories being lost. If you have the option to go back to recent backup to resolve the issue that would be a much better strategy. But when the repository has not been maintained for ears as in my case there is no other way. You'll need to use the Berkeley db utilities that match the version of Berkeley db that is being used in your repository in my case those utilities were found in "/usr/local/bin/db42". The path , and executables on your machine may not match mine. Copy repository to a temporary location. You can use svnadmin hotcopy REPOS_PATH NEW_REPOS_PATH Resolve pure database corruption This resolves error similar to DB_PAGE_NOT_FOUND error. There are files in [repository Path]/db Changes Copies Nodes Representations Revisions Strings Transactions uuids Each of these represent a database table Run "/usr/local/bin/db42/db_dump -r File > File.dump" against each of the files. This dumps the database table to a dump file, and removes instances where corruption has occurred. Note that this will remove corrupted file commits from the database table, but they were irrecoverable anyway so it doesn't matter. If you have a large repository the dump and reload process make take a considerable amount of time, especially for the strings table. Run "rm [File]" for each of the files. Delete the file not the .dump file Run "/usr/local/bin/db42/db_load File < File.dump" for each of the files. This is wonderful in theory, but I have found that the version of db_load we currently have does not like the format of the file output by db_dump on Windows: db_dump puts CR/LF on the end of each line, while db_load does not like this. To correct this: Open the file in Notepad++ View the file in Hex (Alt-Ctrl-Shift-H) Highlight the first 0d 0a. Select Search->Replace (Ctrl-H) Enter "0d" (that's number 0, not letter O) in the "Replace with" field Click on "Replace All" Save the file. Since the process takes a long time, you may want to make a backup of your backup in case you mess up in the next steps, you have a half way point to revert to, since step 2 can take a long time. Resolve repository corruption These errors occur when a transaction for example references a string/commit that no longer exists because you removed it due to database corruption in step 2, a checksum failure, or some other type of error. Start a dump of the database If no errors occur during the dump, you are done, and have resolved all corruption When you experience an error, you'll have to manually update the database table files to resolve the corruption. Here are steps for several types of errors that I came across. Checksum Failure Error along the lines of checksum mismatch on rep "[rep]" expected: checksum actual: checksum There is some reason why the checksum failed because some kind of bad commit. We are going to trick svn into thinking that the checksum has passed. Run "/usr/local/bin/db42/db_dump -kp representations > representations.dump" The "-kp" option dumps the file into a ascii format, so you can open it in a text editor Open representations.dump in a text file Search for the representation id. What is in rep in the error message You will see lines like the below 11ti ((fulltext 1 1 (md5 16 \e3\035\f0\19\c5V\07}\92h\ab\80\aam\02)) 1 0) You want to find the line where the rep id matches on the entire line. You may find the rep id in other lines within the file, you should ignore these. You only want to look at the section where the entire line is equal to the rep id in the error message. Change the section "md5 16 \e3\035\f0\19\c5V\07}\92h\ab\80\aam\02" to "md5 \00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00". This is 16 serious of strings with "\00". This checksum matches everything. Run rm representations Run "/usr/local/bin/db42/db_load representations < representations.dump" Re-run the dump to verify your changes corrected the problem. Only dump the single revision "svnadmin dump -r revision -incremental" Note that the revision that failed is the last revision noted in the output to the dump command +1. The dump outputs the revision when it completes. Continue with the dump until another error occurs Representation not found error Note the rep id. Run "/usr/local/bin/db42/db_dump -kp representations > representations.dump" The "-kp" option dumps the file into a ascii format, so you can open it in a text editor Open representations.dump in a text file Near the top of the file find a representation that is "fulltext" Copy the line of text Insert a representation into the text file that looks like the one you just copies except replace the copied rep id with the rep id in the error message Run rm representations Run "/usr/local/bin/db42/db_load representations < representations.dump" Re-run the dump to verify your changes corrected the problem. Only dump the single revision "svnadmin dump -r revision -incremental" Continue with the dump until another error occurs String not found error This one is a pain. A corrupted string may cause any depending revisions on it to also fail. Open representations.dump in a text file Search for the string id in the text file. You will find the string in the second line / the data text line. If you find it in the id line then you found a representation that has the same id as the string. You should find the string in the last few characters in the data line. "((fulltext 3 2bn (md5 16 \d2\cci\d7\b8\b7\c8\14\b6;St\a0\18\14\b9)) 4 19y4)" in this example 19y4 is the string id Copy the data line from one of the first few lines of the file that is a fulltext. Remove the data line that shows the string id that was not found, and replace it with the one that was copied. Run "rm representations" Run "/usr/local/bin/db42/db_load representations < representations.dump" Re-run the dump to verify your changes corrected the problem. Only dump the single revision "svnadmin dump -r revision -incremental" Continue with the dump until another error occurs Error similar to svndiff failed inconsistency in content length This is an error that results from the fix in the step above Follow the same steps as in the above step to resolve this issue. A difference is you will be looking for rep id now instead of the string id The data line will surely be one that is a delta Error similar to proplist skeleton malformated Part of the dump file will have been generated. Open the dump file, and go to the last few lines. The last file noted is the file that we want to note. Only way I could find to resolve this is to remove all instances of the file from the changes table. If you found the exact revision instances that was corrupted, you might be able to salvage most of the history of the file, but I had a hard time finding the correct one, so just ended up having to wipe it out completely. Run "/usr/local/bin/db42/db_dump -kp changes > changes.dump" The -kp option dumps the file into an ascii format, so you can open it in a text editor Open changes.dump in a text file Search for all instances of the file name, and remove them. Note the id line that is above the line that reads the file name must also be removed. Run "rm changes" Run "/usr/local/bin/db42/db_load changes < changes.dump" Re-run the dump to verify your changes corrected the problem. Only dump the single revision "svnadmin dump -r revision -incremental" Continue with the dump until another error occurs Error similar to file or directory [File name] could not be found Follow the same steps as in "v" above Run the entire dump, and restore it to a new repository Restore any files that had to be removed during the process of fixing the corruption You will lose the history associated with the file that had to be removed, but you can re-import the last version of the file from the older repository. Do a svn checkout of both the old, and new repositories, and do a merge between the two to try and ensure that the most recent version of the files are maintained Run svnadmin verify on a regular basis, and identify the issues before they get out of control. - The contents of this email, and any attachments, are strictly private and confidential. - It may contain legally privileged or sensitive information and is intended solely for the individual or entity to which it is addressed. - Only the intended recipient may review, reproduce, retransmit, disclose, disseminate or otherwise use or take action in reliance upon the information contained in this email and any attachments, with the permission of Australian Arrow Pty. Ltd. - If you have received this communication in error, please reply to the sender immediately and promptly delete the email and attachments, together with any copies, from all computers. - It is your responsibility to scan this communication and any attached files for computer viruses and other defects and we recommend that it be subjected to your virus checking procedures prior to use. - Australian Arrow Pty. Ltd. does not accept liability for any loss or damage of any nature, howsoever caused, which may result directly or indirectly from this communication or any attached files.