Hi, one of our large repositories went corrupt. Maybe someone can help. On Aug 6th we had a Hardware crash of our svn server (mod_dav_svn) / svn 1.7.11.
This caused erros like: [Fri Aug 08 14:00:01 2014] [error] [client ip.ip.ip.ip] (20014)Internal error: Couldn't open rep-cache database [Fri Aug 08 14:00:01 2014] [error] [client ip.ip.ip.ip] (20014)Internal error: -Couldn't perform atomic initialization [Fri Aug 08 14:00:01 2014] [error] [client ip.ip.ip.ip] (20014)Internal error: -database disk image is malformed, executing statement 'PRAGMA synchronous=OFF;PRAGMA recursive_triggers=ON;' This led me to: https://mail-archives.apache.org/mod_mbox/subversion-users/201401.mbox/%3c52d83701.5030...@reser.org%3E As Restoring a dump of the prod-repo would Take about >=8 hours I found a workaround: Taking a light older backup, restoring it on another place and replacing the corrupt rep-cache.db on the production repo and restarting apache httpd... It seems to have worked. None of the error above appears anymore and the rep-cache.db file grows. :) ---------------- Sadly there is a new error where I don't know if the previous issue is linked to this - that's why I explained it. :) Now the httpd-log presents Errors at Checkout like: [Wed Aug 13 17:32:27 2014] [error] [client ip.ip.ip.ip] Unable to deliver content. [500, #0] [Wed Aug 13 17:32:27 2014] [error] [client ip.ip.ip.ip] could not prepare to read the file [500, #160004] [Wed Aug 13 17:32:27 2014] [error] [client ip.ip.ip.ip] Corrupt representation '76625 15736 24 1736 98aa6ec0f63f0cf7bc71f84cb15decf5 106c014975275e25ea0a7efab549472a8e32917e 77054-1ogo/_4x' [500, #160004] [Wed Aug 13 17:32:27 2014] [error] [client ip.ip.ip.ip] Malformed representation header at /srv/svn/repo1/db/revs/76/76625:15751 [500, #160004] Scanning the last revisions throws: > for i in $(seq 77000 $(svnlook youngest /srv/svn/repo1)); do svnadmin verify > /srv/svn/repo1 -r $i; done [...] * Verified revision 77014. svnadmin: E160004: Corrupt representation '75477 1973 1105 7151 a6e0d04828d8d5683c69faf7289c62e0 96f59510929937634de7c7fea27a005db91a31c9 77014-1ofa/_g' svnadmin: E160004: Malformed representation header at /srv/svn/repo1/db/revs/75/75477:1982 * Verified revision 77016. ... * Verified revision 77047. svnadmin: E160004: Corrupt representation '76625 48 29 7011 c2fbbb12f89423213f919c53b926fa82 fc3aba63eaa5672fe664fd47ae5f6db780b9552c 77047-1ogg/_a' svnadmin: E160004: Malformed representation header at /srv/svn/repo1/db/revs/76/76625:52 * Verified revision 77049. [...] * Verified revision 77054. svnadmin: E160004: Corrupt representation '76625 48 29 7011 c2fbbb12f89423213f919c53b926fa82 fc3aba63eaa5672fe664fd47ae5f6db780b9552c 77054-1ogo/_4w' svnadmin: E160004: Malformed representation header at /srv/svn/repo1/db/revs/76/76625:52 * Verified revision 77056. [...] So revisions: 77015, 77048 and 77055 are corrupt - others are okay - so just commiting seems not to be the problem and thus I think it's not linked to the first one. All corrupt commits come from the same user account (authzn_svn_module) I am going to check his environment. Researching I found: https://mail-archives.apache.org/mod_mbox/subversion-dev/201010.mbox/%3C1286360504.2313.132.camel@edith%3E trying the script sadly throws: ./fixer/fix-rev.py /srv/svn/repo1 77015 Traceback (most recent call last): File "./fixer/fix-rev.py", line 237, in <module> fix_rev(repo_dir, rev) File "./fixer/fix-rev.py", line 222, in fix_rev while fix_one_error(repo_dir, rev): File "./fixer/fix-rev.py", line 182, in fix_one_error if handle_one_error(repo_dir, rev, svnadmin_err): File "./fixer/fix-rev.py", line 161, in handle_one_error fix_delta_ref(repo_dir, rev, bad_rev, bad_offset, bad_size) File "./fixer/fix-rev.py", line 111, in fix_delta_ref good_offset = find_good_rep_header(repo_dir, bad_rev, bad_size) File "/root/scripts/fixer/find_good_id.py", line 72, in find_good_rep_header _, texts = rev_file_indexes(repo_dir, rev) File "/root/scripts/fixer/find_good_id.py", line 41, in rev_file_indexes for line in open(rev_file_path(repo_dir, rev)): IOError: [Errno 2] No such file or directory: '/srv/svn/repo1/db/revs/75477 doing a symlink throws following: ]# ln -s /srv/svn/repo1/db/revs/75/75477 /srv/svn/repo1/db/revs/75477 ]# ./fixer/fix-rev.py /srv/svn/repo1 77015 Traceback (most recent call last): File "./fixer/fix-rev.py", line 237, in <module> fix_rev(repo_dir, rev) File "./fixer/fix-rev.py", line 222, in fix_rev while fix_one_error(repo_dir, rev): File "./fixer/fix-rev.py", line 182, in fix_one_error if handle_one_error(repo_dir, rev, svnadmin_err): File "./fixer/fix-rev.py", line 161, in handle_one_error fix_delta_ref(repo_dir, rev, bad_rev, bad_offset, bad_size) File "./fixer/fix-rev.py", line 111, in fix_delta_ref good_offset = find_good_rep_header(repo_dir, bad_rev, bad_size) File "/root/scripts/fixer/find_good_id.py", line 72, in find_good_rep_header _, texts = rev_file_indexes(repo_dir, rev) File "/root/scripts/fixer/find_good_id.py", line 44, in rev_file_indexes id_noderev, id_rev, _ = parse_id(id) File "/root/scripts/fixer/find_good_id.py", line 27, in parse_id _, rev = noderev.split('.r') ValueError: too many values to unpack So script seems to be too old :| I also tried ( from http://www.szakmeister.net/fsfsverify/ ): python fsfsverify.py /srv/svn/repo1/db/revs/75/75477 but the output tells a lot of "Can't check <cryptic string>" like: "b-4028.q-73874.r73874/36007" and it does not seem to get fixed, too. (svnadmin verify throws same above) Now I am at the edge of my competence. Hopefully someone of you can help!? Do you have an Idea how to fix that? As far as I know the commiting guys use a lot of git-svn - if this is a needful information... Thank you very much. Felix --- T-Systems Multimedia Solutions GmbH CU BT Felix Herzog Centralized Applicationmanagement Address: Riesaer Straße 5, 01129 Dresden, Germany Postal address: Postfach 10 02 24, 01072 Dresden, Germany +49 351 2820-2593 (Phone) +49 351 2820-5111 (Fax) +49 151 1483-1647 (Mobile) E-Mail: E-Mail: felix.her...@t-systems.com Internet: http://www.t-systems-mms.com You can find the compulsory statement on: www.t-systems-mms.com/en/compulsory-statement
