subject:"RE\: svnadmin"

Dmitry Minsky wrote on Tue, 28 Jun 2022 13:18 +00:00:
>> What does the "folder with files" contain?
>
> Just a random files on my computer ;) It’s not from working copy or 
> repository or anything else meaningful. Let’s assume that it’s just a 
> bunch of random files which I want to put in the middle of repo and 
> hope that it won’t blow up ;) Is that possible?

And to answer my own question about a simpler solution:

1. Create a copy of the repository up to r7448.

2. Create an empty r7449.

3. Copy over the remaining revisions, discarding (if dump/load: with
svndumpfilter; if svnsync: by using authz to make certain paths
unreadable) all changes to the files that r7449 touched.

This is lossy, but a lot easier.

Actually, you could just do this experiment and in step 2 manually
commit one of the candidate files to the path it was originally
committed in r7449 to, to see whether that candidate is the right one;
but then you'd have to load/sync all revisions between r7450 and the
next revision that touches the file-modified-in-r7449 in order to see
that the candidate isn't the right one.

The important thing is that once you've fabricated an r7449 you can dump
(--deltas --incremental) or svnsync r7450:HEAD on top of it… assuming
all _future_ revisions are fine.

You might want to check that no other revisions are truncated, and which
revisions after r7449 modified the file touched in r7449.  ("file",
singular, per the rep-cache output.)  If no other revision touched that
file, you can just do the "copy up to r7448, commit one of the
candidates, copy r7450:HEAD" thing with whichever candidate you prefer.

Daniel

Command-line tool for applying deltas? (was: Re: svnadmin: E16004: Invalid r4422 footer. How to investigate deeper?)

Good morning dev@,

Anyone has a script that takes as input a file and an svndiff and emits
to stdout the result of applying the latter to the former?  This came up
on users@ in the context of reconstructing a truncated rev file.

I've checked tools/.

Cheers,

Daniel


Daniel Shahaf wrote on Tue, 28 Jun 2022 17:58 +00:00:
> Assuming I haven't missed any simpler solution, you'll want:
⋮
> 2. A script that takes as input a file and a delta, applies the latter
> to the former, and outputs the result.  We don't seem to have one of
> those already.  If you write one, do consider contributing it for our
> tools/ directory.

Re: svnadmin: E16004: Invalid r4422 footer. How to investigate deeper?

Dmitry Minsky wrote on Tue, 28 Jun 2022 14:44 +00:00:
> Soo, here is an output of sql request:
>
> % sqlite3 rep-cache.db '.header on' 'SELECT * FROM rep_cache WHERE 
> revision = 7449'
> hash|revision|offset|size|expanded_size
> a684c1201230ed000e8baf11fcd890efebb059db|7449|3|106064003|111204465
>

OK, so it would seem r7449 added one file and no directories.  That, or
every other added file/directory was a copy.

>
> And here is 7449 file size
>
> % ls -l revs/7/7449
> -r--r--r--. 1 apache apache 106067461 Feb  4  2021 revs/7/7449
>

So the rev file size is the sqlite3 SIZE plus 3458 bytes.  I guess those
could be the dir rep, node-rev header, and so on.

Also:

>>> '%x' % 106067461 
'6527605'
>>> '%x' % 106064003 
'6526883'

No [a-f] in either.  I guess that's just a coincidence.  The probability
of that (disregarding the high two bytes which didn't change) is
(10/16)**8) ≈ 2.3% ≈ 1/43.

>
> Now, what about "links" to 7449 in revisions after 7449. There is 
> something in 7450:
>
> % strings revs/7/7450 | tail -7
> copyroot: 0 /
> minfo-cnt: 61
> _7.0.t7449-67p add-dir false false false 
> /trunk/ProjectB/Art/Characters/Survivors/01/textures/substance_render
> _9.0.t7449-67p add-file true true false 
> /trunk/ProjectB/Art/Characters/Survivors/01/textures/substance_render/01_render.spp

Those are node-rev id's from r7450's transaction.  It's using in-transaction
id's as opposed to final in-revision id's, but I get that on a new test
repo too, which suggests this is an unrelated issue and that nothing
depends on these two values.  Which is to say, "move along, nothing to
see here".

> L2P-INDEX
> P2L-INDEX
> 51995736 59c66b6d95365e6bdb4be4ec3b2d3a34 51995799 
> 72059006b7c456b03efb7f07e0557795S
>
>
> % strings revs/7/7450 | grep 7449
> text: 7450 3 51992815 54791207 b326aa3b7fd0ea02b8e75ac8a8dcc656 
> 1430895ca8250cfb117997d6ee543e7e2c06c265 7449-67p/_b
> props: 2 757 65 53 113136892f2137aa0116093a524ade0b - 7449-67p/_d

That's what the structure file terms "uniquifier".  I don't recall its
semantics off the top of my head.

> DELTA 7449 11 138
> pred: 4-7052.0.r7449/12
> pred: 3-6161.0.r7449/14

Yeah, these matter.  The former is a non-self DELTA rep, i.e., a file
stored as a delta against another file; the latter indicates that a 
node-revision
("a revision of something in the repository") is a newer revision of an
existing "something in the repository" (as opposed to a historyless
add).  When regenerating r7449's rev file you'll want to make sure both
of these pointers remain valid.

The pred: links are easier since you can probably just recommit r7449 to
a copy-up-to-r7448 of the repository and then change them.  Make sure
not to break offsets later in the file.

The delta bases will require more work; see below.

«svnfsfs load-index» might be helpful in regenerating the rev file.
I haven't tried it.  (Or you could use linear addressing for the
restore, if regenerating a linear-addressing file is easier.)

> DELTA 7449 15 24
> pred: 2-6160.0.r7449/16
> DELTA 7449 17 20
> pred: 1-6132.0.r7449/18
> DELTA 7449 19 25
> pred: 3-232.0.r7449/20
> DELTA 7449 21 25
> pred: 0.0.r7449/2

Ditto.

> _7.0.t7449-67p add-dir false false false 
> /trunk/ProjectB/Art/Characters/Survivors/01/textures/substance_render
> _9.0.t7449-67p add-file true true false 
> /trunk/ProjectB/Art/Characters/Survivors/01/textures/substance_render/01_render.spp
>
>
> And there is no "links" to 7449 in 7451 revision and after it, BUT I 
> still can't dump these revisions. Maybe because of "chain" of "links". 
> Like 7449 <- 7450 <- 7451 etc.?
>

That plus the fact that you didn't pass --deltas or --incremental so it
tried to dump the entire contents of ^/@r7450 (what «svn co ^/@7450»
would get, as opposed to «svn diff -c 7450»).

> % svnadmin dump /var/repo_serpico -r7450 > ~/sdb/test.dump
> svnadmin: E160004: Corrupt representation '7449 21 25 159 
> 24ad3bd9d7945c1c7ca3f5e714ea868e - -'
> svnadmin: E160004: Invalid r7449 footer
>

OK, so the next step is to reconstruct bases of the five non-self
DELTAs in r7450.

First, look in the truncated r7449 rev file.  There might be intact reps
in it.  A rep always ends with "ENDREP\n".  (Nothing prevents "ENDREP\n"
from occurring inside the rep itself; parsing a rep requires knowing its
length in advance.)

Second, try the "random files" you mentioned upthread.

Once you have all these candidate files — the reps extracted from the
truncated rev file and the "random files" — try applying each of the
deltas in r7450 to each of the candidate files, and figure out which
combinations produce the md5/sha1 checksums recorded in r7450.

Presumably directory deltification is enabled, meaning those five deltas
comprise one file content delta (based on rep-cache.db) and four
directory deltas — one for each directory level between the modified
file and the repository root — which can be regenerated by hand.
(This is delicate in case the svndiff — meaning the contents of the
DELTA — has "copy" instructions

Re: svnadmin: E16004: Invalid r4422 footer. How to investigate deeper?

2022-06-28 Thread Dmitry Minsky

Soo, here is an output of sql request:

% sqlite3 rep-cache.db '.header on' 'SELECT * FROM rep_cache WHERE revision = 
7449'
hash|revision|offset|size|expanded_size
a684c1201230ed000e8baf11fcd890efebb059db|7449|3|106064003|111204465


And here is 7449 file size

% ls -l revs/7/7449
-r--r--r--. 1 apache apache 106067461 Feb  4  2021 revs/7/7449


Now, what about "links" to 7449 in revisions after 7449. There is something in 
7450:

% strings revs/7/7450 | tail -7
copyroot: 0 /
minfo-cnt: 61
_7.0.t7449-67p add-dir false false false 
/trunk/ProjectB/Art/Characters/Survivors/01/textures/substance_render
_9.0.t7449-67p add-file true true false 
/trunk/ProjectB/Art/Characters/Survivors/01/textures/substance_render/01_render.spp
L2P-INDEX
P2L-INDEX
51995736 59c66b6d95365e6bdb4be4ec3b2d3a34 51995799 
72059006b7c456b03efb7f07e0557795S


% strings revs/7/7450 | grep 7449
text: 7450 3 51992815 54791207 b326aa3b7fd0ea02b8e75ac8a8dcc656 
1430895ca8250cfb117997d6ee543e7e2c06c265 7449-67p/_b
props: 2 757 65 53 113136892f2137aa0116093a524ade0b - 7449-67p/_d
DELTA 7449 11 138
pred: 4-7052.0.r7449/12
pred: 3-6161.0.r7449/14
DELTA 7449 15 24
pred: 2-6160.0.r7449/16
DELTA 7449 17 20
pred: 1-6132.0.r7449/18
DELTA 7449 19 25
pred: 3-232.0.r7449/20
DELTA 7449 21 25
pred: 0.0.r7449/2
_7.0.t7449-67p add-dir false false false 
/trunk/ProjectB/Art/Characters/Survivors/01/textures/substance_render
_9.0.t7449-67p add-file true true false 
/trunk/ProjectB/Art/Characters/Survivors/01/textures/substance_render/01_render.spp


And there is no "links" to 7449 in 7451 revision and after it, BUT I still 
can't dump these revisions. Maybe because of "chain" of "links". Like 7449 <- 
7450 <- 7451 etc.?

% svnadmin dump /var/repo_serpico -r7450 > ~/sdb/test.dump
svnadmin: E160004: Corrupt representation '7449 21 25 159 
24ad3bd9d7945c1c7ca3f5e714ea868e - -'
svnadmin: E160004: Invalid r7449 footer

% svnadmin dump /var/repo_serpico -r7451 > ~/sdb/test.dump
svnadmin: E160004: Corrupt representation '7449 21 25 159 
8f3d18747d3388ff2b35096cafbd57ab - -'
svnadmin: E160004: Invalid r7449 footer


--
Dmitry Minsky

> On 28.06.2022, at 15:50, Daniel Shahaf  wrote:
> 
> Dmitry Minsky wrote on Tue, 28 Jun 2022 13:18 +00:00:
>>> What does the "folder with files" contain?
>> 
>> Just a random files on my computer ;) It’s not from working copy or 
>> repository or anything else meaningful. Let’s assume that it’s just a 
>> bunch of random files which I want to put in the middle of repo and 
>> hope that it won’t blow up ;) Is that possible?
> 
> With enough effort, yes.
> 
> Devs: In attempting to recreate db/revs/7/7449, what needs to be
> matched? Off the top of my head, it's rep-cache.db references, actual
> rep-sharing references in future rev files, and possibly node-rev id's.
> Anything else?
> 
> What's the output of «sqlite3 rep-cache.db '.header on' 'SELECT * FROM
> rep_cache WHERE revision = 7449'»?
> 
> Does any rev file after 7449 contain " 7449 " on a "text:" or
> "props:" line?
> 
> Does any rev file after 7449 contain ".7449/"?
> 
> Daniel
> 
>>> On 28.06.2022, at 15:14, Daniel Shahaf  wrote:
>>> 
>>> Dmitry Minsky wrote on Tue, 28 Jun 2022 11:01 +00:00:
 Ok. I’m pretty sure that db/revs/7/7449 is just truncated. Since there 
 aren’t any signs of any text readable data at the bottom of the file 
 and the top of file looks similar to 7448, 7450 and to any other 
 revision. 
 
 So, let’s say I’m 85.23% sure about content of this particular 
 revision. How can I recreate revision from folder with files? This rev 
 contains only add-dir and add-file changes. 
>>> 
>>> What does the "folder with files" contain?
>>> 
>>> Is it a working copy?  A repository?  An export?  None of the above?
>>> 
>>> Does it contain exactly the files and directories added in r7449 *as
>>> they were in that revision*, and nothing else?

Re: svnadmin: E16004: Invalid r4422 footer. How to investigate deeper?

Dmitry Minsky wrote on Tue, 28 Jun 2022 13:18 +00:00:
>> What does the "folder with files" contain?
>
> Just a random files on my computer ;) It’s not from working copy or 
> repository or anything else meaningful. Let’s assume that it’s just a 
> bunch of random files which I want to put in the middle of repo and 
> hope that it won’t blow up ;) Is that possible?

With enough effort, yes.

Devs: In attempting to recreate db/revs/7/7449, what needs to be
matched? Off the top of my head, it's rep-cache.db references, actual
rep-sharing references in future rev files, and possibly node-rev id's.
Anything else?

What's the output of «sqlite3 rep-cache.db '.header on' 'SELECT * FROM
rep_cache WHERE revision = 7449'»?

Does any rev file after 7449 contain " 7449 " on a "text:" or
"props:" line?

Does any rev file after 7449 contain ".7449/"?

Daniel

>> On 28.06.2022, at 15:14, Daniel Shahaf  wrote:
>> 
>> Dmitry Minsky wrote on Tue, 28 Jun 2022 11:01 +00:00:
>>> Ok. I’m pretty sure that db/revs/7/7449 is just truncated. Since there 
>>> aren’t any signs of any text readable data at the bottom of the file 
>>> and the top of file looks similar to 7448, 7450 and to any other 
>>> revision. 
>>> 
>>> So, let’s say I’m 85.23% sure about content of this particular 
>>> revision. How can I recreate revision from folder with files? This rev 
>>> contains only add-dir and add-file changes. 
>> 
>> What does the "folder with files" contain?
>> 
>> Is it a working copy?  A repository?  An export?  None of the above?
>> 
>> Does it contain exactly the files and directories added in r7449 *as
>> they were in that revision*, and nothing else?

Re: svnadmin: E16004: Invalid r4422 footer. How to investigate deeper?

2022-06-28 Thread Dmitry Minsky

> What does the "folder with files" contain?

Just a random files on my computer ;) It’s not from working copy or repository 
or anything else meaningful. Let’s assume that it’s just a bunch of random 
files which I want to put in the middle of repo and hope that it won’t blow up 
;) Is that possible?

--
Dmitry Minsky

> On 28.06.2022, at 15:14, Daniel Shahaf  wrote:
> 
> Dmitry Minsky wrote on Tue, 28 Jun 2022 11:01 +00:00:
>> Ok. I’m pretty sure that db/revs/7/7449 is just truncated. Since there 
>> aren’t any signs of any text readable data at the bottom of the file 
>> and the top of file looks similar to 7448, 7450 and to any other 
>> revision. 
>> 
>> So, let’s say I’m 85.23% sure about content of this particular 
>> revision. How can I recreate revision from folder with files? This rev 
>> contains only add-dir and add-file changes. 
> 
> What does the "folder with files" contain?
> 
> Is it a working copy?  A repository?  An export?  None of the above?
> 
> Does it contain exactly the files and directories added in r7449 *as
> they were in that revision*, and nothing else?

Re: svnadmin: E16004: Invalid r4422 footer. How to investigate deeper?

Dmitry Minsky wrote on Tue, 28 Jun 2022 11:01 +00:00:
> Ok. I’m pretty sure that db/revs/7/7449 is just truncated. Since there 
> aren’t any signs of any text readable data at the bottom of the file 
> and the top of file looks similar to 7448, 7450 and to any other 
> revision. 
>
> So, let’s say I’m 85.23% sure about content of this particular 
> revision. How can I recreate revision from folder with files? This rev 
> contains only add-dir and add-file changes. 

What does the "folder with files" contain?

Is it a working copy?  A repository?  An export?  None of the above?

Does it contain exactly the files and directories added in r7449 *as
they were in that revision*, and nothing else?

Re: svnadmin: E16004: Invalid r4422 footer. How to investigate deeper?

2022-06-28 Thread Dmitry Minsky

Ok. I’m pretty sure that db/revs/7/7449 is just truncated. Since there aren’t 
any signs of any text readable data at the bottom of the file and the top of 
file looks similar to 7448, 7450 and to any other revision. 

So, let’s say I’m 85.23% sure about content of this particular revision. How 
can I recreate revision from folder with files? This rev contains only add-dir 
and add-file changes. 

--
Dmitry Minsky

> On 21.06.2022, at 09:54, Daniel Shahaf  wrote:
> 
> Dmitry Minsky wrote on Sun, 19 Jun 2022 11:53 +00:00:
>> There is something in revisions before and after corrupted one:
>> 
>> % strings repo/db/revs/7/7448 | tail -1
>> 2244140591 fa0c1a8229575b0ce27ef0c5a8b898b4 2244140730 
>> 7d861109493094a15013c7ea105e33a1W
>> 
>> % strings repo/db/revs/7/7450 | tail -1
>> 51995736 59c66b6d95365e6bdb4be4ec3b2d3a34 51995799 
>> 72059006b7c456b03efb7f07e0557795S
>> 
>> But on corrupted (actual number of corrupted revision is 7449) it doesn't
>> give anything meaningful
>> 
>> % strings repo/db/revs/7/7449 | tail -1
>> Y$Q8
>> 
> 
> OK, so the last line isn't there at all.
> 
> What you now need to do is figure out what happened to the revision
> file.  For starters, check whether the change-list portion (a list of
> lines with "add-file" "modify-dir" and such on them, immediately
> before «L2P-INDEX») is still there.
> 
> - If that's the case, the file was likely truncated, but the lost
>  portion of the file can likely be reconstructed manually, though
>  perhaps with some effort (what's the size of db/revs/7/7449 in bytes
>  and the number of lines `svn log -qvr 7449`'s output should have?).
> 
> - If it got truncated earlier, then it's likely that actual deltas
>  (PLAIN..ENDREP or DELTA..ENDREP) got truncated, i.e., data loss has
>  occurred, so you'd have to reconstruct the revision manually or
>  proceed without the lost data.
> 
> - And if it's not merely truncated but otherwise corrupted, well,
>  it depends.
> 
> Do you have any way to reconstruct the revision or its contents?  Older
> backups, mirrors, commit mails, a working copy that committed this
> revision or updated to it and never got 'svn cleanup' run on it?
> (The thinking being to exploit issue #4071; cf.
>  >.)
> 
> Reminder that revision files contain binary data and should not be
> opened by editors that auto-fix whitespace and so on.  And that
> repositories should be backed up before being manually operated on.
> 
>> % < repo/db/revs/7/7449 xxd -s Y$Q8 -l 9
>> : 4445 4c54 410a 5356 4e   DELTA.SVN
>> 
> 
> That's likely because xxd(1) calls atoi(3) on its argv[2], which
> (assuming the shell parameter «Q8» is unset) has the value «Y», so the
> function call returns 0.
> 
> The output appears sane, though.  These bytes are exactly what the
> start of a revision file might look like.
> 
> Cheers,
> 
> Daniel
> 
> P.S.  For any future FSFS hackers out there, note that it's likely r7449
>  added a file or a directory.  (Why?  Because gung bhgchg fubjf
>  n /frys-pbzcerffrq/ qrygn.)
> 
>> On Sun, Jun 19, 2022 at 12:19 PM Daniel Shahaf 
>> wrote:
>> 
>>> Dmitry Minsky wrote on Sat, 18 Jun 2022 17:16 +00:00:
 I have a pretty old repository and now going to move it to another
>>> machine.
 When I start the dump process I stumbled upon this error in one of the
>>> old
 revisions:
 
 svnadmin: E16004: Invalid r4422 footer
 
>>> 
>>> It's actually E160004.  (Just saying this so search engines will find this
>>> thread.)
>>> 
 I now workaround with a skip over corrupted revision and continue
 --incremental dump. But like every 10-50 revisions I still get this error
 because I suppose there are files that depend on something from this old
 corrupted revision. The question is, how and where to look in this
>>> revision
 so I could manually fix the error by changing some files or checksum or
 anything?
>>> 
>>> At the very end of the file, like the following but on db/revs/4/4422:
>>> 
>>> % strings db/revs/0/1 | tail -1
>>> 417 b4657c89ff644471b6760fd6389d253c 445 ea755737e485eeb03c0012e5d6bc1b49I
>>> % < r/db/revs/0/1 xxd -s 417 -l 9
>>> 01a1: 4c32 502d 494e 4445 58   L2P-INDEX
>>> % < r/db/revs/0/1 xxd -s 445 -l 9
>>> 01bd: 5032 4c2d 494e 4445 58   P2L-INDEX
>>> %
>>> 
>>> (The first command might pick up some of the P2L/L2P data too?  I don't
>>> remember whether it's guaranteed that there'll be a non-printable
>>> character between that and the last line.)

Re: svnadmin: E16004: Invalid r4422 footer. How to investigate deeper?

2022-06-21 Thread Daniel Shahaf

Dmitry Minsky wrote on Sun, 19 Jun 2022 11:53 +00:00:
> There is something in revisions before and after corrupted one:
>
> % strings repo/db/revs/7/7448 | tail -1
> 2244140591 fa0c1a8229575b0ce27ef0c5a8b898b4 2244140730 
> 7d861109493094a15013c7ea105e33a1W
>
> % strings repo/db/revs/7/7450 | tail -1
> 51995736 59c66b6d95365e6bdb4be4ec3b2d3a34 51995799 
> 72059006b7c456b03efb7f07e0557795S
>
> But on corrupted (actual number of corrupted revision is 7449) it doesn't
> give anything meaningful
>
> % strings repo/db/revs/7/7449 | tail -1
> Y$Q8
>

OK, so the last line isn't there at all.

What you now need to do is figure out what happened to the revision
file.  For starters, check whether the change-list portion (a list of
lines with "add-file" "modify-dir" and such on them, immediately
before «L2P-INDEX») is still there.

- If that's the case, the file was likely truncated, but the lost
  portion of the file can likely be reconstructed manually, though
  perhaps with some effort (what's the size of db/revs/7/7449 in bytes
  and the number of lines `svn log -qvr 7449`'s output should have?).

- If it got truncated earlier, then it's likely that actual deltas
  (PLAIN..ENDREP or DELTA..ENDREP) got truncated, i.e., data loss has
  occurred, so you'd have to reconstruct the revision manually or
  proceed without the lost data.

- And if it's not merely truncated but otherwise corrupted, well,
  it depends.

Do you have any way to reconstruct the revision or its contents?  Older
backups, mirrors, commit mails, a working copy that committed this
revision or updated to it and never got 'svn cleanup' run on it?
(The thinking being to exploit issue #4071; cf.
.)

Reminder that revision files contain binary data and should not be
opened by editors that auto-fix whitespace and so on.  And that
repositories should be backed up before being manually operated on.

> % < repo/db/revs/7/7449 xxd -s Y$Q8 -l 9
> : 4445 4c54 410a 5356 4e   DELTA.SVN
>

That's likely because xxd(1) calls atoi(3) on its argv[2], which
(assuming the shell parameter «Q8» is unset) has the value «Y», so the
function call returns 0.

The output appears sane, though.  These bytes are exactly what the
start of a revision file might look like.

Cheers,

Daniel

P.S.  For any future FSFS hackers out there, note that it's likely r7449
  added a file or a directory.  (Why?  Because gung bhgchg fubjf
  n /frys-pbzcerffrq/ qrygn.)

> On Sun, Jun 19, 2022 at 12:19 PM Daniel Shahaf 
> wrote:
>
>> Dmitry Minsky wrote on Sat, 18 Jun 2022 17:16 +00:00:
>> > I have a pretty old repository and now going to move it to another
>> machine.
>> > When I start the dump process I stumbled upon this error in one of the
>> old
>> > revisions:
>> >
>> > svnadmin: E16004: Invalid r4422 footer
>> >
>>
>> It's actually E160004.  (Just saying this so search engines will find this
>> thread.)
>>
>> > I now workaround with a skip over corrupted revision and continue
>> > --incremental dump. But like every 10-50 revisions I still get this error
>> > because I suppose there are files that depend on something from this old
>> > corrupted revision. The question is, how and where to look in this
>> revision
>> > so I could manually fix the error by changing some files or checksum or
>> > anything?
>>
>> At the very end of the file, like the following but on db/revs/4/4422:
>>
>> % strings db/revs/0/1 | tail -1
>> 417 b4657c89ff644471b6760fd6389d253c 445 ea755737e485eeb03c0012e5d6bc1b49I
>> % < r/db/revs/0/1 xxd -s 417 -l 9
>> 01a1: 4c32 502d 494e 4445 58   L2P-INDEX
>> % < r/db/revs/0/1 xxd -s 445 -l 9
>> 01bd: 5032 4c2d 494e 4445 58   P2L-INDEX
>> %
>>
>> (The first command might pick up some of the P2L/L2P data too?  I don't
>> remember whether it's guaranteed that there'll be a non-printable
>> character between that and the last line.)
>>
>>

Re: svnadmin: E16004: Invalid r4422 footer. How to investigate deeper?

2022-06-19 Thread Dmitry Minsky

There is something in revisions before and after corrupted one:

% strings repo/db/revs/7/7448 | tail -1
2244140591 fa0c1a8229575b0ce27ef0c5a8b898b4 2244140730
7d861109493094a15013c7ea105e33a1W

% strings repo/db/revs/7/7450 | tail -1
51995736 59c66b6d95365e6bdb4be4ec3b2d3a34 51995799
72059006b7c456b03efb7f07e0557795S

But on corrupted (actual number of corrupted revision is 7449) it doesn't
give anything meaningful

% strings repo/db/revs/7/7449 | tail -1
Y$Q8

% < repo/db/revs/7/7449 xxd -s Y$Q8 -l 9
: 4445 4c54 410a 5356 4e   DELTA.SVN

On Sun, Jun 19, 2022 at 12:19 PM Daniel Shahaf 
wrote:

> Dmitry Minsky wrote on Sat, 18 Jun 2022 17:16 +00:00:
> > I have a pretty old repository and now going to move it to another
> machine.
> > When I start the dump process I stumbled upon this error in one of the
> old
> > revisions:
> >
> > svnadmin: E16004: Invalid r4422 footer
> >
>
> It's actually E160004.  (Just saying this so search engines will find this
> thread.)
>
> > I now workaround with a skip over corrupted revision and continue
> > --incremental dump. But like every 10-50 revisions I still get this error
> > because I suppose there are files that depend on something from this old
> > corrupted revision. The question is, how and where to look in this
> revision
> > so I could manually fix the error by changing some files or checksum or
> > anything?
>
> At the very end of the file, like the following but on db/revs/4/4422:
>
> % strings db/revs/0/1 | tail -1
> 417 b4657c89ff644471b6760fd6389d253c 445 ea755737e485eeb03c0012e5d6bc1b49I
> % < r/db/revs/0/1 xxd -s 417 -l 9
> 01a1: 4c32 502d 494e 4445 58   L2P-INDEX
> % < r/db/revs/0/1 xxd -s 445 -l 9
> 01bd: 5032 4c2d 494e 4445 58   P2L-INDEX
> %
>
> (The first command might pick up some of the P2L/L2P data too?  I don't
> remember whether it's guaranteed that there'll be a non-printable
> character between that and the last line.)
>
>

Re: svnadmin: E16004: Invalid r4422 footer. How to investigate deeper?

2022-06-19 Thread Daniel Shahaf

Dmitry Minsky wrote on Sat, 18 Jun 2022 17:16 +00:00:
> I have a pretty old repository and now going to move it to another machine.
> When I start the dump process I stumbled upon this error in one of the old
> revisions:
>
> svnadmin: E16004: Invalid r4422 footer
>

It's actually E160004.  (Just saying this so search engines will find this 
thread.)

> I now workaround with a skip over corrupted revision and continue
> --incremental dump. But like every 10-50 revisions I still get this error
> because I suppose there are files that depend on something from this old
> corrupted revision. The question is, how and where to look in this revision
> so I could manually fix the error by changing some files or checksum or
> anything?

At the very end of the file, like the following but on db/revs/4/4422:

% strings db/revs/0/1 | tail -1
417 b4657c89ff644471b6760fd6389d253c 445 ea755737e485eeb03c0012e5d6bc1b49I
% < r/db/revs/0/1 xxd -s 417 -l 9
01a1: 4c32 502d 494e 4445 58   L2P-INDEX
% < r/db/revs/0/1 xxd -s 445 -l 9
01bd: 5032 4c2d 494e 4445 58   P2L-INDEX
% 

(The first command might pick up some of the P2L/L2P data too?  I don't
remember whether it's guaranteed that there'll be a non-printable
character between that and the last line.)

Re: Svnadmin verify question

2021-11-03 Thread Luke Mauldin

Does the Svnadmin verify do checksum checks?  How does it verify validity?  

> On Nov 3, 2021, at 1:44 PM, Nathan Hartman  wrote:
> 
> On Tue, Nov 2, 2021 at 5:09 PM Luke Mauldin  wrote:
>> 
>> Can anyone tell me the details of the svnadmin verify command?  Does 
>> subversion store any internal hash of each commit to verify its correctness 
>> and that it has not changed over time due to disk/memory errors or software 
>> bugs?
>> 
>> Luke Mauldin
> 
> Yes. The purpose of 'svnadmin verify' is to check for exactly those
> kinds of errors.
> 
> By default it runs through each revision one-by-one and stops/reports
> if it finds a problem. There's also a --keep-going option since SVN
> 1.9.
> 
> It is a good idea to run 'svnadmin verify' periodically on
> repositories to ensure their integrity.
> 
> It is also crucial to have a good backup process in place, i.e., keep
> multiple good up-to-date backups of repositories in different physical
> locations and make sure you can restore from them before disaster
> strikes. Obviously it's of no use to detect corruption if you don't
> have backups to restore from.
> 
> Hope this helps,
> Nathan

Re: Svnadmin verify question

2021-11-03 Thread Nathan Hartman

On Tue, Nov 2, 2021 at 5:09 PM Luke Mauldin  wrote:
>
> Can anyone tell me the details of the svnadmin verify command?  Does 
> subversion store any internal hash of each commit to verify its correctness 
> and that it has not changed over time due to disk/memory errors or software 
> bugs?
>
> Luke Mauldin

Yes. The purpose of 'svnadmin verify' is to check for exactly those
kinds of errors.

By default it runs through each revision one-by-one and stops/reports
if it finds a problem. There's also a --keep-going option since SVN
1.9.

It is a good idea to run 'svnadmin verify' periodically on
repositories to ensure their integrity.

It is also crucial to have a good backup process in place, i.e., keep
multiple good up-to-date backups of repositories in different physical
locations and make sure you can restore from them before disaster
strikes. Obviously it's of no use to detect corruption if you don't
have backups to restore from.

Hope this helps,
Nathan

Re: Svnadmin dump with include will lost the svn:log in sometime

2020-03-23 Thread Daniel Shahaf

钱海远(Nathan) wrote on Mon, 23 Mar 2020 05:56 +:
> The version is 1.10.6 , I was try to reproduce this issue.
> 
> You can make a commit like this :
> Revision 1
> A /test
> A /Test/a.c
> A /another
> A /another/b.c
> 
> Revision 2
> M /test/a.c
> M /another/b.c
> 
> Then run command : svnadmin dump svnroot include "/test"
> 
> Then it will happen.

Thanks, I can reproduce this with current trunk:

[[[
% svnadmin create r
% svnmucc -U file://$PWD/r mkdir foo mkdir bar put =(:) foo/iota put =(:) 
bar/kappa -mm
% svnadmin dump r --include=/foo | grep -B1 -A2 svn:log
* Dumped revision 0.
* Dumped revision 1.
% svnadmin dump r | grep -B1 -A2 svn:log
* Dumped revision 0.
K 7
svn:log
V 1
m
* Dumped revision 1.
% 
]]]

I agree it makes sense to keep the log message, at least as an option.

I'll reopen SVN-4850 in a minute.

As to the cause, I stand corrected: it's indeed due to authz, exactly as
you said:

[[[
(lldb) n
Process 24108 stopped
* thread #1, name = 'svnadmin', stop reason = step over
frame #0: 0x77f9ba21 
libsvn_repos-1.so.0`svn_repos_fs_revision_proplist(table_p=0x7fffdd80, 
repos=0x77deb4c0, rev=1, 
authz_read_func=(libsvn_repos-1.so.0`dump_filter_authz_func at dump.c:2010), 
authz_read_baton=0x7fffdf50, pool=0x766a7028) at fs-wrap.c:514
   511authz_read_func, 
authz_read_baton,
   512pool));
   513 
-> 514if (readability == svn_repos_revision_access_none)
   515  {
   516/* Return an empty hash. */
   517*table_p = apr_hash_make(pool);
(lldb) p readability
(svn_repos_revision_access_level_t) $0 = svn_repos_revision_access_partial
]]]

authz was set up internally by svn_repos_dump_fs4():

[[[
  2083/* We use read authz callback to implement dump filtering. If there 
is no
  2084 * read access for some node, it will be excluded from dump as well as
  2085 * references to it (e.g. copy source). */
  2086if (filter_func)
  2087  {
  2088authz_func = dump_filter_authz_func;
  2089authz_baton.filter_func = filter_func;
  2090authz_baton.filter_baton = filter_baton;
  2091  }
  2092else
  2093  {
  2094authz_func = NULL;
  2095  }
]]]

Nevertheless, we can't just make svn_repos_fs_revision_proplist()
include svn:log; that would make svnserve and mod_dav_svn expose
information that currently they do not.

I'm not sure what the easiest way to fix this is…

Cheers,

Daniel

Re: Svnadmin dump with include will lost the svn:log in sometime

2020-03-22 Thread Daniel Shahaf

Daniel Shahaf wrote on Mon, 23 Mar 2020 05:45 +:
> 钱海远(Nathan) wrote on Mon, 23 Mar 2020 05:30 +:
> > The expected behaviour is: 'svnadmin dump' should include all properties.
> > 
> > It happened in command : svnadmin dump --include .
> > 
> > I organized the calling relationship of the function:  
> 
> I can't reproduce that:
> 
> [[[
> % lldb -- svnadmin dump r --include=/foo
> ⋮
> (lldb) n
> Process 10902 stopped
> * thread #1, name = 'svnadmin', stop reason = step over
> frame #0: 0x77d9ba71 
> libsvn_repos-1.so.0`svn_repos_fs_revision_proplist(table_p=0x7fffddf8,
>  repos=0x7765b4c0, rev=0, 
> authz_read_func=(libsvn_repos-1.so.0`dump_filter_authz_func at dump.c:1997), 
> authz_read_baton=0x7fffdf70, pool=0x754b6028) at fs-wrap.c:514
>511authz_read_func, 
> authz_read_baton,
>512pool));
>513 
> -> 514if (readability == svn_repos_revision_access_none)  
>515  {
>516/* Return an empty hash. */
>517*table_p = apr_hash_make(pool);
> (lldb) p readability
> (svn_repos_revision_access_level_t) $1 = svn_repos_revision_access_full
> ]]]
> 
> The value svn_repos_revision_access_full is correct, and indicates that
> the line you patched won't be executed.
> 

It also indicates that all properties on the revision will be included
in the dumpfile.

However, note that nothing guarantees that svn:log exists for every
revision in the repository.  You can check with «svnlook propget --revprop
-r N $REPOS_PATH svn:log».

Note that «svnlook log» does NOT error on revisions that don't have
svn:log set.  (This seems to be a bug.)

Re: Svnadmin dump with include will lost the svn:log in sometime

2020-03-22 Thread Daniel Shahaf

钱海远(Nathan) wrote on Mon, 23 Mar 2020 05:30 +:
> The expected behaviour is: 'svnadmin dump' should include all properties.
> 
> It happened in command : svnadmin dump --include .
> 
> I organized the calling relationship of the function:

I can't reproduce that:

[[[
% lldb -- svnadmin dump r --include=/foo
⋮
(lldb) n
Process 10902 stopped
* thread #1, name = 'svnadmin', stop reason = step over
frame #0: 0x77d9ba71 
libsvn_repos-1.so.0`svn_repos_fs_revision_proplist(table_p=0x7fffddf8, 
repos=0x7765b4c0, rev=0, 
authz_read_func=(libsvn_repos-1.so.0`dump_filter_authz_func at dump.c:1997), 
authz_read_baton=0x7fffdf70, pool=0x754b6028) at fs-wrap.c:514
   511authz_read_func, 
authz_read_baton,
   512pool));
   513 
-> 514if (readability == svn_repos_revision_access_none)
   515  {
   516/* Return an empty hash. */
   517*table_p = apr_hash_make(pool);
(lldb) p readability
(svn_repos_revision_access_level_t) $1 = svn_repos_revision_access_full
]]]

The value svn_repos_revision_access_full is correct, and indicates that
the line you patched won't be executed.

FWIW, the backtrace at that point:

[[[
(lldb) bt
* thread #1, name = 'svnadmin', stop reason = step over
  * frame #0: 0x77d9ba71 
libsvn_repos-1.so.0`svn_repos_fs_revision_proplist(table_p=0x7fffddf8, 
repos=0x7765b4c0, rev=0, 
authz_read_func=(libsvn_repos-1.so.0`dump_filter_authz_func at dump.c:1997), 
authz_read_baton=0x7fffdf70, pool=0x754b6028) at fs-wrap.c:514
frame #1: 0x77d93633 
libsvn_repos-1.so.0`write_revision_record(stream=0x754afbe0, 
repos=0x7765b4c0, rev=0, include_revprops=1, 
authz_func=(libsvn_repos-1.so.0`dump_filter_authz_func at dump.c:1997), 
authz_baton=0x7fffdf70, pool=0x754b6028) at dump.c:1967
frame #2: 0x77d92fd2 
libsvn_repos-1.so.0`svn_repos_dump_fs4(repos=0x7765b4c0, 
stream=0x754afbe0, start_rev=0, end_rev=0, incremental=0, use_deltas=0, 
include_revprops=1, include_changes=1, 
notify_func=(svnadmin`repos_notify_handler at svnadmin.c:1055), 
notify_baton=0x754afc48, filter_func=(svnadmin`dump_filter_func at 
svnadmin.c:1402), filter_baton=0x7fffe128, 
cancel_func=(libsvn_subr-1.so.0`check_cancel at cmdline.c:1659), 
cancel_baton=0x, pool=0x7765b028) at dump.c:2116
frame #3: 0x00406892 
svnadmin`subcommand_dump(os=0x7765b188, baton=0x7fffe3d8, 
pool=0x7765b028) at svnadmin.c:1468
frame #4: 0x004052d2 
svnadmin`sub_main(exit_code=0x7fffe524, argc=4, 
argv=0x7fffe628, pool=0x7765b028) at svnadmin.c:3314
frame #5: 0x00403d02 svnadmin`main(argc=4, argv=0x7fffe628) 
at svnadmin.c:3347
frame #6: 0x7743209b 
libc.so.6`__libc_start_main(main=(svnadmin`main at svnadmin.c:), argc=4, 
argv=0x7fffe628, init=, fini=, 
rtld_fini=, stack_end=0x7fffe618) at libc-start.c:308
frame #7: 0x00403bba svnadmin`_start + 42
(lldb) 
]]]

That's with 1.13.  I don't have 1.10 handy; however, I expect the
same is true in 1.10.

> I was very sure this will make a malformed dump when it work with include.

I expect the dump will load successfully, but TIAS.

Daniel

> 
> Best Regards!
> Haiyuan Qian
> R & D Management Group
> Hangzhou Hikvision Digital Technology Co.,Ltd
> No.555 Qianmo Road, Binjiang District, Hangzhou 310052, China
> M (86)18969199712
> 
> 本邮件及其附件含有海康威视公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
> This e-mail and its attachments contain confidential information from 
> HIKVISION, which is intended only for  the person or entity whose address is 
> listed above. Any use of the information contained herein in any way 
> (including, but not limited to, total or partial disclosure, reproduction, or 
> dissemination) by persons other  than the intended recipient(s) is 
> prohibited. If you receive this e-mail in error, please notify the sender by 
> phone or email immediately and delete it!
> 
> 
> -邮件原件-
> 发件人: Daniel Shahaf 
> 发送时间: 2020年3月22日 1:04
> 收件人: 钱海远(Nathan) 
> 抄送: users@subversion.apache.org
> 主题: Re: Svnadmin dump with include will lost the svn:log in sometime
> 
> 钱海远(Nathan) wrote on Sat, 21 Mar 2020 06:08 +:
> > I found the there is a BUG in subversion 1.10.6.
> >
> > Svnadmin dump with include will lost the svn:log in sometime.
> >
> > I was fix this issue several months ago , I forgot how to the issue 
> > happened again. But I have a patch to fix this issue, the code was lost the 
> > processing for SVN_PROP_REVISION_LOG.
> >
> > If there is no svn:log i

Re: Svnadmin dump with include can not dump the subdir into add when it's parent path was a branch

2020-03-21 Thread Daniel Shahaf

[moving to dev@; please remove users@ from replies]

钱海远(Nathan) wrote on Sat, 21 Mar 2020 06:12 +:
> I found the there is a BUG in subversion 1.10.6.
> 
> Svnadmin dump with include can not dump the subdir into add when it's parent 
> path was a branch:
> 1. 1、/A was copy from/XX , revision was 50, it the first revision of /A; 2. 
> 2、There is a subdir named /A/subdir, and several files and dir under 
> /A/subdir; 3. 3、Try to run “svnadmin dump /data/repos_root  --include 
> /A/subdir >a” . The expected dump file will include /A/subdir(revision 50) 
> into add. But in fact there was nothing.
> 4. 4、I was try to fix this bug in the svnadmin.c  , a function named 
> ary_prefix_match. If the change list is the parent dir for include path, it 
> return true. Then the dump file will include /A/subdir and it's subdir or 
> subfile into add , but regrettably, the dump file will also include /A into 
> add.

钱海远(Nathan) wrote on Sat, 21 Mar 2020 06:12 +:
> --- C:/Users/QIANHA~1/AppData/Local/Temp/svnadmin.c-rev30397.svn001.tmp.c 
>   25 11:51:32 2020
> +++ C:/Users/QIANHA~1/AppData/Local/Temp/svnadmin.c-rev30398.svn000.tmp.c 
>   21 12:58:58 2020
> @@ -1297,3 +1297,3 @@ ary_prefix_match(const apr_array_header_t *pfxlist
> -  if (path_len < pfx_len)
> -continue;
> -  if (strncmp(path, pfx, pfx_len) == 0
> +  /*if (path_len < pfx_len)
> +continue;*/
> +  if ((strncmp(path, pfx, pfx_len) == 0
> @@ -1300,0 +1301,3 @@ ary_prefix_match(const apr_array_header_t *pfxlist
> +|| (strncmp(pfx,path, path_len) == 0
> +  && (path_len == 1 || path[path_len] == '\0' || path[path_len] == 
> '/'))
> +)

Thanks for the patch.  I don't see any obvious problems with this
approach.  However, the patch as it stands causes a regression:

[[[
W: /home/daniel/src/svn/t1/./subversion/libsvn_repos/load.c:667,
W: /home/daniel/src/svn/t1/./subversion/libsvn_repos/load-fs-vtable.c:718,
W: /home/daniel/src/svn/t1/./subversion/libsvn_repos/load-fs-vtable.c:591,
W: /home/daniel/src/svn/t1/./subversion/libsvn_fs/fs-loader.c:1482,
W: /home/daniel/src/svn/t1/./subversion/libsvn_fs_fs/tree.c:2524,
W: /home/daniel/src/svn/t1/./subversion/libsvn_fs_fs/tree.c:1145: 
(apr_err=SVN_ERR_FS_NOT_FOUND)
W: svnadmin: E160013: File not found: transaction '0-0', path '/A/B/F'
W: CWD: /tmp/svn/subversion/tests/cmdline
W: EXCEPTION: Failure: Command failed: "/tmp/svn/subversion/svnadmin/svnadmin 
load --quiet svn-test-work/repositories/svnadmin_tests-60-1"; exit code 1
Traceback (most recent call last):
  File "/home/daniel/src/svn/t1/subversion/tests/cmdline/svntest/main.py", line 
1931, in run
rc = self.pred.run(sandbox)
  File "/home/daniel/src/svn/t1/subversion/tests/cmdline/svntest/testcase.py", 
line 178, in run
result = self.func(sandbox)
  File "/home/daniel/src/svn/t1/subversion/tests/cmdline/svnadmin_tests.py", 
line 3510, in dump_exclude
load_and_verify_dumpstream(sbox2, None, [], None, False, dump)
  File "/home/daniel/src/svn/t1/subversion/tests/cmdline/svnadmin_tests.py", 
line 304, in load_and_verify_dumpstream
'load', '--quiet', sbox.repo_dir, *varargs)
  File "/home/daniel/src/svn/t1/subversion/tests/cmdline/svntest/main.py", line 
666, in run_command_stdin
'"; exit code ' + str(exit_code))
svntest.Failure: Command failed: "/tmp/svn/subversion/svnadmin/svnadmin load 
--quiet svn-test-work/repositories/svnadmin_tests-60-1"; exit code 1
FAIL:  svnadmin_tests.py 60: svnadmin dump with excluded paths
]]]

So, I guess the new condition should be applied to --include's but not
to --exclude's?

Cheers,

Daniel

Re: Svnadmin dump with include will lost the svn:log in sometime

2020-03-21 Thread Daniel Shahaf

钱海远(Nathan) wrote on Sat, 21 Mar 2020 06:08 +:
> I found the there is a BUG in subversion 1.10.6.
> 
> Svnadmin dump with include will lost the svn:log in sometime.
> 
> I was fix this issue several months ago , I forgot how to the issue happened 
> again. But I have a patch to fix this issue, the code was lost the processing 
> for SVN_PROP_REVISION_LOG.
> 
> If there is no svn:log in the dump file , the load will be fail.

Let's take these issues one at a time.  The expected behaviour is:

1. 'svnadmin dump' should include all properties.

2. 'svnadmin load' should succeed even when svn:log is missing.

Now, to your patch.  First of all, the patch shouldn't affect svnadmin's
behaviour in any way, because svnadmin doesn't enter the codepath your
changed: that codepath is only used when authz is in effect, and
svnadmin never does authz.  (That's how #1 is implemented.)  Did you
perchance run 'svnrdump dump' instead?  That'd explain why your patch
had an effect.

Next, omitting svn:log in the svn_repos_revision_partial_access case is
not a bug; it's deliberately this way.

If you can reproduce a case in which either #1 or #2 don't hold, that
would be a bug.  However, given that your patch had an effect, you are
probably using some tool other than svnadmin.  So, please quote the
command lines you entered and the error messages they generated.

Re: svnadmin lslocks ambiguity

2020-01-15 Thread Daniel Shahaf

Trent Fisher wrote on Wed, Jan 15, 2020 at 15:00:56 -0500:
> On 1/6/2020 1:42 PM, Daniel Shahaf wrote:
> > Trent Fisher wrote on Mon, 06 Jan 2020 18:22 +00:00:
> > > This seems like a bug in lslocks, why would it not list *all* the locks?
> > If you run fsfs, it might be 
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__subversion.apache.org_issue3750=DwIBAg=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE=ZbaTjKb7ivmk5rbWkuwu4Mqvrh_gmetmbEk2XMe0cVs=HvcvPbSptrAq78THvePuzylhdDLG4ABfI9Y5NXg15ac=c8ls12Aj8kxq6gQ4jXAltM3ipA4nFCLssJk0XndlTuU=
> >  .
> 
> So, from that, it sounds like copying a repository could result in the locks
> data structures in FSFS becoming inconsistent?  This would make sense as I
> am working with an rsync'ed copy of a repository (though the bug seems to be
> saying that even hotcopy can end up with this problem)

FSFS never blocks readers.  That's (part of) why both rsync and hotcopy can
trigger the problem.

For rsync, you could avoid the problem by running it like this:

rsync -a --delete repo/ repo.copy/
rsync -a --delete repo/ repo.copy/ 
svnadmin freeze -- repo rsync -a --delete repo/ repo.copy/

(You can run rsync any number of times before running it once with freeze.)

> Is there a way to either fix the lock database to be consistent, or just to
> clear it?

Well, I have to give you the obligatory warning never to touch anything under
$REPO_DIR/db/ [except $REPO_DIR/db/fsfs.conf] and that doing so falls under "if
you break it you get to keep both pieces".  With that in mind, if I remember 
correctly, «svnadmin freeze repo -- rm -rf repo/db/locks/*/» will wipe the locks
database cleanly.  But make sure you have backups.

The format of the locks/ directory is one svn_hash_write2() file per path that
is locked or that is an ancestor (path-wise, not revision-wise) of a locked
path, as specified in subversion/libsvn_fs_fs/structure.  Using this
information a tool could be written that repairs the locks/ directory.  I don't
know if anyone has done this exercise already.

Cheers,

Daniel

Re: svnadmin lslocks ambiguity

2020-01-15 Thread Trent Fisher


On 1/6/2020 1:42 PM, Daniel Shahaf wrote:

Trent Fisher wrote on Mon, 06 Jan 2020 18:22 +00:00:

This seems like a bug in lslocks, why would it not list *all* the locks?

If you run fsfs, it might be 
https://urldefense.proofpoint.com/v2/url?u=https-3A__subversion.apache.org_issue3750=DwIBAg=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE=ZbaTjKb7ivmk5rbWkuwu4Mqvrh_gmetmbEk2XMe0cVs=HvcvPbSptrAq78THvePuzylhdDLG4ABfI9Y5NXg15ac=c8ls12Aj8kxq6gQ4jXAltM3ipA4nFCLssJk0XndlTuU=
 .


So, from that, it sounds like copying a repository could result in the 
locks data structures in FSFS becoming inconsistent?  This would make 
sense as I am working with an rsync'ed copy of a repository (though the 
bug seems to be saying that even hotcopy can end up with this problem)


Is there a way to either fix the lock database to be consistent, or just 
to clear it?

Re: svnadmin lslocks ambiguity

2020-01-06 Thread Daniel Shahaf

Trent Fisher wrote on Mon, 06 Jan 2020 18:22 +00:00:
> This seems like a bug in lslocks, why would it not list *all* the locks? 

If you run fsfs, it might be https://subversion.apache.org/issue3750.

Re: svnadmin fails to create empty repository on CSV storage (Windows Cluster Shared Volume)

2019-08-12 Thread Daniel Shahaf

Daniel Watzinger wrote on Mon, 12 Aug 2019 10:56 +00:00:
> Please don't ask why we are using the Windows platform in the first 
> place ;-)

We weren't planning to ask.  Subversion supports any platform that
supports C89, Python, and APR and has a maintained build script.

I'm not familiar with your storage solution so I'll defer to others to
comment on that.

Good luck,

Daniel

Re: svnadmin fails to create empty repository on CSV storage (Windows Cluster Shared Volume)

2019-08-12 Thread Daniel Watzinger

Please let me know if you need any more input on this matter :-)

PS: To give a little bit of background context, we encountered this issue
while trying to set up a HA subversion cluster (svnserve) using Windows
Failover Clustering features behind a nginx reverse proxy load balancer.
The recommended and performant way to share storage between cluster nodes
is to use a CSV volume. Shared storage may of course be implemented, albeit
less performant, by using simple file shares. Please don't ask why we are
using the Windows platform in the first place ;-)

Have a nice day.

Daniel

On Tue, Jul 30, 2019 at 11:13 PM Daniel Watzinger <
daniel.watzin...@gmail.com> wrote:

> The issue at hand may be solved by changing how svn code calls APR.
> However, I strongly suspect that there is more code that relies on data and
> metadata synchronization. I cannot comment on the latter due to
> insufficient knowledge of the APR library and its guarantees on IO
> operations.
>
> On Tue, Jul 30, 2019 at 11:04 PM Daniel Shahaf 
> wrote:
>
>> Daniel Watzinger wrote on Tue, 30 Jul 2019 20:44 +00:00:
>> > I took the liberty to dive into the relevant source code. The issue may
>> > be pinned down to file metadata not being flushed to disk by the
>> > underlying APR library.
>>
>> Do you mean that svn needs to change how it calls APR, or that APR needs
>> to change how it calls the kernel?  In the latter case, I'm afraid I'll
>> have to forward you again, to .
>>
>>

Re: svnadmin fails to create empty repository on CSV storage (Windows Cluster Shared Volume)

2019-07-30 Thread Daniel Watzinger

The issue at hand may be solved by changing how svn code calls APR.
However, I strongly suspect that there is more code that relies on data and
metadata synchronization. I cannot comment on the latter due to
insufficient knowledge of the APR library and its guarantees on IO
operations.

On Tue, Jul 30, 2019 at 11:04 PM Daniel Shahaf 
wrote:

> Daniel Watzinger wrote on Tue, 30 Jul 2019 20:44 +00:00:
> > I took the liberty to dive into the relevant source code. The issue may
> > be pinned down to file metadata not being flushed to disk by the
> > underlying APR library.
>
> Do you mean that svn needs to change how it calls APR, or that APR needs
> to change how it calls the kernel?  In the latter case, I'm afraid I'll
> have to forward you again, to .
>
>

Re: svnadmin fails to create empty repository on CSV storage (Windows Cluster Shared Volume)

2019-07-30 Thread Daniel Shahaf

Daniel Watzinger wrote on Tue, 30 Jul 2019 20:44 +00:00:
> I took the liberty to dive into the relevant source code. The issue may 
> be pinned down to file metadata not being flushed to disk by the 
> underlying APR library.

Do you mean that svn needs to change how it calls APR, or that APR needs
to change how it calls the kernel?  In the latter case, I'm afraid I'll
have to forward you again, to .

Re: svnadmin fails to create empty repository on CSV storage (Windows Cluster Shared Volume)

2019-07-30 Thread Daniel Watzinger

Good evening Daniel :-)

I somehow miserably failed to include a cross-reference to the JIRA issue I
have created.

- https://issues.apache.org/jira/browse/SVN-4819

I took the liberty to dive into the relevant source code. The issue may be
pinned down to file metadata not being flushed to disk by the underlying
APR library. This is a non-issue on many (Windows) filesystems like
NTFS/FAT as a single file handles metadata (like filesize) is synchronized
by the corresponding filesystem drivers, albeit undocumented. However, the
implementation of CSV seems to differ (due to the need to synchronize file
metadata across cluster nodes).

On Tue, Jul 30, 2019 at 9:45 PM Daniel Shahaf 
wrote:

> Good morning Daniel,
>
> I don't have the brainwidth to investigate this, but two quick questions:
>
> - Does it happen on a local disk?
> - Can you post the file foobar/db/revs/0/0 as an attachment?
>
> FWIW, 6b is the correct value of the P2L offset.
>
> Cheers,
>
> Daniel
>
> Daniel Watzinger wrote on Sun, 28 Jul 2019 07:19 +00:00:
> > svnadmin reproducibly fails to initialize an empty repository on a
> > cluster shared volume in a Windows Failover Cluster environment on
> > Windows Server 2019. The initial call seems to be successful.
> >
> > svnadmin create foobar
> > A subsequent
> >
> > svnadmin verify foobar
> > fails with
> >
> > * Verifying metadata at revision 0 ...
> > * Error verifying repository metadata.
> > svnadmin: E160004: P2L offset 6b must be larger than L2P offset 6b in r0
> footer
>

Re: svnadmin fails to create empty repository on CSV storage (Windows Cluster Shared Volume)

2019-07-30 Thread Daniel Shahaf

Good morning Daniel,

I don't have the brainwidth to investigate this, but two quick questions:

- Does it happen on a local disk?
- Can you post the file foobar/db/revs/0/0 as an attachment?

FWIW, 6b is the correct value of the P2L offset.

Cheers,

Daniel

Daniel Watzinger wrote on Sun, 28 Jul 2019 07:19 +00:00:
> svnadmin reproducibly fails to initialize an empty repository on a 
> cluster shared volume in a Windows Failover Cluster environment on 
> Windows Server 2019. The initial call seems to be successful.
> 
> svnadmin create foobar
> A subsequent 
> 
> svnadmin verify foobar
> fails with
> 
> * Verifying metadata at revision 0 ...
> * Error verifying repository metadata.
> svnadmin: E160004: P2L offset 6b must be larger than L2P offset 6b in r0 
> footer

Re: Svnadmin —incremental hotcopy (and other questions)

2018-06-20 Thread Ryan Schmidt



On Jun 19, 2018, at 08:31, Tom Browder> wrote:

> On Tue, Jun 19, 2018 at 06:31 Mark Phippard wrote:
> 
>> On Jun 19, 2018, at 7:25 AM, Tom Browder wrote:
>> ...
>> Finally, now that version 1.10 is out, will the redbook be made current and 
>> turned into a printed book?
> 
> The book is open source.  It is kept somewhat current but it is a volunteer 
> effort.  I think it is extremely unlikely that O'Reilly would ever do another 
> printing.
> 
> Hm, I wonder. I believe it’s been over 10 years since the second edition, and 
> lots of changes.  Maybe worth asking O’Reilly?

The book is a separate project. You should ask them. Scroll down to 
"Feedback/Contributing" on this page:

http://svnbook.red-bean.com

Re: Svnadmin —incremental hotcopy (and other questions)

2018-06-19 Thread Tom Browder

On Tue, Jun 19, 2018 at 06:31 Mark Phippard  wrote:

>
> > On Jun 19, 2018, at 7:25 AM, Tom Browder  wrote:

...

> > Finally, now that version 1.10 is out, will the redbook be made current
> and turned into a printed book?
>
> The book is open source.  It is kept somewhat current but it is a
> volunteer effort.  I think it is extremely unlikely that O'Reilly would
> ever do another printing.

Hm, I wonder. I believe it’s been over 10 years since the second edition,
and lots of changes.  Maybe worth asking O’Reilly?

Thanks, Mark.

Best regards,

-Tom

Re: Svnadmin —incremental hotcopy (and other questions)

2018-06-19 Thread Mark Phippard

> On Jun 19, 2018, at 7:25 AM, Tom Browder  wrote:
> 
> When an incremental hotcopy runs on REPO to BACKUPREPO is BACKUPREPO then 
> always a full backup of REPO?  That is,  could I do a full dump of BACKUPREPO 
> and copy it somewhere else for safekeeping of the full, current REPO?

It is a full backup up to whatever revision it now contains.  Yes you could 
take dumps off the backup.

> 
> Another question, is there any reason an incremental hotcopy could not be 
> used in a post commit hook?

No reason.  I think it would be a solid plan.

> 
> Finally, now that version 1.10 is out, will the redbook be made current and 
> turned into a printed book?

The book is open source.  It is kept somewhat current but it is a volunteer 
effort.  I think it is extremely unlikely that O'Reilly would ever do another 
printing.

Mark

Re: svnadmin dump issue - E200015

2016-08-11 Thread Johan Corveleyn

On Wed, Aug 10, 2016 at 6:43 PM, William Muriithi
 wrote:
> Johan,
>
>>> - Is it safe to use svnadmin dump on the source repository?  As in
>>> does it do it change in the source repo?
>>
>> Yes, it's absolutely safe to run on the source repository, it should
>> not cause anything to change in the source repo.
>
> Thanks.  What about  "svnadmin verify /var/svn/projects"   ?

That's a read-only operation too (just like svnadmin dump), so should
be perfectly safe. BTW, both 'svnadmin dump' and 'svnadmin verify'
should run without problems on a busy repository (it doens't interfere
with commits and other write or read operations that happen in
parellel).

>>
>>> - What causes E200015?  I don't think its permission as I have even
>>> attempted to run svnadmin dump with root permissions? How can I
>>> overcome the issue?
>>
>> Is that the full error message you gave?
>
> Yes, that was it.  There was nothing else that it threw at standard
> error during the dump
>>
>> "Caught signal" would be what you'd get if the process were aborted by
>> a signal, like aborting it with Ctrl-C, or if some other user with
>> enough permissions sends a kill signal to your process or something
>> like that.
>>
>> Is it always aborting at the same revision?
> Yes, actually think its corruption related.  I did manage dumping the
> repo in full but have not been able to restore it.

Hm, the fact that you *were* able to dump it in full (I guess by
splitting it up in chunks, as was suggested by Nico) indicates that
it's probably not a corruption in your source repository. The fact
that it works by dumping in chunks, but not in one giant dump, points
to a resource problem for creating or storing the giant dump.

> The restore fails
> with the same error.
>
>  * editing path :
> condor/trunk/work/design/ic/library/kl_condor/kl_vco_curmir_1to20mA/starrc/layout.oa
> ... done.
> svnadmin: E200015: Caught signal

Here too I would guess you hit some system limit (RAM / swap space,
disk, maybe open file descriptors, ...). Perhaps you can split the
load process in even more steps, by specifying even smaller revision
ranges (but still using the dump files you created earlier). See
'svnadmin help load':

[[[
...
If --revision is specified, limit the loaded revisions to only those
in the dump stream whose revision numbers match the specified range.
]]]

-- 
Johan

Re: svnadmin dump issue - E200015

2016-08-11 Thread William Muriithi

Hi

> > [...] svnadmin dump for 1000 commits at a time[...]
> [...]
>
> > * Dumped revision 2062.
> > WARNING 0x: Referencing data in revision 1696, which is older than
> > the oldest dumped revision (r2001).  Loading this dump into an empty
> > repository will fail.
> >
> > * Dumped revision 3000.
> > WARNING 0x: The range of revisions dumped contained references to
> > copy sources outside that range.
> >
> > WARNING 0x0001: Mergeinfo referencing revision(s) prior to the oldest
> > dumped revision (r57001). Loading this dump may result in invalid
> > mergeinfo.
> >
> > What would this imply?
>
> Standard warning that if you dump part of the history, and that part
references earlier history, the mergeinfo may be invalid (working, but not
meaningfully so). If you load the dumped parts in the same sequence, this
is not an issue.
>
> Andreas
Ah, thanks.  Great to hear this. Kind of now not certain what is triggering
my load failure.

William

Re: svnadmin dump issue - E200015

2016-08-11 Thread Andreas Stieger


> [...] svnadmin dump for 1000 commits at a time[...]
[...]
 
> * Dumped revision 2062.
> WARNING 0x: Referencing data in revision 1696, which is older than
> the oldest dumped revision (r2001).  Loading this dump into an empty
> repository will fail.
> 
> * Dumped revision 3000.
> WARNING 0x: The range of revisions dumped contained references to
> copy sources outside that range.
> 
> WARNING 0x0001: Mergeinfo referencing revision(s) prior to the oldest
> dumped revision (r57001). Loading this dump may result in invalid
> mergeinfo.
> 
> What would this imply?

Standard warning that if you dump part of the history, and that part references 
earlier history, the mergeinfo may be invalid (working, but not meaningfully 
so). If you load the dumped parts in the same sequence, this is not an issue.

Andreas

Re: svnadmin dump issue - E200015

2016-08-10 Thread William Muriithi

Nico,

>>> I have attemped svnadmin  dump, svnadmin hotcopy and  svnrdump.  I am
>>> getting the error before from all these utilities
>
> Break it up. Use svnadmin dump for 1000 commits at a time, to help
> isolte if it's a specific commit that is killing you or if perhaps you
> are running out of resources for such a large dump
>
Thanks for this suggestion.  Its precisely what I did and was able to
dump the full repo successfully.  Well, now that I did look at the
logs, not too cleanly. I have a lot of warnings like this:

* Dumped revision 2062.
WARNING 0x: Referencing data in revision 1696, which is older than
the oldest dumped revision (r2001).  Loading this dump into an empty
repository will fail.

* Dumped revision 3000.
WARNING 0x: The range of revisions dumped contained references to
copy sources outside that range.

WARNING 0x0001: Mergeinfo referencing revision(s) prior to the oldest
dumped revision (r57001). Loading this dump may result in invalid
mergeinfo.

What would this imply?

> In particular, 113 GB is pretty bulky. It's probably a good time to
> think about trimming away some of the fat, of old branches and tags
> that are no longer needed, or even to think about splitting the repo
> up into smaller, individual projects. If that can be trimmed away to
> proper use of svndumpfilter, more power to you. This violates the
> tenet that the history, all of it, is the important thing, but I'm not
> personally a big believer in it.,
>
Thanks, was planning to break it into smaller repos, but the process
hasn't been smooth, so for now, just want a simple dump and restore to
work first.

> There are also several nasty approaches to get something working
> *now*, in parallel with your old repository. One is to do svn
> export/import of the relevant code in a new working repository: you'd
> need to reconcile the inevitable split brain after you get the full
> repository copied, but if you're out of time it can save your ass.
> (And yes, I'm bringing it up again, even though it consistently gets
> me yelled at.)
Thats an interesting advice..  I can see the pain involved due to
split brain, but since the current repo is working, I think its better
to avoid this.  But definally a great advice for firefighting
>
> There is also a nasty, but in a pinch workable, trick for getting
> similar end results for unsophisticated repositories to get code up
> and running if you're out of time. If you're not worried about
> preserving svn:keywords, svn:ignore, or other added repository
> attributes, you can get a fast clone working when svnadmin is giving
> troubling by using git.
>
> * Use "git svn" to make a "fast" copy.
> * Delete unwanted debris, such as unuseful tags and branches.
> * Use "git gc" to obliterate content that is no longer used.
> * Use "git svn" to upload the content to a new, *much smaller* working
> repository.
>
> This flushes a great deal of attribute information and potentially
> useful history, but it's often a much faster way to clear excess
> debris and get a new, non-identical, but workable Subversion
> repository with most of the important content history intact.
>
Interesting..  If I am not able to restore, I may just end up using
this.. I will leave it as last option and will have to see if
everybody can stomach it..

Thanks for sharing all this knowledge.

William

Re: svnadmin dump issue - E200015

2016-08-10 Thread William Muriithi

Johan,

>> - Is it safe to use svnadmin dump on the source repository?  As in
>> does it do it change in the source repo?
>
> Yes, it's absolutely safe to run on the source repository, it should
> not cause anything to change in the source repo.

Thanks.  What about  "svnadmin verify /var/svn/projects"   ?

>
>> - What causes E200015?  I don't think its permission as I have even
>> attempted to run svnadmin dump with root permissions? How can I
>> overcome the issue?
>
> Is that the full error message you gave?

Yes, that was it.  There was nothing else that it threw at standard
error during the dump
>
> "Caught signal" would be what you'd get if the process were aborted by
> a signal, like aborting it with Ctrl-C, or if some other user with
> enough permissions sends a kill signal to your process or something
> like that.
>
> Is it always aborting at the same revision?
Yes, actually think its corruption related.  I did manage dumping the
repo in full but have not been able to restore it.  The restore fails
with the same error.

 * editing path :
condor/trunk/work/design/ic/library/kl_condor/kl_vco_curmir_1to20mA/starrc/layout.oa
... done.
svnadmin: E200015: Caught signal
<<< Started new transaction, based on original revision 7001
 * editing path :
condor/trunk/work/design/ic/library/yt_tmp/DM10_B_DUM ...svnadmin:
E160013: File not found: transaction '6490-50b', path
'/condor/trunk/work/design/ic/library/yt_tmp/DM10_B_DUM'

Note, its attempting to restore revision 7001 but the previous
revision is 6491.  So, the file don't exist because it was created in
the future

>
> Maybe someone else is interfering with your work, and (accidently)
> killing your process? Or maybe your remote (ssh / whatever) session
> gets disconnected which causes the dump process to abort?
No, petty sure, I am the only one who have access to the new SVN
server but have seem E200015 error as seem above
>
> --
> Johan
Thanks a lot Johan

Regards,

William

Re: svnadmin dump issue - E200015

2016-08-07 Thread Nico Kadel-Garcia

On Sun, Aug 7, 2016 at 4:45 PM, Johan Corveleyn  wrote:
> On Sun, Aug 7, 2016 at 8:41 PM, William Muriithi
>  wrote:
>> Hello,
>>
>> I have a repository thats around 113 GB in size.  Its on a VM and the
>> performance hasn't been that ideal.  So we decided to source a new
>> hardware and set it up on its own dedicated system.
>>
>> The current subversion system is subversion-1.8, and plan to move it
>> to subversion-1.9.  I have en
>>
>> source server:
>> subversion-1.8.13-1.x86_64
>>
>> I have attemped svnadmin  dump, svnadmin hotcopy and  svnrdump.  I am
>> getting the error before from all these utilities

Break it up. Use svnadmin dump for 1000 commits at a time, to help
isolte if it's a specific commit that is killing you or if perhaps you
are running out of resources for such a large dump

In particular, 113 GB is pretty bulky. It's probably a good time to
think about trimming away some of the fat, of old branches and tags
that are no longer needed, or even to think about splitting the repo
up into smaller, individual projects. If that can be trimmed away to
proper use of svndumpfilter, more power to you. This violates the
tenet that the history, all of it, is the important thing, but I'm not
personally a big believer in it.,

There are also several nasty approaches to get something working
*now*, in parallel with your old repository. One is to do svn
export/import of the relevant code in a new working repository: you'd
need to reconcile the inevitable split brain after you get the full
repository copied, but if you're out of time it can save your ass.
(And yes, I'm bringing it up again, even though it consistently gets
me yelled at.)

There is also a nasty, but in a pinch workable, trick for getting
similar end results for unsophisticated repositories to get code up
and running if you're out of time. If you're not worried about
preserving svn:keywords, svn:ignore, or other added repository
attributes, you can get a fast clone working when svnadmin is giving
troubling by using git.

* Use "git svn" to make a "fast" copy.
* Delete unwanted debris, such as unuseful tags and branches.
* Use "git gc" to obliterate content that is no longer used.
* Use "git svn" to upload the content to a new, *much smaller* working
repository.

This flushes a great deal of attribute information and potentially
useful history, but it's often a much faster way to clear excess
debris and get a new, non-identical, but workable Subversion
repository with most of the important content history intact.

>> svnadmin: E200015: Caught signal
>>
>> or
>>
>> * Dumped revision 2968.
>> svnrdump: E200015: Caught signal
>>
>> My two questions are:
>>
>> - Is it safe to use svnadmin dump on the source repository?  As in
>> does it do it change in the source repo?
>
> Yes, it's absolutely safe to run on the source repository, it should
> not cause anything to change in the source repo.

The big risk I see with a 113 GB repository is running out of
resources while doing "svnadmin dump".

>> - What causes E200015?  I don't think its permission as I have even
>> attempted to run svnadmin dump with root permissions? How can I
>> overcome the issue?
>
> Is that the full error message you gave?
>
> "Caught signal" would be what you'd get if the process were aborted by
> a signal, like aborting it with Ctrl-C, or if some other user with
> enough permissions sends a kill signal to your process or something
> like that.
>
> Is it always aborting at the same revision?
>
> Maybe someone else is interfering with your work, and (accidently)
> killing your process? Or maybe your remote (ssh / whatever) session
> gets disconnected which causes the dump process to abort?

That... could also be a problem I would use "nohup" to wrap the dump
and get a usable output of both stdout and stderr.

If you're logging out in mid process and use a recent Linux, I'd also
be cautious about this:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394

Recent versions of systemd kill user processes when the original user
login is disconnected, even if some process launched by that user is
backgrounded. It kills nohup'ed jobs, jobs in "screen" and "tmux", and
it doesn't log anywhere that it killed the process.

> --
> Johan

Re: svnadmin dump issue - E200015

2016-08-07 Thread Johan Corveleyn

On Sun, Aug 7, 2016 at 8:41 PM, William Muriithi
 wrote:
> Hello,
>
> I have a repository thats around 113 GB in size.  Its on a VM and the
> performance hasn't been that ideal.  So we decided to source a new
> hardware and set it up on its own dedicated system.
>
> The current subversion system is subversion-1.8, and plan to move it
> to subversion-1.9.  I have en
>
> source server:
> subversion-1.8.13-1.x86_64
>
> I have attemped svnadmin  dump, svnadmin hotcopy and  svnrdump.  I am
> getting the error before from all these utilities
>
> svnadmin: E200015: Caught signal
>
> or
>
> * Dumped revision 2968.
> svnrdump: E200015: Caught signal
>
> My two questions are:
>
> - Is it safe to use svnadmin dump on the source repository?  As in
> does it do it change in the source repo?

Yes, it's absolutely safe to run on the source repository, it should
not cause anything to change in the source repo.

> - What causes E200015?  I don't think its permission as I have even
> attempted to run svnadmin dump with root permissions? How can I
> overcome the issue?

Is that the full error message you gave?

"Caught signal" would be what you'd get if the process were aborted by
a signal, like aborting it with Ctrl-C, or if some other user with
enough permissions sends a kill signal to your process or something
like that.

Is it always aborting at the same revision?

Maybe someone else is interfering with your work, and (accidently)
killing your process? Or maybe your remote (ssh / whatever) session
gets disconnected which causes the dump process to abort?

-- 
Johan

Re: svnadmin hotcopy never ends

2015-05-04 Thread Johan Corveleyn

On Fri, May 1, 2015 at 2:21 PM, Ignacio González (Eliop)
igtorque.el...@googlemail.com wrote:
 svnadmin 1.8.13
 Server: Centos 7 64-bit
 Repo in local xfs disk
 Backups in remote cifs share

 This operation works without problems:
 $ svnadmin hotcopy /path/to/repo /local/path/to/dest
 With a small repo, it takes just a few seconds

 But this one does not:
 $ svnadmin hotcopy /path/to/repo /remote/path/to/dest

 where /remote/path/to/dest is located on a remote cifs share.

 Before detailing the problem, let me tell you that this problem did not
 happen until I upgraded my server from Centos 5.5  Subversion 1.6.16 
 local repository over ext3 to the configuration described above. (The remote
 cifs share for backups is the same).

 Now, the symptons are these: when I launch the hotcopy operation, I can see
 that the repo is completely copied to the destination in a few seconds,
 except for 3 files. After several minutes (even hours), ps ux shows that
 svnadmin hotcopy is still active, but waiting for something that never
 happens... No error or warning message is shown.

 The three files are:
 /path/to/repo/format
 /path/to/repo/db/format
 /path/to/repo/db/fs-type

Can you tell us the contents of those three files please? That will
show us which format of the backend you're using (you've mentioned svn
1.6.16, and I'm not sure if you have 'svnadmin upgrade'd to the latest
backend format).

 It seems to me that these are the last files to be recreated in hotcopy
 operations, but I do not really know.

 The fact that the operation does succed on a local destination hints to some
 problem with permissions, different filesystems, or who knows.

 Here are the relevant parts of /etc/fstab, if they are needed:

 LABEL=repoxfs   /opt/csvn/data/repositories  xfs  defaults  1 2
 //srvr/bkps   /home/csvn/mnt/hotcopies   cifs
 username=csvn,domain=*,password=*,rw,uid=csvn,gid=csvn,dir_mode=0755,file_mode=0755
 0 0

I have seen several hotcopy-fixes mentioned for the upcoming 1.9
release [1]. Maybe, as a test, you can try if the same problem still
happens with 1.9-beta1?

[1] search for 'hotcopy' in
http://svn.apache.org/repos/asf/subversion/trunk/CHANGES

-- 
Johan

Re: svnadmin hotcopy never ends

2015-05-04 Thread Eliop

Thanks for your answer, John.

I dumped and re-loaded the repositories, so the formats are the latest.

In the end, it was a problem with the cifs mount. I had to add nobrl to
the options in /etc/fstab:

//srvr/bkps   /home/csvn/mnt/hotcopies   cifs  nobrl,username=csvn,domain=*,
password=*,rw,uid=csvn,gid=csvn,dir_mode=0755,file_mode=0755 0 0

That did the trick.

2015-05-04 9:33 GMT+02:00 Johan Corveleyn jcor...@gmail.com:

 On Fri, May 1, 2015 at 2:21 PM, Ignacio González (Eliop)
 igtorque.el...@googlemail.com wrote:
  svnadmin 1.8.13
  Server: Centos 7 64-bit
  Repo in local xfs disk
  Backups in remote cifs share
 
  This operation works without problems:
  $ svnadmin hotcopy /path/to/repo /local/path/to/dest
  With a small repo, it takes just a few seconds
 
  But this one does not:
  $ svnadmin hotcopy /path/to/repo /remote/path/to/dest
 
  where /remote/path/to/dest is located on a remote cifs share.
 
  Before detailing the problem, let me tell you that this problem did not
  happen until I upgraded my server from Centos 5.5  Subversion 1.6.16 
  local repository over ext3 to the configuration described above. (The
 remote
  cifs share for backups is the same).
 
  Now, the symptons are these: when I launch the hotcopy operation, I can
 see
  that the repo is completely copied to the destination in a few seconds,
  except for 3 files. After several minutes (even hours), ps ux shows that
  svnadmin hotcopy is still active, but waiting for something that never
  happens... No error or warning message is shown.
 
  The three files are:
  /path/to/repo/format
  /path/to/repo/db/format
  /path/to/repo/db/fs-type

 Can you tell us the contents of those three files please? That will
 show us which format of the backend you're using (you've mentioned svn
 1.6.16, and I'm not sure if you have 'svnadmin upgrade'd to the latest
 backend format).

  It seems to me that these are the last files to be recreated in hotcopy
  operations, but I do not really know.
 
  The fact that the operation does succed on a local destination hints to
 some
  problem with permissions, different filesystems, or who knows.
 
  Here are the relevant parts of /etc/fstab, if they are needed:
 
  LABEL=repoxfs   /opt/csvn/data/repositories  xfs  defaults  1 2
  //srvr/bkps   /home/csvn/mnt/hotcopies   cifs
 
 username=csvn,domain=*,password=*,rw,uid=csvn,gid=csvn,dir_mode=0755,file_mode=0755
  0 0

 I have seen several hotcopy-fixes mentioned for the upcoming 1.9
 release [1]. Maybe, as a test, you can try if the same problem still
 happens with 1.9-beta1?

 [1] search for 'hotcopy' in
 http://svn.apache.org/repos/asf/subversion/trunk/CHANGES

 --
 Johan

Re: svnadmin freeze error

2014-11-14 Thread Imad Sani

Thank you for the reply, but no, this time it says

svnadmin:invalid option: --archive

command: svnadmin freeze test rsync --archive --verbose test test1


-- 

Regards,

*Imad Sani*

Infrastructure Architect


116, Block DD (CCA)

Phase IV, D.H.A. Lahore
Tel: +92.42.37060558

*Bramerz.pk ** |  **Facebook.com/Bramerz.pk ** |  **Twitter.com/Bramerz*

*Save resources - Think before printing *

*Do not print this email unless necessary*

*DISCLAIMER*

This email including any attachment(s) contains information that is
privileged, private and confidential and is intended for the addressee(s)
only. If you receive this email in error, please notify the sender by reply
email and delete it from your system. Unauthorized use, disclosure,
distribution, printing, forwarding or copying is strictly prohibited. The
contents of this email do not necessarily represent the views or policies
of Bramerz (Pvt) Ltd. The email is scanned to be virus free however Bramerz
(Pvt) Ltd cannot accept responsibility for any loss or damage arising from
the use of this email or attachment(s).


On Fri, Nov 14, 2014 at 1:16 PM, Andreas Stieger andreas.stie...@gmx.de
wrote:

 Hi,

 On 14/11/14 07:33, Imad Sani wrote:
  svnadmin freeze /svn/my-repos rsync -av /svn/my-repos /backup/my-repos
 
  This is the error I get: svnadmin: invalid option character: a

 Not Subversion specific. Termination of option parsing for following
 arguments via --:

 svnadmin freeze REPOS_PATH PROGRAM -- ARG...

 Andreas

Re: svnadmin freeze error

2014-11-14 Thread Ryan Schmidt


On Nov 14, 2014, at 1:33 AM, Imad Sani wrote:

 Trying to use the command shown here: 
 http://subversion.apache.org/docs/release-notes/1.8.html#svnadmin-freeze
 
 command: 
 svnadmin freeze /svn/my-repos rsync -av /svn/my-repos /backup/my-repos
 
 This is the error I get: svnadmin: invalid option character: a
 
 I've verified that I'm using the latest version of subversion.

I agree, this command, shown in the Subversion 1.8 release notes, does not work 
with Subversion 1.8.10. The documentation should be updated with a working 
example.

Re: svnadmin freeze error

2014-11-14 Thread Andreas Stieger

Hello,On 14/11/14 08:28, Imad Sani wrote:
 On 14/11/14 07:33, Imad Sani wrote:
  svnadmin freeze /svn/my-repos rsync -av /svn/my-repos /backup/my-repos
 
  This is the error I get: svnadmin: invalid option character: a
 
 Not Subversion specific. Termination of option parsing for following
 arguments via --:
 
 svnadmin freeze REPOS_PATH PROGRAM -- ARG...
 Thank you for the reply, but no, this time it says
 
 svnadmin:invalid option: --archive
 
 command: svnadmin freeze test rsync --archive --verbose test test1

Again:

svnadmin freeze test -- rsync --archive --verbose
svnadmin freeze test -- rsync -av

Andreas

Re: svnadmin freeze error

2014-11-14 Thread Imad Sani

This works, thank you!

-- 

Regards,

*Imad Sani*

Infrastructure Architect


116, Block DD (CCA)

Phase IV, D.H.A. Lahore
Tel: +92.42.37060558

*Bramerz.pk ** |  **Facebook.com/Bramerz.pk ** |  **Twitter.com/Bramerz*

*Save resources - Think before printing *

*Do not print this email unless necessary*

*DISCLAIMER*

This email including any attachment(s) contains information that is
privileged, private and confidential and is intended for the addressee(s)
only. If you receive this email in error, please notify the sender by reply
email and delete it from your system. Unauthorized use, disclosure,
distribution, printing, forwarding or copying is strictly prohibited. The
contents of this email do not necessarily represent the views or policies
of Bramerz (Pvt) Ltd. The email is scanned to be virus free however Bramerz
(Pvt) Ltd cannot accept responsibility for any loss or damage arising from
the use of this email or attachment(s).


On Fri, Nov 14, 2014 at 1:33 PM, Andreas Stieger andreas.stie...@gmx.de
wrote:

 Hello,On 14/11/14 08:28, Imad Sani wrote:
  On 14/11/14 07:33, Imad Sani wrote:
   svnadmin freeze /svn/my-repos rsync -av /svn/my-repos
 /backup/my-repos
  
   This is the error I get: svnadmin: invalid option character: a
 
  Not Subversion specific. Termination of option parsing for following
  arguments via --:
 
  svnadmin freeze REPOS_PATH PROGRAM -- ARG...
  Thank you for the reply, but no, this time it says
 
  svnadmin:invalid option: --archive
 
  command: svnadmin freeze test rsync --archive --verbose test test1

 Again:

 svnadmin freeze test -- rsync --archive --verbose
 svnadmin freeze test -- rsync -av

 Andreas

Re: svnadmin: E200002: Serialized hash missing terminator — but latest show revision is perfectly ok!

2014-10-22 Thread Daniel Shahaf

Daniel Shahaf wrote on Wed, Oct 22, 2014 at 11:17:46 +:
 Stefan Sperling wrote on Mon, Oct 20, 2014 at 20:58:57 +0200:
  On Mon, Oct 20, 2014 at 08:46:03PM +0200, Stefan Sperling wrote:
   On Mon, Oct 20, 2014 at 10:22:05PM +0400, Lev Serebryakov wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 20.10.2014 19:07, Stefan Sperling wrote:

 I would recommend you reset the mirrored repository to the last
 known-good revision (269997?) by moving out of the way all files 
 belonging to newer revisions (in db/revs) and revprops (in
 db/revprops), running 'svnadmin recover' (this should set
 db/current to the number of the last known-good revision)
 I've done this three times, each time removing 100 revisions. Result
the same: before 2 revisions to end svnadmmin verify complains about
Serialized hash missing terminator.

 Now 26 is clear for sure, as it is very old revision. Iad. yes,
I've checked revprops/269/26, it is properly-formatted prop file.

 So, it is NOT problem of revprops, but it is about some other
(which?) file in repo.
   
   It's probably a corrupt entry in db/rep-cache.db which every sync
   keeps picking up again and again.
   
   Try the whole procedure again either with rep-sharing disabled
   in db/fsfs.conf (enable-rep-sharing = false), or with a copy
   of db/rep-cache.db from the master server which you should put
   in place before the 'svnadmin recover' step (because entries newer
   than the recovered HEAD revision are automatically removed from
   the rep-cache during 'svnadmin recover').
  
  Thinking about this some more, it's definitely safer to just
  disable rep-sharing on the slave. Unless you have a bit-perfect
  copy of the master repository on the slave it is probably unsafe
  to copy rep-cache.db between repositories because revision files
  on the slave might contain data in a different order, in which
  case rep-cache entries copied from the master would be invalid.
  
 
 Right, data from the master could be invalid for the slave because of
 different txn names, different zlib versions, or any number of other
 reasons.
 
  If you really want to keep the rep-cache on the svnsync slave
  you'd have to dump/load or sync again from r0 :-/
  Or fiddle with the rep-cache sqlite table directly and remove
  the bad entry (but you'd have to find that yourself).
  
  Disabling rep-caching is a perfectly safe thing to do.
  It might cause the slave to use a little more diskspace than the
  master does for future revisions, but there are no other downsides.
 
 Or you could delete rep-cache.db, that will keep rep-cache working,
 except that revisions younger than current HEAD will not participate
 in it.
 

s/younger/older/.  (That is: new revisions will do rep-sharing only
amongst themselves, and will never references anything that is 
in the rev files that already exist right now.)

Daniel

 Daniel
 (for the record: when backdating HEAD as recommended upthread, one
 should _always_ truncate or delete rep-cache.db too, to prevent
 corruption down the road from references into the undone revisions.)

Re: svnadmin: E200002: Serialized hash missing terminator — but latest show revision is perfectly ok!

2014-10-22 Thread Philip Martin

Daniel Shahaf d...@daniel.shahaf.name writes:

 (for the record: when backdating HEAD as recommended upthread, one
 should _always_ truncate or delete rep-cache.db too, to prevent
 corruption down the road from references into the undone revisions.)

In 1.8 running svnadmin recover is better as it removes any references
to undone revisions but leaves those that are still valid:

http://subversion.tigris.org/issues/show_bug.cgi?id=4077

-- 
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*

Re: svnadmin: E200002: Serialized hash missing terminator — but latest show revision is perfectly ok!

2014-10-20 Thread Stefan Sperling

On Mon, Oct 20, 2014 at 06:41:00PM +0400, Lev Serebryakov wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

I'm trying to recover SVN repository (which is mirror of other repo)
after (physical) server crash in process of svnsync.

svnadmin verify gives me:

* Verifying metadata at revision 269998 ...
svnadmin: E22: Serialized hash missing terminator
#

Please note, that 269998 is almost-round number. It is because I've
renmoved whole 270xxx range (latest one) of revisions (and their
properties) from repository, as them was synced in moment of crash. So,
all 269xxx revisions are OLD ones, which WAS NOT DAMAGED. I've checked
269998/26 revisions properties — they are OK.

Is here any way to learn WHICH Serialized hash is damaged?! Why file
name is not included in error message?

That serialised hash is inside a revprops file. You won't be able
to fix this directly unless you understand how FSFS stores data.
If you really want to know, start reading here:
https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_base/notes/structure
and then this:
https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure

I would recommend you reset the mirrored repository to the
last known-good revision (269997?) by moving out of the way all files
belonging to newer revisions (in db/revs) and revprops (in db/revprops),
running 'svnadmin recover' (this should set db/current to the number
of the last known-good revision), and finally editing the
svn:sync-last-merged-rev revision property with
'svn pe -r0 --revprop REPOS_URL'
to write the number of the last known-good revision in there.

Now you should have a working repository that is synced up to the
last known-good revision and 'svnsync sync' should catch up again.

Re: svnadmin: E200002: Serialized hash missing terminator — but latest show revision is perfectly ok!

2014-10-20 Thread Stefan Sperling

On Mon, Oct 20, 2014 at 10:22:05PM +0400, Lev Serebryakov wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA512
 
 On 20.10.2014 19:07, Stefan Sperling wrote:
 
  I would recommend you reset the mirrored repository to the last
  known-good revision (269997?) by moving out of the way all files 
  belonging to newer revisions (in db/revs) and revprops (in
  db/revprops), running 'svnadmin recover' (this should set
  db/current to the number of the last known-good revision)
  I've done this three times, each time removing 100 revisions. Result
 the same: before 2 revisions to end svnadmmin verify complains about
 Serialized hash missing terminator.
 
  Now 26 is clear for sure, as it is very old revision. Iad. yes,
 I've checked revprops/269/26, it is properly-formatted prop file.
 
  So, it is NOT problem of revprops, but it is about some other
 (which?) file in repo.

It's probably a corrupt entry in db/rep-cache.db which every sync
keeps picking up again and again.

Try the whole procedure again either with rep-sharing disabled
in db/fsfs.conf (enable-rep-sharing = false), or with a copy
of db/rep-cache.db from the master server which you should put
in place before the 'svnadmin recover' step (because entries newer
than the recovered HEAD revision are automatically removed from
the rep-cache during 'svnadmin recover').

Re: svnadmin: E200002: Serialized hash missing terminator — but latest show revision is perfectly ok!

2014-10-20 Thread Stefan Sperling

On Mon, Oct 20, 2014 at 08:46:03PM +0200, Stefan Sperling wrote:
 On Mon, Oct 20, 2014 at 10:22:05PM +0400, Lev Serebryakov wrote:
  -BEGIN PGP SIGNED MESSAGE-
  Hash: SHA512
  
  On 20.10.2014 19:07, Stefan Sperling wrote:
  
   I would recommend you reset the mirrored repository to the last
   known-good revision (269997?) by moving out of the way all files 
   belonging to newer revisions (in db/revs) and revprops (in
   db/revprops), running 'svnadmin recover' (this should set
   db/current to the number of the last known-good revision)
   I've done this three times, each time removing 100 revisions. Result
  the same: before 2 revisions to end svnadmmin verify complains about
  Serialized hash missing terminator.
  
   Now 26 is clear for sure, as it is very old revision. Iad. yes,
  I've checked revprops/269/26, it is properly-formatted prop file.
  
   So, it is NOT problem of revprops, but it is about some other
  (which?) file in repo.
 
 It's probably a corrupt entry in db/rep-cache.db which every sync
 keeps picking up again and again.
 
 Try the whole procedure again either with rep-sharing disabled
 in db/fsfs.conf (enable-rep-sharing = false), or with a copy
 of db/rep-cache.db from the master server which you should put
 in place before the 'svnadmin recover' step (because entries newer
 than the recovered HEAD revision are automatically removed from
 the rep-cache during 'svnadmin recover').

Thinking about this some more, it's definitely safer to just
disable rep-sharing on the slave. Unless you have a bit-perfect
copy of the master repository on the slave it is probably unsafe
to copy rep-cache.db between repositories because revision files
on the slave might contain data in a different order, in which
case rep-cache entries copied from the master would be invalid.

If you really want to keep the rep-cache on the svnsync slave
you'd have to dump/load or sync again from r0 :-/
Or fiddle with the rep-cache sqlite table directly and remove
the bad entry (but you'd have to find that yourself).

Disabling rep-caching is a perfectly safe thing to do.
It might cause the slave to use a little more diskspace than the
master does for future revisions, but there are no other downsides.

Re: svnadmin load fails with Insufficient system resources exist to complete the requested service

2014-09-30 Thread Andreas Stieger

Hi,

On 30/09/14 19:39, jt.mil...@l-3com.com wrote:
 Someone committed a file to the subversion repository that must be
 removed. The server is a virtualized Windows Server 2003 32-bit box with
 4GB RAM and plenty of disk space where the repositories reside. We're
 using SVN 1.4.4 (outdated, I know, not my fault, I was pulled away from
 my day job to fix this issue).
 
 I dumped the repository using svnadmin dump and filtered using
 svndumpfilter. The dump file was around 90GB, and roughly the same size
 filtered (yes, the users are misusing Subversion).
 
 The load of the filtered dump failed with Insufficient system resources
 exist to complete the requested service. 
 This happened after roughly 99% of the filtered dump was loaded
 (r9945/9991). I tried again and it failed earlier (~r7.5k).
 
 There doesn't seem to be an issue with resources... using Process
 Explorer shows svnadmin consuming about 40% of the CPU and roughly 8MB
 in working set memory during the load. During the dump and filter CPU
 usage is lower and memory usage higher, but still entirely reasonable.
 
 I have searched for this error and encountered many instances, but they
 seem to deal with source code and buffer errors. I need some way to
 remove this file and restore the repository.

You may attempt to load the dump using a current version of Apache
Subversion not affected by the issue, but into a repository compatible
with your production version, and then move/sync the FSFS. Specifically:
svnadmin create --pre-1.5-compatible foorepo
svnadmin --compatible-version 1.4.4 foorepo

With kind regards,
Andreas Stieger

RE: svnadmin load fails with Insufficient system resources exist to complete the requested service

2014-09-30 Thread JT.Miller

This worked beautifully, thank you!

Now I just have to compel management to spend the time to upgrade all
the seven year-old software packages on the server...

-Original Message-
From: Andreas Stieger [mailto:andreas.stie...@gmx.de] 
Sent: Tuesday, September 30, 2014 2:22 PM
To: Miller, JT (IE) @ SSG - PE - MT
Cc: users@subversion.apache.org
Subject: Re: svnadmin load fails with Insufficient system resources
exist to complete the requested service

Hi,

On 30/09/14 19:39, jt.mil...@l-3com.com wrote:
 Someone committed a file to the subversion repository that must be 
 removed. The server is a virtualized Windows Server 2003 32-bit box 
 with 4GB RAM and plenty of disk space where the repositories reside. 
 We're using SVN 1.4.4 (outdated, I know, not my fault, I was pulled 
 away from my day job to fix this issue).

 I dumped the repository using svnadmin dump and filtered using 
 svndumpfilter. The dump file was around 90GB, and roughly the same 
 size filtered (yes, the users are misusing Subversion).

 The load of the filtered dump failed with Insufficient system 
 resources exist to complete the requested service.
 This happened after roughly 99% of the filtered dump was loaded 
 (r9945/9991). I tried again and it failed earlier (~r7.5k).

 There doesn't seem to be an issue with resources... using Process 
 Explorer shows svnadmin consuming about 40% of the CPU and roughly 8MB

 in working set memory during the load. During the dump and filter CPU 
 usage is lower and memory usage higher, but still entirely reasonable.

 I have searched for this error and encountered many instances, but 
 they seem to deal with source code and buffer errors. I need some way 
 to remove this file and restore the repository.

You may attempt to load the dump using a current version of Apache
Subversion not affected by the issue, but into a repository compatible
with your production version, and then move/sync the FSFS. Specifically:
svnadmin create --pre-1.5-compatible foorepo svnadmin
--compatible-version 1.4.4 foorepo

With kind regards,
Andreas Stieger

RE: svnadmin pack vs hotcopy --incremental

2014-08-04 Thread Myron A. Semack

 If you don't pack and hotcopy concurrently, it's perfectly safe.
 
 If you pack and hotcopy concurrently, it's almost safe.
 There is a known problem with concurrent hotcopy and pack, the fix for which
 isn't released yet, I believe. Details here: http://svn.apache.org/r1605633

What about the --incremental option?  Any problems using it in conjunction with 
pack?  

My basic plan for the nightly housekeeping script is:

svnadmin verify %repopath%
svnadmin pack %repopath%
svnadmin hotcopy %repopath% %backupdir% --incremental

I should end up with a properly packed hotcopy at the end, right?

Sincerely,
Myron A. Semack

Re: svnadmin pack vs hotcopy --incremental

2014-08-04 Thread Stefan Sperling

On Mon, Aug 04, 2014 at 12:46:06PM -0400, Myron A. Semack wrote:
  If you don't pack and hotcopy concurrently, it's perfectly safe.
  
  If you pack and hotcopy concurrently, it's almost safe.
  There is a known problem with concurrent hotcopy and pack, the fix for which
  isn't released yet, I believe. Details here: http://svn.apache.org/r1605633
 
 What about the --incremental option?  Any problems using it in conjunction 
 with pack?  
 
 My basic plan for the nightly housekeeping script is:
 
   svnadmin verify %repopath%
   svnadmin pack %repopath%
   svnadmin hotcopy %repopath% %backupdir% --incremental
 
 I should end up with a properly packed hotcopy at the end, right?

Yes, this should work. The incremental copy notices if the
source was packed since the last incremental copy, creates
corresponding packs in the destination repository and removes
individual revision files from the destination afterwards.

Just make sure not to run the last two commands in your list
concurrently (i.e. pack the source while a hotcopy is in progress)
until the change made in http://svn.apache.org/r1605633 is available
in a released version of Subversion. With the fix, the hotcopy command
will error out if a pack is in progress. Without it, the hotcopy will
succeed but could create a destination repository which is not packed
consistently, which will cause problems.

Re: svnadmin pack vs hotcopy --incremental

2014-08-01 Thread Stefan Sperling

On Fri, Aug 01, 2014 at 03:17:06PM -0400, Myron A. Semack wrote:
 I have an FSFS repository that is backed up nightly with svnadmin hotcopy 
 --incremental.  It has just crossed the 8 revisions mark, and it has 
 never been packed.  I would like to run svnadmin pack on this repo.
 
 
 
 After I run svandmin pack, will the incremental hotcopy still work?  Will the 
 hotcopy mirror the packing in the backup copy of the repo?  Or do I need to 
 do a full hotcopy after packing the live repo?
 
 
 
 If I schedule svnadmin pack to happen at regular intervals (e.g. weekly), how 
 should I change my nightly backup strategy?
 
 
 
 I just want to make sure I do this right so I don't corrupt anything.
 
 
 
 For reference, this is a Windows server, running SVN 1.8.8, and the repo is 
 at FSFS version 5.
 
 
 
 Sincerely,
 
 Myron A. Semack
 

If you don't pack and hotcopy concurrently, it's perfectly safe.

If you pack and hotcopy concurrently, it's almost safe.
There is a known problem with concurrent hotcopy and pack, the fix for which
isn't released yet, I believe. Details here: http://svn.apache.org/r1605633

Re: svnadmin reports error E160004

2014-03-25 Thread Philip Martin

Martin Bischoff tin...@gmail.com writes:

 I then started svnadmin verify on the repository, which reported the
 following error:

 * Verified revision 2112.
 * Verified revision 2113.
 svnadmin: E160004: r2114's root node's predecessor is r2112 but should be
 r2113

 These revisions were committed in 2008 and so far, svnadmin verify never
 reported any problem (the newest revision is 14331).

http://subversion.apache.org/docs/release-notes/1.8.html#verify-issue4129

-- 
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*

Re: svnadmin reports error E160004

2014-03-25 Thread Martin Bischoff

Thanks a lot for the quick answer.

One additional question:
It seems that after the dump/create/load cycle, I have to manually copy the
hooks, authz and passwd files to the newly created repository.

Is that correct? Are there other files I have to copy?



On Tue, Mar 25, 2014 at 11:11 AM, Philip Martin
philip.mar...@wandisco.comwrote:

 Martin Bischoff tin...@gmail.com writes:

  I then started svnadmin verify on the repository, which reported the
  following error:
 
  * Verified revision 2112.
  * Verified revision 2113.
  svnadmin: E160004: r2114's root node's predecessor is r2112 but should be
  r2113
 
  These revisions were committed in 2008 and so far, svnadmin verify
 never
  reported any problem (the newest revision is 14331).

 http://subversion.apache.org/docs/release-notes/1.8.html#verify-issue4129

 --
 Philip Martin | Subversion Committer
 WANdisco // *Non-Stop Data*

Re: svnadmin reports error E160004

2014-03-25 Thread Daniel Shahaf

Martin Bischoff wrote on Tue, Mar 25, 2014 at 09:34:50 +0100:
 My questions:
 - what exactly is error E160004?

Corruption found.  It's not specific to this problem; it's used for
nearly every error from verify (except for OS-level permission problems
and the like).

Re: svnadmin reports error E160004

2014-03-25 Thread Daniel Shahaf

Martin Bischoff wrote on Tue, Mar 25, 2014 at 11:42:05 +0100:
 Thanks a lot for the quick answer.
 
 One additional question:
 It seems that after the dump/create/load cycle, I have to manually copy the
 hooks, authz and passwd files to the newly created repository.
 
 Is that correct?

Yes.

 Are there other files I have to copy?

You have to copy everything you created yourself that isn't the contents
of a commit.  For example, if your hooks use .conf files, you'd want to
copy those too.

Re: svnadmin verify 1.8.5 errors on old repositories

2014-01-10 Thread kmradke

  I've observed 3 instances of similar errors when using the 1.8.5 
svnadmin
  verify on 3 different old repositories.  (the hundreds of other repos 
on
  this server verified in 1.8.5 without problems.)
  
  svnadmin: E160004: r97's root node's predecessor is r90 but should be 
r96
  svnadmin: E160004: r502's root node's predecessor is r483 but should 
be r501
  svnadmin: E160004: r22033's root node's predecessor is r22031 but 
 should be r22032
  
  All 3 of these repositories verify without errors using 1.7.5.  All 3 
of the
  revisions are from different months in 2009, when I assume the server 
was
  running 1.6.something.  The repositories were upgraded, but not a 
full
  dump/load between major version upgrades.
 
 This is documented in the 1.8. release notes here:
 
https://subversion.apache.org/docs/release-notes/1.8.html#verify-issue4129

I now see the E160004 number is the same, but I had only seen error 
messages
with predecessor count for ... is wrong when I searched for that 
particular
scenario.  Thanks for the clarification that it is indeed the same issue!

Kevin R.

Re: svnadmin verify 1.8.5 errors on old repositories

2014-01-09 Thread Ben Reser

On 1/9/14, 2:48 PM, kmra...@rockwellcollins.com wrote:
 I've observed 3 instances of similar errors when using the 1.8.5 svnadmin
 verify on 3 different old repositories.  (the hundreds of other repos on
 this server verified in 1.8.5 without problems.)
 
 svnadmin: E160004: r97's root node's predecessor is r90 but should be r96
 svnadmin: E160004: r502's root node's predecessor is r483 but should be r501
 svnadmin: E160004: r22033's root node's predecessor is r22031 but should be 
 r22032
 
 All 3 of these repositories verify without errors using 1.7.5.  All 3 of the
 revisions are from different months in 2009, when I assume the server was
 running 1.6.something.  The repositories were upgraded, but not a full
 dump/load between major version upgrades.

This is documented in the 1.8. release notes here:
https://subversion.apache.org/docs/release-notes/1.8.html#verify-issue4129

 Also, on a slightly related note, why does the new svnadmin verify
 medatadata portion not always start at r0 or even show consecutive
 revision numbers on repositories with large numbers of revisions?

That's verifying the representation cache db.  r0 will never have any entry
since r0 is always an empty revision with no file contents (and thus no
represenations).  It will never have an entry for every revision.  There are a
couple reasons why a given revision might not have an entry:

* rep-cache wasn't enabled when a revision was committed (most notably upgrade
repos that existed before 1.6 when this feature was added and that haven't gone
through a dump/load cycle since).

* revision didn't add any new representations when it was committed.  Copies
with history (e.g. branching/tags) will use the representation from the source
for the files.  If a representation has already existed for a given file
content and is found in the rep-cache.db then that representation will be
reused for the file contents in the new revision.

RE: svnadmin hotcopy losing revprops

2013-11-08 Thread Matthews, David

 -Original Message-
 From: Philip Martin [mailto:philip.mar...@wandisco.com] 
 Sent: 08 November 2013 10:19
 To: Matthews, David
 Cc: subversion_us...@googlegroups.com
 Subject: Re: svnadmin hotcopy losing revprops

 dpm david.matth...@metoffice.gov.uk writes:

  I'm using svn 1.8.4 on RHEL 6.3 (we're in the process of trying to upgrade 
  from svn 1.6 to 1.8).
  However, the svnadmin hotcopy command doesn't appear to be working 
  correctly with the
  repositories I've copied over from our 1.6 server - most of the db/revprop 
  directories are missing.

 I can reproduce this, hotcopy fails to copy some of the revprops
 corresponding to packed revisions when the repository is 1.7 
 or earlier
 format:

 svnadmin create repo --compatible-version 1.7
 perl -i -pe s/1000/10/ repo/db/format
 for i in `seq 0 75`;do echo $i  x.x ; svnmucc -mm put x.x 
 file://`pwd`/repo/f ; done
 svnadmin pack repo
 svnadmin hotcopy repo repo2

 The bug was introduced to 1.8 by r1520723 and is still 
 present on trunk.

Thanks for your response (although it doesn't seem to have made it
to the archive yet - perhaps because I used google groups?).
It's good to know that the problem is reproducible.
I'll go back to using 1.8.3 for the moment.

Do you think this bug needs advertising more widely somehow?
Our backup procedure relies on hotcopy and, if I hadn't discovered
this during my testing, we could have ended up having no useable
backups which is a scary thought.

Thanks,
David Matthews

Re: svnadmin hotcopy losing revprops

2013-11-08 Thread Philip Martin

Matthews, David david.matth...@metoffice.gov.uk writes:

 Thanks for your response (although it doesn't seem to have made it
 to the archive yet - perhaps because I used google groups?).

Yes, my reply went to google groups and was bounced because I'm not
subscribed to the google group.

 It's good to know that the problem is reproducible.
 I'll go back to using 1.8.3 for the moment.

 Do you think this bug needs advertising more widely somehow?

I've raised issue 4448
http://subversion.tigris.org/issues/show_bug.cgi?id=4448

 Our backup procedure relies on hotcopy and, if I hadn't discovered
 this during my testing, we could have ended up having no useable
 backups which is a scary thought.

-- 
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*

Re: svnadmin upgrade output message i18n issue

2013-09-06 Thread QXO

The bug found in svn 1.8.3(r1516576) again :(

E:\svnmirrorD:\tools\svn-win32-1.8.3\bin\svnadmin.exe upgrade ZmccPrj
宸插��寰���搴瀹���?璇风�锛绾х�搴��介��瑕�涓�娈垫�堕�?..

完成升级。

E:\svnmirrorD:\tools\svn-win32-1.8.0\bin\svnadmin.exe upgrade ZmccPrj
已取得版本库锁定。
请稍候；升级版本库可能需要一段时间...

完成升级。

2013/6/19 QXO qxodr...@gmail.com

The bug fixed in svn 1.8.0,Thanks:)

2013/5/24 Philip Martin philip.mar...@wandisco.com

Dongsheng Song dongsheng.s...@gmail.com writes:

We do call bind_textdomain_codeset if it is available so we should be
getting UTF8 translations.

For non-autotools system, e.g. Windows, user may not define
HAVE_BIND_TEXTDOMAIN_CODESET.

If you build the software with the wrong settings it may not work
properly. The solution is to build it with the correct settings.
Perhaps you can improve the Windows build.

Or we should call bind_textdomain_codeset as possible, and warn the
user if HAVE_BIND_TEXTDOMAIN_CODESET not defined:

#ifdef HAVE_BIND_TEXTDOMAIN_CODESET
bind_textdomain_codeset(PACKAGE_NAME, UTF-8);
#else
fprintf(sdterr, bind_textdomain_codeset not available, or not
configured. Non-UTF8 locales maybe see garbled output.\n);
#endif /* HAVE_BIND_TEXTDOMAIN_CODESET */

That error would be annoying if it was a false alarm. Perhaps we could
verify the correct behaviour: call gettext and check whether the
returned string is valid UTF8? However, we are not getting reports of
problems so it's probably not worth the effort. The bug that started
this thread is about the exact opposite: gettext was returning UTF8 and
the output code was failing to convert to locale encoding.

--
Certified Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download

Re: svnadmin upgrade output message i18n issue

2013-09-06 Thread Philip Martin

QXO qxodr...@gmail.com writes:

 The bug found in svn 1.8.3(r1516576) again :(

 E:\svnmirrorD:\tools\svn-win32-1.8.3\bin\svnadmin.exe upgrade ZmccPrj
 宸插彇寰楃増鏈簱閿佸畾銆?璇风◢鍊欙紱鍗囩骇鐗堟湰搴撳彲鑳介渶瑕佷竴娈垫椂闂?..
 完成升级。

 E:\svnmirrorD:\tools\svn-win32-1.8.0\bin\svnadmin.exe upgrade ZmccPrj
 已取得版本库锁定。
 请稍候；升级版本库可能需要一段时间...
 完成升级。

 2013/6/19 QXO qxodr...@gmail.com

 The bug fixed in svn 1.8.0,Thanks:)

As I recall you are using a non-UTF8 encoding.  There was a bug in 1.8.0
that caused double UTF8-to-native conversion for svnadmin, that was
fixed in 1.8.3.  Since you say 1.8.0 worked your build must be missing
one of the conversion.  How did you build Subversion?  Was it built with
ENABLE_NLS and HAVE_BIND_TEXTDOMAIN_CODESET?

Is anyone else using Windows in a non-UTF8 setup?

-- 
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*

RE: svnadmin upgrade output message i18n issue

2013-09-06 Thread Bert Huijben

Which build of the Windows binaries do you use?
 
The Subversion project doesn’t produce binaries, so can’t really support build 
issues.
 
Personally I do build the SlikSvn binaries (http://sliksvn.com/en/download), so 
if you find issues in that I might be able to help you.
 
Bert
 
From: QXO [mailto:qxodr...@gmail.com] 
Sent: vrijdag 6 september 2013 08:49
To: Philip Martin
Cc: Dongsheng Song; users@subversion.apache.org; d...@subversion.apache.org
Subject: Re: svnadmin upgrade output message i18n issue
 
The bug found in svn 1.8.3(r1516576) again :(
 
 
E:\svnmirrorD:\tools\svn-win32-1.8.3\bin\svnadmin.exe upgrade ZmccPrj
宸插彇寰楃増鏈簱閿佸畾銆?璇风◢鍊欙紱鍗囩骇鐗堟湰搴撳彲鑳介渶瑕佷竴娈垫椂闂?..
 
完成升级。
 
 
E:\svnmirrorD:\tools\svn-win32-1.8.0\bin\svnadmin.exe upgrade ZmccPrj
已取得版本库锁定。
请稍候；升级版本库可能需要一段时间...
 
完成升级。
 
 
 
2013/6/19 QXO  mailto:qxodr...@gmail.com qxodr...@gmail.com
The bug fixed in svn 1.8.0,Thanks:)
 
2013/5/24 Philip Martin  mailto:philip.mar...@wandisco.com 
philip.mar...@wandisco.com
Dongsheng Song  mailto:dongsheng.s...@gmail.com dongsheng.s...@gmail.com 
writes:

 We do call bind_textdomain_codeset if it is available so we should be
 getting UTF8 translations.


 For non-autotools system, e.g. Windows, user may not define
 HAVE_BIND_TEXTDOMAIN_CODESET.
If you build the software with the wrong settings it may not work
properly.  The solution is to build it with the correct settings.
Perhaps you can improve the Windows build.

 Or we should call bind_textdomain_codeset as possible, and warn the
 user if HAVE_BIND_TEXTDOMAIN_CODESET not defined:

 #ifdef HAVE_BIND_TEXTDOMAIN_CODESET
   bind_textdomain_codeset(PACKAGE_NAME, UTF-8);
 #else
   fprintf(sdterr, bind_textdomain_codeset not available, or not
 configured.  Non-UTF8 locales maybe see garbled output.\n);
 #endif /* HAVE_BIND_TEXTDOMAIN_CODESET */
That error would be annoying if it was a false alarm.  Perhaps we could
verify the correct behaviour: call gettext and check whether the
returned string is valid UTF8?  However, we are not getting reports of
problems so it's probably not worth the effort.  The bug that started
this thread is about the exact opposite: gettext was returning UTF8 and
the output code was failing to convert to locale encoding.

--
Certified  Supported Apache Subversion Downloads:
 http://www.wandisco.com/subversion/download 
http://www.wandisco.com/subversion/download

Re: svnadmin upgrade output message i18n issue

2013-09-06 Thread QXO

http://sourceforge.net/projects/win32svn/


2013/9/6 Bert Huijben b...@qqmail.nl

 Which build of the Windows binaries do you use?

 ** **

 The Subversion project doesn’t produce binaries, so can’t really support
 build issues.

 ** **

 Personally I do build the SlikSvn binaries (http://sliksvn.com/en/download),
 so if you find issues in that I might be able to help you.

 ** **

 Bert

 ** **

 *From:* QXO [mailto:qxodr...@gmail.com]
 *Sent:* vrijdag 6 september 2013 08:49
 *To:* Philip Martin
 *Cc:* Dongsheng Song; users@subversion.apache.org;
 d...@subversion.apache.org
 *Subject:* Re: svnadmin upgrade output message i18n issue

 ** **

 The bug found in svn 1.8.3(r1516576) again :(

 ** **

 ** **

 E:\svnmirrorD:\tools\svn-win32-1.8.3\bin\svnadmin.exe upgrade ZmccPrj

 宸插��寰���搴瀹���?璇风�锛绾х�搴��介��瑕�涓�娈垫�堕�?..

 ** **

 完成升级。

 ** **

 ** **

 E:\svnmirrorD:\tools\svn-win32-1.8.0\bin\svnadmin.exe upgrade ZmccPrj

 已取得版本库锁定。

 请稍候；升级版本库可能需要一段时间...

 ** **

 完成升级。

 ** **

 ** **

 ** **

 2013/6/19 QXO qxodr...@gmail.com

 The bug fixed in svn 1.8.0,Thanks:)

 ** **

 2013/5/24 Philip Martin philip.mar...@wandisco.com

 Dongsheng Song dongsheng.s...@gmail.com writes:

  We do call bind_textdomain_codeset if it is available so we should be
  getting UTF8 translations.
 
 
  For non-autotools system, e.g. Windows, user may not define
  HAVE_BIND_TEXTDOMAIN_CODESET.

 If you build the software with the wrong settings it may not work
 properly.  The solution is to build it with the correct settings.
 Perhaps you can improve the Windows build.


  Or we should call bind_textdomain_codeset as possible, and warn the
  user if HAVE_BIND_TEXTDOMAIN_CODESET not defined:
 
  #ifdef HAVE_BIND_TEXTDOMAIN_CODESET
bind_textdomain_codeset(PACKAGE_NAME, UTF-8);
  #else
fprintf(sdterr, bind_textdomain_codeset not available, or not
  configured.  Non-UTF8 locales maybe see garbled output.\n);
  #endif /* HAVE_BIND_TEXTDOMAIN_CODESET */

 That error would be annoying if it was a false alarm.  Perhaps we could
 verify the correct behaviour: call gettext and check whether the
 returned string is valid UTF8?  However, we are not getting reports of
 problems so it's probably not worth the effort.  The bug that started
 this thread is about the exact opposite: gettext was returning UTF8 and
 the output code was failing to convert to locale encoding.


 --
 Certified  Supported Apache Subversion Downloads:
 http://www.wandisco.com/subversion/download

 ** **

 ** **

Re: svnadmin dump fails with E200020: Pathname not terminated by ':'

2013-08-19 Thread Jan Stürtz

Hi Philip,
sorry for the late reply. Thanks for your details.
That helped and works fine for us.

Kind regards

  - Jan


Am 16.08.2013 11:51, schrieb Philip Martin:
 Jan Stürtz jan.stue...@contact.de writes:
 
 Hi list,
 we have a quite old  grown svn repository here. With some interesting
 svn:mergeinfo Properties. (see the diff below)

 This seems to be been allowed in earlier svn versions.

 For backup Issues svn copy worked fine for all the time,
 but now we want to update our server (svn version  hardware) and
 do a full svn dump/load cycle:

 svnadmin dump --incremental -r 59979 src-100904/  /root/src.dump
 svnadmin: E200020: Pathname not terminated by ':'

 As reading a little bit in the docs  code I found out that there is a
 possibility to bypass the check on svn load (--bypass-prop-validation),
 but not with svn dump.

 It looks like we have mutliple Changes like this one.

 Does anyone have suggestions how to get on with that issue?
 
 The mergeinfo is only parsed for partial dumps, i.e. when the start
 revision greater than one, as this allows the dump process to issue a
 warning when the mergeinfo refers to revisions not in the dump file.
 The only workaround is to do full dumps which bypasses the mergeinfo
 parse and will write the invalid mergeinfo to the dumpfile.  You will
 then need to correct the mergeinfo in the dumpfile as load will fail to
 load a dumpfile with invalid mergeinfo.
 



signature.asc
Description: OpenPGP digital signature

Re: svnadmin dump fails with E200020: Pathname not terminated by ':'

2013-08-16 Thread Philip Martin

Jan Stürtz jan.stue...@contact.de writes:

 Hi list,
 we have a quite old  grown svn repository here. With some interesting
 svn:mergeinfo Properties. (see the diff below)

 This seems to be been allowed in earlier svn versions.

 For backup Issues svn copy worked fine for all the time,
 but now we want to update our server (svn version  hardware) and
 do a full svn dump/load cycle:

 svnadmin dump --incremental -r 59979 src-100904/  /root/src.dump
 svnadmin: E200020: Pathname not terminated by ':'

 As reading a little bit in the docs  code I found out that there is a
 possibility to bypass the check on svn load (--bypass-prop-validation),
 but not with svn dump.

 It looks like we have mutliple Changes like this one.

 Does anyone have suggestions how to get on with that issue?

The mergeinfo is only parsed for partial dumps, i.e. when the start
revision greater than one, as this allows the dump process to issue a
warning when the mergeinfo refers to revisions not in the dump file.
The only workaround is to do full dumps which bypasses the mergeinfo
parse and will write the invalid mergeinfo to the dumpfile.  You will
then need to correct the mergeinfo in the dumpfile as load will fail to
load a dumpfile with invalid mergeinfo.

-- 
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*

Re: svnadmin dump fails with E200020: Pathname not terminated by ':'

2013-08-16 Thread Stefan Sperling

On Fri, Aug 16, 2013 at 10:51:58AM +0100, Philip Martin wrote:
 The mergeinfo is only parsed for partial dumps, i.e. when the start
 revision greater than one, as this allows the dump process to issue a
 warning when the mergeinfo refers to revisions not in the dump file.
 The only workaround is to do full dumps which bypasses the mergeinfo
 parse and will write the invalid mergeinfo to the dumpfile.  You will
 then need to correct the mergeinfo in the dumpfile as load will fail to
 load a dumpfile with invalid mergeinfo.
 
This might be useful for fixing the properties: http://svn.borg.ch/svndumptool/

Re: svnadmin dump fails with E200020: Pathname not terminated by ':'

2013-08-16 Thread Nico Kadel-Garcia

Note that it is sometimes much, much faster and creates much smaller and
cleaner repositories, to simply svn export and svn import  the data
files themselves to a new repository. You don't get  the history, but a
pointer to the old repository kept online in read only mode may be
considered enough. It's also a good opportunity to trim deadwood and bulky,
accidentally committed binaries.

Re: SVNADMIN Crashes

2013-06-23 Thread Andy Levy

On Sun, Jun 23, 2013 at 8:16 PM, Bob Melucci
bob.melu...@claritydesign.com wrote:
 SVNAdmin Crashes with upgrade command- see attached dumps.



 Any help would be appreciated

You've probably run into this bug:
http://svn.haxx.se/users/archive-2013-06/0143.shtml

Re: svnadmin crash

2013-06-19 Thread Branko Čibej

On 19.06.2013 02:04, Mat Berchtold wrote:

 svnadmin upgrade crash

 Divide by zero


Noted. The issue is known and will be fixed in 1.8.1; it's caused by
upgrading unsharded FSFS repositories that were created by moderately
old versions of FSFS.

For now, the best suggestion is to not run svnadmin upgrade; it is not
required for the server to work.


-- 
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. br...@wandisco.com

Re: svnadmin upgrade output message i18n issue

2013-06-18 Thread QXO

The bug fixed in svn 1.8.0,Thanks:)

2013/5/24 Philip Martin philip.mar...@wandisco.com

 Dongsheng Song dongsheng.s...@gmail.com writes:

  We do call bind_textdomain_codeset if it is available so we should be
  getting UTF8 translations.
 
 
  For non-autotools system, e.g. Windows, user may not define
  HAVE_BIND_TEXTDOMAIN_CODESET.

 If you build the software with the wrong settings it may not work
 properly.  The solution is to build it with the correct settings.
 Perhaps you can improve the Windows build.

  Or we should call bind_textdomain_codeset as possible, and warn the
  user if HAVE_BIND_TEXTDOMAIN_CODESET not defined:
 
  #ifdef HAVE_BIND_TEXTDOMAIN_CODESET
bind_textdomain_codeset(PACKAGE_NAME, UTF-8);
  #else
fprintf(sdterr, bind_textdomain_codeset not available, or not
  configured.  Non-UTF8 locales maybe see garbled output.\n);
  #endif /* HAVE_BIND_TEXTDOMAIN_CODESET */

 That error would be annoying if it was a false alarm.  Perhaps we could
 verify the correct behaviour: call gettext and check whether the
 returned string is valid UTF8?  However, we are not getting reports of
 problems so it's probably not worth the effort.  The bug that started
 this thread is about the exact opposite: gettext was returning UTF8 and
 the output code was failing to convert to locale encoding.

 --
 Certified  Supported Apache Subversion Downloads:
 http://www.wandisco.com/subversion/download

Re: svnadmin upgrade output message i18n issue

I have download a binary package from win32svn[1], and confirmed your issue.

I check the subversion.mo file:

msgunfmt.exe  subversion.mo  -o subversion.po

It looks OK. Then I replaced the intl3_svn.dll file with gettext
0.18.2, it give me another output:

C:\var\tmpsvnadmin upgrade test
已取得版本库锁定。
请稍候；升级版本库可能需要一段时间...

?\205?\234?\179?\201?\201?\253?\188?\182?\161?\163

C:\var\tmp

The I write a simple test program:

/*
 * cl /MD /I. t-intl.c libintl-8.lib
 */
#include stdio.h
#include locale.h
#include libintl.h
#include iconv.h

#define _(S) gettext(S)

#define PACKAGE_NAMEsubversion

int main(int argc, char **argv)
{
iconv_t cd;
size_t nc, inbytesleft, outbytesleft;
char *msg, msg2[256];

setlocale(LC_ALL,);
setlocale(LC_CTYPE, );

bindtextdomain(PACKAGE_NAME, ../share/locale);
/* bind_textdomain_codeset(PACKAGE_NAME, UTF-8); */
textdomain(PACKAGE_NAME);

#undef printf

printf(_(Repository lock acquired.\n
Please wait; recovering the repository may take some time...\n));

printf(_(\n
Upgrade completed.\n));

return 0;
}

C:\var\tmp\svn-win32-1.7.9\bincl /nologo /MD /I. t-intl.c libintl-8.lib

C:\var\tmp\svn-win32-1.7.9\bint-intl.exe
已取得版本库锁定。
请稍候；修复版本库可能需要一段时间...

完成升级。

So this is a binary package build issue, not subversion issue.

[1] 
http://sourceforge.net/projects/win32svn/files/1.7.9/apache24/svn-win32-1.7.9-ap24.zip/download

On Thu, May 23, 2013 at 10:13 AM, QXO qxodr...@gmail.com wrote:
 os: windows
 encoding:GBK ( chcp 936 )

 The svnadmin  upgrade command output message  first line encoding
 issue(UTF-8 show in GBK),But the second line is right encoding!

 宸插彇寰楃増鏈簱閿佸畾銆?璇风◢鍊欙紱鍗囩骇鐗堟湰搴撳彲鑳介渶瑕佷竴娈垫椂闂?..

 完成升级。

 if change console encoding to UTF-8 (chcp 65001),output message is :

 Repository lock acquired.
 Please wait; upgrading the repository may take some time...

 Upgrade completed.

Re: svnadmin upgrade output message i18n issue

[bringing in dev@s.a.o]

QXO qxodr...@gmail.com writes:

 os: windows
 encoding:GBK ( chcp 936 )

 The svnadmin  upgrade command output message  first line encoding
 issue(UTF-8 show in GBK),But the second line is right encoding!

 宸插彇寰楃増鏈簱閿佸畾銆?璇风◢鍊欙紱鍗囩骇鐗堟湰搴撳彲鑳介渶瑕佷竴娈垫椂闂?..

 完成升级。

 if change console encoding to UTF-8 (chcp 65001),output message is :

 Repository lock acquired.
 Please wait; upgrading the repository may take some time...

 Upgrade completed.

Those two lines are produced by different code paths.  The first line
is produced by repos_notify_handler:

  svn_error_clear(svn_stream_printf(feedback_stream, scratch_pool,
 _(Repository lock acquired.\n
   Please wait; upgrading the
repository may take some time...\n)));

The second line is produced by:

  SVN_ERR(svn_cmdline_printf(pool, _(\nUpgrade completed.\n)));

and svn_cmdline_printf uses svn_cmdline_cstring_from_utf8 to do a UTF8
to native conversion.

So it appears the UTF8 to native conversion is missing from
repos_notify_handler.  I think repos_notify_handler should be using
svn_stream_printf_from_utf8 rather than svn_stream_printf.

-- 
Certified  Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download

Re: svnadmin upgrade output message i18n issue

On Thu, May 23, 2013 at 4:17 PM, Philip Martin
philip.mar...@wandisco.com wrote:
[bringing in dev@s.a.o]

QXO qxodr...@gmail.com writes:

os: windows
encoding:GBK ( chcp 936 )

The svnadmin upgrade command output message first line encoding
issue(UTF-8 show in GBK),But the second line is right encoding!

宸插彇寰楃増鏈簱閿佸畾銆?璇风◢鍊欙紱鍗囩骇鐗堟湰搴撳彲鑳介渶瑕佷竴娈垫椂闂?..

完成升级。

if change console encoding to UTF-8 (chcp 65001),output message is :

Repository lock acquired.
Please wait; upgrading the repository may take some time...

Upgrade completed.

Those two lines are produced by different code paths. The first line
is produced by repos_notify_handler:

svn_error_clear(svn_stream_printf(feedback_stream, scratch_pool,
_(Repository lock acquired.\n
Please wait; upgrading the
repository may take some time...\n)));

The second line is produced by:

SVN_ERR(svn_cmdline_printf(pool, _(\nUpgrade completed.\n)));

and svn_cmdline_printf uses svn_cmdline_cstring_from_utf8 to do a UTF8
to native conversion.

So it appears the UTF8 to native conversion is missing from
repos_notify_handler. I think repos_notify_handler should be using
svn_stream_printf_from_utf8 rather than svn_stream_printf.

NO. From GETTEXT(3) man pages:

In both cases, the functions also use the LC_CTYPE locale facet in
order to convert the translated message from the translator's
codeset to the ***current locale's codeset***, unless overridden by a
prior call to the bind_textdomain_codeset function.

So svn_cmdline_printf SHOULD NOT assume the input string is UTF-8
coded, it it encoded to the ***current locale's codeset***.

--
Regards,
Dongsheng

Re: svnadmin upgrade output message i18n issue

Philip Martin philip.mar...@wandisco.com writes:

 So it appears the UTF8 to native conversion is missing from
 repos_notify_handler.  I think repos_notify_handler should be using
 svn_stream_printf_from_utf8 rather than svn_stream_printf.

I've fixed trunk to use svn_cmdline_cstring_from_utf8 and proposed it
for 1.8.

-- 
Certified  Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download

Re: svnadmin upgrade output message i18n issue

On Thu, May 23, 2013 at 9:11 PM, Philip Martin
philip.mar...@wandisco.com wrote:
 Philip Martin philip.mar...@wandisco.com writes:

 So it appears the UTF8 to native conversion is missing from
 repos_notify_handler.  I think repos_notify_handler should be using
 svn_stream_printf_from_utf8 rather than svn_stream_printf.

 I've fixed trunk to use svn_cmdline_cstring_from_utf8 and proposed it
 for 1.8.


As GETTEXT(3) man pages said, If and only if
defined(HAVE_BIND_TEXTDOMAIN_CODESET),
your commit is OK.

So you should check HAVE_BIND_TEXTDOMAIN_CODESET when you use
svn_cmdline_cstring_from_utf8.

Re: svnadmin upgrade output message i18n issue

Dongsheng Song dongsheng.s...@gmail.com writes:

 On Thu, May 23, 2013 at 9:11 PM, Philip Martin
 philip.mar...@wandisco.com wrote:
 Philip Martin philip.mar...@wandisco.com writes:

 So it appears the UTF8 to native conversion is missing from
 repos_notify_handler.  I think repos_notify_handler should be using
 svn_stream_printf_from_utf8 rather than svn_stream_printf.

 I've fixed trunk to use svn_cmdline_cstring_from_utf8 and proposed it
 for 1.8.


 As GETTEXT(3) man pages said, If and only if
 defined(HAVE_BIND_TEXTDOMAIN_CODESET),
 your commit is OK.

 So you should check HAVE_BIND_TEXTDOMAIN_CODESET when you use
 svn_cmdline_cstring_from_utf8.

Are you saying there is a problem with my change?  If there is a problem
doesn't already apply to all other uses of svn_cmdline_cstring_from_utf8?

-- 
Certified  Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download

Re: svnadmin upgrade output message i18n issue

On Thu, May 23, 2013 at 9:28 PM, Philip Martin
philip.mar...@wandisco.com wrote:
 Dongsheng Song dongsheng.s...@gmail.com writes:

 On Thu, May 23, 2013 at 9:11 PM, Philip Martin
 philip.mar...@wandisco.com wrote:
 Philip Martin philip.mar...@wandisco.com writes:

 So it appears the UTF8 to native conversion is missing from
 repos_notify_handler.  I think repos_notify_handler should be using
 svn_stream_printf_from_utf8 rather than svn_stream_printf.

 I've fixed trunk to use svn_cmdline_cstring_from_utf8 and proposed it
 for 1.8.


 As GETTEXT(3) man pages said, If and only if
 defined(HAVE_BIND_TEXTDOMAIN_CODESET),
 your commit is OK.

 So you should check HAVE_BIND_TEXTDOMAIN_CODESET when you use
 svn_cmdline_cstring_from_utf8.

 Are you saying there is a problem with my change?  If there is a problem
 doesn't already apply to all other uses of svn_cmdline_cstring_from_utf8?


I thinks so. In the subversion/libsvn_subr/nls.c file:

#ifdef HAVE_BIND_TEXTDOMAIN_CODESET
  bind_textdomain_codeset(PACKAGE_NAME, UTF-8);
#endif /* HAVE_BIND_TEXTDOMAIN_CODESET */

bind_textdomain_codeset only called when HAVE_BIND_TEXTDOMAIN_CODESET
defined. In this case, you can assume GETTEXT(3) returned string is
UTF-8 encoded.

Re: svnadmin upgrade output message i18n issue

Dongsheng Song dongsheng.s...@gmail.com writes:

 On Thu, May 23, 2013 at 9:28 PM, Philip Martin
 philip.mar...@wandisco.com wrote:
 Dongsheng Song dongsheng.s...@gmail.com writes:

 On Thu, May 23, 2013 at 9:11 PM, Philip Martin
 philip.mar...@wandisco.com wrote:
 Philip Martin philip.mar...@wandisco.com writes:

 So it appears the UTF8 to native conversion is missing from
 repos_notify_handler.  I think repos_notify_handler should be using
 svn_stream_printf_from_utf8 rather than svn_stream_printf.

 I've fixed trunk to use svn_cmdline_cstring_from_utf8 and proposed it
 for 1.8.


 As GETTEXT(3) man pages said, If and only if
 defined(HAVE_BIND_TEXTDOMAIN_CODESET),
 your commit is OK.

 So you should check HAVE_BIND_TEXTDOMAIN_CODESET when you use
 svn_cmdline_cstring_from_utf8.

 Are you saying there is a problem with my change?  If there is a problem
 doesn't already apply to all other uses of svn_cmdline_cstring_from_utf8?


 I thinks so. In the subversion/libsvn_subr/nls.c file:

 #ifdef HAVE_BIND_TEXTDOMAIN_CODESET
   bind_textdomain_codeset(PACKAGE_NAME, UTF-8);
 #endif /* HAVE_BIND_TEXTDOMAIN_CODESET */

 bind_textdomain_codeset only called when HAVE_BIND_TEXTDOMAIN_CODESET
 defined. In this case, you can assume GETTEXT(3) returned string is
 UTF-8 encoded.

I still don't understand if you are claiming my change has a problem or
if there is a problem in all uses of svn_cmdline_cstring_from_utf8.

I recall a related thread from last year:

http://svn.haxx.se/dev/archive-2012-08/index.shtml#34
http://mail-archives.apache.org/mod_mbox/subversion-dev/201208.mbox/%3Cop.wilcelggnngjn5@tortoise%3E

I think we assume that the translations are UTF-8.

Is there some code change you think we should make?

-- 
Certified  Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download

Re: svnadmin upgrade output message i18n issue

2013-05-23 Thread Branko Čibej

On 23.05.2013 16:06, Philip Martin wrote:
 Dongsheng Song dongsheng.s...@gmail.com writes:

 On Thu, May 23, 2013 at 9:28 PM, Philip Martin
 philip.mar...@wandisco.com wrote:
 Dongsheng Song dongsheng.s...@gmail.com writes:

 On Thu, May 23, 2013 at 9:11 PM, Philip Martin
 philip.mar...@wandisco.com wrote:
 Philip Martin philip.mar...@wandisco.com writes:

 So it appears the UTF8 to native conversion is missing from
 repos_notify_handler.  I think repos_notify_handler should be using
 svn_stream_printf_from_utf8 rather than svn_stream_printf.
 I've fixed trunk to use svn_cmdline_cstring_from_utf8 and proposed it
 for 1.8.

 As GETTEXT(3) man pages said, If and only if
 defined(HAVE_BIND_TEXTDOMAIN_CODESET),
 your commit is OK.

 So you should check HAVE_BIND_TEXTDOMAIN_CODESET when you use
 svn_cmdline_cstring_from_utf8.
 Are you saying there is a problem with my change?  If there is a problem
 doesn't already apply to all other uses of svn_cmdline_cstring_from_utf8?

 I thinks so. In the subversion/libsvn_subr/nls.c file:

 #ifdef HAVE_BIND_TEXTDOMAIN_CODESET
   bind_textdomain_codeset(PACKAGE_NAME, UTF-8);
 #endif /* HAVE_BIND_TEXTDOMAIN_CODESET */

 bind_textdomain_codeset only called when HAVE_BIND_TEXTDOMAIN_CODESET
 defined. In this case, you can assume GETTEXT(3) returned string is
 UTF-8 encoded.
 I still don't understand if you are claiming my change has a problem or
 if there is a problem in all uses of svn_cmdline_cstring_from_utf8.

 I recall a related thread from last year:

 http://svn.haxx.se/dev/archive-2012-08/index.shtml#34
 http://mail-archives.apache.org/mod_mbox/subversion-dev/201208.mbox/%3Cop.wilcelggnngjn5@tortoise%3E

 I think we assume that the translations are UTF-8.

We do not assume the translations are UTF-8, we require them to be.

http://subversion.apache.org/docs/community-guide/l10n.html#po-mo-requirements

-- Brane 

-- 
Branko Čibej
Director of Subversion | WANdisco | www.wandisco.com

Re: svnadmin upgrade output message i18n issue

On Thu, May 23, 2013 at 10:06 PM, Philip Martin
philip.mar...@wandisco.com wrote:
 Dongsheng Song dongsheng.s...@gmail.com writes:

 On Thu, May 23, 2013 at 9:28 PM, Philip Martin
 philip.mar...@wandisco.com wrote:
 Dongsheng Song dongsheng.s...@gmail.com writes:

 On Thu, May 23, 2013 at 9:11 PM, Philip Martin
 philip.mar...@wandisco.com wrote:
 Philip Martin philip.mar...@wandisco.com writes:

 So it appears the UTF8 to native conversion is missing from
 repos_notify_handler.  I think repos_notify_handler should be using
 svn_stream_printf_from_utf8 rather than svn_stream_printf.

 I've fixed trunk to use svn_cmdline_cstring_from_utf8 and proposed it
 for 1.8.


 As GETTEXT(3) man pages said, If and only if
 defined(HAVE_BIND_TEXTDOMAIN_CODESET),
 your commit is OK.

 So you should check HAVE_BIND_TEXTDOMAIN_CODESET when you use
 svn_cmdline_cstring_from_utf8.

 Are you saying there is a problem with my change?  If there is a problem
 doesn't already apply to all other uses of svn_cmdline_cstring_from_utf8?


 I thinks so. In the subversion/libsvn_subr/nls.c file:

 #ifdef HAVE_BIND_TEXTDOMAIN_CODESET
   bind_textdomain_codeset(PACKAGE_NAME, UTF-8);
 #endif /* HAVE_BIND_TEXTDOMAIN_CODESET */

 bind_textdomain_codeset only called when HAVE_BIND_TEXTDOMAIN_CODESET
 defined. In this case, you can assume GETTEXT(3) returned string is
 UTF-8 encoded.

 I still don't understand if you are claiming my change has a problem or
 if there is a problem in all uses of svn_cmdline_cstring_from_utf8.

 I recall a related thread from last year:

 http://svn.haxx.se/dev/archive-2012-08/index.shtml#34
 http://mail-archives.apache.org/mod_mbox/subversion-dev/201208.mbox/%3Cop.wilcelggnngjn5@tortoise%3E

 I think we assume that the translations are UTF-8.

 Is there some code change you think we should make?


Even ALL the translations are UTF-8, GETTEXT(3) still return the
string encoded by the ***current locale's codeset***.

 Here is sniped from the GETTEXT(3) man pages:

In both cases, the functions also use the LC_CTYPE locale facet  in
order  to  convert  the translated message from the translator's
codeset to the ***current locale's codeset***, unless overridden by a
prior call to the bind_textdomain_codeset function.

So svn_cmdline_printf SHOULD NOT assume the input string is UTF-8
coded, it it encoded to the ***current locale's codeset***.

I think the best solution is: DO NOTconvert the GETTEXT(3) returned
messages, write it ***AS IS***, since GETTEXT(3)  already do the
correct conversion for us.

Re: svnadmin upgrade output message i18n issue

2013-05-23 Thread Erik Huelsmann

sent from my phone
On May 23, 2013 4:43 PM, Dongsheng Song dongsheng.s...@gmail.com wrote:

On Thu, May 23, 2013 at 10:06 PM, Philip Martin
philip.mar...@wandisco.com wrote:
Dongsheng Song dongsheng.s...@gmail.com writes:

On Thu, May 23, 2013 at 9:28 PM, Philip Martin
philip.mar...@wandisco.com wrote:
Dongsheng Song dongsheng.s...@gmail.com writes:

On Thu, May 23, 2013 at 9:11 PM, Philip Martin
philip.mar...@wandisco.com wrote:
Philip Martin philip.mar...@wandisco.com writes:

So it appears the UTF8 to native conversion is missing from
repos_notify_handler. I think repos_notify_handler should be using
svn_stream_printf_from_utf8 rather than svn_stream_printf.

I've fixed trunk to use svn_cmdline_cstring_from_utf8 and proposed
it
for 1.8.

As GETTEXT(3) man pages said, If and only if
defined(HAVE_BIND_TEXTDOMAIN_CODESET),
your commit is OK.

So you should check HAVE_BIND_TEXTDOMAIN_CODESET when you use
svn_cmdline_cstring_from_utf8.

Are you saying there is a problem with my change? If there is a
problem
doesn't already apply to all other uses of
svn_cmdline_cstring_from_utf8?

I thinks so. In the subversion/libsvn_subr/nls.c file:

#ifdef HAVE_BIND_TEXTDOMAIN_CODESET
bind_textdomain_codeset(PACKAGE_NAME, UTF-8);
#endif /* HAVE_BIND_TEXTDOMAIN_CODESET */

bind_textdomain_codeset only called when HAVE_BIND_TEXTDOMAIN_CODESET
defined. In this case, you can assume GETTEXT(3) returned string is
UTF-8 encoded.

I still don't understand if you are claiming my change has a problem or
if there is a problem in all uses of svn_cmdline_cstring_from_utf8.

I recall a related thread from last year:

http://svn.haxx.se/dev/archive-2012-08/index.shtml#34

http://mail-archives.apache.org/mod_mbox/subversion-dev/201208.mbox/%3Cop.wilcelggnngjn5@tortoise%3E

I think we assume that the translations are UTF-8.

Is there some code change you think we should make?

Even ALL the translations are UTF-8, GETTEXT(3) still return the
string encoded by the ***current locale's codeset***.

Here is sniped from the GETTEXT(3) man pages:

So svn_cmdline_printf SHOULD NOT assume the input string is UTF-8
coded, it it encoded to the ***current locale's codeset***.

But we call the codeset function to make sure we do not generate output in
the current locale encoding.

I think the best solution is: DO NOTconvert the GETTEXT(3) returned
messages, write it ***AS IS***, since GETTEXT(3) already do the
correct conversion for us.

Well, even though gettext may want us to believe otherwise, this doesn't
work for cross platform applications: e.g. in windows the locale for output
on the console may be different from the locale for other uses. Back when
we went with gettext (2004?), we've hashed this through pretty thoroughly.
I hope that discussion is still available in the archives.

Bye,

Erik.

Re: svnadmin upgrade output message i18n issue