Re: How to fix corrupt revision in repo?
Guten Tag David Hopkins, am Montag, 19. September 2011 um 04:06 schrieben Sie: At the moment it looks like the nuclear option is to check out the current version of everything and start a new repository with it. You can dump the old repository until the last working rev and start from that with your current working copies, so not loosing all of your history. But if I can purge the broken-ness from the repo and keep the rest of the history, that would obviously be better. My impression on problems like yours, which I had too, is, that in most cases trying to repair broken rev files was simply not successful. Learn from it and take the time to design a proper backup/sync mechanism. Mit freundlichen Grüßen, Thorsten Schöning -- Thorsten Schöning AM-SoFT IT-Systeme - Hameln | Potsdam | Leipzig Telefon: Potsdam: 0331-743881-0 E-Mail: tschoen...@am-soft.de Web: http://www.am-soft.de AM-SoFT GmbH IT-Systeme, Konsumhof 1-5, 14482 Potsdam Amtsgericht Potsdam HRB 21278 P, Geschäftsführer: Andreas Muchow
AW: SVN File Size
Hi all, First of all, I would like to thank you for your dedication to this issue I would like to resume: a) there is work on serverside in 1.7 that cares about file size b) svnlook filesize makes use of it in 1.7 c) there might be issues related to svn:eol and svn:keywords d) It is worth to create a ER for svn log I will try to dig into the code and see if I am able to understand, the rough structure of it. Where should I look to, as a starting point? I am not anymore so fluent in C but I will try my best. For sure I should know more details about the architecture, but I would like to concentrate on the part that deals with this one. Regards Thomas -Ursprüngliche Nachricht- Von: Stefan Sperling [mailto:s...@elego.de] Gesendet: Freitag, 16. September 2011 10:43 An: Stümpfig, Thomas; users@subversion.apache.org Betreff: Re: SVN File Size On Fri, Sep 16, 2011 at 10:31:05AM +0200, Stefan Sperling wrote: So what you're asking amounts to either a performance hit for 'svn log' or to a new feature in the filesystem (record the size of the full text of a file along with the changed-path data which 'svn log' reads). Ooops, I was wrong here, sorry. The expanded size of the full text is already recorded in the filesystem. As Johan pointed out, there is 'svnlook filesize' which prints this number (which is stored in the FS, so there is no need to compute it). There wouldn't be a performance hit for 'svn log'. The below is still valid as-is: I think the feature you are requesting is reasonable. If you like, you can file an 'enhancement request' issue in our issue tracker about this. Maybe somebody will have time to work on it. (In case you know somebody who would like to work on this I'd like to point out that we're always happy to help newcomers get started :)
Re: How to fix corrupt revision in repo?
David Hopkins wrote on Mon, Sep 19, 2011 at 10:06:17 +0800: Daniel Shahaf wrote: One more thing. The fact that in r162 one file was deleted *and no files were added or changed* implies that the only new representations in r162 would be directory representations --- it wouldn't add any *file* representations --- so the reference to r162 in the node-rev header (the sequence of ASCII lines of which the text: line is part) is almost certainly bogus. I'm curious to hear whether the problem was indeed that the noderev referred to r162 instead of r192. Sadly, it wasn't. I've now experimented with that. The offset supposedly within r162 is listed as 670867 bytes, which is well outside the total length of r162 as we've already discussed. But it isn't a valid pointer within r192 either; offset 670867 points to the middle of one of the other rep blocks within the r192 file. I've had a look at the other node-rev headers and it appears that all the rep blocks in the r192 file are fully accounted for by the node-revs which have text: 192. (That is, there are no representations in the r192 file which don't already have a valid node-rev header). So the question is whether the rep is lost or just misplaced. I've had a look through all the revs between 162 and 192 which are at least 600 KiB in size. But I can't find *any* rev files in the whole repository history leading up to 192 where an offset of 670867 points to the beginning of a DELTA or PLAIN representation. So, I'm now assuming that both the reference to r162, and the offset of 670867 bytes, are bogus. But there aren't any obvious candidates for a non-bogus representation of that particular file update. Okay. FWIW, there is no easy way to find an orphaned/unreferenced rep in the revision files. The rightest way would be to parse all node-ids (which _is_ possible, think 'svn ls -R'), collect all the rep pointers they contain, then see if there are any additional reps. And even then, identifying their length would have to be done manually... (reps do not contain their own length) Given that the file with the bogus node-rev is unimportant, and has since been deleted from the repo, is there any way to patch the r192 rev-file so that the repository has enough internal consistency to produce a valid dump file? I suppose you could do it in a number of ways, either by changing what rep the node-rev refers to or by unlinking the node-rev from its parent. Be careful about successors of the file --- whether new revision of the file are expressed as deltas to the node-rev which is now corrupted. You must know whether that's the case before deciding how to alter that node-rev. Also, you must cause no offsets in existing rev files to change (so, add padding as needed). And probably a few other things that I forget right now... At the moment it looks like the nuclear option is to check out the current version of everything and start a new repository with it. This *should* work because the corrupted file isn't included in recent revisions, so SVN won't need to de-reference the invalid reference in r192 when performing the check out. But if I can purge the broken-ness from the repo and keep the rest of the history, that would obviously be better. I certainly don't want to keep using a repo that doesn't validate and can't be dumped, though. You ought to be able to keep the rest of the history even without fixing the brokenness in r192. (as the file is deleted in HEAD, a checkout should work; and you also have the option of dumping the history while excluding the problematic file from it (via authz+svnsync/svnrdump or svndumpfilter).) Daniel Shahaf wrote on Fri, Sep 16, 2011 at 15:37:11 +0300: Quick reply, more verbose one might follow up later. Your reply breaks the nested quoting levels, please try to avoid it, are you sending mail as text/plain? Sorry about breaking the nested quoting. I'm using Outlook which is pretty mediocre as a plain-text email client. I was already using text/plain, but Outlook's quoting style wasn't right, so I was trying to manually fix the text-wrapping and quote marks. Clearly I wasn't getting it right. I've now found a couple more Outlook settings which will hopefully address the problem. Thanks, much better now. Unfortunately, it doesn't look like I'll be able to send you the actual rev file(s), at least not without a lot of inconvenience that I don't want to subject you to (ie an NDA, since we don't actually own the IP to any of the code which may be included in the rev file). Sorry about that. No problems, and good luck. If you need docs about the FS that 'structure' doesn't have, they would be under ../libsvn_fs_base/. Regards, David Hopkins
Re: SVN File Size
Stümpfig, Thomas wrote on Mon, Sep 19, 2011 at 09:18:06 +: Hi all, First of all, I would like to thank you for your dedication to this issue I would like to resume: a) there is work on serverside in 1.7 that cares about file size b) svnlook filesize makes use of it in 1.7 IIRC, in 1.7 svnlook simply exposed a long-existing FS API. c) there might be issues related to svn:eol and svn:keywords The FS returns the length of the repository-normal form, i.e., LF linefeeds and contracted keywords. d) It is worth to create a ER for svn log I'm not sure; where do you draw the line between 'svn log' and 'svn info'? I will try to dig into the code and see if I am able to understand, the rough structure of it. Where should I look to, as a starting point? I am not anymore so fluent in C but I will try my best. For sure I should know more details about the architecture, but I would like to concentrate on the part that deals with this one. First of all you should determine whether your patch needs to extend the wire protocol or not. See subversion/libsvn_ra_svn/protocol or subversion/include/svn_ra.h Regards Thomas -Ursprüngliche Nachricht- Von: Stefan Sperling [mailto:s...@elego.de] Gesendet: Freitag, 16. September 2011 10:43 An: Stümpfig, Thomas; users@subversion.apache.org Betreff: Re: SVN File Size On Fri, Sep 16, 2011 at 10:31:05AM +0200, Stefan Sperling wrote: So what you're asking amounts to either a performance hit for 'svn log' or to a new feature in the filesystem (record the size of the full text of a file along with the changed-path data which 'svn log' reads). Ooops, I was wrong here, sorry. The expanded size of the full text is already recorded in the filesystem. As Johan pointed out, there is 'svnlook filesize' which prints this number (which is stored in the FS, so there is no need to compute it). There wouldn't be a performance hit for 'svn log'. The below is still valid as-is: I think the feature you are requesting is reasonable. If you like, you can file an 'enhancement request' issue in our issue tracker about this. Maybe somebody will have time to work on it. (In case you know somebody who would like to work on this I'd like to point out that we're always happy to help newcomers get started :)
Re: How to fix corrupt revision in repo?
2011/9/19 Thorsten Schöning tschoen...@am-soft.de: Guten Tag David Hopkins, am Montag, 19. September 2011 um 04:06 schrieben Sie: At the moment it looks like the nuclear option is to check out the current version of everything and start a new repository with it. You can dump the old repository until the last working rev and start from that with your current working copies, so not loosing all of your history. You *CAN*. You can also do that to a system without broken revisions. You have to be very careful not to insert a more subtlely corrupted revision, and you *should* change your uuid and check out new working copies lest someone with the old revision in an old working copy wind up severely horking themselves. Simply leaving out the broken revision, renumbering, and keeping the old uuid will break things very, very badly. (Did that once editing out a DVD image that someone added, fortunately it was early in the project before the repository was considered production grade. But if I can purge the broken-ness from the repo and keep the rest of the history, that would obviously be better. My impression on problems like yours, which I had too, is, that in most cases trying to repair broken rev files was simply not successful. Learn from it and take the time to design a proper backup/sync mechanism. This is not as easy as it sounds. Any revision in a core repository that is corrupted online and for which rsync overwrites the backup server's copy is a real problem, and keeping numerous old copies lying around on tape or separate disks is a support issue. And rsync based backups have real problems with revisions in the process of being committed on the primary server. The bulky and lengthy svnadmin dump and load processes are resource intensive and awkward to manage, and they take a *long* time on a big repository, leaving a significant window of opportunity for corruption. The revisions are also vulnerable to filesystem operations by the SVN administrator account, or shared account, on the SVN server. This is one of the reasons I *loathe* shared write permissions for file:// access, it's vulnerable to mistakes. . A backup system that merely replicates those without reliable off-line or distinct disk space storage is vulnerable to all sorts of surprises. My personal favorite solution is a NetApp storage server with filesystem .snapshot directories, and an off-site backup with svnsync running in case the NetApp goes toes-up.
Re: post-commit hook jurisdiction
On Sep 19, 2011, at 11:32, spacegoose wrote: I'm wondering if the post-commit hook applies to the entire SVN repository, Yes. if there are (or can be) multiple post-commit files that apply only to a particular project (or project branch / trunk), No. or if this is all done programmatically via a single post-commit script, e.g. determine which project committed, and based on this do something? Yes.
AW: SVN File Size
Hi all, I found the following when examining the svnlook code svn_fs_file_length where svn look gets the filesize. This is obviously the fs side function. In /libsvn_ra_svn/protocol I found the commands stat and log stat contains a response token of size:number log does not contain such a token I then had a search for svn_ra functions in various header files. But I could not locate an ra api counterpart for svn_fs_file_length. I cannot see if svn_ra_stat is providing filesize info, whilst being defined in /libsvn_ra_svn/protocol. (I hope I am interpreting this correctly) I neither can see if the server_side part encodes filesize information into the stream. Can somebody tell me where the answer is put into the stream? Am I on the right path? Is this already subject to the Dev Mailinglist? This is my starting point of subversion coding. Regards Thomas == stat params: ( path:string [ rev:number ] ) response: ( ? entry:dirent ) dirent: ( name:string kind:node-kind size:number has-props:bool created-rev:number [ created-date:string ] [ last-author:string ] ) === log params: ( ( target-path:string ... ) [ start-rev:number ] [ end-rev:number ] changed-paths:bool strict-node:bool ? limit:number ? include-merged-revisions:bool all-revprops | revprops ( revprop:string ... ) ) Before sending response, server sends log entries, ending with done. If a client does not want to specify a limit, it should send 0 as the limit parameter. rev-props excludes author, date, and log; they are sent separately for backwards-compatibility. log-entry: ( ( change:changed-path-entry ... ) rev:number [ author:string ] [ date:string ] [ message:string ] ? has-children:bool invalid-revnum:bool revprop-count:number rev-props:proplist ? subtractive-merge:bool ) -Ursprüngliche Nachricht- Von: Daniel Shahaf [mailto:d...@daniel.shahaf.name] Gesendet: Montag, 19. September 2011 13:41 An: Stümpfig, Thomas Cc: users@subversion.apache.org Betreff: Re: SVN File Size Stümpfig, Thomas wrote on Mon, Sep 19, 2011 at 09:18:06 +: Hi all, First of all, I would like to thank you for your dedication to this issue I would like to resume: a) there is work on serverside in 1.7 that cares about file size b) svnlook filesize makes use of it in 1.7 IIRC, in 1.7 svnlook simply exposed a long-existing FS API. c) there might be issues related to svn:eol and svn:keywords The FS returns the length of the repository-normal form, i.e., LF linefeeds and contracted keywords. d) It is worth to create a ER for svn log I'm not sure; where do you draw the line between 'svn log' and 'svn info'? I will try to dig into the code and see if I am able to understand, the rough structure of it. Where should I look to, as a starting point? I am not anymore so fluent in C but I will try my best. For sure I should know more details about the architecture, but I would like to concentrate on the part that deals with this one. First of all you should determine whether your patch needs to extend the wire protocol or not. See subversion/libsvn_ra_svn/protocol or subversion/include/svn_ra.h Regards Thomas -Ursprüngliche Nachricht- Von: Stefan Sperling [mailto:s...@elego.de] Gesendet: Freitag, 16. September 2011 10:43 An: Stümpfig, Thomas; users@subversion.apache.org Betreff: Re: SVN File Size On Fri, Sep 16, 2011 at 10:31:05AM +0200, Stefan Sperling wrote: So what you're asking amounts to either a performance hit for 'svn log' or to a new feature in the filesystem (record the size of the full text of a file along with the changed-path data which 'svn log' reads). Ooops, I was wrong here, sorry. The expanded size of the full text is already recorded in the filesystem. As Johan pointed out, there is 'svnlook filesize' which prints this number (which is stored in the FS, so there is no need to compute it). There wouldn't be a performance hit for 'svn log'. The below is still valid as-is: I think the feature you are requesting is reasonable. If you like, you can file an 'enhancement request' issue in our issue tracker about this. Maybe somebody will have time to work on it. (In case you know somebody who would like to work on this I'd like to point out that we're always happy to help newcomers get started :)
Re: SVN File Size
svn_ra_stat() does return the filesize, and it's the API that 'svn info' uses. The next/prev links in the chain are svn_client_info3() and ra_svn_stat() (the latter is a file-private function in libsvn_ra_svn). So, yes, you're on the right path. However, I knew all these APIs would exist, so I'm still where I was before: I'd like to know why the proposed new feature is generally useful enough to be included in the core, where it stands wrt 'svn log' and 'svn info', and how it would be implemented. In particular, implementing a protocol change involves a non-negligible amount of work (to revv four transports, client and server, plus compat code for old servers), while if it doesn't involve a protocol change the question is Why is a bindings script not a sufficient solution?. Stümpfig, Thomas wrote on Mon, Sep 19, 2011 at 21:33:11 +: Hi all, I found the following when examining the svnlook code svn_fs_file_length where svn look gets the filesize. This is obviously the fs side function. In /libsvn_ra_svn/protocol I found the commands stat and log stat contains a response token of size:number log does not contain such a token I then had a search for svn_ra functions in various header files. But I could not locate an ra api counterpart for svn_fs_file_length. I cannot see if svn_ra_stat is providing filesize info, whilst being defined in /libsvn_ra_svn/protocol. (I hope I am interpreting this correctly) I neither can see if the server_side part encodes filesize information into the stream. Can somebody tell me where the answer is put into the stream? Am I on the right path? Is this already subject to the Dev Mailinglist? This is my starting point of subversion coding. Regards Thomas == stat params: ( path:string [ rev:number ] ) response: ( ? entry:dirent ) dirent: ( name:string kind:node-kind size:number has-props:bool created-rev:number [ created-date:string ] [ last-author:string ] ) === log params: ( ( target-path:string ... ) [ start-rev:number ] [ end-rev:number ] changed-paths:bool strict-node:bool ? limit:number ? include-merged-revisions:bool all-revprops | revprops ( revprop:string ... ) ) Before sending response, server sends log entries, ending with done. If a client does not want to specify a limit, it should send 0 as the limit parameter. rev-props excludes author, date, and log; they are sent separately for backwards-compatibility. log-entry: ( ( change:changed-path-entry ... ) rev:number [ author:string ] [ date:string ] [ message:string ] ? has-children:bool invalid-revnum:bool revprop-count:number rev-props:proplist ? subtractive-merge:bool ) -Ursprüngliche Nachricht- Von: Daniel Shahaf [mailto:d...@daniel.shahaf.name] Gesendet: Montag, 19. September 2011 13:41 An: Stümpfig, Thomas Cc: users@subversion.apache.org Betreff: Re: SVN File Size Stümpfig, Thomas wrote on Mon, Sep 19, 2011 at 09:18:06 +: Hi all, First of all, I would like to thank you for your dedication to this issue I would like to resume: a) there is work on serverside in 1.7 that cares about file size b) svnlook filesize makes use of it in 1.7 IIRC, in 1.7 svnlook simply exposed a long-existing FS API. c) there might be issues related to svn:eol and svn:keywords The FS returns the length of the repository-normal form, i.e., LF linefeeds and contracted keywords. d) It is worth to create a ER for svn log I'm not sure; where do you draw the line between 'svn log' and 'svn info'? I will try to dig into the code and see if I am able to understand, the rough structure of it. Where should I look to, as a starting point? I am not anymore so fluent in C but I will try my best. For sure I should know more details about the architecture, but I would like to concentrate on the part that deals with this one. First of all you should determine whether your patch needs to extend the wire protocol or not. See subversion/libsvn_ra_svn/protocol or subversion/include/svn_ra.h Regards Thomas -Ursprüngliche Nachricht- Von: Stefan Sperling [mailto:s...@elego.de] Gesendet: Freitag, 16. September 2011 10:43 An: Stümpfig, Thomas; users@subversion.apache.org Betreff: Re: SVN File Size On Fri, Sep 16, 2011 at 10:31:05AM +0200, Stefan Sperling wrote: So what you're asking amounts to either a performance hit for 'svn log' or to a new feature in the filesystem (record the size of the full text of a file along with the changed-path data which 'svn log'
Ignore development changes.
Hi Everyone, I am just after some advice on how other people solve the following issue. I have a committed file. When it is deployed, it needs to be in a state (lets call it state 1) While testing code locally, it needs to be in some other state (state 2) Short of altering the code to read like; if on dev server have state 1 else have state 2 fi How can I have a local change, but have that change temporarily ignored? My initial thought is to use the svn:ignore But that brings about the issue of, out of mind out sight... and if I ever need to do real changes to that file, they won't be picked up in svn status. The answer is quite likely not even a SVN once - but perhaps a procedure / system one. I am open to any and all suggestions of how people might solve this issue for themselves already. And as always - thanks for any ideas you might have. Gavin Beau Baumanis
Re: Ignore development changes.
The way we do it is: svn:ignore filename.ext filename.local.ext filename.stage.ext filename.prod.ext On local make a symlink to local version: ln -s filename.local.ext filename.ext On staging make a symlink to stage version: ln -s filename.stage.ext filename.ext On prod, make a symlink to prod version: ln -s filename.prod.ext filename.ext Our build scripts do the 2nd and 3rd automagically at deploy. HTH - On Mon, Sep 19, 2011 at 3:29 PM, Gavin Baumanis gav...@thespidernet.comwrote: Hi Everyone, I am just after some advice on how other people solve the following issue. I have a committed file. When it is deployed, it needs to be in a state (lets call it state 1) While testing code locally, it needs to be in some other state (state 2) Short of altering the code to read like; if on dev server have state 1 else have state 2 fi How can I have a local change, but have that change temporarily ignored? My initial thought is to use the svn:ignore But that brings about the issue of, out of mind out sight... and if I ever need to do real changes to that file, they won't be picked up in svn status. The answer is quite likely not even a SVN once - but perhaps a procedure / system one. I am open to any and all suggestions of how people might solve this issue for themselves already. And as always - thanks for any ideas you might have. Gavin Beau Baumanis
Deleting a directory with a space in it.
Hi, I've got a directory in subversion with a space in the name which I'd like to delete. This is what I've tried: svn rm http://subversion/src/Repo/Grad Training svn rm http://subversion/src/Repo/Grad Training svn rm http://subversion/src/Repo/Grad%20Training; svn rm http://subversion/src/Repo/Grad%20Training svn rm http://subversion/src/Repo/Grad\ Training svn rm http://subversion/src/Repo/Grad\ Training and I keep getting: svn: URL 'http://subversion/src/Repo/Grad%2520Training' does not exist svn: Your commit message was left in a temporary file: svn:'svn-commit.9.tmp' I've also tried checking out the parent directory, deleting the directory and committing the changes, but I get a permission error: error: you don't have enough permissions for this transaction: you can't remove Grad Training/ and from the Apache logs: [Tue Sep 20 16:28:00 2011] [error] [client 192.168.1.70] Could not MERGE resource /src/Repo/!svn/act/9f05c85e-f478-4f0e-8d08-d06b5c3d8781 into /src/Repo. [409, #0] [Tue Sep 20 16:28:00 2011] [error] [client 192.168.1.70] Commit blocked by pre-commit hook (exit code 1) with output:\nerror: you don't have enough permissions for this transaction:\nyou can't remove Grad Training/\n [409, #165001] How do you delete directories with a space in subversion? The version of the client I am running is: # svn --version svn, version 1.6.11 (r934486) compiled May 31 2011, 05:46:33 Copyright (C) 2000-2009 CollabNet. Subversion is open source software, see http://subversion.tigris.org/ This product includes software developed by CollabNet ( http://www.Collab.Net/). The following repository access (RA) modules are available: * ra_neon : Module for accessing a repository via WebDAV protocol using Neon. - handles 'http' scheme - handles 'https' scheme * ra_svn : Module for accessing a repository using the svn network protocol. - with Cyrus SASL authentication - handles 'svn' scheme * ra_local : Module for accessing a repository on local disk. - handles 'file' scheme Thank you -- Srđan Đukić
Re: Deleting a directory with a space in it.
Srdan Dukic wrote: I don't think you have a problem with spaces here. [...] error: you don't have enough permissions for this transaction: you can't remove Grad Training/ you should quote the complete error message because the Commit blocked by ... part is essential here and from the Apache logs: [...] [Tue Sep 20 16:28:00 2011] [error] [client 192.168.1.70] Commit blocked by pre-commit hook (exit code 1) with output:\nerror: you don't have enough permissions for this transaction:\nyou can't remove Grad Training/\n [409, #165001] the operation is clearly prohibited by a pre-commit hook script on the server. You will need to talk to the person who put it there. -- Lorenz
