Re: unversioned properties: size limitations?
On 12.08.2014 08:17, Alexey Neyman wrote: > > On Tuesday, August 12, 2014 07:44:03 AM Branko Čibej wrote: > > > > I'm still not sure what you're trying to achieve, though. "Communication > > > between pre- and post-commit hooks" doesn't describe the problem, it > > > describes a solution, and there are of course other ways for hooks to > > > communicate that do not involve the repository. > > > > I've mentioned this in the other thread where you also responded. > There are two problems that are currently (we're using 1.6) solved by > modifying the transaction in the pre-commit hook: > > > > 1. We have a header that needs to reflect the last > modification date of *any* file in the project. Currently, pre-commit > script modifies a property in each commit which touches any file in > /project/trunk. > So why do you need the last-changed revision of the project directory stored in the header file? Are you distributing sources directly from the repository, or are you using that number to identify builds? In either case, there are better ways to do that. > 2. We have a few checks in pre-commit that are performed on text but > not on binary files, and (unless it the type is set explicitly), the > text is distinguished from binaries using simple heuristics. To avoid > running this heuristics over and over, the result is saved into a > property on that file. > I'd just like to point out that Subversion already has heuristics to identify binary files. If you add a binary file to the working copy, it will automagically get the svn:mime-type set to application/octet-stream. There are only a few MIME types that Subversion does not consider binary. Of course, your heuristics might be different from Subversion's, but have you considered just looking at the mime-type property? > So, to avoid modifying the transaction by pre-commit (that no longer > reliably works in 1.7 and 1.8), > Correction: that never reliably worked, because we never promised it would work. > I am changing the pre-commit to list the 'tasks' to be performed by > the post-commit - which will check in a new revision. I don't want to > involve an out-of-repository storage for that list of tasks unless > absolutely necessary - hence, revision property looks like the perfect > place to store the "follow-up tasks" for a particular revision. > > > > > Also I find your approach less than robust: > > > > > > * There's no guarantee that the post-commit hook will ever run, so > > > it's a bad idea to rely on it for anything that's critical to your > > > workflow. > > > > Can you please elaborate on this? I thought that if a transaction was > promoted to a revision, the post-commit hook is always run. I > understand that post-commit may fail and this failure will not roll > back a revision. But when is it not run at all? > It's entirely possible that the commit succeeds, but for any number of reasons -- power failure, full disk, simple misconfiguration -- the post-commit hook doesn't run. Subversion does not make any guarantees there other than making a best effort to run it. > PS. I know that there's an interface, svn_fs_commit_txn, than can > bypass both pre- and post-commit hook. But it is not used in regular > commits from the command line, is it? > Nope. > > * There's no guarantee that other commits won't happen before your > > > post-commit hook is run; so whatever you do with the repository in > > > post-commit may have to deal with conflicts, which is not fun to > > > automate. > > > > I understand that, but I don't expect conflicts: the actions by the > post-commit will only touch certain properties that are not set > manually. After all, I can reject an attempt to set those properties > in the pre-commit. > You mean like this? $ svnadmin create repo $ svn co file://$(pwd)/repo wc1 Checked out revision 0. $ echo a > wc1/foo $ svn add wc1/foo A wc1/foo $ svn ci -mm wc1 Adding wc1/foo Transmitting file data . Committed revision 1. $ svn co file://$(pwd)/repo wc2 Awc2/foo Checked out revision 1. $ echo b > wc1/foo $ svn ci -mm wc1 Sendingwc1/foo Transmitting file data . Committed revision 2. $ svn propset x y wc2/foo property 'x' set on 'wc2/foo' $ svn ci -mm wc2 Sendingwc2/foo svn: E155011: Commit failed (details follow): svn: E155011: File '/Users/brane/src/test/wc2/foo' is out of date svn: E160028: File '/foo' is out of date -- Brane -- Branko Čibej | Director of Subversion WANdisco | Realising the impossibilities of Big Data e. br...@wandisco.com
Re: unversioned properties: size limitations?
On Tuesday, August 12, 2014 07:44:03 AM Branko Čibej wrote: > On 12.08.2014 07:16, Alexey Neyman wrote: > > On Tuesday, August 12, 2014 06:59:20 AM Branko Čibej wrote: > > > On 12.08.2014 03:31, Alexey Neyman wrote: > > > > Hi SVN users/developers, > > > > > > > > > > > > > > > > Is there a limitation in size on the property value that can be set? > > > > > > > > Any scalability traps to be aware of (i.e. non-linear increase in time > > > > > > > > due to increase in size of the property value)? I tried a 4Mb > > > > > > > > property, seems to work fine... > > > > > > One thing to be aware of is that properties were never designed to be > > > > > > large. Property values are always transmitted in full text between > > > > > > client and server (i.e., they're not compressed); they're stored in full > > > > > > text in the repository (not deltified the way file contents are). So the > > > > > > more large properties you have, and the more often you modify them, the > > > > > > less efficient your repository will be, in terms of storage requirements > > > > > > and network bandwidth. > > > > > > > > > > > > So while you should be able to store a 2 gig property value, I really, > > > > > > really recommend not to do that. > > > > I thought of having pre- and post-commit hooks communicate using a > > *revision* property: pre-commit hook would set a revision property > > with the list of files and actions to be performed on them, and the > > post-commit hook will perform these actions by committing a new > > revision (instead of modifying a transaction by pre-commit hook). > > > > > > > > Thus a more specific question - when are arbitrary *revision* > > properties sent from the client to the server? Obviously, svn:* > > properties are used by various SVN commands; but am I right to assume > > that non-standard revision properties are sent only for the 'svn pg > > --revprop' command? > > > > > > > > That said, I expect the property value to be much less than 2Gb. So > > far, the largest commit we've had was ~2 files - with ~150 > > characters per path, that would be about 3Mb for the property value. > > Sure, you'll only transmit revprops with propget --revprop and propset > --revprop. I'm not sure what the implications are of storing large > values in revprops, these are handled a bit differently than versioned > properties on the server. > > And of course, revision properties are not versioned. > > I'm still not sure what you're trying to achieve, though. "Communication > between pre- and post-commit hooks" doesn't describe the problem, it > describes a solution, and there are of course other ways for hooks to > communicate that do not involve the repository. I've mentioned this in the other thread where you also responded. There are two problems that are currently (we're using 1.6) solved by modifying the transaction in the pre-commit hook: 1. We have a header that needs to reflect the last modification date of *any* file in the project. Currently, pre-commit script modifies a property in each commit which touches any file in /project/trunk. 2. We have a few checks in pre-commit that are performed on text but not on binary files, and (unless it the type is set explicitly), the text is distinguished from binaries using simple heuristics. To avoid running this heuristics over and over, the result is saved into a property on that file. So, to avoid modifying the transaction by pre-commit (that no longer reliably works in 1.7 and 1.8), I am changing the pre-commit to list the 'tasks' to be performed by the post- commit - which will check in a new revision. I don't want to involve an out-of-repository storage for that list of tasks unless absolutely necessary - hence, revision property looks like the perfect place to store the "follow-up tasks" for a particular revision. > Also I find your approach less than robust: > > * There's no guarantee that the post-commit hook will ever run, so > it's a bad idea to rely on it for anything that's critical to your > workflow. Can you please elaborate on this? I thought that if a transaction was promoted to a revision, the post-commit hook is always run. I understand that post-commit may fail and this failure will not roll back a revision. But when is it not run at all? PS. I know that there's an interface, svn_fs_commit_txn, than can bypass both pre- and post-commit hook. But it is not used in regular commits from the command line, is it? > * There's no guarantee that other commits won't happen before your > post-commit hook is run; so whatever you do with the repository in > post-commit may have to deal with conflicts, which is not fun to > automate. I understand that, but I don't expect conflicts: the actions by the post-commit will only touch certain properties that are not set manually. After all, I can reject an attempt to set those properties in the pre-commit. Regards,
Re: unversioned properties: size limitations?
On 12.08.2014 07:16, Alexey Neyman wrote: > > On Tuesday, August 12, 2014 06:59:20 AM Branko Čibej wrote: > > > On 12.08.2014 03:31, Alexey Neyman wrote: > > > > Hi SVN users/developers, > > > > > > > > Is there a limitation in size on the property value that can be set? > > > > Any scalability traps to be aware of (i.e. non-linear increase in time > > > > due to increase in size of the property value)? I tried a 4Mb > > > > property, seems to work fine... > > > > > > One thing to be aware of is that properties were never designed to be > > > large. Property values are always transmitted in full text between > > > client and server (i.e., they're not compressed); they're stored in full > > > text in the repository (not deltified the way file contents are). So the > > > more large properties you have, and the more often you modify them, the > > > less efficient your repository will be, in terms of storage requirements > > > and network bandwidth. > > > > > > So while you should be able to store a 2 gig property value, I really, > > > really recommend not to do that. > > > > I thought of having pre- and post-commit hooks communicate using a > *revision* property: pre-commit hook would set a revision property > with the list of files and actions to be performed on them, and the > post-commit hook will perform these actions by committing a new > revision (instead of modifying a transaction by pre-commit hook). > > > > Thus a more specific question - when are arbitrary *revision* > properties sent from the client to the server? Obviously, svn:* > properties are used by various SVN commands; but am I right to assume > that non-standard revision properties are sent only for the 'svn pg > --revprop' command? > > > > That said, I expect the property value to be much less than 2Gb. So > far, the largest commit we've had was ~2 files - with ~150 > characters per path, that would be about 3Mb for the property value. > Sure, you'll only transmit revprops with propget --revprop and propset --revprop. I'm not sure what the implications are of storing large values in revprops, these are handled a bit differently than versioned properties on the server. And of course, revision properties are not versioned. I'm still not sure what you're trying to achieve, though. "Communication between pre- and post-commit hooks" doesn't describe the problem, it describes a solution, and there are of course other ways for hooks to communicate that do not involve the repository. Also I find your approach less than robust: * There's no guarantee that the post-commit hook will ever run, so it's a bad idea to rely on it for anything that's critical to your workflow. * There's no guarantee that other commits won't happen before your post-commit hook is run; so whatever you do with the repository in post-commit may have to deal with conflicts, which is not fun to automate. Why don't you just describe what kind of workflow you're implementing, and more importanly, why? -- Brane -- Branko Čibej | Director of Subversion WANdisco | Realising the impossibilities of Big Data e. br...@wandisco.com
Re: unversioned properties: size limitations?
On Tuesday, August 12, 2014 06:59:20 AM Branko Čibej wrote: > On 12.08.2014 03:31, Alexey Neyman wrote: > > Hi SVN users/developers, > > > > Is there a limitation in size on the property value that can be set? > > Any scalability traps to be aware of (i.e. non-linear increase in time > > due to increase in size of the property value)? I tried a 4Mb > > property, seems to work fine... > > One thing to be aware of is that properties were never designed to be > large. Property values are always transmitted in full text between > client and server (i.e., they're not compressed); they're stored in full > text in the repository (not deltified the way file contents are). So the > more large properties you have, and the more often you modify them, the > less efficient your repository will be, in terms of storage requirements > and network bandwidth. > > So while you should be able to store a 2 gig property value, I really, > really recommend not to do that. I thought of having pre- and post-commit hooks communicate using a *revision* property: pre-commit hook would set a revision property with the list of files and actions to be performed on them, and the post-commit hook will perform these actions by committing a new revision (instead of modifying a transaction by pre-commit hook). Thus a more specific question - when are arbitrary *revision* properties sent from the client to the server? Obviously, svn:* properties are used by various SVN commands; but am I right to assume that non-standard revision properties are sent only for the 'svn pg -- revprop' command? That said, I expect the property value to be much less than 2Gb. So far, the largest commit we've had was ~2 files - with ~150 characters per path, that would be about 3Mb for the property value. Regards, Alexey.
Re: unversioned properties: size limitations?
On 12.08.2014 03:31, Alexey Neyman wrote: > > Hi SVN users/developers, > > > > Is there a limitation in size on the property value that can be set? > Any scalability traps to be aware of (i.e. non-linear increase in time > due to increase in size of the property value)? I tried a 4Mb > property, seems to work fine... > One thing to be aware of is that properties were never designed to be large. Property values are always transmitted in full text between client and server (i.e., they're not compressed); they're stored in full text in the repository (not deltified the way file contents are). So the more large properties you have, and the more often you modify them, the less efficient your repository will be, in terms of storage requirements and network bandwidth. So while you should be able to store a 2 gig property value, I really, really recommend not to do that. -- Brane -- Branko Čibej | Director of Subversion WANdisco | Realising the impossibilities of Big Data e. br...@wandisco.com
Re: svn status/diff false positive
On Aug 11, 2014, at 6:53 AM, Ivan Drobyshevskyi wrote: > On Mon, Aug 11, 2014 at 11:05 AM, Ivan Drobyshevskyi wrote: >> There's a strange behaviour that is reproduced for some files in our >> subversion repository: `touch`ing this file(s) makes `svn diff` show >> entire file as changed. Similar happens when opening-saving file in >> any text editor or copying it back and forth. >> >> touch: >> $ svn status cmVectorStream.h >> $ touch cmVectorStream.h >> $ svn status cmVectorStream.h >> M cmVectorStream.h >> >> copy-restore: >> $ svn status cmVectorStream.h >> $ cp cmVectorStream.h cmVectorStream.h.bak >> $ cp cmVectorStream.h.bak cmVectorStream.h >> $ svn status cmVectorStream.h >> M cmVectorStream.h >> $ diff cmVectorStream.h.bak cmVectorStream.h >> >> >> For the vast majority of files in a repo the above doesn't happen, so >> far I've only encountered two such files. Since svn shows entire file >> as changed it might be related to line endings however I don't see how >> touch'ing file could possibly trigger such behaviour. >> >> I'm running Ubuntu 14.04.1, both stock subersion (1.8.8) and the one >> compiled from source (1.8.9 with only --with-serf option set) expose >> similar behaviour. The repository itself is hosted on Windows machine. >> >> Should I fill in a bug report on this? > > Removing svn:eol-style (was set to 'native') for files in question > property solved the issue. I guess it's some weird interaction between > Tortoise SVN hosted repository and upstream subversion. It is the responsibility of the Subversion client to normalize the line endings to LF if you have svn:eol-style set. The official client and libraries do this; perhaps you committed these files using a reimplementation that does not do that. git-svn comes to mind, for example. TortoiseSVN uses the official Subversion libraries and so should not be affected. If the timestamp of the file and its pristine match, Subversion assumes the contents match as well. When you touch one, then it has to actually compare the contents, which is when it discovers this error.
Re: bug with autoindex on Apache 2.4 when loading SVN modules
On 8/2/14 12:17 PM, Gibson, Brian (IMS) wrote: > I have found that if I have these two configuration statements on my Apache > server, I can no longer use auto index. > > LoadModule dav_svn_module /usr/local/subversion/modules/mod_dav_svn.so > > LoadModule authz_svn_module /usr/local/subversion/modules/mod_authz_svn.so > > > > If I remove those two modules, auto index works perfectly. > > > > Has anyone else seen this? Is there a possible work around? > > > > I’m using Apache 2.4.10, subversion 1.8.9, Apr 1.5.0, Apr-util 1.5.3, and Serf > 1.3.6. > > > > I’m not sure if there is any more information I can give you that would be > helpful. Best thing you can provide is an example configuration and explaining what you expect it to do and what it is doing. Ideally you'd make the configuration as simple as you can while still demonstrating the issue. That said I wouldn't be surprised if what you're seeing isn't related to this: http://subversion.apache.org/docs/release-notes/1.8.html#mod_dav_svn-fsmap But you haven't provided enough information for me to say for sure.
unversioned properties: size limitations?
Hi SVN users/developers, Is there a limitation in size on the property value that can be set? Any scalability traps to be aware of (i.e. non-linear increase in time due to increase in size of the property value)? I tried a 4Mb property, seems to work fine... Thanks, Alexey.
Apache Subversion 1.7.18 released
I'm happy to announce the release of Apache Subversion 1.7.18. This release addresses two security issues: CVE-2014-3522: ra_serf improper validation of wildcards in SSL certs. CVE-2014-3528: credentials cached with svn may be sent to wrong server. Please choose the mirror closest to you by visiting: http://subversion.apache.org/download/#supported-releases The SHA1 checksums are: 09fa636f2c59a5a4bc965def814645a2841e1b91 subversion-1.7.18.zip 96873512eeb196e5ba6435fbffb24ba284bfcf84 subversion-1.7.18.tar.gz 56bd2b413950c916642bb4c280690da875d3c745 subversion-1.7.18.tar.bz2 PGP Signatures are available at: http://www.apache.org/dist/subversion/subversion-1.7.18.tar.bz2.asc http://www.apache.org/dist/subversion/subversion-1.7.18.tar.gz.asc http://www.apache.org/dist/subversion/subversion-1.7.18.zip.asc For this release, the following people have provided PGP signatures: Ben Reser [4096R/16A0DE01] with fingerprint: 19BB CAEF 7B19 B280 A0E2 175E 62D4 8FAD 16A0 DE01 Bert Huijben [4096R/CCC8E1DF] with fingerprint: 3D1D C66D 6D2E 0B90 3952 8138 C4A6 C625 CCC8 E1DF Branko Čibej [2048R/C8628501] with fingerprint: 8769 28CD 4954 EA74 87B6 B96C 29B8 92D0 C862 8501 Branko Čibej [4096R/A347943F] with fingerprint: BA3C 15B1 337C F0FB 222B D41A 1BCA 6586 A347 943F Ivan Zhakov [4096R/F6AD8147] with fingerprint: 4829 8F0F E47F 4B8A 43FD 6525 919F 6F61 F6AD 8147 Paul T. Burba [4096R/56F3D7BC] with fingerprint: 1A0F E7C6 B3C5 F8D4 D0C4 A20B 64DD C071 56F3 D7BC Philip Martin [2048R/ED1A599C] with fingerprint: A844 790F B574 3606 EE95 9207 76D7 88E1 ED1A 599C Stefan Fuhrmann [4096R/57921ACC] with fingerprint: 056F 8016 D9B8 7B1B DE41 7467 99EC 741B 5792 1ACC Release notes for the 1.7.x release series may be found at: http://subversion.apache.org/docs/release-notes/1.7.html You can find the list of changes between 1.7.18 and earlier versions at: http://svn.apache.org/repos/asf/subversion/tags/1.7.18/CHANGES Questions, comments, and bug reports to users@subversion.apache.org. Thanks, - The Subversion Team
Apache Subversion 1.8.10 released
I'm happy to announce the release of Apache Subversion 1.8.10. This release addresses two security issues: CVE-2014-3522: ra_serf improper validation of wildcards in SSL certs. CVE-2014-3528: credentials cached with svn may be sent to wrong server. Please choose the mirror closest to you by visiting: http://subversion.apache.org/download/#recommended-release The SHA1 checksums are: d6896d94bb53c1b4c6e9c5bb1a5c466477b19b2b subversion-1.8.10.tar.bz2 8e1e1e5fd97c3f575a81d66232c62dc902257a17 subversion-1.8.10.tar.gz 963637c9aac7f50b1b8d10a8918c57a88fb6844d subversion-1.8.10.zip PGP Signatures are available at: http://www.apache.org/dist/subversion/subversion-1.8.10.tar.bz2.asc http://www.apache.org/dist/subversion/subversion-1.8.10.tar.gz.asc http://www.apache.org/dist/subversion/subversion-1.8.10.zip.asc For this release, the following people have provided PGP signatures: Ben Reser [4096R/16A0DE01] with fingerprint: 19BB CAEF 7B19 B280 A0E2 175E 62D4 8FAD 16A0 DE01 Bert Huijben [4096R/CCC8E1DF] with fingerprint: 3D1D C66D 6D2E 0B90 3952 8138 C4A6 C625 CCC8 E1DF Branko Čibej [2048R/C8628501] with fingerprint: 8769 28CD 4954 EA74 87B6 B96C 29B8 92D0 C862 8501 Branko Čibej [4096R/A347943F] with fingerprint: BA3C 15B1 337C F0FB 222B D41A 1BCA 6586 A347 943F Ivan Zhakov [4096R/F6AD8147] with fingerprint: 4829 8F0F E47F 4B8A 43FD 6525 919F 6F61 F6AD 8147 Paul T. Burba [4096R/56F3D7BC] with fingerprint: 1A0F E7C6 B3C5 F8D4 D0C4 A20B 64DD C071 56F3 D7BC Philip Martin [2048R/ED1A599C] with fingerprint: A844 790F B574 3606 EE95 9207 76D7 88E1 ED1A 599C Stefan Fuhrmann [4096R/57921ACC] with fingerprint: 056F 8016 D9B8 7B1B DE41 7467 99EC 741B 5792 1ACC Stefan Sperling [2048R/9A59B973] with fingerprint: 8BC4 DAE0 C5A4 D65F 4044 0107 4F7D BAA9 9A59 B973 Release notes for the 1.8.x release series may be found at: http://subversion.apache.org/docs/release-notes/1.8.html You can find the list of changes between 1.8.10 and earlier versions at: http://svn.apache.org/repos/asf/subversion/tags/1.8.10/CHANGES Questions, comments, and bug reports to users@subversion.apache.org. Thanks, - The Subversion Team
Re: svn status/diff false positive
Removing svn:eol-style (was set to 'native') for files in question property solved the issue. I guess it's some weird interaction between Tortoise SVN hosted repository and upstream subversion. On Mon, Aug 11, 2014 at 11:05 AM, Ivan Drobyshevskyi wrote: > There's a strange behaviour that is reproduced for some files in our > subversion repository: `touch`ing this file(s) makes `svn diff` show > entire file as changed. Similar happens when opening-saving file in > any text editor or copying it back and forth. > > touch: > $ svn status cmVectorStream.h > $ touch cmVectorStream.h > $ svn status cmVectorStream.h > M cmVectorStream.h > > copy-restore: > $ svn status cmVectorStream.h > $ cp cmVectorStream.h cmVectorStream.h.bak > $ cp cmVectorStream.h.bak cmVectorStream.h > $ svn status cmVectorStream.h > M cmVectorStream.h > $ diff cmVectorStream.h.bak cmVectorStream.h > > > For the vast majority of files in a repo the above doesn't happen, so > far I've only encountered two such files. Since svn shows entire file > as changed it might be related to line endings however I don't see how > touch'ing file could possibly trigger such behaviour. > > I'm running Ubuntu 14.04.1, both stock subersion (1.8.8) and the one > compiled from source (1.8.9 with only --with-serf option set) expose > similar behaviour. The repository itself is hosted on Windows machine. > > Should I fill in a bug report on this?
svn status/diff false positive
There's a strange behaviour that is reproduced for some files in our subversion repository: `touch`ing this file(s) makes `svn diff` show entire file as changed. Similar happens when opening-saving file in any text editor or copying it back and forth. touch: $ svn status cmVectorStream.h $ touch cmVectorStream.h $ svn status cmVectorStream.h M cmVectorStream.h copy-restore: $ svn status cmVectorStream.h $ cp cmVectorStream.h cmVectorStream.h.bak $ cp cmVectorStream.h.bak cmVectorStream.h $ svn status cmVectorStream.h M cmVectorStream.h $ diff cmVectorStream.h.bak cmVectorStream.h For the vast majority of files in a repo the above doesn't happen, so far I've only encountered two such files. Since svn shows entire file as changed it might be related to line endings however I don't see how touch'ing file could possibly trigger such behaviour. I'm running Ubuntu 14.04.1, both stock subersion (1.8.8) and the one compiled from source (1.8.9 with only --with-serf option set) expose similar behaviour. The repository itself is hosted on Windows machine. Should I fill in a bug report on this?
