Re: unversioned properties: size limitations?
Branko Čibej br...@wandisco.com writes: For a start, this would require a major change in the wire protocol, where the server would, as a response to a successful commit, report any additional magic changes to the client. The problem with this is that it is error prone; the response may never arrive, for any number of reasons. Therefore, the client could not mark commtited items up-to-date until and unless it received the response. Since at least the DAV protocol is stateless, this implies all sorts of complications and the introduction of intermediate states in the working copy. In short: yes, it'd be hard. Another problem: what does the client do when it receives the changes? The client would update the pristine text/properties to match the repository, but what about the working text/properties? After a commit should the working copy match the repository or not? Suppose the user commits a text change and the server replies with a file delete, what happens to the working file? Does the user lose the data? Does the commit produce some sort of conflict? -- Philip Martin | Subversion Committer WANdisco // *Non-Stop Data*
Re: unversioned properties: size limitations?
On Tuesday, August 12, 2014 07:44:03 AM Branko Čibej wrote: On 12.08.2014 07:16, Alexey Neyman wrote: On Tuesday, August 12, 2014 06:59:20 AM Branko Čibej wrote: On 12.08.2014 03:31, Alexey Neyman wrote: Hi SVN users/developers, Is there a limitation in size on the property value that can be set? Any scalability traps to be aware of (i.e. non-linear increase in time due to increase in size of the property value)? I tried a 4Mb property, seems to work fine... One thing to be aware of is that properties were never designed to be large. Property values are always transmitted in full text between client and server (i.e., they're not compressed); they're stored in full text in the repository (not deltified the way file contents are). So the more large properties you have, and the more often you modify them, the less efficient your repository will be, in terms of storage requirements and network bandwidth. So while you should be able to store a 2 gig property value, I really, really recommend not to do that. I thought of having pre- and post-commit hooks communicate using a *revision* property: pre-commit hook would set a revision property with the list of files and actions to be performed on them, and the post-commit hook will perform these actions by committing a new revision (instead of modifying a transaction by pre-commit hook). Thus a more specific question - when are arbitrary *revision* properties sent from the client to the server? Obviously, svn:* properties are used by various SVN commands; but am I right to assume that non-standard revision properties are sent only for the 'svn pg --revprop' command? That said, I expect the property value to be much less than 2Gb. So far, the largest commit we've had was ~2 files - with ~150 characters per path, that would be about 3Mb for the property value. Sure, you'll only transmit revprops with propget --revprop and propset --revprop. I'm not sure what the implications are of storing large values in revprops, these are handled a bit differently than versioned properties on the server. And of course, revision properties are not versioned. I'm still not sure what you're trying to achieve, though. Communication between pre- and post-commit hooks doesn't describe the problem, it describes a solution, and there are of course other ways for hooks to communicate that do not involve the repository. I've mentioned this in the other thread where you also responded. There are two problems that are currently (we're using 1.6) solved by modifying the transaction in the pre-commit hook: 1. We have a version.h header that needs to reflect the last modification date of *any* file in the project. Currently, pre-commit script modifies a property in each commit which touches any file in /project/trunk. 2. We have a few checks in pre-commit that are performed on text but not on binary files, and (unless it the type is set explicitly), the text is distinguished from binaries using simple heuristics. To avoid running this heuristics over and over, the result is saved into a property on that file. So, to avoid modifying the transaction by pre-commit (that no longer reliably works in 1.7 and 1.8), I am changing the pre-commit to list the 'tasks' to be performed by the post- commit - which will check in a new revision. I don't want to involve an out-of-repository storage for that list of tasks unless absolutely necessary - hence, revision property looks like the perfect place to store the follow-up tasks for a particular revision. Also I find your approach less than robust: * There's no guarantee that the post-commit hook will ever run, so it's a bad idea to rely on it for anything that's critical to your workflow. Can you please elaborate on this? I thought that if a transaction was promoted to a revision, the post-commit hook is always run. I understand that post-commit may fail and this failure will not roll back a revision. But when is it not run at all? PS. I know that there's an interface, svn_fs_commit_txn, than can bypass both pre- and post-commit hook. But it is not used in regular commits from the command line, is it? * There's no guarantee that other commits won't happen before your post-commit hook is run; so whatever you do with the repository in post-commit may have to deal with conflicts, which is not fun to automate. I understand that, but I don't expect conflicts: the actions by the post-commit will only touch certain properties that are not set manually. After all, I can reject an attempt to set those properties in the pre-commit. Regards, Alexey.
Re: unversioned properties: size limitations?
On 12.08.2014 08:17, Alexey Neyman wrote: On Tuesday, August 12, 2014 07:44:03 AM Branko Čibej wrote: I'm still not sure what you're trying to achieve, though. Communication between pre- and post-commit hooks doesn't describe the problem, it describes a solution, and there are of course other ways for hooks to communicate that do not involve the repository. I've mentioned this in the other thread where you also responded. There are two problems that are currently (we're using 1.6) solved by modifying the transaction in the pre-commit hook: 1. We have a version.h header that needs to reflect the last modification date of *any* file in the project. Currently, pre-commit script modifies a property in each commit which touches any file in /project/trunk. So why do you need the last-changed revision of the project directory stored in the header file? Are you distributing sources directly from the repository, or are you using that number to identify builds? In either case, there are better ways to do that. 2. We have a few checks in pre-commit that are performed on text but not on binary files, and (unless it the type is set explicitly), the text is distinguished from binaries using simple heuristics. To avoid running this heuristics over and over, the result is saved into a property on that file. I'd just like to point out that Subversion already has heuristics to identify binary files. If you add a binary file to the working copy, it will automagically get the svn:mime-type set to application/octet-stream. There are only a few MIME types that Subversion does not consider binary. Of course, your heuristics might be different from Subversion's, but have you considered just looking at the mime-type property? So, to avoid modifying the transaction by pre-commit (that no longer reliably works in 1.7 and 1.8), Correction: that never reliably worked, because we never promised it would work. I am changing the pre-commit to list the 'tasks' to be performed by the post-commit - which will check in a new revision. I don't want to involve an out-of-repository storage for that list of tasks unless absolutely necessary - hence, revision property looks like the perfect place to store the follow-up tasks for a particular revision. Also I find your approach less than robust: * There's no guarantee that the post-commit hook will ever run, so it's a bad idea to rely on it for anything that's critical to your workflow. Can you please elaborate on this? I thought that if a transaction was promoted to a revision, the post-commit hook is always run. I understand that post-commit may fail and this failure will not roll back a revision. But when is it not run at all? It's entirely possible that the commit succeeds, but for any number of reasons -- power failure, full disk, simple misconfiguration -- the post-commit hook doesn't run. Subversion does not make any guarantees there other than making a best effort to run it. PS. I know that there's an interface, svn_fs_commit_txn, than can bypass both pre- and post-commit hook. But it is not used in regular commits from the command line, is it? Nope. * There's no guarantee that other commits won't happen before your post-commit hook is run; so whatever you do with the repository in post-commit may have to deal with conflicts, which is not fun to automate. I understand that, but I don't expect conflicts: the actions by the post-commit will only touch certain properties that are not set manually. After all, I can reject an attempt to set those properties in the pre-commit. You mean like this? $ svnadmin create repo $ svn co file://$(pwd)/repo wc1 Checked out revision 0. $ echo a wc1/foo $ svn add wc1/foo A wc1/foo $ svn ci -mm wc1 Adding wc1/foo Transmitting file data . Committed revision 1. $ svn co file://$(pwd)/repo wc2 Awc2/foo Checked out revision 1. $ echo b wc1/foo $ svn ci -mm wc1 Sendingwc1/foo Transmitting file data . Committed revision 2. $ svn propset x y wc2/foo property 'x' set on 'wc2/foo' $ svn ci -mm wc2 Sendingwc2/foo svn: E155011: Commit failed (details follow): svn: E155011: File '/Users/brane/src/test/wc2/foo' is out of date svn: E160028: File '/foo' is out of date -- Brane -- Branko Čibej | Director of Subversion WANdisco | Realising the impossibilities of Big Data e. br...@wandisco.com
Re: unversioned properties: size limitations?
On Tuesday, August 12, 2014 08:33:06 AM Branko Čibej wrote: On 12.08.2014 08:17, Alexey Neyman wrote: On Tuesday, August 12, 2014 07:44:03 AM Branko Čibej wrote: I'm still not sure what you're trying to achieve, though. Communication between pre- and post-commit hooks doesn't describe the problem, it describes a solution, and there are of course other ways for hooks to communicate that do not involve the repository. I've mentioned this in the other thread where you also responded. There are two problems that are currently (we're using 1.6) solved by modifying the transaction in the pre-commit hook: 1. We have a version.h header that needs to reflect the last modification date of *any* file in the project. Currently, pre-commit script modifies a property in each commit which touches any file in /project/trunk. So why do you need the last-changed revision of the project directory stored in the header file? Are you distributing sources directly from the repository, or are you using that number to identify builds? In either case, there are better ways to do that. Sometimes, yes. Not directly from a repository but engineers sometimes release semi- official builds to customers. I understand that is less than ideal way (not dealing with mixed-revision WCs, etc.). So, I am curious what are those better ways you mention? 2. We have a few checks in pre-commit that are performed on text but not on binary files, and (unless it the type is set explicitly), the text is distinguished from binaries using simple heuristics. To avoid running this heuristics over and over, the result is saved into a property on that file. I'd just like to point out that Subversion already has heuristics to identify binary files. If you add a binary file to the working copy, it will automagically get the svn:mime-type set to application/octet-stream. There are only a few MIME types that Subversion does not consider binary. Of course, your heuristics might be different from Subversion's, but have you considered just looking at the mime-type property? Yes, I've considered it, there are a few drawbacks: - First, the repository was imported from CVS and many binary files were not marked with the svn:mime-type properly set at the conversion time (most likely, because the oridinal CVS file didn't have -kb keyword expansion set - but that's now history). This could be solved by going over all the project files once and setting svn:mime-type explicitly, though. - Second, some of the MIME types are text even though the MIME type does not start with 'text/*'. image/svg, application/x-sh and application/xml are a few most obvious examples that come to mind. I don't know how to come up with an exhaustive list of such text-despite-MIME-type types. Because of these I decided to use a separate property. So, to avoid modifying the transaction by pre-commit (that no longer reliably works in 1.7 and 1.8), Correction: that never reliably worked, because we never promised it would work. We haven't noticed any issues with 1.4, 1.5 and 1.6 :) I am changing the pre-commit to list the 'tasks' to be performed by the post-commit - which will check in a new revision. I don't want to involve an out-of-repository storage for that list of tasks unless absolutely necessary - hence, revision property looks like the perfect place to store the follow-up tasks for a particular revision. Also I find your approach less than robust: * There's no guarantee that the post-commit hook will ever run, so it's a bad idea to rely on it for anything that's critical to your workflow. Can you please elaborate on this? I thought that if a transaction was promoted to a revision, the post-commit hook is always run. I understand that post-commit may fail and this failure will not roll back a revision. But when is it not run at all? It's entirely possible that the commit succeeds, but for any number of reasons -- power failure, full disk, simple misconfiguration -- the post-commit hook doesn't run. Subversion does not make any guarantees there other than making a best effort to run it. Well, these are force majeure sort of issues. For the uses I described, not running the post- commit is definitely not a deal-breaker for use case [2] (it will only result in non-caching the result of heuristics). For [1], I think I'll just add a monitoring cron job that will re-run post-commit hook if it detects a revision with the actions property from the pre-commit hook still set (which would mean that post-commit didn't run to completion). By the way, does SVN guarantee that such kinds of issues (full disk, power failure) do not result in repository ending up in an inconsistent state? If SVN does not offer such guarantees, it is pointless to require them from a post-commit script anyway. PS. I know that there's an
Re: unversioned properties: size limitations?
On 12.08.2014 09:26, Alexey Neyman wrote: On Tuesday, August 12, 2014 08:33:06 AM Branko Čibej wrote: So why do you need the last-changed revision of the project directory stored in the header file? Are you distributing sources directly from the repository, or are you using that number to identify builds? In either case, there are better ways to do that. Sometimes, yes. Not directly from a repository but engineers sometimes release semi-official builds to customers. I understand that is less than ideal way (not dealing with mixed-revision WCs, etc.). So, I am curious what are those better ways you mention? I'm sort of assuming a defined build (or packaging) workflow. If you have engineers mailing binaries from their mixed-revision development working copies to customers, all bets are off, since they might have locally modified your version property. One solution is to run svnversion as part of the package/build and include the result in the build artefacts. Can't go into any more detail than that, since I have no ides what your build system looks like. This won't always perfectly identify the configuration you're building, but at least it will give some indication of the state of the working copy, and whether (oh horror) it includes local modifications. I'd just like to point out that Subversion already has heuristics to identify binary files. If you add a binary file to the working copy, it will automagically get the svn:mime-type set to application/octet-stream. There are only a few MIME types that Subversion does not consider binary. Of course, your heuristics might be different from Subversion's, but have you considered just looking at the mime-type property? Yes, I've considered it, there are a few drawbacks: - First, the repository was imported from CVS and many binary files were not marked with the svn:mime-type properly set at the conversion time (most likely, because the oridinal CVS file didn't have -kb keyword expansion set - but that's now history). This could be solved by going over all the project files once and setting svn:mime-type explicitly, though. Right. - Second, some of the MIME types are text even though the MIME type does not start with 'text/*'. image/svg, application/x-sh and application/xml are a few most obvious examples that come to mind. I don't know how to come up with an exhaustive list of such text-despite-MIME-type types. Heh, tell me about it. Subversion has the same problem, and we still haven't found a satisfactory solution for it. Because of these I decided to use a separate property. So you really only have to add this property on added files, right? Have you considered using auto-props? They don't currently give you exactly the knob you need, because we don't have an auto-prop setting for all binary or all text files, but maybe that could solve your problem. Especially now that, with 1.8, you can define auto-prop settings in the repository and no longer need local configuration on every client. (Caveat: you do need 1.8 clients.) So, to avoid modifying the transaction by pre-commit (that no longer reliably works in 1.7 and 1.8), Correction: that never reliably worked, because we never promised it would work. We haven't noticed any issues with 1.4, 1.5 and 1.6 :) I always drive on the left side of the road, and I keep seeing all these traffic accidents. You wouldn't believe how many bad drivers there are out there. :) By the way, does SVN guarantee that such kinds of issues (full disk, power failure) do not result in repository ending up in an inconsistent state? If SVN does not offer such guarantees, it is pointless to require them from a post-commit script anyway. We do try to make the repository recoverable, yes. Guaranteeing that post-commit runs is not a priority, though. -- Brane -- Branko Čibej | Director of Subversion WANdisco | Realising the impossibilities of Big Data e. br...@wandisco.com
Re: unversioned properties: size limitations?
On Tuesday, August 12, 2014 09:43:39 AM Branko Čibej wrote: On 12.08.2014 09:26, Alexey Neyman wrote: On Tuesday, August 12, 2014 08:33:06 AM Branko Čibej wrote: So why do you need the last-changed revision of the project directory stored in the header file? Are you distributing sources directly from the repository, or are you using that number to identify builds? In either case, there are better ways to do that. Sometimes, yes. Not directly from a repository but engineers sometimes release semi-official builds to customers. I understand that is less than ideal way (not dealing with mixed-revision WCs, etc.). So, I am curious what are those better ways you mention? I'm sort of assuming a defined build (or packaging) workflow. If you have engineers mailing binaries from their mixed-revision development working copies to customers, all bets are off, since they might have locally modified your version property. One solution is to run svnversion as part of the package/build and include the result in the build artefacts. Can't go into any more detail than that, since I have no ides what your build system looks like. This won't always perfectly identify the configuration you're building, but at least it will give some indication of the state of the working copy, and whether (oh horror) it includes local modifications. 1. svnversion reports the revision of the check-out, not the revision of the last modification. 2. svnversion does not provide the $Date$ information. 3. svnversion won't report anything on 'svn export'-ed sources. All that would be handled if there were a way to modify a transaction being committed :) - Second, some of the MIME types are text even though the MIME type does not start with 'text/*'. image/svg, application/x-sh and application/xml are a few most obvious examples that come to mind. I don't know how to come up with an exhaustive list of such text-despite-MIME-type types. Heh, tell me about it. Subversion has the same problem, and we still haven't found a satisfactory solution for it. Well, if Subversion set an svn:binary to true/false just to report the heuristics, in addition to svn:mime-type, that would've helped somewhat. But Subversion heuristics only runs on newly added files. So you really only have to add this property on added files, right? Have you considered using auto-props? They don't currently give you exactly the knob you need, because we don't have an auto-prop setting for all binary or all text files, but maybe that could solve your problem. No, it is also set on existing files as they're modified. Especially now that, with 1.8, you can define auto-prop settings in the repository and no longer need local configuration on every client. (Caveat: you do need 1.8 clients.) Now you tell me about that. I am still trying to weed out 1.6 clients (which break horribly by nested 'svn mv')- since CentOS does not update the base version of the packages during the release lifetime, CentOS 6.x is stuck with Subversion 1.6. Thanks to WanDisco for providing the CentOS RPMs for Subversion 1.7/1.8 :) It would be nice if there were a 'start-access' script that is run on any access to the repository - that can be used to block such older clients from even accessing the repository. Speaking of 'wish' items: - How hard would be to implement client-side hooks? I know TortoiseSVN has them, but they're local - and to be really useful, they need to be propagated from the repository. And it would be nice to have them supported by the default commandline client. If they were supported, the tasks like the ones described above could be relegated to the client side, and the server-side hooks could then just reject the commit if the client failed to run the script. - How hard would be to allow modifications to a transaction by pre-commit? If it is just the issue of client-side caches going stale - can a shadow transaction be created that records all the modifications by the pre-commit script and then sends it back to the client to apply to the WC? Regards, Alexey.
Re: unversioned properties: size limitations?
On 12.08.2014 18:09, Alexey Neyman wrote: Speaking of 'wish' items: - How hard would be to implement client-side hooks? I know TortoiseSVN has them, but they're local - and to be really useful, they need to be propagated from the repository. And it would be nice to have them supported by the default commandline client. If they were supported, the tasks like the ones described above could be relegated to the client side, and the server-side hooks could then just reject the commit if the client failed to run the script. There are far too many reasons against them to count here, starting with security problems. I don't want to guess why TSVN supports them; I can guess that it has an easier time of it than the generic client, since TSVN is platform-specific. - How hard would be to allow modifications to a transaction by pre-commit? If it is just the issue of client-side caches going stale - can a shadow transaction be created that records all the modifications by the pre-commit script and then sends it back to the client to apply to the WC? For a start, this would require a major change in the wire protocol, where the server would, as a response to a successful commit, report any additional magic changes to the client. The problem with this is that it is error prone; the response may never arrive, for any number of reasons. Therefore, the client could not mark commtited items up-to-date until and unless it received the response. Since at least the DAV protocol is stateless, this implies all sorts of complications and the introduction of intermediate states in the working copy. In short: yes, it'd be hard. -- Brane -- Branko Čibej | Director of Subversion WANdisco | Realising the impossibilities of Big Data e. br...@wandisco.com
Re: unversioned properties: size limitations?
I intended to reply to your other thread given roughly the same svnversion advice that Branko has already given you. On 8/12/14 9:09 AM, Alexey Neyman wrote: 1. svnversion reports the revision of the check-out, not the revision of the last modification. Use one of the alternatives that behaves the way you want. To save retyping info I've already provided elsewhere see this: https://stackoverflow.com/questions/21447373/how-to-get-correct-svn-version-number-on-unix/21448972#21448972 For reference SubWCRev is something that comes with TortoiseSVN: http://tortoisesvn.net/docs/nightly/TortoiseSVN_en/tsvn-subwcrev.html 2. svnversion does not provide the $Date$ information. Patches welcome as always. If you don't want to write the patch you can always ask for it. 3. svnversion won't report anything on 'svn export'-ed sources. Your properties won't be available either. If you script the export then you know what the revision you're exporting is and can determine the information by simply doing `svn log --limit 1` on the same target (path and revision) as you exported. In order to make sure there are no changes between the export and the log if you're not using a revision number you'll probably want to look up the last changed revision using something like the log command given above and then export that revision. All that would be handled if there were a way to modify a transaction being committed :) As Branko has pointed out if you have a mixed revision working copy even your solution doesn't work because you don't necessarily have a state that ever existed in the repository. svnversion exists partly to help resolve that situation. It's difficult to give advice beyond this because I have no idea what language you're working in, what your build system looks like, what you're actually shipping (source or binaries), etc... As I urged you before, you'd be better off asking on this list for advice to solve the problem you're trying to solve than asking about problems you're having with the solutions. This is not an uncommon problem people need to solve. It gets discussed periodically. There are no perfect solutions. See X-Y Problem: http://mywiki.wooledge.org/XyProblem
Re: unversioned properties: size limitations?
On 8/12/14 9:31 AM, Branko Čibej wrote: For a start, this would require a major change in the wire protocol, where the server would, as a response to a successful commit, report any additional magic changes to the client. The problem with this is that it is error prone; the response may never arrive, for any number of reasons. Therefore, the client could not mark commtited items up-to-date until and unless it received the response. Since at least the DAV protocol is stateless, this implies all sorts of complications and the introduction of intermediate states in the working copy. In short: yes, it'd be hard. Ignoring all that... There's a better reason why it won't happen. Subversion clients before whatever version we add it to won't support it. Which leaves those clients with stale caches. You'd have to disallow it with older clients or just ignore the stale cache problem. I frankly do not see the community accepting a change that ignores such a huge problem with old clients. This is the type of change that in my opinion would require a major bump, which is unlikely to happen and even more unlikely to happen for this.
Re: unversioned properties: size limitations?
On Tuesday, August 12, 2014 11:49:39 am Ben Reser wrote: I intended to reply to your other thread given roughly the same svnversion advice that Branko has already given you. On 8/12/14 9:09 AM, Alexey Neyman wrote: 1. svnversion reports the revision of the check-out, not the revision of the last modification. Use one of the alternatives that behaves the way you want. To save retyping info I've already provided elsewhere see this: https://stackoverflow.com/questions/21447373/how-to-get-correct-svn-version -number-on-unix/21448972#21448972 For reference SubWCRev is something that comes with TortoiseSVN: http://tortoisesvn.net/docs/nightly/TortoiseSVN_en/tsvn-subwcrev.html Actually, it's even easier than that. I somehow missed the -c option to svnversion. 2. svnversion does not provide the $Date$ information. Patches welcome as always. If you don't want to write the patch you can always ask for it. I think I can write such patch myself :) 3. svnversion won't report anything on 'svn export'-ed sources. Your properties won't be available either. Properties won't, but expanded keywords will be present in 'svn export'. $Revision$ will change in response to a property change. Please consider these questions closed. For the 'last changed rev/date' I'll modify the 'svnversion -c' and will submit a patch to support reporting the dates as well (hopefully, it can then be applied to 1.8.x branch as well) and for the caching the results of file analysis, I'll use a post-commit script (since not performing that action does not compromise the integrity of the project in the repository). Regards. Alexey.
Re: unversioned properties: size limitations?
On 8/12/14 12:26 PM, Alexey Neyman wrote: Please consider these questions closed. For the 'last changed rev/date' I'll modify the 'svnversion -c' and will submit a patch to support reporting the dates as well (hopefully, it can then be applied to 1.8.x branch as well) and for the caching the results of file analysis, I'll use a post-commit script (since not performing that action does not compromise the integrity of the project in the repository). This isn't something we can backport. We don't add features on release branches. But 1.9.x is wrapping up soon, so if it turns out to be simple there's a chance it could make into a release there. Otherwise it'd have to wait until 1.10.x.
Re: unversioned properties: size limitations?
On Tuesday, August 12, 2014 11:53:09 AM Ben Reser wrote: On 8/12/14 9:31 AM, Branko Čibej wrote: For a start, this would require a major change in the wire protocol, where the server would, as a response to a successful commit, report any additional magic changes to the client. The problem with this is that it is error prone; the response may never arrive, for any number of reasons. Therefore, the client could not mark commtited items up-to-date until and unless it received the response. Since at least the DAV protocol is stateless, this implies all sorts of complications and the introduction of intermediate states in the working copy. In short: yes, it'd be hard. Ignoring all that... There's a better reason why it won't happen. Subversion clients before whatever version we add it to won't support it. Which leaves those clients with stale caches. You'd have to disallow it with older clients or just ignore the stale cache problem. I frankly do not see the community accepting a change that ignores such a huge problem with old clients. Isn't that the same kind of change that happened with version 1.5 and mergeinfo? If one wanted to use mergeinfo, one had to have 1.5+ clients. A capability reporting was added, and a server can check that only mergeinfo-capable clients can start a commit. Same here, if a repository administrator wants to have pre-commit scripts that modify a transaction, he'd better check the clients' ability to handle a change to be applied to WC in server's response. Regards, Alexey.
Re: unversioned properties: size limitations?
On 8/12/14 7:02 PM, Alexey Neyman wrote: Isn't that the same kind of change that happened with version 1.5 and mergeinfo? If one wanted to use mergeinfo, one had to have 1.5+ clients. A capability reporting was added, and a server can check that only mergeinfo-capable clients can start a commit. Same here, if a repository administrator wants to have pre-commit scripts that modify a transaction, he'd better check the clients' ability to handle a change to be applied to WC in server's response. If a server admin changes a transaction and the commit came from an old client then the working copy is potentially broken. If an old client doesn't know about mergeinfo it doesn't have access to that new feature. New clients lose awareness of any merges committed by the old clients but neither side is really broken. You just may be inconvenienced when merging. I think there's a huge difference there.
Re: unversioned properties: size limitations?
On Tuesday, August 12, 2014 10:18:48 PM Ben Reser wrote: On 8/12/14 7:02 PM, Alexey Neyman wrote: Isn't that the same kind of change that happened with version 1.5 and mergeinfo? If one wanted to use mergeinfo, one had to have 1.5+ clients. A capability reporting was added, and a server can check that only mergeinfo-capable clients can start a commit. Same here, if a repository administrator wants to have pre-commit scripts that modify a transaction, he'd better check the clients' ability to handle a change to be applied to WC in server's response. If a server admin changes a transaction and the commit came from an old client then the working copy is potentially broken. If an old client doesn't know about mergeinfo it doesn't have access to that new feature. New clients lose awareness of any merges committed by the old clients but neither side is really broken. You just may be inconvenienced when merging. I think there's a huge difference there. I think this thread quickly turns into a discussion how to roll out a feature that's not going to be implemented in the foreseeable future. I still think there would be feasible ways: Leaving it up to repository admin's configuration of start-commit - which is no worse than the current behavior if that repository's pre-commit modifies a transaction. OR Rejecting modifications to a transaction if the transaction was originated by non-aware client. OR Rejecting to promote the transaction into the revision if it came from non-aware client and was modified by pre-commit. It is pointless to discuss which if these approaches is better and/or acceptable, though, unless there is a decision to implement this feature. Regards, Alexey
Re: unversioned properties: size limitations?
On 12.08.2014 03:31, Alexey Neyman wrote: Hi SVN users/developers, Is there a limitation in size on the property value that can be set? Any scalability traps to be aware of (i.e. non-linear increase in time due to increase in size of the property value)? I tried a 4Mb property, seems to work fine... One thing to be aware of is that properties were never designed to be large. Property values are always transmitted in full text between client and server (i.e., they're not compressed); they're stored in full text in the repository (not deltified the way file contents are). So the more large properties you have, and the more often you modify them, the less efficient your repository will be, in terms of storage requirements and network bandwidth. So while you should be able to store a 2 gig property value, I really, really recommend not to do that. -- Brane -- Branko Čibej | Director of Subversion WANdisco | Realising the impossibilities of Big Data e. br...@wandisco.com
Re: unversioned properties: size limitations?
On Tuesday, August 12, 2014 06:59:20 AM Branko Čibej wrote: On 12.08.2014 03:31, Alexey Neyman wrote: Hi SVN users/developers, Is there a limitation in size on the property value that can be set? Any scalability traps to be aware of (i.e. non-linear increase in time due to increase in size of the property value)? I tried a 4Mb property, seems to work fine... One thing to be aware of is that properties were never designed to be large. Property values are always transmitted in full text between client and server (i.e., they're not compressed); they're stored in full text in the repository (not deltified the way file contents are). So the more large properties you have, and the more often you modify them, the less efficient your repository will be, in terms of storage requirements and network bandwidth. So while you should be able to store a 2 gig property value, I really, really recommend not to do that. I thought of having pre- and post-commit hooks communicate using a *revision* property: pre-commit hook would set a revision property with the list of files and actions to be performed on them, and the post-commit hook will perform these actions by committing a new revision (instead of modifying a transaction by pre-commit hook). Thus a more specific question - when are arbitrary *revision* properties sent from the client to the server? Obviously, svn:* properties are used by various SVN commands; but am I right to assume that non-standard revision properties are sent only for the 'svn pg -- revprop' command? That said, I expect the property value to be much less than 2Gb. So far, the largest commit we've had was ~2 files - with ~150 characters per path, that would be about 3Mb for the property value. Regards, Alexey.
Re: unversioned properties: size limitations?
On 12.08.2014 07:16, Alexey Neyman wrote: On Tuesday, August 12, 2014 06:59:20 AM Branko Čibej wrote: On 12.08.2014 03:31, Alexey Neyman wrote: Hi SVN users/developers, Is there a limitation in size on the property value that can be set? Any scalability traps to be aware of (i.e. non-linear increase in time due to increase in size of the property value)? I tried a 4Mb property, seems to work fine... One thing to be aware of is that properties were never designed to be large. Property values are always transmitted in full text between client and server (i.e., they're not compressed); they're stored in full text in the repository (not deltified the way file contents are). So the more large properties you have, and the more often you modify them, the less efficient your repository will be, in terms of storage requirements and network bandwidth. So while you should be able to store a 2 gig property value, I really, really recommend not to do that. I thought of having pre- and post-commit hooks communicate using a *revision* property: pre-commit hook would set a revision property with the list of files and actions to be performed on them, and the post-commit hook will perform these actions by committing a new revision (instead of modifying a transaction by pre-commit hook). Thus a more specific question - when are arbitrary *revision* properties sent from the client to the server? Obviously, svn:* properties are used by various SVN commands; but am I right to assume that non-standard revision properties are sent only for the 'svn pg --revprop' command? That said, I expect the property value to be much less than 2Gb. So far, the largest commit we've had was ~2 files - with ~150 characters per path, that would be about 3Mb for the property value. Sure, you'll only transmit revprops with propget --revprop and propset --revprop. I'm not sure what the implications are of storing large values in revprops, these are handled a bit differently than versioned properties on the server. And of course, revision properties are not versioned. I'm still not sure what you're trying to achieve, though. Communication between pre- and post-commit hooks doesn't describe the problem, it describes a solution, and there are of course other ways for hooks to communicate that do not involve the repository. Also I find your approach less than robust: * There's no guarantee that the post-commit hook will ever run, so it's a bad idea to rely on it for anything that's critical to your workflow. * There's no guarantee that other commits won't happen before your post-commit hook is run; so whatever you do with the repository in post-commit may have to deal with conflicts, which is not fun to automate. Why don't you just describe what kind of workflow you're implementing, and more importanly, why? -- Brane -- Branko Čibej | Director of Subversion WANdisco | Realising the impossibilities of Big Data e. br...@wandisco.com
