Re: Managing modifications to an open source product
On Fri, 15 Oct 2010 00:08:28 +0200, Stefan Sperling wrote: > On Thu, Oct 14, 2010 at 09:17:59PM +, Dan Nessett wrote: >> My question has to do with bypassing the fetch of the complete new MW >> version into our repository. (NB: MW revisions run about 150MB). Let me >> describe this in more detail. >> >> + When we wish to upgrade to a new MW revision, check out the directory >> in *our* repository that contains our modified files (and only those >> files). >> >> + Merge each of these files with those in the new MW revision. (NB: >> there are only a very few of these). >> >> + After resolving any conflicts, commit these changes into a new >> revision in *our* repository, probably tagging it so it is easy to >> reference. >> >> + svn mkdir to create an empty workspace. >> >> + With svn propedit add an svn:externals property to this directory so >> that the whole source effectively is an external fetch of the >> appropriate MW revision. >> >> + With svn propedit add lines to the svn:externals definition in the >> top- level directory that describe the files we just merged, >> instructing subversion to get these from the new branch we just created >> in *our* repository. (Note: this is where the "overload" occurs, since >> files with the same name and at the same location in the MW repository >> revision exist. We don't want fetch these - we want a checkout to get >> the merged files from *our* repository instead.) >> >> + Commit this to *our* repository under a new tag. >> >> Will this work? > > Unfortunately, this won't work. You cannot use externals like this. The > existing files are in the way. > > But I may be wrong, or maybe I misunderstand your idea. In general, with > a complex tool like Subversion the best way to see if something will > work is to just try it out and see. > >> If so, are there some reasons why it isn't a good idea. > > There's nothing wrong with your idea itself. But Subversion doesn't > support an "overlay" concept like you want to realise with externals. > > Also, I'd recommend not using file externals for anything, because their > implementation is a hack and there are several known issues: > http://subversion.tigris.org/issues/show_bug.cgi?id=3351 > http://subversion.tigris.org/issues/show_bug.cgi?id=3589 > http://subversion.tigris.org/issues/show_bug.cgi?id=3665 > http://subversion.tigris.org/issues/show_bug.cgi?id=3563 Directory > externals are fine though. > > The problem you're trying to solve (vendor branching) isn't easy, and > many have tried before and devised solutions. So I'd suggest that before > trying to devise schemes of your own, shop around for existing > solutions. One of them will probably fit the bill. > > The existing alternatives I know about are: > > - Subversion's vendor branching with svn_load_dirs. > This works fine in general, but handling of renamed files is a bit > clunky. It's what the svnbook describes. > > - Play with Subversion's foreign repository merges. > This is a little-known feature that could help you track the > MediaWiki changes, as long as MediaWiki keeps using Subversion. See > http://blogs.open.collab.net/svn/2008/03/do-not-post-mer.html (which > was written before the 1.5 release but is still relevant). You'd do > something like this in the working copy of your custom MW code: > svn merge \ > http://svn.wikimedia.org/svnroot/mediawiki/trunk/@74770 \ > http://svn.wikimedia.org/svnroot/mediawiki/trunk/@74798 > > - Try Piston. I've never tried it myself, but its interface seems nice > and I've seen it being recommended on this list before. The > dependency on Ruby isn't nice but quite possibly worth it. > > Good luck, > Stefan Thanks. Your description of alternative approaches was very helpful. Regards, -- -- Dan Nessett
Re: Managing modifications to an open source product
On Thu, Oct 14, 2010 at 09:48:21PM +, Dan Nessett wrote: > Bad etiquette to answer my own question, but I just reread the Externals > Definition section of the book and found this: > > "An externals definition can point only to directories, not to files." > So, what I have described won't work even if overloading with > svn:externals is supported. You're probably reading an outdated version of the book. See the nightly edition: http://svnbook.red-bean.com/nightly/en/svn.advanced.externals.html File externals are indeed supported as of 1.6, with the caveats I listed in my other reply (those problems aren't documented in the book). The current plan is to fix file externals in the upcoming 1.7 release. But, as I mentioned, externals cannot be used as "overlays". Stefan
Re: Managing modifications to an open source product
On Thu, Oct 14, 2010 at 09:17:59PM +, Dan Nessett wrote: > My question has to do with bypassing the fetch of the complete new MW > version into our repository. (NB: MW revisions run about 150MB). Let me > describe this in more detail. > > + When we wish to upgrade to a new MW revision, check out the directory > in *our* repository that contains our modified files (and only those > files). > > + Merge each of these files with those in the new MW revision. (NB: there > are only a very few of these). > > + After resolving any conflicts, commit these changes into a new revision > in *our* repository, probably tagging it so it is easy to reference. > > + svn mkdir to create an empty workspace. > > + With svn propedit add an svn:externals property to this directory so > that the whole source effectively is an external fetch of the appropriate > MW revision. > > + With svn propedit add lines to the svn:externals definition in the top- > level directory that describe the files we just merged, instructing > subversion to get these from the new branch we just created in *our* > repository. (Note: this is where the "overload" occurs, since files with > the same name and at the same location in the MW repository revision > exist. We don't want fetch these - we want a checkout to get the merged > files from *our* repository instead.) > > + Commit this to *our* repository under a new tag. > > Will this work? Unfortunately, this won't work. You cannot use externals like this. The existing files are in the way. But I may be wrong, or maybe I misunderstand your idea. In general, with a complex tool like Subversion the best way to see if something will work is to just try it out and see. > If so, are there some reasons why it isn't a good idea. There's nothing wrong with your idea itself. But Subversion doesn't support an "overlay" concept like you want to realise with externals. Also, I'd recommend not using file externals for anything, because their implementation is a hack and there are several known issues: http://subversion.tigris.org/issues/show_bug.cgi?id=3351 http://subversion.tigris.org/issues/show_bug.cgi?id=3589 http://subversion.tigris.org/issues/show_bug.cgi?id=3665 http://subversion.tigris.org/issues/show_bug.cgi?id=3563 Directory externals are fine though. The problem you're trying to solve (vendor branching) isn't easy, and many have tried before and devised solutions. So I'd suggest that before trying to devise schemes of your own, shop around for existing solutions. One of them will probably fit the bill. The existing alternatives I know about are: - Subversion's vendor branching with svn_load_dirs. This works fine in general, but handling of renamed files is a bit clunky. It's what the svnbook describes. - Play with Subversion's foreign repository merges. This is a little-known feature that could help you track the MediaWiki changes, as long as MediaWiki keeps using Subversion. See http://blogs.open.collab.net/svn/2008/03/do-not-post-mer.html (which was written before the 1.5 release but is still relevant). You'd do something like this in the working copy of your custom MW code: svn merge \ http://svn.wikimedia.org/svnroot/mediawiki/trunk/@74770 \ http://svn.wikimedia.org/svnroot/mediawiki/trunk/@74798 - Try Piston. I've never tried it myself, but its interface seems nice and I've seen it being recommended on this list before. The dependency on Ruby isn't nice but quite possibly worth it. Good luck, Stefan
Re: Managing modifications to an open source product
Dan, This isn't that much help but are you aware that svn export is exactly the same as svn checkout & run a script to delete all the .svn folders ...? The problem with stripping .svn from vendor checkout is then you cannot update it; you're forced to export the whole tree every time. The method of having a working copy somewhere pointing to the vendor files, that you can svn update whenever you like, and then diff the changes into your tree is the easiest way to accomplish what you want, I think. What I would recommend is that you have two branches for this (plus trunk of your code): [your stuff] => trunk-working-copy [their stuff] => from-vendor-svn [intermediate] = svn copy [your stuff -r HEAD] Then at any time you can: svn up [their stuff] svn copy [your stuff] => [intermediate] (so you can do your diff/merge to an intermediate branch) test on [intermediate] if it passes merge it to trunk, delete [intermediate] Bear in mind, not sure what OS you're on, but e.g. on Windows using Araxis merge, merging two or three trees is far (several orders of magnitude) simpler than using svn diff. HTH, Geoff
Re: Managing modifications to an open source product
On Thu, 14 Oct 2010 21:17:59 +, Dan Nessett wrote: > On Thu, 14 Oct 2010 21:55:08 +0200, Olivier Sannier wrote: > >> On 14/10/2010 21:45, Dan Nessett wrote: >>> On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote: >>> > I develop for a site that uses Mediawiki (MW). We make some > modifications to it before deployment. Generally, (using subversion) > we check out a tagged version into a workspace, recursively delete > the .svn directories, modify a small number of files, add some of > our own extensions, and then commit the result into our own > repository. We then work with the source from there. > > This approach means we have to track MW bug-fixes and add them to > our modified version. I was wondering if there is a better way to > accomplish the same objective. For example, we can use the > svn:externals property to point to the MW repository version of the > extensions we use, so each time they are updated, all we need to do > is svn up on the externals directory. > > The main source is a different story. Since we modify some of the > files (and have no commit privileges to the MW repository), the > files we modify are not within our purview to change (and > understandably the MW people wouldn't allow it even if we had commit > privileges). > > Is there any way to use the svn:externals property to solve the main > source issue? For example, could we point the revision we keep in > our main repository to the correct revision in the MW repository and > then tag the appropriate directories that contain the files we > modify with svn:external. These latter svn:external properties would > name the individual files we modify and point to the modified > version that we could keep in our repository. My concern is we are > "overloading" the files in the MW repository with files in our > repository and I am not sure subversion allows that. > > There is a whole section in the svn book about this... http://svnbook.red-bean.com/nightly/en/svn- >>> book.html#svn.advanced.vendorbr BOb >>> I have read this section. It is about vendor drops, but it doesn't >>> answer the question I asked. Basically, we are doing a vendor drop >>> now. >>> >>> Dan >> >> >> It actually does, but it's a bit hard to understand. Reread it >> carefully and you'll see that you actually track the public changes in >> a branch that you later merge back into your trunk. > > I just reread the Vendor drop section in the book. It doesn't actually > address the question I raised (which is about overloading files obtained > with the svn:external definition). We don't do exactly what is described > in the book, but logically it is the same. Instead of importing the > vendor drop into the repository, we import it into a workspace (manually > by checking it out and then recursively deleting the .svn directories. > Perhaps we should use the svn import command, but that is a side issue). > We then do the merge in the workspace and commit to a new version in the > repository. > > My question has to do with bypassing the fetch of the complete new MW > version into our repository. (NB: MW revisions run about 150MB). Let me > describe this in more detail. > > + When we wish to upgrade to a new MW revision, check out the directory > in *our* repository that contains our modified files (and only those > files). > > + Merge each of these files with those in the new MW revision. (NB: > there are only a very few of these). > > + After resolving any conflicts, commit these changes into a new > revision in *our* repository, probably tagging it so it is easy to > reference. > > + svn mkdir to create an empty workspace. > > + With svn propedit add an svn:externals property to this directory so > that the whole source effectively is an external fetch of the > appropriate MW revision. > > + With svn propedit add lines to the svn:externals definition in the > top- level directory that describe the files we just merged, instructing > subversion to get these from the new branch we just created in *our* > repository. (Note: this is where the "overload" occurs, since files with > the same name and at the same location in the MW repository revision > exist. We don't want fetch these - we want a checkout to get the merged > files from *our* repository instead.) > > + Commit this to *our* repository under a new tag. > > Will this work? If so, are there some reasons why it isn't a good idea. Bad etiquette to answer my own question, but I just reread the Externals Definition section of the book and found this: "An externals definition can point only to directories, not to files." So, what I have described won't work even if overloading with svn:externals is supported. -- -- Dan Nessett
Re: Managing modifications to an open source product
On Thu, 14 Oct 2010 21:55:08 +0200, Olivier Sannier wrote: > On 14/10/2010 21:45, Dan Nessett wrote: >> On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote: >> I develop for a site that uses Mediawiki (MW). We make some modifications to it before deployment. Generally, (using subversion) we check out a tagged version into a workspace, recursively delete the .svn directories, modify a small number of files, add some of our own extensions, and then commit the result into our own repository. We then work with the source from there. This approach means we have to track MW bug-fixes and add them to our modified version. I was wondering if there is a better way to accomplish the same objective. For example, we can use the svn:externals property to point to the MW repository version of the extensions we use, so each time they are updated, all we need to do is svn up on the externals directory. The main source is a different story. Since we modify some of the files (and have no commit privileges to the MW repository), the files we modify are not within our purview to change (and understandably the MW people wouldn't allow it even if we had commit privileges). Is there any way to use the svn:externals property to solve the main source issue? For example, could we point the revision we keep in our main repository to the correct revision in the MW repository and then tag the appropriate directories that contain the files we modify with svn:external. These latter svn:external properties would name the individual files we modify and point to the modified version that we could keep in our repository. My concern is we are "overloading" the files in the MW repository with files in our repository and I am not sure subversion allows that. >>> There is a whole section in the svn book about this... >>> >>> http://svnbook.red-bean.com/nightly/en/svn- >> book.html#svn.advanced.vendorbr >>> BOb >> I have read this section. It is about vendor drops, but it doesn't >> answer the question I asked. Basically, we are doing a vendor drop now. >> >> Dan > > > It actually does, but it's a bit hard to understand. Reread it carefully > and you'll see that you actually track the public changes in a branch > that you later merge back into your trunk. I just reread the Vendor drop section in the book. It doesn't actually address the question I raised (which is about overloading files obtained with the svn:external definition). We don't do exactly what is described in the book, but logically it is the same. Instead of importing the vendor drop into the repository, we import it into a workspace (manually by checking it out and then recursively deleting the .svn directories. Perhaps we should use the svn import command, but that is a side issue). We then do the merge in the workspace and commit to a new version in the repository. My question has to do with bypassing the fetch of the complete new MW version into our repository. (NB: MW revisions run about 150MB). Let me describe this in more detail. + When we wish to upgrade to a new MW revision, check out the directory in *our* repository that contains our modified files (and only those files). + Merge each of these files with those in the new MW revision. (NB: there are only a very few of these). + After resolving any conflicts, commit these changes into a new revision in *our* repository, probably tagging it so it is easy to reference. + svn mkdir to create an empty workspace. + With svn propedit add an svn:externals property to this directory so that the whole source effectively is an external fetch of the appropriate MW revision. + With svn propedit add lines to the svn:externals definition in the top- level directory that describe the files we just merged, instructing subversion to get these from the new branch we just created in *our* repository. (Note: this is where the "overload" occurs, since files with the same name and at the same location in the MW repository revision exist. We don't want fetch these - we want a checkout to get the merged files from *our* repository instead.) + Commit this to *our* repository under a new tag. Will this work? If so, are there some reasons why it isn't a good idea. -- -- Dan Nessett
Re: Managing modifications to an open source product
On Thu, 14 Oct 2010 22:33:04 +0200, Stefan Sperling wrote: > On Thu, Oct 14, 2010 at 09:55:08PM +0200, Olivier Sannier wrote: >> On 14/10/2010 21:45, Dan Nessett wrote: >> >On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote: >> >>>Generally, (using subversion) we >> >>>check out a tagged version into a workspace, recursively delete the >> >>>.svn directories, modify a small number of files, add some of our >> >>>own extensions, and then commit the result into our own repository. >> >>>We then work with the source from there. > >> >>There is a whole section in the svn book about this... >> >> >> >>http://svnbook.red-bean.com/nightly/en/svn- book.html#svn.advanced.vendorbr > >> >I have read this section. It is about vendor drops, but it doesn't >> >answer the question I asked. Basically, we are doing a vendor drop >> >now. > > It doesn't sound like you're using the approach described in the book. > > With vendor branching, you work with 2 distinct branches, one containing > pristine upstream vendor code, and one containing the modified version. > Rather than adding the upstream source code mixed up with your own > modifications directly. > > You might also want to take a look at Piston: > http://piston.rubyforge.org/ > > Stefan Yes, you're right. We don't store an exact copy of each drop in our repository. Instead, we copy the drop into a workspace (effectively importing by checking out and then recursively deleting the .svn directories), merge there and commit to the new version. We couldn't understand why we need to keep an exact copy of the vendor drop in our repository. Piston looks interesting, but it also appears to require ruby. This isn't a show stopper, but it would be nice if we could do what we want to do with subversion only. If that isn't possible, or it is too complex, then Piston might be the right way to go. Dan -- -- Dan Nessett
Re: Managing modifications to an open source product
On Thu, Oct 14, 2010 at 09:55:08PM +0200, Olivier Sannier wrote: > On 14/10/2010 21:45, Dan Nessett wrote: > >On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote: > >>>Generally, (using subversion) we > >>>check out a tagged version into a workspace, recursively delete the > >>>.svn directories, modify a small number of files, add some of our own > >>>extensions, and then commit the result into our own repository. We then > >>>work with the source from there. > >>There is a whole section in the svn book about this... > >> > >>http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.advanced.vendorbr > >I have read this section. It is about vendor drops, but it doesn't answer > >the question I asked. Basically, we are doing a vendor drop now. It doesn't sound like you're using the approach described in the book. With vendor branching, you work with 2 distinct branches, one containing pristine upstream vendor code, and one containing the modified version. Rather than adding the upstream source code mixed up with your own modifications directly. You might also want to take a look at Piston: http://piston.rubyforge.org/ Stefan
Re: Managing modifications to an open source product
On 14/10/2010 21:45, Dan Nessett wrote: On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote: I develop for a site that uses Mediawiki (MW). We make some modifications to it before deployment. Generally, (using subversion) we check out a tagged version into a workspace, recursively delete the .svn directories, modify a small number of files, add some of our own extensions, and then commit the result into our own repository. We then work with the source from there. This approach means we have to track MW bug-fixes and add them to our modified version. I was wondering if there is a better way to accomplish the same objective. For example, we can use the svn:externals property to point to the MW repository version of the extensions we use, so each time they are updated, all we need to do is svn up on the externals directory. The main source is a different story. Since we modify some of the files (and have no commit privileges to the MW repository), the files we modify are not within our purview to change (and understandably the MW people wouldn't allow it even if we had commit privileges). Is there any way to use the svn:externals property to solve the main source issue? For example, could we point the revision we keep in our main repository to the correct revision in the MW repository and then tag the appropriate directories that contain the files we modify with svn:external. These latter svn:external properties would name the individual files we modify and point to the modified version that we could keep in our repository. My concern is we are "overloading" the files in the MW repository with files in our repository and I am not sure subversion allows that. There is a whole section in the svn book about this... http://svnbook.red-bean.com/nightly/en/svn- book.html#svn.advanced.vendorbr BOb I have read this section. It is about vendor drops, but it doesn't answer the question I asked. Basically, we are doing a vendor drop now. Dan It actually does, but it's a bit hard to understand. Reread it carefully and you'll see that you actually track the public changes in a branch that you later merge back into your trunk.
Re: Managing modifications to an open source product
On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote: >> I develop for a site that uses Mediawiki (MW). We make some >> modifications to it before deployment. Generally, (using subversion) we >> check out a tagged version into a workspace, recursively delete the >> .svn directories, modify a small number of files, add some of our own >> extensions, and then commit the result into our own repository. We then >> work with the source from there. >> >> This approach means we have to track MW bug-fixes and add them to our >> modified version. I was wondering if there is a better way to >> accomplish the same objective. For example, we can use the >> svn:externals property to point to the MW repository version of the >> extensions we use, so each time they are updated, all we need to do is >> svn up on the externals directory. >> >> The main source is a different story. Since we modify some of the files >> (and have no commit privileges to the MW repository), the files we >> modify are not within our purview to change (and understandably the MW >> people wouldn't allow it even if we had commit privileges). >> >> Is there any way to use the svn:externals property to solve the main >> source issue? For example, could we point the revision we keep in our >> main repository to the correct revision in the MW repository and then >> tag the appropriate directories that contain the files we modify with >> svn:external. These latter svn:external properties would name the >> individual files we modify and point to the modified version that we >> could keep in our repository. My concern is we are "overloading" the >> files in the MW repository with files in our repository and I am not >> sure subversion allows that. >> >> > There is a whole section in the svn book about this... > > http://svnbook.red-bean.com/nightly/en/svn- book.html#svn.advanced.vendorbr > > BOb I have read this section. It is about vendor drops, but it doesn't answer the question I asked. Basically, we are doing a vendor drop now. Dan -- -- Dan Nessett
RE: Managing modifications to an open source product
> I develop for a site that uses Mediawiki (MW). We make some > modifications to it before deployment. Generally, (using > subversion) we check out a tagged version into a workspace, > recursively delete the .svn directories, modify a small number of > files, add some of our own extensions, and then commit the result > into our own repository. We then work with the source from there. > > This approach means we have to track MW bug-fixes and add them to > our modified version. I was wondering if there is a better way to > accomplish the same objective. For example, we can use the > svn:externals property to point to the MW repository version of the > extensions we use, so each time they are updated, all we need to do > is svn up on the externals directory. > > The main source is a different story. Since we modify some of the > files (and have no commit privileges to the MW repository), the > files we modify are not within our purview to change (and > understandably the MW people wouldn't allow it even if we had > commit privileges). > > Is there any way to use the svn:externals property to solve the > main source issue? For example, could we point the revision we keep > in our main repository to the correct revision in the MW repository > and then tag the appropriate directories that contain the files we > modify with svn:external. These latter svn:external properties > would name the individual files we modify and point to the modified > version that we could keep in our repository. My concern is we are > "overloading" the files in the MW repository with files in our > repository and I am not sure subversion allows that. > There is a whole section in the svn book about this... http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.advanced.vendorbr BOb
Managing modifications to an open source product
I develop for a site that uses Mediawiki (MW). We make some modifications to it before deployment. Generally, (using subversion) we check out a tagged version into a workspace, recursively delete the .svn directories, modify a small number of files, add some of our own extensions, and then commit the result into our own repository. We then work with the source from there. This approach means we have to track MW bug-fixes and add them to our modified version. I was wondering if there is a better way to accomplish the same objective. For example, we can use the svn:externals property to point to the MW repository version of the extensions we use, so each time they are updated, all we need to do is svn up on the externals directory. The main source is a different story. Since we modify some of the files (and have no commit privileges to the MW repository), the files we modify are not within our purview to change (and understandably the MW people wouldn't allow it even if we had commit privileges). Is there any way to use the svn:externals property to solve the main source issue? For example, could we point the revision we keep in our main repository to the correct revision in the MW repository and then tag the appropriate directories that contain the files we modify with svn:external. These latter svn:external properties would name the individual files we modify and point to the modified version that we could keep in our repository. My concern is we are "overloading" the files in the MW repository with files in our repository and I am not sure subversion allows that. Regards, Dan Nessett
