Re: Upstream source handling
Bastian, I'm really sorry I've taken so long to look at this. On Tue, 2015-09-08 at 22:52 +0200, Bastian Blank wrote: > Moin > > During the linux packaging BoF at DebConf, Ben asked for usefull > upstream source handling. No compeling ones were mentioned. > > Some years ago (yes, years), I proposed some schema based on submodules, > but never got around to actually implement it. I finally managed to do > an initial implementation. It currently lives in the linux.git in > branch waldi/submodule. > > Main principles: > - source of each featureset is a submodule, using version specific > branches > - orig source via version specific tag So genorig.py would be replaced with 'git archive'? > Workflow: > - New upstream version > - Rebase old tag on top of new upstream, tag it with the new version There are no tags in the repository you created. Can you add one as an example? Do you think we would tag when updating to a new upstream version, or only when making the first upload with a new orig tarball (allowing for further DFSG changes before uploading)? > - Rebase old main featureset branch on top of new orig tag as new > branch > - Rebase old other featureset branches on top of new main featureset > branch or replace with new base and create new branch > - Record new top commits and update changelog in main repo That's a lot of rebasing, though not so different from what we do now to refresh the patch series. Presumably we would add a script to support this and ensure it is all done properly? The submodules are checked out with detached heads by default. Is your intent that we would override this? I tried this: $ cd source/orig $ git checkout orig/4.2-rc8 For some reason, this deleted upstream files in the working tree - maybe because of the '*' in .gitignore? ... Branch orig/4.2-rc8 set up to track remote branch orig/4.2-rc8 from origin. Switched to a new branch 'orig/4.2-rc8' $ git reset --hard ... $ git checkout -b orig/4.3 Switched to a new branch 'orig/4.3' $ git fetch ~/src/linux refs/tags/v4.3:refs/tags/v4.3 This fetched all tags; maybe I should have just fetched to FETCH_HEAD? ... $ git rebase v4.3 First, rewinding head to replay your work on top of it... Applying: Remove microcode patches for mgsuvd (not enabled in Debian configs) Applying: dvb-usb-af9005: remove, mark as broken Applying: vs6624: remove, mark as broken Applying: Remove and mark as broken: cops Applying: video: Remove nvidiafb and rivafb Applying: Remove the entire firmware directory Applying: Remove: ft1000 Applying: Remove: Non-free RFC $ cd ../none $ git checkout none/4.2-rc8 ... Branch none/4.2-rc8 set up to track remote branch none/4.2-rc8 from origin. Switched to a new branch 'none/4.2-rc8' $ git reset --hard ... $ git checkout -b none/4.3 Switched to a new branch 'none/4.3' $ git fetch ../orig orig/4.3 From ../orig * branchorig/4.3 -> FETCH_HEAD $ git rebase --onto=FETCH_HEAD origin/orig/4.2-rc8 This rebase took a while, but was definitely easier than rebasing with quilt. There are no featuresets to deal with currently. ... $ cd ../.. $ sed -i 's/4.2-rc8/4.3/' .gitmodules $ emacsclient debian/changelog Waiting for Emacs... $ git commit -a -m 'Update to 4.3' $ git push --set-upstream alioth benh/submodule ... $ cd source/orig $ git push --set-upstream origin orig/4.3 This pushed a whole load of tags too. WTF? ... $ cd ../none $ git push --set-upstream origin none/4.3 ... Is it possible to combine those push commands? Is this something else we would need to script? > - Cherry pick patch > - (Make sure the submodules are on the correct branch, otherwise it > will be hard to push changes or they will go to incorrect locations) > - Cherry pick patch > - Merge changes into all derived featuresets, if any Rather than rebasing? > - Record new top commits and update changelog in main repo So that's two commits rather than one at present. A little annoying. > There are some things not yet implemented or different in my preview: > - debuild from the repo tree does not yet work, the rules are missing > the special directory definitions > - orig is also a submodule > > Please take a look and let me know what you think about this variant. > Most likely I've forgotten something, but I don't know what it is. What I like: 1. Rebasing is easier 2. I can log, diff, etc. through our changes and upstream changes What I don't like: 3. Pushing is more complicated 4. Cherry-picking is more complicated 5. Git working directory looks different from unpacked source package 6. It's not possible to see the history of one of our patches (3) and (4) definitely need to be ad
Re: Upstream source handling
On Sat, 2015-11-07 at 13:48 +, Ben Hutchings wrote: [...] > What I like: > 1. Rebasing is easier > 2. I can log, diff, etc. through our changes and upstream changes > > What I don't like: > 3. Pushing is more complicated > 4. Cherry-picking is more complicated > 5. Git working directory looks different from unpacked source package > 6. It's not possible to see the history of one of our patches > > (3) and (4) definitely need to be addressed, either by documentation > (if I'm missing some git feature or config) or by scripting. > > I think I can live with (5) and (6). For (6), maybe we should start > putting Change-Ids in our changes so that it would be possible to find > all versions using 'git log --tags=debian --grep=...' A couple more things I don't like: 7. The DFSG changes are not documented in the source package 8. Each featureset is reduced to a single patch in the source package (7) should be easy to fix, as the history is linear, except where we delete part of a file. (8) should be easy to fix for the 'none' featureset as its history will also be linear. For any other featuresets, reducing to a single patch may be unavoidable. Ben. -- Ben Hutchings Unix is many things to many people, but it's never been everything to anybody signature.asc Description: This is a digitally signed message part
Re: Upstream source handling
Ben, thanks for the comments. On Sat, Nov 07, 2015 at 01:48:51PM +, Ben Hutchings wrote: > On Tue, 2015-09-08 at 22:52 +0200, Bastian Blank wrote: > > Main principles: > So genorig.py would be replaced with 'git archive'? Yes. > > Workflow: > > - New upstream version > > - Rebase old tag on top of new upstream, tag it with the new version > There are no tags in the repository you created. Can you add one as an > example? Done. > Do you think we would tag when updating to a new upstream version, or > only when making the first upload with a new orig tarball (allowing for > further DFSG changes before uploading)? I think we have to tag while updating. We would need to rebase the already published branches, if we change the orig later. Also it is impossible to build a source package without the tag. > > - Rebase old main featureset branch on top of new orig tag as new > > branch > > - Rebase old other featureset branches on top of new main featureset > > branch or replace with new base and create new branch > > - Record new top commits and update changelog in main repo > That's a lot of rebasing, though not so different from what we do now > to refresh the patch series. Presumably we would add a script to > support this and ensure it is all done properly? Sure, thats support infrastructure. > The submodules are checked out with detached heads by default. Is your > intent that we would override this? Do you know if we can change it? > I tried this: > $ cd source/orig source/orig is supposed to go away. At least that was my plan. > Is it possible to combine those push commands? Is this something else > we would need to script? It is possible to change the way push works via the config, you can specify what is going to be pushed. > > - Cherry pick patch > > - (Make sure the submodules are on the correct branch, otherwise it > > will be hard to push changes or they will go to incorrect locations) > > - Cherry pick patch > > - Merge changes into all derived featuresets, if any > Rather than rebasing? Rebasing published branches is no nice thing to do. > What I don't like: > 3. Pushing is more complicated git push can push all submodules as well, but I don't see a config option for it. > 4. Cherry-picking is more complicated Yeah. That warrants a script. > 6. It's not possible to see the history of one of our patches Which history do you mean? Bastian -- Earth -- mother of the most beautiful women in the universe. -- Apollo, "Who Mourns for Adonais?" stardate 3468.1
Re: Upstream source handling
On Sat, Nov 07, 2015 at 02:16:13PM +, Ben Hutchings wrote: > 7. The DFSG changes are not documented in the source package > 8. Each featureset is reduced to a single patch in the source package > > (7) should be easy to fix, as the history is linear, except where we > delete part of a file. git can generate patch files for deleted files without actually showing the content. > (8) should be easy to fix for the 'none' > featureset as its history will also be linear. For any other > featuresets, reducing to a single patch may be unavoidable. I didn't want to enfore a linear history, but if we say it will be linear (and check that in a hook during push), we can also generate expanded patch series. Bastian -- It would seem that evil retreats when forcibly confronted. -- Yarnek of Excalbia, "The Savage Curtain", stardate 5906.5
Re: Upstream source handling
On Sat, 2015-11-07 at 17:53 +0100, Bastian Blank wrote: > On Sat, Nov 07, 2015 at 02:16:13PM +, Ben Hutchings wrote: > > 7. The DFSG changes are not documented in the source package > > 8. Each featureset is reduced to a single patch in the source package > > > > (7) should be easy to fix, as the history is linear, except where we > > delete part of a file. > > git can generate patch files for deleted files without actually showing > the content. I know, that's why I said where we delete *part* of a file (using unifdef). However, there are currently no cases where we do that. > > (8) should be easy to fix for the 'none' > > featureset as its history will also be linear. For any other > > featuresets, reducing to a single patch may be unavoidable. > > I didn't want to enfore a linear history, but if we say it will be > linear (and check that in a hook during push), we can also generate > expanded patch series. Your description of the workflow did imply the history would be linear for the 'none' featureset. And non-linear history gets linearised when rebasing so it's probably not a good idea to introduce non-linearity in any branch that we expect to rebase later. Ben. -- Ben Hutchings 73.46% of all statistics are made up. signature.asc Description: This is a digitally signed message part
Re: Upstream source handling
On Sat, 2015-11-07 at 17:49 +0100, Bastian Blank wrote: [...] > > Do you think we would tag when updating to a new upstream version, or > > only when making the first upload with a new orig tarball (allowing for > > further DFSG changes before uploading)? > > I think we have to tag while updating. We would need to rebase the > already published branches, if we change the orig later. Also it is > impossible to build a source package without the tag. OK. > > > - Rebase old main featureset branch on top of new orig tag as new > > > branch > > > - Rebase old other featureset branches on top of new main featureset > > > branch or replace with new base and create new branch > > > - Record new top commits and update changelog in main repo > > That's a lot of rebasing, though not so different from what we do now > > to refresh the patch series. Presumably we would add a script to > > support this and ensure it is all done properly? > > Sure, thats support infrastructure. > > > The submodules are checked out with detached heads by default. Is your > > intent that we would override this? > > Do you know if we can change it? No, as I said I'm not familiar with submodules. > > I tried this: > > $ cd source/orig > > source/orig is supposed to go away. At least that was my plan. > > > Is it possible to combine those push commands? Is this something else > > we would need to script? > > It is possible to change the way push works via the config, you can > specify what is going to be pushed. Including sub-modules? From what you wrote below, it sounds like that isn't possible. > > > - Cherry pick patch > > > - (Make sure the submodules are on the correct branch, otherwise it > > > will be hard to push changes or they will go to incorrect locations) > > > - Cherry pick patch > > > - Merge changes into all derived featuresets, if any > > Rather than rebasing? > > Rebasing published branches is no nice thing to do. OK, I suppose this is different from rebasing onto a new upstream as there we're creating a new branch. So in case we want to drop a patch, presumably we would revert the commit and then drop both the original and revert when moving to a new upstream? > > What I don't like: > > 3. Pushing is more complicated > > git push can push all submodules as well, but I don't see a config > option for it. I should use the option --recurse-submodules=on-demand, right? In the absence of a configuration variable, it would presumably be sensible to recommend an alias for this e.g. 'push-sm = push --recurse-submodules=on-demand' > > 4. Cherry-picking is more complicated > > Yeah. That warrants a script. > > > 6. It's not possible to see the history of one of our patches > > Which history do you mean? Currently we can run 'git log debian/patches/a/random.patch' and see when a patch was originally added and how it changed as it was rebased onto different upstream versions. If we were to use a merge-based workflow then 'git log .. patched/file.c' would provide similar information. But with a rebasing workflow, the git object model doesn't provide any association between the commits that apply a patch onto different upstream versions. Ben. -- Ben Hutchings 73.46% of all statistics are made up. signature.asc Description: This is a digitally signed message part
Re: Upstream source handling
On Sat, Sep 12, 2015 at 12:52:13PM +0200, maximilian attems wrote: > shouldn't one use subtrees these days and not submodules. > submodules only live in contrib. subtrees got merged in > main git. The problems solved by subtree and submodule are different. And in the git 2.5.1, subtree is contrib, submodule is main. Subtrees are merged trees into the container repo. They loose all information about there source. They can be manipulated further in the tree and later synced again. I did not look how using subtrees could work. Submodules are only pointers to commits. They don't add any volume of data to the tree. My proposal includes the frequent use to rebase. This mimics the current patch handling, where the applied patches are clearly shown. If we want to export a patch series for the package, we need that anyway. However, if we already maintain our own tree, using subtree does not provide much advantage, but it completely removes the automation around it. Bastian -- It would seem that evil retreats when forcibly confronted. -- Yarnek of Excalbia, "The Savage Curtain", stardate 5906.5
Re: Upstream source handling
On Tue, 2015-09-08 at 22:52 +0200, Bastian Blank wrote: > Please take a look and let me know what you think about this variant. > Most likely I've forgotten something, but I don't know what it is. I had a (very brief) play. The submodules refer to local paths (../source/linux IIRC) rather than URLs on alioth, is that on purpose? Or is it to avoid cloning the full linux history for each submodule? (i.e. git doesn't realise they are the same remote repo and can share stuff, so it would clone the full history multiple times) Also genorig seems to be broken (wants the dfsg series). I suspect this now works completely differently? Manually merging into all derived featuresets when changing the base (==none) featureset sounds annoying, and liable to being forgotten, but I guess that is pretty hard to automate? Would it be possible to arrange for the base featureset to actually be in the main tree at the toplevel rather than as a submodule do you think? That at least would seem to be a far more natural arrangement. Ian.
Re: Upstream source handling
Hi Ian On Sat, Sep 12, 2015 at 10:45:50AM +0100, Ian Campbell wrote: > On Tue, 2015-09-08 at 22:52 +0200, Bastian Blank wrote: > > Please take a look and let me know what you think about this variant. > > Most likely I've forgotten something, but I don't know what it is. > The submodules refer to local paths (../source/linux IIRC) rather than > URLs on alioth, is that on purpose? Relative URL references are expanded from the origin location, so "../source/linux" based on ssh://git.debian.org/git/kernel/linux is ssh://git.debian.org/git/kernel/source/linux. > Or is it to avoid cloning the full > linux history for each submodule? (i.e. git doesn't realise they are > the same remote repo and can share stuff, so it would clone the full > history multiple times) It is not able to use a shared object store if not told by hand. > Also genorig seems to be broken (wants the dfsg series). I suspect this > now works completely differently? I did not change genorig for the time being. > Manually merging into all derived featuresets when changing the base > (==none) featureset sounds annoying, and liable to being forgotten, but > I guess that is pretty hard to automate? We can at least check for such errors. > Would it be possible to arrange for the base featureset to actually be > in the main tree at the toplevel rather than as a submodule do you > think? That at least would seem to be a far more natural arrangement. Nope, a submodule can only live within an existing tree. Bastian -- Those who hate and fight must stop themselves -- otherwise it is not stopped. -- Spock, "Day of the Dove", stardate unknown
Re: Upstream source handling
On Tue, Sep 08, 2015 at 10:52:57PM +0200, Bastian Blank wrote: > During the linux packaging BoF at DebConf, Ben asked for usefull > upstream source handling. No compeling ones were mentioned. > > Some years ago (yes, years), I proposed some schema based on submodules, > but never got around to actually implement it. I finally managed to do > an initial implementation. It currently lives in the linux.git in > branch waldi/submodule. shouldn't one use subtrees these days and not submodules. submodules only live in contrib. subtrees got merged in main git. -- maks
Re: Upstream source handling
On Sat, 2015-09-12 at 12:05 +0200, Bastian Blank wrote: > Hi Ian > > On Sat, Sep 12, 2015 at 10:45:50AM +0100, Ian Campbell wrote: > > On Tue, 2015-09-08 at 22:52 +0200, Bastian Blank wrote: > > > Please take a look and let me know what you think about this > variant. > > > Most likely I've forgotten something, but I don't know what it > is. > > The submodules refer to local paths (../source/linux IIRC) rather > than > > URLs on alioth, is that on purpose? > > Relative URL references are expanded from the origin location, so > "../source/linux" based on ssh://git.debian.org/git/kernel/linux is > ssh://git.debian.org/git/kernel/source/linux. Hrm, this didn't appear to be working for me, and it was happy once I manually cloned something into ../source/linux. Might be dependent on git version or something? > > Would it be possible to arrange for the base featureset to actually be > > in the main tree at the toplevel rather than as a submodule do you > > think? That at least would seem to be a far more natural arrangement. > > Nope, a submodule can only live within an existing tree. I meant making the base featureset part of the main tree, not a submodule. I suppose the fact that it is rebasing knackers that idea though. Ian.
Re: Upstream source handling
On Sat, 2015-09-12 at 12:05 +0200, Bastian Blank wrote: > Hi Ian > > On Sat, Sep 12, 2015 at 10:45:50AM +0100, Ian Campbell wrote: > > On Tue, 2015-09-08 at 22:52 +0200, Bastian Blank wrote: > > > Please take a look and let me know what you think about this > variant. > > > Most likely I've forgotten something, but I don't know what it > is. > > The submodules refer to local paths (../source/linux IIRC) rather > than > > URLs on alioth, is that on purpose? > > Relative URL references are expanded from the origin location, so > "../source/linux" based on ssh://git.debian.org/git/kernel/linux is > ssh://git.debian.org/git/kernel/source/linux. Hrm, this didn't appear to be working for me, and it was happy once I manually cloned something into ../source/linux. Might be dependent on git version or something? > > Would it be possible to arrange for the base featureset to actually be > > in the main tree at the toplevel rather than as a submodule do you > > think? That at least would seem to be a far more natural arrangement. > > Nope, a submodule can only live within an existing tree. I meant making the base featureset part of the main tree, not a submodule. I suppose the fact that it is rebasing knackers that idea though. Ian.
Upstream source handling
Moin During the linux packaging BoF at DebConf, Ben asked for usefull upstream source handling. No compeling ones were mentioned. Some years ago (yes, years), I proposed some schema based on submodules, but never got around to actually implement it. I finally managed to do an initial implementation. It currently lives in the linux.git in branch waldi/submodule. Main principles: - source of each featureset is a submodule, using version specific branches - orig source via version specific tag - 3.0 (custom) source format, script generates a source the archive understands What will work: - debuild -b/-B from the repo tree - a script will create a plain 3.0 (quilt) source, which can be used at will What will not work: - debuild -s and similar calls from the repo tree Workflow: - New upstream version - Rebase old tag on top of new upstream, tag it with the new version - Rebase old main featureset branch on top of new orig tag as new branch - Rebase old other featureset branches on top of new main featureset branch or replace with new base and create new branch - Record new top commits and update changelog in main repo - Cherry pick patch - (Make sure the submodules are on the correct branch, otherwise it will be hard to push changes or they will go to incorrect locations) - Cherry pick patch - Merge changes into all derived featuresets, if any - Record new top commits and update changelog in main repo There are some things not yet implemented or different in my preview: - debuild from the repo tree does not yet work, the rules are missing the special directory definitions - orig is also a submodule Please take a look and let me know what you think about this variant. Most likely I've forgotten something, but I don't know what it is. Regards, Bastian -- Intuition, however illogical, is recognized as a command prerogative. -- Kirk, "Obsession", stardate 3620.7
