Re: [Libreoffice] git submodule
* Norbert Thiebaud nthieb...@gmail.com schrieb: big_snip / Just wondering: do you want to bisect the whole bundle of trees at once or just single packages ? cu -- -- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weig...@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 -- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme -- ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
On Sun, Feb 06, 2011 at 05:44:03AM -0600, Norbert Thiebaud nthieb...@gmail.com wrote: humm... what build.git ? :-) on master I do not even clone build.git anymore... Then bootstrap.git, you get it. :) Con: it seriously increase the barrier of entry. git already scare some people... git + submodule at that scale will be much worse Con: it would be using submodule in a way it was certainly not intended to be used. That means that doc and support for that will be scarse and chance of screwing things up rise big time, with very few 'expert' - if any - to sort things out. Con: how well is it going to mesh with the fact that some git repo are 'optional' (l10n) ? Con: that will probably mess with the hg import script. we would need to make sure that import is still feasible - without breaking the current history Note that the main drawback of our current approach is indeed the bisection problem. It would be certainly be convinient to be able to bisect on the entire set of repos... but how much pain/risk do we want to take to improve that aspect of things ? I already thought about this when the first gitdm script was hacked by Cedric to handle all the repos as a single one (and I managed to came up with a solution there), but that does not help bisecting. If it was not clear from my mail, git-submodule is (in its current form) a no-go for us IMHO. Of course svn externals-like behaviour would not really improve the situation, either. What I can imagine as a working solution is that we already have a tinbuild which can detect if a certain set of git heads (of the repos) is buildable or not. Now tinbuild could publish those working sets to a githeads.git repo and a 'g bisect' could work from that data. Of course that will not find the exact commit like tinbuild does not do, either - but at least point out a few possible ones. (I mean tinbuild would push the contents of git-heads.txt to a githeads.git, then 'g bisect' would call 'git bisect' in githeads.git and after the working directory is updated there, it would check out the commits in the repos based on git-heads.txt so that it's possible to do the actual testing. That's not so hard to implement for bisect start/good/bad/reset.) Does this sound reasonable? I'm happy to first implement the tinbuild part if others agree that the idea is sane. Of course you can call the repo bisect.git or whatever, that's a minor detail. pgp3YqWMadZbo.pgp Description: PGP signature ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
On Mon, Feb 7, 2011 at 9:02 AM, Miklos Vajna vmik...@frugalware.org wrote: On Sun, Feb 06, 2011 at 05:44:03AM -0600, Norbert Thiebaud nthieb...@gmail.com wrote: humm... what build.git ? :-) on master I do not even clone build.git anymore... Then bootstrap.git, you get it. :) ;-) (I mean tinbuild would push the contents of git-heads.txt to a githeads.git, then 'g bisect' would call 'git bisect' in githeads.git and after the working directory is updated there, it would check out the commits in the repos based on git-heads.txt so that it's possible to do the actual testing. That's not so hard to implement for bisect start/good/bad/reset.) How about having a after-push git-hook in the master branch in our git.fdi/git/libreoffice/* git repos that generate the list of HEAD-sha1 that was you should be able to almost-completely bisect. sure in some cases the fact that 2 commit in two git are interdependent would throw off the bisection. but that is not that common. Another problem would be merge of feature branches, which would be counted as just one bisect step for the whole branch (but that is true even with the tinderbuild based solution) Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
On Sun, 2011-02-06 at 05:44 -0600, Norbert Thiebaud wrote: Note that the main drawback of our current approach is indeed the bisection problem. It would be certainly be convinient to be able to bisect on the entire set of repos... but how much pain/risk do we want to take to improve that aspect of things ? Could we do some sort of date based bisection ? or does that not work because the patches are not in chronological order really ;-) Hmm, Michael. -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
On Mon, Feb 07, 2011 at 04:29:23PM +, Michael Meeks
michael.me...@novell.com wrote:
Could we do some sort of date based bisection ? or does that not work
because the patches are not in chronological order really ;-)
It's possible to do g checkout @{2011-02-07}, but then we have to
reinvent our own git-bisect, which won't support this.
pgpNwEq6jCVoV.pgp
Description: PGP signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
On Mon, Feb 07, 2011 at 09:38:14AM -0600, Norbert Thiebaud nthieb...@gmail.com wrote: How about having a after-push git-hook in the master branch in our git.fdi/git/libreoffice/* git repos that generate the list of HEAD-sha1 that was you should be able to almost-completely bisect. sure in some cases the fact that 2 commit in two git are interdependent would throw off the bisection. but that is not that common. You mean when g commit g push creates / pushes multiple commits? For example when you rename a function in a repo and fix the build in 3 other ones. It would be really nice not to try to test the 3 ones where the build will obviously fail (and the tinbuild approach provides this). OTOH of course that's possible - provided that those hooks still push the state to a githeads.git? Another problem would be merge of feature branches, which would be counted as just one bisect step for the whole branch (but that is true even with the tinderbuild based solution) Sure - I don't have an idea either how to not count a merge as a single step with split repos. pgpPIPOn65vtw.pgp Description: PGP signature ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
On Mon, Feb 7, 2011 at 10:29 AM, Michael Meeks michael.me...@novell.com wrote: On Sun, 2011-02-06 at 05:44 -0600, Norbert Thiebaud wrote: Note that the main drawback of our current approach is indeed the bisection problem. It would be certainly be convinient to be able to bisect on the entire set of repos... but how much pain/risk do we want to take to improve that aspect of things ? Could we do some sort of date based bisection ? or does that not work because the patches are not in chronological order really ;-) Even if we could rely on a date (which we can't), that would give you the starting point. the bisection after that is just a bisection of the commit between that point and HEAD (usually) the problem is that we have commit scattered across 20 git repos... so in order to truly bisect, we need an extra layer to 'serialized' all the commit from all the git repos, and then bisect that series of commit Norbert Hmm, Michael. -- michael.me...@novell.com , Pseudo Engineer, itinerant idiot ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
On Mon, Feb 7, 2011 at 11:10 AM, Miklos Vajna vmik...@frugalware.org wrote: On Mon, Feb 07, 2011 at 09:38:14AM -0600, Norbert Thiebaud nthieb...@gmail.com wrote: How about having a after-push git-hook in the master branch in our git.fdi/git/libreoffice/* git repos that generate the list of HEAD-sha1 that was you should be able to almost-completely bisect. sure in some cases the fact that 2 commit in two git are interdependent would throw off the bisection. but that is not that common. You mean when g commit g push creates / pushes multiple commits? For example when you rename a function in a repo and fix the build in 3 other ones. It would be really nice not to try to test the 3 ones where the build will obviously fail (and the tinbuild approach provides this). The problem of the tinderbox approach is the coarse granularity (it is not uncommon to have 1/2 a dozen or more of independent commit between 2 build, the reliability (tinderbox sometime are down and this is not necessarily detected quickly). furthermore if you only record 'good' tinderbox run, you may have a few days of commit in one interval. if you don't then you are in the same case than the scenario above (it doesn't build because of interdependent commit); One thing that we could do is that the pus-hook could do a commit amend if the committer is the same that the very last one. presumably linked commit on multiple repo are pushed 'together' so if a push on repos A is immediately followed by a push on repo B then the hook would do a git commit --amend instead of creating a new version. that ways you usually fold all theses change into one 'list-head' Of course you could have a race between two committer... but that is _really_ rare. Norbert OTOH of course that's possible - provided that those hooks still push the state to a githeads.git? Another problem would be merge of feature branches, which would be counted as just one bisect step for the whole branch (but that is true even with the tinderbuild based solution) Sure - I don't have an idea either how to not count a merge as a single step with split repos. ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
On Mon, Feb 07, 2011 at 12:18:39PM -0600, Norbert Thiebaud nthieb...@gmail.com wrote: The problem of the tinderbox approach is the coarse granularity (it is not uncommon to have 1/2 a dozen or more of independent commit between 2 build, the reliability (tinderbox sometime are down and this is not necessarily detected quickly). furthermore if you only record 'good' tinderbox run, you may have a few days of commit in one interval. if you don't then you are in the same case than the scenario above (it doesn't build because of interdependent commit); Hm, OK. One thing that we could do is that the pus-hook could do a commit amend if the committer is the same that the very last one. presumably linked commit on multiple repo are pushed 'together' so if a push on repos A is immediately followed by a push on repo B then the hook would do a git commit --amend instead of creating a new version. that ways you usually fold all theses change into one 'list-head' Of course you could have a race between two committer... but that is _really_ rare. Automatic git commit --amend is really dangerous, I would not do it. It's not an accident non-fastforwards are rejected while pushing. So - in case post-receive hooks on the freedesktop servers are allowed (CIA is not notified from such a hook at the moment), we could create a githeads.git which is updated after every push and git bisect could run there. Given that there is a githeads.txt in that repo, the g wrapper could update the repos accordingly. Does this sound acceptable? (And there could be multiple branches in githeads.git, just like in the repos.) pgpJhHrvup0ci.pgp Description: PGP signature ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
On Mon, Feb 07, 2011 at 10:33:00PM +0100, Miklos Vajna vmik...@frugalware.org wrote: Automatic git commit --amend is really dangerous, I would not do it. It's not an accident non-fastforwards are rejected while pushing. It turns out I misunderstood it, I thought this is about git commit --amend in the repos, not in githeads.git. So here is what we discussed on IRC: The problem of a logical commit may affect multiple repos and you don't want to bisect a state when such a multi-repo change is not fully applied is similar to the one the cvs2something more modern converters do, where cvs does not handle multi-file commits. That means in case one uses the same commit message for commits in different repos, we can assume it does not make sense to try to build when not all affected repos are updated. As a consequence, a post-receive git hook could be written, which would be executed on the freedesktop servers, which would do the followings: for i in repos: for j in commits: update_githeads_txt() if (j.author, j.commitmsg) in githeads: # the same author already pushed something with # the same commit message to an other repo git commit --amend githeads.txt else: # something new git commit githeads.txt In reality 'g push' executes git push for each affected repo, so the hook in fact would do the inner cycle only. Also, we could add one more constraint: when we use --amend, the timestamp of the old commit should not differ too much (say, 5 minutes) from the new commit. The format of githeads.txt could be still what tinbuild has, we can get other properties of the commits anytime from the repos based on the sha1. Does this sound reasonable? That way we would have a githeads.git that could be used for bisecting (the 'g bisect' part if of course missing, but possible to implement - that would update the state of the repos based on githeads.txt) and it would be quite small. pgpH0NCnr4nwg.pgp Description: PGP signature ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
On Mon, Jan 31, 2011 at 03:39:41PM +0100, Jan Holesovsky ke...@suse.cz wrote:
Can you please do a bit of research in that regard, and summarize how we
would gain from using git submobule?
To my understanding, in short:
git submodules are not like how svn externals are usually used. svn
externals usually checkout the latest revision from a trunk/branch,
while git submodules always checkout a given commit.
This means that in case one pushes to writer.git/master, then somebody
else doing a 'git submodule update' after the push will see no update.
That operation will do something only in case one updates the submodule
link from commit 870df5402bfadae19b9133bdce6a8db05d547925 to
47e1f5dc3b095a36e7837f2d6fc34aabed1d3997 (an example) in the build repo,
and that update will be a commit in the build repo as well.
Pro: it would be easy to bisect, since git bisect knows how to handle
submodules (currently you can just use something like g checkout
@{2011-02-06} and pray that you can something that builds at all)
Con: build.git has to be updated manually (it has to be decided how
frequently), ideally after every 'g push', but that would cause a lot of
bogus commits in build.git, which sounds ugly.
pgpVFmfxOm361.pgp
Description: PGP signature
___
LibreOffice mailing list
LibreOffice@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
* Norbert Thiebaud nthieb...@gmail.com schrieb: Note that the main drawback of our current approach is indeed the bisection problem. It would be certainly be convinient to be able to bisect on the entire set of repos... but how much pain/risk do we want to take to improve that aspect of things ? I wouldn't do that at all, instead actually split the whole thing into smaller modules (really modules - not just subtrees!). The main build process would be driven from an separate build engine, that builds specific versions of the separate packages. See my postings about kicking off the 3rdparty-libs. cu -- -- Enrico Weigelt, metux IT service -- http://www.metux.de/ phone: +49 36207 519931 email: weig...@metux.de mobile: +49 151 27565287 icq: 210169427 skype: nekrad666 -- Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme -- ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
I meant the repos in the bootstrap repo, of course, not the build repo. Am 31.01.2011 16:23, schrieb Norbert Thiebaud: I concur. my limited understanding of how submodule works lead me to believe that it would not be a nice fit with what we do 1/ it is more appropriate when the 'modules' have fairly independent life-cycle, which is not really our case. sure calc/writer/impress/base are fairly isolated from one another, but the coupling with libs-core/libs-gui/ure/... is pretty tight The submodules are independent in that they are seperate repositories and have their own commits, but as I understand it this is the case with LibreOffice already. On the other hand, a certain commit of each submodule is tied to every commit of the superproject. I don't know how this is done right now, maybe in the download script. Or do you get the most recent version of the subrepos no matter which version of the build repo you have checked out? I thought of git submodule when I saw the 'download' and 'g' scripts. I'm not sure what they are doing exactly, but 'make fetch' and 'download' seem to be similar to 'git submodule update --init' and 'g' seems to be similar to 'git submodule foreach'. Tobias ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
Hi Tobias, On 2011-01-30 at 20:56 +0100, Tobias Rosenberger wrote: is there a reason why you don't use git submodule for managing the other repos in the build repo? The reason is that so far nobody researched that enough, to be able to judge if it would help us, or not :-) From what I know about the submodules, it fits more some kind of release tree (our 'build' repository), than the bleeding edge - but I may be completely wrong :-) Can you please do a bit of research in that regard, and summarize how we would gain from using git submobule? Thank you a lot, Kendy ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
Re: [Libreoffice] git submodule
On Mon, Jan 31, 2011 at 8:39 AM, Jan Holesovsky ke...@suse.cz wrote: Hi Tobias, On 2011-01-30 at 20:56 +0100, Tobias Rosenberger wrote: is there a reason why you don't use git submodule for managing the other repos in the build repo? The reason is that so far nobody researched that enough, to be able to judge if it would help us, or not :-) From what I know about the submodules, it fits more some kind of release tree (our 'build' repository), than the bleeding edge - but I may be completely wrong :-) I concur. my limited understanding of how submodule works lead me to believe that it would not be a nice fit with what we do 1/ it is more appropriate when the 'modules' have fairly independent life-cycle, which is not really our case. sure calc/writer/impress/base are fairly isolated from one another, but the coupling with libs-core/libs-gui/ure/... is pretty tight 2/ this would make the barrier of entry, and the possibility of srew-up so much higher. I do agree that it could make a sens for a limited use in the 'build' repo of git submodules... but then again, I'm not sure it will improve the workflow of the people using it... Norbert ___ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
