How many Ubuntu branches share history with upstream?
Hello all, In a previous discussion about Ubuntu distributed development, someone suggested that we graph the number of Ubuntu branches that share history with upstream. I think that's a very interesting thing to graph, but I have absolutely no idea on how to get that information -- even with access to Launchpad. How could we do it? jml -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: How many Ubuntu branches share history with upstream?
On Wed, 2010-02-10 at 13:58 +, Jonathan Lange wrote: In a previous discussion about Ubuntu distributed development, someone suggested that we graph the number of Ubuntu branches that share history with upstream. I think that's a very interesting thing to graph, but I have absolutely no idea on how to get that information -- even with access to Launchpad. How could we do it? The simplest thing to check would just be to see if the first revision of the upstream branch is part of the ancestry of the packaging branches' tip. Are you going to use the database for this or are you fine with calling out to Bazaar and using the branches on disk / remote servers? Cheers, Jelmer signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: How many Ubuntu branches share history with upstream?
On Wed, Feb 10, 2010 at 2:08 PM, Jelmer Vernooij jel...@canonical.com wrote: On Wed, 2010-02-10 at 13:58 +, Jonathan Lange wrote: In a previous discussion about Ubuntu distributed development, someone suggested that we graph the number of Ubuntu branches that share history with upstream. I think that's a very interesting thing to graph, but I have absolutely no idea on how to get that information -- even with access to Launchpad. How could we do it? The simplest thing to check would just be to see if the first revision of the upstream branch is part of the ancestry of the packaging branches' tip. Are you going to use the database for this or are you fine with calling out to Bazaar and using the branches on disk / remote servers? I'd like to use the database, since I think the other option would be grindingly slow. jml -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: How many Ubuntu branches share history with upstream?
On Wed, 2010-02-10 at 14:10 +, Jonathan Lange wrote: On Wed, Feb 10, 2010 at 2:08 PM, Jelmer Vernooij jel...@canonical.com wrote: On Wed, 2010-02-10 at 13:58 +, Jonathan Lange wrote: In a previous discussion about Ubuntu distributed development, someone suggested that we graph the number of Ubuntu branches that share history with upstream. I think that's a very interesting thing to graph, but I have absolutely no idea on how to get that information -- even with access to Launchpad. How could we do it? The simplest thing to check would just be to see if the first revision of the upstream branch is part of the ancestry of the packaging branches' tip. Are you going to use the database for this or are you fine with calling out to Bazaar and using the branches on disk / remote servers? I'd like to use the database, since I think the other option would be grindingly slow. So checking whether a revision is part of another branches' ancestry is not really possible then, if I understand the current database scheme correctly. You should be able to detect the common ancestry in most of the cases by just checking that the first revision of the upstream branch and first revision of the packaging branch have the same revid. This won't allow you to detect the situation where the packaging branch was created first and the upstream merged into it later, but as far as I can tell that's quite rare. Something related to this that has an open bug in lp-code - it is impossible to have common ancestry between upstream and debian packaging branches at the moment because it is impossible to register mirrors / imports for packaging branches. Cheers, Jelmer signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: How many Ubuntu branches share history with upstream?
On Wed, 10 Feb 2010 15:16:18 +0100, Jelmer Vernooij jel...@canonical.com wrote: So checking whether a revision is part of another branches' ancestry is not really possible then, if I understand the current database scheme correctly. You should be able to detect the common ancestry in most of the cases by just checking that the first revision of the upstream branch and first revision of the packaging branch have the same revid. This won't allow you to detect the situation where the packaging branch was created first and the upstream merged into it later, but as far as I can tell that's quite rare. Quite rare due to file id differences. I'd like to start encouraging developers to merge upstream in as a second root, but I don't want to do that until we can handle the file-id differences smoothly. So yes, I'd agree that would be a reasonable approximation currently, but it's not sure to hold over time. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: How many Ubuntu branches share history with upstream?
On Wed, 10 Feb 2010 08:46:37 -0600, John Arbash Meinel j...@arbash-meinel.com wrote: Where is this script going to be running? I wrote a trivial command that lets you run: bzr in-ancestry branch1 branch2 And reports back if the ancestry of branch1 is in branch2. lp:~jameinel/+junk/bzr-in-ancestry Running locally on bzr.dev trees, it takes less than 3 seconds to return true/false. Note also that the answer isn't symmetric. We've merged plugins into bzr.dev, but those plugins have not merged bzr.dev into them. Similarly for packaging branches. I would imagine that the packaging branch might merge upstream, but not the other way around. Comparing a mysql branch with a bzr.dev one seems to take 4s, which still isn't particularly long. I don't know what time scale you were hoping for Certainly less than a day to run. 3s * 10,000 ~= 8 hours. It wouldn't be near that to start with as we don't have anywhere near that number of packaging links, but it's always useful to look at what would happen if you scale. (but my experience with launchpad apis doesn't make it particularly faster than this ...) True, but graphs can be done against the DB directly, which would be much quicker as you don't have the https round trip overhead and it gives you scope for writing queries that act on all projects at once. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: import failures
On Thu, 04 Feb 2010 14:57:25 +1300, Michael Hudson michael.hud...@canonical.com wrote: James Westby wrote: On Wed, 06 Jan 2010 09:41:17 +1300, Michael Hudson michael.hud...@canonical.com wrote: James Westby wrote: Is it possible to get a query of old ones, and just run a bulk-update of them? I have the list of packages, and mwhudson was going to query for the list of branches based on that, and then request server-side upgrade I believe. Well, I've managed to completely drop the ball on this :( No problem. I could have done much of it myself. Can you send me the list of packages again? Attached. Once again, I've not done anything here... can you send me an updated list? Some of the ones from that list are in 2a format and some not, so if you have an up-to-date list it'll make things a bit easier for me. Sorry, forgot to respond to this. Some of them have been upgraded. If it's easier for me to do an info against all of them and filter out those not in 2a then I can do so. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: import failures
On Wed, 2010-02-10 at 21:46 +, James Westby wrote: Some of them have been upgraded. If it's easier for me to do an info against all of them and filter out those not in 2a then I can do so. I think thats easiest. signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
UDD @ Portland
James Westby and I had some time together in Portland to talk about UDD stuff. We talked about a few things: * Looms, their use today and where they should go * The operational issues with the package importer and how the bzr team can help * analysed a few specific bugs and tried to come up with solutions. Firstly though, a couple of overview points: - the udd project has 200 bugs on it. While many of these are 'collision' reports many are not. The collision reports are currently overly noisy, so please ignore them for now. However, the other bugs are open season for people to fix - and every bug fixed there will streamline things for people using bzr to package in Ubuntu. - Measurement error: the hottest 100 has a fairly high error rate for package imports at the moment, OTOH its looking at precisely the packages most likely to fail. James says that overall 96% or so of packages import successfully. - Adoption. I spoke with pitti and seb128 about how the desktop team uses bzr. Mostly they use packaging-only branches, which they prefer precisely because they don't need to download all the history. A quick test showed them getting 2M of data for gnome-panel, and 14M when using the package-imported branch. So we need to do some things here for people that do many drive-by fixes... - make downloading only some history easier/possible - make it possible to be using a mirror of the VCS data so it doesn't have to go over the network (perhaps the Ubuntu mirror network could carry the packaging branches?) - make bzr saturate the network more effectively (the 14M above didn't download at wire speed as far as we could tell). I haven't made these into bugs as yet, pending some feedback from you! Ok, onto the fun stuff. While long term we want a Launchpad control panel for the package importer, James thinks its reasonable that folk helping with the operation of it should be able to directly kick it off - so he has filed an RT ticket to get more access to that machine. James, whats the RT ticket number? Opening this up will let us rerun imports more promptly that appear to have had only spurious failures. Bugs with the importer can and should be debugged on peoples development environment - there is an earlier mail from December documenting how to do this. We should put that in the Wiki I think think. The collisions that are reported as bugs can be divided into three broad groups: - impossible (a collision in debian: at least at the moment, we don't expect people uploading to Debian packaging-branches. Well, *generally speaking* we don't expect this). (Nb: I do it for stuff I maintain in Debian :) - spurious (its not a collision, and a bug caused it) - genuine (it is a collision and it should be a merge proposal) We have a few collision specific tasks: - James is rapidly making new collisions be filed as merge proposals (unless they are in Debian imports) - we need to write a script to analyse the nearly 200 collisions in the bug tracker to highlight the debian imports (must be bugs, might be fixed), and convert the ubuntu ones to merge proposals. - We should delete the stale branches for collisions that we decide are bogus. Membership in the magic group ubuntu-branches is needed for that, and that group needs to be kept locked down (as it is equivalent to upload rights to the archive). So - lets make a list somewhere if you determine a branch isn't needed, and ping James or anyone on the tech board to delete such a branch). We looked at the workflow involved in packaging, and I'm very happy that James has seen the light and will be implementing an 'import-upstream' command to import and make a tarball micro-branch but not do the debian metadata updates. This will be useful for looms, where the two steps occur on different threads. Finally we looked at Looms with mathiaz who is hoping to get the MySQL packages in Looms for both Debian and Ubuntu. We identified some rough spots and a missing command (import-upstream) but it seems doable, if not /nice/ today. After that we talked about a sparser loom merge graph. The basic idea I have is that while the stack seems essential to providing a simple UI, all the merge commits make a lot of noise. So if we only do a merge commit when a conflict has happened, and otherwise depend on 'record' telling us what is incorporated, we can save a lot of commits and make 'log' or 'bzr viz' clearer, as well as making it simpler to cherrypick patches. There seem to be several related issues that tie into this: * Should the ready-to-build on-disk image with patch files be something stored in the loom, or something the loom models *and exports*. * Should someone editing a patch see a working tree with all the lower patches combined, or only their patch (and how does this tie into what gets committed - hairy logic incoming!) * How do people migrate into using Looms? I don't have good answers for all of these, but I'll try and write them up in more detail once
Re: UDD @ Portland
On 11 February 2010 13:18, Robert Collins robert.coll...@canonical.com wrote: James Westby and I had some time together in Portland to talk about UDD stuff. We talked about a few things: * Looms, their use today and where they should go * The operational issues with the package importer and how the bzr team can help * analysed a few specific bugs and tried to come up with solutions. Firstly though, a couple of overview points: - the udd project has 200 bugs on it. While many of these are 'collision' reports many are not. The collision reports are currently overly noisy, so please ignore them for now. However, the other bugs are open season for people to fix - and every bug fixed there will streamline things for people using bzr to package in Ubuntu. Can you explain what a collision is? Finally we looked at Looms with mathiaz who is hoping to get the MySQL packages in Looms for both Debian and Ubuntu. We identified some rough spots and a missing command (import-upstream) but it seems doable, if not /nice/ today. After that we talked about a sparser loom merge graph. I'd like to let looms progress, but not (unless james or others feel differently) add them into the dependency chain for getting UDD going. iow people should be able to try them on particular branches, without mandating them for all package branches, and (perhaps?) without requiring everyone working on that package to use them. -- Martin http://launchpad.net/~mbp/ -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: UDD @ Portland
Martin asks what a collision is. The situation with package imports is that we have a branch B, which both Ubuntu developers and the package importer can commit to. Collisions are what happen when the package importer sees something arrive in the archive which is either not in, or different to, the branch history. E.g. developer A: bzr branch lp:ubuntu/foo # add some changes, call it UNRELEASED # hmm, I won't dput yet, I want to tweak it some more. bzr push :parent developer B: apt-get source foo # add different changes, call it N dput So we get a Y shape import graph, but the archive is official, so the package importer does a push --overwrite to 'win' on the packaging branch, and pushes the old head to a new temporary branch, and files a ticket in launchpad describing that this happened. -Rob signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: UDD @ Portland
On Thu, 11 Feb 2010 13:18:30 +1100, Robert Collins robert.coll...@canonical.com wrote: James Westby and I had some time together in Portland to talk about UDD stuff. Yes, it was good to have the time, thanks for coming and for sending this mail. Firstly though, a couple of overview points: - the udd project has 200 bugs on it. While many of these are 'collision' reports many are not. The collision reports are currently overly noisy, so please ignore them for now. However, the other bugs are open season for people to fix - and every bug fixed there will streamline things for people using bzr to package in Ubuntu. They are now reported as merge proposals if it's an Ubuntu branch, so that should stop the list growing too much. - Measurement error: the hottest 100 has a fairly high error rate for package imports at the moment, OTOH its looking at precisely the packages most likely to fail. James says that overall 96% or so of packages import successfully. They are most likely to fail as they will tend to be larger, uploaded more often etc: making them more likely to trigger bugs. Ok, onto the fun stuff. While long term we want a Launchpad control panel for the package importer, James thinks its reasonable that folk helping with the operation of it should be able to directly kick it off - so he has filed an RT ticket to get more access to that machine. James, whats the RT ticket number? Opening this up will let us rerun imports more promptly that appear to have had only spurious failures. #37368 Feel free to drop me an email with the names of packages you think should be retried in the meantime. I'll do it the next time I read my email. It's no good for debugging issues, but it's not the best way to do it even when you can do it straightaway. Remember that you can run exactly the same code locally and so make use of bzr, pdb, etc. to investigate. Bugs with the importer can and should be debugged on peoples development environment - there is an earlier mail from December documenting how to do this. We should put that in the Wiki I think think. Good idea. The collisions that are reported as bugs can be divided into three broad groups: - impossible (a collision in debian: at least at the moment, we don't expect people uploading to Debian packaging-branches. Well, *generally speaking* we don't expect this). (Nb: I do it for stuff I maintain in Debian :) - spurious (its not a collision, and a bug caused it) - genuine (it is a collision and it should be a merge proposal) We have a few collision specific tasks: - James is rapidly making new collisions be filed as merge proposals (unless they are in Debian imports) Done, but with some slight issues due to the LP API and other things. They are being filed as merge proposals now. - we need to write a script to analyse the nearly 200 collisions in the bug tracker to highlight the debian imports (must be bugs, might be fixed), and convert the ubuntu ones to merge proposals. - We should delete the stale branches for collisions that we decide are bogus. Membership in the magic group ubuntu-branches is needed for that, and that group needs to be kept locked down (as it is equivalent to upload rights to the archive). So - lets make a list somewhere if you determine a branch isn't needed, and ping James or anyone on the tech board to delete such a branch). I'd say comment in the bug report for it. It has all the info I need and I can do it the next time I read mail. We looked at the workflow involved in packaging, and I'm very happy that James has seen the light and will be implementing an 'import-upstream' command to import and make a tarball micro-branch but not do the debian metadata updates. This will be useful for looms, where the two steps occur on different threads. It's currently spelt bzr dh_make, import-upstream would be bzr dh_make --bzr-only. When we get a workflow going with looms we can look at how we it fit in there. I didn't want to have import-upstream straight away as I didn't want confusion arising from the fact that you can run it in a packaging branch and so delete all the packaging. Finally we looked at Looms with mathiaz who is hoping to get the MySQL packages in Looms for both Debian and Ubuntu. We identified some rough spots and a missing command (import-upstream) but it seems doable, if not /nice/ today. After that we talked about a sparser loom merge graph. The basic idea I have is that while the stack seems essential to providing a simple UI, all the merge commits make a lot of noise. So if we only do a merge commit when a conflict has happened, and otherwise depend on 'record' telling us what is incorporated, we can save a lot of commits and make 'log' or 'bzr viz' clearer, as well as making it simpler to cherrypick patches. There seem to be several related issues that tie into this: * Should the ready-to-build on-disk image with patch files be something
Re: UDD @ Portland
On Thu, 11 Feb 2010 13:33:27 +1100, Martin Pool m...@canonical.com wrote: I'd like to let looms progress, but not (unless james or others feel differently) add them into the dependency chain for getting UDD going. No, we don't have to add it to the chain to get it going, but I think it's one ingredient of having a great system. iow people should be able to try them on particular branches, without mandating them for all package branches, and (perhaps?) without requiring everyone working on that package to use them. I think a gradual migration path is something to aim for. What we want is consistency of interaction. I don't want to have to work out what is going on in the packaging branch before I can start work on it. Allowing branch; hack; build; push regardless of what's going on and allowing others to delve more deeply is one way, another would be to have bzr add-patch or something that prepared the tree for working, I'm sure there are more. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
UDD as a product?
UDD now has an active mailing list, a Launchpad project and a bug/task list. Does it make sense to begin thinking about UDD as a product? Would it be valuable to talk about UDD x.y vs x.z? Code wise, I guess the product is a mix of LP features, Bazaar features and Bazaar plugins. OTOH, those things come together to form a system. There's a huge amount of wisdom being imparted each week on this list. Perhaps we should turn some of the threads into tutorials (either in a wiki or bzr branch) or a UDD Hackers Guide say. Is it too early for that? Ian C. -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
