Re: Moving udd away from sqlite
On Tue, 03 Jul 2012 18:12:06 +0100, James Westby james.wes...@canonical.com wrote: I think I've addressed all the comments so far, and I'm keen to move ahead with the deployment before we get too close to the weekend. Therefore if there are no objections I'd like to merge these and deploy Wed morning UK time. Hi, The importer is now running on storm again. There were a couple of bugs that I fixed, but I don't believe bad data should have come from either of them. There have been some database locked errors, but so far everything has kept running without deadlocking. I'll continue to keep an eye on it while we work on the postgres stuff. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd away from sqlite
On Thu, 21 Jun 2012 11:29:08 -0400, James Westby james.wes...@canonical.com wrote: I need to do that, as well as some unicode fixes. I'll get those done and up for review by the end of this week. Belatedly here they are: https://code.launchpad.net/~james-w/udd/storm/+merge/112983 https://code.launchpad.net/~james-w/udd/storm-unicode-fixes/+merge/112984 https://code.launchpad.net/~james-w/udd/storm-sqlite-db-provider/+merge/112985 I'll then spin up an ec2 instance to run in parallel and do everything except push the branches back, and leave this running over the weekend. I spun up three udd instances on ec2, each rigged to not hit codehosting, but to have something like the db access pattern of doing imports. There was one running tip of trunk, one running the code we tried to deploy last time, and one running my current code. After leaving them for a weekend grep database is locked debug_log* gives: trunk: 0 old code: 7199 new code: 0 (with me logging in a couple of times to kill the old code because it was in a deadlocked state.) which is hopefully good evidence that we should have fewer issues this time. The new code also imported roughly the same number of packages as trunk in that time, so there doesn't seem to be any large performance impact from the changes. Then on Tuesday morning my time (starting 1300 UTC) I'll do a deployment of the code to production if (reviews, ec2) don't show any problems. We can then carefully monitor the service for the next few days. I'm ready to do this once the code is reviewed. I'll send another email before I do it. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd away from sqlite
On Thu, 21 Jun 2012 18:22:22 +0200, Vincent Ladeuil vila+...@canonical.com wrote: Rollback is to revert the storm code again Restore the dbs. I don't think we should do that if we have no evidence of data corruption. We'd be repeating work for no benefit. Rollback is to stop the importer, revert the config changes, restore the dbs. No need, as changes would be written to postgres, so just switching back to sqlite is an effective restore. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd away from sqlite
On Mon, 18 Jun 2012 10:45:26 +0200, Vincent Ladeuil vila+...@canonical.com wrote: James Westby james.wes...@canonical.com writes: On Fri, 15 Jun 2012 12:34:12 +0200, Vincent Ladeuil vila+...@canonical.com wrote: It's not magic. It's moving from a database that's not designed for concurrent use to one that is designed for concurrent use. Despite not being designed for concurrent use, it *is* used this way and lock contentions have been encountered leading me to believe that the actual *design* needs to be fixed. The fact that changing the db is triggering more contentions is a symptom of a deeper issue. Changing the db access layer is triggering that, we were still running on the same (non-multi-user db). I agree that the design needs to be fixed, and that's exactly what we're taking about, fixing it by moving a db that is designed for multi-user use. It looks like your understanding of the issue is better than mine here, would you mind sharing that knowledge in an automated test (with the added benefit that we won't regress in this area) ? Just this week-end we had an add-import-jobs failure: Traceback (most recent call last): File /srv/package-import.canonical.com/new/scripts/bin/add-import-jobs, line 5, in module sys.exit(main()) File /srv/package-import.canonical.com/new/scripts/udd/scripts/add_import_jobs.py, line 17, in main icommon.create_import_jobs(lp, status_db) File /srv/package-import.canonical.com/new/scripts/udd/icommon.py, line 304, in create_import_jobs status_db.add_import_jobs(checked, newest_published) File /srv/package-import.canonical.com/new/scripts/udd/icommon.py, line 633, in add_import_jobs self._add_job(c, package, self.JOB_TYPE_NEW) File /srv/package-import.canonical.com/new/scripts/udd/icommon.py, line 615, in _add_job datetime.utcnow(), None, None)) sqlite3.OperationalError: database is locked So we already know that add_import_jobs is involved in the bug (with the current sqlite-based implementation), who is the other user in this case and how can this be reproduced ? Each connection to sqlite is another user, so each of the cron scripts, as well as the imports themselves, and several connections within mass-import are all the users. When a write operation is started a global lock is acquired that locks out any other writers until the operation is complete. If the lock is held then the library will wait up to a timeout (configured to be 30s for udd) for the lock to be released before giving up. The errors like the above occur when the timeout is reached, so either another transaction took more than 30s to release the lock, or there were lots of connections trying to take the lock, and this one didn't win before the 30s was up. When we change to storm it forces pysqlite in to a higher isolation level, so that transactions are started when any statement is executed. My guess is that this means locks are taken more frequently and are held for longer, giving more contention errors. Postgres doesn't have a global lock, it has table or row locks, so that clients will only hit lock contention if they are changing the same data, which will be much less frequent. How can I show that in an automated test? I can write an XFAIL test that if two connections are opened, one starts a transaction and then the other hits an locking exception if it tries to do anything, but that doesn't seem to prove much about the operation of the system. I.e. reproducing the add_import_jobs failure in a test that will fail with sqlite and succeed with your changes will demonstrate we've captured (and fixed) at least one lock contention. We are dealing with probabilistic failure though. I can demonstrate that in a deterministic situation changing two separate tables under sqlite will take global locks, but I can't prove that we will never get contention under postgres. If the test suite cannot be trusted to catch most of the issues that happen in production, the test suite should be fixed. You're not implying that testing in production being needed, the test suite is useless right ? No, I'm saying that the only measure of whether something runs correctly in production is whether it runs in production. From that, can we imagine a test that will import a few packages and compare the corresponding dbs ? We can do that as part of testing the migration script. It can be restarted with the dbs from whenever the transition starts and it will catch up in roughly the between starting the transition and rolling back. There may be a few bugs due to replaying things, but we do it all the time (e.g. removing revids and re-importing when someone does push --overwrite) As in requeue --full ? requeue --zap-revids ? None of them is used on a daily basis but my limited experience there never triggered issues
Re: Upgrading pristine-xz on jubany
On Fri, 15 Jun 2012 10:32:59 +0200, Vincent Ladeuil vila+...@canonical.com wrote: Barry Warsaw ba...@ubuntu.com writes: On Jun 14, 2012, at 05:21 PM, Vincent Ladeuil wrote: - I'm already running successful tests inside a quantal lxc container :) It has become for many of us not just a nice-to-have but a must-have for Ubuntu development. That's my understanding as well. Here are my last achievements for the week: - I got in touch with pristine-tar maintainers resulting in a trivial bugfix included in 1.25. This is a small step in getting *known* as a primary consumer but it also demonstrates that we can get fixes upstream quickly (1.25 has already been uploaded to sid and quantal). - I got in touch with xz maintainers and a fix is on its way there (many thanks to Lasse Collin for its invaluable help here). This will require an additional fix to pristine-xz which I will submit as soon as I can test the xz fix). With these fixes in place, on quantal, it should remain only 10 pristine-tar import failures out of the current 338 on jubany. That's great work Vincent, I look forward to it being deployed, however that happens :-) Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd away from sqlite
On Fri, 15 Jun 2012 12:34:12 +0200, Vincent Ladeuil vila+...@canonical.com wrote: It's not magic. It's moving from a database that's not designed for concurrent use to one that is designed for concurrent use. Despite not being designed for concurrent use, it *is* used this way and lock contentions have been encountered leading me to believe that the actual *design* needs to be fixed. The fact that changing the db is triggering more contentions is a symptom of a deeper issue. Changing the db access layer is triggering that, we were still running on the same (non-multi-user db). I agree that the design needs to be fixed, and that's exactly what we're taking about, fixing it by moving a db that is designed for multi-user use. Well, when the correctness and safety is demonstrated, the context (and hence my own answer) will probably be different but until then I just can't say. And I'm very reluctant to fork without an actual plan for merging back: how to know when it's safe how to actually achieve it. And I have no idea (nor time right now) to debug the fallouts of such a change that the actual package importer doesn't need. Hence my tendency to consider that demonstrating the validity of this change should be achieved first. But you just said above that you *do* think it needs to be fixed? How can we demonstrate the validity of the change? We can only demonstrate that it doesn't break production by running the changes in production. What would satisfy you that it was unlikely to break production? Would there be a script to migrate from sqlite to PG ? Yes. Can the package importer be re-started with empty dbs and catch up (how long will it take ? Days ? Weeks ?). Can this trigger bugs because the importer don't remember what it pushed to lp ? It can be restarted with the dbs from whenever the transition starts and it will catch up in roughly the between starting the transition and rolling back. There may be a few bugs due to replaying things, but we do it all the time (e.g. removing revids and re-importing when someone does push --overwrite) Or do you expect us to see another peek like http://webnumbr.com/ubuntu-package-import-failures.from%282012-01-24%29 ? Hopefully not. Yes, that's why we're not in a position to safely accept such a change ! And all the time spent on integrating these changes is not spent on allowing them to be accepted in good conditions. Sorry, I don't understand this, could you explain? but it demonstrated the locking problem and is how he came up with those options. Then the test improvements are certainly valuable to backport to lp:udd or is there nothing to reuse from the EC2 experiment ? It wasn't a set of unit tests, it was a set of tests of a live system, so nothing to backport. We have had a lot of experience recently working with Canonical IS to get new servers and new staging servers deployed. If you want a staging server, we'd be happy to help you and would gladly advocate for you in their priority queue. Great to hear :) I should come back soon on this topic, just a quick question though: are the new servers running lucid or precise ? New servers are being installed with precise. If an existing server is used it may be lucid, but there is a concerted effort to upgrade all machines to precise currently underway. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: DEB_VENDOR package imports
On Thu, 03 May 2012 14:49:19 +0100, Dmitrijs Ledkovs dmitrij.led...@ubuntu.com wrote: I think, the lp:debian/* imports should be run with export DEB_VENDOR=Debian This should make the lp:debian/package to match the source package as it is unpacked/built on Debian. I'm agreed on this, and Colin pointed it out too and I filed a bug: https://bugs.launchpad.net/udd/+bug/911496 . Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Importer add-import-jobs cronjob temporarily disabled
On Sat, 28 Apr 2012 10:52:44 +0100, Max Bowsher _...@maxb.eu wrote: See https://bugs.launchpad.net/udd/+bug/990394 Thanks for catching this. Fwiw it's not normally an issue as it normally takes longer to run branch-distro.py and re-enable things, and once there has been another package published to the new release it only acts on all the packages once. For whatever reason that didn't happen this time. I'll reply to the bug with some comments about how it could be better handled in future. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: UDD importer making a nuisance of itself with v3 source format branches
On Fri, 13 Apr 2012 01:13:22 +0100, Max Bowsher _...@maxb.eu wrote: I've just had a conversation with cjwatson and slangasek on #ubuntu-release about the importer making a nuisance of itself by declaring a perfectly reasonable commit to be a collision / difference, and replacing it with one of its own. The key pain point here is the .pc/ directory. It's practically impossible to maintain a .pc/ directory checked into VCS without unnatural jumping through hoops. As a result, packages being seriously developed in bzr by humans, rather than being primarily imported, tend NOT to have a .pc/ : cjwatson patches applied doesn't have to imply .pc in vcs; it's unfortunate that the importer took that particular decision cjwatson (I've been using patches-applied-in-bzr since well before the importer did, *without* .pc) I think, as a short term fix, we should modify the collision-is-clean check to ignore the absence of a .pc directory in packager-committed revisions. That sounds reasonable. It does mean that there starts to be some differences in how the branches should be handled, but they exist already depending on whether the importer or a human was last to push. I think there's more to a transition that just ignoring .pc, but it should be considered if that's how humans prefer to deal with the branches. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Importer stopped since Sunday?
On Tue, 10 Apr 2012 01:08:25 +0100, Max Bowsher _...@maxb.eu wrote: It looks like someone stopped the UDD importer on Sunday? James W. has a 'crontab -e' editor open from around that time. Anyone know what's going on, and if it's safe to restart? We're failing rather dismally at providing prompt imports at the moment. Hi, Firstly my apologies for not sending a mail about this at the time. Some time on Saturday night the importer started failing hard with sqlite contention errors. The mass-import process was stuck, so nothing was being imported, and the cronjobs were failing causing two emails to be sent every 5 minutes. Given that nothing was being imported anyway, I stopped the process until I would have more time to investigate after the long weekend. I suspect the changes are somehow related to my storm changes, but I don't know what the relationship is yet. It ran fine for a couple of days, so we should be fine to restart and process the queue. It may well fail again though. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: What to do when a packaging branch is out of date
On Wed, 04 Apr 2012 15:51:21 -0300, Andreas Hasenack andr...@canonical.com
wrote:
Hi, can you take another look? It seems there is a delay again:
$ bzr branch ubuntu:landscape-client precise-already-done
Most recent Ubuntu version: 12.04.3-0ubuntu1
Packaging branch version: 12.04.2-0ubuntu1
Packaging branch status: OUT-OF-DATE
Branched 44 revisions.
There is an error for it in the package importer:
http://package-import.ubuntu.com/status/landscape-client.html#2012-03-30
06:12:01.518169
bzrlib.errors.UnknownErrorFromSmartServer: Server sent an unexpected
error: ('error', 'xmlrpclib.Fault', Fault -1: 'Unexpected Zope
exception: RequestExpired: request expired.')
Hi,
It is now up to date again.
Thanks,
James
--
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: bzr merge-upstream: why delete and add the same unchanged file?
On Thu, 29 Mar 2012 12:21:41 +0200, Jelmer Vernooij jel...@canonical.com wrote: Am 29/03/12 05:14, schrieb James Westby: On Tue, 20 Mar 2012 18:06:52 -0300, Andreas Hasenack andr...@canonical.com wrote: I understand they are isolated and separated branches. I thought supporting a bzr branch for the upstream branch was more of a convenience and that merge-upstream would actually just export it to a temporary tarball and then move on like if I had given it a tarball to work with, but I see now that's not the case. Yeah, there is some old code in bzr-builddeb to do that, but I've forgotten how to activate it now :-) I think I'm missing some context - are you talking about running bzr merge-upstream branch ? That should still work. That will merge in the branch as well as export the tarball right? I think Andreas was asking about a way to just export the tarball and use it in merge-upstream without also merging the branch. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: bzr merge-upstream: why delete and add the same unchanged file?
On Tue, 20 Mar 2012 18:06:52 -0300, Andreas Hasenack andr...@canonical.com wrote: Yes, and I don't have commit or upload rights to ubuntu:landscape-client, so it will always be the first time :) If your sponsor pushes to the branch then it will be there for the next time that you make a change. I understand they are isolated and separated branches. I thought supporting a bzr branch for the upstream branch was more of a convenience and that merge-upstream would actually just export it to a temporary tarball and then move on like if I had given it a tarball to work with, but I see now that's not the case. Yeah, there is some old code in bzr-builddeb to do that, but I've forgotten how to activate it now :-) It also makes it very hard to review the changes this first time, right? The diff isn't helpful. Yeah, the best thing to do I think is to fall back to diff -Nru of two trees so that you diff based on paths rather than file ids. I'm not aware of anyway to get bzr to do this. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: bzr merge-upstream: why delete and add the same unchanged file?
On Tue, 20 Mar 2012 16:20:29 -0300, Andreas Hasenack andr...@canonical.com wrote: Why is it removing and adding the same file? This file (and several others) didn't change between ubuntu:landscape-client and lp:landscape-client, it's exactly the same. I'm assuming that this is the first time you've done any sort of merge between the two? Because they are unrelated branches from bzr's point of view, it has to reconcile the history and file-ids. For the history it joins the two revisions history together, which is fine. The file ids isn't so easy though, as they can't be joined. Therefore it replaces the ubuntu file ids with the upstream ones, which is why you see everything as removed and added. This is annoying, but it allows you to move forward, and only needs to happen once. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: UDD breakdown when building orig.tar.gz from upstream VCS
On Wed, 15 Feb 2012 08:21:37 -0500, Barry Warsaw ba...@ubuntu.com wrote: I just want to put this out there for the historical record. I think this is a rare enough use case that UDD doesn't need to address, certainly not any time soon, if ever. OTOH, maybe there's an easy workaround. I was working on an NBS for the fgfs-atlas package (LP: #903225). The solution was straightforward enough: upstream had all the necessary fixes in their CVS repository, but hadn't done a release in a long time. I twiddled the packaging to build an orig.tar.gz from CVS, and the googlez helped find some good general packaging information on how to do this. Unfortunately, UDD is essentially useless here. The problem is that after creating the tarball from CVS, `bzr bd -S` can't be used because dpkg-source will complain too much about deltas between the tarball and the source tree. It'll warn about a lot of stuff, but then fail with some unrepresentable changes to source. I worked around this by unpacking the new orig.tar.gz and `cp -a` the debian/ directory from the precise version of the package (with my changes) over into the unpacked tarball. After a few rounds of tweaking, and using `debuild -S -sa`, I had a debian/ that built locally, so I uploaded it and will let the importer (hopefully) sort out the mess. I still did all my changes to debian/ in a source branch though, because that made it easier to get a diff for the linked Debian bug. Is there's a magical udd switch or config setting that would have helped me keep all the changes in the source branch? It seems like this is somewhat similar to the merge-upstream issue when upstream has a rather large released tarball delta. What's the merge-upstream issue? I think that using merge-upstream is what should be done here, passing it the tarball you created. That will merge in the new code, keep the debian directory and any changes, with conflicts if necessary. Once all that is resolved `bzr bd -S` should just work. Of course there may be a large gulf between theory and reality :-) Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd to django
On Wed, 14 Dec 2011 09:32:28 +0100, Vincent Ladeuil vila+...@canonical.com wrote: True, but I'm not saying your plan is *bad* for udd, quite the contrary. And yes, sharing some service to query launchpad sounds also like a good idea (I think I mention adding pkgme to mass_import and that's one of the points I had in mind). But this can achieved only by either running both pkgme and udd on the same host or add yet another layer to access the db remotely. In both cases, what I'm saying is that udd has already higher priorities. My plan is delete add-import-jobs, and an a POST handler that gets told when there are new packages to scan. add-import-jobs would then move somewhere else (could be the same machine) That's perhaps what you mean by another layer. It's not a priority for udd, but even while drafting this mail there's a conversation going on about another service that would want the same information. We can take what we have from udd and turn it in to an enabler for lots of other useful things. I would be worried about the risks involved with changing production udd in such a large way at once. The steps I outlined here would involve targeted production changes that should be much easier to debug. But if you follow these steps during your rewrite, nothing forbids following them when deploying once we know you've fully debugged them right ? Except that any feedback from earlier steps has to be incorporated after completing later steps, so you may have to undo lots of work. Both projects will benefit from this separation: - pkgme can go ahead without caring for udd needs, as long as the actual code base evolve by separating old features from the new ones with a reasonable effort to make the new ones easier to integrate. I don't think that's true. It's not like we don't care about udd, and if we make changes without regard for it Why would you do that ? pkgme can go ahead without caring for udd needs The easiest way for us to make changes caring for udd is to have prompt feedback about the way it is affecting udd. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd to django
On Wed, 14 Dec 2011 09:38:53 +0100, Vincent Ladeuil vila+...@canonical.com wrote: it will be less risky to deploy changes gradually, Only if we have tested these changes before deployment which we can't do for now (don't take my word for it, just look at the lp:udd history). Even my recent changes in this regard gave us the ability to *manually* test locally (and there, look at revno 555 history for cases I had to fix on top of our actual test suite). Why not set up a staging instance of udd to test changes before they are deployed to production? and we are better off not forking if we can avoid it. Eventually we can split it into separate services/components. I'm all in favor of splitting, I'm advocating the less risky (and as such the cheapest) way to get there. The way I see it the same code is going to be changing either way, and it can either do so incrementally, or in one big jump. If everything goes well then one big jump is less work, but if things break, and you are saying that they are very likely to, then it's easier to deal with that incrementally, where you have fewer things to reason about. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd to django
On Thu, 15 Dec 2011 08:09:00 +1300, Robert Collins robe...@robertcollins.net wrote: On Thu, Dec 15, 2011 at 7:08 AM, James Westby jw+deb...@jameswestby.net wrote: My plan is delete add-import-jobs, and an a POST handler that gets told when there are new packages to scan. 'there is work to do now' is a classic pub-sub situation. Rather than a post handler, I suggest you want one of: - a webhook http://wiki.webhooks.org/w/page/13385124/FrontPage The concept of a WebHook is simple. A WebHook is an HTTP callback: an HTTP POST that occurs when something happens; a simple event-notification via HTTP POST. Are you referring to a different webhook? Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd to django
On Tue, 13 Dec 2011 09:11:36 +0100, Vincent Ladeuil vila+...@canonical.com wrote: 3. It would also allow for starting to move udd to an SOA, or at least make it easier. Not a concern for udd so far. Actually I'd like to turn add_import_jobs in to a separate service, as it could be shared between udd and pkgme, and other services that people want to build, reducing load on Launchpad. That would be much easier if udd was in django (or twisted, or go, or ...) 4. It would be nice to have a query builder, rather than all the hand-written sql. Not a big problem for udd so far. Indeed, but I consider it technical debt, and it makes the code harder to read and change. 1) Install python-django in production - So that it is there when it is needed. This should be cheap. On the other hand, if newer versions are needed (which would be known during the dev/test phase) this will also be useless. Coding to django 1.1 (lucid) is not too restrictive. IS can also run django 1.3 if absolutely required. What do you think? Is this worth doing? Frankly ? This certainly sounds like a very good plan. I also think udd doesn't need any of that in the short term. Now, if you do it for pkgme, we will certainly be interested in cannibalizing part or all of it. In the mean time, the less disruptive changes we do to udd, the more we can focus on the import failures which is where the paint points are. I do think dealing with import failures is good, but I'm not sure the approach you advocate is the right one. What I mean is that forcing the use of the same code base for two projects with different goals sounds like a sure way to trigger failures for the other project without benefits for the project needing the new features. We've already encountered such issues at a small scale and until the test suite become rock-solid, I fail to see while we won't encounter new issues. I.e.: It sounds easier to separate common parts from specific parts in the actual udd code base while starting a *new* project than doing so in the udd code base that is already in production. Once those parts are clearly separated, the package importer can look at integrating the new and well-separated udd. I would be worried about the risks involved with changing production udd in such a large way at once. The steps I outlined here would involve targeted production changes that should be much easier to debug. Both projects will benefit from this separation: - pkgme can go ahead without caring for udd needs, as long as the actual code base evolve by separating old features from the new ones with a reasonable effort to make the new ones easier to integrate. I don't think that's true. It's not like we don't care about udd, and if we make changes without regard for it then it's likely that either lots of changes will be required to support udd again, or udd will never used the new codebase. - udd won't have to integrate incremental steps that bring no added value but still cost deployment/debugging time or do so *only* when a tangible benefit emerge *and* addresses issues. If you believe this then we can go away and ignore udd, but I fear it will be more work and more risk in the end. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd to django
On Sat, 10 Dec 2011 17:52:40 +1300, Robert Collins robe...@robertcollins.net wrote: On Sat, Dec 10, 2011 at 3:10 PM, James Westby jw+deb...@jameswestby.net wrote: Hi, I think there are a few reasons that we should consider moving udd to django (more on what this actually means later.) I'm curious what data udd stores. There are two sets. There's the bookkeeping data for running import-package each time a package is uploaded to Debian or Ubuntu, keeping track of failures, etc. Then there's the bookkeeping data for import-package itself, about what it thinks the state of bzr is etc. These should be treated differently, with the former the one that is more important for the goals laid out here. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd to django
On Sun, 11 Dec 2011 06:38:25 +1300, Robert Collins robe...@robertcollins.net wrote: It might be interesting - as a thought experiment if nothing else - to consider failures a form of crash and upload them to a crash database rather than processing them inside udd - e.g. toss them out over amqp. The current design is that it also uses them to avoid acting on that package until there is manual intervention. So it may be interesting to use oopses to communicate the problems and do analysis of them, but it won't avoid storing the data as well unless that model is changed. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: bzr bd -S --package-merge
Hi, I think this is likely to be https://bugs.launchpad.net/ubuntu/+source/bzr-builddeb/+bug/876888 Thanks, James On Thu, 27 Oct 2011 07:39:09 -0400, Barry Warsaw ba...@ubuntu.com wrote: How exactly does --package-merge calculate the version it passes to dpkg-genchanges -v? I'm wondering because it doesn't seem to be doing what I'd expect it to do, and I'm reluctant to use it. Latest case in point: I'm merging boost1.46 from testing into precise, so I do the following: $ bzr branch ubuntu:boost1.46 precise $ cd precise $ bzr merge-package debianlp:boost1.46 Most recent Debian version: MISSING Text conflict in debian/control 1 conflicts encountered. The merge resulted in 1 conflicts. Please resolve these and commit the changes with bzr commit. resolve conflict $ bzr diff debian/changelog === modified file 'debian/changelog' --- debian/changelog 2011-06-03 20:28:58 + +++ debian/changelog 2011-10-27 01:57:00 + @@ -1,3 +1,35 @@ +boost1.46 (1.46.1-7ubuntu1) precise; urgency=low + + * Merge with Debian testing. Remaining Ubuntu changes: +- Detect gcc atomic intrinsics, needed for arm spinlock (LP: #513721) +- Drop libboost-mpi, libboost-mpi-python, and libboost-graph-parallel (and + related -dev packages): we don't want to pull all of the mpi packages + into main. These are provided in a separate boost-mpi-source1.46 + package +- Drop libboost1.46-all-dev and provide from boost-mpi-source1.46 +- Adjust debian/rules and debian/control + + -- Barry Warsaw ba...@ubuntu.com Wed, 26 Oct 2011 21:56:45 -0400 + +boost1.46 (1.46.1-7) unstable; urgency=low + + * control: Fix ungrammatical description for iostreams packages. +Closes: #633865. + + * rules: Dump boostrap log file if bootstrapping fails. + + * libboost-doc.README.Debian: Remove reference to packages bjam and +boost-build; now only need boostX.YZ-dev. Closes: #630529. + + -- Steve M. Robbins s...@debian.org Wed, 17 Aug 2011 23:18:52 -0500 + +boost1.46 (1.46.1-6) unstable; urgency=low + + * control(libboost-mpi-python1.46.1, libboost-python1.46.1): Suggests a +python interpreter. Closes: #620775. + + -- Steve M. Robbins s...@debian.org Sun, 12 Jun 2011 00:37:42 -0500 + boost1.46 (1.46.1-5ubuntu2) oneiric; urgency=low * No change rebuild to drop Python 3.1 support. looks good $ bzr bd -S --package-merge If I now look at boost1.46_1.46.1-7ubuntu1_source.changes I see *all* the version entries from 1.46.1-7ubuntu1 all the way back to 1.21.1-1, the first in the changelog. If I drop the --package-merge option, I get a much trimmer changes file, with just the last changelog entry in it. That's not right either though. I'd expect to see entries for -6, -7 and -7ubuntu1. I've pushed the branch to lp:~barry/ubuntu/precise/boost1.46/debian-merge in case you'd like to take a look at see what's going on. I'll hold off on uploading for a bit in case there's something obvious going on. Cheers, -Barry Attachment: signature.asc (application/pgp-signature) -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Collision branches
On Thu, 8 Sep 2011 14:51:44 +1000, Martin Pool m...@canonical.com wrote:
I looked into a few of them and they weren't all clearly due to quilt
problems, but perhaps most of them are (or I didn't understand the
cause from a glance.)
In order to get some data on this I just looked back at 45 of these from
the last month, and found:
9 that looked real, or at least feasible
2 that were caused by updates to .po files
4 that were caused by automatically generated debian/control files
1 caused by updates to config.{guess,sub}
29 caused by quilt interaction
So 64% of these were caused by quilt interactions, and all the other
spurious ones were less than 50% of what's left, indicating that quilt
is the place to focus our efforts to make these merge proposals have
much more signal than noise.
Thanks,
James
--
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Collision branches
On Thu, 8 Sep 2011 14:51:44 +1000, Martin Pool m...@canonical.com wrote: I looked into a few of them and they weren't all clearly due to quilt problems, but perhaps most of them are (or I didn't understand the cause from a glance.) Unfortunately it's necessary to look at the importer log for the package to see why the importer felt it necessary to file the merge proposal. This is because the diff you are seeing is the result of merging the branch in to the new tip, but the importer decides based on the diff of the two revisions. These frequently differ and mean that looking at the merge diff doesn't tell you why the importer chose to do that (particularly if the merge diff is empty.) Perhaps that's the wrong check, but that's the way it is currently. I think we can handle this without blocking on looms by doing a smarter merge that unapplies and reapplies the patches. There is some work towards this in eg https://bugs.launchpad.net/bugs/608675 which Jelmer is working on - we may need extra work to hook it up into the udd importer. Would it also need to be used by LP when generating the merge diffs? What we should probably do next is look at the merge proposals that were filed and work out whether each one - is a real conflict in a sensible form - is not a real conflict and shouldn't be generated at all (some have zero diff) - could be either avoided or better presented by smarter quilt handling or something else That would be good. Is someone able to look at this analysis? Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Collision branches
Hi, Thanks for fixing the bugs that were preventing merge proposals for getting filed for collisions. This had led to a surge in the number of such merge proposals. This is mainly due to a backlog, but there have been 10 or so in the two days since. You can see the extent of this by searching for ubuntu-branches at http://reports.qa.ubuntu.com/reports/sponsoring/index.html Colin has valiantly reviewed some of them (maybe half, thanks Colin,) and has found that in none of the cases so far were the collisions real in the sense that someone pushed and someone else uploaded something different. There was one case at https://code.launchpad.net/~ubuntu-branches/ubuntu/oneiric/ibus/oneiric-201108121834/+merge/73916 which seems to indicate a bug though. From my previous experience going through these merge proposals the majority of issues will be caused by the representation of quilt in the branch. Can the Bazaar team do something to stop this influx of merge proposals that must be sorted, leaving just real ones? Does this have to involve work on looms? Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Terminating the append_revisions_only experiment, for now
On Tue, 23 Aug 2011 09:41:24 +0200, vila v.ladeuil...@free.fr wrote: This is also very close to my feelings and I like to add that we should *really* write tests to capture them and make sure we don't regress again in the future. Overall, I feel we have far too much failures when landing *good* stuff *because* we lack a proper way to test. It's a long known issue that we miss a launchpad test server but *not* having tests to guard against regressions is clearly a path we don't want to pursue as demonstrated here (IMNSHO). Whether we address that by using staging or any sort of local launchpad test server is open to discussion but the sooner we start this discussion the better. I think there are tests that could be written that would have caught some of these issues that don't require a Launchpad instance to run. There are problems when the append-revisions-only is set on the local branches for instance. Whether the code is structured such that you can test those code paths without a Launchpad instance is a different matter though :-) I'm not saying that we shouldn't work out a way to test against Launchpad, just that there may be useful things that can be done without solving that problem. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Wondering how to fix up a category of import failure
On Thu, 23 Jun 2011 10:48:05 +0200, John Arbash Meinel j...@arbash-meinel.com wrote: I personally like C when reasonable. However reasonable means that nobody else has started working on the project (since creating new revisions will break their existing changes). Otherwise, I would go for B. branch-nick doesn't mean all that much. I agree that B is likely the best option here. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: [Launchpad-dev] Removing ubuntu-branches and ubuntu-techboard celebrities
On Mon, 30 May 2011 17:28:14 -0400, Francis J. Lacoste francis.laco...@canonical.com wrote: For official package branch, this change would affect James Westby as he's the only member of ubuntu branches that is not part of the technical board. Maybe what's needed is to fix https://bugs.launchpad.net/launchpad/+bug/365098 and allow anyone who can upload the package to set the official link. The question then becomes, do we need to fix that bug before proceeding (in other word, would there be unwanted fall-outs from restricting this to the current distribution owners.) Yes, that bug needs to be fixed before removing the celebrity, or some other way of keeping the importer working needs to be found. The importer currently runs with my credentials in order to be able to do all of the things that it needs to do (handily I am a core-dev and a member of ~ubuntu-branches, which equates to full permission to do everything that it needs to do.) We want to change this anyway, but haven't ever got to doing it. The importer frequently sets official branches (e.g. when a new package is uploaded to Ubuntu,) and so this needs to continue to work, but I don't feel strongly about the exact mechanics. To put it another way the importer needs to have * ability to push to every official branch for Debian and Ubuntu (to keep them up to date) * ability to set the official branches for Debian and Ubuntu (to add them when something new appears, e.g. new package, first security update for a package to a particular release) and it uses all of those permissions regularly, so any temporary removal would disrupt its operation. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Import layout of Quilt v3 packages
On Tue, 08 Feb 2011 08:00:25 +, Max Bowsher m...@f2s.com wrote: Therefore, what about checking in the patched code, without any quilt metadata (.pc dir) but with a flag file that triggers bzr-builddeb to write out the appropriate metadata whenever a working tree is built for such a branch? (Writing out the metadata would consist of copying the series file to .pc/applied-patches, and reverse-applying each patch in reverse order, stashing the resultant modified file in .pc/patchname/filename for each patch) This would work for checkout. What are the implications for merge etc? Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Import layout of Quilt v3 packages
On Fri, 04 Feb 2011 10:22:45 -0600, John Arbash Meinel j...@arbash-meinel.com wrote: Now that I've described the state as I understand it, here are some questions. 1) As I understand it, most people are in favor of *not* versioning the .pc directory. So that when you do bzr branch lp:ubuntu/foo you'll get a tree with: a) debian/patches/* still intact b) Those patches applied to the working tree c) no .pc directory Well, I'm in favour of *not* versioning the .pc directory, however I don't immediately follow to your a, b and c. I am convinced that check out and immediately start hacking is a property that we want. 2) Doesn't that mean that you have to rebuild the .pc directory right after you get the checkout? Wouldn't it be easier to get it from the beginning? Or is it just introducing an extra place to get conflicts when trying to merge. Yes, you would have to rebuild the .pc directory right after the checkout. Yes it would be easier to get it right from the beginning. That said, if you did end up merging 2 branches that didn't have .pc directories, how would you get the .pc directory rebuilt? (Since presumably the patch series need to be combined in some way, and modifications to the source tree also need to be replicated into the patches themselves.) There is a similar problem with the current state of affairs, where merging two branches attempts to merge everything in .pc and doesn't leave you in a very good state. All this may change again if we switch the importer to use looms, since then you can do stuff like merge the patches one-by-one up the stack until you get to the top, without having to deal with conflicts in .diff format. Exactly. I think that there may be a middle ground, if we can separate storage from presentation to the user. I don't know how that would work without some pretty major changes though. Maybe pursuing looms is the correct way to go. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Making bzr commit more like debcommit
On Mon, 24 Jan 2011 10:54:26 -0500, Barry Warsaw ba...@ubuntu.com wrote: We have 'bzr commit' and we have 'debcommit'. Currently, the UDD guidelines talk about both, but for consistency, I'd like to standardize on recommending 'bzr commit'. One feature that debcommit has: DEBCOMMIT(1) DEBCOMMIT(1) NAME debcommit - commit changes to a package ... VCS SPECIFIC FEATURES ... bzr If the changelog entry used for the commit message closes any bugs then --fixes options to bzr commit will be generated to associate the revision and the bugs. I know we can just do 'bzr commit --fixes lp:12345', though I often forget to do that until after my last commit, so I tend to add --unchanged, which leads to an empty revision. What do you think about making 'bzr commit' act more like debcommit when there's a new debian/changelog entry? I have always wanted to do this. However, none of the current bzr hook points allow for this, except for overriding the commit command, which is why I never got to it. So, +1. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: UDD survey results
Hi Martin, Thanks for running and summarising the survey, very interesting results. On Thu, 18 Nov 2010 18:23:56 +1100, Martin Pool m...@canonical.com wrote: * The patch format from bzr is awkward - I'm not sure what this means; maybe that it is not smart about debdiff stuff I believe this is that they don't like bundles. They want git format-patch/git am as the way to send patches around. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Please check my thinking on bug 646979
On Mon, 04 Oct 2010 17:01:01 -0500, John Arbash Meinel j...@arbash-meinel.com wrote: On 9/29/2010 11:23 PM, James Westby wrote: What merge package does is first merge the two upstream revisions together, taking the tree from whichever has the highest version number. ---B---F / / / / / .-D--. \ A-= H \`-E-` \ \ C--G Currently it will then just merge H in to G (the target). This can generate conflicts, which are very, very confusing to users, as it's incredibly hard to explain why they are getting them. Does merging D E generate conflicts itself? It would seem that if merging to G generates conflicts, then you should have gotten a conflict in the intermediate stage as well. (offhand the best you can usually hope for is more understandable conflicts, unless you have a real 'criss-cross' merge and we are selecting a very poor base.) It is not a real merge. We create a new revision with D E as parents, and the contents of the later of the two (defined in terms of upstream version numbers). So, no, there is no possibility of conflicts at this stage. A /|\ E D B |\|\|\ | H F C \ \| | \ I | \ | \ | \| G Now you have a genuine criss-cross. As the lcas are E and B (ancestors of both I and G that are not superseded by a more recent ancestor.) Just using 3-way merge (vs say --weave) I would expect this to conflict more than merging H = G, because of our specific base selection (when we find a criss-cross 3-way goes to the next base, which will be A, which then will try to merge (I-A) into (G-A). That's unfortunate. Is there a way we could use our increased knowledge about the revisions involved to merge with a strategy that would make this situation better? Would it be beneficial to have some concrete examples to try out? Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Improving merge-upstream
On Tue, 28 Sep 2010 21:39:54 +0200, Vincent Ladeuil v.ladeuil...@free.fr wrote: There is a merge proposal that goes in this direction pending review at: https://code.edge.launchpad.net/~vila/bzr/323111-orphan-config-option/+merge/35690 (with pre-requisites). Great. 4) Remembering the upstream branch. merge-upstream takes an optional upstream branch. Currently there are cases where one person supplies it, but the next doesn't, and that's not ideal, as you get a discontinuity, and there can be extra conflicts in future due to added files. Still related to parallel imports ? It is parallel imports, yes. Martin mentioned recently that we could special-case merge to relax the conflicts for different file-ids by checking the paths involved. That would not solve the parallel import problem but it may help for the packaging and upstream branches relationship. I think so. This is beyond just that problem though, in that it could be recording a richer ancestry that it will if you forget the upstream branch. Therefore I would like if if, similar to e.g. bzr merge, if you supply a branch once it is remembered and used again. That's clearly a case where some config file should be versioned and shared even if it needs to be versioned in a specific way (the reference to the upstream branch is a live data that doesn't flow in the same way than the upstream code or the packaging data (or does it?)). It's slightly independent, yes. I would put it in .bzr-builddeb/default.conf, as that is a good start. Would you qualify this upstream branch as a data shared by a set of branches but for which updates are never reverted ? I'm not sure I follow this. I could concoct a situation in which it is reverted. Couldn't you use an ancestry check against the last merged revision ? That could make sense. It wouldn't stop partially unrelated things being merged. If you are merging a release from a stable branch then you don't want to merge trunk. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Growth of Ubuntu Distributed Development
On Tue, 20 Jul 2010 10:20:03 -0400, Scott Kitterman ubu...@kitterman.com wrote: Specifically for merging, I think an equivalent of grab-merge (I'll go so far as to suggest implementing something in grab-merge and then people can grab- merge --bzr or something) and an easy way to diff just the non-upstream changes are essential to broader adoption. Let's work out how we can do this, and then we can look at implementing it. I would like it to be an operation done entirely within bzr if possible (there are already ways to do it with source packages). What we want to do is compare the old packaged with its upstream version, and then compare that with the changes from the new upstream to the new packaged source. I don't think bzr has any special facilities for interdiffs right now? In that case we could start by generating those two diffs and piping them to interdiff. Getting the diffs themselves is relatively straightforward. What should the command be? Just an option to diff? diff-upstream? How much would you want to be able to tweak things beyond the default of showing the differences between the last two versions in debian/changelog? Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: [Launchpad-users] upstream + packaging + looms + lp != happiness
On Thu, 27 May 2010 16:18:20 -0400, Barry Warsaw ba...@canonical.com wrote:
This is the sanest way I can currently think of for organizing the upstream
and packaging branches for two distros and a PPA. If it works, it seems like
a nice way to manage things.
I agree that your sketch looks like a good way to organise things.
Unfortunately, this doesn't work because round tripping the branch through
Launchpad throws away all the loom information. Here's the transcript from
two different machines ('bzr looms' is an alias for 'bzr show-loom'; the
'split' thread is an artifact that will go away).
limelight% bzr looms
=ubuntu
debian
split
trunk
limelight% bzr record
Loom recorded.
limelight% bzr push lp:~barry/computer-janitor/loomified
Using default stacking branch
/~computer-janitor-hackers/computer-janitor/trunk at
lp-69637584:///~barry/computer-janitor
Created new stacked branch referring to
/~computer-janitor-hackers/computer-janitor/trunk.
heresy% bzr branch lp:~barry/computer-janitor/loomified
Branched 240 revision(s).
heresy% cd loomified/
heresy%
All the threads disappeared when the branch was pulled to machine heresy.
Maybe Launchpad doesn't support looms yet? Maybe the stacking is messing
things up? Any other suggestions or comments?
No idea, sorry. Have you filed a bug for tracking purposes?
Thanks,
James
--
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Operational changes and documentation
On Fri, 19 Feb 2010 01:34:35 +, James Westby jw+deb...@jameswestby.net wrote: And just as I send this I notice that something has gone screwy and so I've stopped the importer. If those people could resist the urge to play until I have had a chance to investigate after sleeping :-) (Feel free to investigate, but I would advise against restarting the daemon until the problem is sorted.) I forgot to note to the list that I fixed that issue and we are up and running again. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: import failures
On Thu, 18 Feb 2010 12:54:01 +1100, Robert Collins robe...@robertcollins.net
wrote:
Ah:
CHKInventoryRepository('lp-hosted:///~ubuntu-branches/debian/sid/camlp5/sid/.bzr/repository',
fallback_repositories=[CHKInventoryRepository('lp-hosted:///~ubuntu-branches/debian/squeeze/camlp5/squeeze/.bzr/repository')])
has no revision james.wes...@ubuntu.com-20091029134422-acd9dzuxa44whcct
We have a bug open on this - I think we should make it critical.
bug 507040 was the only one that looked similar on a search, is that the
one?
Thanks,
James
--
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Operational changes and documentation
Hi, Our wonderful sysadmins today helped migrate the package importer to a shared environment so that others can help administer it. Unfortunately that can only be Canonical employees currently as it requires machine access. So, the following people can now twiddle with the importer if you need something doing, you don't have to go through me: * igc (once he gets added to the group, sorry Ian) * jam * lifeless * poolie * spiv * vila Bear with us while we work out how to do it, as some requests will still be routed through me while they learn the ins and outs of the system. To help them I wrote some docuementation, but it is more widely applicable, just getting in to the details at the end. You can find it at https://wiki.ubuntu.com/DistributedDevelopment/UnderTheHood and the links therein. Help improving this would be appreciated. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: import failures
On Mon, 15 Feb 2010 19:19:13 +1100, Martin Pool m...@canonical.com wrote: So, just to be sure, we want to upgrade all the non-2a ones to 2a, on Launchpad? Probably by running the upgrade either on the lp machine itself or at least on a nearby machine? Yep. Differing formats cause pain, and there's no need for these to be in an inferior format. Doing the upgrade from the codehosting machine would be much more efficient for these three thousand branches. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: UDD as a product?
On Sat, 13 Feb 2010 07:06:37 +1000, Ian Clatworthy ian.clatwor...@canonical.com wrote: Would these differ from the phases in the overall specification? Is there a URL for this? https://wiki.ubuntu.com/DistributedDevelopment/Specification (linked from https://wiki.ubuntu.com/DistributedDevelopment) Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: How many Ubuntu branches share history with upstream?
On Wed, 10 Feb 2010 15:16:18 +0100, Jelmer Vernooij jel...@canonical.com wrote: So checking whether a revision is part of another branches' ancestry is not really possible then, if I understand the current database scheme correctly. You should be able to detect the common ancestry in most of the cases by just checking that the first revision of the upstream branch and first revision of the packaging branch have the same revid. This won't allow you to detect the situation where the packaging branch was created first and the upstream merged into it later, but as far as I can tell that's quite rare. Quite rare due to file id differences. I'd like to start encouraging developers to merge upstream in as a second root, but I don't want to do that until we can handle the file-id differences smoothly. So yes, I'd agree that would be a reasonable approximation currently, but it's not sure to hold over time. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: How many Ubuntu branches share history with upstream?
On Wed, 10 Feb 2010 08:46:37 -0600, John Arbash Meinel j...@arbash-meinel.com wrote: Where is this script going to be running? I wrote a trivial command that lets you run: bzr in-ancestry branch1 branch2 And reports back if the ancestry of branch1 is in branch2. lp:~jameinel/+junk/bzr-in-ancestry Running locally on bzr.dev trees, it takes less than 3 seconds to return true/false. Note also that the answer isn't symmetric. We've merged plugins into bzr.dev, but those plugins have not merged bzr.dev into them. Similarly for packaging branches. I would imagine that the packaging branch might merge upstream, but not the other way around. Comparing a mysql branch with a bzr.dev one seems to take 4s, which still isn't particularly long. I don't know what time scale you were hoping for Certainly less than a day to run. 3s * 10,000 ~= 8 hours. It wouldn't be near that to start with as we don't have anywhere near that number of packaging links, but it's always useful to look at what would happen if you scale. (but my experience with launchpad apis doesn't make it particularly faster than this ...) True, but graphs can be done against the DB directly, which would be much quicker as you don't have the https round trip overhead and it gives you scope for writing queries that act on all projects at once. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: import failures
On Thu, 04 Feb 2010 14:57:25 +1300, Michael Hudson michael.hud...@canonical.com wrote: James Westby wrote: On Wed, 06 Jan 2010 09:41:17 +1300, Michael Hudson michael.hud...@canonical.com wrote: James Westby wrote: Is it possible to get a query of old ones, and just run a bulk-update of them? I have the list of packages, and mwhudson was going to query for the list of branches based on that, and then request server-side upgrade I believe. Well, I've managed to completely drop the ball on this :( No problem. I could have done much of it myself. Can you send me the list of packages again? Attached. Once again, I've not done anything here... can you send me an updated list? Some of the ones from that list are in 2a format and some not, so if you have an up-to-date list it'll make things a bit easier for me. Sorry, forgot to respond to this. Some of them have been upgraded. If it's easier for me to do an info against all of them and filter out those not in 2a then I can do so. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: UDD @ Portland
On Thu, 11 Feb 2010 13:18:30 +1100, Robert Collins robert.coll...@canonical.com wrote: James Westby and I had some time together in Portland to talk about UDD stuff. Yes, it was good to have the time, thanks for coming and for sending this mail. Firstly though, a couple of overview points: - the udd project has 200 bugs on it. While many of these are 'collision' reports many are not. The collision reports are currently overly noisy, so please ignore them for now. However, the other bugs are open season for people to fix - and every bug fixed there will streamline things for people using bzr to package in Ubuntu. They are now reported as merge proposals if it's an Ubuntu branch, so that should stop the list growing too much. - Measurement error: the hottest 100 has a fairly high error rate for package imports at the moment, OTOH its looking at precisely the packages most likely to fail. James says that overall 96% or so of packages import successfully. They are most likely to fail as they will tend to be larger, uploaded more often etc: making them more likely to trigger bugs. Ok, onto the fun stuff. While long term we want a Launchpad control panel for the package importer, James thinks its reasonable that folk helping with the operation of it should be able to directly kick it off - so he has filed an RT ticket to get more access to that machine. James, whats the RT ticket number? Opening this up will let us rerun imports more promptly that appear to have had only spurious failures. #37368 Feel free to drop me an email with the names of packages you think should be retried in the meantime. I'll do it the next time I read my email. It's no good for debugging issues, but it's not the best way to do it even when you can do it straightaway. Remember that you can run exactly the same code locally and so make use of bzr, pdb, etc. to investigate. Bugs with the importer can and should be debugged on peoples development environment - there is an earlier mail from December documenting how to do this. We should put that in the Wiki I think think. Good idea. The collisions that are reported as bugs can be divided into three broad groups: - impossible (a collision in debian: at least at the moment, we don't expect people uploading to Debian packaging-branches. Well, *generally speaking* we don't expect this). (Nb: I do it for stuff I maintain in Debian :) - spurious (its not a collision, and a bug caused it) - genuine (it is a collision and it should be a merge proposal) We have a few collision specific tasks: - James is rapidly making new collisions be filed as merge proposals (unless they are in Debian imports) Done, but with some slight issues due to the LP API and other things. They are being filed as merge proposals now. - we need to write a script to analyse the nearly 200 collisions in the bug tracker to highlight the debian imports (must be bugs, might be fixed), and convert the ubuntu ones to merge proposals. - We should delete the stale branches for collisions that we decide are bogus. Membership in the magic group ubuntu-branches is needed for that, and that group needs to be kept locked down (as it is equivalent to upload rights to the archive). So - lets make a list somewhere if you determine a branch isn't needed, and ping James or anyone on the tech board to delete such a branch). I'd say comment in the bug report for it. It has all the info I need and I can do it the next time I read mail. We looked at the workflow involved in packaging, and I'm very happy that James has seen the light and will be implementing an 'import-upstream' command to import and make a tarball micro-branch but not do the debian metadata updates. This will be useful for looms, where the two steps occur on different threads. It's currently spelt bzr dh_make, import-upstream would be bzr dh_make --bzr-only. When we get a workflow going with looms we can look at how we it fit in there. I didn't want to have import-upstream straight away as I didn't want confusion arising from the fact that you can run it in a packaging branch and so delete all the packaging. Finally we looked at Looms with mathiaz who is hoping to get the MySQL packages in Looms for both Debian and Ubuntu. We identified some rough spots and a missing command (import-upstream) but it seems doable, if not /nice/ today. After that we talked about a sparser loom merge graph. The basic idea I have is that while the stack seems essential to providing a simple UI, all the merge commits make a lot of noise. So if we only do a merge commit when a conflict has happened, and otherwise depend on 'record' telling us what is incorporated, we can save a lot of commits and make 'log' or 'bzr viz' clearer, as well as making it simpler to cherrypick patches. There seem to be several related issues that tie into this: * Should the ready-to-build on-disk image with patch files be something
Re: UDD @ Portland
On Thu, 11 Feb 2010 13:33:27 +1100, Martin Pool m...@canonical.com wrote: I'd like to let looms progress, but not (unless james or others feel differently) add them into the dependency chain for getting UDD going. No, we don't have to add it to the chain to get it going, but I think it's one ingredient of having a great system. iow people should be able to try them on particular branches, without mandating them for all package branches, and (perhaps?) without requiring everyone working on that package to use them. I think a gradual migration path is something to aim for. What we want is consistency of interaction. I don't want to have to work out what is going on in the packaging branch before I can start work on it. Allowing branch; hack; build; push regardless of what's going on and allowing others to delve more deeply is one way, another would be to have bzr add-patch or something that prepared the tree for working, I'm sure there are more. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: hottest100 (was Re: Bazaar focus for 2.1 and 2.2)
On Wed, 3 Feb 2010 16:40:12 +, Martin Pool m...@canonical.com wrote: * help james_w with some of the bugs opened against the package importer (assuming he wants it) I would love it. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: [Launchpad-dev] RFC on build from branch UI
On Tue, 2 Feb 2010 11:01:49 +0100, Michael Nelson michael.nel...@canonical.com wrote: Hi all, I've been putting together some mockups for the build-from-branch UI work (with the help of jml, wgrant and mwhudson), and would love to hear ideas on how they could be improved or re-worked etc. You can see the details at: https://dev.launchpad.net/BuildBranchToArchiveUI Thanks for working on this Michael, it's great to be able to review it as a whole at this time, rather than making comments as the changes arrive. I have some comments that are quite fundamental, but let me say first that I like the design for interaction, and I'm confident that whatever you end up going with will be a pleasure to use. The main point that underlies all my comemnts is that your focus is on building a branch using a recipe, rather than building a recipe. I think that it is important to have something on a branch page to get started, but I'm not sure that driving everything from the point of view of the branch is the right thing to do. For instance, I might go to the branch page for the packaging branch, and want to do a build from there, but for daily builds that won't be the base branch. If I were to make a suggestion it may be for the following: * A branch page has a list of recipes that mention the branch, and a link to add a new one. * You can navigate to the +recipe page from there, and from there there is a build this button, which launches the archive picker. It's extra clicks I think, but it makes recipes more of a first-class object, rather than trying to make them an implementation detail, which is what your layout feels like. This isn't just me wanting my work to be front and center :-). The recipe can be more than just a pointer to the packaging branch. I would be anxious for people that don't really care to get something that works as quickly as possible though. Some further unstructured comments: When viewing a source package recipe build within their PPA, users can: easily go to the recipe that was used and from there to the branches. They should also be able to go to the manifest and so see the revisions that were used. I'm keen for these links to be more common in Launchpad, from e.g. bugs to the revisions that fix them etc. Being able to go binary package-source package-manifest- branches will be useful. Having the recipe editable from the recipe selector seems a little odd to me. Reducing the rounds trips would be good, but it may confuse some people who don't realise they are editing a shared object. Given that they are series-specific this won't cause too many problems in practice though I expect. Should a recipe handle all distro series?! - If we are going to do this then why bother building the source package on each release? Build the source once and then rebuild it tweaking the binary package version? It may be slightly more work to go from a branch that can build a source package for each release and a branch that can build a source package that can be built on each release (small difference). Would be a great feature to have, but I'm not sure that it's the right place to implement it. Slow moving projects, duplicate rejects either do what Aaron suggests, or provide the manifest to the buildd (which I want to do anyway), and use the mode where if the recipe resolves to the same revisions from the same branches as last time it wont build. Do we currently detect when *none* of the branches associated with a recipe add a debian directory? We could add a better error message to bzr-builder so that when they get the build failure email they will know what to do. I'm assuming that part of our validation when creating a build should be that the evaluated deb-version will be greater than all current versions in the target PPA? I'm not sure we can do that? The destination target needs a progress indicator. For PPAs, the detail packages page should show the source package in the construction (builddeb) phase, which conceptually is before anything it shows right now. How will we do this? I think adding a line, perhaps with a different background, and with (building) or something added, but only in the maintainer's view of the PPA. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: New overview page for the Bazaar importer
On Mon, 01 Feb 2010 20:10:15 +, Max Bowsher m...@f2s.com wrote: On 08/01/10 00:52, James Westby wrote: Hi, After a few days spent doing some infrastructure work I'm now about to present a more coherent interface to the current status of the bzr importer. You can find it at http://package-import.ubuntu.com/failures/.bzr/failures/index.html This seems to have gone away. Where does it live now? Yes, sorry, the sysadmins accidentally obscured it by cleaning up some redirections as they didn't realise it was there. It does mean we can have sensible URLs now though :-) http://package-import.ubuntu.com/status/ I've added an index.html at the top level as well now, which should make it easier to find. I also added http://package-import.ubuntu.com/merges/ http://package-import.ubuntu.com/diffs/ which, while the presentation is really poor, aim to be replacements for patches.ubuntu.com and merges.ubuntu.com. I don't think they are working correctly yet, suggestions for how to imporove it (or even patches) welcome. James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: New overview page for the Bazaar importer
On Fri, 15 Jan 2010 10:50:39 -0500, Francis J. Lacoste francis.laco...@canonical.com wrote: Looking at the signature today, I only see the first one as network related: 2 packages failed to many times to retry with key launchpadlib.errors.HTTPError:module:main:get_versions:iterate_collection:get_collection_slice:get:_request Being discussed in https://bugs.edge.launchpad.net/soyuz/+bug/513491 It's not network related, but is often a transient error on the LP side. The soyuz team have been helping to reduce the number of timeouts we get, and it is much better now. too many times to retry meant that they were auto-retried several times, but they hit the limit and so are awaiting investigation in case they are not transient errors. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Feedback on merging via bzr
On Thu, 28 Jan 2010 09:38:20 -0500 (EST), Scott Kitterman ubu...@kitterman.com wrote: Then maybe the description of what is happening just needs to be improved because as I said in the bug, it sure appears to be looking upstream and getting the tarball from there before it checks Debian. I understand the theory, it isn't clear to me that's what's happening. Both of you are correct. pristine-tar is supposed to make the problems go away, but it isn't perfect. When it doesn't work bzr-builddeb will try some other methods to grab the tarball, but I didn't think to try Debian explicitly (if you have deb-src lines for debian in your sources.list it should work). I intend to add this feature when I get a little time, or I would happily help someone else add it. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Where to see the list of merges/uploads that need review?
On Mon, 01 Feb 2010 12:14:44 +1100, Robert Collins robe...@robertcollins.net wrote: On Sun, 2010-01-31 at 14:56 -0800, James Westby wrote: I've been bugging the LP developers about this at every opportunity for the past few weeks, it's a rather large hole currently. The links posted are the best we have right now, but they are not API accessible, so it's not easy to fold it in to Daniel's page. Its not API accessible? Is there a bug open we can look at? https://bugs.edge.launchpad.net/launchpad-code/+bug/411357 https://bugs.edge.launchpad.net/launchpad-code/+bug/483931 Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Feedback on merging via bzr
On Mon, 18 Jan 2010 15:33:30 -0500, Barry Warsaw ba...@canonical.com wrote: Unfortunately, the diff that's automatically generated on the merge proposal isn't what Scott wants for the review. As shown above, he really wants the diffs between the branch and current Ubuntu, and branch and current Debian. Perhaps merge proposals for distro packages need a different kind of automatic diff? It could do that, but we would only want it for merge proposals for merges from Debian, and that's tricky to detect, and there is currently no facility to select they type of proposal. In addition, you often want to look at any of a few different types of diffs. To that end I filed a bug against launchpad-code the other day requesting attachments on merge-proposals, which will allow us to attach the extra diffs so the reviewer doesn't have to generate them. We can then provide a wrapper command that proposes the merge and attaches the diffs. If you like this idea then I suggest you go subscribe to the bug report and comment with any further suggestions you have for how the feature should work. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Feedback on merging via bzr
On Sun, 17 Jan 2010 23:35:05 -0500, Andrew SB a.star...@gmail.com wrote: Ideally, the script would also produce something along the lines of MoM's REPORT, listing the conflicts and giving instructions on how to proceed. I'm working on something to do this centrally to replace merges.ubuntu.com, but it's not ready for use yet. I'm not sure writing a file when used locally would be all that useful would it? Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Feedback on merging via bzr
On Mon, 18 Jan 2010 14:26:48 -0500, Barry Warsaw ba...@canonical.com wrote: I think it would make sense to submit bug reports to help streamline things, but I'm not sure where those bug reports would go. On the bzr-builddeb plugin on Launchpad? Yes, I appreciate everyone's feedback here. I would love to get more feedback in terms of bug reports. Even if you aren't sure that it is a bug as such, if you think something is too complicated then it shows that we probably need to make some changes, and I'm happy to work with you in a bug report to work out what would solve the problem. Plus, bug reports stick around as reminders of what needs to be fixed. bzr-builddeb is a fine place to report them, if you know somewhere more specific then you can do that, but I am happy to reassign if needed. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Feedback on merging via bzr
On Wed, 20 Jan 2010 09:36:38 +, Dmitrijs Ledkovs dmitrij.led...@gmail.com wrote: Hmmm = I *think* lp can do this already In the merge proposals options set lp:debian/sid/package as pre-requisite. Then the diff will be the one sponsors require! Wow, that's a cunning idea. I'm not sure it's the right thing to do as it's abusing that field somewhat, and also means it is not free for its intended use should the need arise. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Feedback on merging via bzr
On 18 Jan 2010 12:07:41 +0100, Michael Bienia mich...@bienia.de wrote: On 2010-01-17 22:45:53 -0500, Scott Kitterman wrote: Currently I often insert a step 0 before starting on merging with bzr: - Check if the Debian branch is uptodate. Perhaps I got unlucky when picking a package to merge but several of the packages I wanted to merge had outdated Debian branches so I had to do it by hand (download the Debian .dsc and the Ubuntu delta from PTS and merge it). Yes, this has been a problem due to operational issues. I hope that it will be far less frequent in future. I'm sorry you got hit repeatedly. I usually do a bzr init-repo before and branch both the Ubuntu and Debian branch and use the local Debian branch for merging and diffing. It would be really nice if bzr warned one if the format of local branch and the remote repository differ *before* I start working on something. Please file a bug against bzr. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Feedback on merging via bzr
On Mon, 18 Jan 2010 06:56:13 -0600, charlie cj...@cableone.net wrote: On Mon, 2010-01-18 at 12:07 +0100, Michael Bienia wrote: On 2010-01-17 22:45:53 -0500, Scott Kitterman wrote: Currently I often insert a step 0 before starting on merging with bzr: - Check if the Debian branch is uptodate. This has bitten me a few times. When you go to a packages launchpad page under branches, it would be nice if there were links to the ubuntu and debian branches. This way one could quickly determine if the debian branch has been updated. Please file a bug against launchpad-code requesting this. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Feedback on merging via bzr
On Sun, 17 Jan 2010 22:45:53 -0500, Scott Kitterman ubu...@kitterman.com wrote: I spent some time over the holidays giving merging via bzr and the UDD tools. I understand that development of the tools to support this is still a work in progress and the much of this feedback probably represents work that you already know needs doing. This mail is based on notes I took recently while doing a merge of regina-normal. As merges go, it was not a complex one. Hi Scott, Thanks for taking the time to try Bazaar and writing up this detailed feedback, it is very valuable. I'm writing this from the perspective of someone who is reasonably familiar with merging using merge-o-matic (~3 years of experience), but relatively knew to bzr. I would have been dead almost immediately if not for the w.u.c documentation [1]. I'm glad there was at least documentation to help you through it. Step 1: Getting the source: Using MoM, I would have grab-merge.sh regina-normal and then had local copies of the common ancestor package, the current Debian package, the current Ubuntu package, and a proposed merge with any conflicting files listed. The UDD equivalent seems to be: $bzr branch lp:ubuntu/regina-normal regina-normal $password for ssh key $cd regina-normal $bzr merge-package lp:debian/squeeze/regina-normal Text conflict in debian/changelog 1 conflicts encountered. The merge resulted in 1 conflicts. Please resolve these and commit the changes with bzr commit. This gives me the proposed merge. The conflict in this case seems to occur with every merge I try. The most recent Debian and Ubuntu debian/changelog entries conflict with each other and the file has to be edited to move the most recent Debian entry above the most recent Ubuntu entry. Additionally, MoM would create a stub changelog entry for the new merged package which was quite convenient. None of this additional work is difficult, but it is tedious and seems ripe for automation. Indeed it is. I'm more than happy to help someone implement these things, they won't be particularly hard, I just have more pressing issues before I get to them. At least the changelog stuff will be resolved before we release Lucid either way. I would love to have something more akin to grab-merge, and again I would be happy to help someone work on this. I would suggest it get added to bzr-builddeb as that will make some things easier. Step 3 checking the package: At this point I want to check the package against the previous Debian and Ubuntu packages to make sure I have it correct. Traditionally, I would locally debdiff the proposed merge with both the previous Debian and Ubuntu packages to make sure I had documented all of the Ubuntu diff and not lost any needed changes in the merge. To do it the new way, I did: $bzr diff --old lp:debian/squeeze/regina-normal | less ssh key (repeat, including redownloading each time the diff is done) I assume bzr diff --old lp:ubuntu/regina-normal would have worked for the diff from the previous Ubuntu package, but it isn't documented and I didn't think to try it at the time. Indeed it would have. Then commit the result: $ bzr ci -m * Update debian/changelog to include undocumented differences from Debian * Remove extraneous whitespace changes Committing to: ~/devel/boost/merge/regina-normal/ modified debian/changelog modified python/testsuite/trigeneral.test Committed revision 20. This method of diffing works fine, except that the previous branch has to be re- downloaded each time the diff is done. In this case I was trying to remove extraneous white space changes that had crept into the packages so it took several tries to get them all. For larger packages or people with a slow internet connection, the need to re-download the diff will have a substantial negative impact. Additionally, I miss a way to diff just the debian directories. For new upstream releases (which this wasn't, so I didn't hit it) I find this a critical way to review the packaging differences between the old and new packages. How did you diff just ./debian/ before if not for filterdiff? Why can't you do that now? Step 4: Building the package: In the old method, I would debuild -S (-sa if needed) -v and have a package ready for uploading. bzr builddeb -S -- -v4.6-1ubuntu1 Now this one looks easy, but has the hidden trap of bzr builddeb providing only -S from dpkg-buildpackage and requiring an extra flag to pass other options to dpkg-buildpackage (--). I think this needs a rethink [2]. Indeed it does. It was never intended to be a lasting solution, but it was a stop-gap from the bzr builddeb -S --builder debuild -S -v4.6-1ubuntu1 that it was before. I knew it was non-obvious, but at least it was there. Again, I will happily help someone who wants to fix this, it should be fairly straightforward. Step 6: Uploading
Re: Feedback on merging via bzr
On Mon, 25 Jan 2010 03:35:40 +, Dmitrijs Ledkovs dmitrij.led...@gmail.com wrote: 2010/1/25 James Westby jw+deb...@jameswestby.net: bzr-builddeb is a fine place to report them, if you know somewhere more specific then you can do that, but I am happy to reassign if needed. I guess launchpad.net/udd is not nice place for bugreports? Swamped with what it seems like automatic bug reports It works, and I plan to clear out the automatic reports soon. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Breakdown of import failures in the hottest100
On Fri, 08 Jan 2010 13:49:54 +1000, Ian Clatworthy ian.clatwor...@canonical.com wrote: Packages without products - gst0.10-python - gstreamer Python bindings? http://gstreamer.freedesktop.org/modules/gst-python.html linux-restricted-modules Cobbled together from various places. nspluginwrapper http://gwenole.beauchesne.info/projects/nspluginwrapper/ http://svn.beauchesne.info/svn/gwenole/projects/nspluginwrapper/trunk sysvinit http://changelogs.ubuntu.com/changelogs/pool/main/s/sysvinit/sysvinit_2.86.ds1-61ubuntu16/copyright Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: hottest100 (was Re: Bazaar focus for 2.1 and 2.2)
On Tue, 5 Jan 2010 18:33:29 +1100, Martin Pool m...@canonical.com wrote: I put jml's query output into a Google spreadsheet, so that we can annotate lines with the relevant bug etc. http://spreadsheets.google.com/ccc?key=0Ag3S65cphSMHdG1VckNSRXI4OHBmVmxGaklGVW4tcWchl=en_GB. I'll put another plug in for https://answers.edge.launchpad.net/launchpad/+question/94298 which will push the percentage of branches in that spreadsheet much higher. Thanks, James P.S. Naming a project also helps with searching for the mails about it, thanks. -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: hottest100 (was Re: Bazaar focus for 2.1 and 2.2)
On Tue, 05 Jan 2010 10:32:08 -0600, John Arbash Meinel j...@arbash-meinel.com wrote: So... how do we do that? Who gets assigned that task? I certainly don't feel like I have any ability to make that change. I think it's normally the job of the LP CHR. I was hoping one of the LP people on the list would help with that. I'll also note that all of the move these branches to ~vcs-imports links are 'broken', so I assume they have already been done. Yep, Tom did those for us. However, I can see stuff like: https://code.edge.launchpad.net/~vcs-imports/wine/git-trunk is listed, but https://edge.launchpad.net/wine has a 'trunk series' here: https://edge.launchpad.net/wine/trunk But has not associated branch. So the associating development focus does not seem to have been done. Again, I don't think I have access to actually change any of this stuff. (And obviously neither do you, or you probably would have done it. :) Exactly. It will generally need someone in Registry Admin to do it for us. I'm just bringing it up here as the question itself isn't getting the attention, and doing these operations will help with the hottest100 goal. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Bazaar focus for 2.1 and 2.2
On Tue, 22 Dec 2009 10:49:14 +1100, Martin Pool m...@canonical.com wrote: I think after the break we should focus on the vcs-imports of the top 100 Ubuntu packages until they're all working well. jml and spm helped with some scripts to query their current state, and we can map that into a spreadsheet showing the root cause for each failure. If you can get someone to move on https://answers.edge.launchpad.net/launchpad/+question/94298 then it will make your numbers more representative. You can use https://edge.launchpad.net/ubuntu/+upstreamreport to track which packages don't have a VCS by looking for a lack of a Bazaar icon. Checking the ones that do to ensure that it points to somewhere useful would probably be wise too. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: your thoughts wanted on bzr team UDD focus
On Thu Dec 03 01:05:00 -0500 2009 Andrew Bennetts wrote: Are there any other UDD-related bugs that should be filed (or existing ones tagged)? Is the Bazaar team looking for all issues to be filed as bugs. I'm happy to do that, but a lot of things we have been discussing would currently considered to be too imprecise for a good bug report. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: your thoughts wanted on bzr team UDD focus
On Mon Nov 30 20:19:13 -0500 2009 Martin Pool wrote: I'd like to get a sanity check from UDD people on what the Bazaar team is going to do for our 2.1 release, which will have a feature freeze in February and go into Karmic. From the previous threads, it seems that the main large things we need to do are: Hi, Thanks for requesting feedback. There were a bunch of other things discussed, and scenarios considered, and these two items are the only things that came out the other end. You are obviously the one that should decide how much work to take on given the developer time available, so I'm not complaining that the list is short. I do however wonder about the other features that were discussed. Did you decide that these gave the best return on investment, or was it just that these were the items most discussed? * get more code imports working -- investigation of failure causes is continuing Are you setting any sort of targets, or is it too early to know how varied the causes are and so know what a sensible target would be? I am currently distilling UDS discussions in to specs, so I will be sending them to the list shortly, this may inspire some discussion of needed features. There are a couple of items that I will highlight for discussion. Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Bugs for daily build packages
Hi, If we are planning to massively increase the number of daily build packages in PPAs then I think it behooves us to also solve this problem, other wise we are not only throwing away useful information, but also pouring buckets of water on to the slope when we know there will be a freeze tonight. Currently there is no certain place to report bugs for packages from PPAs. They are not an LP bug target. Sometimes they should go against the Ubuntu package (some Ubuntu developers request this now for their bleeding edge PPAs), more often against the upstream. There is no way to indicate this currently either, so even if we know then it's not easy to tell e.g. apport where to file the bugs. When I said we are throwing away useful information I meant this: install a package from a PPA, make it crash. Apport will pop-up and tell you that it crashed and ask you if you would like to proceed with filing a bug, you answer “yes.” It will think for a minute and then tell you that you can't as it is not an “official Ubuntu package.” You can install apport-retrace, retrace the crash report manually, gather the needed bits, find where to report the bug and do so yourself, but that's so 2009. Therefore I think we need to look at improving the situation. We can't make it all perfect without some very big changes across a lot of projects, but we can make some big practical differences. The aim is to have lots of people testing the latest code, let's enable them to report issues with it easily, otherwise we run the risk of it being a big waste of time. Here is a strawman proposal for how it should work, please amend or propose alternatives where you know of something better: 1. Allow a PPA to specify a bug target. Either an LP project or package, or a URL of an external bug tracker. 2. Export this information in the Packages file using the existing Bugs field. 3. Extend apport to use this information when it detects the package is not from Ubuntu. 4. Where the target is on LP use much the same system as it currently does. Where the target is not on LP, but uses a bugtracker it understands, make use of that knowledge to file the bug. Otherwise dump the useful information to a text file, open the specified URL and tell the user to attach the file(s). 5. Possibly extend apport to retrace the reports locally, so that this does not have to be done after they are reported. (Duplicating the infrastructure we have in Ubuntu for everything would probably be a mistake.) Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: udd team on launchpad
On Thu Nov 19 20:09:56 -0600 2009 Robert Collins wrote: What does the udd team on launchpad govern? I think it's just the maintainer of the udd project, to get bug mail etc. IIRC you created, what purpose did you have in mind? Who should be in it? We need to know the purpose before we can answer that. who should govern who is in it (Currently James Westby and I do, but perhaps it should be delegated from e.g. the TB)/ If it is used in a structural way within Ubuntu then the TB is appropriate. If it is just used for organising our work then I don't think it is. does it need to be moderated or should it be open for anyone to join? Again we need to know the purpose. The description just says 'folk that are interested in udd', which suggests making it open would be safe. I would agree if that is all it is for. Do we have any need for a controlled team that we can grant permissions to? You have made me think of the ~ubuntu-branches team, should that be owned by the TB jml? Thanks, James -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: A rather tricky UDD merge
On Wed Nov 18 18:26:35 -0600 2009 Max Bowsher wrote:
I have merged the subversion package from sid aiming for lucid. The
merge was ... interesting ... and I'm initially chronicling my
experiences here - they can go into a bug once I understand exactly what
bug I'm filing.
In a nutshell, bzr merge-package ../debian/sid produced lots of
spurious conflicts. This turned out to be because it was trying re-merge
changes from 1.6.{3,4,5}dfsg-1 into 1.6.5dfsg-1ubuntu1 which already
contained them.
The underlying cause here was that the karmic|lucid branch contains an
import of the 1.6.5-1ubuntu1 version without any recorded bzr ancestry
tying it to Debian revisions. It seems potentially relevant that the
1.6.5-1ubuntu1 upload was a merge from sid, where the previous merge was
from experimental.
Thanks for tracking this down. I can't say without looking what the
cause of this was, it's possible the merge from experimental is
involved, though in theory the code doesn't really care where the
merges come from. I'll be able to look at the logs of the import and
hopefully that may give a clue.
I was able piece the history back into a sane state by:
1) Making a commit containing no changes with left-parent the Debian
upstream-1.6.5dfsg revision, right-parent the Ubuntu upstream-1.6.5dfsg
revision - i.e. tying together the two upstream-1.6.5dfsg imports in
history.
2) Merging said commit into the Ubuntu packaging branch, having the
effect of replacing all the differing file-ids in the Ubuntu branch with
the ones from the Debian branch.
3) Merging the Debian 1.6.5dfsg-1 revision into the Ubuntu branch (i.e.
the version which the Ubuntu branch was based on, but did not record
ancestry for), and dealing with the conflicts mainly by bzr reverting
them, except where there was a file-id conflict, in which case picking
the text of the Ubuntu version *but* the file-id of the Debian version.
At which point I was in a position to simply bzr merge (not
merge-package) the debian/sid branch into karmic|lucid branch, resolve
the genuine conflicts between ubuntu and debian packagings and commit.
It sounds like you took the right approach to fixing it to me, does anyone
on the bzr team have any suggestions for ways it could be improved?
I hope that no-one else has to go through this, but I'm at a loss for ways
to do this except for ensuring that all merges are accurately tracked.
I'm not sure that merge-package should be automating those steps for you
if it can detect the situation.
The branch, by the way, is at lp:~maxb/ubuntu/lucid/subversion/merge. I
recommend you examine it with bzr qlog if at all - the history is too
tortuous for a non-graphical visualization.
Thanks, I'll take a look, and also try the original merge-package myself.
So the open questions:
1) What should the importer have actually done? Why didn't it record any
ancestry from Debian against the 1.6.5-1ubuntu1 import?
I will investigate this.
2) Did my way of tying the diverged ancestry back together make sense?
To me, yes.
3) Do we want to retroactively replace history with a less contorted
version?
I don't think we do.
4) When an upload to unstable occurs with a higher version than the
version in experimental, should the importer pull experimental into
unstable before importing? That would seem to make sense to me, and
would eliminate one notable source of diverged upstream import branches
when the importer needed to pull the upstream import into an ubuntu branch.
This is what is supposed to happen, assuming that the experimental
changelog entries are included in the unstable upload. If not then
I don't think we should be inferring a relationship, as it would cause
other issues if a merge was later attempted.
(Consider someone doing some long-term work in experimental when a new
upstream is released. As the work in experimental wasn't ready for
unstable they would just upload the new upstream to unstable and
then merge back. Granted, they wouldn't be likely to be using the
importer, but it illustrates the issues with inferring relationships
that the changelog doesn't include.)
Thanks,
James
--
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
