Re: Moving udd away from sqlite
On Tue, Jun 19, 2012 at 3:09 AM, James Westby james.wes...@canonical.com wrote: Since https://bugs.launchpad.net/udd/+bug/724893 has been filed, we haven't understood *when* the importer fails but we know it can fail and it failed more often recently (and a lot during your first attempt if I remember correctly). If you have a fix for that, great ! Show us how it's fixed with a test. I'm still not sure what test would be satisfactory for doing this. The above setup isn't amendable to a fast unit test. And the more tests we add the easier it becomes to add a new one. Agreed, but I'm not sure that we are working in an area where unit tests are a good fix. A unit test can't prove the absence of deadlocks in a multi-process system. Further to that, no amount of testing can show whether an architecture is good or bad: the architecture talks to things like *tendency to fail* and expected or predictable emergent behaviour. Testing, of various sorts, can help you measure particular aspects like - 'in situation X, does a failure occur', but it cannot tell you which situation X's to test, nor which situation Y's a test isn't needed because the structure supports the desired use. So in this case its very simple: SQLite is designed for a single-process embedded DB use case. The import is not such a use case, and *all* the failure modes you are experiencing are predicted by the statement 'using SQLite with multiple processes'. Postgresql is designed for multiple processes working on the system at once, as long as no schema change are being made: schema changes will cause lock contention, as will reading the same rows at maximum isolation - which is why LP, for instance, explicitly loosens the isolation we have. AIUI we want, and we know we want, to move to postgresql, in production, as soon as possible. The storm migration is an attempt to do that. Some routes are: - move to postgresql using the lower level APIs and add storm later - work on the storm patchset to make it reliable with sqlite - do both postgresql and sqlite at the same time. My recommendation, given the existing investment, is to bite the bullet: bring up a postgresql instance in staging and prod, dump and import the data to it, and switch over to postgresql. Debug any issues in real-time, and move on. The depth of potential issues is sharply limited, because postgresql handles many more combinations of client interaction than SQLite (by design and necessity). -Rob -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd to django
On Thu, Dec 15, 2011 at 7:08 AM, James Westby jw+deb...@jameswestby.net wrote: My plan is delete add-import-jobs, and an a POST handler that gets told when there are new packages to scan. 'there is work to do now' is a classic pub-sub situation. Rather than a post handler, I suggest you want one of: - a webhook - a pubsubhubbub subscription - or an AMQP consumer Depending on whether: - the thing calling you back has direct access - or you can get an rss feed of it - or its in-dc only and we can use our canned solution for things eventlike. -Rob -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd to django
On Sun, Dec 11, 2011 at 4:28 AM, James Westby jw+deb...@jameswestby.net wrote: On Sat, 10 Dec 2011 17:52:40 +1300, Robert Collins robe...@robertcollins.net wrote: On Sat, Dec 10, 2011 at 3:10 PM, James Westby jw+deb...@jameswestby.net wrote: Hi, I think there are a few reasons that we should consider moving udd to django (more on what this actually means later.) I'm curious what data udd stores. There are two sets. There's the bookkeeping data for running import-package each time a package is uploaded to Debian or Ubuntu, keeping track of failures, etc. It might be interesting - as a thought experiment if nothing else - to consider failures a form of crash and upload them to a crash database rather than processing them inside udd - e.g. toss them out over amqp. -Rob -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd to django
On Sun, Dec 11, 2011 at 7:19 AM, James Westby jw+deb...@jameswestby.net wrote:
On Sun, 11 Dec 2011 06:38:25 +1300, Robert Collins
robe...@robertcollins.net wrote:
It might be interesting - as a thought experiment if nothing else - to
consider failures a form of crash and upload them to a crash database
rather than processing them inside udd - e.g. toss them out over amqp.
The current design is that it also uses them to avoid acting on that
package until there is manual intervention.
So it may be interesting to use oopses to communicate the problems and
do analysis of them, but it won't avoid storing the data as well unless
that model is changed.
I'm sure the data needs to be stored somewhere. If its in a crash
database, then you can ask 'has this crash been handled' where I guess
handled means flagging the crash signature in someway ('retry', 'bug
X', ..). The poential benefit I see is one less place writing fault
aggregation heuristics.
-Rob
--
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Moving udd to django
On Sat, Dec 10, 2011 at 3:10 PM, James Westby jw+deb...@jameswestby.net wrote: Hi, I think there are a few reasons that we should consider moving udd to django (more on what this actually means later.) I'm curious what data udd stores. I'm +1000 on moving it towards a service, something that can be driven without access to the machine will make making it fully webops much simpler. You could consider storm rather than the django ORM, but I don't think we have a compelling alternative to Django for html templates at the moment (other than 'zope' which is fairly ill defined - but these days probably means pyramid. HTH -Rob -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: bzr get-orig-source
On Tue, Oct 4, 2011 at 11:10 PM, Jonathan Riddell jridd...@ubuntu.com wrote: I added a command to builddeb, bzr get-orig-source. This helps with my workflow .. Others prefer to use bzr builddeb [-- -nc] but I find all the changing directories that requires is more hassle than gain. What directory changing? -Rob -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: The append-revisions-only can of worms
On Tue, Jul 19, 2011 at 11:06 AM, Max Bowsher _...@maxb.eu wrote: On 18/07/11 21:24, James Westby wrote: And of course, if the package has *ever* had an Ubuntu specific upload, that's an AppendRevisionsOnlyViolation. So we've actually found a bug here :-( Yeah, I think the logic should be to do a merge taking all the content from Ubuntu, with the parents being (debian, ubuntu)? Well... The logic *should* be to have imported into the Debian branch first and then merged into the Ubuntu branch, such that this situation never arises. I don't think thats correct. Firstly, some devs upload the same version to both Ubuntu and Debian (typically identically), and we need to handle that (both identical and not). Secondly, if someone reconciles Debian and Ubuntu they may discard interim versions when merging into Debian (not recommended, but happens) - if that happens we still want the Ubuntu series-of-uploads to be correct [that is, we need to keep the tags for Ubuntu intact, and the history intact]. Rob -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Loom merge
On Sat, Feb 5, 2011 at 6:00 AM, John Arbash Meinel j...@arbash-meinel.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I've been thinking about what it should mean when merging looms, and I was wondering what other people think. My specific thought is that loom merge should operate on the loom itself, step by step. Consider this example: Thats basically it, Note that the working loom has room to record the tips of each thread in much the same way a working tree can record pending merges, with the intent of the same basic workflow: $ bzr merge otherloom - does as much as can be done until conflicts occur; writes all state to disk, $ bzr conflicts - tells you about conflicts (we could add bzr loom-conflicts, or union the output) $ bzr up-thread/down-thread can put you into a pending-merge state or whatever as appropriate -Rob -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Making bzr commit more like debcommit
Should be easy enough to add a hook to get fixes from plugins; the tricky bit is allowing users to confirm that the detected things are correct (or perhaps thats overkill?) Anyhow the bzr-builddeb, commitfromnews, launchpad plugins would all probably like this. -Rob -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Looking into builddeb bugs a bit
I'm going to spent a little bit of time looking into bzr builddeb bugs. Are there any (ignore the bug tracker) of particular interest to folk on either of these two lists? Thanks, Rob -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Lots of conflicts when trying to merge protobuf to maverick
Hi Elliot, as we discussed on IRC - please make this a bug, and also store the two branches somewhere so that the package importer won't destroy the evidence. There's clearly something wrong, even if its just data, we should fix it and make it more obvious whats up. -Rob -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
diagnosing collisions
Is there any docs on doing this? Many, if not most of the udd bugs are auto generated bug reports like https://bugs.edge.launchpad.net/udd/+bug/414095. I'd like to analyse and fix these, but currently I'm starting from first principles. -Rob signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: import failures
On Thu, 2010-02-18 at 14:46 +1300, Michael Hudson wrote:
Robert Collins wrote:
On Thu, 2010-02-18 at 13:22 +1300, Michael Hudson wrote:
James Westby wrote:
On Wed, 17 Feb 2010 08:14:26 +1300, Michael Hudson
michael.hud...@canonical.com wrote:
For one reason and another, Tim ran the script in the end, but it looks
like it's finished now but you might want to rerun your script to check.
Thanks to both of you.
However, the attached didn't seem to stick. Only 57 branches this time
though.
So a bit of poking has got almost all of these up to 2a. There are 5
that seem to be broken:
Details please.
Clicking the links will get you as many details as I have.
Ah:
CHKInventoryRepository('lp-hosted:///~ubuntu-branches/debian/sid/camlp5/sid/.bzr/repository',
fallback_repositories=[CHKInventoryRepository('lp-hosted:///~ubuntu-branches/debian/squeeze/camlp5/squeeze/.bzr/repository')])
has no revision james.wes...@ubuntu.com-20091029134422-acd9dzuxa44whcct
We have a bug open on this - I think we should make it critical.
-Rob
signature.asc
Description: This is a digitally signed message part
--
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: import failures
On Wed, 2010-02-10 at 21:46 +, James Westby wrote: Some of them have been upgraded. If it's easier for me to do an info against all of them and filter out those not in 2a then I can do so. I think thats easiest. signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
UDD @ Portland
James Westby and I had some time together in Portland to talk about UDD stuff. We talked about a few things: * Looms, their use today and where they should go * The operational issues with the package importer and how the bzr team can help * analysed a few specific bugs and tried to come up with solutions. Firstly though, a couple of overview points: - the udd project has 200 bugs on it. While many of these are 'collision' reports many are not. The collision reports are currently overly noisy, so please ignore them for now. However, the other bugs are open season for people to fix - and every bug fixed there will streamline things for people using bzr to package in Ubuntu. - Measurement error: the hottest 100 has a fairly high error rate for package imports at the moment, OTOH its looking at precisely the packages most likely to fail. James says that overall 96% or so of packages import successfully. - Adoption. I spoke with pitti and seb128 about how the desktop team uses bzr. Mostly they use packaging-only branches, which they prefer precisely because they don't need to download all the history. A quick test showed them getting 2M of data for gnome-panel, and 14M when using the package-imported branch. So we need to do some things here for people that do many drive-by fixes... - make downloading only some history easier/possible - make it possible to be using a mirror of the VCS data so it doesn't have to go over the network (perhaps the Ubuntu mirror network could carry the packaging branches?) - make bzr saturate the network more effectively (the 14M above didn't download at wire speed as far as we could tell). I haven't made these into bugs as yet, pending some feedback from you! Ok, onto the fun stuff. While long term we want a Launchpad control panel for the package importer, James thinks its reasonable that folk helping with the operation of it should be able to directly kick it off - so he has filed an RT ticket to get more access to that machine. James, whats the RT ticket number? Opening this up will let us rerun imports more promptly that appear to have had only spurious failures. Bugs with the importer can and should be debugged on peoples development environment - there is an earlier mail from December documenting how to do this. We should put that in the Wiki I think think. The collisions that are reported as bugs can be divided into three broad groups: - impossible (a collision in debian: at least at the moment, we don't expect people uploading to Debian packaging-branches. Well, *generally speaking* we don't expect this). (Nb: I do it for stuff I maintain in Debian :) - spurious (its not a collision, and a bug caused it) - genuine (it is a collision and it should be a merge proposal) We have a few collision specific tasks: - James is rapidly making new collisions be filed as merge proposals (unless they are in Debian imports) - we need to write a script to analyse the nearly 200 collisions in the bug tracker to highlight the debian imports (must be bugs, might be fixed), and convert the ubuntu ones to merge proposals. - We should delete the stale branches for collisions that we decide are bogus. Membership in the magic group ubuntu-branches is needed for that, and that group needs to be kept locked down (as it is equivalent to upload rights to the archive). So - lets make a list somewhere if you determine a branch isn't needed, and ping James or anyone on the tech board to delete such a branch). We looked at the workflow involved in packaging, and I'm very happy that James has seen the light and will be implementing an 'import-upstream' command to import and make a tarball micro-branch but not do the debian metadata updates. This will be useful for looms, where the two steps occur on different threads. Finally we looked at Looms with mathiaz who is hoping to get the MySQL packages in Looms for both Debian and Ubuntu. We identified some rough spots and a missing command (import-upstream) but it seems doable, if not /nice/ today. After that we talked about a sparser loom merge graph. The basic idea I have is that while the stack seems essential to providing a simple UI, all the merge commits make a lot of noise. So if we only do a merge commit when a conflict has happened, and otherwise depend on 'record' telling us what is incorporated, we can save a lot of commits and make 'log' or 'bzr viz' clearer, as well as making it simpler to cherrypick patches. There seem to be several related issues that tie into this: * Should the ready-to-build on-disk image with patch files be something stored in the loom, or something the loom models *and exports*. * Should someone editing a patch see a working tree with all the lower patches combined, or only their patch (and how does this tie into what gets committed - hairy logic incoming!) * How do people migrate into using Looms? I don't have good answers for all of these, but I'll try and write them up in more detail once
Re: UDD @ Portland
Martin asks what a collision is. The situation with package imports is that we have a branch B, which both Ubuntu developers and the package importer can commit to. Collisions are what happen when the package importer sees something arrive in the archive which is either not in, or different to, the branch history. E.g. developer A: bzr branch lp:ubuntu/foo # add some changes, call it UNRELEASED # hmm, I won't dput yet, I want to tweak it some more. bzr push :parent developer B: apt-get source foo # add different changes, call it N dput So we get a Y shape import graph, but the archive is official, so the package importer does a push --overwrite to 'win' on the packaging branch, and pushes the old head to a new temporary branch, and files a ticket in launchpad describing that this happened. -Rob signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Recipes vs. Looms vs. pipelines
On Tue, 2010-01-05 at 10:30 -0600, John Arbash Meinel wrote: If you have a purely 'stack' model, and have: - - feature2 - - feature1 - - upstream If someone wants just 'feature2' they have to cherrypick or get feature1. Only if we have merged feature1 stuff into feature2; and its quite possible to skip doing that while preserving a stack description - remember that each revision has dag info, so we don't need a dag at the list of branches level. The changes for loom itself, are stuff like 'up-thread' from feature1 = feature2 is just a 'switch'. But up-thread from feature2 = integration (or feature1) requires a merge from both feature1 and feature2 (probably committing inbetween.) I think I filed a bug, but I plan to reduce the noop commits massively here. -Rob signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: your thoughts wanted on bzr team UDD focus
On Thu, 2009-12-03 at 17:05 +1100, Andrew Bennetts wrote: Are there any other UDD-related bugs that should be filed (or existing ones tagged)? I suspect there are many, but we'll need to find them as we join dots up. I haven't read James' latest email, but I suspect it is relevant. -Rob signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: your thoughts wanted on bzr team UDD focus
On Tue, 2009-12-01 at 11:29 -0500, James Westby wrote: http://people.canonical.com/~jamesw/merge_changelog is one, there are other versions around. Note that with the merge-package command we don't need the hooks to be able to do this (though it would be nice to have anyway). I don't see anything on this list that would move us towards being able to drop merge-package. https://bugs.edge.launchpad.net/udd/+bug/491711 is a bug I've filed about merging specific files better; please provide feedback about whether you think it might work for you. I've marked it high, because things that make bzr-builddeb etc easier and simpler will help in maintenance of that code and its understandability. What do you need to be able to drop merge-package? -Rob signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: your thoughts wanted on bzr team UDD focus
On Tue, 2009-12-01 at 02:39 +0100, Jelmer Vernooij wrote: Hi Martin, On Tue, 2009-12-01 at 12:19 +1100, Martin Pool wrote: I'd like to get a sanity check from UDD people on what the Bazaar team is going to do for our 2.1 release, which will have a feature freeze in February and go into Karmic. From the previous threads, it seems that the main large things we need to do are: * get more code imports working -- investigation of failure causes is continuing * support imports of git submodules or svn externals -- this is not moving yet As for bzr-git/bzr-svn support for git submodules / svn externals: fwiw git submodules can (as of UDS when I discussed this with Dustin and Vincent) now be imported with newer versions of bzr-git, and are converted to by-reference nested trees. This will of course only work if the target repository supports storing nested trees (e.g. the experimental development-subtree format). This makes me extremely nervous. The last time a plugin did something the core didn't /support/, it took us 4 and a half years or so to fix it, because we were constrained by having lots of data around, and we didn't get to do the fix that would have been best (and been maybe 10% smaller on the converted data). _please please please please_ contribute to bzr's support first, and only enable this in bzr-git when there is a supported format that does nested trees. Until then, I suggest outputting one of: - a bzr-builder recipe - scm-proj - config-manager - by-value nested tree. -Rob signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
udd team on launchpad
What does the udd team on launchpad govern? Who should be in it? who should govern who is in it (Currently James Westby and I do, but perhaps it should be delegated from e.g. the TB)/ does it need to be moderated or should it be open for anyone to join? The description just says 'folk that are interested in udd', which suggests making it open would be safe. -Rob signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: bzr loves Ubuntu @ UDS
On Wed, 2009-11-18 at 18:26 -0600, Jelmer Vernooij wrote: I'm not suggesting getting *all* of the imports working before doing anything else. But getting the import for a particular project for which you're going to do daily builds working properly before doing something else with it seems like a better idea than fixing the branch up later. Right now we have no daily build infrastructure. On a case by case basis I don't disagree with you, but neither do I massively agree with you: - we have 16K+ packages to address eventually - 1-2K in 'main' - perhaps 100 key ones that really need this badly I think we should join the dots end to end on a few packages as soon as possible and then iterate sideways adding features to the daily builds and more imports to the mix. But we have enough imports already to permit focusing on the daily build aspects (a couple of K imports, 0 daily builds from trunk using lp infrastructure). Remember that the #1 priority is 'daily builds from trunk'. That /clear/y implies an import of those trunks we're trying to build. If you say #1 priority is imports of trunk, you get a totally different emergent effect. And thats what we are avoiding! Lastly, as we have to have good handling of history rewrites, I really don't care at the per-branch level whether we incur a rewrite or not: we have (already) 160K branches that eventually need rewrites. 1 here or there is noise. I don't think other folk should care as well; its a bridge we need to cross as part of this overall task. I don't think the failure rates are as bad as 25%-50%. For bzr-git the only real issue I'm aware of at this point are submodules (this is e.g. problematic for kvm/qemu); for bzr-svn the two main issues seem to be libsvn http bugs and odd history involving bzr-svn roundtripping. I don't know what the failure rates are like for cscvs? Fortunately, I was quoting an adhoc report we got done, rather than guessing. IIRC cscvs has the lowest failure rate, cscvs/svn the second and git the worst at the moment. (OTOH there are less git samples so bias may be present). when bzr-svn is rolled out I'd expect the bzr-svn failure rate to go way down. Also some of the failures may still be bad / moved URLS. Its a tricky figure to qualify at the moment. -Rob signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Merging with Bazaar -- Missing import of binutils-avr 2.20-1 and -2
On Thu, 2009-11-19 at 06:59 +, Max Bowsher wrote: Another import needing a nudge from you - neither 2.20-1 or -2 have been imported: I think it would be good to have a team of folk able to nudge things along, rather than bottlenecking on one person. -Rob signature.asc Description: This is a digitally signed message part -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
