Re: UDD @ Portland

[Aaron Bentley]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Robert Collins wrote:
>  - the udd project has 200 bugs on it. While many of these are
> 'collision' reports many are not.

What do you mean by collision?

Aaron
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkt0Pr0ACgkQ0F+nu1YWqI1nCgCeLhF9YJbZLWZ9GfixjrvXXCz1
+TwAn2DSBrmMcmeBlTH1Qi6ghrqnLaQh
=f0Tk
-END PGP SIGNATURE-

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD @ Portland

[Barry Warsaw]

On Feb 11, 2010, at 02:57 AM, James Westby wrote:

>On Thu, 11 Feb 2010 13:33:27 +1100, Martin Pool  wrote:
>> I'd like to let looms progress, but not (unless james or others feel
>> differently) add them into the dependency chain for getting UDD going.
>
>No, we don't have to add it to the chain to get it going, but I think
>it's one ingredient of having a great system.

I agree.  Looms help both the developer and reviewer focus in on the parts
that they really care about, and I think could help manage mostly boring
bookkeeping details (such has how to play nicely in whichever patch system is
being used).

Looms probably aren't essential for udd, but seem very important for really
fantastic (and fun!) opportunistic hacking.

-Barry


signature.asc
Description: PGP signature
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD @ Portland

[Barry Warsaw]

On Feb 11, 2010, at 01:59 PM, Robert Collins wrote:

>looms and pipelines have very different goals:
> - looms aim to manage *and version* a collection of
>usually-orthogonal-things with no defined merge order (what upstream
>accept, they accept). And many things will never go upstream (e.g.
>branding issues, workarounds for other platform issues). Specifically
>looms have no intrinsic desire or need for fully-merged pipes, and a
>need to let people collaborate on the structure of the loom.
> - pipelines aim to let individual developers factor out different
>aspects of a feature they are working on, to ease review and provide
>clarity about development; they need fully merged to be the normal
>situation, don't generally need cherrypicking, and have no need for the
>structure of the pipe to be versioned.

This is a very valuable paragraph; it's the first time I can remember reading
such a succinct description of the goals of these two features.  Thanks!

>I'm keen to share more code with pipelines, but they really are
>different things, and I think it's likely harmful to both to push to
>hard on them to become similar.

From the outside, I think that looms and pipelines /feel/ very similar, except
when they don't, which can be jarring.  Maybe the focus that you're bringing
on their different use cases will help users understand when to use one over
the other.  Right now it just seems like they're attacking the same problem.

Great stuff!
-Barry


signature.asc
Description: PGP signature
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD @ Portland

[Robert Collins]

On Thu, 2010-02-11 at 15:42 +1300, Tim Penhey wrote:
> 
> Changing looms to use co-located branches seems like a win to me.
> This would 
> also bring looms and pipelines closer.  I'd love to see a unified
> model here. 

looms and pipelines have very different goals:
 - looms aim to manage *and version* a collection of
usually-orthogonal-things with no defined merge order (what upstream
accept, they accept). And many things will never go upstream (e.g.
branding issues, workarounds for other platform issues). Specifically
looms have no intrinsic desire or need for fully-merged pipes, and a
need to let people collaborate on the structure of the loom.
 - pipelines aim to let individual developers factor out different
aspects of a feature they are working on, to ease review and provide
clarity about development; they need fully merged to be the normal
situation, don't generally need cherrypicking, and have no need for the
structure of the pipe to be versioned.

I'm keen to share more code with pipelines, but they really are
different things, and I think it's likely harmful to both to push to
hard on them to become similar.

-Rob


signature.asc
Description: This is a digitally signed message part
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD @ Portland

[James Westby]

On Thu, 11 Feb 2010 13:33:27 +1100, Martin Pool  wrote:
> I'd like to let looms progress, but not (unless james or others feel
> differently) add them into the dependency chain for getting UDD going.

No, we don't have to add it to the chain to get it going, but I think
it's one ingredient of having a great system.

>  iow people should be able to try them on particular branches, without
> mandating them for all package branches, and (perhaps?) without
> requiring everyone working on that package to use them.

I think a gradual migration path is something to aim for. What we want
is consistency of interaction. I don't want to have to work out what is
going on in the packaging branch before I can start work on it. Allowing
"branch; hack; build; push" regardless of what's going on and allowing
others to delve more deeply is one way, another would be to have "bzr
add-patch" or something that prepared the tree for working, I'm sure
there are more.

Thanks,

James

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD @ Portland

[James Westby]

On Thu, 11 Feb 2010 13:18:30 +1100, Robert Collins 
 wrote:
> James Westby and I had some time together in Portland to talk about UDD
> stuff.

Yes, it was good to have the time, thanks for coming and for sending
this mail.

> Firstly though, a couple of overview points:
>  - the udd project has 200 bugs on it. While many of these are
> 'collision' reports many are not. The collision reports are currently
> overly noisy, so please ignore them for now. However, the other bugs are
> open season for people to fix - and every bug fixed there will
> streamline things for people using bzr to package in Ubuntu.

They are now reported as merge proposals if it's an Ubuntu branch, so
that should stop the list growing too much.

>  - Measurement error: the hottest 100 has a fairly high error rate for
> package imports at the moment, OTOH its looking at precisely the
> packages most likely to fail. James says that overall 96% or so of
> packages import successfully.

They are most likely to fail as they will tend to be larger, uploaded
more often etc: making them more likely to trigger bugs.

> Ok, onto the fun stuff. While long term we want a Launchpad control
> panel for the package importer, James thinks its reasonable that folk
> helping with the operation of it should be able to directly kick it off
> - so he has filed an RT ticket to get more access to that machine.
> James, whats the RT ticket number? Opening this up will let us rerun
> imports more promptly that appear to have had only spurious failures.

#37368

Feel free to drop me an email with the names of packages you think
should be retried in the meantime. I'll do it the next time I read my
email. It's no good for debugging issues, but it's not the best way to
do it even when you can do it straightaway. Remember that you can run
exactly the same code locally and so make use of bzr, pdb, etc. to
investigate.

> Bugs with the importer can and should be debugged on peoples development
> environment - there is an earlier mail from December documenting how to
> do this. We should put that in the Wiki I think think.

Good idea.

> The collisions that are reported as bugs can be divided into three broad
> groups:
>  - impossible (a collision in debian: at least at the moment, we don't
> expect people uploading to Debian packaging-branches. Well, *generally
> speaking* we don't expect this). (Nb: I do it for stuff I maintain in
> Debian :)
>  - spurious (its not a collision, and a bug caused it)
>  - genuine (it is a collision and it should be a merge proposal)
> 
> We have a few collision specific tasks:
>  - James is rapidly making new collisions be filed as merge proposals
> (unless they are in Debian imports)

Done, but with some slight issues due to the LP API and other
things. They are being filed as merge proposals now.

>  - we need to write a script to analyse the nearly 200 collisions in the
> bug tracker to highlight the debian imports (must be bugs, might be
> fixed), and convert the ubuntu ones to merge proposals.
>  - We should delete the stale branches for collisions that we decide are
> bogus. Membership in the magic group ubuntu-branches is needed for that,
> and that group needs to be kept locked down (as it is equivalent to
> upload rights to the archive). So - lets make a list somewhere if you
> determine a branch isn't needed, and ping James or anyone on the tech
> board to delete such a branch).

I'd say comment in the bug report for it. It has all the info I need and
I can do it the next time I read mail.

> We looked at the workflow involved in packaging, and I'm very happy that
> James has seen the light and will be implementing an 'import-upstream'
> command to import and make a tarball micro-branch but not do the debian
> metadata updates. This will be useful for looms, where the two steps
> occur on different threads.

It's currently spelt "bzr dh_make", "import-upstream" would be "bzr
dh_make --bzr-only". When we get a workflow going with looms we can look
at how we it fit in there. I didn't want to have "import-upstream"
straight away as I didn't want confusion arising from the fact that you
can run it in a packaging branch and so delete all the packaging.

> Finally we looked at Looms with mathiaz who is hoping to get the MySQL
> packages in Looms for both Debian and Ubuntu. We identified some rough
> spots and a missing command (import-upstream) but it seems doable, if
> not /nice/ today. After that we talked about a sparser loom merge graph.
> 
> The basic idea I have is that while the stack seems essential to
> providing a simple UI, all the merge commits make a lot of noise. So if
> we only do a merge commit when a conflict has happened, and otherwise
> depend on  'record' telling us what is incorporated, we can save a lot
> of commits and make 'log' or 'bzr viz' clearer, as well as making it
> simpler to cherrypick patches.
> 
> There seem to be several related issues that tie into this:
> 
> * Should the ready-to-build on-di

Re: UDD @ Portland

[Tim Penhey]

On Thu, 11 Feb 2010 15:18:30 Robert Collins wrote:
> James Westby and I had some time together in Portland to talk about UDD
> stuff.

I just have two comments below:

>  - make downloading only some history easier/possible

Shallow branches FTW.  This would be awesome to focus on, and should be able 
to build on stacked branch work.  Make bzr handle ghosts on mainline history, 
and enabling committing to a stacked branch.  This would rock!

> Another open issue is how looms might look in a colocated branch world:
> would they be N branches with a metadata structure glueing them
> together, or still an all-in-one structure? Specifically it would be
> nice for threads to *version and propogate* some key concepts like 'bzr
> pull from lp:myupstream or lp:my-redhat-patch-branch-import'.

Changing looms to use co-located branches seems like a win to me.  This would 
also bring looms and pipelines closer.  I'd love to see a unified model here.

Tim

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD @ Portland

[Robert Collins]

Martin asks what a collision is.

The situation with package imports is that we have a branch B, which
both Ubuntu developers and the package importer can commit to.
Collisions are what happen when the package importer sees something
arrive in the archive which is either not in, or different to, the
branch history.

E.g. 
developer A:
bzr branch lp:ubuntu/foo
# add some changes, call it UNRELEASED
# hmm, I won't dput yet, I want to tweak it some more.
bzr push :parent

developer B:
apt-get source foo
# add different changes, call it N
dput


So we get a Y shape import graph, but the archive is official, so the
package importer does a push --overwrite to 'win' on the packaging
branch, and pushes the old head to a new temporary branch, and files a
ticket in launchpad describing that this happened.

-Rob


signature.asc
Description: This is a digitally signed message part
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: UDD @ Portland

[Martin Pool]

On 11 February 2010 13:18, Robert Collins  wrote:
> James Westby and I had some time together in Portland to talk about UDD
> stuff.
>
> We talked about a few things:
>
> * Looms, their use today and where they should go
> * The operational issues with the package importer and how the bzr team
> can help
> * analysed a few specific bugs and tried to come up with solutions.
>
> Firstly though, a couple of overview points:
>  - the udd project has 200 bugs on it. While many of these are
> 'collision' reports many are not. The collision reports are currently
> overly noisy, so please ignore them for now. However, the other bugs are
> open season for people to fix - and every bug fixed there will
> streamline things for people using bzr to package in Ubuntu.

Can you explain what a collision is?

> Finally we looked at Looms with mathiaz who is hoping to get the MySQL
> packages in Looms for both Debian and Ubuntu. We identified some rough
> spots and a missing command (import-upstream) but it seems doable, if
> not /nice/ today. After that we talked about a sparser loom merge graph.

I'd like to let looms progress, but not (unless james or others feel
differently) add them into the dependency chain for getting UDD going.
 iow people should be able to try them on particular branches, without
mandating them for all package branches, and (perhaps?) without
requiring everyone working on that package to use them.

-- 
Martin 

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

UDD @ Portland

[Robert Collins]

James Westby and I had some time together in Portland to talk about UDD
stuff.

We talked about a few things:

* Looms, their use today and where they should go
* The operational issues with the package importer and how the bzr team
can help
* analysed a few specific bugs and tried to come up with solutions.

Firstly though, a couple of overview points:
 - the udd project has 200 bugs on it. While many of these are
'collision' reports many are not. The collision reports are currently
overly noisy, so please ignore them for now. However, the other bugs are
open season for people to fix - and every bug fixed there will
streamline things for people using bzr to package in Ubuntu.

 - Measurement error: the hottest 100 has a fairly high error rate for
package imports at the moment, OTOH its looking at precisely the
packages most likely to fail. James says that overall 96% or so of
packages import successfully.

 - Adoption. I spoke with pitti and seb128 about how the desktop team
uses bzr. Mostly they use packaging-only branches, which they prefer
precisely because they don't need to download all the history. A quick
test showed them getting 2M of data for gnome-panel, and 14M when using
the package-imported branch. So we need to do some things here for
people that do many drive-by fixes...
 - make downloading only some history easier/possible
 - make it possible to be using a mirror of the VCS data so it doesn't
have to go over the network (perhaps the Ubuntu mirror network could
carry the packaging branches?)
 - make bzr saturate the network more effectively (the 14M above didn't
download at wire speed as far as we could tell).
 I haven't made these into bugs as yet, pending some feedback from you!


Ok, onto the fun stuff. While long term we want a Launchpad control
panel for the package importer, James thinks its reasonable that folk
helping with the operation of it should be able to directly kick it off
- so he has filed an RT ticket to get more access to that machine.
James, whats the RT ticket number? Opening this up will let us rerun
imports more promptly that appear to have had only spurious failures.

Bugs with the importer can and should be debugged on peoples development
environment - there is an earlier mail from December documenting how to
do this. We should put that in the Wiki I think think.

The collisions that are reported as bugs can be divided into three broad
groups:
 - impossible (a collision in debian: at least at the moment, we don't
expect people uploading to Debian packaging-branches. Well, *generally
speaking* we don't expect this). (Nb: I do it for stuff I maintain in
Debian :)
 - spurious (its not a collision, and a bug caused it)
 - genuine (it is a collision and it should be a merge proposal)

We have a few collision specific tasks:
 - James is rapidly making new collisions be filed as merge proposals
(unless they are in Debian imports)
 - we need to write a script to analyse the nearly 200 collisions in the
bug tracker to highlight the debian imports (must be bugs, might be
fixed), and convert the ubuntu ones to merge proposals.
 - We should delete the stale branches for collisions that we decide are
bogus. Membership in the magic group ubuntu-branches is needed for that,
and that group needs to be kept locked down (as it is equivalent to
upload rights to the archive). So - lets make a list somewhere if you
determine a branch isn't needed, and ping James or anyone on the tech
board to delete such a branch).

We looked at the workflow involved in packaging, and I'm very happy that
James has seen the light and will be implementing an 'import-upstream'
command to import and make a tarball micro-branch but not do the debian
metadata updates. This will be useful for looms, where the two steps
occur on different threads.

Finally we looked at Looms with mathiaz who is hoping to get the MySQL
packages in Looms for both Debian and Ubuntu. We identified some rough
spots and a missing command (import-upstream) but it seems doable, if
not /nice/ today. After that we talked about a sparser loom merge graph.

The basic idea I have is that while the stack seems essential to
providing a simple UI, all the merge commits make a lot of noise. So if
we only do a merge commit when a conflict has happened, and otherwise
depend on  'record' telling us what is incorporated, we can save a lot
of commits and make 'log' or 'bzr viz' clearer, as well as making it
simpler to cherrypick patches.

There seem to be several related issues that tie into this:

* Should the ready-to-build on-disk image with patch files be something
stored in the loom, or something the loom models *and exports*.

* Should someone editing a patch see a working tree with all the lower
patches combined, or only their patch (and how does this tie into what
gets committed - hairy logic incoming!)

* How do people migrate into using Looms?

I don't have good answers for all of these, but I'll try and write them
up in more detail once I