Re: Giving developers access to requeue package imports [Was: Ubuntu Platform developers BOF session?]

2013-11-14 Thread Barry Warsaw 
On Nov 13, 2013, at 10:48 PM, Stéphane Graber wrote:

>I push those commits to the main branch (ubuntu:) as I go.
>It's true that the head of the branch doesn't reflect the source
>package, however the latest tag does, so that's not a real concern.
>
>What I like is that when I have a few simple changes or fixes that don't
>seem to warrant an upload on their own either because of a very long
>build time or because I know someone else is going to merge the package
>from Debian in the next few days, I can just commit stuff in there, add
>an UNRELEASED changelog entry and let whoever does the next major change,
>just pick up my changes at the same time.
>
>That also means people don't need to go looking for MP before uploading
>something as the change is already right there in the branch, all they
>need to do is to add theirs and upload.
>
>I've noticed that I'm not the only one doing that, I know that at least Steve
>is doing something similar with some packages ;)

Thanks for the workflow explanation - this is really interesting.

It seems to me like you're using the UDD branches in a very similar way as say
Debian Python uses its svn branches, as the primary location to make
incremental packaging changes in preparation for upload.  I.e. it's your main
location for doing package development.

I think I better understand yours, Steve's, and others objections to my
suggestion.  It's not how I think of the UDD branches but I'll retract my
suggestion because I wouldn't want to prohibit your workflows.

But this *is* getting off-topic from the original request, which is: how can
we empower developers to fix their own broken UDD branches?

Let's leave the future of UDD, and the reality of the given resources to
improve the situation and get closer to our dream world, for another day.

-Barry


signature.asc
Description: PGP signature
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Giving developers access to requeue package imports [Was: Ubuntu Platform developers BOF session?]

2013-11-13 Thread Barry Warsaw 
On Nov 13, 2013, at 04:51 PM, Stéphane Graber wrote:

>Well, to be fair my fallback process when not doing UDD is:
> - pull-lp-source  
> - cd */
> - bzr init && bzr add && bzr commit -m "current"
> - Do whatever changes, commit when needed, revert, ...
> - bzr bd -S
> - dput

For my UDD branches, I always branch them into a shared repository, under the
assumption that if I've grabbed the package once, the next time I'll have most
of the revisions already downloaded, so the branching is quicker.

I guess pull-lp-source essentially fills the roll of `chdist apt-get series
blah` although the former has a bit more flexibility in how you specify the
exact branch you want.

>Which based on what you described about commitless UDD seems pretty much
>identical with the significant improvement that I don't have to grab the
>whole branch on top of that :)

Could be, although I have to type less :).

>I could also push and share that temporary branch with others and
>there'd be no downside to this since I wouldn't be able to merge that
>branch back into the UDD one anyway.

For out-of-date branches, I think they'd be roughly equivalent.  For
up-to-date branches, not getting the UDD branch means you wouldn't be able to
merge propose, which also makes sponsoring more difficult (patches/debdiffs
are like dead things, branches are alive! :).

>At least for me, UDD without commit rights, would mean lost granularity
>in some changes I'm doing in the archive, for example for some of the
>edubuntu packages I've had dozens of commits before an actual upload,
>and I quite enjoy having that present in the UDD history, loosing that
>ability would be loosing much of UDD's benefits.

I'm a little unclear if you mean that you do pushes to the parent UDD branch
for each of those commits.  Doesn't that mean that the UDD branch won't mirror
the contents of the source package?  If so, is that a good thing?  Or maybe
you only push once the package is dput'd.

I am concerned about workflows where the UDD branch is not a reflection of the
contents of the source package, modulo short importer lag.

It's too bad there's no way to capture those non-pushed intermediate commits
in the source package you upload, such that the importer would apply them, and
they would be preserved in the master UDD branch.

-Barry


signature.asc
Description: PGP signature
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Giving developers access to requeue package imports [Was: Ubuntu Platform developers BOF session?]

2013-11-13 Thread Steve Langasek 
On Wed, Nov 13, 2013 at 03:32:38PM -0500, Barry Warsaw wrote:
> On Nov 13, 2013, at 11:32 AM, Steve Langasek wrote:

> >But I think it would be more interesting to get a permanent fix for this
> >bug:

> >  https://bugs.launchpad.net/udd/+bug/714622

> >This accounts for the problem people have mentioned, that core packages are
> >much more likely to have failed imports.  The importer badly needs fixed to
> >not throw up its hands when the revision ID of a tag has changed; it should
> >only care about this when the branch contents are wrong.

> >This single bug accounts for just under half of all importer failures, and
> >is a failure scenario that the importer *could*, with sufficient smarts,
> >resolve automatically.

> This may be controversial, but (except for trying to fix error
> conditions), I think we should disallow all developer pushes to UDD
> branches and only let the importer write to them.  It's simply too error
> prone otherwise, and there's no good reason for it.

I disagree 100%.  To me, this is the only reason to *have* UDD branches.  If
we're never going to push to the branches, then the whole thing is a futile
exercise.  Being able to use bzr blame / bzr diff to traverse history at
per-upload granularity is not sufficiently interesting to justify the effort
of supporting UDD.

As long as there are other VCS branches for core packages that are richer
and more authoritative with finer-grained history, UDD branches will
continue to be confusingly redundant.  If we've resigned ourselves to the
notion that they will never be a suitable replacement for maintainer
branches, then we should kill UDD off entirely instead of continuing to
invest time and hardware resources in giving developers a poor VCS
experience that requires you to play a game of 20 questions to figure out
the "right" place to get the package's source.

> One possible reason for developers to push to UDD branches is to share the
> code with other people, or to avoid the lag in importer runs.

The reason for pushing to the UDD branches is so that changes to the package
are grouped logically, and you can use the branch history as a useful
forensic tool, or as a source for cherry-picking changes.  Having UDD
branches that exist but are *not* an authoritative source for this
information is actively harmful.  At the time UDD was conceived.  this was
considered an acceptable temporary downside on the path to a full
branch-based workflow.  Now that it's clear that having imports of full
upstream history into UDD is not on the horizon, we should work out what to
do to avoid continuing to provide UDD as a bad service.

> Of course the former can be easily handled by pushing to a personal
> branch.  The latter?  Oh well, I can live with that for error-free
> branches.  ;)

This error is not with the branches, but with the importer for imposing
unnecessary constraints on the branches.  In nearly all of the cases where
that bug hits us, the branch *contents* are identical to what the importer
would have generated, which means the importer should accept the ID change
and move on.  In cases where the contents don't match, the importer should
win and move the branch aside - in fact, it's supposed to already do this.
But if we no longer have anyone who can fix this importer problem, then we
should admit defeat and scrap UDD altogether.

> A long time ago I decided never to push UDD branches and always let the
> importer update them.  I've never regretted that or encountered problems
> with that discipline.

That works ok for packages that are only touched once in a blue moon.  But
making this a policy means that UDD is no longer useful for core packages
like upstart, and all our rich branch history is going to go somewhere else.

This doesn't solve the problem that people are actually complaining about,
namely that UDD branches are not useful for packages that are actively
maintained - it merely codifies it, and exacerbates the related problem that
UDD branches only sometimes provide a useful workflow for outside
contributors.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Giving developers access to requeue package imports [Was: Ubuntu Platform developers BOF session?]

2013-11-13 Thread Barry Warsaw 
On Nov 13, 2013, at 04:28 PM, Stéphane Graber wrote:

>For UDD, if we can't commit to the branch, then there's zero benefit in
>even using it as the source branch as I could just as well use apt-get
>source, which will get me the current package from the archive (which
>UDD doesn't necessarily give me...), then apply changes and push that.

For simple package changes, you could have a point, but I rarely encounter
simple package changes specifically in Ubuntu.  Usually I'm merging a new
upstream, or Debian version, and then local version control is often a
godsend.  Sometimes the merge goes badly, or you have conflicts, or it's not
enough to get a working Ubuntu package.  I can do the merge, commit that local
change, and then do further commits locally as I refine the package to build
and work properly.  I can diff between revisions, revert changes, etc.
E.g. all the benefits of version control.  I can create side branches of my
main branch to try things out, then merge them back into my main branch.  All
this is especially useful if you are working on a package over some span of
time.

apt-get source is like performing without a net.  Let's say you head down the
wrong path while updating a package.  It's very difficult to backup and try
again, take side travels for experimentation, etc.  Oh, and chdist is nice,
but I prefer having ubuntu:{,-proposed}/ branches.

>Not having commit rights to the UDD branch would make UDD a simple
>archiving service and based on its current state, a pretty bad one at
>that.
>
>To clarifiy my position, I really like UDD and I think that kind of VCS
>integration is what we want for the distro, but it's never been working
>reliably enough to gain sufficient momentum.
>
>In a perfect world, I'd have a VCS repository containing branches for
>all Ubuntu series and pockets, for the various Debian releases and for
>the upstream code, making any rebase/cherry-pick trivial and having LP
>trigger builds on either a special signed commit or a signed tag.
>
>Also in that perfect world, the inner workings of our upload process
>would be tied to that, so that it wouldn't be possible for the branch to
>be out of sync with the archive. This could be achieved by either making
>this the only way to upload or making the "legacy" upload path, commit
>to the branch and export from there prior to processing.

I'll agree with you there.  I'd love to live in this world. :)

-Barry


signature.asc
Description: PGP signature
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Giving developers access to requeue package imports [Was: Ubuntu Platform developers BOF session?]

2013-11-13 Thread Barry Warsaw 
On Nov 13, 2013, at 09:54 PM, Sebastien Bacher wrote:

>That's a good question ... what's the goal of UDD? Having an history of the
>changes that is easy to query? Make possible to stage changes without
>uploading? Have an easy way to integrate into launchpad for reviews (by using
>the merge requests)?

Yes, merge proposals, sponsoring, collaborative branches, etc.  Also, local
version control while you're developing changes, being able to better manage
upstream or Debian merges, etc.

In my previous 'bot analogy, there's one difference: your local commits won't
show up as a side line-of-development once the importer lands your changes.
That'll make the git fast-forward fans happy though. :)

-Barry

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Giving developers access to requeue package imports [Was: Ubuntu Platform developers BOF session?]

2013-11-13 Thread Barry Warsaw 
On Nov 13, 2013, at 03:43 PM, Stéphane Graber wrote:

>Hmm, so if we can't planned changes to UDD branches and have to use a
>separate user-owned branch for that, then what's the use of the UDD
>branch?
>
>It sounds to me like it'd then be much easier for me to just maintain my
>own branch on the side and upload from there, ignoring UDD entirely,
>which surely isn't what we want there.

We're all familiar with workflows where landings to the master branch are
guarded by robots like Jenkins or Tarmac.  I think of this exactly the same
way, with the 'bot being the importer.

-Barry


signature.asc
Description: PGP signature
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Giving developers access to requeue package imports [Was: Ubuntu Platform developers BOF session?]

2013-11-13 Thread Barry Warsaw 
On Nov 13, 2013, at 11:32 AM, Steve Langasek wrote:

>But I think it would be more interesting to get a permanent fix for this
>bug:
>
>  https://bugs.launchpad.net/udd/+bug/714622
>
>This accounts for the problem people have mentioned, that core packages are
>much more likely to have failed imports.  The importer badly needs fixed to
>not throw up its hands when the revision ID of a tag has changed; it should
>only care about this when the branch contents are wrong.
>
>This single bug accounts for just under half of all importer failures, and
>is a failure scenario that the importer *could*, with sufficient smarts,
>resolve automatically.

This may be controversial, but (except for trying to fix error conditions), I
think we should disallow all developer pushes to UDD branches and only let the
importer write to them.  It's simply too error prone otherwise, and there's no
good reason for it.

One possible reason for developers to push to UDD branches is to share the
code with other people, or to avoid the lag in importer runs.  Of course the
former can be easily handled by pushing to a personal branch.  The latter?  Oh
well, I can live with that for error-free branches. ;)

A long time ago I decided never to push UDD branches and always let the
importer update them.  I've never regretted that or encountered problems with
that discipline.

Cheers,
-Barry


signature.asc
Description: PGP signature
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Giving developers access to requeue package imports [Was: Ubuntu Platform developers BOF session?]

2013-11-13 Thread Steve Langasek 
On Wed, Nov 13, 2013 at 07:17:56PM +, Dmitrijs Ledkovs wrote:
> On 13 November 2013 17:39, Andrew Starr-Bochicchio  
> wrote:
> > On Fri, Nov 8, 2013 at 1:33 PM, Andrew Starr-Bochicchio
> >  wrote:
> >> Though since we're talking about it, the one stop gap fix that would
> >> make me happy would be if all Ubuntu Developers could trigger the
> >> equivalent of the local 'requeue_package.py --full' command that UDD
> >> admins can run. Some history might get lost, but at least out of date
> >> branches could be made usable.

> > This seems to have been the topic that has generated the most
> > interest. It seems to be a bit of an overkill to have a vUDS session
> > on it, especially if we don't have the right people in the room. So
> > maybe we can try to hammer out the requirements here?

> > Currently you need shell access to Jubany in order to run the command.
> > [0] I know that this request has come up in the past, but my Google-fu
> > is failing me now. Adding the (seemingly dormant) UDD list to the
> > conversation in hopes of catching the right person.

> > [0] https://bugs.launchpad.net/udd/+bug/713719

> I requeue packages from time to time, upon request. But I don't keep a
> track of packages for which full requeue doesn't help.
> Nor have a good way to process such requests.

> Maybe a request wiki page? I'd subscribe to it, and would comment
> which one requeued and whether that fixes / not fixes the branch.

I would suggest using https://bugs.launchpad.net/udd/ itself for tracking
this.

But I think it would be more interesting to get a permanent fix for this
bug:

  https://bugs.launchpad.net/udd/+bug/714622

This accounts for the problem people have mentioned, that core packages are
much more likely to have failed imports.  The importer badly needs fixed to
not throw up its hands when the revision ID of a tag has changed; it should
only care about this when the branch contents are wrong.

This single bug accounts for just under half of all importer failures, and
is a failure scenario that the importer *could*, with sufficient smarts,
resolve automatically.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature
-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Re: Giving developers access to requeue package imports [Was: Ubuntu Platform developers BOF session?]

2013-11-13 Thread Dmitrijs Ledkovs 
On 13 November 2013 17:39, Andrew Starr-Bochicchio  wrote:
> On Fri, Nov 8, 2013 at 1:33 PM, Andrew Starr-Bochicchio
>  wrote:
>> Though since we're talking about it, the one stop gap fix that would
>> make me happy would be if all Ubuntu Developers could trigger the
>> equivalent of the local 'requeue_package.py --full' command that UDD
>> admins can run. Some history might get lost, but at least out of date
>> branches could be made usable.
>
> This seems to have been the topic that has generated the most
> interest. It seems to be a bit of an overkill to have a vUDS session
> on it, especially if we don't have the right people in the room. So
> maybe we can try to hammer out the requirements here?
>
> Currently you need shell access to Jubany in order to run the command.
> [0] I know that this request has come up in the past, but my Google-fu
> is failing me now. Adding the (seemingly dormant) UDD list to the
> conversation in hopes of catching the right person.
>
> [0] https://bugs.launchpad.net/udd/+bug/713719


I requeue packages from time to time, upon request. But I don't keep a
track of packages for which full requeue doesn't help.
Nor have a good way to process such requests.

Maybe a request wiki page? I'd subscribe to it, and would comment
which one requeued and whether that fixes / not fixes the branch.

Regards,

Dmitrijs.

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel

Giving developers access to requeue package imports [Was: Ubuntu Platform developers BOF session?]

2013-11-13 Thread Andrew Starr-Bochicchio 
On Fri, Nov 8, 2013 at 1:33 PM, Andrew Starr-Bochicchio
 wrote:
> Though since we're talking about it, the one stop gap fix that would
> make me happy would be if all Ubuntu Developers could trigger the
> equivalent of the local 'requeue_package.py --full' command that UDD
> admins can run. Some history might get lost, but at least out of date
> branches could be made usable.

This seems to have been the topic that has generated the most
interest. It seems to be a bit of an overkill to have a vUDS session
on it, especially if we don't have the right people in the room. So
maybe we can try to hammer out the requirements here?

Currently you need shell access to Jubany in order to run the command.
[0] I know that this request has come up in the past, but my Google-fu
is failing me now. Adding the (seemingly dormant) UDD list to the
conversation in hopes of catching the right person.

[0] https://bugs.launchpad.net/udd/+bug/713719

-- Andrew Starr-Bochicchio

   Ubuntu Developer 
   Debian Developer 
   PGP/GPG Key ID: D53FDCB1

-- 
ubuntu-distributed-devel mailing list
ubuntu-distributed-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel