Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Johannes Schauer]

Quoting James Clarke (2017-05-22 16:25:38)
> But I notice that for the sbuild path, schroot is completely missing,

Maybe I should also point out that schroot is just the *default* sbuild chroot
backend.  It also supports the "sudo" mode (which essentially just uses "sudo
chroot") and the autopkgtest mode which allows sbuild to build packages in any
environment that autopkgtest provides like lxc, lxd, qemu, ssh...

And then there is also a patch for adding a backend using Linux user namespaces
which is a lightweight replacement of lxc-usernsexec and unshare that allows to
easily do unprivileged package builds without the need of an suid binary like
schroot:

https://anonscm.debian.org/cgit/buildd-tools/sbuild.git/commit/?h=user/josch/uchroot=8c40e83a63195983dadad7616b7b916e084cf7e4

And then there is also the effort to add this as a backend to autopkgtest:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844459

Work on all of this stalled a bit because of lack of time. I don't want to drag
this thread further into this direction, so if you are interested in any of
this, just get in touch with me off-list.

Thanks!

cheers, josch


signature.asc
Description: signature

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Ben Finney]

Ian Jackson  writes:

> Use [dgit] to publish your git history, by doing your uploads with
> dgit push.
>
> The root goal is this: Debian should publish the source for all our
> packages, as git branches, in a format that is directly useable by
> ordinary people.

Thanks. That is what I think anyone reading the package description
needs to know; it's context that simply isn't there.

> I would be very happy to hear suggestions for improving the
> descriptions.  Ideally without making them longer.

Making it longer is unavoidable, I think, since the issue as I see it is
the context is missing. So I hope you'll be convinced to add that
context to the package description.

Adding a paragraph like the above one about “the root goal” would be an
excellent start.

-- 
 \ “Why doesn't Python warn that it's not 100% perfect? Are people |
  `\ just supposed to “know” this, magically?” —Mitya Sirenef, |
_o__) comp.lang.python, 2012-12-27 |
Ben Finney

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Philipp Kern]

On 22.05.2017 16:25, James Clarke wrote:
> You say that, but this is incredibly biased. Even he admits that in the
> colour choice. Disclaimer: as the cowbuilder maintainer (which comes
> from the cow*dancer* source package, for historical reasons, despite
> what the diagram may tell you) I am of course also biased. But I notice
> that for the sbuild path, schroot is completely missing, which is
> implemented in C++ and therefore probably deserves a bright red colour
> just like cowbuilder in his second picture (or maybe even something
> worse, depending on your views of C++). Then there's the extremely
> unhelfpul, hostile comment at the end:

schroot is in theory just a tool in the toolbox, even if it came from
the same developer. The same is in theory true for cowdancer, but it
inherently requires cooperation from the chroot as well. At the same
time it won't properly work for programs that do not use the C library's
file interface (e.g. if they use syscalls directly like Go).

So cowdancer is in a way a very cute hack of which you might need to
know the semantics whereas schroot aimed to be a fancy chroot done
right. The aim of one tool was to speed-up the build (hence cowdancer
and cowbuilder being built from the same source) and of the other other
to get a chroot tool that can be used by unpriviledged users, in a way
that allows reuse of chroot templates.

I somewhat doubt that you end up hacking on a C++ setuid binary as part
of Debian packaging work. And the same is likely true for cowdancer
itself - although it's theoretically more brittle - with the twist that
for some reason pbuilder does not have native support for it.

Kind regards
Philipp Kern



signature.asc
Description: OpenPGP digital signature

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Ian Jackson]

Ben Finney writes ("Re: infinite number of Debian workflows (Re: Moving away 
from (unsupportable) FusionForge on Alioth?)"):
> Ian Jackson <ijack...@chiark.greenend.org.uk> writes:
> > I want every maintainer who is using git to be able to use dgit.
> 
> Use it to do what, though? The package description is currently:

Use it to publish your git history, by doing your uploads with
dgit push.

The root goal is this: Debian should publish the source for all our
packages, as git branches, in a format that is directly useable by
ordinary people.

A user should not have to be an expert on Debian source formats to
clone the source to a program on your computer, git-cherry-pick a
patch from upstream (or develop their own patch to the upstream code),
and rebuild and install it.  Debian should provide a sensible git
branch, for every package, that a user can use this way.

> git interoperability with the Debian archive
>  dgit (with the associated infrastructure) makes it possible to
>  treat the Debian archive as a git repository.
>  .
>  dgit push constructs uploads from git commits
>  .
>  dgit clone and dgit fetch construct git commits from uploads.
> 
> That sounds to me like it isn't a tool for maintainers, but rather a
> tool for “interoperability with the Debian archive” which AFAICT is
> already provided by the tools I am using.

You have only one-way interoperability, and only from a nonstandard
git layout.

> If the package does something that should be of interest to package
> maintainers in general, I'd expect the description to be a lot clearer
> what that is and why it's of interest.

I would be very happy to hear suggestions for improving the
descriptions.  Ideally without making them longer.


It's awkward, because dgit is different things to different people.

1. For a user or a downstream, dgit is a way to get Debian source code
   as directly useable git branches in a standard format.

2. For a maintainer, dgit is a way to publish and share your actual
   git branch with users, downstreams and other contributors.  It
   can also be a way to handle an NMU without needing to ever deal
   with a source package.

3. For sponsors/sponsees, etc., it is a way to replace the
   soruce-package-based sponsorship workflow with one based on git
   branches.  The sponsee publishes their git branch, and the sponsor
   (when satisfied) invokes dgit to upload it.

4. In the abstract, much of the functionality is a bidirectional
   converter between git trees and source packages [4]; but it's also
   a client (read and write) for a standardised git publication
   infrastructure.


We have other tools that do these things.  So let me address the
competition:

1. Obtaining Debian package source code as a git branch.  The
   principal competition here is the Vcs-Git dsc header.  One big
   problem with Vcs-Git is that the git tree you find there is largely
   unstandardised:
  - The git tree might be patches-unapplied.  Usually it is.  Such
trees are useable only by people who are already Debian
experts, and have many weird properties.
  - The git tree might not have the upstream code at all.
  - There might be other arbitrary weirdnesses; perhaps
some script needs to be run to make it buildable, for
example.

   (It can also sometimes be a problem to find the git commit
   corresponding to a particular Debian package.  Luckily we have
   DEP-14 tags which are largely used now; even git-dpm, whose author
   rejects the DEP-14 tag encoding, creates similar enough tags out of
   most versions.)

   There are other potential problems: the git branch you find might
   not be fast-forwarding; or there might not be a suitable DEP-14
   tag; or the last upload might have been done `out of maintainer
   workflow' for some reason and not be reflected in the git
   repository.  This means that the Vcs-Git repository might lack
   important security patches.  The repository might have moved.

   And of course it may not exist at all.  A user wants a single tool,
   not to be told to flail about trying various options.   

2. For working within Debian maintainer teams, our existing git
   workflows mostly work.  However, they have the problem that they do
   not, generally, produce externally-useable git branches.  See (1).

   Some workflows (eg git-dpm) do produce extenally-useable git
   branches; but before dgit there was no way to publish your git
   branch along with a promise that it was directly useable.  And
   there was no fallback for packages that didn't have a useable git
   branch.

   In a sense I am hoping that maintainers, who do not suffer from the
   lack of a solution to (1), are willing to do a bit of work to make
   everyone else's lives easier.

   There are some advantages for the maintainer of course: if a
   maintainer and an NMUer both use dgit, then the

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Holger Levsen]

On Tue, May 23, 2017 at 06:34:09PM +0100, Sean Whitton wrote:
> On Tue, May 23, 2017 at 06:38:29PM +0200, Emilio Pozuelo Monfort wrote:
> > I have to deal with packages in svn, git-bp and plain git, and have started 
> > to
> > write a set of (ugly) scripts that perform common actions in each of those
> > formats, and a generic wrapper that calls the right one depending on which 
> > repo
> > I'm on, so that I can just do e.g. 'deb update; dch; deb commit; deb build; 
> > deb
> > release; deb tag' and not having to remember all the different command line
> > switches and options and so on (I have a bad memory and am lazy). This 
> > shouldn't
> > be necessary in the first place; at least a bit of homogenization would make
> > sense imho.
> This is pretty much exactly the point of dgit.

I actually thought that Emilio reinvented mr (but then realized it's mr plus
yet another release-wrapper ;-)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Ben Finney]

Ian Jackson  writes:

> I want every maintainer who is using git to be able to use dgit.

Use it to do what, though? The package description is currently:

git interoperability with the Debian archive
 dgit (with the associated infrastructure) makes it possible to
 treat the Debian archive as a git repository.
 .
 dgit push constructs uploads from git commits
 .
 dgit clone and dgit fetch construct git commits from uploads.

That sounds to me like it isn't a tool for maintainers, but rather a
tool for “interoperability with the Debian archive” which AFAICT is
already provided by the tools I am using.

If the package does something that should be of interest to package
maintainers in general, I'd expect the description to be a lot clearer
what that is and why it's of interest.


My apologies for publicly pointing to a package description for
criticism, but it seems relevant to the claim that the package is for
“every maintainer who uses git” that the description should explain why
that is.

-- 
 \ “Pinky, are you pondering what I'm pondering?” “I think so, but |
  `\  where will we find an open tattoo parlor at this time of |
_o__)   night?” —_Pinky and The Brain_ |
Ben Finney

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Sean Whitton]

On Tue, May 23, 2017 at 06:38:29PM +0200, Emilio Pozuelo Monfort wrote:
> I have to deal with packages in svn, git-bp and plain git, and have started to
> write a set of (ugly) scripts that perform common actions in each of those
> formats, and a generic wrapper that calls the right one depending on which 
> repo
> I'm on, so that I can just do e.g. 'deb update; dch; deb commit; deb build; 
> deb
> release; deb tag' and not having to remember all the different command line
> switches and options and so on (I have a bad memory and am lazy). This 
> shouldn't
> be necessary in the first place; at least a bit of homogenization would make
> sense imho.

This is pretty much exactly the point of dgit.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Sean Whitton]

On Tue, May 23, 2017 at 10:21:27AM +0100, Jonathan Dowland wrote:
> On Mon, May 22, 2017 at 10:07:20PM +0100, James Clarke wrote:
> > There already effectively is a semi-"primary" implementation given that
> > sbuild is used on the buildds.
> 
> Yes that is a very strong fact in favour of sbuild.

If by "in favour" you mean in favour of using sbuild instead of pbuilder
on your own machine, I think James' point was that the fact that sbuild
is run on the buildds is a reason *not* to run it locally.

Your package will be built using sbuild on the buildds anyway, so
testing with pbuilder locally could expose bugs.

(disclaimer: I use sbuild)

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Russ Allbery]

Jonathan Dowland  writes:

> Fair enough, cowbuilder was one of the ones in my hazy peripheral vision
> as "another", along with some tools to use things like docker that I am
> aware of but couldn't remember the names. None of them have the same
> traction as pbuilder or sbuild. I've only used pbuilder myself
> personally.

If you're using git-buildpackage, using cowbuilder is indistinguishable
from using pbuilder (in fact, someone may well be using cowbuilder and not
know it if they used the default recommendations in git-buildpackage).
cowbuilder is nearly identical to pbuilder; it just constructs the work
directory for the build differently, in a way that at least used to be way
faster.

-- 
Russ Allbery (r...@debian.org)   

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Ian Jackson]

Emilio Pozuelo Monfort writes ("Re: infinite number of Debian workflows (Re: 
Moving away from (unsupportable) FusionForge on Alioth?)"):
> Besides, the sbuild/pbuilder duplicity is the least of your problems
> in terms of multiple workflows, because once you choose one of those
> and set it up, you can build all packages, and don't have to set the
> other one up or learn it.

Right!

> Compare that to hacking on a package which may use
>  - debhelper, dh, cdbs or no helper at all (!) for debian/rules
>  - quilt, dpatch, simple-patchsys, single-debian-patch for patch management
>  - format 1.0, 3.0, native, non-native, multiple tarballs... for the source
>  - git-buildpackage, git-dpm, other git repo structure with no helper tools,
> svn, bzr, etc for the repository
>  - ...

Of these, debhelper/dh/cdbs/raw is difficult to deal with with better
tooling because it's inherint in the source code.  However:
  * dh $@ is a more automatic way of doing dehelper
  * cdbs is on the way out
  * no helper at all is definitely on the way out
So at least the last two are a form of technical debt.

> Some of those are easy to learn, and some you don't have to deal
> with if you are doing an NMU (e.g. the repository). It's still a
> pretty complicated picture and a steep learning curve if you want to
> start contributing, or want to join a team that has a completely
> different setup.

Yes.

> I have to deal with packages in svn, git-bp and plain git, and have
> started to write a set of (ugly) scripts that perform common actions
> in each of those formats, and a generic wrapper that calls the right
> one depending on which repo I'm on, so that I can just do e.g. 'deb
> update; dch; deb commit; deb build; deb release; deb tag' and not
> having to remember all the different command line switches and
> options and so on (I have a bad memory and am lazy). This shouldn't
> be necessary in the first place; at least a bit of homogenization
> would make sense imho.

Part of the problem is that many of the workflows have been pretty
poor: at least, they all have flaws that some people regard as
important.

Ian.

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Emilio Pozuelo Monfort]

On 22/05/17 16:25, James Clarke wrote:
> On Mon, May 22, 2017 at 03:06:48PM +0100, Jonathan Dowland wrote:
>> On Mon, May 22, 2017 at 12:47:51PM +0100, Sean Whitton wrote:
>>> Someone else already had this idea:
>>>
>>> https://people.debian.org/~stapelberg//2016/11/25/build-tools.html
>>
>> Excellent, this is a great start, and seeing "Michael Stapelberg" for me is 
>> an
>> indication of quality.
> 
> You say that, but this is incredibly biased. Even he admits that in the
> colour choice. Disclaimer: as the cowbuilder maintainer (which comes
> from the cow*dancer* source package, for historical reasons, despite
> what the diagram may tell you) I am of course also biased. But I notice
> that for the sbuild path, schroot is completely missing, which is
> implemented in C++ and therefore probably deserves a bright red colour
> just like cowbuilder in his second picture (or maybe even something
> worse, depending on your views of C++). Then there's the extremely
> unhelfpul, hostile comment at the end:
> 
>> I propose to eliminate complexity in Debian by deprecating the
>> pbuilder toolchain in favor of sbuild.
> 
> Now, I am not necessarily against simplifying workflows, but choosing
> one to rule the all arbitrarily because you prefer the language it's
> written in is not the way to go about it. There are also benefits to
> having a variety of implementations; they behave slightly differently,
> sometimes not implementing policy in the same way, and this can be a
> good thing, allowing policy to be clarified, or finding mistakes in the
> other builder, or just exposing flaws in your package. For example,
> pbuilder will by default use a new isolated network namespace for the
> build, whereas sbuild won't, so if you just use sbuild you may not be
> aware of violating policy (4.9). I'm not trying to sell pbuilder over
> sbuild here though; there are also many ways in which sbuild is better
> than pbuilder.

Besides, the sbuild/pbuilder duplicity is the least of your problems in terms of
multiple workflows, because once you choose one of those and set it up, you can
build all packages, and don't have to set the other one up or learn it.

Compare that to hacking on a package which may use
 - debhelper, dh, cdbs or no helper at all (!) for debian/rules
 - quilt, dpatch, simple-patchsys, single-debian-patch for patch management
 - format 1.0, 3.0, native, non-native, multiple tarballs... for the source
 - git-buildpackage, git-dpm, other git repo structure with no helper tools,
svn, bzr, etc for the repository
 - ...

Some of those are easy to learn, and some you don't have to deal with if you are
doing an NMU (e.g. the repository). It's still a pretty complicated picture and
a steep learning curve if you want to start contributing, or want to join a team
that has a completely different setup.

I have to deal with packages in svn, git-bp and plain git, and have started to
write a set of (ugly) scripts that perform common actions in each of those
formats, and a generic wrapper that calls the right one depending on which repo
I'm on, so that I can just do e.g. 'deb update; dch; deb commit; deb build; deb
release; deb tag' and not having to remember all the different command line
switches and options and so on (I have a bad memory and am lazy). This shouldn't
be necessary in the first place; at least a bit of homogenization would make
sense imho.

Cheers,
Emilio

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Ian Jackson]

Sean Whitton writes ("Re: infinite number of Debian workflows (Re: Moving away 
from (unsupportable) FusionForge on Alioth?)"):
> A way to set the version during the build, as you suggest, would be
> sufficient to cover this.  It is hard to see how we could relieve the
> user of the need to understand how to choose a version number for a .deb
> for testing.  An option to set the version in the build command line
> would remove the need for Debian source package knowledge.

It would be best if the user would just pass an option to say `pls
pick version number' and some tool would make a reasonable stab at it.

> >  * Pull request workflow for submitting changes.  This should
> >eventually turn into a bug submission to the Debian BTS.
> >This sounds to me like it probabl needs to be a web service, but
> >perhaps some local client utility that looked enough like a web
> >page would do.
> 
> We basically already have all the pieces:
> 
> - git-request-pull(1)
> - reportbug(1)
> - git hosting on alioth / our shiny pagure git hosting (coming soon)
> 
> dgit could ship a script that ties these together.  (The reason I suggest
> using our own git hosting is so that the branch doesn't disappear -- one
> advantage of patches in the BTS is that they can't 404.)

I'm not sure that a command-line tool is what our target audience for
this would be looking for.  But contributions certainly welcome.

Ian.

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Ian Jackson]

Sean Whitton writes ("Re: infinite number of Debian workflows (Re: Moving away 
from (unsupportable) FusionForge on Alioth?)"):
> On Mon, May 22, 2017 at 09:22:00PM +0100, Sean Whitton wrote:
> > On Mon, May 22, 2017 at 01:42:54PM -0400, Jeremy Bicha wrote:
> > > Of course, dgit is yet another workflow and my understanding is that
> > > git-buildpackage (without dgit) is far more commonly used in Debian.

Jeremy: you're right that dgit is not yet used for the majority of
uploads.  I think this is a problem because I think we should all be
sharing our git trees in a way that is directly useable by
non-Debian-experts.

But I think it is not true that dgit `is yet another workflow'.  Of
course it depends a bit on what you mean by `workflow', but in this
context I think it means things like what your git branches look like,
where you keep them, how you deal with new upstream versions; and it
also means what build tools you use (eg cowbuilder vs sbuild).


>From a maintainer's point of view, dgit tries not to impose
workflow requirements.  You can use dgit:

   - with patches-applied branches,
 or with patches-unapplied branches

   - with git-buildpackage; with git-dpm; with raw git

   - with sbuild or with pbuilder (although I am working on
  smoothing out some wrinkles with the use of pbuilder)

   - if you generate your orig tarballs from git, or if you
  download them from upstream, or if you repack them
  yourself using eg dpkg-repack

etc. etc.

I want every maintainer who is using git to be able to use dgit.
We're not quite there yet, but the exceptional cases are much rarer
now.  (The main obvious one is packaging-only git trees.)

So I think the `yet another standard' jibe is unfair.  dgit is trying
to *work with* maintainers' existing packaging tools, not supplant
them.


> Ian did claim that dgit gives you one workflow for all Debian packages.
> It does indeed provide that.  What I should have said was that it does
> not enforce /that/ workflow; it just makes it available, primarily for
> the benefit of NMUers and those who aren't already Debian contributors.

Precisely.  From a _user_'s point of view (or an NMUer, or security
responder) dgit offers the same workflow for all packages.

This is possible because dgit is mostly a converter between different
source package representations, along with knowledge of where to find
these representations and when to do what kind of conversion.


Ian.

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Jonathan Dowland]

On Mon, May 22, 2017 at 10:07:20PM +0100, James Clarke wrote:
> There already effectively is a semi-"primary" implementation given that
> sbuild is used on the buildds.

Yes that is a very strong fact in favour of sbuild.

> And as for making these "secondary" implementations not geared for real
> users, for whom would they then be?

I was thinking of things like 'dash' which are pedantically POSIX compliant, 
serve to find
bugs in other scripts/shells but are not themselves recommended for end-user 
use (at least
interactively)

> There are lots of areas where Debian has far too many tools to
> accomplish the same thing, but I don't think this is one of them; there
> are only two main tools for building in chroots (sbuild and
> pbuilder[0]), both of which have significant user bases.
> [0] cowbuilder is a thin wrapper that behaves (almost) identically, so
> it doesn't really count as something different

Fair enough, cowbuilder was one of the ones in my hazy peripheral vision as
"another", along with some tools to use things like docker that I am aware of
but couldn't remember the names. None of them have the same traction as
pbuilder or sbuild. I've only used pbuilder myself personally.

> Anyway, I'm done with this debate; it's clear I have very different
> views from some on this matter.

The points you have made are a valuable contribution IMHO, thanks for making 
them.


-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄ Please do not CC me, I am subscribed to the list.

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Sean Whitton]

On Mon, May 22, 2017 at 09:22:00PM +0100, Sean Whitton wrote:
> On Mon, May 22, 2017 at 01:42:54PM -0400, Jeremy Bicha wrote:
> > Of course, dgit is yet another workflow and my understanding is that
> > git-buildpackage (without dgit) is far more commonly used in Debian.
> 
> This isn't fair to dgit.  While it does impose some minimal requirements
> upon git trees, it is otherwise workflow neutral.  This is evidenced by
> the fact that there are already several distinct dgit-compatible
> workflows documented in the dgit-maint-*(7) namespace.

I re-read Ian's e-mail and realised I wasn't quite fair to you, either :)

Ian did claim that dgit gives you one workflow for all Debian packages.
It does indeed provide that.  What I should have said was that it does
not enforce /that/ workflow; it just makes it available, primarily for
the benefit of NMUers and those who aren't already Debian contributors.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Marc Haber]

On Mon, 22 May 2017 22:20:29 +0200, Andreas Tille 
wrote:
>On Mon, May 22, 2017 at 10:46:59PM +0500, Andrey Rahmatullin wrote:
>> On Mon, May 22, 2017 at 09:07:52AM +, Holger Levsen wrote:
>> > there's just a hundred Debian workflows to maintain a package and 200
>> > manuals for that.
>> No, no, there are more workflows than manuals for them.
>
>For instance there is no manual for cdbs (at least I have not yet found
>one that helps you over the real hurdles).

cdbs docs used to be pretty good. Or at least I remember them to be
that way before I switched to dh.

Greetings
Marc
-- 
-- !! No courtesy copies, please !! -
Marc Haber |   " Questions are the | Mailadresse im Header
Mannheim, Germany  | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[James Clarke]

On Mon, May 22, 2017 at 05:10:26PM +0100, Jonathan Dowland wrote:
> On Mon, May 22, 2017 at 03:25:38PM +0100, James Clarke wrote:
> > On Mon, May 22, 2017 at 03:06:48PM +0100, Jonathan Dowland wrote:
> > > Excellent, this is a great start, and seeing "Michael Stapelberg" for me 
> > > is an
>^^^
> > > indication of quality.
>
> emphasis on *start*

To qualify as "great" I think you should be accurate, but hey, maybe I
have too high standards. Anyway the main point wasn't to attack you but
to provide another side to that blog post.

> > You say that, but this is incredibly biased. Even he admits that in the
> > colour choice.
>
> I was completely ignoring the colour choices and implementation language.
>
> > that for the sbuild path, schroot is completely missing
>
> Indeed there are a lot of omissions. I think the next step, whether this
> is the basis for what goes next, or not, would be to get the source of
> this picture (or another) into a version control system so it can be
> iteratively and collaboratively improved.

That sounds sensible (for the uncoloured version).

> [ Stapelberg via James Clarke ]
> > > I propose to eliminate complexity in Debian by deprecating the
> > > pbuilder toolchain in favor of sbuild.
> >
> > Now, I am not necessarily against simplifying workflows, but choosing
> > one to rule the all arbitrarily because you prefer the language it's
> > written in is not the way to go about it.
>
> Language should be a factor, as there are practical considerations that can't
> be avoided, independent of personal preferences – but I got the impression
> that he was not coming to this conclusion soley on the basis of implementation
> language anyway, nor soley in terms of his own preferences for implementation
> language.

Sure, language can be important, but I would say Bash and C are *more*
accessible to the average developer these days than Perl, and that's
only going to become more true over time. He didn't give any other
reasons, but if there are, these would be welcome in the form of
constructive bug reports so we can address them and make the situation
better for everyone.

> > There are also benefits to having a variety of implementations; they behave
> > slightly differently, sometimes not implementing policy in the same way, and
> > this can be a good thing, allowing policy to be clarified, or finding
> > mistakes in the other builder, or just exposing flaws in your package.
>
> These are good points. However, considering just these points in isolation,
> the "secondary" implementation(s) would not need to be something that was
> actually geared towards real users, or advertised as such.

There already effectively is a semi-"primary" implementation given that
sbuild is used on the buildds. And as for making these "secondary"
implementations not geared for real users, for whom would they then be?
Designating a certain tool as "primary" will just lead its behaviour
becoming the de facto policy, so everyone will be required to build with
it locally, and so nobody would even consider using a secondary
alternative.

> > I'm not trying to sell pbuilder over sbuild here though; there are also many
> > ways in which sbuild is better than pbuilder.
>
> I am not (yet) familiar enough with either of them (or the other dozen or so
> similar tools for the same job) to come to a conclusion on what I think would
> be the correct number/recommendations for Debian contributors, but I am of the
> initial opinion that there are too many. I've heard a lot of good things about
> sbuild.

There are lots of areas where Debian has far too many tools to
accomplish the same thing, but I don't think this is one of them; there
are only two main tools for building in chroots (sbuild and
pbuilder[0]), both of which have significant user bases.

Anyway, I'm done with this debate; it's clear I have very different
views from some on this matter.

James

[0] cowbuilder is a thin wrapper that behaves (almost) identically, so
it doesn't really count as something different

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Sean Whitton]

On Mon, May 22, 2017 at 03:07:42PM +0100, Ian Jackson wrote:
> Areas of work that could do with attention from people with relevant
> expertise and effort:
> 
>  * Getting rid of the need to mess with the changelog.  That might
>involve changes to Debian changelog practice, or better tooling (eg
>yet another wrapper around dpkg-buildpackage - maybe a way to set
>the version without committing? - etc. etc.)

A way to set the version during the build, as you suggest, would be
sufficient to cover this.  It is hard to see how we could relieve the
user of the need to understand how to choose a version number for a .deb
for testing.  An option to set the version in the build command line
would remove the need for Debian source package knowledge.

>  * Pull request workflow for submitting changes.  This should
>eventually turn into a bug submission to the Debian BTS.
>This sounds to me like it probabl needs to be a web service, but
>perhaps some local client utility that looked enough like a web
>page would do.

We basically already have all the pieces:

- git-request-pull(1)
- reportbug(1)
- git hosting on alioth / our shiny pagure git hosting (coming soon)

dgit could ship a script that ties these together.  (The reason I suggest
using our own git hosting is so that the branch doesn't disappear -- one
advantage of patches in the BTS is that they can't 404.)

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Sean Whitton]

On Mon, May 22, 2017 at 01:42:54PM -0400, Jeremy Bicha wrote:
> Of course, dgit is yet another workflow and my understanding is that
> git-buildpackage (without dgit) is far more commonly used in Debian.

This isn't fair to dgit.  While it does impose some minimal requirements
upon git trees, it is otherwise workflow neutral.  This is evidenced by
the fact that there are already several distinct dgit-compatible
workflows documented in the dgit-maint-*(7) namespace.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Andreas Tille]

On Mon, May 22, 2017 at 10:46:59PM +0500, Andrey Rahmatullin wrote:
> On Mon, May 22, 2017 at 09:07:52AM +, Holger Levsen wrote:
> > there's just a hundred Debian workflows to maintain a package and 200
> > manuals for that.
> No, no, there are more workflows than manuals for them.

For instance there is no manual for cdbs (at least I have not yet found
one that helps you over the real hurdles).

SCNR

  Andreas.

-- 
http://fam-tille.de

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Zlatan Todoric]

On 05/22/2017 07:42 PM, Jeremy Bicha wrote:
> On Mon, May 22, 2017 at 10:07 AM, Ian Jackson
> <ijack...@chiark.greenend.org.uk> wrote:
>> Holger Levsen writes ("infinite number of Debian workflows (Re: Moving away 
>> from (unsupportable) FusionForge on Alioth?)"):
>> I would encourage anyone who has effort to work on this end of the
>> contributor experience to consider what dgit has to offer.  dgit
>> provides a single git workflow for all Debian packages.
> Of course, dgit is yet another workflow and my understanding is that
> git-buildpackage (without dgit) is far more commonly used in Debian.
>
> Thanks,
> Jeremy Bicha
>
https://xkcd.com/927/

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Andrey Rahmatullin]

On Mon, May 22, 2017 at 09:07:52AM +, Holger Levsen wrote:
> there's just a hundred Debian workflows to maintain a package and 200
> manuals for that.
No, no, there are more workflows than manuals for them.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Jeremy Bicha]

On Mon, May 22, 2017 at 10:07 AM, Ian Jackson
<ijack...@chiark.greenend.org.uk> wrote:
> Holger Levsen writes ("infinite number of Debian workflows (Re: Moving away 
> from (unsupportable) FusionForge on Alioth?)"):
> I would encourage anyone who has effort to work on this end of the
> contributor experience to consider what dgit has to offer.  dgit
> provides a single git workflow for all Debian packages.

Of course, dgit is yet another workflow and my understanding is that
git-buildpackage (without dgit) is far more commonly used in Debian.

Thanks,
Jeremy Bicha

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Jonathan Dowland]

On Mon, May 22, 2017 at 03:25:38PM +0100, James Clarke wrote:
> On Mon, May 22, 2017 at 03:06:48PM +0100, Jonathan Dowland wrote:
> > Excellent, this is a great start, and seeing "Michael Stapelberg" for me is 
> > an
 ^^^
> > indication of quality.

emphasis on *start*

> You say that, but this is incredibly biased. Even he admits that in the
> colour choice.

I was completely ignoring the colour choices and implementation language.

> that for the sbuild path, schroot is completely missing

Indeed there are a lot of omissions. I think the next step, whether this
is the basis for what goes next, or not, would be to get the source of
this picture (or another) into a version control system so it can be
iteratively and collaboratively improved.

[ Stapelberg via James Clarke ]
> > I propose to eliminate complexity in Debian by deprecating the
> > pbuilder toolchain in favor of sbuild.
> 
> Now, I am not necessarily against simplifying workflows, but choosing
> one to rule the all arbitrarily because you prefer the language it's
> written in is not the way to go about it.

Language should be a factor, as there are practical considerations that can't
be avoided, independent of personal preferences – but I got the impression
that he was not coming to this conclusion soley on the basis of implementation
language anyway, nor soley in terms of his own preferences for implementation
language.

> There are also benefits to having a variety of implementations; they behave
> slightly differently, sometimes not implementing policy in the same way, and
> this can be a good thing, allowing policy to be clarified, or finding
> mistakes in the other builder, or just exposing flaws in your package.

These are good points. However, considering just these points in isolation,
the "secondary" implementation(s) would not need to be something that was
actually geared towards real users, or advertised as such.

> I'm not trying to sell pbuilder over sbuild here though; there are also many
> ways in which sbuild is better than pbuilder.

I am not (yet) familiar enough with either of them (or the other dozen or so
similar tools for the same job) to come to a conclusion on what I think would
be the correct number/recommendations for Debian contributors, but I am of the
initial opinion that there are too many. I've heard a lot of good things about
sbuild.

-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄ Please do not CC me, I am subscribed to the list.

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[James Clarke]

On Mon, May 22, 2017 at 03:06:48PM +0100, Jonathan Dowland wrote:
> On Mon, May 22, 2017 at 12:47:51PM +0100, Sean Whitton wrote:
> > Someone else already had this idea:
> >
> > https://people.debian.org/~stapelberg//2016/11/25/build-tools.html
>
> Excellent, this is a great start, and seeing "Michael Stapelberg" for me is an
> indication of quality.

You say that, but this is incredibly biased. Even he admits that in the
colour choice. Disclaimer: as the cowbuilder maintainer (which comes
from the cow*dancer* source package, for historical reasons, despite
what the diagram may tell you) I am of course also biased. But I notice
that for the sbuild path, schroot is completely missing, which is
implemented in C++ and therefore probably deserves a bright red colour
just like cowbuilder in his second picture (or maybe even something
worse, depending on your views of C++). Then there's the extremely
unhelfpul, hostile comment at the end:

> I propose to eliminate complexity in Debian by deprecating the
> pbuilder toolchain in favor of sbuild.

Now, I am not necessarily against simplifying workflows, but choosing
one to rule the all arbitrarily because you prefer the language it's
written in is not the way to go about it. There are also benefits to
having a variety of implementations; they behave slightly differently,
sometimes not implementing policy in the same way, and this can be a
good thing, allowing policy to be clarified, or finding mistakes in the
other builder, or just exposing flaws in your package. For example,
pbuilder will by default use a new isolated network namespace for the
build, whereas sbuild won't, so if you just use sbuild you may not be
aware of violating policy (4.9). I'm not trying to sell pbuilder over
sbuild here though; there are also many ways in which sbuild is better
than pbuilder.

James

infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Ian Jackson]

Holger Levsen writes ("infinite number of Debian workflows (Re: Moving away 
from (unsupportable) FusionForge on Alioth?)"):
> I can totally confirm this. When people ask me how to get foo fixed in Debian
> and I start explaining the above, people role their eyes and point to this:

Yes.

> there have been a few trying to stick around but often they dont
> know where to move on, as there is no "Debian workflow", no "Debian
> manual", there's just a hundred Debian workflows to maintain a
> package and 200 manuals for that.

I would encourage anyone who has effort to work on this end of the
contributor experience to consider what dgit has to offer.  dgit
provides a single git workflow for all Debian packages.

Depending on the maintainer's workflow practices, the history may be
poor, but the code is there in your git tree just like a non-Debian
git user would expect, and patches can be cherry-picked straight onto
it off upstream branches.

The dgit user does not need to know anything about the Debian
packaging teams or workflow rules or to read package-specific
documentation from the maintainer; nor do they need to know anything
about Debian source *packages* (as opposed to source *trees*).

To do a test build and install the user does need to know
some runes, and they do need to mess with the changelog:
  https://manpages.debian.org/jessie-backports/dgit/dgit-user.7.en.html
(And of course if they want to change packaging they will need to know
or learn about whatever it is they're changing.)

Areas of work that could do with attention from people with relevant
expertise and effort:

 * Getting rid of the need to mess with the changelog.  That might
   involve changes to Debian changelog practice, or better tooling (eg
   yet another wrapper around dpkg-buildpackage - maybe a way to set
   the version without committing? - etc. etc.)

 * Pull request workflow for submitting changes.  This should
   eventually turn into a bug submission to the Debian BTS.
   This sounds to me like it probabl needs to be a web service, but
   perhaps some local client utility that looked enough like a web
   page would do.

 * User/contributor testing to discover other roadblocks that make
   contribution tricky.

An area that I am working on myself is:

 * A way to do a new upstream version that does not involve the user
   having to mess with Debian source packages.  Sadly I can't see how
   to do this in a reasonable way without interacting with the
   maintainers' git workflows; for now, I am working on a way for
   maintainer(s) to cooperate using git branches as the interchange
   format, with the patch stacks on upstream worked on as a rebasing
   git branch.

Ian.

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Jonathan Dowland]

On Mon, May 22, 2017 at 12:47:51PM +0100, Sean Whitton wrote:
> Someone else already had this idea:
> 
> https://people.debian.org/~stapelberg//2016/11/25/build-tools.html

Excellent, this is a great start, and seeing "Michael Stapelberg" for me is an
indication of quality.



-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄ Please do not CC me, I am subscribed to the list.

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Jonathan Dowland]

On Mon, May 22, 2017 at 12:28:01PM +0200, Emilio Pozuelo Monfort wrote:
> Do you mean cdbs rather than cmake? I'm struggling to make a connection 
> between
> dh and cmake.

Yes, I do; thanks for the correction.

(although it does remind me that this issue of more-than-one-way-to-do-it 
extends
up to upstreams, too)

-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄ Please do not CC me, I am subscribed to the list.

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Sean Whitton]

On Mon, May 22, 2017 at 10:29:24AM +0100, Jonathan Dowland wrote:
> I often think about this problem, and I start to wonder if step 0 is to try 
> and
> enumerate it properly. That is: I picture in my mind some kind of huge diagram
> (perhaps generated from more structured data, I dunno, something into a 
> graphviz)
> of a landscape of debian developer tools, grouped by some kind of
> categorisation that might overlap (venn diagram style), so you'd have dh and
> cmake in one place, git-buildpackage, dgit, possibly others in another; and
> also our documentation: not just maint-guide but developers-reference, wiki
> pages, etc.

Someone else already had this idea:

https://people.debian.org/~stapelberg//2016/11/25/build-tools.html

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Emilio Pozuelo Monfort]

On 22/05/17 11:29, Jonathan Dowland wrote:
> I often think about this problem, and I start to wonder if step 0 is to try 
> and
> enumerate it properly. That is: I picture in my mind some kind of huge diagram
> (perhaps generated from more structured data, I dunno, something into a 
> graphviz)
> of a landscape of debian developer tools, grouped by some kind of
> categorisation that might overlap (venn diagram style), so you'd have dh and
> cmake in one place, git-buildpackage, dgit, possibly others in another; and
> also our documentation: not just maint-guide but developers-reference, wiki
> pages, etc.
> 
> Such a thing could potentially be annotated with relevant statistics (number
> of source packages in archive using dh; cmake; etc.; number with Vcs headers,
> pointing at git or svn or ... repos;)

Do you mean cdbs rather than cmake? I'm struggling to make a connection between
dh and cmake.

Cheers,
Emilio

Re: infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Jonathan Dowland]

I often think about this problem, and I start to wonder if step 0 is to try and
enumerate it properly. That is: I picture in my mind some kind of huge diagram
(perhaps generated from more structured data, I dunno, something into a 
graphviz)
of a landscape of debian developer tools, grouped by some kind of
categorisation that might overlap (venn diagram style), so you'd have dh and
cmake in one place, git-buildpackage, dgit, possibly others in another; and
also our documentation: not just maint-guide but developers-reference, wiki
pages, etc.

Such a thing could potentially be annotated with relevant statistics (number
of source packages in archive using dh; cmake; etc.; number with Vcs headers,
pointing at git or svn or ... repos;)

I'm inspired by the Debian Women wiki's diagrams of packaging workflows which
took existing flows and presented them in a way that made them much easier to
understand as a whole (and also made it clearer how complex they are, or aren't)

Then we could look to see what should be eliminated in order to make the diagram
simpler.

We could re-calculate/draw the diagram on a regular basis: annually (or even
monthly if we had much movement behind this idea of simplifying the landscape)
to see what the current state of the art is, and compare it to the case last
time.

We could even attempt to formulate the 'ideal picture' diagram, as something to
work towards, although we would not likely get a complete consensus on any one
version of that.

-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄ Please do not CC me, I am subscribed to the list.

infinite number of Debian workflows (Re: Moving away from (unsupportable) FusionForge on Alioth?)

[Holger Levsen]

On Mon, May 22, 2017 at 07:52:34AM +, Riku Voipio wrote:
> Right now, if you have a minor change - such fixing Homepage: or typo on
> definition, it's not as straitforward as submitting a pull request. And
> it gets much worse if you want to patch against upstream or build a new
> upstream version. Every maintainer has their own preferred workflow which
> a new contributor needs to adapt to.

_this_!

I can totally confirm this. When people ask me how to get foo fixed in Debian
and I start explaining the above, people role their eyes and point to this:

> [...] But for better or worse, there is now a generation that has
> become used to github style use. And if we want avoid Debian becoming an
> old grumpy mens club, we have cater for them.

and then most of these people move on without bothering trying to fix it.

there have been a few trying to stick around but often they dont know where
to move on, as there is no "Debian workflow", no "Debian manual", there's just
a hundred Debian workflows to maintain a package and 200 manuals for that.

The nice thing about standards is *not* that there are so many to choose from.


-- 
cheers,
Holger


signature.asc
Description: Digital signature