Re: The Difference between debcheckout and dgit and what they try to accomplish

[Sean Whitton]

Hello,

On Wed 19 Jun 2019 at 11:51pm +0200, Wouter Verhelst wrote:

> On Mon, Jun 17, 2019 at 11:47:38AM +0100, Sean Whitton wrote:
>> We could try to write a tool which tries to guess and convert (e.g.) the
>> dgit view with your changes into a maintainer workflow, but there are
>> large obstacles to this working reliably.  For example, there exist edge
>> cases such that there is no algorithm which can determine, for any
>> possible Debian source package tree committed to git, whether it is
>> patches-applied or patches-unapplied.  And there are so many small
>> variations on workflows that such a tool would have to be very complex.
>
> What if you took away the necessary guesswork?
>
> Have dgit support a field in debian/control that, if it exists, explains
> to dgit (and any other tool that might care) what the workflow type is.
> This would require a categorization of all the possible git layouts, and
> would initially obviously not support all of them.
>
> But once it exists, it would allow behaviour for a hypothetical "dgit
> create-merge-request" command or some such, where if the field exists in
> debian/control a merge request is posted to salsa in the correct
> fashion; and if the field does *not* exist in debian/control, it would
> then instead simply be a wrapper around 'git format-patch' and/or 'git
> send-email'.
>
> This would, as an aside, also allow maintainers to express that they
> would prefer merge requests in salsa over patches in the bts, so would
> have use beyond dgit.

This is a cool idea.  I'd just like to note that I don't think we would
want it to be in any way dgit-specific or labelled with 'dgit', and
`dgit create-merge-request` would probably be better something in
devscripts than a dgit subcommand.

dgit it meant to be only for interoperation between git and the
archive.  If this metadata has uses beyond that, it should not be
labelled 'dgit'.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Philip Hands]

Ansgar Burchardt  writes:

> On Thu, 2019-06-20 at 10:52 +0200, Enrico Zini wrote:
>> This reminds me of something that popped up in a dinner discussion a few
>> days ago: mandate documenting workflow in debian/README.source no matter
>> what, and allow to symlink that file to a repository in
>> /usr/share/doc/somewhere/ as we do for common licenses.
>
> My workflow also includes getting changes merged upstream, so any such
> documentation would need to include the upstream workflow. Information
> how to contribute also includes code style (spaces vs tabs), naming
> conventions and so on.
>
> I'm not willing to write such things down for Debian for the rare case
> someone might read it; there are enough other things that need to be
> done.

I'm glad for you that all those details are currently retained by your
brain, but I suspect that if you're fortunate enough to live long enough
there will come a time when such things manage to evaporate from memory.

That being the case, you might want to consider writing such
documentation while it's trivially easy for you to do so from memory, so
as to save the frustration of your future self forgetting them.

As a side effect, you'll also be saving anyone else from the effort of
finding all that out about the upstreams etc. in case your interests
change, and someone else has to take over your packages.

You also seem to be making the perfect the enemy of the good here.  I'd
imagine that a partial description of your workflow would be better than
none, and if anyone ever asks about the extra bits you could answer them
by adding missing details to the package and thus get closer to full
documentation without significant additional effort.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Chris Lamb]

Hi Enrico,

> This reminds me of something that popped up in a dinner discussion a few
> days ago: mandate documenting workflow in debian/README.source no matter
> what, and allow to symlink that file to a repository in
> /usr/share/doc/somewhere/ as we do for common licenses.

I do like the symlink part of this idea as it could mostly address the
"boy who cried wolf"-like boilerplate concerns that I raised in 2009
about previous potential overuses of README.source, although in this
particular case with respect to quilt patches:

  https://bugs.debian.org/543417
  
(*silently cringes at past-lamby's usage of the hackneyed "considered
harmful" suffix*)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org  chris-lamb.co.uk
   `-

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Ansgar Burchardt]

On Thu, 2019-06-20 at 10:52 +0200, Enrico Zini wrote:
> This reminds me of something that popped up in a dinner discussion a few
> days ago: mandate documenting workflow in debian/README.source no matter
> what, and allow to symlink that file to a repository in
> /usr/share/doc/somewhere/ as we do for common licenses.

My workflow also includes getting changes merged upstream, so any such
documentation would need to include the upstream workflow. Information
how to contribute also includes code style (spaces vs tabs), naming
conventions and so on.

I'm not willing to write such things down for Debian for the rare case
someone might read it; there are enough other things that need to be
done.

Ansgar

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Enrico Zini]

On Wed, Jun 19, 2019 at 11:51:14PM +0200, Wouter Verhelst wrote:

> What if you took away the necessary guesswork?
> 
> Have dgit support a field in debian/control that, if it exists, explains
> to dgit (and any other tool that might care) what the workflow type is.
> This would require a categorization of all the possible git layouts, and
> would initially obviously not support all of them.

This reminds me of something that popped up in a dinner discussion a few
days ago: mandate documenting workflow in debian/README.source no matter
what, and allow to symlink that file to a repository in
/usr/share/doc/somewhere/ as we do for common licenses.

That way, there is a standard, machine readable way to detect standard
workflows, everything is documented no matter what, common workflow
documentation can grow without changes to packages, and people are
encouraged to adopt a standard workflow, because they come already
documented.


Enrico

-- 
GPG key: 4096R/634F4BD1E7AD5568 2009-05-08 Enrico Zini 

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Ansgar Burchardt]

Russ Allbery writes:
> Colin Watson writes:
>> Is it at all likely that the ftpmaster api service might migrate away
>> from Let's Encrypt at this point?  I would assume probably not.  In that
>> case, you could at least make the situation substantially better with no
>> further DSA work required by pinning the appropriate LE root certificate
>> in dgit.
>
> debian.org already publishes a CAA record, which conveys that information
> (although has its own verification concerns, but I think debian.org is
> using DNSSEC so you can verify the record that way).  It says that all
> debian.org hosts will only use certificates from either LE or Amazon:

The CAA record does not indicate a future commitment and could change at
any time.  If you want to rely on debian.org to use some specific CAs,
you would have to ask DSA.

(Nor does the CAA record indicate that all currently valid certificates
must come from the listed CAs; the CAA record could have been different
when those were issued.)

Ansgar

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Wouter Verhelst]

On Mon, Jun 17, 2019 at 11:47:38AM +0100, Sean Whitton wrote:
> We could try to write a tool which tries to guess and convert (e.g.) the
> dgit view with your changes into a maintainer workflow, but there are
> large obstacles to this working reliably.  For example, there exist edge
> cases such that there is no algorithm which can determine, for any
> possible Debian source package tree committed to git, whether it is
> patches-applied or patches-unapplied.  And there are so many small
> variations on workflows that such a tool would have to be very complex.

What if you took away the necessary guesswork?

Have dgit support a field in debian/control that, if it exists, explains
to dgit (and any other tool that might care) what the workflow type is.
This would require a categorization of all the possible git layouts, and
would initially obviously not support all of them.

But once it exists, it would allow behaviour for a hypothetical "dgit
create-merge-request" command or some such, where if the field exists in
debian/control a merge request is posted to salsa in the correct
fashion; and if the field does *not* exist in debian/control, it would
then instead simply be a wrapper around 'git format-patch' and/or 'git
send-email'.

This would, as an aside, also allow maintainers to express that they
would prefer merge requests in salsa over patches in the bts, so would
have use beyond dgit.

-- 
To the thief who stole my anti-depressants: I hope you're happy

  -- seen somewhere on the Internet on a photo of a billboard

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Russ Allbery]

Sam Hartman  writes:
>> "Russ" == Russ Allbery  writes:

> Russ> debian.org already publishes a CAA record, which conveys that
> Russ> information (although has its own verification concerns, but I
> Russ> think debian.org is using DNSSEC so you can verify the record
> Russ> that way).  It says that all debian.org hosts will only use
> Russ> certificates from either LE or Amazon:

> Russ, you may be more up to date on webpki than I am.
> Does that say anything about which root letsencrypt will chain to?
> I.E. can letsencrypt change what their chain looks like?

It does, but that's because it's not the greatest for clients, which makes
it a touch hard to use on the dgit side.

My understanding is that the CAA record is intended as a constraint on
CAs, not really a verification step for clients.  A CAA is not supposed to
issue certificates for a domain that has a CAA record and that doesn't
list that issuer.  Because of that, the interpretation of the CAA record
is somewhat up to the domain.  It's intended to protect you against
legitimate CAs being socially-engineered to issue a certificate for your
domain.

That also means there's no easy way to map a CAA record value to a
specific certificate or certificate chain.  It therefore doesn't solve the
dgit problem directly, but it does provide some verification that
debian.org domains are only going to use LE (and Amazon) certs unless that
CAA record changes, and therefore dgit could do client-side certificate
pinning to those two certificate chains, at the cost of having to manually
track changes to those chains.

-- 
Russ Allbery (r...@debian.org)   

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Sam Hartman]

> "Russ" == Russ Allbery  writes:

Russ> Colin Watson  writes:
>> Is it at all likely that the ftpmaster api service might migrate
>> away from Let's Encrypt at this point?  I would assume probably
>> not.  In that case, you could at least make the situation
>> substantially better with no further DSA work required by pinning
>> the appropriate LE root certificate in dgit.

Russ> debian.org already publishes a CAA record, which conveys that
Russ> information (although has its own verification concerns, but I
Russ> think debian.org is using DNSSEC so you can verify the record
Russ> that way).  It says that all debian.org hosts will only use
Russ> certificates from either LE or Amazon:

Russ, you may be more up to date on webpki than I am.
Does that say anything about which root letsencrypt will chain to?
I.E. can letsencrypt change what their chain looks like?

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Russ Allbery]

Colin Watson  writes:

> Is it at all likely that the ftpmaster api service might migrate away
> from Let's Encrypt at this point?  I would assume probably not.  In that
> case, you could at least make the situation substantially better with no
> further DSA work required by pinning the appropriate LE root certificate
> in dgit.

debian.org already publishes a CAA record, which conveys that information
(although has its own verification concerns, but I think debian.org is
using DNSSEC so you can verify the record that way).  It says that all
debian.org hosts will only use certificates from either LE or Amazon:

gwaihir:~$ host -t caa debian.org
debian.org has CAA record 0 iodef "mailto:d...@debian.org;
debian.org has CAA record 128 issuewild ";"
debian.org has CAA record 128 issue "letsencrypt.org"
debian.org has CAA record 128 issue "amazon.com"

-- 
Russ Allbery (r...@debian.org)   

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Colin Watson]

On Wed, Jun 19, 2019 at 01:57:39PM +0100, Ian Jackson wrote:
> FTAOD: I have a memory that in response to Hector Oron's message #20
> in that bug, I did try to have a conversation on debian-admin, but
> that I found that conversation very frustrating.  I did not feel that
> the DSA members I was talking to were listening very well.  Probably,
> they felt I was rude.  I gave up. [1]

Is it at all likely that the ftpmaster api service might migrate away
from Let's Encrypt at this point?  I would assume probably not.  In that
case, you could at least make the situation substantially better with no
further DSA work required by pinning the appropriate LE root certificate
in dgit.  While that still means that LE could subvert the service, it
would prevent anyone else who operates one of the many CAs in
ca-certificates from doing the same.  Does that help?

-- 
Colin Watson   [cjwat...@debian.org]

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Ian Jackson]

Ian Jackson writes ("Re: The Difference between debcheckout and dgit and what 
they try to accomplish"):
> Hector: If it would be useful to you, I could tell you a set of dgit
> configuration settings to have it use the apt repository fetching
> method.  That would rely, therefore, on the existing apt dsc
> verification system (based on Release files etc.), and not on the
> ftpmaster api service.

I meant "Helmut".  Sorry to get that wrong!

(I will 

Ian.

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Ian Jackson]

> On Mon 17 Jun 2019 at 06:21pm +0200, Helmut Grohne wrote:
> > Presently, no. I attempted using it, but I feel that the extra
> > complexity did not help my use case. dgit solves a difficult problem and
> > that comes at a cost. Verification of source integrity is much more
> > difficult to understand with dgit (and it presently seems to have a
> > trust root in the ca business). The integrity checking performed by
> > apt-get source on the other hand is quite easily explained (if you
> > assume gpgv).

On this, Sean writes:

> Can I ask whether you think it would help if dgit was more verbose about
> the verification it was doing?  Telling you what the ftpmaster API was
> telling it, or something.

This might be of some use but I don't think it is a real solution for
Helmut.

> The commercial SSL thing is indeed a problem (#790093).

FTAOD: I have a memory that in response to Hector Oron's message #20
in that bug, I did try to have a conversation on debian-admin, but
that I found that conversation very frustrating.  I did not feel that
the DSA members I was talking to were listening very well.  Probably,
they felt I was rude.  I gave up. [1]

I would really appreciate it if someone else (who understands the
problem) would have a go.

Hector: If it would be useful to you, I could tell you a set of dgit
configuration settings to have it use the apt repository fetching
method.  That would rely, therefore, on the existing apt dsc
verification system (based on Release files etc.), and not on the
ftpmaster api service.

This is not the default because (1) it means downloading the whole of
the Sources for a distribution just to obtain one package and (2) it
has a much greater risk of skew due to getting stale data.  If more
people want this mode of operation I could provide a more cooked way
to specify it, although I think it is suboptimal to fixing #790093.

> >  I
> > occasionally look into history of packages to figure something out. For
> > this case, dgit is not useful due to its low adoption and being young.
> > On the other hand, debian/changelog often suffices here.

dgit is not young.  It was invented in August 2013 in Vaumarcus by
Joey Hess and I.  So it is nearly 6 years old.

Low adoption of "dgit push" is indeed a serious problem.

> For the history thing, after you `dgit clone`, `git fetch vcs-git` will
> get you the maintainer's history for browsing.  That's about as easy as
> debcheckout.

It's not really what Helmut wants, though.

Ian.

[1] I don't appear to have saved a transcript of the conversation.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Sean Whitton]

Hello Helmut,

On Mon 17 Jun 2019 at 06:21pm +0200, Helmut Grohne wrote:

> Presently, no. I attempted using it, but I feel that the extra
> complexity did not help my use case. dgit solves a difficult problem and
> that comes at a cost. Verification of source integrity is much more
> difficult to understand with dgit (and it presently seems to have a
> trust root in the ca business). The integrity checking performed by
> apt-get source on the other hand is quite easily explained (if you
> assume gpgv). Then, I'm not interested in commits that are not yet
> uploaded. I want to reproduce the exact failure that QA saw. I
> occasionally look into history of packages to figure something out. For
> this case, dgit is not useful due to its low adoption and being young.
> On the other hand, debian/changelog often suffices here.
>
> So it's not like dgit would be bad. It just seems to solve a different
> problem than the one I have.

This is good feedback, thank you.

Can I ask whether you think it would help if dgit was more verbose about
the verification it was doing?  Telling you what the ftpmaster API was
telling it, or something.

The commercial SSL thing is indeed a problem (#790093).

For the history thing, after you `dgit clone`, `git fetch vcs-git` will
get you the maintainer's history for browsing.  That's about as easy as
debcheckout.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Helmut Grohne]

Hi,

Let me try to answer multiple mails as one:

On Mon, Jun 17, 2019 at 07:44:39AM +0200, Andreas Tille wrote:
> In other words: If vcs-git would be uniform would this be more
> attractive to you?

Maybe. Uniformity certainly is an important property to me. Having to
figure out different workflows is simply too much when the average
budget per patch is supposed to be less than a quarter of an hour.
Uniformity is not sufficient though. QA-tooling tests what is uploaded
to unstable. As long as it tests unstable and not vcs-git master, I want
that unstable tree and not the vcs-git master.

On Mon, Jun 17, 2019 at 11:27:07AM +0200, IOhannes m zmölnig (Debian/GNU) wrote:
> out of curiosity (and because i usually quite enjoy your patches): do
> you do use dgit in *your* workflow?

Presently, no. I attempted using it, but I feel that the extra
complexity did not help my use case. dgit solves a difficult problem and
that comes at a cost. Verification of source integrity is much more
difficult to understand with dgit (and it presently seems to have a
trust root in the ca business). The integrity checking performed by
apt-get source on the other hand is quite easily explained (if you
assume gpgv). Then, I'm not interested in commits that are not yet
uploaded. I want to reproduce the exact failure that QA saw. I
occasionally look into history of packages to figure something out. For
this case, dgit is not useful due to its low adoption and being young.
On the other hand, debian/changelog often suffices here.

So it's not like dgit would be bad. It just seems to solve a different
problem than the one I have.

On Mon, Jun 17, 2019 at 11:47:38AM +0100, Sean Whitton wrote:
> It would seem, then, that what we want is merge requests against our
> interchange format: the source packages that actually get uploaded.  Or,

Yes. And as much as people may dislike it, a working way presently is
sending a .debdiff to the bts. We require maintainers to receive such
mail. (When the maintainer address bounces, we file rc bugs.) We require
maintainers to interoperate with the bts. Presently, you cannot easily
say "please only use salsa" and ignore the bts. If you do, changes are
that the autoremover quickly takes care of your package. This
process/policy is what makes the bts useful to me.

So Michael Stapelberg constructively expressed some discomfort[1] with
the bts. Given that I'm a heavy bts user, I don't quite agree with the
details. On the other hand, I can hardly deny that not everyone is happy
about the current bts-debdiff-based workflow. Improving it certainly is
an important matter. It's just not obvious to me what the future should
look like.

Whatever we change here, it'll make some people unhappy. That is
unavoidable. Very likely, I'm part of those some people, because I
integrate heavily with the bts. That still may be a reasonable
trade-off. Arguing that "no you cannot this, because it works fine for
me" would be making innovation impossible.

It got a bit longer than originally intended. Hope this helps someone.

Helmut

[1] https://michael.stapelberg.ch/posts/2019-03-10-debian-winding-down/

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Ian Jackson]

Sean Whitton writes ("Re: The Difference between debcheckout and dgit and what 
they try to accomplish"):
> It would seem, then, that what we want is merge requests against our
> interchange format: the source packages that actually get uploaded.  Or,
> in other words, we want merge requests against the dgit view of
> packages.  Presumably package maintainers to massage such a thing into a
> merge request against the maintainer view ... ?

For most[1] git workflows, every MR to the dgit view could be
automatically converted to a MR against the maintainer view.  The
conversion is generally quite simple - indeed, it is usually simply a
particular arrangement of existing maintainer workflow tooling.

The exception would be new-upstream-version.  It is *possible* to use
dgit and git-debrebase together to do a new-upstream-version of *any*
package but the resulting git history is not very sane, and I think
not really suitable for such an automatic conversion.  I think in
practice, at least unless we want to do a lot of extra work,
new-upstream-version must be done with the maintainer view.

The resulting maintainer-MR would not be FF from the submitter-MR, but
it is not uncommon with "MR" approaches outside-Debian, for the
maintainer to rebase or squash or otherwise mangle the submitted MR
branch, so that the master does not become FF from the submitted MR.
So I don't think this is a significant problem.

Ian.

[1] Some workflows have automatically generated files - eg d/control -
in the dgit/.dsc view.  Edits to the dgit view which change
automatically generated files are not automatically back-convertable.
Indeed some such changes may not be representable at all in the
maintainer view...

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Ian Jackson]

Sam Hartman writes ("Re: The Difference between debcheckout and dgit and what 
they try to accomplish"):
> But my reading of the discussion so far is that people see getting
> patches actually integrated as a pain point.
...
> In the debian-vote discussion people were talking about a eutopia where
> every DD could push to every package.  And if not push, then submit a
> merge request.

Ah, I think I see better now what you are getting at.

If I may repeat back to you what I am hearing.  I proposed the
following solution-neutral problem statement:

 * There is far too much friction and makework caused by Debian's
   antiquated source code management practices.

You feel that this is a big topic, too big to handle all at once, so
you want to narrow your focus.  You wish to prioritise the following
aspect of the above:

 * ... Specifically, there is too much friction involved, after an
   contributor has prepared source code changes to a package, in
   getting those changes actually incorporated in the binary packages
   published by Debian.  (Reducing "friction" does not mean reducing
   review.)

I agree that this is important.  I also think it is possible to
address this *without* compromising the other goals.  Indeed, if we
get this right, it can (at least to an extent) *help* achieve those
other goals.

I do note that my proposed SNPS above does *not* imply that the
contributor is trying to use the maintainer's git workflow.  I think a
key question you are trying to answer with your message is whether we
should try to develop a solution which involves the contributor using
the maintainer's git branch and workflow, or one which involves the
contributor using a dgit view branch.

Am I right about that ?  If so I have an answer but I'm not sure you
will like it...

> If I'm understanding our private conversations, this is not your focus.

I'm not sure "Ian's focus" is the way I look at it.  I have an
ideology of prioritising helping users and downstreams realise their
so-far-often-theoretical software freedom, over improving Debian's
internal processes.

This is because pain from Debian's internal processes is felt by
Debian maintainers who are in a position to change things and highly
motivated to do so.  Whereas the pain felt by users and downstreams is
sometimes neglected.  Users and downstreams need champions within
Debian.

This seems to me to follow fairly objectively from the project's
common ideology as expressed in our foundation documents, combined
with some obvious observations about people scratching their own
itches.

That doesn't mean I don't think we can also help our users and
downstreams (by providing better software), and have more fun
ourselves, by improving our processes within Debian.  And I am very
happy to help with that.

>  But I think that some of your arguments in your response about
> whether something was or was not an advantage made it harder rathre
> than easier for me to see whether I understood the space.

I'm sorry that you evidently found my responses not very helpful.

> > Unfortunately I don't think it is possible to have the
> > conversation about how to reduce within-Debian friction without
> > at least considering how the proposals support or impede the
> > goal of providing users the source for the code on their
> > computers.
> 
> We are agreed.
> If we do start with discussing in-Debian friction, it is important that
> the impact of that discussion on other things needs to be inscope.
> If we're solving one problem at a time, we need to include discussion of
> future related work as it impacts our current discussion.

In that case, I have no problem with the approach you are taking.

While I am trying to do more (and think more is achievable), I can try
to confine my comments to the narrower scope.

> > That's not going to be possible because some possible answers
> > to what you call "the salsa discussion" make it harder to solve
> > our users' problems.
> 
> And I think arguments like that need to be in scope.

Great.

I'll wait to see what you say to this "scoping" mail.  Hopefully I can
then go back and read your initial mail again and try to discuss
technical details in a more useful way.

Thanks,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Sean Whitton]

Hello,

On Sun 16 Jun 2019 at 08:08pm -0400, Sam Hartman wrote:

> Dgit absolutely can be used to prepare a bunch of patches.  But unless
> the maintainer actually uses dgit, it's not clear that the result is
> easier for the maintainer to merge than a well submitted patch via the
> bts produced from apt-get source.
>
> (Yes, if you actually did an nmu, dgit clone users will benefit from the
> dgit push).
>
>
> But my reading of the discussion so far is that people see getting
> patches actually integrated as a pain point.  If I'm understanding our
> private conversations, this is not your focus.
> This is an area where I think your focus and the project's desires for
> improvements are not aligned.
>
> In the debian-vote discussion people were talking about a eutopia where
> every DD could push to every package.  And if not push, then submit a
> merge request.
>
> I think clarifying people's desires and understanding how critical being
> able to push to a package and/or submit merge requests to it is will be
> an important next step.

This has not been at the forefront of the minds of those of us working
on dgit, indeed, probably just because Ian and I are the kind of people
who are quite happy with patches in the BTS rather than merge requests.
However, ISTM that dgit as it already exists has a role to play in
making it possible to submit merge requests against packages.

Let's assume that we are not going to standardise on a single git
workflow for packages on salsa -- the amount of time and effort to
convert all maintainer views to a single workflow would be immense, and
I doubt that the project actually wants to do this, having positive
reasons, regarding technical diversity, for not doing it.  (It's a dgit
design goal not to require standardising the workflows of maintainer
views, for roughly these reasons.)

With that assumption made, however, submitting merge requests against
maintainer views on salsa now faces the problem that doing so requires
knowledge of the workflow the maintainer is using.

We could try to write a tool which tries to guess and convert (e.g.) the
dgit view with your changes into a maintainer workflow, but there are
large obstacles to this working reliably.  For example, there exist edge
cases such that there is no algorithm which can determine, for any
possible Debian source package tree committed to git, whether it is
patches-applied or patches-unapplied.  And there are so many small
variations on workflows that such a tool would have to be very complex.

It would seem, then, that what we want is merge requests against our
interchange format: the source packages that actually get uploaded.  Or,
in other words, we want merge requests against the dgit view of
packages.  Presumably package maintainers to massage such a thing into a
merge request against the maintainer view ... ?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Debian/GNU]

On 17.06.19 06:53, Helmut Grohne wrote:
> Whether I use apt-get source and debdiff or dgit
> and git format-patch is a detail on my side.


out of curiosity (and because i usually quite enjoy your patches): do
you do use dgit in *your* workflow?

fgmadsr
IOhannes

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Andreas Tille]

On Mon, Jun 17, 2019 at 06:53:05AM +0200, Helmut Grohne wrote:
> 
> Still, vcs-git is not my preferred interface to the archive, because its
> lack of uniformity makes using it unproductive for me.

In other words: If vcs-git would be uniform would this be more
attractive to you?

Kind regards

  Andreas.

-- 
http://fam-tille.de

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Helmut Grohne]

Hi Sam,

On Sun, Jun 16, 2019 at 08:08:51PM -0400, Sam Hartman wrote:
> Ian> That depends, I think, on whether you are trying to (a) join a
> Ian> maintainer team, or do "maintainer-like" work; or, (b) do
> Ian> cross-archive work with small changes to many packages, or
> Ian> archive-wide testing, or similar.
> 
> Ian> Any case where the change you want to make is simple, but you
> Ian> want to make it to many packages, is much easier to achieve
> Ian> with a dgit-based NMU workflow.
> 
> Ian> RC bugfixing and other kinds of "passer-by" QA work is usually
> Ian> easier with dgit too.  You only need to understand the bug, and
> Ian> the relevant parts of the package, not hold some git workflow
> Ian> stuff in your head too.
> Apologies for being a bit repetative.
> 
> I understand this is your opinion.  I am not sure this is generally
> accepted.  What I think I'm hearing from people is that they want to
> actually be able to get closer to being able to change maintainer
> sources than is supported by patches into the bts.

As someone who performs archive-wide changes, I fear I need to somewhat
support Ian's view here. If you are making small changes to many
packages, the maintainer's git tree is not the thing you want to use.

In that kind of situation, dealing with commits that are not yet
uploaded is a disadvantage to your performance. Having to figure out the
maintainer's workflow is a disadvantage to your performance.

My personal conclusion here is that I send patches via the bts. It may
be suboptimal from a maintainer pov (some express a strong preference to
salsa merge requests). Whether I use apt-get source and debdiff or dgit
and git format-patch is a detail on my side.

Still, vcs-git is not my preferred interface to the archive, because its
lack of uniformity makes using it unproductive for me.

I hope this data point helps a little

Helmut

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Sam Hartman]

When I outlined disadvantages and advantages I tried to follow what
people who have been participating in the discussion to date described
as advantages and disadvantages.  For the things I listed I have high
confidence that the people involved understood the tools they were using
and their own needs to accurately categorize what they valued.

Clearly we'll need to talk about values for the project as a whole.  But
I think that some of your arguments in your response about whether
something was or was not an advantage made it harder rathre than easier
for me to see whether I understood the space.
> "Ian" == Ian Jackson  writes:



Ian> Also: if all you are planning to do is prepare and test a
Ian> patch, then usually the easiest workflow is dgit clone followed
Ian> by some git-format-patch.  I have submitted many patches this
Ian> way and no-one has seemed to think them inconvenient.

They are certainly no less convenient than patches submitted through the
bts any other way.
It would not make sense to express inconvenience to an individual
submitter who properly follows our established procedure even if that
procedure is frustrating.
However a lot of patches do languish in the bts for non-rc issues.  I
think we've been hearing on blogs, in the debian-vote discussion, and in
feedback from people who do make changes across the archive that they
want improvement here.
I acknowledge it is my job to see if I'm reading comments correctly.

[My comments about being able to interact with the maintainer's work
flow being an advantage]

Ian> That depends, I think, on whether you are trying to (a) join a
Ian> maintainer team, or do "maintainer-like" work; or, (b) do
Ian> cross-archive work with small changes to many packages, or
Ian> archive-wide testing, or similar.

Ian> Any case where the change you want to make is simple, but you
Ian> want to make it to many packages, is much easier to achieve
Ian> with a dgit-based NMU workflow.

Ian> RC bugfixing and other kinds of "passer-by" QA work is usually
Ian> easier with dgit too.  You only need to understand the bug, and
Ian> the relevant parts of the package, not hold some git workflow
Ian> stuff in your head too.
Apologies for being a bit repetative.

I understand this is your opinion.  I am not sure this is generally
accepted.  What I think I'm hearing from people is that they want to
actually be able to get closer to being able to change maintainer
sources than is supported by patches into the bts.

Dgit absolutely can be used to prepare a bunch of patches.  But unless
the maintainer actually uses dgit, it's not clear that the result is
easier for the maintainer to merge than a well submitted patch via the
bts produced from apt-get source.

(Yes, if you actually did an nmu, dgit clone users will benefit from the
dgit push).


But my reading of the discussion so far is that people see getting
patches actually integrated as a pain point.  If I'm understanding our
private conversations, this is not your focus.
This is an area where I think your focus and the project's desires for
improvements are not aligned.

In the debian-vote discussion people were talking about a eutopia where
every DD could push to every package.  And if not push, then submit a
merge request.

I think clarifying people's desires and understanding how critical being
able to push to a package and/or submit merge requests to it is will be
an important next step.






>> Conclusions ===
>> 
>> I think that for the discussion I'm hoping to start soon, we're
>> focused on development within Debian.  That is, we do care about
>> code not yet released to the archive.  We do care about getting
>> future changes integrated, possibly even with little action on
>> the part of the maintainer.

Ian> I think these things are important but:

Ian> for the salsa discussion coming out of the DPL campaign

Ian> I think you are narrowing the scope unduly, perhaps because you
Ian> already have an idea what kind of a solution you want, or
Ian> because you want to solve some problems and leave others for
Ian> later.

I want to talk about solving some problems and leaving others for later.
My prioritization is based on what I've read in discussions so far.

Ian> Obviously it's up to you what you want to prioritise but I
Ian> think we need a solution-neutral problem statement.

Ian> For me, that is:

Ian>  * There is far too much friction and makework caused by
Ian> Debian's antiquated source code management practices.

I don't think I could run a discussion that broad and get anywhere.  If
you or someone else would like to take a crack at running a discussion
that broad, I'm open to stepping back for a month or so and seeing where
the discussion gets.  I'd think a month ought to be enough time to see
if things get stuck or if we're making forward progress.

I don't even 

Re: The Difference between debcheckout and dgit and what they try to accomplish

[Ian Jackson]

I haven't really organised my thoughts and have been travelling a lot
and am very tired, so this is a bit bitty.  I may respond more
usefully in a day or two:

Sam Hartman writes ("The Difference between debcheckout and dgit and what they 
try to accomplish"):
> Debcheckout allows you to get the latest version of the maintainer's
> preferred branch.

"debcheckout" is just a convenience script for getting at whatever
Vcs-Git provides.  In this context I should note that "dgit clone"
will set up a git remote called "vcs-git" which you can "git fetch",
to access the same thing[1] as "debcheckout" gives you.

[1] There is a problem that the Vcs-Git metadata is in the wrong place
(like the Maintainer and Uploaders fields are) with complicated
implications in the presence of skew.  (Some of these you mention.)


> If you want to contribute to future development in Debian and make
> things easy for the maintainer, it seems like this is the best tool we
> have today.

That depends, I think, on whether you are trying to (a) join a
maintainer team, or do "maintainer-like" work; or, (b) do
cross-archive work with small changes to many packages, or
archive-wide testing, or similar.

Any case where the change you want to make is simple, but you want to
make it to many packages, is much easier to achieve with a dgit-based
NMU workflow.

RC bugfixing and other kinds of "passer-by" QA work is usually easier
with dgit too.  You only need to understand the bug, and the relevant
parts of the package, not hold some git workflow stuff in your head
too.

Also: if all you are planning to do is prepare and test a patch, then
usually the easiest workflow is dgit clone followed by some
git-format-patch.  I have submitted many patches this way and no-one
has seemed to think them inconvenient.


> Advantages:
> 
> * You get to see things in the maintainer's work flow.  You can submit
>   patches in the form most familiar to the maintainer.

You say "submit patches".  Maybe you mean "submit changes" ?

If you intend to submit *patches*, I think git-format-patch on a
dgitish git branch will do something that is (1) very standard and
expected by a maintainer and (2) easy for the maintainer to
incorporate.

However, what you cannot do without grappling with the maintainer's
workflow is submit a salsa git merge request.  I think it would be
possible to improve this - see below.

> * You get to see ongoing development that may not have made it into
>   Debian yet.

This may or may not be an advantage, depending on what you are trying
to do.

If we had the right metadata, we could automatically synthesize a dgit
view from the salsa maintainer branch, as desired.  But what you would
do with that next is not 100% clear to me...

> * You may see branches that have never been (and possibly never will be)
>   in Debian.

This is unrelated to using the debcheckout branch vs using the dgit
clone branch.  dgit makes access to the debcheckout history easy.
So this is nothing to do with `dgit clone' vs debcheckout.


> Compare and contrast with dgit clone.  Dgit always (assuming there are
...
> * Regardless of what maintainer work flow and data model are used, you
>   get a consistent way of interacting with the sources.

You missed:

Regardless of what maintainer work flow and data model are used, you
can prepare completely competent
  - NMU
  - QA upload
  - patches for sending to the BTS
without interacting with the maintainer workflow, with dgit clone.

> * May give you history sufficient for viewing without you needing to
>   understand how the maintainer works with git.  Interacting with those
>   commits may sometimes require knowledge of work flows other than
>   dgit's native work flow/data model.  Example: you can probably review
>   the history enough to read a patch.  But cherry-picking patches (or
>   reverting) from maintainer view commits may sometimes require
>   knowledge beyond standard git knowledge.

Whether maintainer view commits can be "git-revert"'d or
"git-cherry-pick"'d does indeed depend on the maintainer's workflow.

I don't think this problem is avoidable, other than by forbidding
workflows which do not have whatever properties are desirable.

> * When the dgit view is not the same as the maintainer view,
>   contributing merge requests requires knowledge on your part or that of
>   the maintainer.  You always have the bts and raw patches as a
>   fallback, but it is often easier for maintainers to process things
>   closer to their workflow.

As a matter of equity, I think it should be up to the maintainer to
deal with their workflow.  They chose it.

Unfortunately we lack a standardised way for a submitter using a dgit
view to present a branch with their proposed changes.

If we did, then the maintainer could use manual processing, or
tooling, to integrate the patches in whatever way they prefer.

To do this on salsa might need some special salsa behaviours.

We would of course need a way to turn "sane" (dgit view) git 

The Difference between debcheckout and dgit and what they try to accomplish

[Sam Hartman]

I'm trying to organize my thoughts leading towards a discussion about
git on salsa.

Last month we had a wide ranging discussion that started from a
discussion of preferred formats for git branch structure.
During that discussion, we explored the differences in what a tool like
debcheckout gives you vs a tool like dgit.

Here's my understanding of those differences.  I'd appreciate comments
on whether I've accurately understood things as well as more general
comments.


Debcheckout allows you to get the latest version of the maintainer's
preferred branch.

If you want to contribute to future development in Debian and make
things easy for the maintainer, it seems like this is the best tool we
have today.

Advantages:

* You get to see things in the maintainer's work flow.  You can submit
  patches in the form most familiar to the maintainer.

* You get to see ongoing development that may not have made it into
  Debian yet.

* You may see branches that have never been (and possibly never will be)
  in Debian.

Disadvantages when contributing to future Debian development:

* May not work at all or may provide out of date information if the
  vcs-git is outdated etc.

* We have many different git work flows and data models.  You may not be
  familiar with or like the one the maintainer uses.

* It may not be obvious how to build the sources.


Additional Disadvantages for other use cases:

* It may be challenging to find the source corresponding to binaries you
  have



Compare and contrast with dgit clone.  Dgit always (assuming there are
no dscs it cannot process) will give you a uniform git view of the
package in the Debian distribution of your choice.  It may give you
history, which may be more or less complete, depending on when and how
often dgit push has been used.
It is well defined how to build a dgit view of a source package.

Advantages:

* Lets you as a downstream or user get the source to modify the binaries
  you have so you can make changes.

* Regardless of what maintainer work flow and data model are used, you
  get a consistent way of interacting with the sources.

* May give you history sufficient for viewing without you needing to
  understand how the maintainer works with git.  Interacting with those
  commits may sometimes require knowledge of work flows other than
  dgit's native work flow/data model.  Example: you can probably review
  the history enough to read a patch.  But cherry-picking patches (or
  reverting) from maintainer view commits may sometimes require
  knowledge beyond standard git knowledge.

Disadvantages for Debian Development:

* Only gives you code that has actually been uploaded to the archive.

* When the dgit view is not the same as the maintainer view,
  contributing merge requests requires knowledge on your part or that of
  the maintainer.  You always have the bts and raw patches as a
  fallback, but it is often easier for maintainers to process things
  closer to their workflow.

* History diverges from maintainer's history if dgit push is never or
  not always used.

Conclusions
===


I think that for the discussion I'm hoping to start soon, we're focused
on development within Debian.  That is, we do care about code not yet
released to the archive.  We do care about getting future changes
integrated, possibly even with little action on the part of the
maintainer.

That is, for the salsa discussion coming out of the DPL campaign, we're
focused more on those use cases than on finding the source to a given
binary.  And we explicitly do care about how proposed changes get into
the git data model/workflow that the maintainer uses.

I think the question of finding sources for binaries and of users having
a uniform approach for making changes is valuable.  It's just not the
focus of the discussion I said I would have during the DPL campaign.
I'm not even sure which is more valuable: I don't think all  the people
involved in the debian-vote discussion earlier this year understood the
difference adequately.

What I do know is that I think I'm already worried about whether the
discussion is too big.  I'm going to try and focus on what I said I'd
focus on because that's valuable.  There are other things that are also
valuable; by trying to narrow my own thinking in this instance I'm
trying not to make a judgment about them.