Bug#1007717: Draft resolution for "Native source package format with non-native version"
Hello, At today's ctte meeting we considered whether we can start a vote on this, but two committee members were missing, and someone else at the meeting reported that they hadn't yet been able to spend enough time thinking through the issue to be ready to vote. We came up with the following plan. Below I've drafted a ballot. Once each of those three individuals has let me know that they've had a chance to catch up, I'll start a vote. The hope is that this can happen well in advance of our next meeting. So, ctte members, if I don't already know that you're caught up, please write to me once you are. ~ DRAFT Using its powers under constitution 6.1.5, the Technical Committee issues the following advice: 1. It is not a bug of any severity for a package with a non-native version number to use a native source package format. 2. Thus, we think that dpkg shouldn't issue warnings, or otherwise complain, when a non-native version number is used w/ 3.0 (native). 3. We suggest that the wontfix tag on #737634 be reconsidered. 4. We believe that there are indeed circumstances in which 1.0-with-diff is the best choice for a particular source package, including, but not limited to, git-first packaging workflows. 5. We decline to comment on the recent source package format MBF. Option A -- issue items 1--5 Option B -- issue items 1, 2, 3 and 5, but not 4 Option N -- none of the above. END DRAFT -- Sean Whitton signature.asc Description: PGP signature
Bug#1007717: Draft resolution for "Native source package format with non-native version"
On 10/05/22 at 17:29 -0700, Sean Whitton wrote: > Hello, > > At today's ctte meeting we considered whether we can start a vote on > this, but two committee members were missing, and someone else at the > meeting reported that they hadn't yet been able to spend enough time > thinking through the issue to be ready to vote. > > We came up with the following plan. Below I've drafted a ballot. Once > each of those three individuals has let me know that they've had a > chance to catch up, I'll start a vote. The hope is that this can happen > well in advance of our next meeting. So, ctte members, if I don't > already know that you're caught up, please write to me once you are. > > ~ > > DRAFT > > Using its powers under constitution 6.1.5, the Technical Committee > issues the following advice: > > 1. It is not a bug of any severity for a package with a non-native > version number to use a native source package format. > > 2. Thus, we think that dpkg shouldn't issue warnings, or otherwise > complain, when a non-native version number is used w/ 3.0 (native). > > 3. We suggest that the wontfix tag on #737634 be reconsidered. > > 4. We believe that there are indeed circumstances in which > 1.0-with-diff is the best choice for a particular source package, > including, but not limited to, git-first packaging workflows. > > 5. We decline to comment on the recent source package format MBF. > > Option A -- issue items 1--5 > > Option B -- issue items 1, 2, 3 and 5, but not 4 > > Option N -- none of the above. > > END DRAFT Hi, If it was possible to use 3.0 (native) with non-native revisions, would there be remaining circumstances where 1.0-with-diff is the best choice? If not, maybe the fact that this is the blocking issue should be made explicit in (4)? That would be a way to state: "either dpkg maints refuses to support 3.0 (native) with non-native revs, and 1.0-with-diff must not be considered deprecated; or dpkg maints supports it, and it might be possible to deprecate 1.0". Lucas
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Lucas Nussbaum writes ("Re: Bug#1007717: Draft resolution for "Native source
package format with non-native version""):
> If it was possible to use 3.0 (native) with non-native revisions, would
> there be remaining circumstances where 1.0-with-diff is the best choice?
Yes, unfortunately.
If you have a package whose orig source code is large, and the delta
is representable with 1.0-with-diff (which is quite likely), then you
don't want to be reuploading the whole tarball each time. That's
wasteful of everyone's bandwidth and disk space.
So you want a representation that offers delta compression. That is
offered by the non-native formats. The non-native formats supported
by the archive are 1.0-with-diff and "3.0 (quilt)". The latter has
the problem with debian/patches/ living inside the source tree, which
is quite undesirable especially for "git-first workflows" (as the
draft text puts it).
> If not, maybe the fact that this is the blocking issue should be made
> explicit in (4)?
4. We believe that there are indeed circumstances in which
1.0-with-diff is the best choice for a particular source package,
including, but not limited to, git-first packaging workflows.
+This is because there is currently no other source format which avoids
+reuploading the whole source (including upstream) for each upload,
+and also avoids having to maintain debian/patches/ inside the
+package tree.
Or something.
> That would be a way to state: "either dpkg maints refuses to support 3.0
> (native) with non-native revs, and 1.0-with-diff must not be considered
> deprecated; or dpkg maints supports it, and it might be possible to
> deprecate 1.0".
I would love for there to be something like 3.0-with-git-diff. Indeed,
I filed this wishlist bug to ask if contribution would be welcome:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1007781
but have not had any response.
The code in dpkg-source for 1.0-with-diff is quite crusty particularly
in respect of the implied behaviour wrt scanning your ".." for stuff.
The *format* of 1.0-with-diff is quite reasonable, but it lacks
support more kinds of delta. That could be done as an extension to
1.0-with-diff, but I doubt that would be a popular direction.
Ian.
--
Ian JacksonThese opinions are my own.
Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Thanks for your answer. On 11/05/22 at 12:38 +0100, Ian Jackson wrote: > I would love for there to be something like 3.0-with-git-diff. Indeed, > I filed this wishlist bug to ask if contribution would be welcome: > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1007781 > but have not had any response. > > The code in dpkg-source for 1.0-with-diff is quite crusty particularly > in respect of the implied behaviour wrt scanning your ".." for stuff. > The *format* of 1.0-with-diff is quite reasonable, but it lacks > support more kinds of delta. That could be done as an extension to > 1.0-with-diff, but I doubt that would be a popular direction. Out of curiosity, if 3.0 (native) supported multiple tarballs, wouldn't it be a good solution? The main limitation I see is that it would not allow to represent efficiently small changes to large text files (which a git-based diff would allow). I'm asking because if 3.0 (native) gets more generic by allowing non-native revisions, it might be an easier sell to introduce multi-tarballs support, than to introduce a completely different source format. Lucas
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Lucas Nussbaum writes ("Re: Bug#1007717: Draft resolution for "Native source
package format with non-native version""):
> Out of curiosity, if 3.0 (native) supported multiple tarballs, wouldn't
> it be a good solution?
Oh, I hadn't thought of that.
> The main limitation I see is that it would not allow to represent
> efficiently small changes to large text files (which a git-based diff
> would allow).
That may not be important. My feeling is that it wouldn't be.
However, I think there are some other difficulties with this idea.
*Deletion* of files *is* important. Something would have to be done
to support that. (Tarball repacking is an abominable workflow which
we should only do when we must.)
It is important that packing and unpacking these things works roughly
the same way that things work right now for the diffish formats. Ie,
for a package with two tarballs, the first tarball would have to omit
the Debian revision from its filename (so that it needn't be
re-uploaded), and the second tarball would have to *contain* it.
dpkg-source -b would have to calculate the appropriate second tarball
as a diff from the first. (GNU tar has an incremental option that
could perhaps be used.)
I think, having followed this line of throught, the result looks quite
like a "3.0 (diff)" only the diff is in the form of an incremental
tarball rather than a textual patch file.
This could definitely work.
But I think that might not meet ftpmaster's review needs. AIUI
ftpmaster like to review the diff qua diff, and a tarball isn't so
straightforward. I had a similar idea to use an rsync batchfile as
the delta, which foundered on the same issue.
And I'm not sure that it will find better favour with dpkg upstream
than my "3.0 (git-diff)".
> I'm asking because if 3.0 (native) gets more generic by allowing
> non-native revisions, it might be an easier sell to introduce
> multi-tarballs support, than to introduce a completely different source
> format.
Mmmm. I think there are many possibilities here which would suffice.
Right now, though, it's a bit hard to make progress without feedback
on what general direction would be most well received.
Ian.
--
Ian JacksonThese opinions are my own.
Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.
Bug#1007717: Draft resolution for "Native source package format with non-native version"
On 11/05/22 at 17:29 +0100, Ian Jackson wrote:
> Lucas Nussbaum writes ("Re: Bug#1007717: Draft resolution for "Native source
> package format with non-native version""):
> > Out of curiosity, if 3.0 (native) supported multiple tarballs, wouldn't
> > it be a good solution?
>
> Oh, I hadn't thought of that.
>
> > The main limitation I see is that it would not allow to represent
> > efficiently small changes to large text files (which a git-based diff
> > would allow).
>
> That may not be important. My feeling is that it wouldn't be.
>
> However, I think there are some other difficulties with this idea.
>
> *Deletion* of files *is* important. Something would have to be done
> to support that. (Tarball repacking is an abominable workflow which
> we should only do when we must.)
>
> It is important that packing and unpacking these things works roughly
> the same way that things work right now for the diffish formats. Ie,
> for a package with two tarballs, the first tarball would have to omit
> the Debian revision from its filename (so that it needn't be
> re-uploaded), and the second tarball would have to *contain* it.
> dpkg-source -b would have to calculate the appropriate second tarball
> as a diff from the first. (GNU tar has an incremental option that
> could perhaps be used.)
>
> I think, having followed this line of throught, the result looks quite
> like a "3.0 (diff)" only the diff is in the form of an incremental
> tarball rather than a textual patch file.
>
> This could definitely work.
FWIW, GNU tar's incremental mode supports file deletions.
Example:
-->8
# create a first archive
mkdir t
echo foo > t/f1
echo bar > t/f2
echo baz > t/f3
tar --listed-increment=snapshot-file -czvvvf archive.tgz t
# remove a file, create an incremental archive
rm t/f2
tar --listed-increment=snapshot-file -czvvvf archive.1.tgz t
# remove everything
rm -rf snapshot-file t
# extract incremental archives
tar Gvvvxf archive.tgz
tar Gvvvxf archive.1.tgz
# listing of archive.1.tgz
tar Gvvvtf archive.1.tgz
-->8
This is implemented using the GNU-specific D entry type. According to
https://www.freebsd.org/cgi/man.cgi?query=tar&apropos=0&sektion=5&format=html :
D This indicates a directory entry. Unlike the POSIX-stan-
dard "5" typeflag, the header is followed by data records
listing the names of files in this directory. Each name
is preceded by an ASCII "Y" if the file is stored in this
archive or "N" if the file is not stored in this archive.
Each name is terminated with a null, and an extra null
marks the end of the name list. The purpose of this en-
try is to support incremental backups; a program restor-
ing from such an archive may wish to delete files on disk
that did not exist in the directory when the archive was
made.
Note that the "D" typeflag specifically violates POSIX,
which requires that unrecognized typeflags be restored as
normal files. In this case, restoring the "D" entry as a
file could interfere with subsequent creation of the
like-named directory.
> But I think that might not meet ftpmaster's review needs. AIUI
> ftpmaster like to review the diff qua diff, and a tarball isn't so
> straightforward. I had a similar idea to use an rsync batchfile as
> the delta, which foundered on the same issue.
Note that it's not worse than using native formats, where ftpmasters get
a single tarball anyway...
Lucas
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Lucas Nussbaum writes ("Re: Bug#1007717: Draft resolution for "Native source
package format with non-native version""):
> On 11/05/22 at 17:29 +0100, Ian Jackson wrote:
> > But I think that might not meet ftpmaster's review needs. AIUI
> > ftpmaster like to review the diff qua diff, and a tarball isn't so
> > straightforward. I had a similar idea to use an rsync batchfile as
> > the delta, which foundered on the same issue.
>
> Note that it's not worse than using native formats, where ftpmasters get
> a single tarball anyway...
Indeed. But I don't necessarily expect to be able to predict what
ftpmaster (or, for that matter, the dpkg maintainers) will like.
Thanks for exploring the design space! What would be really good
would be to get agreement in principle from the key stakeholders...
Ian.
--
Ian JacksonThese opinions are my own.
Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Hallo, On Tue, May 10, 2022 at 05:29:57PM -0700, Sean Whitton wrote: > DRAFT > > Using its powers under constitution 6.1.5, the Technical Committee > issues the following advice: I've given this some thought and feel uneasy about one item. > 4. We believe that there are indeed circumstances in which > 1.0-with-diff is the best choice for a particular source package, > including, but not limited to, git-first packaging workflows. While I can agree with this item on a technical level, I think there is more to it than that and I am wondering whether it sends the "right" message. Sometimes, things we do are technically possible and fill a niche well. Yet, we decide that it is no longer reasonable to continue supporting them and remove their support despite the feature being useful to some. Quite clearly, there is a trade-off involved here. Continuing to support 1.0-with-diff comes with a cost that reduces uniformity inside the archive. Evidently, this is what motivated Lucas to file the MBF initially. My experience is that lack of uniformity is a significant barrier to prospective contributors to Debian. Exploring different technical approaches does have value as well, but I think we've had sufficient time to consider the various advantages and disadvantages of various source packages formats. On a whole, it seems to me that that the number of packages benefiting from 1.0-with-diff is relatively small. What would you think about adding an alternative option 4? 4b. We believe that there are indeed circumstances in which 1.0-with-diff is the best choice for a particular source package. Given that the number of packages for which this is relevant is fairly small, we recommend discontinuing use of 1.0-with-diff to gain more uniformity. Helmut
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Hello, On Tue 07 Jun 2022 at 07:43am +02, Helmut Grohne wrote: > While I can agree with this item on a technical level, I think there is > more to it than that and I am wondering whether it sends the "right" > message. > > Sometimes, things we do are technically possible and fill a niche well. > Yet, we decide that it is no longer reasonable to continue supporting > them and remove their support despite the feature being useful to some. > Quite clearly, there is a trade-off involved here. Continuing to support > 1.0-with-diff comes with a cost that reduces uniformity inside the > archive. Evidently, this is what motivated Lucas to file the MBF > initially. My experience is that lack of uniformity is a significant > barrier to prospective contributors to Debian. I think this argument needs to be made more precise -- we should be clearer about why this particular un-uniformity is bad. I don't think the issue for new contributors is persuasive enough, as new contributors can mostly ignore source packages. It's not like, e.g., debhelper vs. cdbs. I haven't yet seen an argument that the lack of uniformity is doing anyone's work any harm comparable to the harm done to things like what Ian and Sam want to do. > Exploring different technical approaches does have value as well, but I > think we've had sufficient time to consider the various advantages and > disadvantages of various source packages formats. On a whole, it seems > to me that that the number of packages benefiting from 1.0-with-diff is > relatively small. I agree, it's not about the benefits of the source format, we do indeed understand all the trade-offs by now. It's that certain ideas and workflows *which are not really about source packages* are made inconvenient or impossible if we remove this option. In other words, it needs to be replaced before it can be deprecated. > What would you think about adding an alternative option 4? > > 4b. We believe that there are indeed circumstances in which > 1.0-with-diff is the best choice for a particular source package. > Given that the number of packages for which this is relevant is > fairly small, we recommend discontinuing use of 1.0-with-diff to > gain more uniformity. Thanks for coming up with the text. I'd say that as uniformity is not good in itself, it would be good to have more concrete reasons for wanting uniformity in this case documented in this bug (not necessarily in the resolution text) before we add it to the ballot. -- Sean Whitton
Bug#1007717: Draft resolution for "Native source package format with non-native version"
On 07/06/22 at 07:43 +0200, Helmut Grohne wrote: > Hallo, > > On Tue, May 10, 2022 at 05:29:57PM -0700, Sean Whitton wrote: > > DRAFT > > > > Using its powers under constitution 6.1.5, the Technical Committee > > issues the following advice: > > I've given this some thought and feel uneasy about one item. > > > 4. We believe that there are indeed circumstances in which > > 1.0-with-diff is the best choice for a particular source package, > > including, but not limited to, git-first packaging workflows. > > While I can agree with this item on a technical level, I think there is > more to it than that and I am wondering whether it sends the "right" > message. > > Sometimes, things we do are technically possible and fill a niche well. > Yet, we decide that it is no longer reasonable to continue supporting > them and remove their support despite the feature being useful to some. > Quite clearly, there is a trade-off involved here. Continuing to support > 1.0-with-diff comes with a cost that reduces uniformity inside the > archive. Evidently, this is what motivated Lucas to file the MBF > initially. My experience is that lack of uniformity is a significant > barrier to prospective contributors to Debian. > > Exploring different technical approaches does have value as well, but I > think we've had sufficient time to consider the various advantages and > disadvantages of various source packages formats. On a whole, it seems > to me that that the number of packages benefiting from 1.0-with-diff is > relatively small. > > What would you think about adding an alternative option 4? > > 4b. We believe that there are indeed circumstances in which > 1.0-with-diff is the best choice for a particular source package. > Given that the number of packages for which this is relevant is > fairly small, we recommend discontinuing use of 1.0-with-diff to > gain more uniformity. Hi, A middle ground between (4) and (4b) could be to discourage the use of 1.0-with-diff in circumstances where it is not justified. Something like: 4c. We believe that there are indeed circumstances in which 1.0-with-diff is the best choice for a particular source package, including, but not limited to, git-first packaging workflows. However, we recommend discontinuing use of 1.0-with-diff in other circumstances to gain more uniformity. That opens the path to an archive where the only remaining 1.0 packages are the ones where there's a reason to use 1.0. (as opposed to currently, where we have a mix of packages using 1.0 for a good reason, and packages using 1.0 because nobody cared to update them to modern practices). Lucas
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Re: Lucas Nussbaum > A middle ground between (4) and (4b) could be to discourage the use of > 1.0-with-diff in circumstances where it is not justified. Something > like: > > 4c. We believe that there are indeed circumstances in which > 1.0-with-diff is the best choice for a particular source package, > including, but not limited to, git-first packaging workflows. > However, we recommend discontinuing use of 1.0-with-diff in other > circumstances to gain more uniformity. > > That opens the path to an archive where the only remaining 1.0 packages > are the ones where there's a reason to use 1.0. (as opposed to > currently, where we have a mix of packages using 1.0 for a good reason, > and packages using 1.0 because nobody cared to update them to modern > practices). The bit I was missing is something like "we would welcome changes to the 3.0 format to make it usable for the remaining cases where 1.0 is still better today". Did anyone investigate if that would be feasible? Christoph
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Helmut Grohne writes ("Re: Bug#1007717: Draft resolution for "Native source
package format with non-native version""):
> What would you think about adding an alternative option 4?
>
> 4b. We believe that there are indeed circumstances in which
> 1.0-with-diff is the best choice for a particular source package.
> Given that the number of packages for which this is relevant is
> fairly small, we recommend discontinuing use of 1.0-with-diff to
> gain more uniformity.
It seems to me that the lack of uniformity (and indeed technical
superiority) here is due entirely due to our inability to move forward
with a replacement for 1.0-with-diff that is actually superior for a
wider range of use cases.
I know that the TC has a history of favouring what looks like
"progress" over other factors (such as contributor happiness, software
diversity, and indeed inconvenient details).
In this case I would like to suggest that progress would be better
served by trying to unblock a better source format that (i) has some
kind of delta representation (ii) does not put a
needing-to-be-maintained copy of the delta metadata inside the working
tree.
There are several proposals for how to do that which are obviously
possible to implement. If the TC wants to unblock that, you could
look at them and pick one.
(And, I quextion whether .dsc format is the right place for Debian to
pursue uniformity. .dsc is a legacy format we are maintaining because
our git transition is stalled.)
Ian.
--
Ian JacksonThese opinions are my own.
Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.
Bug#1007717: Draft resolution for "Native source package format with non-native version"
On 07/06/2022 07:08, Sean Whitton wrote: I agree, it's not about the benefits of the source format, we do indeed understand all the trade-offs by now. It's that certain ideas and workflows *which are not really about source packages* are made inconvenient or impossible if we remove this option. In other words, it needs to be replaced before it can be deprecated. Mmm. And I'd be pretty reluctant to talk about deprecation until the replacement is good to go. What would you think about adding an alternative option 4? 4b. We believe that there are indeed circumstances in which 1.0-with-diff is the best choice for a particular source package. Given that the number of packages for which this is relevant is fairly small, we recommend discontinuing use of 1.0-with-diff to gain more uniformity. Thanks for coming up with the text. I'd say that as uniformity is not good in itself, it would be good to have more concrete reasons for wanting uniformity in this case documented in this bug (not necessarily in the resolution text) before we add it to the ballot. I would certainly vote against 4b as drafted; I would need considerable persuasion that "more uniformity" here is a concrete benefit. Regards, Matthew
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Christoph Berg writes ("Re: Bug#1007717: Draft resolution for "Native source
package format with non-native version""):
> Re: Lucas Nussbaum
> > 4c. We believe that there are indeed circumstances in which
> > 1.0-with-diff is the best choice for a particular source package,
> > including, but not limited to, git-first packaging workflows.
> > However, we recommend discontinuing use of 1.0-with-diff in other
> > circumstances to gain more uniformity.
...
> The bit I was missing is something like "we would welcome changes to
> the 3.0 format to make it usable for the remaining cases where 1.0 is
> still better today". Did anyone investigate if that would be feasible?
A format that solves this problem well is entirely feasible on a
technical level, and not even particularly difficult.
Contenders for the details that have been mentioned in this thread[1]
are 3.0-with-git-diff (#1007781) and Lucas's suggestion for
incremental tarballs in this conversation
(https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1007717#185).
That could certainly fit within "changes to 3.0" since 3.0 already
encompasses a very wide variety of formats.
The real diifficulty isn't technical. It's that it isn't worth
anyone's time *implementing* it because of the strong possibility that
its support and deployment would be blocked.
Perhaps TC would be willing to say something like this
We hope for the development of a version of the 3.0 format which is
usable for the remaining cases where 1.0 is still better today.
The "3.0 with git diff" proposal in #1007781 seems to us to be a
good option, and we encourage its implementation. If the
implementation quality is good, we feel it should be supported
within Debian (subject to an appropriate transition plan).
I think 3.0-with-git-diff is the most promising candidate because it
fits into the existing ftpmaster workflow exactly as 1.0-with-diff
does now; this isn't so true of Lucas's incremental tarballs, and it
is even less true of my earlier 3.0-with-rsync-delta.
If the TC wants to give its blessing to one of the other format
proposals then that would be fine too from my point of view, but my
personal feeling is that it would be setting the TC up for a fight
with ftpmaster. From what I know of the ftpmaster workflow I think
even Lucas's incremental tarball proposal would be a retrograde step
for them.
Ian.
--
Ian JacksonThese opinions are my own.
Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Hi Sean, On Mon, Jun 06, 2022 at 11:08:48PM -0700, Sean Whitton wrote: > I think this argument needs to be made more precise -- we should be > clearer about why this particular un-uniformity is bad. I don't think > the issue for new contributors is persuasive enough, as new contributors > can mostly ignore source packages. It's not like, e.g., debhelper > vs. cdbs. I haven't yet seen an argument that the lack of uniformity is > doing anyone's work any harm comparable to the harm done to things like > what Ian and Sam want to do. It seems to me that Ian et al works from the premise that source packages are an export format and that git is the truth. In a sense, that is correct as that's what most maintainers work with. Unfortunately, packaging repositories have very diverse workflows, lack discoverability and miss a simple trust root. A number of people have tried to reconcile the various workflows, but that helps only so much. On the other hand, source packages offer a very simple trust root and uniform interface. If I am to modify an arbitrary package (with no prior knowledge about it), I have some options to go about it. I can use debcheckout. For a few packages, there is no repository declared. Yet others point to non-existent servers. Still others point to an obsolete repository that has been moved. After checking it out, I have a git tree of something. It may come without upstream sources (e.g. DHG) or with upstream sources. Patches may be applied or may be not. It may be for the version I'm looking at or including unreleased changes or pointing at some other suite. The representation of patches in git is another story. The source repository may provide the ability to create pull requests. Maybe the maintainer looks at them or maybe I'll have to send a mail to the bts. I have no chance of figuring out this mess without investing significant amount of time. Then I can use dgit. It is way more uniform in a lot of aspects. Its trust root resides with the CA cartel. The resulting tree may or may not be representing the version I'm interested in. The history may be useful or autogenerated for my checkout as initial checkin. The Debian maintainer may be completely ignoring this representation. Most likely, I won't be able to send a pull request. And then, I can apt source the package. This is really simple and has an easy to understand trust root. It also is reproducible. Unless using a 1.0 source format, debian/source/format tells me what kind of patch system to expect and gladly there aren't too many options. I can concentrate on the substance of my change without being distracted by the packaging workflow details. Please keep in mind that this is about trade-offs. It is a question of how we value "package ownership". If we favour the strong ownership approach that Debian used for a long time, then yes accommodating the needs of maintainers is paramount. If we want to lessen the concept of ownership, embrace drive-by contributions and decentralize maintenance, then we need a stronger focus on uniformity. I suppose that my own opinion on this matter is fairly obvious at this point. Annoyingly enough, 3.0 (quilt) can be abused as well by deleting the series file and managing patch application inside debian/rules. We do have packages doing just that. > I agree, it's not about the benefits of the source format, we do indeed > understand all the trade-offs by now. It's that certain ideas and > workflows *which are not really about source packages* are made > inconvenient or impossible if we remove this option. In other words, it > needs to be replaced before it can be deprecated. I admit being somewhat ignorant towards workflows, because there are just too many for me to be bothered to learn them all. In any case, I do take issue with the premise that workflows need to continue to work. We're deleting packages from unstable and stop supporting unique use cases all the time. In most cases, we could continue supporting them for longer, but we choose not to. I hope we all agree that there are too many competing workflows. How would you go about reducing them to a sane number? > Thanks for coming up with the text. I'd say that as uniformity is not > good in itself, it would be good to have more concrete reasons for > wanting uniformity in this case documented in this bug (not necessarily > in the resolution text) before we add it to the ballot. You can rephrase it as reducing complexity if that helps. It's not the one additional source package format that is too much. It's that we continue using all the failed experiments while adding new ones and when some Lucas comes around and tries to clean up the mess he gets referred to us. I have anecdotal evidence that people find Debian's processes too complex. Unless you closely work withing a particular packaging team (with unified processes), onboarding people into Debian takes very long compared to other projects. Helmut
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Hello, On Tue 07 Jun 2022 at 09:31am +02, Lucas Nussbaum wrote: > A middle ground between (4) and (4b) could be to discourage the use of > 1.0-with-diff in circumstances where it is not justified. Something > like: > > 4c. We believe that there are indeed circumstances in which > 1.0-with-diff is the best choice for a particular source package, > including, but not limited to, git-first packaging workflows. > However, we recommend discontinuing use of 1.0-with-diff in other > circumstances to gain more uniformity. > > That opens the path to an archive where the only remaining 1.0 packages > are the ones where there's a reason to use 1.0. (as opposed to > currently, where we have a mix of packages using 1.0 for a good reason, > and packages using 1.0 because nobody cared to update them to modern > practices). This would be a good ballot option to include, thank you. -- Sean Whitton
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Hello, On Tue 07 Jun 2022 at 08:19pm +02, Helmut Grohne wrote: > Please keep in mind that this is about trade-offs. It is a question of > how we value "package ownership". If we favour the strong ownership > approach that Debian used for a long time, then yes accommodating the > needs of maintainers is paramount. If we want to lessen the concept of > ownership, embrace drive-by contributions and decentralize maintenance, > then we need a stronger focus on uniformity. I suppose that my own > opinion on this matter is fairly obvious at this point. I disagree with you that this is primarily about package ownership, and I think that we agree on more than you realise we do :) The git-first people are not making a trade-off between the maintainer's convenience and the convenience of others who want to work with the package. The most important reason for wanting both (i) git-first workflows and (ii) uploads done with dgit, is to make it easier for people other than the maintainer to work with the package. So, your priorities are quite in agreement with those of Ian, Sam, Russ and I. In other words, I would like to make weaker package ownership more possible, in a project with Debian's history, by improving our tools for dealing with packages for which you're not primary maintainer. What we disagree about is the extent to which the current tooling amounts to a failed experiment, such that we should give up on it and fall back to 'apt-get source'. Many people strongly prefer 'dgit clone' to 'apt-get source' already, and the number of people switching to upload with 'dgit [--gbp] push-source' is steadily rising. Against this backdrop, some of us are interested in git-first workflows for reducing the distance between the output of 'debcheckout' and 'dgit clone'. It's completely reasonable that you're more sceptical about the prospects of solving the outstanding issues in this programme than others are, but surely we can agree that this is an ongoing piece of work rather than something we're sure we can reject? And if keeping an old source package format around is needed for that work to continue, then that's a compelling reason to keep it around. > How would you go about reducing them to a sane number? The goal is to attack this problem on two fronts: 1. reduce the need for uniformity by making it possible for people to get their cross-archive work done using 'dgit clone' 2. develop git-first workflows that solve all the existing usecases. > You can rephrase it as reducing complexity if that helps. It's not the > one additional source package format that is too much. It's that we > continue using all the failed experiments while adding new ones and when > some Lucas comes around and tries to clean up the mess he gets referred > to us. As I said, I don't think it's fair to characterise the git-first work as a failed experiment. It's ongoing work. And the source package format is just a way to keep it going, at the present moment. > I have anecdotal evidence that people find Debian's processes too > complex. Unless you closely work withing a particular packaging team > (with unified processes), onboarding people into Debian takes very long > compared to other projects. Right, hence why we want 'dgit clone' to be as useful as possible. -- Sean Whitton
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Hello, On Tue 07 Jun 2022 at 11:26am +01, Ian Jackson wrote: > In this case I would like to suggest that progress would be better > served by trying to unblock a better source format that (i) has some > kind of delta representation (ii) does not put a > needing-to-be-maintained copy of the delta metadata inside the working > tree. > > There are several proposals for how to do that which are obviously > possible to implement. If the TC wants to unblock that, you could > look at them and pick one. > > (And, I quextion whether .dsc format is the right place for Debian to > pursue uniformity. .dsc is a legacy format we are maintaining because > our git transition is stalled.) I think that this would be out-of-scope for this bug as presented to us: we'd need someone to present those options for us to choose between. Let's stick with the current resolution text for a bug that's probably been open too long already. -- Sean Whitton
Bug#1007717: Draft resolution for "Native source package format with non-native version"
On Tue, Jun 07, 2022 at 08:19:29PM +0200, Helmut Grohne wrote: > Please keep in mind that this is about trade-offs. It is a question of > how we value "package ownership". If we favour the strong ownership > approach that Debian used for a long time, then yes accommodating the > needs of maintainers is paramount. If we want to lessen the concept of > ownership, embrace drive-by contributions and decentralize maintenance, > then we need a stronger focus on uniformity. I suppose that my own > opinion on this matter is fairly obvious at this point. This is also a significant issue for downstreams. With my Ubuntu hat on, if I have to touch packages downstream, I loathe it everytime I get a non-descript blob of all the changes. I suspect this is the same for other downstream distributions that want to modify packages. We cannot cater to everyone's individual packaging repository approach, and downstream repositories if they exist are separate anyhow. Using 1.0 instead of 3.0 (quilt) is just being a hostile upstream, like Red Hat is with dumping all their kernel patches together. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Hello, On Wed 08 Jun 2022 at 12:06pm +02, Julian Andres Klode wrote: > On Tue, Jun 07, 2022 at 08:19:29PM +0200, Helmut Grohne wrote: >> Please keep in mind that this is about trade-offs. It is a question of >> how we value "package ownership". If we favour the strong ownership >> approach that Debian used for a long time, then yes accommodating the >> needs of maintainers is paramount. If we want to lessen the concept of >> ownership, embrace drive-by contributions and decentralize maintenance, >> then we need a stronger focus on uniformity. I suppose that my own >> opinion on this matter is fairly obvious at this point. > > This is also a significant issue for downstreams. With my Ubuntu hat > on, if I have to touch packages downstream, I loathe it everytime I > get a non-descript blob of all the changes. This is not inherent to all of the workflows we are discussing here, just one of them. Indeed, this is one of the primary motivations for one of the others. -- Sean Whitton
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Hi Sean, On Tue, Jun 07, 2022 at 04:35:24PM -0700, Sean Whitton wrote: > I disagree with you that this is primarily about package ownership, and > I think that we agree on more than you realise we do :) Hmm. It's not that obvious. While it would be possible to remove the choice of workflow from strong package ownership, the way we practice package ownership presently grants that freedom. Therefore I think they still are closely related. > The git-first people are not making a trade-off between the maintainer's > convenience and the convenience of others who want to work with the > package. The most important reason for wanting both (i) git-first > workflows and (ii) uploads done with dgit, is to make it easier for > people other than the maintainer to work with the package. So, your > priorities are quite in agreement with those of Ian, Sam, Russ and I. I find it interesting that you seem to equate git-first with dgit. My mental model separated those concepts and considered git-first workflows on salsa as well. And once you equate them, you can derive a lot of conclusions. In my previous mail, I stated that dgit provides the sort of uniformity I was looking for quite many aspects. I'm less sure that all git-first users are dgit users though. > In other words, I would like to make weaker package ownership more > possible, in a project with Debian's history, by improving our tools for > dealing with packages for which you're not primary maintainer. I do see how this is a dgit goal. It just seems to me that dgit bolts a secondary workflow onto packages that maintainers are free to ignore entirely. Some choose to use that dgit workflow exclusively and others don't. In a sense, the problem with dgit is that it leaves too much freedom to maintainers (and in a project like Debian, it really has to do exactly that or it would run into straight opposition). > What we disagree about is the extent to which the current tooling > amounts to a failed experiment, such that we should give up on it and > fall back to 'apt-get source'. Many people strongly prefer 'dgit clone' > to 'apt-get source' already, and the number of people switching to > upload with 'dgit [--gbp] push-source' is steadily rising. Against this > backdrop, some of us are interested in git-first workflows for reducing > the distance between the output of 'debcheckout' and 'dgit clone'. Indeed, dgit kinda faces a chicken & egg problem and it seems that it is quite good at solving it: The usefulness of dgit grows with its adoption. I have looked into replacing my apt source workflow with dgit clone a number of times already and will continue trying. > It's completely reasonable that you're more sceptical about the > prospects of solving the outstanding issues in this programme than > others are, but surely we can agree that this is an ongoing piece of > work rather than something we're sure we can reject? And if keeping an > old source package format around is needed for that work to continue, > then that's a compelling reason to keep it around. I think I take more issue with non-dgit git-first workflows than with dgit ones, because dgit is so well documented and is a workflow that is already shared by a noticeable fraction of the archive. What is not entirely clear to me is why dgit requires the 1.0-with-diff format features. It seems to me that it quite happily deals with a number of 3.0 (quilt) packages already. I suppose that you'll now explain to me how some git trees are not representable as 3.0 (quilt) packages, but do those exist in practice or are those mostly pathological? > > How would you go about reducing them to a sane number? > > The goal is to attack this problem on two fronts: > > 1. reduce the need for uniformity by making it possible for people to >get their cross-archive work done using 'dgit clone' I've seen this argument multiple times already and indeed dgit solves part of the uniformity issues. However, dgit history often lacks maintainer history (and in that way is little better than apt source) and it also lacks a collaboration feature that would save me from sending a .debdiff to the bts. Possibly, such a collaboration feature is out-of-scope for dgit, but then maybe it is not the kind of tool solving the problem of streamlining cross-archive work. Either way, my understanding is that unless maintainers switch to using dgit primarily, we just gain a secondary workflow here. > 2. develop git-first workflows that solve all the existing usecases. Can i rephrase that as follows? Implement so many features into dgit that its workflow covers the needs of existing workflows and hope for maintainers to migrate to dgit. It feels a bit like https://xkcd.com/927/ (14 competing standards -> 15). On the other hand, dgit is remarkably successful already. I had hoped that we could kinda trim the available workflows eventually. It seems like you want to let them all die slowly and concurrently. > > You can rephrase
Bug#1007717: Draft resolution for "Native source package format with non-native version"
On 08/06/22 at 21:07 +0200, Helmut Grohne wrote: > Now we've turned a discussion about source package formats into a > discussion about workflows and git. So when I reason about uniformity, I > effectively want those idiosyncratic workflows to go away. If dgit > requires 1.0-with-diff for now, then maybe we could phrase it as an > exception that is specific to dgit and limited until a better solution > (such as the format proposed by Ian) is mature. If there are more > git-first workflows beyond dgit that we want to support, maybe we can > require declaring a working Vcs-Git for 1.0-with-diff uploads? I think that the workflow used by the Debian X team is such a git-first workflow that is not using dgit. That workflow combines Debian-specific patches managed by quilt, and commits cherry-picked from upstream directly applied to the source in git (outside of quilt). See There are 89 packages maintained by Debian X among the 607 packages in testing still using 1.0. Lucas
Bug#1007717: Draft resolution for "Native source package format with non-native version"
On 08/06/22 at 21:07 +0200, Helmut Grohne wrote: > I think I take more issue with non-dgit git-first workflows than with > dgit ones, because dgit is so well documented and is a workflow that is > already shared by a noticeable fraction of the archive. I'm curious: how do you measure dgit usage? You cannot just look at the list of repositories at https://browse.dgit.debian.org/, since that just says that at some point in the past, someone used dgit to work on that package, right? For example, the keepass2 package has a repo there, but it is completely outdated compared to what is in unstable. Lucas
Bug#1007717: Draft resolution for "Native source package format with non-native version"
Hello, On Wed 08 Jun 2022 at 09:07pm +02, Helmut Grohne wrote: > I find it interesting that you seem to equate git-first with dgit. My > mental model separated those concepts and considered git-first workflows > on salsa as well. And once you equate them, you can derive a lot of > conclusions. In my previous mail, I stated that dgit provides the sort > of uniformity I was looking for quite many aspects. I'm less sure that > all git-first users are dgit users though. > > I think I take more issue with non-dgit git-first workflows than with > dgit ones, because dgit is so well documented and is a workflow that is > already shared by a noticeable fraction of the archive. Didn't mean to equate them, sorry. We can state the following: dgit should have support for all git-first workflows. I am not sure whether there are any git-first workflows in use on salsa for which you can't 'dgit push-source' atm. If there are, those are dgit bugs. > What is not entirely clear to me is why dgit requires the 1.0-with-diff > format features. It seems to me that it quite happily deals with a > number of 3.0 (quilt) packages already. I suppose that you'll now > explain to me how some git trees are not representable as 3.0 (quilt) > packages, but do those exist in practice or are those mostly > pathological? Sorry, didn't mean to suggest that dgit requires 1.0-with-diff. It does not. >> The goal is to attack this problem on two fronts: >> >> 1. reduce the need for uniformity by making it possible for people to >>get their cross-archive work done using 'dgit clone' > > I've seen this argument multiple times already and indeed dgit solves > part of the uniformity issues. However, dgit history often lacks > maintainer history (and in that way is little better than apt source) Right, (1) is about making it easy for people to be using 'dgit push-source' such that the maintainer history is there when you 'dgit clone'. > and it also lacks a collaboration feature that would save me from > sending a .debdiff to the bts. Possibly, such a collaboration feature > is out-of-scope for dgit, but then maybe it is not the kind of tool > solving the problem of streamlining cross-archive work. Not out of scope, we have some notes for 'dgit nmudiff' in the BTS. Would be cool to collaborate with you on finishing that up. > Either way, my understanding is that unless maintainers switch to > using dgit primarily, we just gain a secondary workflow here. No -- (1) is all about ensuring that maintainers don't have to change their workflows aside from how they do the actual upload. >> 2. develop git-first workflows that solve all the existing usecases. > > Can i rephrase that as follows? Implement so many features into dgit > that its workflow covers the needs of existing workflows and hope for > maintainers to migrate to dgit. > > It feels a bit like https://xkcd.com/927/ (14 competing standards -> > 15). On the other hand, dgit is remarkably successful already. I don't think it can be rephrased that way. Firstly, (1) is about dgit, but (2) is about tools like git-debrebase(1) and workflows such as dgit-maint-merge(7). More substantially, (1) is about attaching maintainer histories to the dgit view, and (2) is about developing workflows which enables the maintainer and dgit views to be identical. That's what I mean when I say that (1) and (2) are attacking the problem on two fronts. (2) is particularly important for submitting NMU changes to the maintainer -- it enables using salsa merge requests, for example. > I had hoped that we could kinda trim the available workflows eventually. > It seems like you want to let them all die slowly and concurrently. > > [...] > > I don't think there is "the git-first work". Instead there is lots of > concurrent git-first workflows that are all somewhat similar and yet > subtly different. Those subtle differences is what I would like to get > rid of. > > Now we've turned a discussion about source package formats into a > discussion about workflows and git. So when I reason about uniformity, I > effectively want those idiosyncratic workflows to go away. If dgit > requires 1.0-with-diff for now, then maybe we could phrase it as an > exception that is specific to dgit and limited until a better solution > (such as the format proposed by Ian) is mature. If there are more > git-first workflows beyond dgit that we want to support, maybe we can > require declaring a working Vcs-Git for 1.0-with-diff uploads? My own appraisal of the situation is that we do not know enough yet to be thinking about cutting back on features and workflows. But I certainly agree that it would be better to have a better solution, like Ian's sketch. I think Ian has already suggested adding text to our resolution recommending the development of such a source format. To be honest I don't think it's really necessary as everyone agrees it would be better to have it, but if you think it should be said explicitly then let's
