Re: OpenCL / handling of hardware-specific packages

[Rebecca N. Palmer]

On 01/07/2019 18:10, Enrico Weigelt, metux IT consult wrote:

On 19.06.19 09:09, Rebecca N. Palmer wrote:

I proposed [0] fixing this by creating a metapackage for "all OpenCL
drivers" (similar to the ones for graphics).  However, having unusable
OpenCL drivers installed can trigger bugs: [1] in llvm, and some
applications that treat "no hardware for this driver" as "fatal error"
instead of "try the next driver".


So, installing an opencl-based package pulls in *all* cl driver stacks ?


If we do the above, yes by default, but the user can prevent this by 
explicitly installing any one.




Please don't do that. This IMHO clearly belongs into the operator's
hands


Do you mean "not as long as it would cause the above bugs" or "not 
ever"?  If the latter, is it because of wasted storage/bandwidth or 
something else?


Do you also believe that the existing hardware-related -all packages 
(printer-driver-all(-enforce), va-driver-all, vdpau-driver-all, 
xserver-xorg-input-all, xserver-xorg-video-all) should not exist / 
should not be installed by default?



(or possibly some installer magic).


Do we have such a mechanism?  I agree this would be better if it existed.

The AppStream metadata format includes a field for "hardware this works 
with", and beignet-opencl-icd has one, but I don't know if any existing 
tools use this field.

Re: ZFS in Buster

[Sam Hartman]

> "Enrico" == Enrico Weigelt, metux IT consult  writes:

Enrico> On 01.07.19 21:06, Enrico Weigelt, metux IT consult wrote:
Enrico> Hi,

>> IIRC the whole things is actually about crypto stuff. Why don't
>> zfs> folks just use the standard linux crypto api (potentially
>> introduce
a> 
Enrico> new algo if the existing ones aren't sufficient) ?
Enrico> Addendum: just had a quick scan through the code and found a
Enrico> completely own AES implementation.

Almost certainly because the crypto apis in linux are gpl only and so
 module cannot actually call them.

I'd like to  echo Ian's comments from earlier today.
You do seem to be dominating the list without managing to display
adequate understanding of the points of view of other participants.
It might be valuable to respond  fewer times to the same thread and to
spend a bit more time trying to understand others in the thread.
If you're having trouble doing that it might be good to reach out to see
if you could have an IRC, phone or real-life meeting with participants
to get more onto the same page to participate more effectively.
I think this is much more obvious on the git threads than the zfs
threads.

--Sam

Re: ZFS in Buster

[Enrico Weigelt, metux IT consult]

On 01.07.19 21:06, Enrico Weigelt, metux IT consult wrote:

Hi,

> IIRC the whole things is actually about crypto stuff. Why don't zfs> folks 
> just use the standard linux crypto api (potentially introduce a>
new algo if the existing ones aren't sufficient) ?
Addendum: just had a quick scan through the code and found a completely
own AES implementation.

Seriously ?

Completely redundant cipher implementations from an likely understaffed
project is something I wouldn't like to have on my machines, not in the
kernel. There're just too many pitfalls (especially w/ buggy x86 CPUs)
in crypto programming - I wouldn't dare to implement that myself and
run it on production systems.

And for performance, many folks like to use hw acceleration. Linux
crypto api already provides that.


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: ZFS in Buster

[Enrico Weigelt, metux IT consult]

On 07.06.19 10:16, Philipp Kern wrote:

Hi,

> This would not be the case here. But crippling the performance would be> 
> indeed an option, even though this would make Debian much less
relevant> for ZFS deployments and people would just go and use Ubuntu
instead.
Is it really necessary to to have some competition w/ Ubuntu on who's
got the larger user base ? In which way is that relevant for the
progress on Debian ?

For me, personally, when working on FOSS, it never mattered how many
users are out there - don't need to "sell" anything.

> I personally wonder why a kernel which provides a module interface does 
> not provide a way to save FPU state, but alas, they made their decision.

Because that's really low-level, arch specific stuff. I don't even
recall any platform driver that ever cares about such things. From a
kernel hacker/maintainer pov, the idea of having an arch specific
filesystem driver, sounds really weird.

This function IIRC just was a workaround for kvm, which always had been
suboptimal and was replaced by a better solution. Since nobody used it
anymore for quite some time, it got removed. And regarding LTS, I don't
recall that Greg ever made any committment on not removing obsolete and
used stuff (he's just reluctant of putting too much extra work into
that for his lts trees).

Of course, as users of kernel-internal APIs, only the in-tree stuff
matters - this has always been the policy in Linux development
(at least as far as I remember).

IIRC the whole things is actually about crypto stuff. Why don't zfs
folks just use the standard linux crypto api (potentially introduce a
new algo if the existing ones aren't sufficient) ?


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: AMDGPU+OpenCL with Debian?

[Enrico Weigelt, metux IT consult]

On 19.06.19 09:09, Rebecca N. Palmer wrote:

Hi,

> I proposed [0] fixing this by creating a metapackage for "all OpenCL
> drivers" (similar to the ones for graphics).  However, having unusable
> OpenCL drivers installed can trigger bugs: [1] in llvm, and some
> applications that treat "no hardware for this driver" as "fatal error"
> instead of "try the next driver".

So, installing an opencl-based package pulls in *all* cl driver stacks ?

Please don't do that. This IMHO clearly belongs into the operator's
hands (or possibly some installer magic).

--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: scratch buildds

[Enrico Weigelt, metux IT consult]

On 15.06.19 11:20, Helmut Grohne wrote:

Hi,

> Unlike the ${dist}-proposed variant, the scratch distribution can be set> up 
> entirely outside Debian. It only needs someone doing the work with
no> involvement of DSA. Wait, this reminds me of something. Luca
Falavigna> put up debomatic-${arch}.debian.net. And it has piuparts and
lintian!
As somebody who does backports stuff and project/client specific repos,
I've created something on my own, which can build whole stacks of
packages and create apt repos. it also allows fine control on what is
in the base image, extra repos, etc.

The bad thing for me is: I've only got limited computing power and
and very limited in available archs (just x86 and some older arm).
So, having a CI that can build for all the debian supported archs and
allows using extra repos, tailored base images, work on git directly
(fully debianized branch) and publishes the repos to the outside world,
would be a really cool thing for me. IIRC that should also cover this
'scratch builds' usecase.

I admit, I haven't checked whether gitlab-ci can already do that.


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: Survey: git packaging practices / repository format

[Enrico Weigelt, metux IT consult]

On 01.07.19 15:09, Andrey Rahmatullin wrote:
> On Mon, Jul 01, 2019 at 03:04:26PM +0200, Enrico Weigelt, metux IT consult 
> wrote:
>> On 29.05.19 17:41, Andrey Rahmatullin wrote:
>>
 Perhaps we should update policy to say that the .orig tarball may (or
 even "should") be generated from an upstream release tag where
 applicable.
>>> This conflicts with shipping tarball signatures.
>>
>> Does that really need to be the upstream's tarballs ?
> The idea is checking the sig that the upstream made, with the key the
> upstream published.

Okay, but is that actually used (by somebody except the maintainers) ?

>> If it's about validating the source integrity all along the path from
>> from upstream to deb-src repo, we could do that by auditable process
>> (eg. fully automatic, easily reproducable transformations)
> Sounds very complicated.

I don't think so, at least if we're considering the whole workflow.

In the end, it's just a matter of trust-chains:

* upstream should used signed tags - we can collect their pubkeys
  in some suitable place (what we should do anyway).
* if upstream doesn't sign, the maintainer has to trust them blindly,
  or needs to verify the code anyways. we could use some half-automated
  process for verifying the diff between the upstream tarball and the
  scm repo (we could add our own signatures here)
* finally the maintainer signs his final tree (the one that's used for
  actual building the final packages)

I believe that 99% can be done automatically, with a little bit of
tooling.

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: Survey results: git packaging practices / repository format

[Enrico Weigelt, metux IT consult]

On 28.06.19 23:42, Ian Jackson wrote:

>https://wiki.debian.org/GitPackagingSurvey
> 
> Unfortunately due to bugs in the Debian and Wiki CSS styles, it is
> displayed very suboptimally unless you log into the wiki and set your
> style to "Classic".  (I have filed bug reports.)

Very nice work.

For the next stage I think it would be nice to assign some kind of
canonical names for the individual workflows and try to create some
precise documentation.

Once we have that, we could add some machine readable file to the d/
trees stating the exact workflow and possibly extra metadata
(eg. where to find certain repos/branches, etc).

> [1] This does *not* include the one response from a Debian downstream.

Talking about me ? ;-)

> The task of being a Debian downstream is rather different and it
> doesn't make sense to try to represent that in the same table.

I don't think it's so different in that regard. IMHO the only difference
is that I'm not an official debian maintainer and not using buildd.
But I believe the same approach could be used by any actual Debian
maintainer if he just also creates dsc's and pushes them onto buildd.

Maybe it would be better to just differenciate the statistics between
official debian and others (such as downstreams).


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: Survey: git packaging practices / repository format [and 1 more messages]

[Ian Jackson]

Picking just two examples:

Enrico Weigelt, metux IT consult writes ("Re: Survey: git packaging practices / 
repository format"):
> I think for each supported upstream version there should be ...

Enrico Weigelt, metux IT consult writes ("Re: Survey: git packaging practices / 
repository format"):
> ACK. But there should also be some definition or at least guideline ...
>  and some rules on how the ... shall be done. ...

I appreciate that you feel you have a lot of knowledge and experience,
and some good practices to share, but:

* This thread was not really the place for sharing best practices.
* As previously noted, this thread was about maintenance within
   Debian, but you are primarily discussing being a downstream.
* You have posted quite a lot, covering a wide set of subtopics,
   and your messages are starting to dominate the thread.
* You're trying to get Debian to change the way it does things but
   this is not really the right way to go about it.

I am sympathetic to many of your points, but can you please reconsider
your approach ?

Many of the things you say here are worth discussing but burying them
in this thread, in a scattered series of messages, is not very likely
to produce useful outcomes.  And some off-thread-topic messages are to
be expected, but it is a problem if the thread gets derailed.

Thanks,
Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Survey: git packaging practices / repository format

[Enrico Weigelt, metux IT consult]

On 29.05.19 23:39, David Bremner wrote:

>> Also, how do you move to a new upstream version ?
> 
> use git merge, typically from an upstream tag, or from a debian specific
> upstream branch with tarballs imported on top of upstream history.

Uh, that creates an pretty ugly, unreadable git repo and makes
interacting w/ upstream (eg. submitting patches) unncessarily hard.

That's something I regularily see w/ crappy vendor kernels, which then
take lots of time to bring them into some somewhat usable state :o


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: Survey: git packaging practices / repository format

[Andrey Rahmatullin]

On Mon, Jul 01, 2019 at 03:04:26PM +0200, Enrico Weigelt, metux IT consult 
wrote:
> On 29.05.19 17:41, Andrey Rahmatullin wrote:
> 
> >> Perhaps we should update policy to say that the .orig tarball may (or
> >> even "should") be generated from an upstream release tag where
> >> applicable.
> > This conflicts with shipping tarball signatures.
> 
> Does that really need to be the upstream's tarballs ?
The idea is checking the sig that the upstream made, with the key the
upstream published.

> If it's about validating the source integrity all along the path from
> from upstream to deb-src repo, we could do that by auditable process
> (eg. fully automatic, easily reproducable transformations)
Sounds very complicated.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Survey: git packaging practices / repository format

[Enrico Weigelt, metux IT consult]

On 29.05.19 17:41, Andrey Rahmatullin wrote:

>> Perhaps we should update policy to say that the .orig tarball may (or
>> even "should") be generated from an upstream release tag where
>> applicable.
> This conflicts with shipping tarball signatures.

Does that really need to be the upstream's tarballs ?
Why not just automatically generating the orig tarballs and fingerprint
*them* (not caring about the upstream's tarball at all) ?

If it's about validating the source integrity all along the path from
from upstream to deb-src repo, we could do that by auditable process
(eg. fully automatic, easily reproducable transformations)


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: Survey: git packaging practices / repository format

[Enrico Weigelt, metux IT consult]

On 29.05.19 15:14, Ben Hutchings wrote:

> Perhaps we should update policy to say that the .orig tarball may (or
> even "should") be generated from an upstream release tag where
> applicable.

ACK. But there should also be some definition or at least guideline on
what is considiered "applicable" (or better: when is it okay not doing
that) and some rules on how the generation process shall be done.
(maybe having some script with a defined name ?)


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: Survey: git packaging practices / repository format

[Enrico Weigelt, metux IT consult]

On 02.06.19 16:22, Sam Hartman wrote:
>> "Nikolaus" == Nikolaus Rath  writes:
> 
> 
> Yes, but the lack of similarity is the thing I find interesting.
> In git-pdm (and git-debrebase), you retain all the rebases and stitch
> them together with various pseudo-merges into a combined history.
> 
> If you could avoid that and have a more pure rebase workflow, I think it
> would be nice.
> As Ian points out, we don't know how to do that because we don't know
> how to figure out whether you have the latest rebase.

Could you give me some more details on the intended workflow ?
Why does one need that information at all ?


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: Survey: git packaging practices / repository format

[Enrico Weigelt, metux IT consult]

On 02.06.19 00:57, Ian Jackson wrote:

Hi,

> The difficulty with this as a collaboration approach is that you can't
> tell whether a rebase is "the newest", at least without a lot of
> additional information.  That additional information is the "clutter"
> if you like which the "cleaner" history doesn't contain.

Depends on what you actually call "new".

I think for each supported upstream version there should be a separate
maintenance branch (we can track the maintenance status in some other
place, maybe some extra metadata branch). These individual branches are
never rebased, just created once on the corresponding upstream tag.
(well, there could be extra devel branches that indeed are rebased on
upstream's master, but that has to be explicitly communicated)

> Both git-debrebase and git-dpm use a special history structure to
> record what the most recent rebase is.  Obviously I prefer
> git-debrebase since I wrote it - using a different data model - even
> after I knew about git-dpm and its data model.  But maybe this isn't
> the thread for that advocacy conversation.

What exactly does this tool do ?


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: Survey: git packaging practices / repository format

[Enrico Weigelt, metux IT consult]

On 29.05.19 14:47, Sam Hartman wrote:

Hi,

> I'm certainly going to look at dck-buildpackage now, because what he
> describes is a workflow I'd like to be using within Debian.

:)

Maybe you'd also find that useful: https://github.com/metux/deb-pkg

It's a little tool for automatically creating whole repos, eg.
automatically cloning repos (even w/ multiple remotes), building
whole dependency trees (deps are explicitly configured instead of
fetched from the packages, intentionally), etc.

Note: the "master" branch is pretty hackish, basically an example - the
idea is to branch off from "base" for each project and do the necessary
changes directly in git. (from time to time it's worth rebasing).

> For some projects I want to ignore orig tarballs as much as I can.  I'm
> happy with native packages, or 3.0 quilt with single-debian-patch.

single-patch isn't so nice for interacting w/ upstreams.

> I don't want merge artifacts from Debian packaging on my branches.
> I'm happy to need to give the system an upstream tag.

I'd prefer always having a debian branch ontop of the upstream release
tag and doing all the debianziation there, possibly per-release or
dist release, flavour, etc.

> I'm happy for a dsc to fall out the bottom, and so long as it
> corresponds to my git tree I don't care how that happens.

ACK. I see dsc just as an autogenerated itermediate stage for certain
build systems (eg. builldd) or providing src repos.

> I have a slight preference for 3.0 format over 1.0 format packages.  3.0
> makes it possible to deal with binaries, better compression and a couple
> of things like that.  The quilt bits are (in this workflow) an annoyance
> to be conquered, not a value.

ACK. That's why I do everything in git only.
I don't really care how the src packages look like, as long as I've got
an easy and fully automatic way for getting a clean git tree with all
the necessary changes already applied as readable (and documented)
git commits.

> The thing his approach really seems to have going for it is that he
> gives up on the debian history fast forwarding and instead rebases a lot
> for a cleaner history.

ACK. Personally, I don't see any actual value in an separate Debian
history, or even an history of the text-based patches.

git-rebase is one of my primary daily tools.

> If we could figure out a way to collaborate on something like that well,
> it might be a very interesting tool to have.

ACK.

I believe we should set some computable policies on how orig trees are
generated from actual upstream repos and patches are handled, so we can
do imports/transformations fully automatically.


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: Survey results: git packaging practices / repository format

[Ian Jackson]

Simon McVittie writes ("Re: Survey results: git packaging practices / 
repository format"):
> The precise names of the branches are not immediately relevant to
> GitPackagingSurvey: it's the content of those branches that matters.

Thanks for this mail which I think will be very helpful.

> From https://perl-team.pages.debian.net/git.html#Patches it appears
> the Perl team is using what the survey page has labelled as "unapplied"
> (also seen referred to as "patches-unapplied" elsewhere):
> 
> - a git checkout of the packaging branch contains debian/control, etc.
> - a git checkout of the packaging branch contains upstream files like
>   Makefile.PL
...

I wonder if it would be worth making it much clearer that the survey
page's main table is talking only about the contents of the main
packaging branch, and ...

> default gbp branch name DEP-14 branch name
> (e.g. Perl and systemd teams)   (e.g. GNOME team)
> --- --
> master  debian/master
> stretch, etc. (not standardized)debian/stretch, etc.
> upstreamupstream/latest, upstream/2.32.x, etc.
> pristine-tarpristine-tar

... to discuss or even include this.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Survey results: git packaging practices / repository format

[Ian Jackson]

Ian Jackson writes ("Re: Survey results: git packaging practices / repository 
format"):
> What it doesn't say is how changes to the upstream files are
> represented.  Can you explain ?

Also: I had hoped that the table's focus on particularly these
questions:
 - representation of changes to upstream files
 - upstream files in the git tree: modified, unmodified, or absent
 - is there debian/patches and what does it contain
would allow you to see which row applies.

Evidently that's not true.

Can you explain to me in your own terminology what is going on, or
explain what was confusing about the page ?

We have, generally, a very serious lack of understanding between
various different people about how all this stuff is done.  I am
trying to bridge these gaps but apparently not yet successfully
enough.

Ian.

Re: Survey results: git packaging practices / repository format

[Ian Jackson]

Andreas Tille writes ("Re: Survey results: git packaging practices / repository 
format"):
> On Fri, Jun 28, 2019 at 10:42:29PM +0100, Ian Jackson wrote:
> > Thanks to everyone who responded.  I (with some help from Sean and
> > some review from Sam) have worked the answers into a wiki page:
> > 
> >https://wiki.debian.org/GitPackagingSurvey
> 
> I admit I did not joined the dgit discussion - may be that's the reason
> I can not really match the branches that are used in Perl team policy
> 
>  https://perl-team.pages.debian.net/git.html#repository_layout
> 
> (which is used by several other teams I'm working in) to the table in
> the Wiki.

My table is supposed to be useable without knowing anything about
dgit.

I looked at the perl page you link to, and it says:

  repository layout

  We're using the typical git-buildpackage branch layout: Upstream
  sources are kept (in plain, uncompressed form) in the upstream
  branch. The data needed to regenerate original source tarballs from
  the upstream branch is kept with the help of pristine-tar(1) in the
  pristine-tar branch. Upstream sources are merged with Debian-specific
  changes in the master branch, which is the usual place to work in.

I think this means you `git merge' the `upstream' branch into your
`master' branch.

What it doesn't say is how changes to the upstream files are
represented.  Can you explain ?

I tried reading the web page but I am trying to avoid reading and
understanding the manpages for every one of our hundreds of
git/package/branch management utilities, and the bulk of the page was
just runes for dpt or gbp.

There's a section "Patches" which mentions quilt(1).  So I think the
answer to the question "how are changes to upstream files represented"
is "the upstream files are not directly modified in the master branch;
instead, there are patches to them, in patches in debian/patches" ?

That would mean you are using the form I call "unapplied".

> > Please let us know if we have missed one.  It is probably better if
> > you ask us rather than just adding it, unless you're sure that what
> > you are adding isn't the same as one of the existing ones.  In
> > particular it seems that "unapplied" is used by a large number of
> > people with disjoint tooling and disjoint terminology.
> 
> So I'm just asking if its just me not understanding the table properly
> or whether the layout I'm used to in close to all teams I'm working in
> is not mentioned.

I think we have a terminological problem.

Ian.

-- 
Ian JacksonThese opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Survey: git packaging practices / repository format

[Enrico Weigelt, metux IT consult]

On 29.05.19 13:50, Ian Jackson wrote:

Hi,

> Oh.  I think I have misunderstood.  I think you are describing a git
> workflow you use as a *downstream* of Debian, not as a maintainer
> *within* Debian.

Yes, something like that. I'm maintaining additional repos for certain
projects, usually backports or packages that aren't in Debian at all.
Pretty often it's pretty much rebasing Debianziation onto latest
upstream releases.

A very annoying aspect for me is the way upstream souces and patches
are managed, eg.

* I need to reproduce how exactly orig trees are constructed from the
  actual upstream (git) trees. We often have autogenerated stuff in
  here (eg. autotools stuff), often files are missing, stuff moved
  around, etc, etc.
  --> here we at least should have some fully automatic transformation
  system within git. probably not within the actual source packages
  themselves, possibly as a cross-package project.
* text-based patches are costly to reproduce into a git repository
  * many just don't apply via git-am --> at least we should fix them
and add a policy that all patches need to be git-am compatible
(no, quilt isn't so much helpful here, and I find it pretty complex
to use - compared with git - I need rebase)
  * we don't have any clear (machine-readable) distinction between types
of patches, eg. whether they're generic or really debian specific
  * somethimes we even have patches against autogenerated stuff
  * many patches lack any clear description
* sometimes we even have weird transformations on the source tree
  (usually on the orig tree, but sometimes also within rules file)

Few days ago, I tried to rebuild recent rustc (which I need for tbird),
but got a lot of strange fails. It also lacked the source code for some
library where that specific version even doesn't exist in the upstream
git repo anymore. I know that rust is an ugly beast, but those things
just should not happen. The rust toolchain seems to be a good candidate
for creating an fully automatic git transformation (eg. transform
submodules into merges, etc).

I'd like to propose some additions to the packaging policies:

* there shall be no source tree transformations in the build process,
  all necessary changes shall be done by patches
* the upstream build process shall be used as-is whenever possible,
  if necessary patch it. (eg. no manual build or install rules, etc)
* there shall be no conditional applying of patches - the queue shall
  be always applied a as a whole. if certain code changes are only
  applicable for certain build conditions (eg. different flavours like
  with the kernel), proper build-time config flags shall be introduced.
* all patches shall be git-am compatible and have a clear description
  of what they're actually doing
* patches shall be written in an generic/upstreamable way if possible
  (eg. introduce new build-time config flags if necessary)
* patches shall be grouped into generic/upstreamable and distro specific
  ones, the differenciation shall be easily machine-readable (eg.
  message headers), and the generic ones shall be first in the queue.
* no patching of autogenerated files
* autogenerated files shall be removed from source and always
  regenerated within the build process
* the debian/ directory shall contain a machine readable file for
  telling exactly the upstream source tree (eg. canonical version, url
  to tarball and gitrepo, tag name + commit id, etc)
* minimum required debhelper version shall be the one present in stable,
  (or better oldstable) - unless threre's really no other sane way

> And I think what you are saying is that you don't use source packages
> (.dsc) except maybe in the innards somewhere of your machinery.
> I think that is a good way for a downstream to work.  Certainly when I
> modify anything locally I don't bothere with that .dsc stuff.

Right, I've always found that very complicated and hard to maintain.
Actually, I'd like to propose phasing it out that old relic. With git
we just don't need that anymore.

> But my aim in this thread was to capture how people work *within*
> Debian, where a maintainer is still required to produce a .dsc.

I don't think that .dsc really makes the picture so different. It always
can be generated automatically. IMHO, it's only needed as an output
format for creating src repos and as an intermediate transport for
traditional tools like buildd.


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287

Re: Survey results: git packaging practices / repository format

[Simon McVittie]

On Mon, 01 Jul 2019 at 07:30:39 +0200, Andreas Tille wrote:
> I admit I did not joined the dgit discussion - may be that's the reason
> I can not really match the branches that are used in Perl team policy
> 
>  https://perl-team.pages.debian.net/git.html#repository_layout
> 
> (which is used by several other teams I'm working in) to the table in
> the Wiki.

The precise names of the branches are not immediately relevant to
GitPackagingSurvey: it's the content of those branches that matters.

>From https://perl-team.pages.debian.net/git.html#Patches it appears
the Perl team is using what the survey page has labelled as "unapplied"
(also seen referred to as "patches-unapplied" elsewhere):

- a git checkout of the packaging branch contains debian/control, etc.
- a git checkout of the packaging branch contains upstream files like
  Makefile.PL
- if the Debian maintainer has patched upstream files like Makefile.PL,
  those changes appear as a quilt-style patch series in debian/patches/*
- if the Debian maintainer has patched upstream files like Makefile.PL,
  those changes are *not* reflected in the copy of Makefile.PL you get
  by just checking out the packaging branch (hence "unapplied" - in the
  representation you check out, your patches exist as patches but have
  not been applied yet)

(If you read those and you think "well, obviously... what else would
we do?" then you probably haven't encountered the workflows represented
by the other rows in the survey's table.)

This is the same setup that gbp-pq assumes, and is also used by the
GNOME, systemd and Python modules teams, among others, although with some
variation in branch naming. The main variation that I've seen is that
some teams (e.g. Perl) consistently use the default gbp branch names,
some teams (e.g. GNOME) consistently use the names recommended in DEP-14
, and some teams vary
per-repository (e.g. games, Python modules).

default gbp branch name DEP-14 branch name
(e.g. Perl and systemd teams)   (e.g. GNOME team)
--- --
master  debian/master
stretch, etc. (not standardized)debian/stretch, etc.
upstreamupstream/latest, upstream/2.32.x, etc.
pristine-tarpristine-tar

I personally think DEP-14 is a good idea, and I advocated its use when
the GNOME team switched from svn to git. It's particularly useful when
more than one upstream branch is packaged in parallel (for example GNOME
3.30 in unstable and 3.32 in experimental at the moment), or when there
is an active downstream distro, which can simply branch from the Debian
packaging and replace debian/ with their own name (for example Ubuntu
uses branches like ubuntu/cosmic for automated package source imports
on Launchpad, and various smaller Debian derivatives like Apertis,
Kali Linux and SteamOS use DEP-14 branch names like apertis/v2019,
kali/master and steamos/brewmaster to track their modified packages).

smcv