Re: Difficult Packaging Practices

[Niels Thykier]

Vincent Bernat:
>  ❦ 28 mai 2019 06:30 +00, Niels Thykier :
> 
>> I.e. with the proper implementation of "make-it-work" (in the lack of a
>> better name - maybe something "fetch-and-build"), the following should
>> be possible
>>
>> """
>> #!/usr/bin/make -f
>>
>> # DISTRIBUTION = $(shell sed -n "s/^VERSION_CODENAME=//p" /etc/os-release)
>> export VERSION=1.5.16
>> # PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
>> export TARBALL = memcached-$(VERSION).tar.gz
>> export URL = https://www.memcached.org/files/$(TARBALL)
>>
>> %:
>> dh $@ --with make-it-work
>>
>> # override_dh_gencontrol:
>> # dh_gencontrol -- -v$(PACKAGEVERSION)
>> """
>>
>> This could inject a tool to run the wget + tar extraction (incl. the
>> distro version sed'ing) after dh_testdir and before
>> dh_update_autotools_config.  From there, the dh's standard tools would
>> "just work(tm)" until dh_gencontrol where we have to inject the full
>> version (the commented lines works around that if re-enabled).
>>
>> @Vincent: Do you feel something like this would be helpful, useful and
>> doable? My only reference in the memcached, so the above is tailored to
>> the above.
> 
> That would be very useful as it would reduce the boilerplate but also
> solve some of the problems of hijacking the clean target, notably the
> fact it is executed twice by some tools (like pbuilder, once outside,
> once inside). I didn't propose anything like that as I was thinking it
> was very likely to be rejected as it is of no use for proper Debian
> packages. While my Perl is a bit rusty, I can propose some "dh_fetch"
> helper for this if there is no huge opposition against this approach.
> 

I think it would be fine for a stand-alone debhelper utility/add-on
package to contain such a tool and a dh-sequence to enable it. :)  I am
happy with assisting with the technical bits of this utility.

>> Would it help if we could remove the dependency on having a d/changelog
>> (i.e. make it optional)?  I have not fully checked if it is doable, but
>> I can look into it if it makes sense and if someone wants to help me
>> test this.
> 
> Yes, it would help too.
> 

I had a stab at it and the results are not too positive.  Already before
you reach dh, dpkg-buildpackage goes "I want a d/changelog".
Furthermore, dpkg-gencontrol also insist on a changelog (but the
dh_fetch/dh_uscan tool could have generated it is by then).

Thanks,
~Niels

Simplified Debianization Process (was: Re: Difficult Packaging Practices (OT)

[Mo Zhou]

Hi Thomas,

On 2019-05-28 07:48, Thomas Dettbarn wrote:
> Debian was quite more complicated, and the documentation on that 
> topic was scattered all over the interwebs. Here I had to download
> the sources, rename the directories, rename the package, repackage,
> change some files, count the number of spaces at the beginning of a
> line... I did it, but it is rather frustrating for new developers. 
> 
> Please please help them! :)

That complexity comes from various reasons.

There is an on-going project named duprkit[1], which aims to provide
a simplified Debian packaging experience. Specifically, it defines
a "Recipe" format (YAML+HFT), which can be automatically converted
into a Debianized source tree (including downloading the source).
This tool set allows:

  * automatic Recipe guessing (nothing -> Recipe)
  * translating Recipe into debian/ directory (Recipe -> debian/ dir)
  * automatic debian/ directory generation (nothing -> Recipe -> debian/
dir)
  * Recipe -> .deb/.dsc convertion (download, extract, debianize, build)

I plan to beat both dh-make and debmake in the future.
(although you might be interested in them)

However, the problem is, even if this "duprkit" aims at
"Simplification" and "Automization", documentation is still
a problem due to lack of time and energy. The only thing
you can read is the code and some examples[2]. Plus, due
to the same reason y I'm still dragging the next major release
after a significant redesign.

[1] https://github.com/dupr/duprkit
[2] https://github.com/dupr/duprkit/tree/master/examples

Re: Difficult Packaging Practices (OT)

[Alex Mestiashvili]

On 5/28/19 9:48 AM, Thomas Dettbarn wrote:
> Hello.
> 
> 
> This is slightly off-topic, and I have not been on this mailinglist
> since this thread started. Nevertheless, I would like to express my
> support for changing the packaging practices.
> 
> Over the last two weeks, I tried creating ports and packages for my
> project "dMagnetic". It was rather easy for the OpenBSD and FreeBSD 
> ports system. All that was needed there were some changes to some 
> existing files, and my port was ready. FreeBSD was the easiest: 
> Sourcecode URL, SHA256 sum and a short description was all that was 
> needed. Now it is already part of the offical ports collection.
> 
> Debian was quite more complicated, and the documentation on that 
> topic was scattered all over the interwebs. Here I had to download
> the sources, rename the directories, rename the package, repackage,
> change some files, count the number of spaces at the beginning of a
> line... I did it, but it is rather frustrating for new developers. 
> 
> Please please help them! :)
> 
> 
> Thomas Dettbarn
> 

This thread reminded me the Debian User Repository thread:
 https://lists.debian.org/debian-devel/2019/04/msg00064.html

Such a repository can be a "easy" packaging zone, possibly attracting
more contributing people. Eventually some people will try to improve the
packages and get them into official Debian.

Re: Difficult Packaging Practices (OT)

[Thomas Dettbarn]

Hello.


This is slightly off-topic, and I have not been on this mailinglist
since this thread started. Nevertheless, I would like to express my
support for changing the packaging practices.

Over the last two weeks, I tried creating ports and packages for my
project "dMagnetic". It was rather easy for the OpenBSD and FreeBSD 
ports system. All that was needed there were some changes to some 
existing files, and my port was ready. FreeBSD was the easiest: 
Sourcecode URL, SHA256 sum and a short description was all that was 
needed. Now it is already part of the offical ports collection.

Debian was quite more complicated, and the documentation on that 
topic was scattered all over the interwebs. Here I had to download
the sources, rename the directories, rename the package, repackage,
change some files, count the number of spaces at the beginning of a
line... I did it, but it is rather frustrating for new developers. 

Please please help them! :)


Thomas Dettbarn

Re: Difficult Packaging Practices

[Vincent Bernat]

 ❦ 28 mai 2019 06:50 +00, PICCA Frederic-Emmanuel 
:

>> packages. While my Perl is a bit rusty, I can propose some "dh_fetch"
>> helper for this if there is no huge opposition against this approach.
>
> why not a dh_uscan ?
>
> what is the fundamental difference between dh_fetch and what you can
> achieve by using uscan from the rules file ?

Yes, it would work too.
-- 
Use statement labels that mean something.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Difficult Packaging Practices

[Vincent Bernat]

 ❦ 28 mai 2019 06:30 +00, Niels Thykier :

> I.e. with the proper implementation of "make-it-work" (in the lack of a
> better name - maybe something "fetch-and-build"), the following should
> be possible
>
> """
> #!/usr/bin/make -f
>
> # DISTRIBUTION = $(shell sed -n "s/^VERSION_CODENAME=//p" /etc/os-release)
> export VERSION=1.5.16
> # PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
> export TARBALL = memcached-$(VERSION).tar.gz
> export URL = https://www.memcached.org/files/$(TARBALL)
>
> %:
> dh $@ --with make-it-work
>
> # override_dh_gencontrol:
> # dh_gencontrol -- -v$(PACKAGEVERSION)
> """
>
> This could inject a tool to run the wget + tar extraction (incl. the
> distro version sed'ing) after dh_testdir and before
> dh_update_autotools_config.  From there, the dh's standard tools would
> "just work(tm)" until dh_gencontrol where we have to inject the full
> version (the commented lines works around that if re-enabled).
>
> @Vincent: Do you feel something like this would be helpful, useful and
> doable? My only reference in the memcached, so the above is tailored to
> the above.

That would be very useful as it would reduce the boilerplate but also
solve some of the problems of hijacking the clean target, notably the
fact it is executed twice by some tools (like pbuilder, once outside,
once inside). I didn't propose anything like that as I was thinking it
was very likely to be rejected as it is of no use for proper Debian
packages. While my Perl is a bit rusty, I can propose some "dh_fetch"
helper for this if there is no huge opposition against this approach.

> Would it help if we could remove the dependency on having a d/changelog
> (i.e. make it optional)?  I have not fully checked if it is doable, but
> I can look into it if it makes sense and if someone wants to help me
> test this.

Yes, it would help too.
-- 
Use library functions.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Difficult Packaging Practices

[Niels Thykier]

Vincent Bernat:
>  ❦ 28 mai 2019 08:59 +08, Paul Wise :
> 
>>> People using tools like fpm will never get familiar with our tools and
>>> will never be contributors.
>>
>> I enjoyed your blog post about pragmatic packaging using Debian's
>> tools instead of fpm, it seems like a good approach if one is
>> committed to using Debian, especially since it allows for incremental
>> improvement towards a package that could even enter Debian.
>>
>> https://vincent.bernat.ch/en/blog/2019-pragmatic-debian-packaging
>>
>> OTOH I get the feeling that most FLOSS upstreams are interested in
>> cross-OS and cross-distro packaging rather than Debian-specific
>> packaging. I don't have a good feeling for how "sticky" OS/distro
>> choices are for folks who are building non-FLOSS services on top of
>> FLOSS distros.
> 
> Yes, upstreams are more likely to prefer one tool. However, if you
> contribute a native implementation that works accross the same set of
> Debian-derivative distributions, in my experience, they are usually
> eager to accept it it stays simple enough to not be a maintenance
> burden. Of course, it helps if they didn't start with fpm.
> 

It seems to me that it should be doable to write one or two helper tools
that could handle the fetch and then pass the downloaded code on to
dh_auto_* which would then DTRT (in case of standard build systems).


I.e. with the proper implementation of "make-it-work" (in the lack of a
better name - maybe something "fetch-and-build"), the following should
be possible

"""
#!/usr/bin/make -f

# DISTRIBUTION = $(shell sed -n "s/^VERSION_CODENAME=//p" /etc/os-release)
export VERSION=1.5.16
# PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
export TARBALL = memcached-$(VERSION).tar.gz
export URL = https://www.memcached.org/files/$(TARBALL)

%:
dh $@ --with make-it-work

# override_dh_gencontrol:
# dh_gencontrol -- -v$(PACKAGEVERSION)
"""

This could inject a tool to run the wget + tar extraction (incl. the
distro version sed'ing) after dh_testdir and before
dh_update_autotools_config.  From there, the dh's standard tools would
"just work(tm)" until dh_gencontrol where we have to inject the full
version (the commented lines works around that if re-enabled).

@Vincent: Do you feel something like this would be helpful, useful and
doable? My only reference in the memcached, so the above is tailored to
the above.

Would it help if we could remove the dependency on having a d/changelog
(i.e. make it optional)?  I have not fully checked if it is doable, but
I can look into it if it makes sense and if someone wants to help me
test this.

Thanks,
~Niels

Re: Difficult Packaging Practices

[Vincent Bernat]

 ❦ 28 mai 2019 08:59 +08, Paul Wise :

>> People using tools like fpm will never get familiar with our tools and
>> will never be contributors.
>
> I enjoyed your blog post about pragmatic packaging using Debian's
> tools instead of fpm, it seems like a good approach if one is
> committed to using Debian, especially since it allows for incremental
> improvement towards a package that could even enter Debian.
>
> https://vincent.bernat.ch/en/blog/2019-pragmatic-debian-packaging
>
> OTOH I get the feeling that most FLOSS upstreams are interested in
> cross-OS and cross-distro packaging rather than Debian-specific
> packaging. I don't have a good feeling for how "sticky" OS/distro
> choices are for folks who are building non-FLOSS services on top of
> FLOSS distros.

Yes, upstreams are more likely to prefer one tool. However, if you
contribute a native implementation that works accross the same set of
Debian-derivative distributions, in my experience, they are usually
eager to accept it it stays simple enough to not be a maintenance
burden. Of course, it helps if they didn't start with fpm.
-- 
Make sure your code "does nothing" gracefully.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Difficult Packaging Practices

[Paul Wise]

On Mon, May 27, 2019 at 2:52 PM Vincent Bernat wrote:

> People using tools like fpm will never get familiar with our tools and
> will never be contributors.

I enjoyed your blog post about pragmatic packaging using Debian's
tools instead of fpm, it seems like a good approach if one is
committed to using Debian, especially since it allows for incremental
improvement towards a package that could even enter Debian.

https://vincent.bernat.ch/en/blog/2019-pragmatic-debian-packaging

OTOH I get the feeling that most FLOSS upstreams are interested in
cross-OS and cross-distro packaging rather than Debian-specific
packaging. I don't have a good feeling for how "sticky" OS/distro
choices are for folks who are building non-FLOSS services on top of
FLOSS distros.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Difficult Packaging Practices

[Sam Hartman]

> "Ben" == Ben Finney  writes:

Ben> Sam Hartman  writes:
>> If you just want to get upstream's idea of their package onto a
>> system with their release schedule and their recommended
>> dependency versions, there are better ways than getting a package
>> into Debian.

Ben> In the Debian mentors forum (that is, the chat channel, the
Ben> mailing list, etc.) we make a point of saying: that's fine!

Ben> Not every package needs to be in Debian, for its users to
Ben> install that package.

I think I was trying to agree with you.


I'd actually recommend understanding people's pain points for other
approaches and making sure there aren't things we can do to make it
easier.

As an example, Ubuntu makes it easy to add side-loaded channels (ppas).
we don't.

The fast-paced repository discussion from earlier this year is another
example of where we could address a pain point.

Another pain point is people who would like to do the work of making
something stable but who need a fixed set of dependencies; the space
where containers might be a solution.
We as a distribution haven't done a lot there beyond including flatpak
and snap.

--Sam

Re: Difficult Packaging Practices

[Sam Hartman]

> "Russ" == Russ Allbery  writes:

Russ> This is my experience as well.  I've occasionally tried to get
Russ> people at work (at various jobs) interested in packaging
Russ> software for Debian, without all that much success.  The
Russ> problem seems less any one specific thing and more that
Russ> they're perfectly content with a Debian package created by
Russ> running fpm on some install tree, and don't see the point in
Russ> doing any more work than that.  This is usually in the context
Russ> where there's some other config management system in use, so
Russ> to them all the packaging format is good for is as a container
Russ> of files with a version number attached.

I've actually had somewhat better experience getting Debian packages
built at work.
At least in environments where we had a fairly pure Debian ecosystem (or
say Debian+windows) I've been able to get people to build packages using
.dscs.  And if I set up the gitlab CI integration, that works fairly
well.

At my current job we even have interest in making some of the stuff
comply with policy and actually make its way back into Debian.
And I think we're possibly getting close to activation energy to do some
of that.

But the challenge is that the quality bar you need for a local
environment is much lower than for inclusion in a global OS.

I think the only reason I've had different experience than Russ is two
factors.  First, I didn't show people fpm (and they weren't aware).
Second, I think the environments I'm talking about are a lot smaller
than Russ where some of the social concerns really do matter.

At the first Cloud sprint we saw something similar.  The folks from
Google wanted to talk to us about getting some of their cloud APIs into
Debian.
But they wanted an approach that allowed them to build for all
distributions.  They wanted us to accept fpm input as source.
They didn't actually want to do the work that is Debian specific in
packaging.
I ended up saying that unfortunately, unless someone wanted to commit to
doing that work, it wasn't a package we as a community want to accept.
I tried to be more polite than that and to explain why those were our
norms.

To be clear, I'm sure there are a lot of ways our processes and
documentation can be improved.
However there are some reasons why what we're asking people to do is
hard.  And I don't think we want to give up on those.

Re: Difficult Packaging Practices

[Vincent Bernat]

 ❦ 27 mai 2019 16:15 +10, Ben Finney :

>> If you just want to get upstream's idea of their package onto a system
>> with their release schedule and their recommended dependency versions,
>> there are better ways than getting a package into Debian.
>
> In the Debian mentors forum (that is, the chat channel, the mailing
> list, etc.) we make a point of saying: that's fine!
>
> Not every package needs to be in Debian, for its users to install that
> package. Someone who wants what you describe above can learn how to make
> a Debian package that will only be side-loaded from that community's own
> repository, or whatever.

People using tools like fpm will never get familiar with our tools and
will never be contributors.
-- 
The man who sets out to carry a cat by its tail learns something that
will always be useful and which never will grow dim or doubtful.
-- Mark Twain


signature.asc
Description: PGP signature

Re: Difficult Packaging Practices

[Ben Finney]

Sam Hartman  writes:

> If you just want to get upstream's idea of their package onto a system
> with their release schedule and their recommended dependency versions,
> there are better ways than getting a package into Debian.

In the Debian mentors forum (that is, the chat channel, the mailing
list, etc.) we make a point of saying: that's fine!

Not every package needs to be in Debian, for its users to install that
package. Someone who wants what you describe above can learn how to make
a Debian package that will only be side-loaded from that community's own
repository, or whatever.

Perhaps that needs to be stated louder or more consistently?

-- 
 \   “Truth is stranger than fiction, but it is because fiction is |
  `\ obliged to stick to possibilities, truth isn't.” —Mark Twain, |
_o__)  _Following the Equator_ |
Ben Finney

Re: Difficult Packaging Practices

[Russ Allbery]

Sam Hartman  writes:

> I think it's a combination of a lot of things.  We have high standards,
> a lot of complexity, and you have to get most or all of that right to
> contribute.  You have to have a package that meets our standards.  You
> have to have a copyright file that meets our standards.  You have to be
> able to figure out our processes.  You have to be willing to follow our
> processes.  And you eventually have to deal with the PGP mess.

> If you don't find value in the things where we have high standards,
> Debian doesn't make a lot of sense.  If you just want to get upstream's
> idea of their package onto a system with their release schedule and
> their recommended dependency versions, there are better ways than
> getting a package into Debian.

This is my experience as well.  I've occasionally tried to get people at
work (at various jobs) interested in packaging software for Debian,
without all that much success.  The problem seems less any one specific
thing and more that they're perfectly content with a Debian package
created by running fpm on some install tree, and don't see the point in
doing any more work than that.  This is usually in the context where
there's some other config management system in use, so to them all the
packaging format is good for is as a container of files with a version
number attached.

It's not that they don't understand the merits of having what they think
of as the base operating system properly maintained and integrated; it's
more that they don't see any value in doing that work for the incremental
thing that they're adding.  They cobble together some combination of local
config management and a deployment method until it works and then move on
to some (from their perspective) more interesting problem.

-- 
Russ Allbery (r...@debian.org)   

Re: Difficult Packaging Practices

[Jérémy Lal]

Le dim. 26 mai 2019 à 23:01, Sam Hartman  a écrit :

> > "Adrian" == Adrian Bunk  writes:
>
> Adrian> It is a problem for people making their first contributions
> Adrian> to Debian to get them into unstable. The problem here is
> Adrian> lack of sponsors willing to do proper reviews and then
> Adrian> uploads. Usually the package in question is already using
> Adrian> dh.
>
> My experience is consistent with the above.
> I can't send links because most of my conversations with people getting
> started contributing to Debian  have been that: actual conversations
> with lungs compressing air to blow out mouths.
>
> Unfortunately, I've found that it's often a combination of factors.
>
> The one first cited is  the difficulty finding a sponsor.
>

There's a gap between getting some unknown software into debian,
and getting well-known software into debian.
Some well-determined upstream develop can manage to get software
into debian, and even get it into stable, and then just quit maintaining it
!
On the other hand, when a fellow DD uploads a software it usually means
that piece is worth it. So i'm not that keen on making it easier to get any
kind
of software into debian.


> But in at least some cases it's more complex than that.
>
> for example in one case someone was talking about finding a sponsor.  I
> said that I didn't normally sponsor packages I couldn't adequately test,
> so that limited what I'd sponsor, but if the contributor could find
> something within that set I'd sponsor it.
>
> He came up with something.
> Then he said but really the hard part there was not the sponsorship, but
> the dependency problem
> Apparently upstream had forked some library already in Debian and you
> had to use the fork for the package to work.
>

It might also mean the upstream software is still maturing and not ready for
prime time.

I think it's a combination of a lot of things.  We have high standards,
> a lot of complexity, and you have to get most or all of that right to
> contribute.  You have to have a package that meets our standards.  You
> have to have a copyright file that meets our standards.


When it's hard to make upstream conform to DFSG it's either easy to fix,
or impossible to fix. So it's not that hard to deal with.


> You have to be
> able to figure out our processes.  You have to be willing to follow our
> processes.  And you eventually have to deal with the PGP mess.
>

Are you referring to the identity check ?
That mess is onto the uploader's hand.
You don't need to have your identity checked as an upstream author, and
the identity check is the best part of subscribing to a community.


> If you don't find value in the things where we have high standards,
> Debian doesn't make a lot of sense.
> If you just want to get upstream's idea of their package onto a system
> with their release schedule and their recommended dependency versions,
> there are better ways than getting a package into Debian.
>

There are even ways that are supported by software available in Debian !
(thinking of flatpak but many other ways to allow users to install software
easily).

--
Jérémy