Bug#762638: metaconfig source control/distribution and Debian's DFSG

[Dominic Hargreaves]

Hi all,

Apologies for another long delay in replying; other matters have
taken up most of my recent energy. I'm replying to both
H.Merijn Brand and Andy Dougherty's most recent messages on the
subject here.

I've snipped some quoted parts for brevity, but kept what I hope
are the most pertinent bits of conversation, to help summarise.

On Sat, Oct 04, 2014 at 08:23:10PM +0200, H.Merijn Brand wrote:
> On Sat, 4 Oct 2014 17:50:51 +0100, Dominic Hargreaves 
> > On Thu, Sep 25, 2014 at 09:07:59PM -0400, Andy Dougherty wrote:

> > > If we include all the units in the source tree
> > > instead, I think end users will have an expectation of greater support.
> > > They will expect it to work, and work easily, right out of the box.
> > > Getting it to that point will require significant effort, and I really
> > > don't see the benefit.  Of course, if someone else wants to do it and
> > > support it, that's fine with me.
> > 
> > One benefit would be that we wouldn't this acknowledged bus factor
> > issue with the metaconfig wrangling.
> 
> Disagree. Areas in the perl core are picked up by people that like that
> area:
> 
> • Threading
> • 64bit/32bit
> • Regex
> • COW
> • Speed
> • Debugging
> • Toolchain
> • Configuration
> • …
> 
> It just happens that Configuration is NOT an area that appeals to
> developers. It works, and they are happy it works the way it does.
> Improving the configuration or the configuration process does NOT make
> the language itself more appealing or more interesting, and eventually,
> that is what the developers want (to achieve).
> 
> Yes, the bus-factor is (too) low, but it is not caused by
> (un)availability of the pieces

Okay, I accept this.
 
> > > > If we have to, we can make a tarball of that repository and include
> > > > it with the perl source package, but it seems like it would we should
> > > > explore the possibility of fixing this discrepancy upstream, rather
> > > > than working around it in Debian. In fact, it's likely we'd also have
> > > > to supply the patches between the current metaconfig output and what's
> > > > actually in the perl release tarball, since Configure is explicitly
> > > > allowed to be patched even though it's generated.
> > > 
> > > To be complete, you would have to include both perl's metaconfig
> > > repository and an appropriately patched 'dist' source tree, which would
> > > likely differ somewhat from Debian's existing 'dist' tree.  If the
> > > intent is to actually use it, there are probably a bunch of installation
> > > issues to work out.  If you want to check your results, you'll have to
> > > confront the reshuffling problem.
> > 
> > > Short summary:  I think it would be a lot of work for very little
> > > gain.
> > 
> > On Fri, Sep 26, 2014 at 08:34:58AM +0200, H.Merijn Brand wrote:
> > 
> > > The main problem with generating Configure from meta is not the
> > > availability of the metaunits used for perl, but the way meta/dist has
> > > to be installed to make them work the way it does.
> > 
> > Do you mean the fact that dist is being patched, or some other reason?
> 
> Just like learning a new language, the whole *process* is rather
> complicated. It is not like most GNU tools "sh autogen.sh",
> "configure", "make check", "make install", but - for people building
> perl - "Configure", "make test", "make install"
> 
> The source code for autogen is widely available, but it works fine only
> on Linux. (I'd invite you to the hell of using the auto* and (even
> worse) libtool on OS's like AIX and HP-UX.
> 
> To make the developers' life of our Perl community *less* of a hell, we
> are shipping "Configure", which is a autoconf/autogen/configure in one,
> that runs on almost every *nix like platform (VMS and Windows have
> special cased support).
> 
> For how many of the GNU tools that you ship, do you include the
> autogen.sh stuff that is only used *before* the configure is included
> in the source package? I only see the auto-stuff if I clone the source
> repo for the tool itself.

Most of the autotools stuff in Debian ships the source files and
it's generally encourages to rerun autoconf/automake to generate
configure these days, so that changes needed to support new architectures,
etc, can be reflected without any work on the source of the package.

> The fact that we patched and extended meta is NO reason for it to be
> more complex than using meta/dist from scratch. It is like installing a
> patched version of autoconf. The changes are reasonable well
> documented, and the source is widely and openly available.

Or at least it is now, thank you :)

> > > As the current maint for this process, I communicate with the
> > > maintainer of dist/meta on a rather regular basis. I give feedback of
> > > what we changed, and I look at the changes he makes to see if those
> > > will be usable for perl.
> > > 
> > > As part of the metaunits is still used from dist, part is a set of
> > > (slightly) modified units from dist, and part is the un

Bug#762638: metaconfig source control/distribution and Debian's DFSG

[Dominic Hargreaves]

On Sat, Nov 01, 2014 at 02:41:58PM +, Dominic Hargreaves wrote:
> I will propose within Debian that we package up the modified dist
> package and metaconfig units, and include documentation in the perl
> source package about this.

After discussions with Helmut at Debconf, it seems like all we can
reasonably do in Debian is to make it clear that the current situation
is explicitly supported by Debian, rather than spending effort with
metaconfig. As such, I propose we close this bug with something like the
attached documentation patch.

Cheers,
Dominic.
>From 72c6dc60df6cb46a77986f96ed29643a8ec9b55c Mon Sep 17 00:00:00 2001
From: Dominic Hargreaves 
Date: Mon, 17 Aug 2015 18:45:25 +0200
Subject: [PATCH] Document the special case of modifying Configure in in
 debian/README.Source (Closes: #762638)

---
 debian/README.source | 12 
 debian/changelog |  7 +++
 2 files changed, 19 insertions(+)

diff --git a/debian/README.source b/debian/README.source
index f4b7dbb..7cbbba8 100644
--- a/debian/README.source
+++ b/debian/README.source
@@ -125,6 +125,18 @@ then
 prove debian/t/*.t
 fi
 
+A special note on patching Configure
+
+
+The Configure file is a special case of a file which is autogenerated
+upstream by specialised tools, but which upstream explicitly declares
+it as being (a) preferred form of modification. As far as Debian goes,
+this means that we should generally not try to regenerate it, but patch
+it directly (forwarding patches upstream when needed).
+
+Please see 
+for a discussion about this issue.
+
 Credits
 ---
 
diff --git a/debian/changelog b/debian/changelog
index 14f18df..539206c 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+perl (5.22.0-3) UNRELEASED; urgency=medium
+
+  * Document the special case of modifying Configure in
+in debian/README.Source (Closes: #762638)
+
+ -- Dominic Hargreaves   Mon, 17 Aug 2015 18:44:44 +0200
+
 perl (5.22.0-2) experimental; urgency=medium
 
   * Drop the ExtUtils::MakeMaker changes we've been carrying for much too long
-- 
2.1.4

Bug#762638: metaconfig source control/distribution and Debian's DFSG

[Dominic Hargreaves]

Hi,

I'm including Helmut, the submitter to the Debian bug report, as a
CC, in case he can help provide additional context that I'm missing.

On Thu, Sep 25, 2014 at 09:07:59PM -0400, Andy Dougherty wrote:
> On Thu, Sep 25, 2014 at 10:16:32PM +0100, Dominic Hargreaves wrote:
> > Hi,
> > 
> > The Debian project has recently received a bug report indicating that
> > the perl source package we distribute does not comply the the DFSG[1]
> > since the preferred form of modification of Configure is not provided.
> > You can see the bug report at [2].
> 
> I'd quibble with that -- though I am no longer the primary Configure
> maintainer, I actually did prefer most users to directly modify Configure.
> It ended up being much easier than supporting users using metaconfig to
> generate Configure.  Users are of course welcome to fetch and install
> metaconfig and use it, but I wouldn't say that was the "preferred"
> form.

I don't have the exact details, but Helmut suggests that this wasn't
the case for him, as the changes would have had to be made in several
places. In general I think a healthy free software project should aim to
minimise cases where a casual hacker would be discouraged from doing
things the same way a seasoned developer would.

> Generating Configure requires both the 'dist' package (which contains
> metaconfig) and the perl-specific metaconfig units.  Historically, we have
> often ended up using a patched 'dist', or a version that lagged behind
> the main package.  We definitely can't simply assume the version in the
> Debian archive will work.  Generating Configure also sometimes involves
> patching or fixing the output of metaconfig in various ways.  All the
> information to do this should be in perl's metaconfig git repository.
> 
> It is also sometimes difficult to check or merge the generated output.
> The metaconfig units are assembled in order based on dependencies,
> but those dependencies don't fully constrain the order.  The result is
> that the order of units can vary depending on who generates Configure.
> Thus even a tiny patch to a single metaconfig unit can result in a
> Configure patch with hundreds of lines that are just shuffling
> parts around.
> 
> That's another reason why modifying the metaconfig units is not
> necessarily the "preferred" form of modification of Configure.
> 
> Obviously, it's all a bit involved, and a fair amount of effort to
> figure out.
> 
> The way it's set up now, we encourage people to simply patch Configure.
> If someone wants to go the metaconfig route instead, it's a lot of
> extra work, but presumably the person deliberately chooses that route
> knowing it's extra work.

Again, there is an ideological point here. It *shouldn't* be a lot of
extra work to do things in the way that upstream developers would.
Clearly perl isn't going to be kicked out of Debian because of this,
but a less important package might well be.

> If we include all the units in the source tree
> instead, I think end users will have an expectation of greater support.
> They will expect it to work, and work easily, right out of the box.
> Getting it to that point will require significant effort, and I really
> don't see the benefit.  Of course, if someone else wants to do it and
> support it, that's fine with me.

One benefit would be that we wouldn't this acknowledged bus factor
issue with the metaconfig wrangling.
 
> > If we have to, we can make a tarball of that repository and include
> > it with the perl source package, but it seems like it would we should
> > explore the possibility of fixing this discrepancy upstream, rather
> > than working around it in Debian. In fact, it's likely we'd also have
> > to supply the patches between the current metaconfig output and what's
> > actually in the perl release tarball, since Configure is explicitly
> > allowed to be patched even though it's generated.
> 
> To be complete, you would have to include both perl's metaconfig
> repository and an appropriately patched 'dist' source tree, which would
> likely differ somewhat from Debian's existing 'dist' tree.  If the
> intent is to actually use it, there are probably a bunch of installation
> issues to work out.  If you want to check your results, you'll have to
> confront the reshuffling problem.

> Short summary:  I think it would be a lot of work for very little
> gain.


On Fri, Sep 26, 2014 at 08:34:58AM +0200, H.Merijn Brand wrote:

> The main problem with generating Configure from meta is not the
> availability of the metaunits used for perl, but the way meta/dist has
> to be installed to make them work the way it does.

Do you mean the fact that dist is being patched, or some other reason?
 
> As Andy already stated, it proves to be much easier to "backport" the
> direct patches to Configure and friends to meta than it is to instruct
> (new) users to patch metaunits the way it should. Even seasoned users
> seem to have problems doing so. I had two sessions past year explaining

Bug#762638: metaconfig source control/distribution and Debian's DFSG

[H.Merijn Brand]

On Sat, 4 Oct 2014 17:50:51 +0100, Dominic Hargreaves 
wrote:

> Hi,
> 
> I'm including Helmut, the submitter to the Debian bug report, as a
> CC, in case he can help provide additional context that I'm missing.
> 
> On Thu, Sep 25, 2014 at 09:07:59PM -0400, Andy Dougherty wrote:
> > On Thu, Sep 25, 2014 at 10:16:32PM +0100, Dominic Hargreaves wrote:
> > > Hi,
> > > 
> > > The Debian project has recently received a bug report indicating that
> > > the perl source package we distribute does not comply the the DFSG[1]
> > > since the preferred form of modification of Configure is not provided.
> > > You can see the bug report at [2].
> > 
> > I'd quibble with that -- though I am no longer the primary Configure
> > maintainer, I actually did prefer most users to directly modify Configure.
> > It ended up being much easier than supporting users using metaconfig to
> > generate Configure.  Users are of course welcome to fetch and install
> > metaconfig and use it, but I wouldn't say that was the "preferred"
> > form.
> 
> I don't have the exact details, but Helmut suggests that this wasn't
> the case for him, as the changes would have had to be made in several
> places. In general I think a healthy free software project should aim to
> minimise cases where a casual hacker would be discouraged from doing
> things the same way a seasoned developer would.
> 
> > Generating Configure requires both the 'dist' package (which contains
> > metaconfig) and the perl-specific metaconfig units.  Historically, we have
> > often ended up using a patched 'dist', or a version that lagged behind
> > the main package.  We definitely can't simply assume the version in the
> > Debian archive will work.  Generating Configure also sometimes involves
> > patching or fixing the output of metaconfig in various ways.  All the
> > information to do this should be in perl's metaconfig git repository.
> > 
> > It is also sometimes difficult to check or merge the generated output.
> > The metaconfig units are assembled in order based on dependencies,
> > but those dependencies don't fully constrain the order.  The result is
> > that the order of units can vary depending on who generates Configure.
> > Thus even a tiny patch to a single metaconfig unit can result in a
> > Configure patch with hundreds of lines that are just shuffling
> > parts around.
> > 
> > That's another reason why modifying the metaconfig units is not
> > necessarily the "preferred" form of modification of Configure.
> > 
> > Obviously, it's all a bit involved, and a fair amount of effort to
> > figure out.
> > 
> > The way it's set up now, we encourage people to simply patch Configure.
> > If someone wants to go the metaconfig route instead, it's a lot of
> > extra work, but presumably the person deliberately chooses that route
> > knowing it's extra work.
> 
> Again, there is an ideological point here. It *shouldn't* be a lot of
> extra work to do things in the way that upstream developers would.
> Clearly perl isn't going to be kicked out of Debian because of this,
> but a less important package might well be.
> 
> > If we include all the units in the source tree
> > instead, I think end users will have an expectation of greater support.
> > They will expect it to work, and work easily, right out of the box.
> > Getting it to that point will require significant effort, and I really
> > don't see the benefit.  Of course, if someone else wants to do it and
> > support it, that's fine with me.
> 
> One benefit would be that we wouldn't this acknowledged bus factor
> issue with the metaconfig wrangling.

Disagree. Areas in the perl core are picked up by people that like that
area:

• Threading
• 64bit/32bit
• Regex
• COW
• Speed
• Debugging
• Toolchain
• Configuration
• …

It just happens that Configuration is NOT an area that appeals to
developers. It works, and they are happy it works the way it does.
Improving the configuration or the configuration process does NOT make
the language itself more appealing or more interesting, and eventually,
that is what the developers want (to achieve).

Yes, the bus-factor is (too) low, but it is not caused by
(un)availability of the pieces

> > > If we have to, we can make a tarball of that repository and include
> > > it with the perl source package, but it seems like it would we should
> > > explore the possibility of fixing this discrepancy upstream, rather
> > > than working around it in Debian. In fact, it's likely we'd also have
> > > to supply the patches between the current metaconfig output and what's
> > > actually in the perl release tarball, since Configure is explicitly
> > > allowed to be patched even though it's generated.
> > 
> > To be complete, you would have to include both perl's metaconfig
> > repository and an appropriately patched 'dist' source tree, which would
> > likely differ somewhat from Debian's existing 'dist' tree.  If the
> > intent is to actually use it, there are probably a bunch of installation
> 

Bug#762638: metaconfig source control/distribution and Debian's DFSG

[Helmut Grohne]

Dear Perl maintainers,

On Sat, Oct 04, 2014 at 05:50:51PM +0100, Dominic Hargreaves wrote:
> I'm including Helmut, the submitter to the Debian bug report, as a
> CC, in case he can help provide additional context that I'm missing.

Yes, I do have additional context.

> I don't have the exact details, but Helmut suggests that this wasn't
> the case for him, as the changes would have had to be made in several
> places. In general I think a healthy free software project should aim to
> minimise cases where a casual hacker would be discouraged from doing
> things the same way a seasoned developer would.

I have to admit that I am no longer 100% sure that I do indeed want to
modify metaconfig units rather than just patching Configure, but given
my experience with autoconf that initially seemed like the logical way
to do it.

I can give background on what I would like to see changed about Perl's
Configure, but it'll be a longer read and it may not help solving the
question as to whether metaconfig is to be considered part of the Perl
source (from a Debian pov). The one extra bit I can give here is that
reading metaconfig units helps with understanding Configure. I do not
understand much of the other details involved, so the rest of this mail
concentrates on my particular use case:

Cross building Perl is hard. When Perl is cross built, it most often is
a retrospective. In other words, you built it natively and collected the
Configure results to be able to reproduce them as part of a real cross
build. I am aware of two cross building efforts for Perl.  One is Neil
William's perl-cross-debian[1] and the other is OpenWRT, which
apparently provides a microperl. Neil William's effort basically gathers
the Configure results an treats them as source. Since these results
change with Perl releases, perl-cross-debian is hard to keep up to date
and it does not work with any non-ARM architectures at the moment.
Ideally, Perl would cross just like any other language, but we are very
very far from that.

So one way to make crossing Perl more feasible is to reduce the effort
of maintaining the Configure results and one way to reduce that effort
is to reduce the number of checks that need to execute host[2] arch
code.  Indeed, there is very much low hanging fruit here. Checks such as
the *size checks or d_open3 and o_nonblock can be turned into
compile-only checks (i.e. without running host[2] arch code). Since the
*size checks are very much repetitive, it seemed natural to me that
modifying metaconfig is the way to go. Once a good portion of these
checks is turned into compile-only checks, the remaining checks may
become maintainable enabling reasonable cross compilation.

The documented method of cross compiling Perl is a bad joke. It presumes
that one actually has a host[2] architecture machine, but for a long time
arm64 and ppc64el hardware was unavailable and it is still hard to come
by today.  It also presumes that one has a ssh server on the host[2]
architecture machine, but building openssh requires Perl. So in my view,
Perl currently makes cross compilation unnecessarily hard and the place
to fix that is Configure.

Helmut

[1] https://github.com/codehelp/perl-cross-debian
[2] I am using GNU terminology here, because it is most widely accepted.
Perl's documentation refers to this as "target".


-- 
To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Bug#762638: metaconfig source control/distribution and Debian's DFSG

[Andy Dougherty]

On Sat, Oct 04, 2014 at 05:50:51PM +0100, Dominic Hargreaves wrote:

> On Thu, Sep 25, 2014 at 09:07:59PM -0400, Andy Dougherty wrote:
> > On Thu, Sep 25, 2014 at 10:16:32PM +0100, Dominic Hargreaves wrote:

> > The way it's set up now, we encourage people to simply patch Configure.
> > If someone wants to go the metaconfig route instead, it's a lot of
> > extra work, but presumably the person deliberately chooses that route
> > knowing it's extra work.
> 
> Again, there is an ideological point here. It *shouldn't* be a lot of
> extra work to do things in the way that upstream developers would.
> Clearly perl isn't going to be kicked out of Debian because of this,
> but a less important package might well be.

I think we may have confused you here such that you have it exactly
backwards.  It *isn't* a lot of extra work for anyone to do things in
the way the upstream developers would.  Everyone has access to the exact
same source.  What a few upstream developers do have is *experience*
using the package, so that they can do that work somewhat more easily.

Those few upstream developers (well, really only H.Merijn these days) have
volunteered to help do that work for people who would rather spend their
time fixing something else rather than learning a complete configuration
system.  Thus if a casual hacker wants to make a simple Configure change,
they can simply make it directly to Configure.  Thus it is *easier* for
the casual hacker to get involved.  If they want to do it the way the
upstream developer would, then they have to do the same hard work that
the upstream deveveloper does, and they are certainly free to do so.

Both the dist subversion repository and the perl metaconfig git
repository are freely available, so anyone can check out the appropriate
versions to recreate Configure (subject to machine-dependent ordering).
I hadn't realized that the precise versions used weren't clearly labeled
because I don't recall anybody ever asking before.  Encouraged by this
request, I will try to remember and document that more clearly in perl's
documentation.  If someone else wants to do it first, the patches would
certainly be welcome.

> Okay, so clearly from a pragmatic view we would need to ship our own version
> of dist along with the rest of the stuff from the metaconfig repository.
> Depressingly this violates another part of Debian policy relating to
> embedding copies of code not intended to be embedded, but the freedom
> to modify the code using the preferred form clearly overrides that in
> my mind.

You could instead separately package dist-3.5.20 and depend on that,
if you liked.  (Presumably, that package would point to the standard
'dist' package as the recommended starting point for new projects,
and would not overwrite the standard 'dist' package files.)  I don't
see how this situation is fundamentally any different from that of any
other program built with a particular version of an external tool (such
as bison, for example).

> > Rebuilding Configure would not be easy or automatic, but all the necessary
> > files would be available.  Would that satisfy the Debian guidelines?
> 
> I think it would satisfy the letter of the law, if not the full spirit.

I'm sorry, but I really fail to see why, and suspect that there is
a lingering misunderstanding.  Rebuilding Configure is not easy or
automatic for *anyone*, but that's not an issue of freedom.  I want to
be helpful and share what we have, but we can only share what we have.

Cheers,

-- 
Andy Dougherty  dough...@lafayette.edu


-- 
To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org