Re: Getting dh_install to do what we need

[Gergely Nagy]

Goswin von Brederlow  writes:

>>> My implementation copies the file to the desired destination, which may
>>> or may not be a good idea - I'll do some more tests to see which one's
>>> less painful and more safe.
>>
>> That breaks -X, --fail-missing, --list-missing, --sourcedir, and --tmpdir
>
> That is part of what I fear will happen with executable config files.
>
> How do you get access to the parameters passed to dh_*? Scripts my need
> to honor them.

For now, my latest implementation should not break -X, --fail-missing
and --list-missing, the three options out of five that are used
regularly.

It won't work with --sourcedir and --tmpdir, though, and that's
something I can't change, unless debhelper starts to pass down options,
or export them via an environment variable.

These are corner cases, though, which can be appropriately
documented. Neither --sourcedir nor --tmpdir is all that common I think,
so the number of cases where one would want to use an executable
.install file together with either of the two problematic options would
be minimal.

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/874nx07rkh.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Goswin von Brederlow]

Joey Hess  writes:

> Gergely Nagy wrote:
>> At the moment, I have something that works like this:
>> 
>> ,
>> | #! /usr/bin/dh-exec-install
>> | # The next one will simply echo it back to dh_install
>> | source-file /dest-dir/
>> |
>> | # This one will copy the file itself, following similar heuristics as
>> | # dh_install: it will first try the source file directly, and if it's
>> | # not found, try the same path under debian/tmp/. The destination is
>> | # relative to debian/${PACKAGE} (as per dh compat level 7+)
>> | #
>> | # Since dh-exec-install does the copying itself, this line is NOT
>> | # echoed back to dh_install.
>> | source-file /dest-dir/new-name
>> `
>
> Of course the reason I didn't add this to dh_install 10 years ago is
> that this syntax sucks. It's really horrible; either the trailing slashes
> are much more significant than makes sense, or what it does depends on
> the state of the filesystem(ie, checking whether /dest-dir/new-name is
> a directory).

I disagree that it is horrible. Yes, trailing slashes become much more
significant. But that is already common behaviour for tools like cp,
scp, rsync, ... And having to rename outside of dh_install or replacing
dh_install completly is usualy worse.

>> My implementation copies the file to the desired destination, which may
>> or may not be a good idea - I'll do some more tests to see which one's
>> less painful and more safe.
>
> That breaks -X, --fail-missing, --list-missing, --sourcedir, and --tmpdir

That is part of what I fear will happen with executable config files.

How do you get access to the parameters passed to dh_*? Scripts my need
to honor them.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87hb106e29.fsf@frosties.localnet

Re: Getting dh_install to do what we need

[Gergely Nagy]

Josselin Mouette  writes:

> Le vendredi 09 décembre 2011 à 11:50 -0400, Joey Hess a écrit : 
>> I have made hundreds of changes to debhelper that broke buggy packages
>> without using compat levels; that is not what compat levels are for.
>
> So, breaking a dozen packages with +x on their files in debian/ is fine,
> but breaking 0 packages by adding variable substitution in them is
> not?

Those dozen packages were slightly buggy to begin with. Fixing them to
remove the +x is a good thing, which should've been done anyway.

I still did not see any compelling argument against executable files,
except for personal distaste, which is, well, an opinion only.

There's plenty of examples in the archive where great flexibility was
given to maintainers, and they did not abuse it. I can't see why this
would be an exception.

(There's also cases of much more brain damage in various packages all
over the archive, which noone seems to mind all that much. Just look at
how long yada survived.)

-- 
|8]


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8762hibtpr@luthien.mhp

Re: Getting dh_install to do what we need

[Gergely Nagy]

Josselin Mouette  writes:

> Le vendredi 09 décembre 2011 à 16:05 +0100, Gergely Nagy a écrit : 
>> 'lo and behold, I started implementing dh-subst, and while it's fairly
>> rough and underdocumented, there's some usable code out there in my
>> github repo[1].
>
> If it’s a joke, it’s a bad one.
> Two wrongs don’t make a right. We need to find a way to fix debhelper,
> not to work around its brokenness.

Your perception of wrong is not universal.

There's nothing to fix in debhelper (you're free to ignore executable
config files, and just do your own magic, noone's forcing you to use
this feature).

-- 
|8]


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87aa6ubtxu@luthien.mhp

Re: Getting dh_install to do what we need

[Josselin Mouette]

Le vendredi 09 décembre 2011 à 11:50 -0400, Joey Hess a écrit : 
> I have made hundreds of changes to debhelper that broke buggy packages
> without using compat levels; that is not what compat levels are for.

So, breaking a dozen packages with +x on their files in debian/ is fine,
but breaking 0 packages by adding variable substitution in them is not?

-- 
 .''`.  Josselin Mouette
: :' :
`. `'
  `-


signature.asc
Description: This is a digitally signed message part

Re: Getting dh_install to do what we need

[Josselin Mouette]

Le vendredi 09 décembre 2011 à 16:05 +0100, Gergely Nagy a écrit : 
> 'lo and behold, I started implementing dh-subst, and while it's fairly
> rough and underdocumented, there's some usable code out there in my
> github repo[1].

If it’s a joke, it’s a bad one.
Two wrongs don’t make a right. We need to find a way to fix debhelper,
not to work around its brokenness.

-- 
 .''`.  Josselin Mouette
: :' :
`. `'
  `-


signature.asc
Description: This is a digitally signed message part

Re: Getting dh_install to do what we need

[Gergely Nagy]

Joey Hess  writes:

> Gergely Nagy wrote:
>> At the moment, I have something that works like this:
>> 
>> ,
>> | #! /usr/bin/dh-exec-install
>> | # The next one will simply echo it back to dh_install
>> | source-file /dest-dir/
>> |
>> | # This one will copy the file itself, following similar heuristics as
>> | # dh_install: it will first try the source file directly, and if it's
>> | # not found, try the same path under debian/tmp/. The destination is
>> | # relative to debian/${PACKAGE} (as per dh compat level 7+)
>> | #
>> | # Since dh-exec-install does the copying itself, this line is NOT
>> | # echoed back to dh_install.
>> | source-file /dest-dir/new-name
>> `
>
> Of course the reason I didn't add this to dh_install 10 years ago is
> that this syntax sucks. It's really horrible; either the trailing slashes
> are much more significant than makes sense, or what it does depends on
> the state of the filesystem(ie, checking whether /dest-dir/new-name is
> a directory).

Mhm, fair enough. I'll scratch that for now then, and try to figure out
something else.

>> My implementation copies the file to the desired destination, which may
>> or may not be a good idea - I'll do some more tests to see which one's
>> less painful and more safe.
>
> That breaks -X, --fail-missing, --list-missing, --sourcedir, and --tmpdir

Ow. That's not good. Thanks for pointing these out!

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87k4604x3l.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Joey Hess]

Gergely Nagy wrote:
> At the moment, I have something that works like this:
> 
> ,
> | #! /usr/bin/dh-exec-install
> | # The next one will simply echo it back to dh_install
> | source-file /dest-dir/
> |
> | # This one will copy the file itself, following similar heuristics as
> | # dh_install: it will first try the source file directly, and if it's
> | # not found, try the same path under debian/tmp/. The destination is
> | # relative to debian/${PACKAGE} (as per dh compat level 7+)
> | #
> | # Since dh-exec-install does the copying itself, this line is NOT
> | # echoed back to dh_install.
> | source-file /dest-dir/new-name
> `

Of course the reason I didn't add this to dh_install 10 years ago is
that this syntax sucks. It's really horrible; either the trailing slashes
are much more significant than makes sense, or what it does depends on
the state of the filesystem(ie, checking whether /dest-dir/new-name is
a directory).

> My implementation copies the file to the desired destination, which may
> or may not be a good idea - I'll do some more tests to see which one's
> less painful and more safe.

That breaks -X, --fail-missing, --list-missing, --sourcedir, and --tmpdir

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Gergely Nagy]

Goswin von Brederlow  writes:

>>> At a glance what does this do?
>> [...]
>>
>> It triggers my "slap the maintainer silly" button. Other than that, it's
>> dead simple: copy a file from one place to the other (with possibly
>> renaming the file), with the file list following the while loop.
>
> Not quite. It implements support for renaming files during dh_install,
> implementing the following syntax:
>
> file path/
> file path/new_name
>
> The hack it does is to copy the file in the source directory to the new
> name and then letting dh_install install it to the destination
> directory.

Yeah... I realized my mistake a minute after sending the message, but at
that point, I was already trying to come up with a she-bang tool that
helps with this case.

This, though, is a valid point against executable scripts, unless
they become somewhat standardized.

> And yes, like this it is a totaly ugly hack. Does it become better as
> #!/usr/bin/debhelper-exec-rename or something?

At the moment, I have something that works like this:

,
| #! /usr/bin/dh-exec-install
| # The next one will simply echo it back to dh_install
| source-file /dest-dir/
|
| # This one will copy the file itself, following similar heuristics as
| # dh_install: it will first try the source file directly, and if it's
| # not found, try the same path under debian/tmp/. The destination is
| # relative to debian/${PACKAGE} (as per dh compat level 7+)
| #
| # Since dh-exec-install does the copying itself, this line is NOT
| # echoed back to dh_install.
| source-file /dest-dir/new-name
`

My implementation copies the file to the desired destination, which may
or may not be a good idea - I'll do some more tests to see which one's
less painful and more safe.

I'm leaning towards copying to the destination with dh-exec-install
directly, as that avoids the possibility of naming conflicts within the
source (however unlikely that may be).

>> This also gave me an idea, and since this use-case seems to be common,
>> I'll probably rename my dh_subst thing to debhelper-exec-extras or
>> something similar, and include a little tool that will cover this
>> use-case aswell.
>
> What if I want to do more than one thing? For example rename and do the
> variable substitution. Should we have something like
>
> #! /usr/bin/debhelper-exec-pipe debhelper-exec-subst debhelper-exec-rename
>
> which would pipe the input file through debhelper-exec-subst and
> debhelper-exec-rename both?
>
> Does any Debian port have the limit of only one argument for shebangs?

Good question!

I thought about this too, and the current idea is a dh-exec tool, that
wraps the rest, and puts them in the appropriate order that makes sense
in 99% of the cases (with the current two tools: this will be
dh-exec-subst | dh-exec-install, so that we get variable expansion
before copy-stuff). This, however, is not completely implemented
yet. I'll fix that in a couple of hours when I near a computer.

And if so need be, this can be later extended with --with-$tool and
--without-$tool options, so that one can say:

#! /usr/bin/dh-exec --without-subst

Or:

#! /usr/bin/dh-exec --with-install

By the way, the current state of my work (renamed from dh-subst to
dh-exec) is available at https://github.com/algernon/dh-exec/ (the URL
may change, if the project gets renamed again), if you or anyone else
feels like giving it a test ride.

There's even a debian/ dir in there, and a test suite (just don't look
at the test scripts... I need to rewrite those)!

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87vcpl4ps4.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Goswin von Brederlow]

Gergely Nagy  writes:

> Goswin von Brederlow  writes:
>
>>> So, in this case, the difference is negligible, both can be trivially
>>> understood.
>>>
>>> However, it gives more flexibility to the maintainer, to do more complex
>>> stuff, if so needs be. But, that won't be the common case. Why? Because
>>> there's no point in overcomplicating things.
>>
>> Lets hope it stays at the level where you have a shebang (one of few
>> well known tools) followed by the normal input. That would be relatively
>> easy to get used to and understand. So you look at the .install file and
>> then man dh_subst and you are good.
>
> That's the idea. Similarly to the normal dh_* tools, where you look at
> the man page, and you're good. ;)
>
>> At a glance what does this do?
> [...]
>
> It triggers my "slap the maintainer silly" button. Other than that, it's
> dead simple: copy a file from one place to the other (with possibly
> renaming the file), with the file list following the while loop.

Not quite. It implements support for renaming files during dh_install,
implementing the following syntax:

file path/
file path/new_name

The hack it does is to copy the file in the source directory to the new
name and then letting dh_install install it to the destination
directory.

And yes, like this it is a totaly ugly hack. Does it become better as
#!/usr/bin/debhelper-exec-rename or something?

> This also gave me an idea, and since this use-case seems to be common,
> I'll probably rename my dh_subst thing to debhelper-exec-extras or
> something similar, and include a little tool that will cover this
> use-case aswell.

What if I want to do more than one thing? For example rename and do the
variable substitution. Should we have something like

#! /usr/bin/debhelper-exec-pipe debhelper-exec-subst debhelper-exec-rename

which would pipe the input file through debhelper-exec-subst and
debhelper-exec-rename both?

Does any Debian port have the limit of only one argument for shebangs?

 On the other hand we do have packages with executable debhelper files
 that are NOT scripts. Debhelper currently breaks those. The execute
 scripts feature should use a compat level. And then you would have the
 situation you described, where you have to check the compat level and
 every script.
>>>
>>> I'd treat executable files that are not scripts as a bug. They most
>>> probably don't have a shebang line, and that's just wrong.
>>>
>>> Breaking buggy packages is not something I'd worry about. Especially if
>>> the number is fairly low (and I expect it to be so, but feel free to
>>> correct me with numbers).
>>
>> Around 17 sources for 3.0 (quilt) format. So nothing major. Could be
>> some more for native packages.
>
> So, the number is not fairly low, it's so low that it doesn't even
> matter. :P

Yes. Don't make a big deal out of this. It was just tickling my funny
bone. It isn't a showstopper.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87wra1hk7k.fsf@frosties.localnet

Re: Getting dh_install to do what we need

[Gergely Nagy]

Goswin von Brederlow  writes:

>> So, in this case, the difference is negligible, both can be trivially
>> understood.
>>
>> However, it gives more flexibility to the maintainer, to do more complex
>> stuff, if so needs be. But, that won't be the common case. Why? Because
>> there's no point in overcomplicating things.
>
> Lets hope it stays at the level where you have a shebang (one of few
> well known tools) followed by the normal input. That would be relatively
> easy to get used to and understand. So you look at the .install file and
> then man dh_subst and you are good.

That's the idea. Similarly to the normal dh_* tools, where you look at
the man page, and you're good. ;)

> At a glance what does this do?
[...]

It triggers my "slap the maintainer silly" button. Other than that, it's
dead simple: copy a file from one place to the other (with possibly
renaming the file), with the file list following the while loop.

This also gave me an idea, and since this use-case seems to be common,
I'll probably rename my dh_subst thing to debhelper-exec-extras or
something similar, and include a little tool that will cover this
use-case aswell.

>>> On the other hand we do have packages with executable debhelper files
>>> that are NOT scripts. Debhelper currently breaks those. The execute
>>> scripts feature should use a compat level. And then you would have the
>>> situation you described, where you have to check the compat level and
>>> every script.
>>
>> I'd treat executable files that are not scripts as a bug. They most
>> probably don't have a shebang line, and that's just wrong.
>>
>> Breaking buggy packages is not something I'd worry about. Especially if
>> the number is fairly low (and I expect it to be so, but feel free to
>> correct me with numbers).
>
> Around 17 sources for 3.0 (quilt) format. So nothing major. Could be
> some more for native packages.

So, the number is not fairly low, it's so low that it doesn't even
matter. :P

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87zkey3zy1.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Gergely Nagy]

Nicolas Boulenguez  writes:

> 3/ The way debhelper splits its work in small tools does not always
> fit the separation of human concerns. For example, the following is
> IMHO more readable that generating/executing a single debian/*.install
> file in which each line deals with a different library.
>
> # foo stuff
> override_dh_install::
>  dh_install --package=libfoo$(FOO_SOVERSION) 
> lib/libfoo.so.$(FOO_SOVERSION) usr/lib/$(DEB_HOST_MULTIARCH)
> override_dh_link::
>  dh_link --package=libfoo-dev 
> usr/lib/$(DEB_HOST_MULTIARCH)/libfoo.so.$(FOO_SOVERSION) 
> usr/lib/$(DEB_HOST_MULTIARCH)/libfoo.so
> … more foo stuff
>
> # bar stuff
> override_dh_install::
>  dh_install --package=libbar$(BAR_SOVERSION) 
> lib/libbar.so.$(BAR_SOVERSION) usr/lib/$(DEB_HOST_MULTIARCH)
> … more bar stuff
>
> # At the end
> override_dh_install::
>  dh_install --remaining-packages
> override_dh_link::
>  dh_link --remaining-packages
> … and so on

We'll have to disagree here, I find the above overrides awkward, and
harder to follow than this:

debian/rules:
,
| export FOO_SOVERSION=1.0
| %:
|   dh $@
`

debian/libfoo1.0.install:
,
| #!/usr/bin/dh_subst
| lib/libfoo.so.${FOO_VERSION} usr/lib/${DEB_HOST_MULTIARCH}
`

And so on...

Why do I find it more readable? Because there aren't any
overrides. Overrides are a horrible, horrible way to customize a
build. They don't really follow the make flow one normally
expects.

Going the executable script route, you can get rid of the overrides, and
I honestly fail to see how a she-bang line would make it less
understandable.

Okay, okay, you will have to trust dh_subst to do the right thing. But
you already trust debhelper, and dh_subst will be a *very* lightweight
and dead-simple tool.

> In short, I think that debian/*.install dynamic content should be
> discouraged.

I disagree, and think that dynamic debhelper files should be encouraged
over overrides.

But each to his own, I guess. We all can use whichever method we prefer,
as both work, and none of us is forced to use the other method if we
don't want to.

In the end, both ways work, and no matter how I happen to feel about
overrides, neither them, nor executable scripts should be discouraged.

-- 
|8]


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/874nx65ev1.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Nicolas Boulenguez]

If I understand well, this thread deals, among other things, with
different ways to transmit a variable from debian/rules to dh_install.
- via override_dh_install: and command line arguments
- via debian/*.install static content
- via debian/*.install.in content, preprocessed (with environment variables)
- via debian/*.install execution output (based on environment)

1/ From this point of view, FOO_SOVERSION is similar to
DEB_HOST_MULTIARCH. As the soversion is mentioned in the library
binary package name, only the first solution applies to it, unless the
file is preprocessed *and renamed*.

override_dh_install:
 dh_install --package=libfoo$(FOO_SOVERSION) lib/libfoo.so.$(FOO_SOVERSION) 
usr/lib/$(DEB_HOST_MULTIARCH)
 dh_install

2/ As a novice packager, I had a hard time figuring the execution flow
of the debhelper tools, and specially variable values transmission
across the hierarchy of subprocesses, even if each step is well
documented. It is quite instructive to read, in this very thread,
experimented DD arguing whether some dpkg-buildflag variable is
available to a particular program called by a dh_tool at a specific
compatibility level, and having to run the program once to know for
sure… The first solution provides, in debian/rules, the equivalent of
a method call with an explicit list of named actual parameters. In
this analogy, environment variables are global variables, and should
be avoided when possible.

3/ The way debhelper splits its work in small tools does not always
fit the separation of human concerns. For example, the following is
IMHO more readable that generating/executing a single debian/*.install
file in which each line deals with a different library.

# foo stuff
override_dh_install::
 dh_install --package=libfoo$(FOO_SOVERSION) lib/libfoo.so.$(FOO_SOVERSION) 
usr/lib/$(DEB_HOST_MULTIARCH)
override_dh_link::
 dh_link --package=libfoo-dev 
usr/lib/$(DEB_HOST_MULTIARCH)/libfoo.so.$(FOO_SOVERSION) 
usr/lib/$(DEB_HOST_MULTIARCH)/libfoo.so
… more foo stuff

# bar stuff
override_dh_install::
 dh_install --package=libbar$(BAR_SOVERSION) lib/libbar.so.$(BAR_SOVERSION) 
usr/lib/$(DEB_HOST_MULTIARCH)
… more bar stuff

# At the end
override_dh_install::
 dh_install --remaining-packages
override_dh_link::
 dh_link --remaining-packages
… and so on

In short, I think that debian/*.install dynamic content should be
discouraged.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111212013038.GA2955@pegase

Re: Getting dh_install to do what we need

[Goswin von Brederlow]

Peter Samuelson  writes:

> [Kees Cook]
>> This doesn't work with source-format-1 packages without adding
>> "chmod" lines for the scripted debhelper config files in the rules
>> file. Perhaps this isn't a big deal since we should all be using
>> source-format-3 anyway.
>
> We should?  I prefer to think of this whole debacle as yet another
> reason to never switch to format 3.  So long as I stay with format 1,
> "features" like this one will never come back to bite me.
>
> (Well, yes, it could still happen to me with native packages.  But
> that's a much smaller attack surface than if I used format 3.)

This has nothing to do with the source format you use other than that
tarballs will retain the x flags and diff.gz will not.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87obvh2w52.fsf@frosties.localnet

Re: Getting dh_install to do what we need

[Goswin von Brederlow]

Gergely Nagy  writes:

> Goswin von Brederlow  writes:
>
>>> Compared to writing overrides, it's less effort. Compared to just
>>> writing the variable and expecting it to work, it's two commands more. I
>>> believe that's not much.
>>>
>>> On the other hand, though, making it obvious that it's a script, and
>>> there's magic (= variable substitution) going on, it's much more clear
>>> to have it executable. Otherwise you'd have to look at the compat level
>>> to see if it's supposed to do some tricks or not.
>>>
>>> I don't know about you, but I hardly ever look at compat levels. An
>>> executable file in debian/ is much easier for me to notice.
>>
>> Would you really need to look at the compat level? Wouldn't seeing a
>> ${DEB_HOST_MULTIARCH} in the .install file be obvious to everyone that
>> there is substitution going on and what it does? It is a rather common
>> syntax that should be familiar to every maintainer. Do you even need to
>> read the .install files? You do know it will copy and only copy files
>> around.
>
> The same will be true for dh-subst using packages. Does it start with #!
> $BLAH/dh_subst? Yes? Then it will copy and only copy files around, and
> substitution works just like you'd expect it to work.

Oh, no. This was ment as debhelper directly supporting the substitution
for a select set of known variables (dpkg-architecture output basically
or even just the multiarch var). If you have "#! /usr/bin/dh_subst" then
you do have the executable script way.

> So, in this case, the difference is negligible, both can be trivially
> understood.
>
> However, it gives more flexibility to the maintainer, to do more complex
> stuff, if so needs be. But, that won't be the common case. Why? Because
> there's no point in overcomplicating things.

Lets hope it stays at the level where you have a shebang (one of few
well known tools) followed by the normal input. That would be relatively
easy to get used to and understand. So you look at the .install file and
then man dh_subst and you are good.

At a glance what does this do?

#!/bin/bash
while read SRC DEST; do
  case "$SRC" in
(\#*|) continue;;
  esac
  case "$DEST" in
(*/) ;;
(*)  cp -- "$SRC" "${SRC%/*}/${DEST##*/}";;
  esac;
  printf -- "%s %s\n" "$SRC" "${DEST%/*}"
done << EOF
bin/foo bin/baz
bin/bla bin/
EOF

>> On the other hand we do have packages with executable debhelper files
>> that are NOT scripts. Debhelper currently breaks those. The execute
>> scripts feature should use a compat level. And then you would have the
>> situation you described, where you have to check the compat level and
>> every script.
>
> I'd treat executable files that are not scripts as a bug. They most
> probably don't have a shebang line, and that's just wrong.
>
> Breaking buggy packages is not something I'd worry about. Especially if
> the number is fairly low (and I expect it to be so, but feel free to
> correct me with numbers).

Around 17 sources for 3.0 (quilt) format. So nothing major. Could be
some more for native packages.

Till now joey has always taken great care of not breaking existing
sources when introducing new features. He has been verry good with using
the compat level to restrict known and unknown side effects of new
features from affecting existing sources. It seems with the latest one
he slipped up slightly.

MfG
Goswin

PS: Thanks Joey for all the good work you have done with debhelper. A
minor disagreement about ONE new feature in no way lessens the gratitude
you deserve for the rest.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87sjkt2w86.fsf@frosties.localnet

Re: Getting dh_install to do what we need

[Tollef Fog Heen]

]] Russ Allbery 

> Peter Samuelson  writes:
> 
> > Not for native packages.
> > Not for packages in format 3.0 (quilt).
> 
> > In both cases, execute permission in debian/ is preserved, with the
> > obvious exception of debian/rules, for which dpkg-source forces the +x
> > bit.
> 
> But still, how do you end up with random text files being executable in
> the debian directory?

At a guess: because you're careless and don't notice that you've run
chmod +x debian/* instead of chmod +x debian/rules.

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/871usd8jde@qurzaw.varnish-software.com

Re: Getting dh_install to do what we need

[Russ Allbery]

Peter Samuelson  writes:

> Not for native packages.
> Not for packages in format 3.0 (quilt).

> In both cases, execute permission in debian/ is preserved, with the
> obvious exception of debian/rules, for which dpkg-source forces the +x
> bit.

But still, how do you end up with random text files being executable in
the debian directory?  Are people packaging on Windows systems using a
remote file system implementation that uses a 000 umask or something?
That's the only time I've seen regular text files be marked randomly
executable.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87d3bxmui9@windlord.stanford.edu

Re: Getting dh_install to do what we need

[Chow Loong Jin]

On 10/12/2011 01:36, Bernd Zeimetz wrote:
> On 12/08/2011 07:00 PM, Chow Loong Jin wrote:
>> On 09/12/2011 01:40, Gergely Nagy wrote:
>>> Adam Borowski  writes:
>>>
>> *tad*
>> It would need to be a compiled program, since you can’t use scripts in
>> shebangs.
>
> Wrong, you can.

 On Linux and Hurd, yeah.
 On kFreeBSD, you can't.

 But hey, FreeBSD folks learned about basic niceties like tab completion 
 just
 last year, give them a decade or two and you'll have recursive shebangs 
 too.

 Unless we cheat and s|^#!|#!/usr/bin/env |, that is.  This works.
>>>
>>> See my workaround in the mail you quoted. "#! /bin/sh $PATH" should work
>>> for kFreeBSD and pretty much anything else out there too. An extra
>>> /bin/sh never hurt anybody!
>>
>> Except that it forces your interpreter to be written in sh, which Debian 
>> doesn't
>> like[1][2].
>>
>> [1] http://lintian.debian.org/tags/script-with-language-extension.html
>> [2] http://www.debian.org/doc/debian-policy/ch-files.html#s-scripts
> 
> Wrong, debian likes sh if you don't try to use !sh stuff in sh scripts.

I was referring to the fixing of implementation language, rather than sh itself.

-- 
Kind regards,
Loong Jin



signature.asc
Description: OpenPGP digital signature

Re: Getting dh_install to do what we need

[Peter Samuelson]

[Bernd Zeimetz]
> So there are sources which have executable debhelper files already? I
> doubt it as you'd have to chmod them manually.

Not for native packages.
Not for packages in format 3.0 (quilt).

In both cases, execute permission in debian/ is preserved, with the
obvious exception of debian/rules, for which dpkg-source forces the +x
bit.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111209182117.gc3...@p12n.org

Re: Getting dh_install to do what we need

[Bernd Zeimetz]

On 12/09/2011 05:01 PM, Jakub Wilk wrote:
> * Joey Hess , 2011-12-09, 11:50:
>>> On the other hand we do have packages with executable debhelper files that
>>> are NOT scripts. Debhelper currently breaks those. The execute scripts
>>> feature should use a compat level.
>> I have made hundreds of changes to debhelper that broke buggy packages 
>> without
>> using compat levels; that is not what compat levels are for.
> 
> Huh? "Some files in debian/* are executable, but they don't need to" would
> hardly qualifies even for a severity:minor bug.

I think it should, might be even a higher severity, depending on what the
content of these files is - especially .install and similar files might produce
not so funny results if you execute them accidentally.


-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4ee2482f.4060...@bzed.de

Re: Getting dh_install to do what we need

[Bernd Zeimetz]

On 12/08/2011 07:00 PM, Chow Loong Jin wrote:
> On 09/12/2011 01:40, Gergely Nagy wrote:
>> Adam Borowski  writes:
>>
> *tad*
> It would need to be a compiled program, since you can’t use scripts in
> shebangs.

 Wrong, you can.
>>>
>>> On Linux and Hurd, yeah.
>>> On kFreeBSD, you can't.
>>>
>>> But hey, FreeBSD folks learned about basic niceties like tab completion just
>>> last year, give them a decade or two and you'll have recursive shebangs too.
>>>
>>> Unless we cheat and s|^#!|#!/usr/bin/env |, that is.  This works.
>>
>> See my workaround in the mail you quoted. "#! /bin/sh $PATH" should work
>> for kFreeBSD and pretty much anything else out there too. An extra
>> /bin/sh never hurt anybody!
> 
> Except that it forces your interpreter to be written in sh, which Debian 
> doesn't
> like[1][2].
> 
> [1] http://lintian.debian.org/tags/script-with-language-extension.html
> [2] http://www.debian.org/doc/debian-policy/ch-files.html#s-scripts

Wrong, debian likes sh if you don't try to use !sh stuff in sh scripts.


-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4ee2470c.9080...@bzed.de

Re: Getting dh_install to do what we need

[Bernd Zeimetz]

On 12/09/2011 07:42 AM, Goswin von Brederlow wrote:
> Bernd Zeimetz  writes:
> 
>> On 12/07/2011 11:47 PM, Josselin Mouette wrote:
>>> Now that we’ve made incredible progress in terms of obfuscation, I’d
>>> appreciate if we could have a working solution that does not require
>>> scripting for the most trivial operations. So what remains? 
>>>   * Convincing Joey to revert this useless change and actually
>>> commit something useful. 
>>>   * NMU debhelper. 
>>>   * Technical committee. 
>>>   * Fork dh_install in a new package.
>>
>> It is up to you to use debhelper or not, so just use something else -
>> you have experience in writing
>> tools-which-do-the-same-thing-as-existing-tools. You could report a
>> bug report if you don't like it, but it is up to Joey to follow it or
>> not. You could do a NMU if you want people to complain about your
>> behaviour to DAM. You could even talk to the CTTE if you want to waste
>> more people's time.  And yes, you could even fork (its open source!) a
>> dh_install package - or instead of wasting ftp master resources, just
>> ship your own dh_install script in your debian folders.
> 
> I find it kind of funny that just recently Joey complained about Ubuntu
> adding a small patch to their debhelper (for DEB_HOST_MULTIARCH support
> in .install files) that he claimed would break some packages (although
> no examples have been given to substantiate that claim) and he therefore
> would ignore the patch and issue completly from then on. To teach Ubuntu
> a lesson.

You want to read that bug again.

> And now he does the same. Even worse because his change DOES break
> sources.

So there are sources which have executable debhelper files already? I doubt it
as you'd have to chmod them manually.


-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4ee2469b.9050...@bzed.de

Re: Getting dh_install to do what we need

[Gergely Nagy]

Jakub Wilk  writes:

> * Gergely Nagy , 2011-12-09, 16:05:
>> Comments, critique, testing and whatnot is more than welcomed, feel
>> free to shout my head off if you see something remarkably stupid.
>>
>> [1]: https://github.com/algernon/dh-subst
>
> As I already told you on IRC: this name (both package name and binary
> name) is wrong, or at least highly confusing.
>
> Quoting /usr/share/doc/debhelper/PROGRAMMING.gz:
> | This file documents things you should know to write a new debhelper
> | program. Any program with a name that begins with dh_ should conform
> | to these guidelines (with the historical exception of dh_make).

Fair enough. I'll try to come up with something else before I
upload. (Suggestions still welcome, unless we want to go with
/usr/bin/${boom} which I'd find funny.)

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87pqfx4pkj@luthien.mhp

Re: Getting dh_install to do what we need

[Jakub Wilk]

* Joey Hess , 2011-12-09, 11:50:
On the other hand we do have packages with executable debhelper files 
that are NOT scripts. Debhelper currently breaks those. The execute 
scripts feature should use a compat level.
I have made hundreds of changes to debhelper that broke buggy packages 
without using compat levels; that is not what compat levels are for.


Huh? "Some files in debian/* are executable, but they don't need to" 
would hardly qualifies even for a severity:minor bug.


--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111209160112.ga2...@jwilk.net

Re: Getting dh_install to do what we need

[Joey Hess]

Goswin von Brederlow wrote:
> On the other hand we do have packages with executable debhelper files
> that are NOT scripts. Debhelper currently breaks those. The execute
> scripts feature should use a compat level.

I have made hundreds of changes to debhelper that broke buggy packages
without using compat levels; that is not what compat levels are for.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Jakub Wilk]

* Gergely Nagy , 2011-12-09, 16:05:
Comments, critique, testing and whatnot is more than welcomed, feel 
free to shout my head off if you see something remarkably stupid.


[1]: https://github.com/algernon/dh-subst


As I already told you on IRC: this name (both package name and binary 
name) is wrong, or at least highly confusing.


Quoting /usr/share/doc/debhelper/PROGRAMMING.gz:
| This file documents things you should know to write a new debhelper
| program. Any program with a name that begins with dh_ should conform
| to these guidelines (with the historical exception of dh_make).

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111209152031.ga9...@jwilk.net

Re: Getting dh_install to do what we need

[Gergely Nagy]

Gergely Nagy  writes:

> Kees Cook  writes:
>
>> Which means I can't use DEB_HOST_MULTIARCH in the config-scripts,
>> unfortunately.
>
> Not to worry! dh-subst will be coming in a day or two (+ NEW waiting
> time), which will allow you to use DEB_HOST_MULTIARCH as you'd expect
> to.

'lo and behold, I started implementing dh-subst, and while it's fairly
rough and underdocumented, there's some usable code out there in my
github repo[1].

I'm not entirely sure it will do the right thing with DEB_HOST_MULTIARCH
in all cases, but I plan to test (and possibly fix) the corner cases
tonight.

Comments, critique, testing and whatnot is more than welcomed, feel free
to shout my head off if you see something remarkably stupid.

 [1]: https://github.com/algernon/dh-subst

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87borh6b0i.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Peter Samuelson]

[Kees Cook]
> This doesn't work with source-format-1 packages without adding
> "chmod" lines for the scripted debhelper config files in the rules
> file. Perhaps this isn't a big deal since we should all be using
> source-format-3 anyway.

We should?  I prefer to think of this whole debacle as yet another
reason to never switch to format 3.  So long as I stay with format 1,
"features" like this one will never come back to bite me.

(Well, yes, it could still happen to me with native packages.  But
that's a much smaller attack surface than if I used format 3.)


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111209105857.gb3...@p12n.org

Re: Getting dh_install to do what we need

[Gergely Nagy]

Goswin von Brederlow  writes:

>> Compared to writing overrides, it's less effort. Compared to just
>> writing the variable and expecting it to work, it's two commands more. I
>> believe that's not much.
>>
>> On the other hand, though, making it obvious that it's a script, and
>> there's magic (= variable substitution) going on, it's much more clear
>> to have it executable. Otherwise you'd have to look at the compat level
>> to see if it's supposed to do some tricks or not.
>>
>> I don't know about you, but I hardly ever look at compat levels. An
>> executable file in debian/ is much easier for me to notice.
>
> Would you really need to look at the compat level? Wouldn't seeing a
> ${DEB_HOST_MULTIARCH} in the .install file be obvious to everyone that
> there is substitution going on and what it does? It is a rather common
> syntax that should be familiar to every maintainer. Do you even need to
> read the .install files? You do know it will copy and only copy files
> around.

The same will be true for dh-subst using packages. Does it start with #!
$BLAH/dh_subst? Yes? Then it will copy and only copy files around, and
substitution works just like you'd expect it to work.

So, in this case, the difference is negligible, both can be trivially
understood.

However, it gives more flexibility to the maintainer, to do more complex
stuff, if so needs be. But, that won't be the common case. Why? Because
there's no point in overcomplicating things.

> On the other hand we do have packages with executable debhelper files
> that are NOT scripts. Debhelper currently breaks those. The execute
> scripts feature should use a compat level. And then you would have the
> situation you described, where you have to check the compat level and
> every script.

I'd treat executable files that are not scripts as a bug. They most
probably don't have a shebang line, and that's just wrong.

Breaking buggy packages is not something I'd worry about. Especially if
the number is fairly low (and I expect it to be so, but feel free to
correct me with numbers).

> Also an executable can do anything in any number of verry obscure ways. You
> have to actualy read the script, check the shebang, see what the shebang
> does before you can say wether the script will, for example, rm -rf / or
> not.

Yes. But once you see a familiar shebang, you're fine. And familiar
shebang is what you will see 99.9% of the time.

If you're so paranoid that you need to double-check a package so
throughly, you have to read debian/rules and control throughly too: you
never know when a package depends on something that provides a DH
sequence command that would happily rm -rf / away...

> Think of what this means for e.g the security team if they have to fix a
> package that uses a .install file written in haskell. Debian stopped
> allowing perl scripts for debian/rules for a reason and now dh brings
> that back, at least in part.

Debian stopped allowing non-make debian/rules due to an unrelated
correction, and since then is trying to explain it wasn't so. (It
certainly wasn't intentional at the time.)

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87y5umqkfu@luthien.mhp

Re: Getting dh_install to do what we need

[Goswin von Brederlow]

Gergely Nagy  writes:

> Thomas Goirand  writes:
>
>> Disclaimer: I didn't write any multiarch packaging (yet).
>>
>> The incentive for doing all this seems to be multiarch. Why
>> instead don't we have a mechanism to have variables in
>> debian/*.install instead, or a dh_helper to move things to
>> the multiarch folder instead of /usr/lib (or anything you would
>> find a better mechanism), so that we don't have to use tricks to
>> make files reach the correct destination?
>
> Apart from the reason explained in debhelper's git history, I can offer
> another: variable substitution would be an incompatible change, as it's
> perfectly legit to have '/usr/bin/${oh hai I broke your system}' or
> '/usr/lib/${DEB_HOST_MULTIARCH}' files.
>
> So, we'd need a flag to turn on variable substitution. Perhaps a compat
> level would be enough - I don't know, nor do I care.

Except that none of the existing sources do have one and it is verry
verry unlikely any 3rd party sourced have one either. It was also
discussed to even limit the substitution to a select few variables so
'/usr/bin/${oh hai I broke your system}' would not get substituted, even
if you had an evironment variable called "oh hai I broke your system".

So a compat level for that would be overkill. But totaly possible.

>> Putting manual efforts into moving things to the correct
>> multiarch folder seem to be quite annoying, the fact that
>> it would be in debian/rules or in debian/*.install doesn't
>> change much the issue: in both case we need to put efforts
>> into it, which can lead to all sorts of various issues which
>> wouldn't happen if we had the correct automation.
>
> How much effort is a chmod +x, and adding 1(!) line to the .install
> file?

Times 2000 library packages. But that isn't really the point. Adding
${DEB_HOST_MULTIARCH} to the .install files isn't any less work.

> Compared to writing overrides, it's less effort. Compared to just
> writing the variable and expecting it to work, it's two commands more. I
> believe that's not much.
>
> On the other hand, though, making it obvious that it's a script, and
> there's magic (= variable substitution) going on, it's much more clear
> to have it executable. Otherwise you'd have to look at the compat level
> to see if it's supposed to do some tricks or not.
>
> I don't know about you, but I hardly ever look at compat levels. An
> executable file in debian/ is much easier for me to notice.

Would you really need to look at the compat level? Wouldn't seeing a
${DEB_HOST_MULTIARCH} in the .install file be obvious to everyone that
there is substitution going on and what it does? It is a rather common
syntax that should be familiar to every maintainer. Do you even need to
read the .install files? You do know it will copy and only copy files
around.

On the other hand we do have packages with executable debhelper files
that are NOT scripts. Debhelper currently breaks those. The execute
scripts feature should use a compat level. And then you would have the
situation you described, where you have to check the compat level and
every script.

Also an executable can do anything in any number of verry obscure ways. You
have to actualy read the script, check the shebang, see what the shebang
does before you can say wether the script will, for example, rm -rf / or
not.

Think of what this means for e.g the security team if they have to fix a
package that uses a .install file written in haskell. Debian stopped
allowing perl scripts for debian/rules for a reason and now dh brings
that back, at least in part.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8762hq4486.fsf@frosties.localnet

Re: Getting dh_install to do what we need

[Gergely Nagy]

Goswin von Brederlow  writes:

>> I'll happily use the new dh_install feature instead of whining.
>
> You are forgetting that there are more people than just you looking at
> your package. By using an obviously verry controversial feature you will
> make everybody else life more difficult.

You're making it sound far worse than it is. If someone can't make heads
and tails out of what most of the executable files will be, he shouldn't
be looking at the sources to begin with.

(Especially since said executable files will differ from the
Ubuntu-proposed multi-arch stuff by exactly one line and a +x bit, and
that is far easier to understand than an override maze.)

> I don't think he is whining. He is trying to find a solution that
> everybody can live and work with. And while a package is generally the
> Maintainers domain a package also belongs to its users and has to work
> together with Debian as a whole. If a major piece of our toolchain for
> debian sources suddenly breaks existing sources and pisses of half the
> maintainer that is not a good state to have.

Yeah, some maintainers then need to take a deep breath and realize
they're overreacting.

What exactly does this break, that wasn't buggy before, by the way?

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8739cus02e@luthien.mhp

Re: Getting dh_install to do what we need

[Goswin von Brederlow]

Bernd Zeimetz  writes:

> On 12/07/2011 11:47 PM, Josselin Mouette wrote:
>> Now that we’ve made incredible progress in terms of obfuscation, I’d
>> appreciate if we could have a working solution that does not require
>> scripting for the most trivial operations. So what remains? 
>>   * Convincing Joey to revert this useless change and actually
>> commit something useful. 
>>   * NMU debhelper. 
>>   * Technical committee. 
>>   * Fork dh_install in a new package.
>
> It is up to you to use debhelper or not, so just use something else -
> you have experience in writing
> tools-which-do-the-same-thing-as-existing-tools. You could report a
> bug report if you don't like it, but it is up to Joey to follow it or
> not. You could do a NMU if you want people to complain about your
> behaviour to DAM. You could even talk to the CTTE if you want to waste
> more people's time.  And yes, you could even fork (its open source!) a
> dh_install package - or instead of wasting ftp master resources, just
> ship your own dh_install script in your debian folders.

I find it kind of funny that just recently Joey complained about Ubuntu
adding a small patch to their debhelper (for DEB_HOST_MULTIARCH support
in .install files) that he claimed would break some packages (although
no examples have been given to substantiate that claim) and he therefore
would ignore the patch and issue completly from then on. To teach Ubuntu
a lesson.

And now he does the same. Even worse because his change DOES break
sources.

> I'll happily use the new dh_install feature instead of whining.

You are forgetting that there are more people than just you looking at
your package. By using an obviously verry controversial feature you will
make everybody else life more difficult.

I don't think he is whining. He is trying to find a solution that
everybody can live and work with. And while a package is generally the
Maintainers domain a package also belongs to its users and has to work
together with Debian as a whole. If a major piece of our toolchain for
debian sources suddenly breaks existing sources and pisses of half the
maintainer that is not a good state to have.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87aa72456l.fsf@frosties.localnet

Re: Getting dh_install to do what we need

[Goswin von Brederlow]

Gergely Nagy  writes:

> Steve Langasek  writes:
>
>> On Thu, Dec 08, 2011 at 10:07:12PM +0100, Gergely Nagy wrote:
>>> Kees Cook  writes:
>>
>>> > Which means I can't use DEB_HOST_MULTIARCH in the config-scripts,
>>> > unfortunately.
>>
>>> Not to worry! dh-subst will be coming in a day or two (+ NEW waiting
>>> time), which will allow you to use DEB_HOST_MULTIARCH as you'd expect
>>> to.
>>
>> Will dh-subst automatically query dpkg-architecture to grab variables such
>> as DEB_HOST_MULTIARCH into the environment?
>
> Yes, it will. The intention is to be able to use DEB_*_MULTIARCH and
> similar stuff out of the box, without further work.

How will it know the architecture to grab them for? E.g. does

dpkg-buildpackage -aarmel

get the ones for armel?

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ehwe467a.fsf@frosties.localnet

Re: Getting dh_install to do what we need

[Jonas Smedegaard]

On 11-12-08 at 05:48pm, Russ Allbery wrote:
> Jonas Smedegaard  writes:
> 
> > Well, in a way this one is too: compat level 9 is not yet finalized, 
> > as I understand it.  Just prematurely gotten into use due to its 
> > upcoming ability to handle multiarch.
> 
> Yeah, and that was a little unfortunate.  There are packages I've not 
> converted to multiarch yet because I don't really want to use an 
> experimental compat level.  Normally debhelper sticks to one compat 
> level per release, but it would have been nice (for my uses at least) 
> to just call 9 done with multiarch and start 10.

That's my reason to hesitate with multiarch as well.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Russ Allbery]

Jonas Smedegaard  writes:

> Well, in a way this one is too: compat level 9 is not yet finalized, as
> I understand it.  Just prematurely gotten into use due to its upcoming
> ability to handle multiarch.

Yeah, and that was a little unfortunate.  There are packages I've not
converted to multiarch yet because I don't really want to use an
experimental compat level.  Normally debhelper sticks to one compat level
per release, but it would have been nice (for my uses at least) to just
call 9 done with multiarch and start 10.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8739cucy6a@windlord.stanford.edu

Re: Getting dh_install to do what we need

[Jonas Smedegaard]

On 11-12-08 at 03:02pm, Russ Allbery wrote:
> Gergely Nagy  writes:
> 
> > Apart from the reason explained in debhelper's git history, I can 
> > offer another: variable substitution would be an incompatible 
> > change, as it's perfectly legit to have '/usr/bin/${oh hai I broke 
> > your system}' or '/usr/lib/${DEB_HOST_MULTIARCH}' files.
> 
> > So, we'd need a flag to turn on variable substitution. Perhaps a 
> > compat level would be enough - I don't know, nor do I care.
> 
> This would be a similar change to the change of supporting wildcards 
> in all debhelper files, which was done with a compat level change.

Well, in a way this one is too: compat level 9 is not yet finalized, as 
I understand it.  Just prematurely gotten into use due to its upcoming 
ability to handle multiarch.

 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Russ Allbery]

Gergely Nagy  writes:

> Apart from the reason explained in debhelper's git history, I can offer
> another: variable substitution would be an incompatible change, as it's
> perfectly legit to have '/usr/bin/${oh hai I broke your system}' or
> '/usr/lib/${DEB_HOST_MULTIARCH}' files.

> So, we'd need a flag to turn on variable substitution. Perhaps a compat
> level would be enough - I don't know, nor do I care.

This would be a similar change to the change of supporting wildcards in
all debhelper files, which was done with a compat level change.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/871useis5i@windlord.stanford.edu

Re: Getting dh_install to do what we need

[Gergely Nagy]

Steve Langasek  writes:

> On Thu, Dec 08, 2011 at 10:07:12PM +0100, Gergely Nagy wrote:
>> Kees Cook  writes:
>
>> > Which means I can't use DEB_HOST_MULTIARCH in the config-scripts,
>> > unfortunately.
>
>> Not to worry! dh-subst will be coming in a day or two (+ NEW waiting
>> time), which will allow you to use DEB_HOST_MULTIARCH as you'd expect
>> to.
>
> Will dh-subst automatically query dpkg-architecture to grab variables such
> as DEB_HOST_MULTIARCH into the environment?

Yes, it will. The intention is to be able to use DEB_*_MULTIARCH and
similar stuff out of the box, without further work.

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87obvismc0@luthien.mhp

Re: Getting dh_install to do what we need

[Steve Langasek]

On Thu, Dec 08, 2011 at 01:36:48PM -0800, Steve Langasek wrote:
> On Thu, Dec 08, 2011 at 10:07:12PM +0100, Gergely Nagy wrote:
> > Kees Cook  writes:

> > > Which means I can't use DEB_HOST_MULTIARCH in the config-scripts,
> > > unfortunately.

> > Not to worry! dh-subst will be coming in a day or two (+ NEW waiting
> > time), which will allow you to use DEB_HOST_MULTIARCH as you'd expect
> > to.

> Will dh-subst automatically query dpkg-architecture to grab variables such
> as DEB_HOST_MULTIARCH into the environment?

> What *I* would expect is that these dpkg-architecture variables are already
> put into the environment by dh, because this is a common use case.  If
> dh-subst is going to take care of it automatically then I suppose that's ok,
> but that doesn't really feel like the right layer for it.

> If I have to manually export them in debian/rules, then that's yet another
> step away from rules.tiny, and that makes me sad.

Of course, the original bug report about substitutions in .install files to
support multiarch never said anything about auto-exporting DEB_* variables
from dh anyway, so if I really want that I suppose I should file another bug
against debhelper instead of babbling on debian-devel.  So please ignore the
above. :)

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Gergely Nagy]

Thomas Goirand  writes:

> P.S: Is there any valid lintian clean package in the archive that doesn't
> use any helper tools at all? That's would be very instructive...

dpatch, for example. There are plenty of other examples in the archive.

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87sjkusq1o@luthien.mhp

Re: Getting dh_install to do what we need

[Steve Langasek]

On Thu, Dec 08, 2011 at 10:07:12PM +0100, Gergely Nagy wrote:
> Kees Cook  writes:

> > Which means I can't use DEB_HOST_MULTIARCH in the config-scripts,
> > unfortunately.

> Not to worry! dh-subst will be coming in a day or two (+ NEW waiting
> time), which will allow you to use DEB_HOST_MULTIARCH as you'd expect
> to.

Will dh-subst automatically query dpkg-architecture to grab variables such
as DEB_HOST_MULTIARCH into the environment?

What *I* would expect is that these dpkg-architecture variables are already
put into the environment by dh, because this is a common use case.  If
dh-subst is going to take care of it automatically then I suppose that's ok,
but that doesn't really feel like the right layer for it.

If I have to manually export them in debian/rules, then that's yet another
step away from rules.tiny, and that makes me sad.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Thomas Goirand]

On 12/09/2011 03:58 AM, Gergely Nagy wrote:
> [...] I don't know about you [...]
>   
If it goes down to me, I would like using install -D -m [...] manually in my
debian/rules files, but as others don't, I try avoiding such hacks these
days... :)

Note that using the BSD install is compatible with

/usr/lib/${DEB_HOST_MULTIARCH}

without too much hacking... :)

Thomas

P.S: Is there any valid lintian clean package in the archive that doesn't
use any helper tools at all? That's would be very instructive...


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4ee123d5.2080...@debian.org

Re: Getting dh_install to do what we need

[Gergely Nagy]

Kees Cook  writes:

> Which means I can't use DEB_HOST_MULTIARCH in the config-scripts,
> unfortunately.

Not to worry! dh-subst will be coming in a day or two (+ NEW waiting
time), which will allow you to use DEB_HOST_MULTIARCH as you'd expect
to.

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87wra6srgf@luthien.mhp

Re: Getting dh_install to do what we need

[Joey Hess]

Kees Cook wrote:
> We must be talking about separate things:

Yes, there are so many ill-advised variables floating around the build
system these days that I have trouble keeping them all straight.

> Which means I can't use DEB_HOST_MULTIARCH in the config-scripts,
> unfortunately.

Happily there is a standard way for such a script to set
DEB_HOST_MULTIARCH on its own.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Joey Hess]

Steve Langasek wrote:
> Also, I'm pretty sure the *FLAGS export happens only for dh_auto_* and
> not for dh itself (set_buildflags is called from
> /usr/share/perl5/Debian/Debhelper/Dh_Buildsystems.pm), so it still doesn't
> help when called from dh_install.

joey@gnu:~/src/debhelper>grep set_buildflags dh
set_buildflags();

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Gergely Nagy]

Thomas Goirand  writes:

> Disclaimer: I didn't write any multiarch packaging (yet).
>
> The incentive for doing all this seems to be multiarch. Why
> instead don't we have a mechanism to have variables in
> debian/*.install instead, or a dh_helper to move things to
> the multiarch folder instead of /usr/lib (or anything you would
> find a better mechanism), so that we don't have to use tricks to
> make files reach the correct destination?

Apart from the reason explained in debhelper's git history, I can offer
another: variable substitution would be an incompatible change, as it's
perfectly legit to have '/usr/bin/${oh hai I broke your system}' or
'/usr/lib/${DEB_HOST_MULTIARCH}' files.

So, we'd need a flag to turn on variable substitution. Perhaps a compat
level would be enough - I don't know, nor do I care.

> Putting manual efforts into moving things to the correct
> multiarch folder seem to be quite annoying, the fact that
> it would be in debian/rules or in debian/*.install doesn't
> change much the issue: in both case we need to put efforts
> into it, which can lead to all sorts of various issues which
> wouldn't happen if we had the correct automation.

How much effort is a chmod +x, and adding 1(!) line to the .install
file?

Compared to writing overrides, it's less effort. Compared to just
writing the variable and expecting it to work, it's two commands more. I
believe that's not much.

On the other hand, though, making it obvious that it's a script, and
there's magic (= variable substitution) going on, it's much more clear
to have it executable. Otherwise you'd have to look at the compat level
to see if it's supposed to do some tricks or not.

I don't know about you, but I hardly ever look at compat levels. An
executable file in debian/ is much easier for me to notice.

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/871useu97k@luthien.mhp

Re: Getting dh_install to do what we need

[Gergely Nagy]

Chow Loong Jin  writes:

> On 09/12/2011 02:10, Gergely Nagy wrote:
>> Chow Loong Jin  writes:
>> 
 See my workaround in the mail you quoted. "#! /bin/sh $PATH" should work
 for kFreeBSD and pretty much anything else out there too. An extra
 /bin/sh never hurt anybody!
>>>
>>> Except that it forces your interpreter to be written in sh, which Debian 
>>> doesn't
>>> like[1][2].
>>>
>>> [1] http://lintian.debian.org/tags/script-with-language-extension.html
>> 
>> This is irrelevant, as it's only about the extension, and only about
>> scripts on the path. My proposal has no extension, and isn't on the
>> path, either.
>
> Sorry, I mixed it up with another post that mentioned 
> /usr/bin/dh_multiarchify.
> I didn't realize you changed the path to /usr/lib/debhelper/dh_multiarchify.

It wouldn't apply in the /usr/bin case, either, since there is no
extension.

> I linked to the policy and lintian tag not because of an implied policy
> violation, but the general reasoning behind not permitting extensions in 
> scripts
> on $PATH, i.e. that it causes the implementation language of the binary/script
> to be locked into the #!s of all its users.

Yup, and my script didn't have an extension.

> I'm guessing that dpatch-run had never needed its implementation language
> changed, but that doesn't necessarily justify locking the implementation
> language of your proposed dh_multiarchify into the #!s of all .install files
> which use it.

If the language ever needs to be changed, no problem: make the former
shell script a wrapper, that calls the real deal.

> Between "#!/usr/bin/env /usr/lib/debhelper/dh_multiarchify" and "#!/bin/sh
> /usr/lib/debhelper/dh_multiarchify", I think the former is obviously the
> superior choice, regardless of what dpatch used for the past 7 years.

Perhaps. In the long run, it doesn't matter, both get the job done, and
whether it's /bin/sh or /usr/bin/env is such a minor detail. And policy
still doesn't say a word about this case. Not even remotely.

But, I'm convinced: I'll use /usr/bin/env in the upcoming dh-subst
thingy.

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8762hqu9hp@luthien.mhp

Re: Getting dh_install to do what we need

[Wookey]

+++ Thomas Goirand [2011-12-09 02:39 +0800]:
> On 12/08/2011 06:47 AM, Josselin Mouette wrote:

> >  Closes: #235302, #372310, #235302, #614731,
> >  Closes: #438601, #477625, #632860, #642129
> >
> 
> The incentive for doing all this seems to be multiarch. 

It's not just multiarch. The list of bugs above covers a range of
cases where people want to do 'interesting' things with .install files.

The way this one changes solves all those bugs demonstrates the
power/flexibility of the idea. The question is, is it overkill for the
common case? 

Yes, multiarch is the case that is by far the most common
requirement and appears to have prompted this change (
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=614731 is the
relevant bug )

> Why
> instead don't we have a mechanism to have variables in
> debian/*.install instead, 

That was my original suggestion. It has the advantage of being easy to
understand, but of course is not as flexible as this general solution
for 'do whatever crazy shit you need to do'. 

There is often a tradoff between simplicity and flexibility. You could
argue that this errs somewhat on the side of being too flexible, but
maybe it's just really clever?

> or a dh_helper to move things to
> the multiarch folder instead of /usr/lib (or anything you would
> find a better mechanism), so that we don't have to use tricks to
> make files reach the correct destination?
> 
> Putting manual efforts into moving things to the correct
> multiarch folder seem to be quite annoying, the fact that
> it would be in debian/rules or in debian/*.install doesn't
> change much the issue: in both case we need to put efforts
> into it, which can lead to all sorts of various issues which
> wouldn't happen if we had the correct automation.
> 
> Comments anyone?

It does seem that multiarch is such a common case that there is a good
argument for having it dealt with in a standard way. That could be a
standard dh_multiarchify script as suggested somewhere else in this
thread (can we make a one-size-fits-all reliable multiarchify
script?). I'd have been perfectly happy with a special token,
substvar, or env var substitution, and I guess that could still be
done if it eventually seems still to be a good idea.

Seems to me that it'd be nicer to have installs something
like:
debian/tmp/usr/lib/${DEB_HOST_MULTIARCH}/libpopt.so.*   
lib/${DEB_HOST_MULTIARCH}

But, ultimately I'm happy that there is now mechanism to solve this issue
fairly cleanly (assuming that the DEB_* env vars really are exported -
see elsewhere in this thread).

I wouldn't have done it this way, but then I'm nothing like as clever
as Joey is. 

Wookey
-- 
Principal hats:  Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111208193425.gj28...@dream.aleph1.co.uk

Re: Getting dh_install to do what we need

[Kees Cook]

On Thu, Dec 08, 2011 at 02:47:00PM -0400, Joey Hess wrote:
> Steve Langasek wrote:
> > While I had originally believed this to be the case when drafting
> > , feedback from those
> > implementing this in practice is that dh does *not* export these variables,
> > it only passes them to autoconf.
> 
> That's not correct, dh exports the variables. (You made me look though.)
> 
> joey@gnu:~/src/filters>head -n2 Makefile
> build:
>   echo ${CPPFLAGS}
> joey@gnu:~/src/filters>echo 9 > debian/compat
> joey@gnu:~/src/filters>debian/rules build
> dh build
>dh_testdir
>dh_auto_configure
>dh_auto_build
> make[1]: Entering directory `/home/joey/src/filters'
> echo -D_FORTIFY_SOURCE=2
> -D_FORTIFY_SOURCE=2

We must be talking about separate things:

$ head -n3 Makefile
build:
echo CPPFLAGS: $(CPPFLAGS)
echo DEB_HOST_MULTIARCH: $(DEB_HOST_MULTIARCH)
$ cat debian/compat 
9
$ fakeroot debian/rules build
dh build 
   dh_testdir
   dh_auto_configure
   dh_auto_build
make[1]: Entering directory `/tmp/dhtest-1.0'
echo CPPFLAGS: -D_FORTIFY_SOURCE=2
CPPFLAGS: -D_FORTIFY_SOURCE=2
echo DEB_HOST_MULTIARCH: 
DEB_HOST_MULTIARCH:

Which means I can't use DEB_HOST_MULTIARCH in the config-scripts,
unfortunately.

-Kees

-- 
Kees Cook@debian.org


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111208190633.gw5...@outflux.net

Re: Getting dh_install to do what we need

[Steve Langasek]

On Thu, Dec 08, 2011 at 02:47:00PM -0400, Joey Hess wrote:
> Steve Langasek wrote:
> > While I had originally believed this to be the case when drafting
> > , feedback from those
> > implementing this in practice is that dh does *not* export these variables,
> > it only passes them to autoconf.

> That's not correct, dh exports the variables. (You made me look though.)

> joey@gnu:~/src/filters>head -n2 Makefile
> build:
>   echo ${CPPFLAGS}
> joey@gnu:~/src/filters>echo 9 > debian/compat
> joey@gnu:~/src/filters>debian/rules build
> dh build
>dh_testdir
>dh_auto_configure
>dh_auto_build
> make[1]: Entering directory `/home/joey/src/filters'
> echo -D_FORTIFY_SOURCE=2
> -D_FORTIFY_SOURCE=2

Well, it exports CPPFLAGS as part of dpkg-buildflags handling, sure, but not
DEB_{HOST,BUILD}_*.

(I did actually test this myself before posting:

$ cat debian/rules
#!/usr/bin/make -f

%:
dh $@

override_dh_auto_configure:
echo $(DEB_HOST_MULTIARCH)
false

$ echo 9 > debian/compat
$ ./debian/rules build
dh build 
   dh_testdir
   debian/rules override_dh_auto_configure
make[1]: Entering directory `/tmp/testing'
echo 

false 

)

Also, I'm pretty sure the *FLAGS export happens only for dh_auto_* and
not for dh itself (set_buildflags is called from
/usr/share/perl5/Debian/Debhelper/Dh_Buildsystems.pm), so it still doesn't
help when called from dh_install.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Thomas Goirand]

On 12/08/2011 06:47 AM, Josselin Mouette wrote:
> For those who haven’t followed, the latest debhelper upload includes the
> following change:
>
>* Debhelper config files may be made executable programs that output the
>  desired configuration. No further changes are planned to the config file
>  format; those needing powerful syntaxes may now use a programming 
> language
>  of their choice. (Be careful aiming that at your feet.)
>  Closes: #235302, #372310, #235302, #614731,
>  Closes: #438601, #477625, #632860, #642129
>
> So, to sum it up. Before, you would do in debian/rules: 
> sed s/@DEB_HOST_MULTIARCH@/${DEB_HOST_MULTIARCH}/ 
> debian/libfoo.install.in > debian/libfoo.install
>
> Now, you will do in debian/foo.install:
> #! /bin/sh 
> sed s/@DEB_HOST_MULTIARCH@/$(dpkg-architecture -qDEB_HOST_MULTIARCH)/ 
> << EOF
> …
> EOF
> (Bonus points for implementing this in a different language for each
> package.)
>
> Now that we’ve made incredible progress in terms of obfuscation, I’d
> appreciate if we could have a working solution that does not require
> scripting for the most trivial operations. So what remains? 
>   * Convincing Joey to revert this useless change and actually
> commit something useful. 
>   * NMU debhelper. 
>   * Technical committee. 
>   * Fork dh_install in a new package.
>
> Any comments before someone starts doing either of these?
>
>   
Disclaimer: I didn't write any multiarch packaging (yet).

The incentive for doing all this seems to be multiarch. Why
instead don't we have a mechanism to have variables in
debian/*.install instead, or a dh_helper to move things to
the multiarch folder instead of /usr/lib (or anything you would
find a better mechanism), so that we don't have to use tricks to
make files reach the correct destination?

Putting manual efforts into moving things to the correct
multiarch folder seem to be quite annoying, the fact that
it would be in debian/rules or in debian/*.install doesn't
change much the issue: in both case we need to put efforts
into it, which can lead to all sorts of various issues which
wouldn't happen if we had the correct automation.

Comments anyone?

Thomas


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4ee10460.7040...@debian.org

Re: Getting dh_install to do what we need

[Chow Loong Jin]

On 09/12/2011 02:10, Gergely Nagy wrote:
> Chow Loong Jin  writes:
> 
>>> See my workaround in the mail you quoted. "#! /bin/sh $PATH" should work
>>> for kFreeBSD and pretty much anything else out there too. An extra
>>> /bin/sh never hurt anybody!
>>
>> Except that it forces your interpreter to be written in sh, which Debian 
>> doesn't
>> like[1][2].
>>
>> [1] http://lintian.debian.org/tags/script-with-language-extension.html
> 
> This is irrelevant, as it's only about the extension, and only about
> scripts on the path. My proposal has no extension, and isn't on the
> path, either.

Sorry, I mixed it up with another post that mentioned /usr/bin/dh_multiarchify.
I didn't realize you changed the path to /usr/lib/debhelper/dh_multiarchify.
> 
>> [2] http://www.debian.org/doc/debian-policy/ch-files.html#s-scripts
> 
> This also doesn't talk about restricting the #! interpreter to
> binaries. The only restriction it makes is that csh scripts should be
> avoided. (And that's a should, not a must)
> 
> Besides, dpatch has been using #! /bin/sh $PATH for more than 7 years
> now. Please, pretty please, find a policy violation in that, and we can
> suddenly break a thousand packages in one swift swing of an arm!

I linked to the policy and lintian tag not because of an implied policy
violation, but the general reasoning behind not permitting extensions in scripts
on $PATH, i.e. that it causes the implementation language of the binary/script
to be locked into the #!s of all its users.

I'm guessing that dpatch-run had never needed its implementation language
changed, but that doesn't necessarily justify locking the implementation
language of your proposed dh_multiarchify into the #!s of all .install files
which use it.

Between "#!/usr/bin/env /usr/lib/debhelper/dh_multiarchify" and "#!/bin/sh
/usr/lib/debhelper/dh_multiarchify", I think the former is obviously the
superior choice, regardless of what dpatch used for the past 7 years.

-- 
Kind regards,
Loong Jin



signature.asc
Description: OpenPGP digital signature

Re: Getting dh_install to do what we need

[Joey Hess]

Steve Langasek wrote:
> While I had originally believed this to be the case when drafting
> , feedback from those
> implementing this in practice is that dh does *not* export these variables,
> it only passes them to autoconf.

That's not correct, dh exports the variables. (You made me look though.)

joey@gnu:~/src/filters>head -n2 Makefile
build:
echo ${CPPFLAGS}
joey@gnu:~/src/filters>echo 9 > debian/compat
joey@gnu:~/src/filters>debian/rules build
dh build
   dh_testdir
   dh_auto_configure
   dh_auto_build
make[1]: Entering directory `/home/joey/src/filters'
echo -D_FORTIFY_SOURCE=2
-D_FORTIFY_SOURCE=2

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Steve Langasek]

On Thu, Dec 08, 2011 at 06:51:51PM +0100, Andreas Metzler wrote:
> Josselin Mouette  wrote:
> [...]
> > So, to sum it up. Before, you would do in debian/rules: 
> >sed s/@DEB_HOST_MULTIARCH@/${DEB_HOST_MULTIARCH}/ 
> > debian/libfoo.install.in > debian/libfoo.install
> [...]

> a little bit beside the point, but since the example seems to propagate
> through the whole thread I feel the need to mention it anyway:

> 999 out of a 1000 library packages will not need to use tricks like
> this for conversion to multi-arch. Usually a simple wildcard works:

Your estimate is off by several orders of magnitude.  Out of approximately
200 libraries converted so far, I'm sure at least 6-10 of them have needed
this trick.

Usually this is because they need to create symlinks with .links files, so
you have to know the exact name, not a wildcard; or because they need to use
dh_install to move files to a directory other than where the upstream target
installs them, so they need to be able to represent the target directory
name.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Bernd Zeimetz]

On 12/07/2011 11:47 PM, Josselin Mouette wrote:
> Now that we’ve made incredible progress in terms of obfuscation, I’d
> appreciate if we could have a working solution that does not require
> scripting for the most trivial operations. So what remains? 
>   * Convincing Joey to revert this useless change and actually
> commit something useful. 
>   * NMU debhelper. 
>   * Technical committee. 
>   * Fork dh_install in a new package.

It is up to you to use debhelper or not, so just use something else - you have
experience in writing tools-which-do-the-same-thing-as-existing-tools. You could
report a bug report if you don't like it, but it is up to Joey to follow it or
not. You could do a NMU if you want people to complain about your behaviour to
DAM. You could even talk to the CTTE if you want to waste more people's time.
And yes, you could even fork (its open source!) a dh_install package - or
instead of wasting ftp master resources, just ship your own dh_install script in
your debian folders.

I'll happily use the new dh_install feature instead of whining.

-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4ee1000b.40...@bzed.de

Re: Getting dh_install to do what we need

[Gergely Nagy]

Chow Loong Jin  writes:

>> See my workaround in the mail you quoted. "#! /bin/sh $PATH" should work
>> for kFreeBSD and pretty much anything else out there too. An extra
>> /bin/sh never hurt anybody!
>
> Except that it forces your interpreter to be written in sh, which Debian 
> doesn't
> like[1][2].
>
> [1] http://lintian.debian.org/tags/script-with-language-extension.html

This is irrelevant, as it's only about the extension, and only about
scripts on the path. My proposal has no extension, and isn't on the
path, either.

> [2] http://www.debian.org/doc/debian-policy/ch-files.html#s-scripts

This also doesn't talk about restricting the #! interpreter to
binaries. The only restriction it makes is that csh scripts should be
avoided. (And that's a should, not a must)

Besides, dpatch has been using #! /bin/sh $PATH for more than 7 years
now. Please, pretty please, find a policy violation in that, and we can
suddenly break a thousand packages in one swift swing of an arm!

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87hb1bue70.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Chow Loong Jin]

On 09/12/2011 01:40, Gergely Nagy wrote:
> Adam Borowski  writes:
> 
 *tad*
 It would need to be a compiled program, since you can’t use scripts in
 shebangs.
>>>
>>> Wrong, you can.
>>
>> On Linux and Hurd, yeah.
>> On kFreeBSD, you can't.
>>
>> But hey, FreeBSD folks learned about basic niceties like tab completion just
>> last year, give them a decade or two and you'll have recursive shebangs too.
>>
>> Unless we cheat and s|^#!|#!/usr/bin/env |, that is.  This works.
> 
> See my workaround in the mail you quoted. "#! /bin/sh $PATH" should work
> for kFreeBSD and pretty much anything else out there too. An extra
> /bin/sh never hurt anybody!

Except that it forces your interpreter to be written in sh, which Debian doesn't
like[1][2].

[1] http://lintian.debian.org/tags/script-with-language-extension.html
[2] http://www.debian.org/doc/debian-policy/ch-files.html#s-scripts

-- 
Kind regards,
Loong Jin



signature.asc
Description: OpenPGP digital signature

Re: Getting dh_install to do what we need

[Andreas Metzler]

Josselin Mouette  wrote:
[...]
> So, to sum it up. Before, you would do in debian/rules: 
>sed s/@DEB_HOST_MULTIARCH@/${DEB_HOST_MULTIARCH}/ 
> debian/libfoo.install.in > debian/libfoo.install
[...]

Hello,

a little bit beside the point, but since the example seems to propagate
through the whole thread I feel the need to mention it anyway:

999 out of a 1000 library packages will not need to use tricks like
this for conversion to multi-arch. Usually a simple wildcard works:

libfoo42.install
-debian/tmp/usr/lib/lib*.so.*
+debian/tmp/usr/lib/*/lib*.so.*

libfoo-dev.install
-debian/tmp/usr/lib/lib*.so
-debian/tmp/usr/lib/lib*.a
-debian/tmp/usr/lib/pkgconfig
+debian/tmp/usr/lib/*/lib*.so
+debian/tmp/usr/lib/*/lib*.a
+debian/tmp/usr/lib/*/pkgconfig

cu andreas


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/n2a7r8-rm8@argenau.downhill.at.eu.org

Re: Getting dh_install to do what we need

[Gergely Nagy]

Adam Borowski  writes:

>> > *tad*
>> > It would need to be a compiled program, since you can’t use scripts in
>> > shebangs.
>> 
>> Wrong, you can.
>
> On Linux and Hurd, yeah.
> On kFreeBSD, you can't.
>
> But hey, FreeBSD folks learned about basic niceties like tab completion just
> last year, give them a decade or two and you'll have recursive shebangs too.
>
> Unless we cheat and s|^#!|#!/usr/bin/env |, that is.  This works.

See my workaround in the mail you quoted. "#! /bin/sh $PATH" should work
for kFreeBSD and pretty much anything else out there too. An extra
/bin/sh never hurt anybody!

-- 
|8]


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87liqnufm2.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Adam Borowski]

On Thu, Dec 08, 2011 at 05:12:08PM +0100, Gergely Nagy wrote:
> Josselin Mouette  writes:
> 
> > Le jeudi 08 décembre 2011 à 00:12 +0100, Gergely Nagy a écrit : 
> >> ,
> >> | #! /usr/bin/dh_multiarchify
> >> | /usr/lib/${DEB_HOST_MULTIARCH}/*
> >> `
> >> 
> >> The /usr/bin/dh_multiarchify script
> >
> > *tad*
> > It would need to be a compiled program, since you can’t use scripts in
> > shebangs.
> 
> Wrong, you can.

On Linux and Hurd, yeah.
On kFreeBSD, you can't.

But hey, FreeBSD folks learned about basic niceties like tab completion just
last year, give them a decade or two and you'll have recursive shebangs too.

Unless we cheat and s|^#!|#!/usr/bin/env |, that is.  This works.


-- 
1KB // Yo momma uses IPv4!


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111208171054.ga7...@angband.pl

Re: Getting dh_install to do what we need

[Gergely Nagy]

Goswin von Brederlow  writes:

>>> override_dh_auto_install:
>>> debian/libfoo.my-install-script
>> [..]
>>
>>> This new feature stinks of black-box magic that will make people crazy
>>> trying to find/fix a prolem in somebody elses package. The thing that
>>> make cdbs so bad.
>>
>>
>> I beg to disagree. You made  very good example why the former (your)
>> approach was a black-box indeed, whereas the newer one in fact
>> standardizes things in parts.
>>
>> See, in your case libfoo.my-install-script could be doing anything,
>
> The libfoo.install file can do anything too. It's an executable, which
> are turing complete.

So, neither solution is any better or worse than the other in this
regard. Both can do pretty much anything they want.

>> including but not limited to copying, moving, creating, changing a file
>> without any information on what's going on.
>> Now, by using that feature you are forced to generate a (dynamic) file
>> listing instead and everyone can execute that script and see the results
>> without /actually/ installing anything to a binary package.
>
> But the debian/rules file makes it clear that there is something more
> than copying going on. It points a big fat arrow at the
> debian/libfoo.my-install-script.

So does a +x on the install script. And if it needs to do anything more
elaborate than expanding a variable or two and copying, it can be better
documented, and have a less convoluted syntax than Makefile rules.

> With the automatic execution feauture you never know if there is magic
> going on under the hood or not unless you carefully check every single
> files permissions.

ls --color debian/, or find debian/ -type f -executable give good enough
hints, in my opinion, and quickly glancing over the results is quicker
than finding your way through a bunch of overrides.

>> Of course you could discuss whether executing scripts is necessarily a
>> better idea than having some semantically parsed *.install file magic
>> instead, but that's an implementation detail.
>> Up to now, you are all discussing why "chmod +x foo.install" is so much
>
> Note: With a debian.tar file you don't need the chmod I think. So
> debian/rules would give you no indication that there is a black-box
> thing happening via dh_install.

There's alredy a lot of magic happening with debhelper. If it learned to
expand variables instead of executable scripts, there'd still be
magic. Just different kind of it.

If one's curious, the answer is a find away. It's not hard. And most
executable scripts WILL be trivial. Especially if we manage to
standardise on executable-debhelper-file-helper-tool usage.

Again, look at dpatch. I belive that despite all its warts, it is a good
demonstration that executable magic stuff can be standardised.

>> worse than overriding dh_install by your own dark magic but you should
>> be realizing you just traded one black-box for another
>> not-so-black-but-still-very-dark-box.
>
> Let there be light.
>
> A lot of cases will also be like
>
> override_dh_auto_install:
>  sed 's/@DEB_HOST_MULTIARCH@/$(DEB_HOST_MULTIARCH)/g' < 
> debian/libfoo.install.in > debian/libfoo.install
>  dh_install
>
> Something I would find more readable than libfoo.install being a executable.

And I find this more readable:

$ cat debian/libfoo.install
#! /usr/lib/debhelper/dh_subst
debian/tmp/usr/lib/${DEB_HOST_MULTIARCH}/libfoo.so.*
$

But, the good thing is: you can do either! You can opt to not make your
files executable, and do the override magic, if so you prefer. Those of
us, who hate doing that over and over again, can use convenient helper
scripts to make the executable files trivial.

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87pqfzug4p.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Steve Langasek]

On Thu, Dec 08, 2011 at 01:03:18PM -0400, Joey Hess wrote:
> Kees Cook wrote:
> > - Export DEB_* environment variables to the script. This really feels
> >   like the missing piece to me.

> dh already does that in v9 mode.

While I had originally believed this to be the case when drafting
, feedback from those
implementing this in practice is that dh does *not* export these variables,
it only passes them to autoconf.  So the DEB_* variables are set in the
environment when debian/rules is invoked via dpkg-buildpackage because
dpkg-buildpackage itself sets them; they are not set in the environment when
debian/rules is called directly.

If dh were to export the dpkg-architecture variables by default in v9, I
think that would be a big help here.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Joey Hess]

Kees Cook wrote:
> - Export DEB_* environment variables to the script. This really feels
>   like the missing piece to me.

dh already does that in v9 mode.

-- 
see shy jo


signature.asc
Description: Digital signature

Re: Getting dh_install to do what we need

[Goswin von Brederlow]

Arno Töll  writes:

> Hello,
>
> On 08.12.2011 10:44, Goswin von Brederlow wrote:
>> Or for the more general case:
>> 
>> override_dh_auto_install:
>> debian/libfoo.my-install-script
> [..]
>
>> This new feature stinks of black-box magic that will make people crazy
>> trying to find/fix a prolem in somebody elses package. The thing that
>> make cdbs so bad.
>
>
> I beg to disagree. You made  very good example why the former (your)
> approach was a black-box indeed, whereas the newer one in fact
> standardizes things in parts.
>
> See, in your case libfoo.my-install-script could be doing anything,

The libfoo.install file can do anything too. It's an executable, which
are turing complete.

> including but not limited to copying, moving, creating, changing a file
> without any information on what's going on.
> Now, by using that feature you are forced to generate a (dynamic) file
> listing instead and everyone can execute that script and see the results
> without /actually/ installing anything to a binary package.

But the debian/rules file makes it clear that there is something more
than copying going on. It points a big fat arrow at the
debian/libfoo.my-install-script.

With the automatic execution feauture you never know if there is magic
going on under the hood or not unless you carefully check every single
files permissions.

I'm not objecting to the magic part, I'm objecting to the black-box
part.

> Of course you could discuss whether executing scripts is necessarily a
> better idea than having some semantically parsed *.install file magic
> instead, but that's an implementation detail.
> Up to now, you are all discussing why "chmod +x foo.install" is so much

Note: With a debian.tar file you don't need the chmod I think. So
debian/rules would give you no indication that there is a black-box
thing happening via dh_install.

> worse than overriding dh_install by your own dark magic but you should
> be realizing you just traded one black-box for another
> not-so-black-but-still-very-dark-box.

Let there be light.

A lot of cases will also be like

override_dh_auto_install:
 sed 's/@DEB_HOST_MULTIARCH@/$(DEB_HOST_MULTIARCH)/g' < 
debian/libfoo.install.in > debian/libfoo.install
 dh_install

Something I would find more readable than libfoo.install being a executable.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87iplrdn1d.fsf@frosties.localnet

Re: Getting dh_install to do what we need

[Gergely Nagy]

Kees Cook  writes:
> - Export DEB_* environment variables to the script. This really feels
>   like the missing piece to me.

I'd love this too. I already have a half-baked patch, implementing some
generic substitution-foo that could use this (see #651393 for the
details).

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87y5unui7x.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Kees Cook]

After looking at the bugs that are solved with this change, I don't think
it's an unreasonable solution. That said, I think it feels incomplete.

On Wed, Dec 07, 2011 at 11:47:49PM +0100, Josselin Mouette wrote:
> So, to sum it up. Before, you would do in debian/rules: 
> sed s/@DEB_HOST_MULTIARCH@/${DEB_HOST_MULTIARCH}/ 
> debian/libfoo.install.in > debian/libfoo.install
> 
> Now, you will do in debian/foo.install:
> #! /bin/sh 
> sed s/@DEB_HOST_MULTIARCH@/$(dpkg-architecture -qDEB_HOST_MULTIARCH)/ 
> << EOF
> …
> EOF

This doesn't work with source-format-1 packages without adding "chmod"
lines for the scripted debhelper config files in the rules file. Perhaps
this isn't a big deal since we should all be using source-format-3 anyway.

The debhelper config scripts do not run in the same environment as the
rules file, so none of the environment variables available there are
available to the config scripts. This means either adding "export" lines to
the rules file (which means you can't have a nice 2-line rules file), or
it means manually re-establishing the environment in the config scripts,
which seems wasteful since dh has already done all the magic to establish
the environment.

I think it would make sense to support the common case, and that appears to
be environment variable replacements (235302, 477625, 614731, 642129), with
general filtering a close second (372310, 438601, 632860).

So, how about:

- Have debhelper do the executable marking in some way. (Yet another
  config to list the config scripts?) Or, I guess, just ignore this
  problem since it's only a problem in source-format-1.

- Export DEB_* environment variables to the script. This really feels
  like the missing piece to me.

-Kees

-- 
Kees Cook@debian.org


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111208161348.gv5...@outflux.net

Re: Getting dh_install to do what we need

[Gergely Nagy]

Josselin Mouette  writes:

> Le jeudi 08 décembre 2011 à 00:12 +0100, Gergely Nagy a écrit : 
>> ,
>> | #! /usr/bin/dh_multiarchify
>> | /usr/lib/${DEB_HOST_MULTIARCH}/*
>> `
>> 
>> The /usr/bin/dh_multiarchify script
>
> *tad*
> It would need to be a compiled program, since you can’t use scripts in
> shebangs.

Wrong, you can.

But even if you couldn't, there'd be a workaround:
#! /bin/sh /usr/lib/debhelper/dh_multiarchify

-- 
|8]


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87aa73vy93.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Igor Pashev]

08.12.2011 18:49, Josselin Mouette пишет:

Le jeudi 08 décembre 2011 à 00:12 +0100, Gergely Nagy a écrit :

,
| #! /usr/bin/dh_multiarchify
| /usr/lib/${DEB_HOST_MULTIARCH}/*
`

The /usr/bin/dh_multiarchify script


*tad*
It would need to be a compiled program, since you can’t use scripts in
shebangs.



(18:52:00)
[pashev@moon:~]
# cat shbang script
#!/bin/sh

echo "$@"

#!/home/pashev/shbang

dfwuefnwiuef


(18:52:06)
[pashev@moon:~]
# ./script
./script


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4ee0cf32.9090...@gmail.com

Re: Getting dh_install to do what we need

[Josselin Mouette]

Le jeudi 08 décembre 2011 à 00:12 +0100, Gergely Nagy a écrit : 
> ,
> | #! /usr/bin/dh_multiarchify
> | /usr/lib/${DEB_HOST_MULTIARCH}/*
> `
> 
> The /usr/bin/dh_multiarchify script

*tad*
It would need to be a compiled program, since you can’t use scripts in
shebangs.

-- 
 .''`.  Josselin Mouette
: :' :
`. `'
  `-


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1323355743.31948.642.camel@pi0307572

Re: Getting dh_install to do what we need

[Igor Pashev]

08.12.2011 13:49, Goswin von Brederlow пишет:

Arno Töll  writes:


Your own script-fu in debian/rules or external scripts isn't exactly the
next best thing to read and learn how a foreign package works and there
/are/ use cases where dh_install isn't flexible enough to deal with the
problem by using the possibilities you had before. Renaming files and
multi-arch support is what comes me in mind immediately.


Yes, there are cases where dh_install isn't the tool you should or even
can use.


As for http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=632860
I ended with dh_movefiles


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4ee098a4.50...@gmail.com

Re: Getting dh_install to do what we need

[Gergely Nagy]

Jon Dowland  writes:

> On Thu, Dec 08, 2011 at 08:14:52AM +0100, Gergely Nagy wrote:
>> I disagree. Look at dpatch. It had executable patches since the
>> beginning, and a standardised script from 2.0 onwards.
>
> One problem with executable patches was the fact you couldn't reason
> about the packaging without first executing them, in the general case,
> which was frowned upon by security-concious people who like automated
> package testing tools.
>
> Does that not apply here?

Not really, no. At least, executable debhelper files are not worse than
the already existing hacks and the proposed override mazes.

They actually have the potential to be much more understandable.

Compared to simple, static files, they're harder to understand and
reason about. But compared to overrides, n+1 variants of them, nope,
they're not worse in any way. However, this presents us with an
opportunity to standardise, and that's great.

Also, debhelper files are considerably easier to understand, even when
executable, than the few dpatch scripts that weren't patches are. ;)

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87sjkvwee6.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Jon Dowland]

On Thu, Dec 08, 2011 at 08:14:52AM +0100, Gergely Nagy wrote:
> I disagree. Look at dpatch. It had executable patches since the
> beginning, and a standardised script from 2.0 onwards.

One problem with executable patches was the fact you couldn't reason
about the packaging without first executing them, in the general case,
which was frowned upon by security-concious people who like automated
package testing tools.

Does that not apply here?

-- 
Jon Dowland


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111208101158.GB22474@pris

Re: Getting dh_install to do what we need

[Arno Töll]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hello,

On 08.12.2011 10:44, Goswin von Brederlow wrote:
> Or for the more general case:
> 
> override_dh_auto_install:
> debian/libfoo.my-install-script
[..]

> This new feature stinks of black-box magic that will make people crazy
> trying to find/fix a prolem in somebody elses package. The thing that
> make cdbs so bad.


I beg to disagree. You made  very good example why the former (your)
approach was a black-box indeed, whereas the newer one in fact
standardizes things in parts.

See, in your case libfoo.my-install-script could be doing anything,
including but not limited to copying, moving, creating, changing a file
without any information on what's going on.
Now, by using that feature you are forced to generate a (dynamic) file
listing instead and everyone can execute that script and see the results
without /actually/ installing anything to a binary package.

Of course you could discuss whether executing scripts is necessarily a
better idea than having some semantically parsed *.install file magic
instead, but that's an implementation detail.
Up to now, you are all discussing why "chmod +x foo.install" is so much
worse than overriding dh_install by your own dark magic but you should
be realizing you just traded one black-box for another
not-so-black-but-still-very-dark-box.


- -- 
with kind regards,
Arno Töll
IRC: daemonkeeper on Freenode/OFTC
GnuPG Key-ID: 0x9D80F36D
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJO4I1WAAoJEMcrUe6dgPNtyWQQAMy6IsiNwKUT+iGhTa/XzL4i
Cfrr4hmOp/pxrVNrbAE7S+c4uljcDaYzWjx381SQNGD0YRFIM/yI8A5n0rmXxWCd
YP1jVgbmCRw8hI3DwJh+i2LaM7DwjtrgFu8V62AUzwVH66UsKolw4csPFPvkfZh4
CzLQdYYcQ5aMqUY4gVFfbl7e/Zvdxj8T/QB3eKgV7BgpRxpV/EcpgSPpicIdWRxW
96kafQIJnyPF7do8DR0XzEareNCfj9jfly6DeiHbvmxeVl+qNU0AwMqwztzdgbmc
dZfKr3RHKEX15jHwmKBWEcQ6dP2q9QiKteoKefRLNFPxBTBHG75tirrBGXuW673G
n9nh8masrKcKUCTbI6TO6+vNK8qe5nAECxoHhzhBx6EN99n8IVmytxarDhvU1imT
2bq1WubtNQ3dwCGWysYm9NjKMhgb5tHf+Ytd1q1UHbX0tbazAvWYUg4soduFpU9q
TMbuvNexXkw2EnBSm6s6cAYOdv99v+aFpEMoVniX+48yNo7t8qc93kFQqCOBhY3T
s3BDSAotA9gwQbVlC0SOf+QK44I+GnCnDiTeBVEPNJdVeAonyJl07JTkXdbr7TwB
BPwVSbgn2Ugz5V6XnmrUDMdUz1GeQ8HP7loU81KENqjuVrH+PWUI1ykjMltXfO4T
Clc/Sos/fuWuA/2PRDwZ
=Vfnx
-END PGP SIGNATURE-


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4ee08d56.30...@toell.net

Re: Getting dh_install to do what we need

[Gergely Nagy]

Goswin von Brederlow  writes:

> Arno Töll  writes:
>
>> Your own script-fu in debian/rules or external scripts isn't exactly the
>> next best thing to read and learn how a foreign package works and there
>> /are/ use cases where dh_install isn't flexible enough to deal with the
>> problem by using the possibilities you had before. Renaming files and
>> multi-arch support is what comes me in mind immediately.
>
> Yes, there are cases where dh_install isn't the tool you should or even
> can use.
>
> But that is what override is for. All this feature does it replace an
> override for dh_install with one to chmod +x the script.

To be honest, I find executable debhelper files far cleaner and more
understandable than a maze of overrides in something that is very far
from your old trusty Makefile by now.

At least, if the various tasks one wishes to do with the executable
config files is standardised one way or the other (and that can be done:
see dpatch, which provided executable patches for *years*, and we never
saw wide-spread abuse, did we?), then all it takes is having a look at
the executable config file, perhaps the manpage of the interpreter tool,
and done.

It can make perfect sense.

And in the end, I prefer this:

,
| #!/usr/lib/debhelper/dh_subst
| debian/tmp/usr/lib/${DEB_HOST_MULTIARCH}/libfoo.so.*
| debian/${VARIANT}/foo.conf /etc/foo/
`

Over this:

,
| override_dh_install:
|   install -d debian/libfoo0/usr/lib/${DEB_HOST_MULTIARCH}
|   cp debian/tmp/usr/lib/${DEB_HOST_MULTIARCH}/libfoo.so.* 
debian/libfoo0/usr/lib/${DEB_HOST_MULTIARCH}/ 
|
|   cp debian/${VARIANT}/foo.conf debian/libfoo0/etc/foo/
`

Perhaps it is just me, but the former makes much more sense to me, and
is easier to modify, and easier to copy to other similar packages (or
even duplicate it, if one's building multiple binaries from the same
source, all of which require similar treatment).

-- 
|8]


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/874nxbxtvo.fsf@algernon.balabit

Re: Getting dh_install to do what we need

[Goswin von Brederlow]

Arno Töll  writes:

> Your own script-fu in debian/rules or external scripts isn't exactly the
> next best thing to read and learn how a foreign package works and there
> /are/ use cases where dh_install isn't flexible enough to deal with the
> problem by using the possibilities you had before. Renaming files and
> multi-arch support is what comes me in mind immediately.

Yes, there are cases where dh_install isn't the tool you should or even
can use.

But that is what override is for. All this feature does it replace an
override for dh_install with one to chmod +x the script.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87liqn4clv.fsf@frosties.localnet

Re: Getting dh_install to do what we need

[Goswin von Brederlow]

Don Armstrong  writes:

> On Wed, 07 Dec 2011, Josselin Mouette wrote:
>> So, to sum it up. Before, you would do in debian/rules: 
>> sed s/@DEB_HOST_MULTIARCH@/${DEB_HOST_MULTIARCH}/ 
>> debian/libfoo.install.in > debian/libfoo.install
>> 
>> Now, you will do in debian/foo.install:
>> #! /bin/sh 
>> sed s/@DEB_HOST_MULTIARCH@/$(dpkg-architecture 
>> -qDEB_HOST_MULTIARCH)/ << EOF
>> …
>> EOF
>
> Although, honestly, you could always do this in debian/rules with 
>
>  override_dh_auto_install:
> debian/libfoo.install.sh > debian/libfoo.install
> dh_install;
>
> so besides the breakage generated by extraneously executable
> libfoo.install files, not much has changed in terms of obfuscation.
> That said, explicitly calling the appropriate script to generate the
> install file before the various dh_* bits were engaged is a bit
> clearer.
>
>
> Don Armstrong

Or for the more general case:

override_dh_auto_install:
debian/libfoo.my-install-script

At least with an override it would be clear from debian/rules that there
is more going on there than just copying a few files.

This new feature stinks of black-box magic that will make people crazy
trying to find/fix a prolem in somebody elses package. The thing that
make cdbs so bad.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87pqfz4cto.fsf@frosties.localnet

Re: Getting dh_install to do what we need

[Russ Allbery]

Mike Hommey  writes:

> Note you also need to add a chmod +x debian/*.sh in debian/rules, since
> apart from debian/rules, nothing has the executable bit after
> dpkg-source -x.

I believe the 3.0 (quilt) format fixes this.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ty5bv69p@windlord.stanford.edu

Re: Getting dh_install to do what we need

[Mike Hommey]

On Wed, Dec 07, 2011 at 11:47:49PM +0100, Josselin Mouette wrote:
> For those who haven’t followed, the latest debhelper upload includes the
> following change:
> 
>* Debhelper config files may be made executable programs that output the
>  desired configuration. No further changes are planned to the config file
>  format; those needing powerful syntaxes may now use a programming 
> language
>  of their choice. (Be careful aiming that at your feet.)
>  Closes: #235302, #372310, #235302, #614731,
>  Closes: #438601, #477625, #632860, #642129
> 
> So, to sum it up. Before, you would do in debian/rules: 
> sed s/@DEB_HOST_MULTIARCH@/${DEB_HOST_MULTIARCH}/ 
> debian/libfoo.install.in > debian/libfoo.install
> 
> Now, you will do in debian/foo.install:
> #! /bin/sh 
> sed s/@DEB_HOST_MULTIARCH@/$(dpkg-architecture -qDEB_HOST_MULTIARCH)/ 
> << EOF
> …
> EOF
> (Bonus points for implementing this in a different language for each
> package.)

Note you also need to add a chmod +x debian/*.sh in debian/rules, since
apart from debian/rules, nothing has the executable bit after
dpkg-source -x.

Mike


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111208065319.gb4...@glandium.org

Re: Getting dh_install to do what we need

[Gergely Nagy]

Chow Loong Jin  writes:

> On 08/12/2011 07:12, Gergely Nagy wrote:
>> 
>> And by extend, I mean something like:
>> 
>> ,
>> | #! /usr/bin/dh_multiarchify
>> | /usr/lib/${DEB_HOST_MULTIARCH}/*
>> `
>> 
>> The /usr/bin/dh_multiarchify script would do the sed magic. This would
>> make the simple cases simple, and would still allow one to do
>> complicated magic, if so need be.
>
> I would much rather prefer a substvars-ish approach, similar to what
> dpkg-gencontrol does. Making .install executable at all (even if it has a
> standard #! interpreter) makes things a lot harder to standardize. Like 
> Josselin
> mentioned, bonus points for implementing this in a different language for each
> package.

I disagree. Look at dpatch. It had executable patches since the
beginning, and a standardised script from 2.0 onwards.

Too many different implementations were never a problem in that case,
and shouldn't be in this, either. I believe most of the people who do
need this kind of feature realize that reinventing the wheel would be
counter-productive, and will try to come to an agreement, and use very
similar approaches.

-- 
|8]


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/874nxby1oz@luthien.mhp

Re: Getting dh_install to do what we need

[Chow Loong Jin]

On 08/12/2011 07:12, Gergely Nagy wrote:
> 
> And by extend, I mean something like:
> 
> ,
> | #! /usr/bin/dh_multiarchify
> | /usr/lib/${DEB_HOST_MULTIARCH}/*
> `
> 
> The /usr/bin/dh_multiarchify script would do the sed magic. This would
> make the simple cases simple, and would still allow one to do
> complicated magic, if so need be.

I would much rather prefer a substvars-ish approach, similar to what
dpkg-gencontrol does. Making .install executable at all (even if it has a
standard #! interpreter) makes things a lot harder to standardize. Like Josselin
mentioned, bonus points for implementing this in a different language for each
package.

-- 
Kind regards,
Loong Jin



signature.asc
Description: OpenPGP digital signature

Re: Getting dh_install to do what we need

[Chow Loong Jin]

On 08/12/2011 07:12, Gergely Nagy wrote:
> 
> And by extend, I mean something like:
> 
> ,
> | #! /usr/bin/dh_multiarchify
> | /usr/lib/${DEB_HOST_MULTIARCH}/*
> `
> 
> The /usr/bin/dh_multiarchify script would do the sed magic. This would
> make the simple cases simple, and would still allow one to do
> complicated magic, if so need be.

I would much rather prefer a substvars-ish approach, similar to what
dpkg-gencontrol does. Making .install executable at all (even if it has a
standard #! interpreter) makes things a lot harder to standardize. Like Josselin
mentioned, bonus points for implementing this in a different language for each
package.

-- 
Kind regards,
Loong Jin



signature.asc
Description: OpenPGP digital signature

Re: Getting dh_install to do what we need

[Arno Töll]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

I am pretty sure I'll regret to comment on this thread soonish, but ...

On 08.12.2011 00:12, Gergely Nagy wrote:
> I'd like to offer another option: deal with it, and instead of doing any
> of the above (none of which would be a good idea in the long run, except
> maybe the first - but I quite like this feature as it is), embrace
> it, and extend it.

I am one of those who see the upsides of this change as well.

There are quite a few limitations in dh_install you can come over by
using such a script in a yet semi-expected behavior as you get a
well-defined output to anyone trying to understand a foreign package.

Your own script-fu in debian/rules or external scripts isn't exactly the
next best thing to read and learn how a foreign package works and there
/are/ use cases where dh_install isn't flexible enough to deal with the
problem by using the possibilities you had before. Renaming files and
multi-arch support is what comes me in mind immediately.

Gergely's idea to support standardized scripts to be executed also
sounds like a good idea.

That said, I'd appreciate if we could limit the usage to scripts in a
more sane dimension. For example for dh_install only (I fail to see how
it would be useful to other debhelper scripts - does any of you see a
legit use case for another debhelper script?), and, most important, if
we could limit this behavior to debhelper compatibility level 9 to not
break any existing source package out there which might accidentally
have +x on such a file.



- -- 
with kind regards,
Arno Töll
IRC: daemonkeeper on Freenode/OFTC
GnuPG Key-ID: 0x9D80F36D
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJO3/m5AAoJEMcrUe6dgPNtHjEQAI1+wFJADokc1VdY7L/miCIt
1ISaSWceOK1CHMQfDg1UK6N+MueWuF/KSIdrIACprjaYA2mm67ongyNWONT1MgeM
lkkdqVeDL0LgJSSO3/DGkKNKXiAOO37j61S8aCSI/rfXVZU0WKsovUiyzv9T2EK/
IjzAKBKPCltOnqzMQMUnx91fotLD5Shh155q/rs02fB7wmN5xT1L3BTO7LLAC1Kj
DOfpRVBLucL/rK4gSFpBJ3eL3zeSqtD3PXeOaqr8j3CwECThzT9KIVjq4ZeTGseT
ky24qDmAS0odpoEhfNXev5EsP7oysLiX61ZTgSycB4y+BssRMo15HuJyD3KjFxvE
9LiPsYDUal2d/+Za2yDkfCMHd3aQmwn/n0SsFc1kqcgsCgX2PxL7s2w+ndAI1vYP
g0FF9sizx6HLUndC2uXP4HvupKb+1YqkH14fPBLR62cDXJNRKZA8vcEegZ6+x/CL
M7IUhuYxoFs1jEBmSXUV5YGNvldmAdNfIObLI18wGxNVfUpahf8/spYNxtcXFt4H
wlJkX82i/nf2XeL3R2lSGBwYy+i/fQnV2m2R+eyM2CgomYumeqE7VXT4J5H6977l
1ptPaIm8PG+wL7X0g9UP4/qgPlvRJEMzwctEBHT3EfZ3iIMuc/3Hsclyo2MV2Yv8
mOMf1H/w9T3695QS5lwC
=hmbG
-END PGP SIGNATURE-


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4edff9b9.1010...@toell.net

Re: Getting dh_install to do what we need

[Gergely Nagy]

Josselin Mouette  writes:

> So, to sum it up. Before, you would do in debian/rules: 
> sed s/@DEB_HOST_MULTIARCH@/${DEB_HOST_MULTIARCH}/ 
> debian/libfoo.install.in > debian/libfoo.install
>
> Now, you will do in debian/foo.install:
> #! /bin/sh 
> sed s/@DEB_HOST_MULTIARCH@/$(dpkg-architecture -qDEB_HOST_MULTIARCH)/ 
> << EOF
> …
> EOF
> (Bonus points for implementing this in a different language for each
> package.)
>
> Now that we’ve made incredible progress in terms of obfuscation, I’d
> appreciate if we could have a working solution that does not require
> scripting for the most trivial operations. So what remains? 
>   * Convincing Joey to revert this useless change and actually
> commit something useful. 
>   * NMU debhelper. 
>   * Technical committee. 
>   * Fork dh_install in a new package.
>
> Any comments before someone starts doing either of these?

I'd like to offer another option: deal with it, and instead of doing any
of the above (none of which would be a good idea in the long run, except
maybe the first - but I quite like this feature as it is), embrace
it, and extend it.

And by extend, I mean something like:

,
| #! /usr/bin/dh_multiarchify
| /usr/lib/${DEB_HOST_MULTIARCH}/*
`

The /usr/bin/dh_multiarchify script would do the sed magic. This would
make the simple cases simple, and would still allow one to do
complicated magic, if so need be.

Something similar has worked well for dpatch in the past, and could be
used in this case too.

-- 
|8]


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87hb1cugc6@luthien.mhp

Re: Getting dh_install to do what we need

[Don Armstrong]

On Wed, 07 Dec 2011, Josselin Mouette wrote:
> So, to sum it up. Before, you would do in debian/rules: 
> sed s/@DEB_HOST_MULTIARCH@/${DEB_HOST_MULTIARCH}/ 
> debian/libfoo.install.in > debian/libfoo.install
> 
> Now, you will do in debian/foo.install:
> #! /bin/sh 
> sed s/@DEB_HOST_MULTIARCH@/$(dpkg-architecture -qDEB_HOST_MULTIARCH)/ 
> << EOF
> …
> EOF

Although, honestly, you could always do this in debian/rules with 

 override_dh_auto_install:
debian/libfoo.install.sh > debian/libfoo.install
dh_install;

so besides the breakage generated by extraneously executable
libfoo.install files, not much has changed in terms of obfuscation.
That said, explicitly calling the appropriate script to generate the
install file before the various dh_* bits were engaged is a bit
clearer.


Don Armstrong

-- 
A people living under the perpetual menace of war and invasion is very
easy to govern. It demands no social reforms. It does not haggle over
expenditures on armaments and military equipment. It pays without
discussion, it ruins itself, and that is an excellent thing for the
syndicates of financiers and manufacturers for whom patriotic terrors
are an abundant source of gain.
 -- Anatole France

http://www.donarmstrong.com  http://rzlab.ucr.edu


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111207231113.ge21...@rzlab.ucr.edu

Getting dh_install to do what we need

[Josselin Mouette]

For those who haven’t followed, the latest debhelper upload includes the
following change:

   * Debhelper config files may be made executable programs that output the
 desired configuration. No further changes are planned to the config file
 format; those needing powerful syntaxes may now use a programming language
 of their choice. (Be careful aiming that at your feet.)
 Closes: #235302, #372310, #235302, #614731,
 Closes: #438601, #477625, #632860, #642129

So, to sum it up. Before, you would do in debian/rules: 
sed s/@DEB_HOST_MULTIARCH@/${DEB_HOST_MULTIARCH}/ 
debian/libfoo.install.in > debian/libfoo.install

Now, you will do in debian/foo.install:
#! /bin/sh 
sed s/@DEB_HOST_MULTIARCH@/$(dpkg-architecture -qDEB_HOST_MULTIARCH)/ 
<< EOF
…
EOF
(Bonus points for implementing this in a different language for each
package.)

Now that we’ve made incredible progress in terms of obfuscation, I’d
appreciate if we could have a working solution that does not require
scripting for the most trivial operations. So what remains? 
  * Convincing Joey to revert this useless change and actually
commit something useful. 
  * NMU debhelper. 
  * Technical committee. 
  * Fork dh_install in a new package.

Any comments before someone starts doing either of these?

-- 
 .''`.  Josselin Mouette
: :' :
`. `'
  `-


signature.asc
Description: This is a digitally signed message part