Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Ian Jackson]

Thanks to everyone for the comments and review.  I have written things
up on the wiki:
  https://wiki.debian.org/BuildProfile/upstream-cargo
and added the entry here:
  https://wiki.debian.org/BuildProfileSpec

Please comment here, if you think any of this doesn't make sense.

Thanks,
Ian.

-- 
Ian JacksonThese opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Simon McVittie]

On Fri, 16 Dec 2022 at 11:01:58 +, Ian Jackson wrote:
> With nopython, we want to *avoid doing the Python things at all*.  But
> "the Python things" here isn't "all Python things" - it's "certain
> Python things that appear in the outputs".  So that can't be done as a
> blanket exclusion on B-d.

As a concrete example of this distinction, src:vulkan-loader runs a
Python script during build to regenerate some mechanically-generated C
code, but its only end result is a C library and some C/C++ headers. In
principle it would be possible to have a build-profile that makes it use
the pre-generated version provided by upstream[1] instead of regenerating
it, perhaps  or something, but that would be outside the scope
of  (and apparently not useful enough in this case for anyone
to want to implement it).

Conversely, the main end result of compiling src:avahi is a collection
of C libraries (and a daemon and some tools), but it optionally also
produces a Python module (and a non-essential Python program that uses
it). Excluding those Python parts with  is in-scope: it's
reasonable to say "I want a C/C++ library stack with libavahi and others,
but I don't care about any Python bindings" (for instance for use inside
an app-container that is designed for C/C++ code).

smcv

[1] in the past the pre-generated version was used unconditionally,
until #981362 changed the build to regenerate it unconditionally

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Jonas Smedegaard]

Quoting Russ Allbery (2022-12-16 17:07:00)
> Jonas Smedegaard  writes:
> > Quoting Russ Allbery (2022-12-15 17:41:15)
> 
> >> This is why I don't agree with Jonas: yes, there *are* other ways to
> >> achieve the same goal, but they're more complicated and harder to
> >> explain.  The user experience of this build profile looks a lot simpler
> >> and cleaner.
> 
> > I don't have a firm opinion here, so wonder what you disagree with,
> > Russ.
> 
> > For the record, prior to this thread Ian reached out to me privately
> > where we discussed my work on forking dh-cargo to behave more
> > "Debian-ish" than what is developed by the Rust team.
> 
> > Perhaps this is a language thing: When I "question" Ians proposed
> > approach, I do not imply that I reject it or am against it, only that I
> > want to better understand it.
> 
> Oh, I'm sorry, this is probably just a sequence of misunderstandings.  I
> think I misunderstood your message as being somewhat opposed, and then I
> think you read my "disagree" as stronger than I meant it.  All I was
> trying to say is that I think (based on my very sketchy understanding of
> build profiles) that this would have a lower bar of entry for building
> non-Debian packages as a stepping stone to proper Debian packages than
> pre-downloading the dependencies.

Ah, indeed I failed to read nuance into your post.  Sorry for my
unfounded criticism!

 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/
 * Sponsorship: https://ko-fi.com/drjones

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Russ Allbery]

Jonas Smedegaard  writes:
> Quoting Russ Allbery (2022-12-15 17:41:15)

>> This is why I don't agree with Jonas: yes, there *are* other ways to
>> achieve the same goal, but they're more complicated and harder to
>> explain.  The user experience of this build profile looks a lot simpler
>> and cleaner.

> I don't have a firm opinion here, so wonder what you disagree with,
> Russ.

> For the record, prior to this thread Ian reached out to me privately
> where we discussed my work on forking dh-cargo to behave more
> "Debian-ish" than what is developed by the Rust team.

> Perhaps this is a language thing: When I "question" Ians proposed
> approach, I do not imply that I reject it or am against it, only that I
> want to better understand it.

Oh, I'm sorry, this is probably just a sequence of misunderstandings.  I
think I misunderstood your message as being somewhat opposed, and then I
think you read my "disagree" as stronger than I meant it.  All I was
trying to say is that I think (based on my very sketchy understanding of
build profiles) that this would have a lower bar of entry for building
non-Debian packages as a stepping stone to proper Debian packages than
pre-downloading the dependencies.

-- 
Russ Allbery (r...@debian.org)  

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Jonas Smedegaard]

Quoting Russ Allbery (2022-12-15 17:41:15)
> Ian Jackson  writes:
> 
> > I would like to add the following entry to the table of build
> > profiles in BuildProfileSpec:
> 
> >  Profile anme: `cargo-upstream`
> >  Can cause changes to built binaries, including functional chnges. (Y)
> >  Should not cause changes to set of build debs. (N)
> >  Description:
> >Obtain Rust dependencies from crates.io, not from Debian; use a
> >`Cargo.lock` from the source package if there is one.  Will usually
> >cause cargo to make network accesses.  Ideally, behaviour of the
> >resulting programs will be the same, but this is not guaranteed.
> 
> > This is useful when developing packages which are to be uploaded to
> > Debian, but whose build dependencies are not yet available in the
> > targeted suite.
> 
> I think this is a great idea.
> 
> In addition to the benefits you list (and less importantly, because
> there's less direct affect on Debian), having this feature available would
> make it easier to create a first pass of a package of a Rust tool intended
> for local use.  I often build semi-broken short-cut packages locally of
> things I need on some system but haven't yet found the time to make them
> properly ready for Debian, and then gradually finish the work to get them
> into Debian.  It would be very useful to have this support to get over the
> initial barrier to making a Debian package.
> 
> I think it would increase the likelihood that I'd eventually do the work
> to get all the way to an uploadable Debian package, whereas if it's
> difficult to bootstrap from Cargo, I'm much more likely to let laziness
> win, just build a static binary, and copy it around.
> 
> This is why I don't agree with Jonas: yes, there *are* other ways to
> achieve the same goal, but they're more complicated and harder to explain.
> The user experience of this build profile looks a lot simpler and cleaner.

I don't have a firm opinion here, so wonder what you disagree with, Russ.

For the record, prior to this thread Ian reached out to me privately
where we discussed my work on forking dh-cargo to behave more
"Debian-ish" than what is developed by the Rust team.

Perhaps this is a language thing: When I "question" Ians proposed
approach, I do not imply that I reject it or am against it, only that I
want to better understand it.

Since my last post in this thread, Ian reached out to me privately, but
have chosen not to share that conversation here, which leaves me in an
odd situation appearing as being against this proposal.  I am not!


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/
 * Sponsorship: https://ko-fi.com/drjones

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Ian Jackson]

Helmut Grohne writes ("Re: Proposed `cargo-upstream` dpkg-buildpackage etc. 
build profile"):
> On Thu, Dec 15, 2022 at 11:44:34PM +, Ian Jackson wrote:
> > I'm not sure precisely how this feature could (or should) be made
> > available to *all* application packages in a central way.  Having
> > tools like debcargo automatically add the profile to the build deps
> > produces a lot of bloat.  I'm hoping there is a better way.
> 
> I think this becomes the crux of the matter. I see basically two
> distinct ways to move forward:
...
> What are your thoughts on updating Build-Depends?

I think that in packages where the maintainer has deliberately chosen
to support this build profile, the maintainer should annotate the
Build-Depends.

> B
> 
> In this case, the profile description should be more clear about the
> intended scope (rust application packages).

Yes.

Is "application package" precisely the right term?  It could also
apply to: packages which provide shared objects (eg a .so plugin);
ABI/API libraries (eg trad C libraries whose innards are now rewritten
in Rust); packages which provide an executable that isn't an
application.

The possible scope includes any package whose .deb contains
executables (including possibly WASM, as well as native code) made by
compiling Rust.  It might even include packages where Rust is used
only as part of the build system.

One way to describe the exception would be to say something like
  (except Rust library packages)
or
  (except Rust library packages themselves)

> I also do not see us automating the update to Build-Depends.

I agree but for different reasons to yours.  I don't want this done to
every Rust leaf package routinely, which would be the natural end
point of automating this.

> > This could be made easier.  Maybe tools like sbuild could have a
> > sepaarate option to disregard build deps matching a wildcard pattern,
> > or something.
> 
> The idea is good in principle. Let me do a brief excursion into
> Multi-Arch for comparison. By default, we have an architecture
> constraint on dependencies. It can be lifted either by M-A:foreign on
> the target of the edge or by :any on the source of the edge. A build
> profile is a bit like :any and annotates the source of an edge. Your
> proposed wildcard pattern would be a target annotation for build
> profiles in this analogy while the current profile implementation has no
> way to do such target annotations.

Yes.  The information would ideally be centralised: "if you have a
Build-Dependency matching librust-*-dev, disregard it with this
build-profile".  So the edges would be "annotated" but implicitly.

Failing that I guesss we could add a general facility of this shape
  Disregard-build-depends-with-build-profile: upstream-cargo
a la M-A.

> This idea is also applicable beyond rust. With the nopython build
> profile, we might want to automatically annotate Python dependencies.
> Unfortunately, we are now in a place where a significant portion of
> packages uses Python modules during build (e.g. sphinx) while not
> producing any Python modules or extensions as output. As such, we cannot
> mechanically tell how such a dependency is being used. Maybe a package
> later contains a rust crate that is not shipped, but only used during
> build. In this case, we wouldn't want to annotate the relevant
> dependencies either.

The analogy with "nopython" is rather misleading.  (You could have
"norust" too; not sure if that's in the list right now.)

With nopython, we want to *avoid doing the Python things at all*.  But
"the Python things" here isn't "all Python things" - it's "certain
Python things that appear in the outputs".  So that can't be done as a
blanket exclusion on B-d.

With upstream-cargo, we want to *do all the Rust things, but get them
all from upstream crates.io*.  This should affect *every* dependency
on a cargo package (except ones that are within the source package
itself).

So if a package that previously didn't do any Rust things, gains a
Rust crate that is not "shipped" (to /usr/share/cargo/registry), then
this buid profile does become nontrivially applicable.  (Of course
this build profile is trivially applicable to packages which don't
involve cargo at all).

The reason why the upstream-cargo profile ought not to be provided for
a Rust library crate is not becuase it's incoherent or wrong in a
narrow sense.

A Rust library crate typically *does* actually run rustc during rules
build, as a check:
  https://deb.debian.org/debian/pool/main/r/rust-cxx/rust-cxx_1.0.73-1.dsc
And in principle it is perfectly coherent to say "please build this
source package, but do the check with upstream Rust Dependencies".

But this is not *useful* to support as a build profile.  Narr

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Helmut Grohne]

Hi Ian,

On Thu, Dec 15, 2022 at 11:44:34PM +, Ian Jackson wrote:
> I'm not sure precisely how this feature could (or should) be made
> available to *all* application packages in a central way.  Having
> tools like debcargo automatically add the profile to the build deps
> produces a lot of bloat.  I'm hoping there is a better way.

I think this becomes the crux of the matter. I see basically two
distinct ways to move forward:

A

Do not annotate Build-Depends. We've seen a number of reasons for not
doing it, but the most significant are bloating sources and that it is
quite mechanical work while editing debian/control is usually manual.

In this case, a build profile is the wrong tool for the job. The value
for draw from build profiles precisely comes from formalizing their
effects in debian/control. If your profile skips this step, then it is
not the right tool.

> But even without modifying the build dependencies, it is a lot easier
> to cause a build to run without the stated build deps being satisfied,
> than it is to do violence to the package build system.

I concur, but in this case, it can be a DEB_BUILD_OPTIONS or some other
variable.

Admittedly, I think that the value of this feature significantly reduces
without annotating Build-Depends, because you can no longer just use
sbuild or pbuilder. The ease of use suffers.

B

In this case, the profile description should be more clear about the
intended scope (rust application packages).

I also do not see us automating the update to Build-Depends.

> This could be made easier.  Maybe tools like sbuild could have a
> sepaarate option to disregard build deps matching a wildcard pattern,
> or something.

The idea is good in principle. Let me do a brief excursion into
Multi-Arch for comparison. By default, we have an architecture
constraint on dependencies. It can be lifted either by M-A:foreign on
the target of the edge or by :any on the source of the edge. A build
profile is a bit like :any and annotates the source of an edge. Your
proposed wildcard pattern would be a target annotation for build
profiles in this analogy while the current profile implementation has no
way to do such target annotations.

This idea is also applicable beyond rust. With the nopython build
profile, we might want to automatically annotate Python dependencies.
Unfortunately, we are now in a place where a significant portion of
packages uses Python modules during build (e.g. sphinx) while not
producing any Python modules or extensions as output. As such, we cannot
mechanically tell how such a dependency is being used. Maybe a package
later contains a rust crate that is not shipped, but only used during
build. In this case, we wouldn't want to annotate the relevant
dependencies either.

So I think annotating rust Build-Depends has to be a semi-manual task at
least. Updating dh-cargo or debcargo to automatically adapt its
behaviour still seems useful though as that reduces the amount of work
to updating Build-Depends.

What are your thoughts on updating Build-Depends?

Helmut

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Ian Jackson]

Helmut Grohne writes ("Re: Proposed `cargo-upstream` dpkg-buildpackage etc. 
build profile"):
> Thank you very much for immediately taking this to the list.

You're welcome.

Thanks for the review and the penetrating questions.

> I think that you imply here that all of the rust libraries would be
> annotated . Do you confirm?

Yes.

> Yes:
> 
> I caution that we increase our (uncompressed) sources by 200KB for
> adding this. This assumes changing every package, but the scope
> isn't entirely clear, see below. Do you confirm that this is the
> intention and that you think this increase is fine?

No, this is not my intention.  And IMO that increase would be
undesirable.

This facility is certainly cxompletely useless for a Rust library
package - those whose .debs are the Rust text and metadata, for
building other programs against.  Those things don't even have "real"
Rust build-dependencies - the Rust build deps are just there for
testing etc.  And there is no use to anyone building such a package if
they are using upstream crates.io packages.

So at the very least we're limiting this to leaf packages containing
binaries.  And, probably even then, this will only be useful when the
Debian or upstream maintainers want this hybrid or stepping stone
workflow.

I was imagining that this would be done ad-hoc by a maintainer when
they had a reason to do it, not that it would be supported by even
every leaf package.  In some cases it might be a transitional state.

I do think it would be useful for the central tooling to offer this
build profile for every package, but I don't think it is appropriate
to annotate all the Rust build depends for every package this way.
Tools like debcargo ought *not* to add this build profile to the Rust
deps.

If this profile turns out to be widely desired for "leaf packages at
random", filtering (or dummy-satisfying) the build dependencies should
be done some other way.

> Which packages should support this profile? I see the value for
> applications. Do you also intend it for libraries? If yes, how do they
> benefit?

They don't.

> How do you intend to transition packages to support this profile?
> Should that happen on an as-needed basis? Should it happen as a mass
> commit? Do you want it mass uploaded?

As needed.  No mass commit, no mass upload, no automated change.

> Given all of the mails in the thread thus far, you've convinced me:
>  * That the requested feature is useful.
>  * That it should be ecosystem-specific.
>  * That it needs to be easy to use (and that build profiles satisfy this
>requirement).
> 
> On the bike coloring front, I do prefer putting the package ecosystem
> last (i.e. upstream-cargo > cargo-upstream) for the consistency reasons
> that you gave in a reply.

I think I'm sold on that name question.

> I would also question the "upstream" part as it wasn't obvious to me
> initially. Good alternatives that aren't too long are not easy to come
> by, but maybe you have a reason to reject "external"? I think "external"
> would more strongly highlight that a build is no longer self-contained.

I don't have strong feelings about this.  I chose "upstream" because
that seems to be the term of art that people working in this space
use, to distinguish the general public package repo, from the Debian
one.

> > However, this could be a generally useful feature for Debian's cargo
> > tooling to support, and I think it could do so in a general way so
> > that this profile would be available in most Debian packages
> > containing Rust code, without package-specific work.  Whether to
> > implement such a feature is a matter for the maintainers of dh_cargo
> > et al.; IMO the build profile registration is useful even ad-hoc,
> > without any general feature.
> 
> Yeah, having this supported generically seems very useful. It would be
> nice to have a supportive reply from one of the dh-cargo maintainers
> here. (I do not see this as a requirement.)
> 
> I note that "without package-specific work" implies that you wouldn't
> attach build profiles to Build-Depends, which was my initial question.
> I'll stop second guessing here and wait for your answer.

I'm not sure precisely how this feature could (or should) be made
available to *all* application packages in a central way.  Having
tools like debcargo automatically add the profile to the build deps
produces a lot of bloat.  I'm hoping there is a better way.

But even without modifying the build dependencies, it is a lot easier
to cause a build to run without the stated build deps being satisfied,
than it is to do violence to the package build system.

This could be made easier.  Maybe tools like sbuild could have a
sepaarate option to disregard build deps matching a wi

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Helmut Grohne]

Hi Ian,

Thank you very much for immediately taking this to the list. The
discussion I've seen thus far seems very constructive and useful to me.
Thanks to the other participants as well.

On Thu, Dec 15, 2022 at 10:41:13AM +, Ian Jackson wrote:
> I would like to add the following entry to the table of build
> profiles in BuildProfileSpec:
> 
>  Profile anme: `cargo-upstream`
>  Can cause changes to built binaries, including functional chnges. (Y)
>  Should not cause changes to set of build debs. (N)
>  Description:
>Obtain Rust dependencies from crates.io, not from Debian; use a
>`Cargo.lock` from the source package if there is one.  Will usually
>cause cargo to make network accesses.  Ideally, behaviour of the
>resulting programs will be the same, but this is not guaranteed.


I think that you imply here that all of the rust libraries would be
annotated . Do you confirm?

Yes:

I caution that we increase our (uncompressed) sources by 200KB for
adding this. This assumes changing every package, but the scope
isn't entirely clear, see below. Do you confirm that this is the
intention and that you think this increase is fine?

No:

Would this profile show up in debian/control in any way? If not, why
use a build profile rather than an option?

(Please do skip questions that do not apply.)

Which packages should support this profile? I see the value for
applications. Do you also intend it for libraries? If yes, how do they
benefit?

How do you intend to transition packages to support this profile?
Should that happen on an as-needed basis? Should it happen as a mass
commit? Do you want it mass uploaded?

> This is useful when developing packages which are to be uploaded to
> Debian, but whose build dependencies are not yet available in the
> targeted suite.

Given all of the mails in the thread thus far, you've convinced me:
 * That the requested feature is useful.
 * That it should be ecosystem-specific.
 * That it needs to be easy to use (and that build profiles satisfy this
   requirement).

On the bike coloring front, I do prefer putting the package ecosystem
last (i.e. upstream-cargo > cargo-upstream) for the consistency reasons
that you gave in a reply.

I would also question the "upstream" part as it wasn't obvious to me
initially. Good alternatives that aren't too long are not easy to come
by, but maybe you have a reason to reject "external"? I think "external"
would more strongly highlight that a build is no longer self-contained.

In any case, I think the naming questions are solvable one way or
another, so please focus on the non-naming aspects first. I don't feel
too strongly about naming.

> However, this could be a generally useful feature for Debian's cargo
> tooling to support, and I think it could do so in a general way so
> that this profile would be available in most Debian packages
> containing Rust code, without package-specific work.  Whether to
> implement such a feature is a matter for the maintainers of dh_cargo
> et al.; IMO the build profile registration is useful even ad-hoc,
> without any general feature.

Yeah, having this supported generically seems very useful. It would be
nice to have a supportive reply from one of the dh-cargo maintainers
here. (I do not see this as a requirement.)

I note that "without package-specific work" implies that you wouldn't
attach build profiles to Build-Depends, which was my initial question.
I'll stop second guessing here and wait for your answer.

Helmut

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Russ Allbery]

Ian Jackson  writes:

> I would like to add the following entry to the table of build
> profiles in BuildProfileSpec:

>  Profile anme: `cargo-upstream`
>  Can cause changes to built binaries, including functional chnges. (Y)
>  Should not cause changes to set of build debs. (N)
>  Description:
>Obtain Rust dependencies from crates.io, not from Debian; use a
>`Cargo.lock` from the source package if there is one.  Will usually
>cause cargo to make network accesses.  Ideally, behaviour of the
>resulting programs will be the same, but this is not guaranteed.

> This is useful when developing packages which are to be uploaded to
> Debian, but whose build dependencies are not yet available in the
> targeted suite.

I think this is a great idea.

In addition to the benefits you list (and less importantly, because
there's less direct affect on Debian), having this feature available would
make it easier to create a first pass of a package of a Rust tool intended
for local use.  I often build semi-broken short-cut packages locally of
things I need on some system but haven't yet found the time to make them
properly ready for Debian, and then gradually finish the work to get them
into Debian.  It would be very useful to have this support to get over the
initial barrier to making a Debian package.

I think it would increase the likelihood that I'd eventually do the work
to get all the way to an uploadable Debian package, whereas if it's
difficult to bootstrap from Cargo, I'm much more likely to let laziness
win, just build a static binary, and copy it around.

This is why I don't agree with Jonas: yes, there *are* other ways to
achieve the same goal, but they're more complicated and harder to explain.
The user experience of this build profile looks a lot simpler and cleaner.

-- 
Russ Allbery (r...@debian.org)  

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Jonas Smedegaard]

Quoting Ian Jackson (2022-12-15 14:05:32)
> Jonas Smedegaard writes ("Re: Proposed `cargo-upstream` dpkg-buildpackage 
> etc. build profile"):
> > What is the benefit of introducing a standardized flag for this
> > relatively narrow scope, compared to doing non-standardized fetching of
> > crates _before_ package build and building with those embedded?
> > Example of doing that is here: https://salsa.debian.org/debian/helvum
> > (essentially doing `cargo vendor --versioned-dirs debian/vendorlibs`).
> 
> IDK what precisely you mean there, and you've liked to the repo as a
> whole so I'm not sure what to look for.  Maybe you mean "what's wrong
> with pretending to vendor the dependencies" ?
> 
> Whatever approach is taken, it has to be controlled somehow.  For
> example, the paths to dependencies need to be adjusted, or the use of
> the Debian cargo wrapper enabled/disabled.
> 
> That control can be done by: (i) modifying the package source code
> (which is much more a pain, even if it's a toggle in a single place),
> (ii) ad-hoc environment variables or something (which don't survive
> sbuild) or (iii) a build profile.
> 
> ISTM that this kind of "build this package in s funky way" situation
> is precisely what build profiles are good for.

I disagree that it "has to be controlled somehow" *during build*, which
build profiles is about.

Looking at current list of profile names at
https://wiki.debian.org/BuildProfileSpec#Registered_profile_names it
seems to me that none of them involve violating the principle of source
package providing source for the build.

Essentially, you are proposing to formalize a way for injecting source
during build.

Debian currently has tooling for helping fetch source _before_ build,
most notably uscan, which does not involve defining a build profile
because what is built is static.

What I question is the sense of introducing dynamic-source builds, as
opposed to mangle-source-before-build which as a proof of concept is
possible to do with Rust creates through calling `cargo vendor ...`
_outside_ of regular dpkg-source mandated build targets.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/
 * Sponsorship: https://ko-fi.com/drjones

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Ian Jackson]

Wouter Verhelst writes ("Re: Proposed `cargo-upstream` dpkg-buildpackage etc. 
build profile"):
> I have just one question: why make this rust-specific? I think a similar
> thing might be useful for golang packages (who also don't support shared
> libraries), or, heck, even Perl (if I'm willing to test that the package
> works while I have the not-yet-packaged dependencies in my ~/perl5/lib)

You are absolutely right that many other upstream (language-specifc)
package managers will have analogous situations.

I think *each* of these should have their *own* build-profile, rather
than one portmanteau one.  If a package has both Rust and NPM
build-dependencies, say, a user may very well want to use Rust
dependencies from Debian and NPM dependencies from upstream, or vice
versa.

The profile name should be named after the upstream package manager
being influenced, not after the programming language, since some
languages have several.

So I think that we will probably want
   cargo-upstream
   npm-upstream
   golang-upstream (is this the right name?)
etc.

Maybe this means that the name ought to be the other way arond, so
they all group together in the table, like the `no*` options.
   upstream-cargo
   upstream-npm
   upstream-golang (is this the right name?)

I don't want to speak for those working with these other package
managers, so I don't propose to add all of those right away.  But
establishing the precedent will hopefulloy be helpful for them

So for now I'm proposing to add just the cargo one.  If someone
working with Node packages (say) tells me "we want this for npm too"
then great and we should do that right away.  Otherwise we should
wait until ti's wanted.

Ian.

-- 
Ian JacksonThese opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Ian Jackson]

Jonas Smedegaard writes ("Re: Proposed `cargo-upstream` dpkg-buildpackage etc. 
build profile"):
> Similar pain for other upstream ecosystems as well - I know about npm
> for NodeJS modules, but imagine it is similar for Java and others as
> well.

Yes.

> What is the benefit of introducing a standardized flag for this
> relatively narrow scope, compared to doing non-standardized fetching of
> crates _before_ package build and building with those embedded?
> Example of doing that is here: https://salsa.debian.org/debian/helvum
> (essentially doing `cargo vendor --versioned-dirs debian/vendorlibs`).

IDK what precisely you mean there, and you've liked to the repo as a
whole so I'm not sure what to look for.  Maybe you mean "what's wrong
with pretending to vendor the dependencies" ?

Whatever approach is taken, it has to be controlled somehow.  For
example, the paths to dependencies need to be adjusted, or the use of
the Debian cargo wrapper enabled/disabled.

That control can be done by: (i) modifying the package source code
(which is much more a pain, even if it's a toggle in a single place),
(ii) ad-hoc environment variables or something (which don't survive
sbuild) or (iii) a build profile.

ISTM that this kind of "build this package in s funky way" situation
is precisely what build profiles are good for.

> If there is really a need for a standardized flag, then what is the
> benefit of a narrow one specific to cargo, compared to a more general
> "fetches-source-during-build" that would be suitable also for e.g.
> NodeJS fetching npm modules?

Wouter made the same point - I will reply there, to both the CC lists.

Ian.

-- 
Ian JacksonThese opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Wouter Verhelst]

Hi Ian,

On Thu, Dec 15, 2022 at 10:41:13AM +, Ian Jackson wrote:
> I would like to add the following entry to the table of build
> profiles in BuildProfileSpec:
> 
>  Profile anme: `cargo-upstream`
>  Can cause changes to built binaries, including functional chnges. (Y)
>  Should not cause changes to set of build debs. (N)
>  Description:
>Obtain Rust dependencies from crates.io, not from Debian; use a
>`Cargo.lock` from the source package if there is one.  Will usually
>cause cargo to make network accesses.  Ideally, behaviour of the
>resulting programs will be the same, but this is not guaranteed.
[...]

I have just one question: why make this rust-specific? I think a similar
thing might be useful for golang packages (who also don't support shared
libraries), or, heck, even Perl (if I'm willing to test that the package
works while I have the not-yet-packaged dependencies in my ~/perl5/lib)

-- 
 w@uter.{be,co.za}
wouter@{grep.be,fosdem.org,debian.org}

I will have a Tin-Actinium-Potassium mixture, thanks.

Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Jonas Smedegaard]

[ reply to d-devel only, as more suitable for Debian-wide issue ]

Quoting Ian Jackson (2022-12-15 11:41:13)
> I would like to add the following entry to the table of build
> profiles in BuildProfileSpec:
> 
>  Profile anme: `cargo-upstream`
>  Can cause changes to built binaries, including functional chnges. (Y)
>  Should not cause changes to set of build debs. (N)
>  Description:
>Obtain Rust dependencies from crates.io, not from Debian; use a
>`Cargo.lock` from the source package if there is one.  Will usually
>cause cargo to make network accesses.  Ideally, behaviour of the
>resulting programs will be the same, but this is not guaranteed.
> 
> This is useful when developing packages which are to be uploaded to
> Debian, but whose build dependencies are not yet available in the
> targeted suite.

Similar pain for other upstream ecosystems as well - I know about npm
for NodeJS modules, but imagine it is similar for Java and others as
well.

What is the benefit of introducing a standardized flag for this
relatively narrow scope, compared to doing non-standardized fetching of
crates _before_ package build and building with those embedded?
Example of doing that is here: https://salsa.debian.org/debian/helvum
(essentially doing `cargo vendor --versioned-dirs debian/vendorlibs`).

If there is really a need for a standardized flag, then what is the
benefit of a narrow one specific to cargo, compared to a more general
"fetches-source-during-build" that would be suitable also for e.g.
NodeJS fetching npm modules?


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/
 * Sponsorship: https://ko-fi.com/drjones

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Proposed `cargo-upstream` dpkg-buildpackage etc. build profile

[Ian Jackson]

I would like to add the following entry to the table of build
profiles in BuildProfileSpec:

 Profile anme: `cargo-upstream`
 Can cause changes to built binaries, including functional chnges. (Y)
 Should not cause changes to set of build debs. (N)
 Description:
   Obtain Rust dependencies from crates.io, not from Debian; use a
   `Cargo.lock` from the source package if there is one.  Will usually
   cause cargo to make network accesses.  Ideally, behaviour of the
   resulting programs will be the same, but this is not guaranteed.

This is useful when developing packages which are to be uploaded to
Debian, but whose build dependencies are not yet available in the
targeted suite.

It allows a Debian maintainer or user to more easily rebuild the
package in a different suite to the one targeted, and can allow
unification of upstream (crates.io-based) and Debian development
without needing to carry a build system patch.

Particularly, it will sometimes ease the task of getting (leaf)
packages in Debian, as other aspects of the packaging can more easily
be tested separately from working on getting the dependencies in.

This build profile ought only to be used in packages that are intended
to appear in Debian (or a derivative) with .debs built from in-distro
dependencies (ie the dh_cargo-style local cargo replacement
repository, containing).  In such packages, obtaining dependencies
locally from within-distro is the default behaviour without
-Pcargo-upstream.

Invocation of this build profile will generally only be appropriate in
the context of manual development, or package-specific near-upstream
release workflows.  Certainly this profile ought not to be activated
for uploads to Debian, as it makes the package build depend on
out-of-distro inputs (and relies on external network access at build
time).

For now, I expect to use this only in an ad hoc fashion in affected
packages.  (I have a number of candidates.)

However, this could be a generally useful feature for Debian's cargo
tooling to support, and I think it could do so in a general way so
that this profile would be available in most Debian packages
containing Rust code, without package-specific work.  Whether to
implement such a feature is a matter for the maintainers of dh_cargo
et al.; IMO the build profile registration is useful even ad-hoc,
without any general feature.

Ian.

-- 
Ian JacksonThese opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

Re: Unmet :native dependency error when cross-compiling with dpkg-buildpackage

[Guillem Jover]

Hi!

On Wed, 2021-12-08 at 15:26:10 +0300, Uladzimir Bely wrote:
> The conditions are the following:
> Build Architecture: amd64
> Host Architecture: arm64

> The package I'm trying to cross-build is 'u-boot' with some patches. It has 
> build-time dependency 'swig:native'. But swig from Debian repos doesn't fit 
> all needs, so it's also compiled from sources with some patches applied. 
> Than, 
> local Debian repository is created (using reprepro), where 'swig:amd64' is 
> available.

> Later, 'u-boot' itself is being compiled using sbuild. During "Install 
> package 
> build dependencies" stage, 'swig:native' is correctly resolved to 
> 'swig:amd64'. All deps are correctly installed, then. But later, when it 
> comes 
> to dpkg-buildpackage stage, it fails with the following error:
> > Command: dpkg-buildpackage --sanitize-env -aarm64 -Pcross,nocheck -us -uc 
> > -b 
> -rfakeroot
> > "dpkg-checkbuilddeps: error: Unmet build dependencies: swig:native:

The swig package is arch:all m-a:foreign, which is disallowed as the
target of a :native arch-qualified dependency, and thus the dependency
is not satisfiable. See:

  
https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/tree/doc/multiarch.txt?h=pu/doc-multiarch-spec

BTW, you might want to rise this kind of question in the
debian-cross@l.d.o mailing list.

Thanks,
Guillem

Unmet :native dependency error when cross-compiling with dpkg-buildpackage

[Uladzimir Bely]

Hello.

While experimenting with cross-compilation I faced an error that looks like a 
bug in dpkg-buildpackage for me.

The conditions are the following:
Build Architecture: amd64
Host Architecture: arm64

The package I'm trying to cross-build is 'u-boot' with some patches. It has 
build-time dependency 'swig:native'. But swig from Debian repos doesn't fit 
all needs, so it's also compiled from sources with some patches applied. Than, 
local Debian repository is created (using reprepro), where 'swig:amd64' is 
available.

Later, 'u-boot' itself is being compiled using sbuild. During "Install package 
build dependencies" stage, 'swig:native' is correctly resolved to 
'swig:amd64'. All deps are correctly installed, then. But later, when it comes 
to dpkg-buildpackage stage, it fails with the following error:
> Command: dpkg-buildpackage --sanitize-env -aarm64 -Pcross,nocheck -us -uc -b 
-rfakeroot
> "dpkg-checkbuilddeps: error: Unmet build dependencies: swig:native:

To avoid this error, I have to add '-d' flag to dpkg-buildpackage which just 
makes it not to invoke dpkg-checkbuilddeps. In this case compilation of u-boot 
goes OK.

For me, it looks like an error in dpkg-buildpackage. I think, that in case of 
cross-compilation, ':native' should be resolved as ':amd64' when passed to 
dpkg-buildpackage.

I've attached a build log with an error. Here, previously compiled 'swig' is 
available from local 'isar' repository, while other build deps are downloaded 
from Debian mirrors.

-- 
Uladzimir Bely
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov+ sbuild -A -n -c isar-builder-2226 --extra-repository=deb [trusted=yes] 
file:///home/builder/u-boot-iot2050/isar-apt/iot2050-debian-arm64/apt/iot2050-debian
 isar main --host=arm64 --build=amd64 --starting-build-commands=runuser -u 
builder -- sh -c ":" --no-run-lintian --no-run-piuparts --no-run-autopkgtest 
--chroot-setup-commands=cp -n --no-preserve=owner 
/home/builder/u-boot-iot2050/rootfs//var/cache/apt/archives//*.deb -t 
/var/cache/apt/archives// || : --finished-build-commands=rm -f 
/var/cache/apt/archives//sbuild-build-depends-main-dummy_*.deb 
--finished-build-commands=cp -n --no-preserve=owner 
/var/cache/apt/archives//*.deb -t 
/home/builder/u-boot-iot2050/rootfs//var/cache/apt/archives// || : 
--build-dir=/build/tmp/work/iot2050-debian-arm64/u-boot-iot2050/2021.04-r0 
/build/tmp/work/iot2050-debian-arm64/u-boot-iot2050/2021.04-r0/u-boot-2021.04

dh clean --parallel

dh: No packages to build. Architecture mismatch: amd64, want: arm64

dpkg-source: warning: no source format specified in debian/source/format, see 
dpkg-source(1)

dpkg-source: warning: source directory 'u-boot-2021.04' is not 
- 'u-boot-iot2050-2021.04'

dpkg-source: info: using source format '1.0'
dpkg-source: info: building u-boot-iot2050 in u-boot-iot2050_2021.04.tar.gz

dpkg-source: info: building u-boot-iot2050 in u-boot-iot2050_2021.04.dsc

sbuild (Debian sbuild) 0.81.2 (31 January 2021) on 1420c20bd3c8

+==+
| u-boot-iot2050 2021.04 (arm64)   Wed, 08 Dec 2021 12:11:52 + |
+==+

Package: u-boot-iot2050
Version: 2021.04
Source Version: 2021.04
Distribution: UNRELEASED
Machine Architecture: amd64
Host Architecture: arm64
Build Architecture: amd64
Build Profiles: cross nocheck
Build Type: binary


I: NOTICE: Log filtering will replace 
'var/run/schroot/mount/isar-builder-2226-49e19571-410c-4f31-97b3-ae7435a8a993' 
with '<>'


+--+
| Chroot Setup Commands|
+--+



cp -n --no-preserve=owner 
/home/builder/u-boot-iot2050/rootfs//var/cache/apt/archives//*.deb -t 
/var/cache/apt/archives// || :
--



I: Finished running 'cp -n --no-preserve=owner 
/home/builder/u-boot-iot2050/rootfs//var/cache/apt/archives//*.deb -t 
/var/cache/apt/archives// || :'.

Finished processing commands.


I: NOTICE: Log filtering will replace 
'build/u-boot-iot2050-PwYChH/resolver-Ivfcqo' with '<>'

+--+
| Update chroot|
+--+



Get:1 
file:/home/builder/u-boot-iot2050/isar-apt/iot2050-debian-arm64/apt/iot2050-debian
 isar InRelease
Ign:1 
file:/h

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[David Kalnischkies]

(Disclaimer: This is a xkcd:386-like response to this subthread)

> Here's the current list of these packages on my system:
> 
>   $ aptitude -F '%p' search '~prequired !~E'

The list omits 'apt' as a libapt internally flags it as essential to
grant it the utmost protection by all clients along with its (due to
that) pseudo-essential dependencies both in terms of user actions as
well as (re)solver and installation ordering algorithms.

So, to see the "real" list, you need something like:
$ aptitude -F '%p' search '~prequired !~E' -o pkgCacheGen::ForceEssential=',' 
-o Dir::Cache=/dev/null

(The second -o is needed to prevent libapt from using its binary caches
and forces it to reparse everything in memory; the first -o is the knob
defaulting to 'apt' if unset. And yes, it is really ',' and you probably
don't want to know why and just accept it as meaning empty list)


Now, suggesting that apt is not an integral part of a Debian system and
could henceforth be removed is of course heresy! The only thing saving
you vile heretics is apts heavy involvement in the creation of these
chroots.


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Holger Levsen]

On Fri, Jan 17, 2020 at 03:02:06AM +, Paul Wise wrote:
> Personally, I think there is value in Daniel's work on bootstrapping
> Debian from other operating systems and Helmut's work on bootstrapping
> Debian from existing Debian architectures. Both are important projects
> and we need Debian and FLOSS in general to be more bootstrappable than
> it is now [...]

indeed!

> At some point I'd even like to see Debian
> bootstrapped from whatever comes out of the the Bootstrappable Builds
> project:
> 
> https://bootstrappable.org/

Vagrant (cc:ed) was working on this at the last reproducible builds summit.


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C



signature.asc
Description: PGP signature

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Paul Wise]

On Fri, 2020-01-17 at 10:58 +0100, Johannes Schauer wrote:
> Quoting Simon McVittie (2020-01-16 19:47:02)
> > I think I dimly remember someone setting up "the buildd from hell" which
> > deliberately did this as a QA mechanism, but it doesn't seem to have
> > continued in any systematic way.
> 
> is there more information about it somewhere? My inbox has only two emails 
> from
> xnox and pabs (in CC) about it. How did it work?

I found a reference to it on the wiki:

https://wiki.debian.org/qa.debian.org/ArchiveTesting#TODO

Some discussions about it that I found:

Subject: rebuilding the archive in a dirty chroot: results
<20080125142515.ga18...@xanadu.blop.info>

Subject: Meaning of the "Altering package upload rules"
<20080216130228.ga32...@pcpool00.mathematik.uni-freiburg.de>
<20080216181247.GA13475@garfield>

Subject: Use of the Build-Conflicts field


Subject: Re: Please drop anacron from task-desktop


-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Daniel Schepler]

On Fri, Jan 17, 2020 at 2:05 AM Johannes Schauer  wrote:
> yes, probably a communication problem. I think you are talking about [1] from
> August 11 last year? I replied the same day, telling you to please file a bug
> with your patches to continue discussion there. But then there was no response
> from you anymore and I don't find your bug in the BTS. Maybe my mail got lost
> somehow or I fail finding the bug you filed?

It seems the reply must have gotten lost somehow - I don't see it anywhere.

> I'm also very excited about having yet another chroot backend being integrated
> into sbuild! Though my first comment would be the same as I gave those asking
> for a docker backend in #867176: maybe try adding that backend to autopkgtest
> first. Then more people would profit from having that type of backend 
> available
> and no additional changes would be needed in sbuild because sbuild can already
> use autopkgtest backends for package building.

OK, I'll try to get the patch submitted either tonight or tomorrow.
(What I need to clean up is that it's interspersed with changes I made
to have it run with a personal distribution build I've been tinkering
with.  On a quick review, I now notice it's also interspersed with
changes to support using eatmydata only on the apt install commands
and only if not in schroot-update mode, instead of having to put it
into schroot config to apply to all commands - which seems reasonable
to split out into a separate patch to submit.  I also haven't yet
updated documentation files.)

One quick question: I don't see any mention of
$options->{'DISABLE_NETWORK'} in lib/Sbuild/ChrootAutopkgtest.pm,
whereas my new lib/Sbuild/ChrootNspawn.pm does support it.  If I'm not
missing something, then I wonder what it would take to add
DISABLE_NETWORK support to the autopkgtest sbuild engine.
-- 
Daniel Schepler

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Johannes Schauer]

Hi,

Quoting Daniel Schepler (2020-01-17 17:58:09)
> > I'm also very excited about having yet another chroot backend being
> > integrated into sbuild! Though my first comment would be the same as I gave
> > those asking for a docker backend in #867176: maybe try adding that backend
> > to autopkgtest first. Then more people would profit from having that type
> > of backend available and no additional changes would be needed in sbuild
> > because sbuild can already use autopkgtest backends for package building.
> OK, I'll try to get the patch submitted either tonight or tomorrow.

thank you!! :)

> One quick question: I don't see any mention of $options->{'DISABLE_NETWORK'}
> in lib/Sbuild/ChrootAutopkgtest.pm, whereas my new lib/Sbuild/ChrootNspawn.pm
> does support it.  If I'm not missing something, then I wonder what it would
> take to add DISABLE_NETWORK support to the autopkgtest sbuild engine.

it would require some way to tell the autopkgtest backends to disable network.
To my knowledge that is not possible. So as of today, the only backend where
you can disable network for the build (not the apt install) is the "unshare"
backend.

More info here:
https://wiki.debian.org/sbuild#Disabling_network_access_for_dpkg-buildpackage

Thanks!

cheers, josch


signature.asc
Description: signature

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Helmut Grohne]

Hi Daniel,

On Thu, Jan 16, 2020 at 08:50:25AM -0800, Daniel Schepler wrote:
> However, I've been getting push back on some of these, with
> maintainers of the opinion that it isn't actually a bug.  So, I
> thought I'd consult here to get more opinions on whether these are
> true bugs, or whether I'm at fault for trying to run dpkg-buildpackage
> manually instead of using it through pbuilder or sbuild.

Given that I'm also bootstrapping Debian (in a different setting), I'm
running into much the same problems. However, when I file bugs about
build failures in non-standard environments, I don't receive the
push-back that you describe. I cannot reproduce your experience. It's a
rare thing for maintainers to tell me that I'm filing a non-bug and I do
file a *lot* of bugs. Could it be that your sample is very small or
biased? Maybe you can give examples?

In any case, the consensus seems to be that FTBFS in a non-standard
environment is a bug, but not an RC one. And in general, you get to
discuss less with maintainers if your bug includes a patch. ;)

Helmut

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Helmut Grohne]

Hi Daniel,

On Thu, Jan 16, 2020 at 12:04:12PM -0800, Daniel Schepler wrote:
> Also, by the way, the amd64 -> i386 cross built core packages largely
> worked OK, with the exception of gcc-9, which ended up with incorrect
> include-fixed/limits.h, and with a compiler that required -lssp when
> building with -fstack-protector-strong or -fstack-protector as almost
> all Debian packages do.  To anybody on the list: is there something I
> did wrong with the cross build there, which was to run
> "dpkg-buildpackage -a i386 -B -Pcross"?)

This sounds a lot like you're running into
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80677.

Cross building gcc tends to not work in my experience, see:
http://crossqa.debian.net/src/gcc-8
http://crossqa.debian.net/src/gcc-9

For gcc-9, the --host flag is not properly passed down to the gm2
component. This is reported as
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92336.

I think the expectation that it works is misguided.

Adding DEB_BUILD_OPTIONS=nocheck might increase your chances of a
successful build.

Quite certainly, more work is needed here. Unfortunately, we sorely lack
porters and everyone seems to expect that things just work without
anyone doing the work.

Helmut

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Guillem Jover]

On Fri, 2020-01-17 at 11:12:50 +0100, Ansgar wrote:
> Johannes Schauer writes:
> > My advice would also be to switch from debootstrap to mmdebstrap because the
> > latter is able to create a chroot with only Essential:yes packages in it 
> > while
> > debootstrap includes Priority:required packages as well. Alternatively,
> > debootstrap could gain a variant only installing Essential:yes packages and
> > their dependencies (why doesn't it have that already?).
> 
> Debian doesn't support systems that don't have "required" packages
> installed. So buildds should have them installed.

If these statements are based on the policy quote below, then I do
not agree. I don't see why e2fsprogs would need to be installed on
a chroot, as long as there's nothing depending on it, for example.

> Policy states:
> "Removing a required package may cause your system to become totally
> broken and you may not even be able to use dpkg to put things back, so
> only do so if you know what you are doing."

That seems wrong, or inaccurate at best. dpkg should never depend on
anything that is not part of the pseudo-essential set (strictly
speaking only Essential:yes + awk-virtual), or that it depends on
explicitly. So as long as a package has not been forced out, dpkg
must work.

Removing a required package (that is not Essential:yes) might indeed
render your system broken, but what broken means or in what context is
not qualified there. This could apply to systems that get booted for
example, but not to chroots. A package that relies on another package
that is Priority:required and not Essential:yes, and that it does not
depend on, is just broken.

Here's the current list of these packages on my system:

  $ aptitude -F '%p' search '~prequired !~E'
  debconf
  e2fsprogs
  gcc-10-base
  gcc-9-base
  libpam-modules
  libpam-modules-bin
  libpam-runtime
  mawk
  mount
  passwd
  tzdata

And most of these are actually part of the pseudo-essential set
anyway, and cannot be removed w/o force.

That passage in policy might have made sense some time ago, when
Priority:required almost matched the pseudo-essential set, and when we
didn't have a separation of "dpkg-essential" and "boot-essential".

Regards,
Guillem

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Simon Richter]

Hi,

On Thu, Jan 16, 2020 at 08:50:25AM -0800, Daniel Schepler wrote:

> I've been running a manual test bootstrap of Debian (starting with
> cross-compiled packages amd64 -> i386 up to the point I was able to
> install debhelper), and posting a few bugs I've found along the way.
> These are where I found that having extra packages installed during
> the dpkg-buildpackage run either failed or resulted in broken
> packages.  (Some examples of the type of thing I mean: #948522,
> #887902.)

I would expect a sensible package build to not depend on package
relationships that aren't declared, in either way, but things in /usr/local
are outside the control of the package system.

So a package that enables an optional feature if a particular other package
is installed, but there is neither a build dependency nor a build conflict
declared is a bug to me.

The Build-Conflicts should still be avoided, by an explicit configure
option "disable this feature".

   Simon

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Ansgar]

Johannes Schauer writes:
> My advice would also be to switch from debootstrap to mmdebstrap because the
> latter is able to create a chroot with only Essential:yes packages in it while
> debootstrap includes Priority:required packages as well. Alternatively,
> debootstrap could gain a variant only installing Essential:yes packages and
> their dependencies (why doesn't it have that already?).

Debian doesn't support systems that don't have "required" packages
installed.  So buildds should have them installed.  Policy states:
"Removing a required package may cause your system to become totally
broken and you may not even be able to use dpkg to put things back, so
only do so if you know what you are doing."

"Essential" packages just have additional requirements (in particular
essential packages must work even in the "unpacked" state).

Ansgar

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Johannes Schauer]

Hi Daniel, Sam and all,

Quoting Sam Hartman (2020-01-17 01:27:52)
> > "Daniel" == Daniel Schepler  writes:
> 
> Daniel> (Incidentally, another offshoot was creating local patches to 
> sbuild
> Daniel> which add an operation mode using systemd-nspawn --ephemeral to 
> start
> Daniel> a container (along with the base being a BTRFS subvolume to speed 
> up
> Daniel> the cloning), systemd-run -M debian-sid-amd64-xxx
> Daniel> [--property=PrivateNetwork=yes] cmd..., etc.  When I sent a 
> message to
> Daniel> sbu...@packages.debian.org there didn't seem to be any interest in
> Daniel> having me clean up the patches and send them on.
> 
> This sounds like a communication problem or something.  Having
> systemd-nspawn container support in sbuild would be really helpful.
> Would you be willing to try reaching out to the sbuild maintainers
> again?  If you don't get an answer, in a month or so, please reach out
> to me as DPL.  My job would not to be to convince the maintainer to
> accept your patches, but rather to facilitate communication so you can
> get a clear answer.  The DPL does that from time to time when things seem to
> have broken down.

yes, probably a communication problem. I think you are talking about [1] from
August 11 last year? I replied the same day, telling you to please file a bug
with your patches to continue discussion there. But then there was no response
from you anymore and I don't find your bug in the BTS. Maybe my mail got lost
somehow or I fail finding the bug you filed?

I'm also very excited about having yet another chroot backend being integrated
into sbuild! Though my first comment would be the same as I gave those asking
for a docker backend in #867176: maybe try adding that backend to autopkgtest
first. Then more people would profit from having that type of backend available
and no additional changes would be needed in sbuild because sbuild can already
use autopkgtest backends for package building.

Let me also use this opportunity to ask for help with maintaining sbuild. If
any of you reading this ever felt that sbuild was the thing that is worth your
time, please feel free to reach out to me. We need to make a new release fixing
some easy but important bugs that accumulated in the BTS. I would be very happy
to review and apply people's patches. :)

Thanks!

cheers, josch

[1] cadf0c45pjydq+hmqetg6atdqp8ojw8cr3z1kz2ktu9z3ua3...@mail.gmail.com


signature.asc
Description: signature

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Johannes Schauer]

Hi,

Quoting Simon McVittie (2020-01-16 19:47:02)
> I think I dimly remember someone setting up "the buildd from hell" which
> deliberately did this as a QA mechanism, but it doesn't seem to have
> continued in any systematic way.

is there more information about it somewhere? My inbox has only two emails from
xnox and pabs (in CC) about it. How did it work?

> I think in an ideal world, we'd have better tools for those users to build
> modified packages in a way that more closely resembles what happens on the
> production Debian infrastructure - which might for instance mean CI services,
> or my vectis[1] tool, or sbuild-debian-developer-setup, or even autopkgtest
> (which is really for testing packages, but as a side-effect, it knows how to
> build packages in an environment that in practice is going to be
> close-to-minimal).

My advice would also be to switch from debootstrap to mmdebstrap because the
latter is able to create a chroot with only Essential:yes packages in it while
debootstrap includes Priority:required packages as well. Alternatively,
debootstrap could gain a variant only installing Essential:yes packages and
their dependencies (why doesn't it have that already?).

> pbuilder and the usual sbuild+schroot setup have the disadvantage of
> requiring root privileges and crossing privilege boundaries, but vectis uses
> virtual machines (in practice this means kvm group membership or udev/logind
> uaccess, to get write access to /dev/kvm) and as namespace/container stuff
> gets more powerful and more trusted, I'd hope that we'll get a better ability
> to install build-dependencies and do builds in unprivileged containers.

That can be done with sbuild. With --chroot-mode=unshare, sbuild looks for
rootfs tarballs in ~/.cache/sbuild or you can give it your own tarball with the
--chroot option. Since you can create rootfs tarballs without sudo using
mmdebstrap, you can setup arbitrary chroots and build packages without any
process running with root privileges.

If you don't like the security implications of unshared user namespaces, then
you might want to try --chroot-mode=autopkgtest and
--autopkgtest-virt-server=qemu. Using --autopkgtest-virt-server-opts you can
then supply any qemu compatible system image. Using mmdebstrap with fakechroot
mode and guestfish you can create these system images without becoming root and
without having to enable kernel.unprivileged_userns_clone.

Or if lxc/lxd are the unprivileged containers you are talking about, then you
can just use --autopkgtest-virt-server=lxc and let sbuild do builds in your
existing lxc container.

Thanks!

cheers, josch


signature.asc
Description: signature

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Paul Wise]

On Thu, Jan 16, 2020 at 7:18 PM Ondřej Surý wrote:

> while your effort is valiant, I see a little value in it as there’s no real 
> world use case. While your arguments are valid, you are imposing additional 
> work on generally already overloaded maintainers with unclear goal and 
> purpose.

Personally, I think there is value in Daniel's work on bootstrapping
Debian from other operating systems and Helmut's work on bootstrapping
Debian from existing Debian architectures. Both are important projects
and we need Debian and FLOSS in general to be more bootstrappable than
it is now, just like the Reproducible Builds folks made everything
more reproducible. At some point I'd even like to see Debian
bootstrapped from whatever comes out of the the Bootstrappable Builds
project:

https://bootstrappable.org/

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Sam Hartman]

> "Daniel" == Daniel Schepler  writes:

Daniel> (Incidentally, another offshoot was creating local patches to sbuild
Daniel> which add an operation mode using systemd-nspawn --ephemeral to 
start
Daniel> a container (along with the base being a BTRFS subvolume to speed up
Daniel> the cloning), systemd-run -M debian-sid-amd64-xxx
Daniel> [--property=PrivateNetwork=yes] cmd..., etc.  When I sent a message 
to
Daniel> sbu...@packages.debian.org there didn't seem to be any interest in
Daniel> having me clean up the patches and send them on.

This sounds like a communication problem or something.  Having
systemd-nspawn container support in sbuild would be really helpful.
Would you be willing to try reaching out to the sbuild maintainers
again?  If you don't get an answer, in a month or so, please reach out
to me as DPL.  My job would not to be to convince the maintainer to
accept your patches, but rather to facilitate communication so you can
get a clear answer.  The DPL does that from time to time when things
seem to have broken down.

--Sam

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Daniel Schepler]

On Thu, Jan 16, 2020 at 11:18 AM Ondřej Surý  wrote:
> while your effort is valiant, I see a little value in it as there’s no real 
> world use case. While your arguments are valid, you are imposing additional 
> work on generally already overloaded maintainers with unclear goal and 
> purpose.
>
> Perhaps your energy and enthusiasm (which I appreciate) could be spent on 
> helping fixing reproducible builds in packages or cross-building. Those are 
> practical and you won’t find any resistance in accepting patches for these 
> two use cases.

OK, makes sense.  This was, in fact, an offshoot of the beginning
stages of a project to create some way to automate bootstrapping an
architecture starting with automated cross builds of the core
packages.

(Incidentally, another offshoot was creating local patches to sbuild
which add an operation mode using systemd-nspawn --ephemeral to start
a container (along with the base being a BTRFS subvolume to speed up
the cloning), systemd-run -M debian-sid-amd64-xxx
[--property=PrivateNetwork=yes] cmd..., etc.  When I sent a message to
sbu...@packages.debian.org there didn't seem to be any interest in
having me clean up the patches and send them on.  Still, do you think
if I posted bug reports for issues I found due to the builds running
under seccomp filters, as wishlist bugs and with either suggested
patches or a request for advice on further debugging it myself where I
got stuck, that maintainers might be willing to consider them?

Also, by the way, the amd64 -> i386 cross built core packages largely
worked OK, with the exception of gcc-9, which ended up with incorrect
include-fixed/limits.h, and with a compiler that required -lssp when
building with -fstack-protector-strong or -fstack-protector as almost
all Debian packages do.  To anybody on the list: is there something I
did wrong with the cross build there, which was to run
"dpkg-buildpackage -a i386 -B -Pcross"?)
-- 
Daniel Schepler

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Ondřej Surý]

Daniel,

while your effort is valiant, I see a little value in it as there’s no real 
world use case. While your arguments are valid, you are imposing additional 
work on generally already overloaded maintainers with unclear goal and purpose.

Perhaps your energy and enthusiasm (which I appreciate) could be spent on 
helping fixing reproducible builds in packages or cross-building. Those are 
practical and you won’t find any resistance in accepting patches for these two 
use cases.

Ondrej
--
Ondřej Surý 

> On 16 Jan 2020, at 18:21, Daniel Schepler  wrote:
> 
> I've been running a manual test bootstrap of Debian (starting with
> cross-compiled packages amd64 -> i386 up to the point I was able to
> install debhelper), and posting a few bugs I've found along the way.
> These are where I found that having extra packages installed during
> the dpkg-buildpackage run either failed or resulted in broken
> packages.  (Some examples of the type of thing I mean: #948522,
> #887902.)
> 
> However, I've been getting push back on some of these, with
> maintainers of the opinion that it isn't actually a bug.  So, I
> thought I'd consult here to get more opinions on whether these are
> true bugs, or whether I'm at fault for trying to run dpkg-buildpackage
> manually instead of using it through pbuilder or sbuild.
> 
> My arguments in favor of such things being bugs:
> 
> 1. I've been using Debian since before pbuilder or sbuild even
> existed, and I don't remember ever seeing any announcements along the
> lines of "using pbuilder or sbuild is now mandatory, running
> dpkg-buildpackage manually is forbidden".  (Just announcements that of
> course, testing package builds using one of those before uploading is
> strongly encouraged.)
> 2. The mere existence of the Build-Conflicts field.
> 3. The general principle that the Build-Depends are meant not to
> describe every possible way the package *could* be built, but to pin
> down the exact environment in which the package *should* be built, in
> order to avoid unnecessary differences in the resulting packages
> between architectures.
> 4. The build-essential package set could evolve over time, and in a
> few cases that could come back to bite maintainers.  For a
> hypothetical example: what if Debian eventually decided to add cmake,
> ninja, and meson to the build-essential set?  Or, what if there were a
> source package that made a build time check of whether a libpopt.so.2
> file exists to be dlopen()ed, but if it's found enabled broken code;
> and then, eventually, one of the build-essential packages added a
> dependency on libpopt?
> 
> Possible arguments I can anticipate against such things being considered bugs:
> 
> 1. It would be very difficult to impossible to test for every possible
> combination of packages that satisfy the Build-Depends.  (Though I
> would think a vast majority of such bugs would be detected by a
> reproducible-builds type setup with one build being the standard
> minimal chroot, and the other build using a chroot with as many
> packages as possible installed.)
> 2. It would be pointless to worry about such things, especially now
> that all uploads to the archives must be source only.  (To which my
> answer would be: requiring use of pbuilder or sbuild would place a
> burden on users who previously would have made local patches by a
> sequence of "apt-get source package; cd package-*/; edit;
> dpkg-buildpackage -b -uc; sudo apt-get install ../*.deb )
> 
> (Somewhat related to this: I've also found a few packages that hang
> when building from the command line, waiting for input from stdin;
> whereas in pbuilder or sbuild, with input redirected to /dev/null or
> similar, the builds succeed.  Would these be considered bugs as well?
> Of course, in some situations it looks like it detects an
> incontrovertible bug, such as when an "rm" command hangs on the prompt
> for confirmation on a read-only file, and the /dev/null stdin case
> would just result in those files being left in place.  I've especially
> been seeing the latter sort of thing related to Perl packages now that
> recent Perls install lots of files as read only.)
> -- 
> Daniel Schepler
> 

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Simon McVittie]

On Thu, 16 Jan 2020 at 08:50:25 -0800, Daniel Schepler wrote:
> I've been running a manual test bootstrap of Debian (starting with
> cross-compiled packages amd64 -> i386 up to the point I was able to
> install debhelper), and posting a few bugs I've found along the way.
> These are where I found that having extra packages installed during
> the dpkg-buildpackage run either failed or resulted in broken
> packages.  (Some examples of the type of thing I mean: #948522,
> #887902.)
> 
> However, I've been getting push back on some of these, with
> maintainers of the opinion that it isn't actually a bug.  So, I
> thought I'd consult here to get more opinions on whether these are
> true bugs, or whether I'm at fault for trying to run dpkg-buildpackage
> manually instead of using it through pbuilder or sbuild.

I would say it's often a bug, but usually (always?) a non-RC bug, because
there's a straightforward workaround (build in a minimal environment).

Opening bugs as RC tends to put maintainers on the defensive (if nothing
else because RC bugs have automatic consequences, like autoremovals), so
it's often better to err on the side of opening bugs with a lower severity
(and escalating if there's later consensus that this should happen)
rather than starting with a severity that could be seen as unjustified.

FTBFS and broken packages when built in a non-minimal environment are
special cases of the more general problem of getting a different package
when built in a non-minimal environment, which is sometimes trivial
to solve, but somewhat intractable to detect and solve in general,
because there are so many different non-minimal environments - as Ansgar
mentioned, not just packages that we can capture in Build-Depends and
Build-Conflicts, but also the contents of directories like /usr/local,
/opt and ~/bin, and various properties of your system such as whether it
has undergone the /usr merge, whether alternatives have been reconfigured,
the filesystem you're running on, whether an LSM is in use, ...

I think it's best-practice for package maintainers to explicitly disable
optional features that are autodetected but aren't wanted in the package
(./configure --disable-weasels --without-libstoat, or equivalent), but
this isn't always possible without relatively extensive patches to the
upstream build system, which themselves carry a cost.

> The mere existence of the Build-Conflicts field [implies that we are
> meant to use it]

Yes to a point, but here's a somewhat devil's-advocate response to that:

Suppose we lived in a parallel timeline where there is a serious QA effort
put into making sure packages produce the same results, regardless of
how they are built, as long as their Build-Depends and Build-Conflicts
(and their -Arch, -Indep flavours) are satisfied.

In that world, one rational thing for maintainers of packages with
optional features to do would be to add an ever-increasing number of
Build-Conflicts (every time someone notices a package that can alter or
breaks the build). I could easily imagine it becoming no longer feasible
to build large sets of packages unless you use sbuild or equivalent,
because at least a few of those packages will have a Build-Conflicts on
something you needed for another of the packages - which might not even
be necessary any more, but as an individual package maintainer, there's
little incentive to *remove* Build-Conflicts on things that you yourself
don't use, particularly if insufficient Build-Conflicts are considered RC.

So, pragmatically, it might be a good idea to optimize away the steps
where we put lots of effort into QA, report lots of bugs, get them
fixed by addition of Build-Conflicts, and find ourselves having to use
a minimal build environment to get your desired set of packages built -
by using a minimal build environment to start with. And we're back to
the advice to use pbuilder or sbuild.

I don't think it's coincidence that there has been so much parallel
evolution between Debian's pbuilder and sbuild (among others), Fedora's
mock, openSUSE's obs-build, Flatpak apps' flatpak-builder, and lots of
projects' Docker-based build systems - getting a reliable and reproducible
build of a complex package/module in minimal environments is already quite
difficult, which makes the harder task of making it reliably reproducible
in arbitrary non-minimal environments look rather unappealing.

> It would be very difficult to impossible to test for every possible
> combination of packages that satisfy the Build-Depends.  (Though I
> would think a vast majority of such bugs would be detected by a
> reproducible-builds type setup with one build being the standard
> minimal chroot, and the other build using a chroot with as many
> packages as possible installed.)

I think I dimly remember someone setting up "the buildd from hell"
which deliberately did this as a QA mechanism, but it doesn't seem to

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Sam Hartman]

>>>>> "Daniel" == Daniel Schepler  writes:

Daniel> However, I've been getting push back on some of these, with
Daniel> maintainers of the opinion that it isn't actually a bug.  So, I
Daniel> thought I'd consult here to get more opinions on whether these are
Daniel> true bugs, or whether I'm at fault for trying to run 
dpkg-buildpackage
Daniel> manually instead of using it through pbuilder or sbuild.

I think it is often a bug, but rarely if ever a serious bug.

I'd say that a good test is whether you can produce a clean, simple
patch.
(adding build-conflicts seems like a fine answer in a lot of places).
If something like build-conflicts is sufficient, then sure, file the bug
with a patch if it  actually annoys you.

If the patch is going to be ugly, then perhaps it's not worth it.

Re: Is running dpkg-buildpackage manually from the command line forbidden?

[Ansgar]

On Thu, 2020-01-16 at 08:50 -0800, Daniel Schepler wrote:
> These are where I found that having extra packages installed during
> the dpkg-buildpackage run either failed or resulted in broken
> packages.  (Some examples of the type of thing I mean: #948522,
> #887902.)

If you build outside a controlled environment the set of additional
packages are just a small part of your worries.  There are others such
as

- more creative solutions to satisfy Build-Depends,
- anything in /usr/local,
- anything in /opt (if PATH or other configuration includes /opt),
- which alternative is active,
- diversions,
- local configuration, including environment variables (PATH,
  variables changing behavior of the Python interpreter, ...),
- modified files,
- local packages that do not exist in Debian's archive, or
- modified packages.

In many cases this will not be a problem, but trying to make any
guarantees that the package building process will still work reliably
seems to be a lost cause to me.

It is fine to ask a maintainer to include some `--disable-something`
flag if this is easy to do and reasonably likely to be a problem, but I
would not expect people to actively spend effort on this.

Ansgar

Is running dpkg-buildpackage manually from the command line forbidden?

[Daniel Schepler]

I've been running a manual test bootstrap of Debian (starting with
cross-compiled packages amd64 -> i386 up to the point I was able to
install debhelper), and posting a few bugs I've found along the way.
These are where I found that having extra packages installed during
the dpkg-buildpackage run either failed or resulted in broken
packages.  (Some examples of the type of thing I mean: #948522,
#887902.)

However, I've been getting push back on some of these, with
maintainers of the opinion that it isn't actually a bug.  So, I
thought I'd consult here to get more opinions on whether these are
true bugs, or whether I'm at fault for trying to run dpkg-buildpackage
manually instead of using it through pbuilder or sbuild.

My arguments in favor of such things being bugs:

1. I've been using Debian since before pbuilder or sbuild even
existed, and I don't remember ever seeing any announcements along the
lines of "using pbuilder or sbuild is now mandatory, running
dpkg-buildpackage manually is forbidden".  (Just announcements that of
course, testing package builds using one of those before uploading is
strongly encouraged.)
2. The mere existence of the Build-Conflicts field.
3. The general principle that the Build-Depends are meant not to
describe every possible way the package *could* be built, but to pin
down the exact environment in which the package *should* be built, in
order to avoid unnecessary differences in the resulting packages
between architectures.
4. The build-essential package set could evolve over time, and in a
few cases that could come back to bite maintainers.  For a
hypothetical example: what if Debian eventually decided to add cmake,
ninja, and meson to the build-essential set?  Or, what if there were a
source package that made a build time check of whether a libpopt.so.2
file exists to be dlopen()ed, but if it's found enabled broken code;
and then, eventually, one of the build-essential packages added a
dependency on libpopt?

Possible arguments I can anticipate against such things being considered bugs:

1. It would be very difficult to impossible to test for every possible
combination of packages that satisfy the Build-Depends.  (Though I
would think a vast majority of such bugs would be detected by a
reproducible-builds type setup with one build being the standard
minimal chroot, and the other build using a chroot with as many
packages as possible installed.)
2. It would be pointless to worry about such things, especially now
that all uploads to the archives must be source only.  (To which my
answer would be: requiring use of pbuilder or sbuild would place a
burden on users who previously would have made local patches by a
sequence of "apt-get source package; cd package-*/; edit;
dpkg-buildpackage -b -uc; sudo apt-get install ../*.deb )

(Somewhat related to this: I've also found a few packages that hang
when building from the command line, waiting for input from stdin;
whereas in pbuilder or sbuild, with input redirected to /dev/null or
similar, the builds succeed.  Would these be considered bugs as well?
Of course, in some situations it looks like it detects an
incontrovertible bug, such as when an "rm" command hangs on the prompt
for confirmation on a read-only file, and the /dev/null stdin case
would just result in those files being left in place.  I've especially
been seeing the latter sort of thing related to Perl packages now that
recent Perls install lots of files as read only.)
-- 
Daniel Schepler

Re: make dpkg-buildpackage default locale UTF-8

[Andreas Tille]

On Mon, Sep 04, 2017 at 01:45:25PM +0800, 殷啟聰 | Kai-Chung Yan wrote:
> +1 to setting UTF-8 as default.
> 
> Some Java packages that I worked with contain source files with symbols not 
> recognized by compilers unless the encoding is set to UTF-8. Mostly these 
> symbols are a copyright sign "©" apprearing in the license section, 
> occasionally CJK letters in the author names. This change to dpkg should be 
> able to solve the problem conveniently.

I can confirm that due to the same experience the d/rules template
I'm using for Java files contains

  export LC_ALL=C.UTF-8

I'd love to see this go - so +1 for UTF-8 as default.

Kind regards

 Andreas.

-- 
http://fam-tille.de

Re: make dpkg-buildpackage default locale UTF-8

[殷啟聰 | Kai-Chung Yan]

+1 to setting UTF-8 as default.

Some Java packages that I worked with contain source files with symbols not 
recognized by compilers unless the encoding is set to UTF-8. Mostly these 
symbols are a copyright sign "©" apprearing in the license section, 
occasionally CJK letters in the author names. This change to dpkg should be 
able to solve the problem conveniently.

Hans-Christoph Steiner 於 2017/9/1 下午4:23 寫道:
> Package: dpkg-dev
>
> More and more packages are adding unicode files as unicode support has
> become more reliable and available.  The package building process is not
> guaranteed to happen in a unicode locale since the Debian default locale
> is LC_ALL=C, which is ASCII not UTF-8.  Reading UTF-8 filenames when the
> system is using ASCII causes errors (Python makes them very visible, for
> example).
>
> mbiebl, youpi, wRAR, bunk, and I had a discussion in #debian-devel.  It
> looks like setting the default locale to C.UTF-8 in dpkg-buildpackage is
> an easy way to improve this situation a lot.  Any package that needs an
> encoding besides UTF-8 could always set it by adding something like this
> to debian/rules:
>
>   export LC_ALL = C
>
> Setting C.UTF-8 as the global default in Debian would be the best
> solution to this and many other issues, but that's a much much larger
> project:
> https://sourceware.org/glibc/wiki/Proposals/C.UTF-8
>




signature.asc
Description: OpenPGP digital signature

Re: make dpkg-buildpackage default locale UTF-8

[Tollef Fog Heen]

]] Ivan Shmakov 

> > Tollef Fog Heen  writes:
> > Ivan Shmakov
> > Hans-Christoph Steiner  writes:
> 
>  >>> Package: dpkg-dev
> 
>  >>> More and more packages are adding unicode files
> 
>  >> I assume you mean “UTF-8 filenames” here (per below), right?
> 
>  >>> as unicode support has become more reliable and available.
> 
>  >> What are the use cases for such filenames?
> 
>  > Accurate representation of what they contain.  wnorwegian contains a
>  > file «bokmål» which is the word list for the form of Norwegian known
>  > as bokmål.  There's a convenience link between it and bokmaal so that
>  > people without Norwegian keyboard (or without compose keys) can type
>  > it too, but the canonical name is bokmål, not bokmaal.
> 
>   It does indeed seem natural, when it comes to packages providing
>   support for a specific language, to use filenames in that same
>   language; I’m not going to strongly object to that.  However,
>   I’d like to note that other wordlist packages appear to stick to
>   English (and ASCII) filenames.  For instance, wfrench uses
>   ‘french’ (instead of français) and witalian uses ‘italian’
>   (instead of italiano.)

Since Norwegian has two forms of the language, we ship two wordlists.  I
could have gone with «/usr/share/dict/norwegian - New Norwegian» and
«/usr/share/dict/norwegian - book tongue», but I suspect that'd be
considered less helpful than just using bokmål and nynorsk.

>   I’m still curious, are there any other uses in the archive
>   beside the above?

There are at least some files in the ca-certificate package that have
non-ASCII names, presumably since the CAs have non-ASCII names.

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

Re: make dpkg-buildpackage default locale UTF-8

[Ivan Shmakov]

> Tollef Fog Heen  writes:
> Ivan Shmakov
> Hans-Christoph Steiner  writes:

 >>> Package: dpkg-dev

 >>> More and more packages are adding unicode files

 >> I assume you mean “UTF-8 filenames” here (per below), right?

 >>> as unicode support has become more reliable and available.

 >> What are the use cases for such filenames?

 > Accurate representation of what they contain.  wnorwegian contains a
 > file «bokmål» which is the word list for the form of Norwegian known
 > as bokmål.  There's a convenience link between it and bokmaal so that
 > people without Norwegian keyboard (or without compose keys) can type
 > it too, but the canonical name is bokmål, not bokmaal.

It does indeed seem natural, when it comes to packages providing
support for a specific language, to use filenames in that same
language; I’m not going to strongly object to that.  However,
I’d like to note that other wordlist packages appear to stick to
English (and ASCII) filenames.  For instance, wfrench uses
‘french’ (instead of français) and witalian uses ‘italian’
(instead of italiano.)

At the same time, were these files intended for mainly ‘machine’
use, I guess I’d rather prefer them to stick to ISO 639 instead.
Just like the ‘locale’ system already does.

I’m still curious, are there any other uses in the archive
beside the above?

 > (I see there's a small bug where the symlink is the wrong way around,
 > I'll get that fixed.)

 > å is in latin1, though so fonts should not be a problem in your
 > particular case.

Yes.

-- 
FSF associate member #7257  np. Graceful Flame — Radiarc   3013 B6A0 230E 334A

Re: make dpkg-buildpackage default locale UTF-8

[Tollef Fog Heen]

]] Ivan Shmakov 

> > Hans-Christoph Steiner  writes:
> 
>  > Package: dpkg-dev
> 
>  > More and more packages are adding unicode files
> 
>   I assume you mean “UTF-8 filenames” here (per below), right?
> 
>  > as unicode support has become more reliable and available.
> 
>   What are the use cases for such filenames?

Accurate representation of what they contain.  wnorwegian contains a
file «bokmål» which is the word list for the form of Norwegian known as
bokmål.  There's a convenience link between it and bokmaal so that
people without Norwegian keyboard (or without compose keys) can type it
too, but the canonical name is bokmål, not bokmaal.

(I see there's a small bug where the symlink is the wrong way around,
I'll get that fixed.)

å is in latin1, though so fonts should not be a problem in your
particular case.

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

Re: make dpkg-buildpackage default locale UTF-8

[Ivan Shmakov]

> Hans-Christoph Steiner  writes:

 > Package: dpkg-dev

 > More and more packages are adding unicode files

I assume you mean “UTF-8 filenames” here (per below), right?

 > as unicode support has become more reliable and available.

What are the use cases for such filenames?

FWIW, I more than just occasionally use Debian in environments
with fonts lacking good (as in: ≥ 90%) Unicode, or even BMP,
coverage.  (Specifically, I’m for the most part interested in
Latin-1, -3, and Cyrillic characters only.)

Do you suggest that there’re filenames in Debian packages that
cannot be rendered in such environments?  If so, that’d
certainly be a nuisance for me.

 > The package building process is not guaranteed to happen in a unicode
 > locale since the Debian default locale is LC_ALL=C, which is ASCII
 > not UTF-8.  Reading UTF-8 filenames when the system is using ASCII
 > causes errors (Python makes them very visible, for example).

[…]

-- 
FSF associate member #7257  np. Fear of the Dark — Iron Maiden  B6A0 230E 334A

Re: make dpkg-buildpackage default locale UTF-8

[Ian Jackson]

Hans-Christoph Steiner writes ("make dpkg-buildpackage default locale UTF-8"):
> More and more packages are adding unicode files as unicode support has
> become more reliable and available.  The package building process is not
> guaranteed to happen in a unicode locale since the Debian default locale
> is LC_ALL=C, which is ASCII not UTF-8.  Reading UTF-8 filenames when the
> system is using ASCII causes errors (Python makes them very visible, for
> example).

I think this is a bug in Python.  The default "C" locale should be
just-send-8 for filenames and data.

Ian.

Re: Bug#873919: make dpkg-buildpackage default locale UTF-8

[Guillem Jover]

Control: forcemerge -1 843776

Hi!

On Fri, 2017-09-01 at 10:23:59 +0200, Hans-Christoph Steiner wrote:
> Package: dpkg-dev
> 
> More and more packages are adding unicode files as unicode support has
> become more reliable and available.  The package building process is not
> guaranteed to happen in a unicode locale since the Debian default locale
> is LC_ALL=C, which is ASCII not UTF-8.  Reading UTF-8 filenames when the
> system is using ASCII causes errors (Python makes them very visible, for
> example).
> 
> mbiebl, youpi, wRAR, bunk, and I had a discussion in #debian-devel.  It
> looks like setting the default locale to C.UTF-8 in dpkg-buildpackage is
> an easy way to improve this situation a lot.  Any package that needs an
> encoding besides UTF-8 could always set it by adding something like this
> to debian/rules:
> 
>   export LC_ALL = C

As long as the project considers debian/rules the main entry point to a
package build, I'm not planning on predefining new general purpose
environment variables from dpkg-buildpackage that would otherwise not
be set by the builder.

The distinction here would be something like LC_*, against something
like CC on a cross-compilation, which you need to set no matter what.

But please, see the rationale on the other bug. I think I'll be adding
an entry to the dpkg FAQ, because it seems this has become a recurring
request. :)

> Setting C.UTF-8 as the global default in Debian would be the best
> solution to this and many other issues, but that's a much much larger
> project:
> https://sourceware.org/glibc/wiki/Proposals/C.UTF-8

That _might_ help on buildds, but not on maintainer systems for example.

Thanks,
Guillem

make dpkg-buildpackage default locale UTF-8

[Hans-Christoph Steiner]

Package: dpkg-dev

More and more packages are adding unicode files as unicode support has
become more reliable and available.  The package building process is not
guaranteed to happen in a unicode locale since the Debian default locale
is LC_ALL=C, which is ASCII not UTF-8.  Reading UTF-8 filenames when the
system is using ASCII causes errors (Python makes them very visible, for
example).

mbiebl, youpi, wRAR, bunk, and I had a discussion in #debian-devel.  It
looks like setting the default locale to C.UTF-8 in dpkg-buildpackage is
an easy way to improve this situation a lot.  Any package that needs an
encoding besides UTF-8 could always set it by adding something like this
to debian/rules:

  export LC_ALL = C

Setting C.UTF-8 as the global default in Debian would be the best
solution to this and many other issues, but that's a much much larger
project:
https://sourceware.org/glibc/wiki/Proposals/C.UTF-8

Re: Mass bug filing: dpkg-buildpackage -A

[Santiago Vila]

Hi.

I will put all the bugs that I report about this issue here:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=binary-indep;users=sanv...@debian.org

So far there are one that I reported 19 days ago as a "test report"
and 162 that I reported today. I'm intentionally excluding packages
having a different version in stretch and sid, since I'm only building
packages in stretch (too many FTBFS even without using -A, fortunately
Chris Lamb and my other reproducible builds friends usually report
those before they reach testing :-).

There are already quite a few fixed packages, which is great.

Will continue reporting in the next days/weeks/months, depending on my
free time.

Thanks a lot.

Re: Mass bug filing: dpkg-buildpackage -A

[Julien Cristau]

On Tue, Nov 24, 2015 at 00:41:35 +0100, Santiago Vila wrote:

> On Mon, Nov 23, 2015 at 08:42:14PM +0100, Julien Cristau wrote:
> > On Tue, Nov 24, 2015 at 06:14:43 +1100, Ben Finney wrote:
> > 
> > > If it's a “severe violation of Debian policy”, the bug is at least
> > > “serious” severity.
> > > 
> > The release team's RC policy decides which policy violations we consider
> > "severe" in the sense of "gets a serious severity bug".
> 
> Indeed, this is how I thought the severities were being used. Not in
> theory but in practice.
> 
> Please don't worry about severities. I will try to be "nice" and will
> report them as "important", as there are a lot of affected packages.
> 
> But will use some kind of usertags so that release managers are able
> to change severities easily. I would love to see this as a release
> goal for stretch, but I think it's soon to decide about that.
> 
Sounds great, thanks!

Cheers,
Julien


signature.asc
Description: PGP signature

Re: Mass bug filing: dpkg-buildpackage -A

[Ben Finney]

Santiago Vila  writes:

> On Mon, Nov 23, 2015 at 11:29:19AM +0100, Jakub Wilk wrote:
> > I'd use "severity: serious", just like for a normal FTBFS.
>
> The problem with making them "severity: serious" is that I would be
> deciding on my own that those bugs are RC. Normally, the release team
> decides about what bugs are RC and which ones are not.

The release team decides what bugs they consider to be release-critical.
The severity of the bug is one criterion that they can use.

The determination of what severity to assign the bug, though, is a
function of the effects of the bug behaviour. Its value should be set
truthfully, independent of whether or not the result of a severity level
would affect the release.

You make a determination of a bug's severity by the definitions of each
severity level [0]: if a bug's behaviour best meets the definition of
“serious” the correct value is “Severity: serious”.

If “in the package maintainer's or release manager's opinion, [the bug]
makes the package unsuitable for release”, the bug is at least “serious”
severity.

If it's a “severe violation of Debian policy”, the bug is at least
“serious” severity.

Either of those are sufficient for a reporter to truthfully set the bug
report as “Severity: serious” or higher. If you determine that either of
the above definitions describe the bug you're reporting, IMO you should
truthfully set that value for the bug report.

It's still up to the release team whether the bug is critical for the
release. That's not an attribute you get to decide, and IMO you are not
making the decision for them by setting the severity of a bug report.


[0] 

-- 
 \  “It is well to remember that the entire universe, with one |
  `\   trifling exception, is composed of others.” —John Andrew Holmes |
_o__)  |
Ben Finney

Re: Mass bug filing: dpkg-buildpackage -A

[Julien Cristau]

On Tue, Nov 24, 2015 at 06:14:43 +1100, Ben Finney wrote:

> If it's a “severe violation of Debian policy”, the bug is at least
> “serious” severity.
> 
The release team's RC policy decides which policy violations we consider
"severe" in the sense of "gets a serious severity bug".

Cheers,
Julien


signature.asc
Description: PGP signature

Mass bug filing: dpkg-buildpackage -A

[Santiago Vila]

Greetings.

I've noticed that some packages fail to build from source when built
using "dpkg-buildpackage -A".

This is not only a violation of policy, but also prevents the package
from being uploaded in source-only form, as the architecture-independent
packages are generated by a dedicated "Arch: all" autobuilder which
does exactly "dpkg-buildpackage -A" and nothing else.

[ The problem with source-only uploads is aggravated by this bug in
  ftp.debian.org:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=798413

  which apparently it's only a "normal" bug ].

As the number of bugs is going to be greater than ten, I hereby
announce my intent to report as many bugs of this type as I found.

I think those are clearly bugs, and they should be reported.
The only thing that may be unclear is the severity. I would like this
to be RC in some not too distant future (to be on par with ordinary
"dpkg-buildpackage"), but for now I think "severity: important" would
be appropriate here.

Thanks.

Re: Mass bug filing: dpkg-buildpackage -A

[Jakub Wilk]

* Santiago Vila <sanv...@unex.es>, 2015-11-23, 10:52:
I've noticed that some packages fail to build from source when built 
using "dpkg-buildpackage -A".

[...]
I think those are clearly bugs, and they should be reported. The only 
thing that may be unclear is the severity. I would like this to be RC 
in some not too distant future (to be on par with ordinary 
"dpkg-buildpackage"), but for now I think "severity: important" would 
be appropriate here.


I'd use "severity: serious", just like for a normal FTBFS.

--
Jakub Wilk

Re: Mass bug filing: dpkg-buildpackage -A

[Santiago Vila]

On Mon, Nov 23, 2015 at 11:29:19AM +0100, Jakub Wilk wrote:
> * Santiago Vila <sanv...@unex.es>, 2015-11-23, 10:52:
> >I've noticed that some packages fail to build from source when built using
> >"dpkg-buildpackage -A".
> [...]
> >I think those are clearly bugs, and they should be reported. The only
> >thing that may be unclear is the severity. I would like this to be RC in
> >some not too distant future (to be on par with ordinary
> >"dpkg-buildpackage"), but for now I think "severity: important" would be
> >appropriate here.
> 
> I'd use "severity: serious", just like for a normal FTBFS.

Ok, I was in doubt between "normal" and "important", so your email
tells me that I should probably forget about "normal".

The problem with making them "severity: serious" is that I would be
deciding on my own that those bugs are RC. Normally, the release team
decides about what bugs are RC and which ones are not.

Maybe later (but still during this release cycle) when we can be sure
that the number of packages is low enough, we can consider them to be
"severity: serious". The bugs will be already reported and it would be
just a matter of changing the severity.

Thanks.

Re: Mass bug filing: dpkg-buildpackage -A

[Adam D. Barratt]

On Tue, 2015-11-24 at 08:48 +1100, Ben Finney wrote:
> Julien Cristau  writes:
> 
> > On Tue, Nov 24, 2015 at 06:14:43 +1100, Ben Finney wrote:
> >
> > > If it's a “severe violation of Debian policy”, the bug is at least
> > > “serious” severity.
> 
> This is one way that a bug can be determined “Severity: serious”, by
> definition.
> 
> In other words: “severe violation of Debian policy” implies “bug is
> Severity: serious”.
> 
> > The release team's RC policy decides which policy violations we
> > consider "severe" in the sense of "gets a serious severity bug".
> 
> And this is another, by definition.

I think you may have missed Julien's point - there is no specific
definition of "severe violation of Debian policy" that is distinct from
"is listed in the Release Team's RC policy".

Neither policy nor the BTS defines exactly what is meant by "severe
violation". Policy indicates that violations are "roughly equivalent" to
particular bug severities and https://www.debian.org/Bugs/Developer.html
similarly uses "roughly".

The BTS used to have a more precise definition, most recently by
explicitly referring to the separate RC policy - see
http://anonscm.debian.org/viewvc/webwml/webwml/english/Bugs/Developer.wml?r1=1.46=1.47
 and #695531.

(Whether there should be such a distinction is something on which there
are differing opinions; that's also subtly different from whether there
is one.)

Regards,

Adam

Re: Mass bug filing: dpkg-buildpackage -A

[Santiago Vila]

On Mon, Nov 23, 2015 at 08:42:14PM +0100, Julien Cristau wrote:
> On Tue, Nov 24, 2015 at 06:14:43 +1100, Ben Finney wrote:
> 
> > If it's a “severe violation of Debian policy”, the bug is at least
> > “serious” severity.
> > 
> The release team's RC policy decides which policy violations we consider
> "severe" in the sense of "gets a serious severity bug".

Indeed, this is how I thought the severities were being used. Not in
theory but in practice.

Please don't worry about severities. I will try to be "nice" and will
report them as "important", as there are a lot of affected packages.

But will use some kind of usertags so that release managers are able
to change severities easily. I would love to see this as a release
goal for stretch, but I think it's soon to decide about that.

Thanks.

Re: Mass bug filing: dpkg-buildpackage -A

[Santiago Vila]

On Mon, Nov 23, 2015 at 05:51:44PM +0100, Thomas Goirand wrote:
> > As the number of bugs is going to be greater than ten
> 
> Do you have the exact figures? How many packages are affected?

I don't have exact figures yet, but I estimate about 300, more or less.

I considered the source packages generating at least one "Architecture: amd64"
package and at least one "Architecture: all" package. There are about 3100
of those in stretch/main, and the failure rate I'm getting is about 10%.

Re: Mass bug filing: dpkg-buildpackage -A

[Thomas Goirand]

On 11/23/2015 10:52 AM, Santiago Vila wrote:
> Greetings.
> 
> I've noticed that some packages fail to build from source when built
> using "dpkg-buildpackage -A".
> 
> This is not only a violation of policy, but also prevents the package
> from being uploaded in source-only form, as the architecture-independent
> packages are generated by a dedicated "Arch: all" autobuilder which
> does exactly "dpkg-buildpackage -A" and nothing else.
> 
> [ The problem with source-only uploads is aggravated by this bug in
>   ftp.debian.org:
> 
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=798413
> 
>   which apparently it's only a "normal" bug ].
> 

Hi Santiago,

Thanks for working on this.

> As the number of bugs is going to be greater than ten

Do you have the exact figures? How many packages are affected?

Cheers,

Thomas Goirand (zigo)

Re: Mass bug filing: dpkg-buildpackage -A

[Josh Triplett]

Santiago Vila wrote:
> I've noticed that some packages fail to build from source when built
> using "dpkg-buildpackage -A".
> 
> This is not only a violation of policy, but also prevents the package
> from being uploaded in source-only form, as the architecture-independent
> packages are generated by a dedicated "Arch: all" autobuilder which
> does exactly "dpkg-buildpackage -A" and nothing else.

Thanks for checking this and for filing bugs on this.

> As the number of bugs is going to be greater than ten, I hereby
> announce my intent to report as many bugs of this type as I found.

Please do.

> I think those are clearly bugs, and they should be reported.
> The only thing that may be unclear is the severity. I would like this
> to be RC in some not too distant future (to be on par with ordinary
> "dpkg-buildpackage"), but for now I think "severity: important" would
> be appropriate here.

These bugs represent policy violations; by definition, policy violations
default to "Severity: serious".

I would suggest filing these bugs as "serious", but also explicitly
tagging the versions affected, as the bugs almost certainly affect the
versions in testing as well.  As long as you tag the affected versions,
the bugs won't actually prevent migration to testing.

- Josh Triplett

Re: Mass bug filing: dpkg-buildpackage -A

[Ben Finney]

Julien Cristau  writes:

> On Tue, Nov 24, 2015 at 06:14:43 +1100, Ben Finney wrote:
>
> > If it's a “severe violation of Debian policy”, the bug is at least
> > “serious” severity.

This is one way that a bug can be determined “Severity: serious”, by
definition.

In other words: “severe violation of Debian policy” implies “bug is
Severity: serious”.

> The release team's RC policy decides which policy violations we
> consider "severe" in the sense of "gets a serious severity bug".

And this is another, by definition.

In other words: “release team determined the bug is RC” implies “bug is
Severity: serious”.

The reverse implication is not true: one cannot infer from “Severity:
serious” that the release team made that determination. The “Severity”
field doesn't *only* mean that, so it's not a valid inference.

By my reading of the “serious” severity definition, any of its “or”
clauses makes for sufficient justification to set “Severity: serious”
without considering the other clauses.

-- 
 \“If this is your first visit to the USSR, you are welcome to |
  `\  it.” —hotel room, Moscow |
_o__)  |
Ben Finney

Re: Heads up: Upcoming dpkg-buildpackage -j precedence change

[Guillem Jover]

Hi!

On Wed, 2015-05-13 at 14:21:54 +0200, Guillem Jover wrote:
 On Tue, 2015-05-12 at 10:02:27 +0100, Jonathan Dowland wrote:
  On Mon, May 11, 2015 at 08:40:16PM +0200, Guillem Jover wrote:
   $ make -jN -f debian/rules build
   
   and
   
   $ DEB_BUILD_OPTIONS=parallel=N debian/rules build
  
  I prefer the latter behaviour but the former brevity. Would you consider
  something like -J in dpkg-buildpackage adjusting parallel= in
  DEB_BUILD_OPTIONS, and perhaps in the future phasing out -j?
 
 Sure, adding either an option to disable the forced parallel runs in
 combination with -j, or a new option such as -J seems fine to me.

I've now pushed a patch to dpkg git master that adds -J to
dpkg-buildpackage, will be included in the upcoming dpkg 1.18.2.

Thanks,
Guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150730044738.ga2...@gaara.hadrons.org

Re: Heads up: Upcoming dpkg-buildpackage -j precedence change

[Wookey]

+++ Jonas Smedegaard [2015-05-13 15:44 +0200]:
 Quoting Julian Taylor (2015-05-13 14:48:02)
  Are those still parallel or does the flag override all submakes? The 
  option is not documented in make's manpage.
 
 From make man page:
 
  SEE ALSO
 
  The full documentation for make is maintained as a Texinfo manual.  If 
  the info and make programs are properly installed at your site, the 
  command
 
  info make
 
  should give you access to the complete manual.

pinfo, pinfo! Never make anyone use info, it's mean :-)


Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150602135625.gm26...@halon.org.uk

Re: Heads up: Upcoming dpkg-buildpackage -j precedence change

[Guillem Jover]

[ reproducible-builds people, please see below. ]

Hi!

On Tue, 2015-05-12 at 10:02:27 +0100, Jonathan Dowland wrote:
 On Mon, May 11, 2015 at 08:40:16PM +0200, Guillem Jover wrote:
  $ make -jN -f debian/rules build
  
  and
  
  $ DEB_BUILD_OPTIONS=parallel=N debian/rules build
 
 I prefer the latter behaviour but the former brevity. Would you consider
 something like -J in dpkg-buildpackage adjusting parallel= in
 DEB_BUILD_OPTIONS, and perhaps in the future phasing out -j?

Sure, adding either an option to disable the forced parallel runs in
combination with -j, or a new option such as -J seems fine to me.

The “nice” thing about -j is that it allows anyone to naturally exercise
parallel builds (like they'd do with an upstream project) and possibly
notice if something does not support them. Getting rid of it would mean
that the current marking of packages might be even slower than it is.
Because we have to opt-in for every and each package in the archive,
with the usual multi-year Debian adoption rates and probably longer in
this case. So I think having both semantics at hand is useful.

And of course «dpkg-buildpackage -jN» does not cover build systems that
do not use make, but I've always thought of DEB_BUILD_OPTIONS=paralle=N
as a way to request that.

  Actually Makefiles that do not support parallel builds and do not
  declare so with .NOTPARALLEL: are buggy, because upstreams do not
  have any standardized opt-in way to honor a parallel build request.
 
 Now that rules files have to be makefiles; I wonder if this should be 
 clarified
 in policy, one way or the other. That is: should debian/rules be assumed to
 support parallel building by default, or not? I imagine the saner default for
 us would be not (even though I am a big fan of parallel builds).

IMO dpkg-buildpackage should assume yes, in the same way it assumes
packages are cross-buildable, and we don't go around marking them as
such. But I guess for Debian that depends on how much of our packaging
and upstream build systems are parallel buildable. I'd have assumed that
with projects like Gentoo around and multicore systems being so common
nowadays, many things would be parallel buildable already, although I
don't have numbers…

… but, we do have a way to check if a package is parallel buildable now,
through reproducible builds. If a package outputs exactly the same with
dpkg-buildpackage -j1 and -jauto (w/o DEB_BUILD_OPTIONS) then it means
the package is good, otherwise it might need fixing one way or another.

Thanks,
Guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150513122154.ga17...@gaara.hadrons.org

Re: Heads up: Upcoming dpkg-buildpackage -j precedence change

[Julian Taylor]

On Wed, May 13, 2015 at 2:21 PM, Guillem Jover guil...@debian.org wrote:

 IMO dpkg-buildpackage should assume yes, in the same way it assumes
 packages are cross-buildable, and we don't go around marking them as
 such. But I guess for Debian that depends on how much of our packaging
 and upstream build systems are parallel buildable. I'd have assumed that
 with projects like Gentoo around and multicore systems being so common
 nowadays, many things would be parallel buildable already, although I
 don't have numbers…

 … but, we do have a way to check if a package is parallel buildable now,
 through reproducible builds. If a package outputs exactly the same with
 dpkg-buildpackage -j1 and -jauto (w/o DEB_BUILD_OPTIONS) then it means
 the package is good, otherwise it might need fixing one way or another.



I'd argue assume no.
I know none of my packages profit from parallel debian/rules execution
as they are all either IO bound or already sufficiently parallelized
via the upstream build system (which is handled by DEB_BUILD_OPTIONS).

Also I know that at least one of my packages will fail with parallel
debian/rules execution, because I ran into that dpkg-buildpackage -j
issue before.
Fixing the file to work in parallel gains me nothing, so I don't see
why I should do it.

I could add .NOTPARALLEL to all my packages, though how does that
affect $(MAKE) -j(buildoptions) calls in the rules file?
Are those still parallel or does the flag override all submakes? The
option is not documented in make's manpage.


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/CAK5FAtFwdMD9R-�evf6_5-t1sogmzaqoeu0kjhmdfdaq9...@mail.gmail.com

Re: Heads up: Upcoming dpkg-buildpackage -j precedence change

[Jonas Smedegaard]

Quoting Julian Taylor (2015-05-13 14:48:02)
 Are those still parallel or does the flag override all submakes? The 
 option is not documented in make's manpage.

From make man page:

 SEE ALSO

 The full documentation for make is maintained as a Texinfo manual.  If 
 the info and make programs are properly installed at your site, the 
 command

 info make

 should give you access to the complete manual.


 - Jonas

-- 
 * Jonas Smedegaard - idealist  Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: signature

Re: Heads up: Upcoming dpkg-buildpackage -j precedence change

[Jonathan Dowland]

On Mon, May 11, 2015 at 08:40:16PM +0200, Guillem Jover wrote:
 $ make -jN -f debian/rules build
 
 and
 
 $ DEB_BUILD_OPTIONS=parallel=N debian/rules build

I prefer the latter behaviour but the former brevity. Would you consider
something like -J in dpkg-buildpackage adjusting parallel= in
DEB_BUILD_OPTIONS, and perhaps in the future phasing out -j?

 Actually Makefiles that do not support parallel builds and do not
 declare so with .NOTPARALLEL: are buggy, because upstreams do not
 have any standardized opt-in way to honor a parallel build request.

Now that rules files have to be makefiles; I wonder if this should be clarified
in policy, one way or the other. That is: should debian/rules be assumed to
support parallel building by default, or not? I imagine the saner default for
us would be not (even though I am a big fan of parallel builds).

-- 
Jonathan Dowland


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150512090227.gb14...@chew.redmars.org

Re: Heads up: Upcoming dpkg-buildpackage -j precedence change

[Guillem Jover]

On Sun, 2015-05-10 at 19:56:49 +0200, Julien Cristau wrote:
 On Sun, May 10, 2015 at 18:49:25 +0100, Wookey wrote:
  I'm happy if you change this - it seems like fixing a bug to me, but I
  will just throw in this observation from recent arm64 archive-rebuilds,
  that -j and DEB_BUILD_OPTIONS=parallel= are not exactly the same. Is
  that expected?

Yes, this is expected, there was a bug report recently asking to
clarify this in the man page, fixed in master and will be included
in the upcoming dpkg 1.18.0 release.

 dpkg-buildpackage -j is buggy.

No. This is the equivalent distinction between:

$ make -jN -f debian/rules build

and

$ DEB_BUILD_OPTIONS=parallel=N debian/rules build

You might not like the behavior, and that is fine, but that does not
make it buggy. (And BTW debuild also sets MAKEFLAGS.)

 It sets MAKEFLAGS whether the package
 supports parallel builds or not.  Whereas setting DEB_BUILD_OPTIONS lets
 the package know it's allowed to use more processes if it wants to /
 can, but doesn't have to.

Actually Makefiles that do not support parallel builds and do not
declare so with .NOTPARALLEL: are buggy, because upstreams do not
have any standardized opt-in way to honor a parallel build request.

Regards,
Guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150511184016.ga18...@gaara.hadrons.org

Re: Heads up: Upcoming dpkg-buildpackage -j precedence change

[Wookey]

+++ Guillem Jover [2015-05-10 04:53 +0200]:
 Hi!
 
 “Recently” when adding support for «-jauto» to dpkg-buildpackage, I
 noticed that the semantics for the -j option were quite unorthodox.
 The value from the DEB_BUILD_OPTIONS paralle= option takes precedence
 and overrides any explicit value passed on the commend-line via -j,
 (when -j should be overriding the environment instead).
 
 I'm not entirely sure what I was thinking when I did that change in
 dpkg 1.14.15 (2008-01), but that specific part seems entirely broken.
 But even then given that this behavior has been like that for a long
 time now, I'm a bit hesitant to suddenly change it.
 
 If you happen to depend on this behavior and stuff would break due to
 that change, please shout and I'll deploy a staged transition with
 warnings and similar, otherwise I'll just go ahead with the change.

I'm happy if you change this - it seems like fixing a bug to me, but I
will just throw in this observation from recent arm64 archive-rebuilds, that
-j and DEB_BUILD_OPTIONS=parallel= are not exactly the same. Is that
expected? If not then it should perhaps be considered/investigated in
case other builds are sensitive to the difference?

here is a message from Ed Grimley-Evans, checking the FTBFS:
---
 freecdb illustrates the problem repeatably:

 works: 
   DEB_BUILD_OPTIONS=parallel=4 dpkg-buildpackage -b

 fails:
   dpkg-buildpackage -b -j4

I haven't worked out precisely what goes wrong, but it seems to have
something to do with a version of debian/implicit from 2005/2006,
which was perhaps written with the assumption that dependencies are
built one at a time in order. The whole package is that old, in fact.
  
Anyway, what's the bug? Are packages that won't build with
dpkg-buildpackage -j4 buggy, or is it a bug that man pages suggest
using dpkg-buildpackage -j4 rather than
DEB_BUILD_OPTIONS=parallel=4 dpkg-buildpackage -b? 


This seems to be reproducible even on a dual-core amd64 machine. 

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150510174924.gu24...@halon.org.uk

Re: Heads up: Upcoming dpkg-buildpackage -j precedence change

[Julien Cristau]

On Sun, May 10, 2015 at 18:49:25 +0100, Wookey wrote:

 I'm happy if you change this - it seems like fixing a bug to me, but I
 will just throw in this observation from recent arm64 archive-rebuilds, that
 -j and DEB_BUILD_OPTIONS=parallel= are not exactly the same. Is that
 expected? If not then it should perhaps be considered/investigated in
 case other builds are sensitive to the difference?
 
 here is a message from Ed Grimley-Evans, checking the FTBFS:
 ---
  freecdb illustrates the problem repeatably:
 
  works: 
DEB_BUILD_OPTIONS=parallel=4 dpkg-buildpackage -b
 
  fails:
dpkg-buildpackage -b -j4
 
 I haven't worked out precisely what goes wrong, but it seems to have
 something to do with a version of debian/implicit from 2005/2006,
 which was perhaps written with the assumption that dependencies are
 built one at a time in order. The whole package is that old, in fact.
   
 Anyway, what's the bug? Are packages that won't build with
 dpkg-buildpackage -j4 buggy, or is it a bug that man pages suggest
 using dpkg-buildpackage -j4 rather than
 DEB_BUILD_OPTIONS=parallel=4 dpkg-buildpackage -b? 
 
 
 This seems to be reproducible even on a dual-core amd64 machine. 
 
dpkg-buildpackage -j is buggy.  It sets MAKEFLAGS whether the package
supports parallel builds or not.  Whereas setting DEB_BUILD_OPTIONS lets
the package know it's allowed to use more processes if it wants to /
can, but doesn't have to.

Cheers,
Julien


signature.asc
Description: Digital signature

Heads up: Upcoming dpkg-buildpackage -j precedence change

[Guillem Jover]

Hi!

“Recently” when adding support for «-jauto» to dpkg-buildpackage, I
noticed that the semantics for the -j option were quite unorthodox.
The value from the DEB_BUILD_OPTIONS paralle= option takes precedence
and overrides any explicit value passed on the commend-line via -j,
(when -j should be overriding the environment instead).

I'm not entirely sure what I was thinking when I did that change in
dpkg 1.14.15 (2008-01), but that specific part seems entirely broken.
But even then given that this behavior has been like that for a long
time now, I'm a bit hesitant to suddenly change it.

If you happen to depend on this behavior and stuff would break due to
that change, please shout and I'll deploy a staged transition with
warnings and similar, otherwise I'll just go ahead with the change.

Thanks,
Guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20150510025347.ga15...@gaara.hadrons.org

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Benjamin Drung]

Am Sonntag, den 05.01.2014, 15:09 +0100 schrieb Guillem Jover:
 On Thu, 2014-01-02 at 17:22:33 +, Jonathan Dowland wrote:
  You raise some very valid points and §I appreciate your concerns and
  perhaps should rephrase my request so that I'm suggesting subsuming the
  most common used features of debsign and perhaps as part of a staged
  migration (compat symlink to debsign binary name in the phase 1, real
  name dpkg-sign or whatever), to try and avoid further complicating the
  debian package development universe.
 
  On Thu, Jan 02, 2014 at 11:03:04AM +0100, Guillem Jover wrote:
   IMO having debsign become a thiner wrapper around this new tool would
   be the goal, as it would simplify its code,
 
 (Obviously, assuming devscripts maintainers would agree, I was just
 inferring from previous interactions regarding debuild for example; in
 any case what happens with debsign would be their decision entirely.)

I agree with this proposal. It can become a thin wrapper around the new
tool (assuming someone does the work). Then it can either fade away (if
the new tool can replace it completely) or stay forever (like debuild)
if it provides additional features.

-- 
Benjamin Drung
Debian  Ubuntu Developer


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1389221512.3369.6.camel@deep-thought

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Didier 'OdyX' Raboud]

Le samedi, 4 janvier 2014, 19.54:05 Stefano Rivera a écrit :
 Hi Jonathan (2014.01.02_19:22:33_+0200)
 
* having to support remote signing
  
  It would be fair enough to stderr not supported, please use the
  older tool in devscripts and error 1 if such an argument was
  provided. That would be pragmatic if (as I suspect) -r is rarely
  used.
 
 Aww. I'm a frequent user of -r.
 I have better places to build big packages than on my lap, and I
 prefer to keep my GPG key in as few places as possible.

I concur. I build my packages on a server's sbuild and then debsign -r 
from my laptop, then dput from the server.

Please keep this workflow possible with in-archive tools.

Cheers,
OdyX

signature.asc
Description: This is a digitally signed message part.

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Lisandro Damián Nicanor Pérez Meyer]

On Sunday 05 January 2014 11:58:07 Didier 'OdyX' Raboud wrote:
 Le samedi, 4 janvier 2014, 19.54:05 Stefano Rivera a écrit :
  Hi Jonathan (2014.01.02_19:22:33_+0200)
  
 * having to support remote signing
   
   It would be fair enough to stderr not supported, please use the
   older tool in devscripts and error 1 if such an argument was
   provided. That would be pragmatic if (as I suspect) -r is rarely
   used.
  
  Aww. I'm a frequent user of -r.
  I have better places to build big packages than on my lap, and I
  prefer to keep my GPG key in as few places as possible.
 
 I concur. I build my packages on a server's sbuild and then debsign -r
 from my laptop, then dput from the server.
 
 Please keep this workflow possible with in-archive tools.

Indeed, my case here too. -r is crucial for my workflow :-/

-- 

Lisandro Damián Nicanor Pérez Meyer
http://perezmeyer.com.ar/
http://perezmeyer.blogspot.com/


signature.asc
Description: This is a digitally signed message part.

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Guillem Jover]

On Thu, 2014-01-02 at 17:22:33 +, Jonathan Dowland wrote:
 You raise some very valid points and §I appreciate your concerns and
 perhaps should rephrase my request so that I'm suggesting subsuming the
 most common used features of debsign and perhaps as part of a staged
 migration (compat symlink to debsign binary name in the phase 1, real
 name dpkg-sign or whatever), to try and avoid further complicating the
 debian package development universe.

 On Thu, Jan 02, 2014 at 11:03:04AM +0100, Guillem Jover wrote:
  IMO having debsign become a thiner wrapper around this new tool would
  be the goal, as it would simplify its code,

(Obviously, assuming devscripts maintainers would agree, I was just
inferring from previous interactions regarding debuild for example; in
any case what happens with debsign would be their decision entirely.)

  people not wanting to use
  debsign could use the dpkg tool directly, and people not wanting to
  learn new stuff could keep using the wrapper w/o regressions.

 That would not address the concern I raise: the surface area of debian
 package development tools would continue to grow, meaning people would
 use the non-recommended tool if they did not know better (or relied on
 documentation which had not been updated).

Honestly, I've been struggling a bit trying to understand this concern,
because to me that reads as suggesting no new features, command-line
options or tools should be added (to dpkg or devscripts or similar
packages), to avoid increasing the packaging tools surface area, when
in this case usage of this tool would be completely optional, and might
make life easier for some people. I don't see either the problem of using
one or the other, or one being more recommended than the other. If a new
tool would get added to dpkg every second week, then I could see it, but
not with the current pace.

If it was just a “hey, please consider creating a single interfaces that
can cover all current usages, and drop the old one if possible”, sure
I'm all for integrating back stuff that has been developed or prototyped
elsewhere, and trying to end up with a single interface by deprecating
the old interface if necessary. In this case though, I think debsign
(and debuild, and other similar wrappers) add value, and have features
that I don't see fit in dpkg proper, or might serve as future playground
to try out new stuff that could get merged back too. So in this case I
don't see its deprecation as desirable (but again that's something for
the devscripts maintainers to decide).

 [1] haven't checked whether they, in turn, rely on debsign.

They do, as well as many tools that need to control the signing
process, some mentioned in this thread.

Thanks,
Guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20140105140914.ga5...@gaara.hadrons.org

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Stefano Rivera]

Hi Jonathan (2014.01.02_19:22:33_+0200)
   * having to support remote signing
 
 It would be fair enough to stderr not supported, please use the older
 tool in devscripts and error 1 if such an argument was provided. That
 would be pragmatic if (as I suspect) -r is rarely used.

Aww. I'm a frequent user of -r.
I have better places to build big packages than on my lap, and I prefer
to keep my GPG key in as few places as possible.

But of course, this could be replaced by a new remote signing wrapper.
And, if popular enough, end up in devscripts...

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  H: +27 21 461 1230 C: +27 72 419 8559


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20140104175405.ga7...@bach.rivera.co.za

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Jonathan Dowland]

On Mon, Dec 30, 2013 at 12:27:29PM +0100, Guillem Jover wrote:
  * signing would get rafactored into a new program so that users do
not need to manually mangle the .changes file, rebuild or require
devscripts for something that was possible out-of-the-box.

I chose to read that as debsign will move from devscripts to src:dpkg
and felt happier. I actually don't care if debsign is reimplemented, but
I'd prefer if the Debian package development brain space was not further
complicated with more command line tools to learn[1], so I suggest that
the existing name and arguments are preserved.

[1] or ignore


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20140102092455.ga25...@bryant.redmars.org

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Guillem Jover]

Hi!

On Thu, 2014-01-02 at 09:24:55 +, Jonathan Dowland wrote:
 On Mon, Dec 30, 2013 at 12:27:29PM +0100, Guillem Jover wrote:
   * signing would get rafactored into a new program so that users do
 not need to manually mangle the .changes file, rebuild or require
 devscripts for something that was possible out-of-the-box.
 
 I chose to read that as debsign will move from devscripts to src:dpkg
 and felt happier. I actually don't care if debsign is reimplemented, but
 I'd prefer if the Debian package development brain space was not further
 complicated with more command line tools to learn[1], so I suggest that
 the existing name and arguments are preserved.

I'd feel very uncomfortable doing that, because it would mean, at least:

 * introducing a core dpkg tool within a different namespace
 * having to parse a devscripts specific configuration file
 * having to support DAK specific .commands signing
 * having to support remote signing

IMO having debsign become a thiner wrapper around this new tool would
be the goal, as it would simplify its code, people not wanting to use
debsign could use the dpkg tool directly, and people not wanting to
learn new stuff could keep using the wrapper w/o regressions.

Thanks,
Guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20140102100304.ga10...@gaara.hadrons.org

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Jonathan Dowland]

You raise some very valid points and §I appreciate your concerns and
perhaps should rephrase my request so that I'm suggesting subsuming the
most common used features of debsign and perhaps as part of a staged
migration (compat symlink to debsign binary name in the phase 1, real
name dpkg-sign or whatever), to try and avoid further complicating the
debian package development universe.

On Thu, Jan 02, 2014 at 11:03:04AM +0100, Guillem Jover wrote:
 I'd feel very uncomfortable doing that, because it would mean, at least:
 
  * introducing a core dpkg tool within a different namespace

So dpkg-sign (or dpkg sign if you ever decided to move to internal
namespacing, which seems to be fashionable) with a compatibility symlink
to debsign would resolve this one,

  * having to parse a devscripts specific configuration file

That's more awkward, I'd agree, but you could support the command-line
arguments without supporting the devscripts configuration file. It could
be confusing for those who have relied upon it, though. A more
considered migration would be necessary.

  * having to support DAK specific .commands signing

That could be awkward, although the format is very similar (if not
identical) to debian/control and only one header is of interest. I
imagine most people use dcut/dput nowadays[1]

  * having to support remote signing

It would be fair enough to stderr not supported, please use the older
tool in devscripts and error 1 if such an argument was provided. That
would be pragmatic if (as I suspect) -r is rarely used.

 IMO having debsign become a thiner wrapper around this new tool would
 be the goal, as it would simplify its code, people not wanting to use
 debsign could use the dpkg tool directly, and people not wanting to
 learn new stuff could keep using the wrapper w/o regressions.

That would not address the concern I raise: the surface area of debian
package development tools would continue to grow, meaning people would
use the non-recommended tool if they did not know better (or relied on
documentation which had not been updated).

[1] haven't checked whether they, in turn, rely on debsign.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20140102172233.ga7...@bryant.redmars.org

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Roger Leigh]

On Mon, Dec 30, 2013 at 12:27:29PM +0100, Guillem Jover wrote:
 I guess it's probably a good idea to switch the default, becuse I
 assume most maintainers do more test builds than final ones. Or users
 who either don't have gpg installed or don't have a gpg key. Although
 with the current no-signing-UNRELEASED behaviour, the need for -us -uc
 should have dropped in many cases.

On the sbuild/buildd side, we have run dpkg-buildpackage with
-us -uc by default for years.  If you do enable signing, as is
the case for buildd uploads, we run debsign explicitly after
dpkg-buildpackage completed.  This avoids any need for GPG keys
to be present in the build chroot.  So from the POV of making
-us -uc the default, I think that's a good plan and matches
the requirements of the majority of both manual and automated
builds.  And from the POV of having a replacement for debsign,
we can conditionally switch to using it as soon as it becomes
available.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?   http://gutenprint.sourceforge.net/
   `-GPG Public Key: 0x25BFB848   Please GPG sign your mail.


signature.asc
Description: Digital signature

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Ian Jackson]

Roger Leigh writes (Re: Bug#733029: dpkg-buildpackage: disable signing by 
default (-us -uc should be the default)):
 On the sbuild/buildd side, we have run dpkg-buildpackage with
 -us -uc by default for years.  If you do enable signing, as is
 the case for buildd uploads, we run debsign explicitly after
 dpkg-buildpackage completed.

Likewise dgit, which does the signing only in dgit push.  It does it
by itself without the use of debsign or anything from dpkg-dev.

I have no objection to the proposed change to dpkg-buildpackage, or
the moving of functionality from devscripts.  

I do of course want debsign's command line interface and functionality
to continue to work in its entirity.  But I also need the arrangements
for making the signature remain part of the defined interface, so that
dgit keeps working: i.e. even if this functionality becomes part of
dpkg-dev, its use should not be mandatory.

Thanks,
Ian.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/21189.47230.191871.195...@chiark.greenend.org.uk

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Guillem Jover]

Hi!

[ CCing debian-devel to get input from possibly afftected users. ]

On Tue, 2013-12-24 at 15:14:22 +0800, Paul Wise wrote:
 Package: dpkg-dev
 Severity: wishlist
 File: /usr/bin/dpkg-buildpackage
 X-Debbugs-CC: Arno Töll a...@debian.org
 
 Please disable signing by default. Generally the right thing to do for
 most dpkg-dev users is to have dpkg-buildpackage *not* auto-sign
 anything. Almost all dpkg-dev users (except buildds) should test and
 check the package *before* signing and uploading the package. We have a
 volunteer for adjusting the buildds (in CC) if needed and I volunteer to
 adjust any documentation outside of dpkg-dev that needs changing.

The difference between this request and the one about not signing
UNRELEASED builds, is that this one has way greater user impact, and
the latter is supposedly not meant to be signed (in the general case)
anyway.

I guess it's probably a good idea to switch the default, becuse I
assume most maintainers do more test builds than final ones. Or users
who either don't have gpg installed or don't have a gpg key. Although
with the current no-signing-UNRELEASED behaviour, the need for -us -uc
should have dropped in many cases.

In any case, this is about the tool best global default, irrespective of
any future possibility of setting local defaults through a configuration
file or similar.

If most people agree this would be an improvement, and not just an
annoyance, I'm happy to do the change. But there's few things that I'd
want to tackle first:

 * signing would get rafactored into a new program so that users do
   not need to manually mangle the .changes file, rebuild or require
   devscripts for something that was possible out-of-the-box.
 * (possibly) new options would get added to specify signing, although
   there's --force-sign since 1.17.0, which could be used already on
   the buildds.
 * (possibly) wait for a grace period with warnings and the above
   available so that users can switch scripts/programs gracefully,
   before the flag day.

Thanks,
Guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20131230112729.ga16...@gaara.hadrons.org

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Jakub Wilk]

* Guillem Jover guil...@debian.org, 2013-12-30, 12:27:
I guess it's probably a good idea to switch the default, becuse I 
assume most maintainers do more test builds than final ones.


ACK

If most people agree this would be an improvement, and not just an 
annoyance, I'm happy to do the change. But there's few things that I'd 
want to tackle first:



[...]
* (possibly) new options would get added to specify signing, although 
there's --force-sign since 1.17.0, which could be used already on the 
buildds.


I took a look at a random build log[0], and it looks like buildds don't 
use dpkg-buildpackage for signing, but debsign. (It would be nice if 
someone from the wb-team could confirm that this is the case.)



[0] 
https://buildd.debian.org/status/fetch.php?pkg=dpkgarch=i386ver=1.17.5stamp=1386841160

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20131230180054.ga5...@jwilk.net

Re: Bug#733029: dpkg-buildpackage: disable signing by default (-us -uc should be the default)

[Raphael Hertzog]

On Mon, 30 Dec 2013, Jakub Wilk wrote:
 * Guillem Jover guil...@debian.org, 2013-12-30, 12:27:
 I guess it's probably a good idea to switch the default, becuse I
 assume most maintainers do more test builds than final ones.
 
 ACK

And also, it should be friendly to non-maintainers who are just trying
to rebuild the package grabbed from Debian. The assumption that the
builder is the maintainer is inadequate to start with.

So I'm in favor of a transition. I also agree that it makes sense to
add a dedicated signing tool in dpkg-dev (merging debsign as dpkg-sign?)
and that this change should dealt together with a general revamp of
dpkg-buildpackage.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20131230182316.gb24...@x230-buxy.home.ouaza.com

Re: dpkg-buildpackage creating uninstallable packages?

[Norbert Preining]

On So, 29 Sep 2013, Stephen Kitt wrote:
  Uninstall the libc6-amd64:i386 package.
  See http://lists.debian.org/debian-mentors/2013/03/msg00139.html.
 
 But watch out for http://bugs.debian.org/699206 - make sure you have a root
 sash running somewhere so you can relink /lib64/ld-linux-x86-64.so.2...

Indeed, suddenly my system was hosed ... nothing did run again. Umpf.
I managed to get it back without knowing the above bug, but
it did cost me some nerves.

Sorry, might I ask *why* bugs like this that break *all* other software
are tagged as important and open since *January* ???

Norbert


PREINING, Norbert   http://www.preining.info
JAIST, Japan TeX Live  Debian Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130930075943.gb17...@gamma.logic.tuwien.ac.at

Re: dpkg-buildpackage creating uninstallable packages?

[Dominik George]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512



Norbert Preining prein...@logic.at schrieb:
On So, 29 Sep 2013, Stephen Kitt wrote:
  Uninstall the libc6-amd64:i386 package.
  See http://lists.debian.org/debian-mentors/2013/03/msg00139.html.

 But watch out for http://bugs.debian.org/699206 - make sure you have
a root
 sash running somewhere so you can relink
/lib64/ld-linux-x86-64.so.2...

Indeed, suddenly my system was hosed ... nothing did run again. Umpf.
I managed to get it back without knowing the above bug, but
it did cost me some nerves.

Sorry, might I ask *why* bugs like this that break *all* other software
are tagged as important and open since *January* ???

Norbert


PREINING, Norbert
http://www.preining.info
JAIST, Japan TeX Live  Debian
Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5
B094


Simply put: Because you made no effort to fix it :).

- -nik
- --
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
-BEGIN PGP SIGNATURE-
Version: APG v1.0.8-fdroid

iQFNBAEBCgA3BQJSST1JMBxEb21pbmlrIEdlb3JnZSAobW9iaWxlIGtleSkgPG5p
a0BuYXR1cmFsbmV0LmRlPgAKCRAvLbGk0zMOJQMXB/954UMli1fbKU4qTASJ+mOw
4D7txAdR0MUUgKHrelZeJ4MNPsstOvGybqtd14NdrG0WnCZM3w1hWv9kyYtX76n4
ot7N79zReheZJsSj/uQ0nVjPL6N9nut5ONzd+suLQhThg0dHCzuUiPUC7hPNmKEC
h3r7pLw3zw/f8cNAn4QA4XvBfoU2TS5+Il6YZ0ODxGvJE6mdeGYO3SXh09HmkABA
Ec1KNDNOs5zOHQjNnb75+9WGZXs/5DJnTDxrMAkPS8qbgSfT+N+RfTU9WBD2f/Wv
u2JjbsNWqsDZLkegtKgOsrWuGIA52inSD+jIaXhGPH6Aviv4bcw+5qYdTj8F5jn2
=fgDw
-END PGP SIGNATURE-


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/20155ad1-6366-460d-86da-0388793af...@email.android.com

Re: dpkg-buildpackage creating uninstallable packages?

[Norbert Preining]

Hi Dominik,

 Simply put: Because you made no effort to fix it :).

Thanks for the very useful comment.

Yes, I care for RC bugs in my own packages ... and that are quite
a lot. So no time to fix RC bugs of other maintainers.

Norbert


PREINING, Norbert   http://www.preining.info
JAIST, Japan TeX Live  Debian Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130930123939.ga25...@gamma.logic.tuwien.ac.at

Re: dpkg-buildpackage creating uninstallable packages?

[Dominik George]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512



Norbert Preining prein...@logic.at schrieb:
Hi Dominik,

 Simply put: Because you made no effort to fix it :).

Thanks for the very useful comment.

Yes, I care for RC bugs in my own packages ... and that are quite
a lot. So no time to fix RC bugs of other maintainers.

Norbert


PREINING, Norbert
http://www.preining.info
JAIST, Japan TeX Live  Debian
Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5
B094


Hi Norbert,

I do not filter my replies for DDs or Non-DDs. If you accuse everyone else in 
the community of not caring for something, I accuse you of not noticing it 
earlier. I am certain everyone here does a great job, so I feel disappointed by 
such accusations.

- -nik
- --
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
-BEGIN PGP SIGNATURE-
Version: APG v1.0.8-fdroid

iQFNBAEBCgA3BQJSSXHtMBxEb21pbmlrIEdlb3JnZSAobW9iaWxlIGtleSkgPG5p
a0BuYXR1cmFsbmV0LmRlPgAKCRAvLbGk0zMOJUXwCAC0WiHHGm1MU0pkxiCiXOAh
SXxItktOPHms/FTnp+CTxp+ZNwEtfwH59e4UpkFYdUIkiK9uxIwoDXhJ6icY7rV3
H7/IG+Tx/3L8+pSIrMAQEvXcBrscXfKjrQexPHTlO8wgASP0BL3hUq2YHzEn9dRI
up13o2LP/lCb4w0dSV4CG9QeBlJuwEteVdNnLw5csnKYhTNWRY+wvPmYJHEiGJit
0olbEUWwqX4JZd+TFmHNnbG2xGqzgOR3EXUQApxkgRpzEL+rdXowZjmmc/AEdXya
imT7WL+ckycSLb4dwUMT9B23gQTwqaOdt771e/JEsRYHT1TD2cBh7IJInl5pL2sK
=xuJw
-END PGP SIGNATURE-


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/c8929909-a597-4928-82dd-4f29ae8ce...@email.android.com

Re: dpkg-buildpackage creating uninstallable packages?

[Norbert Preining]

severity 699206 serious
thanks

Hi Dominik,

first of all, please stop including all the email and bottom-posting,
this is a pain and against usual netiquette.

Then ...

On Mo, 30 Sep 2013, Dominik George wrote:
 If you accuse everyone else in the community
  [...]

I did not accuse anyone, I asked why a RC bug is tagged as important
and not as RC, and why there is no activity since month, although
easily reproducible.

Anyway, I have raised the severity of this bug. Removing a package
must not leave the system in an non-operable state.


Norbert


PREINING, Norbert   http://www.preining.info
JAIST, Japan TeX Live  Debian Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130930155031.ga30...@gamma.logic.tuwien.ac.at

Re: dpkg-buildpackage creating uninstallable packages?

[Norbert Preining]

Hi everyone,

second try, with more data ..


default package texinfo, I am importing a new upstream into my git,
no changes to debian/rules or debian/control, rebuild.
From the debian/control:
..
Package: info
...
Architecture: any
Multi-Arch: foreign
...

After building the package looks like:
info_5.2.0.dfsg.1-1_amd64.deb:
 new debian package, version 2.0.
 
 Version: 5.2.0.dfsg.1-1
 Architecture: amd64
 Maintainer: Debian TeX maintainers debian-tex-ma...@lists.debian.org
 Installed-Size: 451
 Depends: libc6-amd64 (= 2.15), libtinfo5, install-info
 ...

--

When installing the just built deb on my machine for testing I get:

dpkg: dependency problems prevent configuration of info:
 info depends on libc6-amd64 (= 2.15).


For completeness, I am on amd64, and on uptodate sid.

$ dpkg --print-architecture
amd64

$ dpkg -l libc6-amd64
ii  libc6-amd642.17-93  i386 Embedded GNU C Library: 64bit S..

$ apt-cache policy libc6-amd64
libc6-amd64:i386:
  Installed: 2.17-93
  Candidate: 2.17-93
  Version table:
 *** 2.17-93 0
500 http://ftp2.jp.debian.org/debian/ sid/main i386 Packages
100 /var/lib/dpkg/status


What is going wrong here?

Norbert


PREINING, Norbert   http://www.preining.info
JAIST, Japan TeX Live  Debian Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130929064022.ga11...@gamma.logic.tuwien.ac.at

Re: dpkg-buildpackage creating uninstallable packages?

[Sven Joachim]

On 2013-09-28 22:18 +0200, Norbert Preining wrote:

 since a short time when I build a binary package on my running system,
 I cannot install the created .deb anymore because it depends on
 libc-amd64 (= some.version) which somehow is not what I have although
 I am running amd64 sid.

Uninstall the libc6-amd64:i386 package.
See http://lists.debian.org/debian-mentors/2013/03/msg00139.html.

Cheers,
   Sven


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87eh88nmg3@turtle.gmx.de

Re: dpkg-buildpackage creating uninstallable packages?

[Paul Gevers]

On 29-09-13 08:40, Norbert Preining wrote:
 What is going wrong here?

For whatever reason, the amd64 build is picking up i386 paths. I don't
know how that happens, except that I expect it is some multi-arch
twitch. I recommend you build your packages in a chroot to avoid this
(an other) issues. I use pbuilder for all my builds, but other solutions
exist.

Paul




signature.asc
Description: OpenPGP digital signature

Re: dpkg-buildpackage creating uninstallable packages?

[Stephen Kitt]

On Sun, 29 Sep 2013 08:58:36 +0200, Sven Joachim svenj...@gmx.de wrote:
 On 2013-09-28 22:18 +0200, Norbert Preining wrote:
  since a short time when I build a binary package on my running system,
  I cannot install the created .deb anymore because it depends on
  libc-amd64 (= some.version) which somehow is not what I have although
  I am running amd64 sid.
 
 Uninstall the libc6-amd64:i386 package.
 See http://lists.debian.org/debian-mentors/2013/03/msg00139.html.

But watch out for http://bugs.debian.org/699206 - make sure you have a root
sash running somewhere so you can relink /lib64/ld-linux-x86-64.so.2...

Regards,

Stephen


signature.asc
Description: PGP signature

dpkg-buildpackage creating uninstallable packages?

[Norbert Preining]

Hi everyone,

since a short time when I build a binary package on my running system, I cannot 
install the created .deb anymore because it depends on
   libc-amd64 (= some.version)
which somehow is not what I have although I am running amd64 sid.

Any suggestions?

Thanks

Norbert


PREINING, Norbert  http://www.preining.info
JAIST, Japan TeX Live  Debian Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094

Re: dpkg-buildpackage creating uninstallable packages?

[Andrey Rahmatullin]

On Sun, Sep 29, 2013 at 12:18:03AM +0400, Norbert Preining wrote:
 since a short time when I build a binary package on my running system, I 
 cannot install the created .deb anymore because it depends on
libc-amd64 (= some.version)
 which somehow is not what I have although I am running amd64 sid.
You need to provide more data. 

-- 
WBR, wRAR


signature.asc
Description: Digital signature

Re: debuild/dpkg-buildpackage behaves not as expected

[Goswin von Brederlow]

debian-de...@liska.ath.cx (Olе Streicher) writes:

 Goswin von Brederlow goswin-...@web.de writes:
 debian-de...@liska.ath.cx (Ole Streicher) writes:
 I think the best way would be that debuild/dpkg-buildpackage would not
 automatically unapply the patches (so it would leave the source in the

 It doesn't automatically unapply the patches. It only restores the state
 you had before the dpkg-buildpackage was called.

 It does not since it keeps the compiled files. If you mean it serious
 with restoring the state, you should call clean here, too.

The state of the patches.

 or hook that does this for those who really need it (and know what they
 are doing).
 Which would mean that you would have to unapply patches every time you
 try to build while working on a patch. With the current behaviour I can do

 quilt push foo.patch   (foo.patch being somewhere in the middle)
 edit file
 quilt refresh
 debuild
 [...]

 You can do the same even if either clean is called before the
 unpatching was done, or if neither clean nor unpatching was done, since
 quilt recognizes the state.

If the patches aren't rolled back to foo.patch after build then the
files will differ. If you still have the file open in an editor then
editing it will destroy stuf. And quilt refresh will refresh the topmost
patch and not foo.patch as intended.

So no, you can't do the same without the patch state being restored.

 The point is really: The state
 * compiled files (*.o etc.) from a patched package, but
 * unpatched source files
 is inconsistent. 

In a good way. :)

 Best regards

 Ole

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87k3zmdxvd.fsf@frosties.localnet

Re: debuild/dpkg-buildpackage behaves not as expected

[Goswin von Brederlow]

debian-de...@liska.ath.cx (Olе Streicher) writes:

 Goswin von Brederlow goswin-...@web.de writes:
 If you need to change a file then that means that file isn't source
 anymore but generated. Try switching to out-of-tree builds if you have
 something like that.

 What is the advantage of that? From the Debian policy, I don't see a
 need why sources should kept untouched during the build process.

 Less surprises by someone unfamiliar with the source. For example:

 - you (as in some porter, not the maintainer) build the source to
   reproduce a FTBS 
 - it fails as expected
 - you edit the broken file
 - you build again and it works
 - you call clean so you can make a patch
 - clean restores the original source file destroying hours of your
   work

 If I would be unfamilar with the source, I would just not expect that
 the package behaves as I would do it myself. Instead, I would be ready
 for the case that it does tricky, undocumented things with the source --
 and create the debian patch before I am going to build the package.

 Why should a porter expect more from a package than the requirements
 specified in the policy?

Because I'm an optimist. When I work on a new source package I naively
assume that the source is nice and does no evil, ugly, hackish things.
Obviously that leaves me disapointed when it does.

But all of that doesn't mean a source isn't better if it doesn't do
evil, ugly, hackish things. Policy might not require it but common sense
encourages it.

 It is just the better design: the package was built with a patched
 source, so only the patched version knows for sure how to clean it up. 

 Note that it only calls clean with unpatched sources if you
 specifically unpatched your source before calling it.

 If you insist so much on this standard: why does debuild (or
 dpkg-buildpackage) undo the patches if they were not applied before? In
 this case, it would be up to the (rare) people to unpatch if they need
 this. One could even provide a debuild/dpkg-buildpackage option for that
 (like --unpatch-after-build or so), or do it in a hook.

There already is the uapply-patches option. But then the patch is always
unpatched after build instead of returning to the state prior to build.

 So I think having the clean target make sure patches are applied if
 needed is the better design.

  which again does not behave as expected: if clean depends on
 patch, then after debuild clean the packages is in the patched
 state even if it was unpatched before. 

Yes, if you just depend on patch then that is the result.

clean:
$(PATCH)
clean up everything
$(UNDO_PATCH)

where PATCH is a makro that records the current patch state (like dpkg
itself does) and UNDO_PATCH a makro to return the patch state to what
was saved. I haven't tried this but you can probably make those makros
use exactly the state file and format dpkg uses for abest results.

 I think the best way would be that debuild/dpkg-buildpackage would not
 automatically unapply the patches (so it would leave the source in the

It doesn't automatically unapply the patches. It only restores the state
you had before the dpkg-buildpackage was called.

 way that is described as standard for Debian), with a special option

Which means that if you start with the standard of patches applied
then they will remain applied. But if you manually deviate from the
standard then it will preserve that deviation too.

 or hook that does this for those who really need it (and know what they
 are doing).

Which would mean that you would have to unapply patches every time you
try to build while working on a patch. With the current behaviour I can
do

quilt push foo.patch   (foo.patch being somewhere in the middle)
edit file
quilt refresh
debuild
edit file
quilt refresh
debuild
edit file
quilt refresh
debuild

 I was describing the case of having changes commited to the RCS and
 generating debian/patches/* automatically (or a single debian-changes
 patch).

 A single debian-changes patch is evil -- even Lintian complains there. 
 Handling a single debian-changes patch is something I would explicitely
 *not* take as a valid use case.

 Is there a way to automatically handle a bunch of individual patches
 trough an RCS?

git-pkg has something for that for example.

 Best regards

 Ole

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87txyvmidm.fsf@frosties.localnet

Re: debuild/dpkg-buildpackage behaves not as expected

[Olе Streicher]

Goswin von Brederlow goswin-...@web.de writes:
 debian-de...@liska.ath.cx (Ole Streicher) writes:
 I think the best way would be that debuild/dpkg-buildpackage would not
 automatically unapply the patches (so it would leave the source in the

 It doesn't automatically unapply the patches. It only restores the state
 you had before the dpkg-buildpackage was called.

It does not since it keeps the compiled files. If you mean it serious
with restoring the state, you should call clean here, too.

 or hook that does this for those who really need it (and know what they
 are doing).
 Which would mean that you would have to unapply patches every time you
 try to build while working on a patch. With the current behaviour I can do

 quilt push foo.patch   (foo.patch being somewhere in the middle)
 edit file
 quilt refresh
 debuild
[...]

You can do the same even if either clean is called before the
unpatching was done, or if neither clean nor unpatching was done, since
quilt recognizes the state.

The point is really: The state
* compiled files (*.o etc.) from a patched package, but
* unpatched source files
is inconsistent. 

Best regards

Ole



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/ytzsjef6zu1@news.ole.ath.cx

Re: debuild/dpkg-buildpackage behaves not as expected

[Olе Streicher]

Goswin von Brederlow goswin-...@web.de writes:
 If you need to change a file then that means that file isn't source
 anymore but generated. Try switching to out-of-tree builds if you have
 something like that.

 What is the advantage of that? From the Debian policy, I don't see a
 need why sources should kept untouched during the build process.

 Less surprises by someone unfamiliar with the source. For example:

 - you (as in some porter, not the maintainer) build the source to
   reproduce a FTBS 
 - it fails as expected
 - you edit the broken file
 - you build again and it works
 - you call clean so you can make a patch
 - clean restores the original source file destroying hours of your
   work

If I would be unfamilar with the source, I would just not expect that
the package behaves as I would do it myself. Instead, I would be ready
for the case that it does tricky, undocumented things with the source --
and create the debian patch before I am going to build the package.

Why should a porter expect more from a package than the requirements
specified in the policy?

 It is just the better design: the package was built with a patched
 source, so only the patched version knows for sure how to clean it up. 

 Note that it only calls clean with unpatched sources if you
 specifically unpatched your source before calling it.

If you insist so much on this standard: why does debuild (or
dpkg-buildpackage) undo the patches if they were not applied before? In
this case, it would be up to the (rare) people to unpatch if they need
this. One could even provide a debuild/dpkg-buildpackage option for that
(like --unpatch-after-build or so), or do it in a hook.

 So I think having the clean target make sure patches are applied if
 needed is the better design.

 which again does not behave as expected: if clean depends on
patch, then after debuild clean the packages is in the patched
state even if it was unpatched before. 

I think the best way would be that debuild/dpkg-buildpackage would not
automatically unapply the patches (so it would leave the source in the
way that is described as standard for Debian), with a special option
or hook that does this for those who really need it (and know what they
are doing).

 I was describing the case of having changes commited to the RCS and
 generating debian/patches/* automatically (or a single debian-changes
 patch).

A single debian-changes patch is evil -- even Lintian complains there. 
Handling a single debian-changes patch is something I would explicitely
*not* take as a valid use case.

Is there a way to automatically handle a bunch of individual patches
trough an RCS?

Best regards

Ole


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/ytzzk926l53@news.ole.ath.cx

Re: debuild/dpkg-buildpackage behaves not as expected

[Goswin von Brederlow]

debian-de...@liska.ath.cx (Olе Streicher) writes:

 Goswin von Brederlow goswin-...@web.de writes:
 debian-de...@liska.ath.cx (Olе Streicher) writes:
 James McCoy vega.ja...@gmail.com writes:
 On Wed, May 16, 2012 at 04:23:05PM +0200, Olе Streicher wrote:
 Unpatching the sources *before* the build process was cleaned up makes
 no sense to me at all. Could you provide a use case for that?
 As was described in #649531:

   vcs clone repository with unpatched source
   cd repo
   ... tweak a little ...
   dpkg-buildpackage; # applies patches, builds, and unapplies patches
   vcs diff; # looks good?
   vcs commit

 This works only for the special case that build does not change any
 source file. Otherwise you would also commit the changed source files.

 And it better not. There is no excuse for changing source files during
 build. 

 Welcome to reality. Usually, configure scripts are distributed with
 (upstream) source on purpose (since they shall provide an easy adoption
 to the current enviroment without the need of additional packages), but
 they are going to be changed during the packaging.

In which case I remove them in clean since they are not source
files. I'm not a fan of backup+restore for them.

 Another examples are convienence copies of generated files from a parser

Even more so not source. If I generate a .c/.h files from .ll/.y then
they should be removed in the clean targets.

 generator. Or, sometimes Makefiles are going to be *changed* (not even
 regenerated!) by adding/changing dependencies at their end.

Evil. Make support include directives. Put the depends into a seperate
file that is completly generated.

 Would you really require for all packages using autoconf that they
 repackage upstream source in order to get rid of the regenerated files?
 And how would you handle the case that a Makefile gets a system
 dependent dependency extension?

I don't require that upstream sources are repackaged. But I don't find
it unreasonable for the clean target to remove non source files even if
upstream does ship them. Note that there is a big difference between
regenerating files and modifying them. Generated files aren't source and
can just be removed in clean.

As for Makefiles: use include.

 If you need to change a file then that means that file isn't source
 anymore but generated. Try switching to out-of-tree builds if you have
 something like that.

 What is the advantage of that? From the Debian policy, I don't see a
 need why sources should kept untouched during the build process.

Less surprises by someone unfamiliar with the source. For example:

- you (as in some porter, not the maintainer) build the source to
  reproduce a FTBS 
- it fails as expected
- you edit the broken file
- you build again and it works
- you call clean so you can make a patch
- clean restores the original source file destroying hours of your work

Or you make a patch before calling clean and that won't apply to a clean
copy due to the changes made during build.

 patch was meant as a target that *applies* the patches. Therefore,
 it does not leave the sources in the same state (since it applies the
 patches).

 I ment: It leaves the source in the same state it found it other than
 the side effects the called target(s) have themself.

 Why would you need to have a local patch target? If it is somehow
 generally useful, it should be common to all packages -- and than it
 could just be builtin as an option into dpkg-buildpackage. Or just use
 quilt directly.

In my case I have a patch target since that is easier to type than
QUILT_PATCHES=debian/patches quilt push -a and because under Lucid
debuild/dpkg-buildpackage does not apply patches before build or
unapplies the after build so I need to do that myself in debian/rules.

Nowadays it is built-in as dpkg-source --before-build. The patch
target is just a convenience and backward compatibility. And for those
that need a patched source when calling debuild clean.

 My main point, which I didn't explicitly state, is this:

 The way debuild/dpkg-buildpackage/dpkg-source currently behave allow
 maintainers to modify the behaviour by adding something to
 debian/rules. If the clean target needs the patches applied then
 debian/rules can easily make sure that they are.

 Since clean usually calls the upstream cleanup, its work depends on
 whether the upstream cleanup would actually work on the unchanged
 package. Trying to apply the clean target from the unpatched source on
 a directory that is built by the patched source seems to me buggy by
 design and just works on accident.

 On the other hand if the clean target doesn't need the patches applied,
 as is the case for 99.9% of all packages then applying them would be
 wastefull.

 It is just the better design: the package was built with a patched
 source, so only the patched version knows for sure how to clean it up. 

Note that it only calls clean with unpatched sources if you specifically
unpatched your source

Re: debuild/dpkg-buildpackage behaves not as expected

[Goswin von Brederlow]

debian-de...@liska.ath.cx (Olе Streicher) writes:

 James McCoy vega.ja...@gmail.com writes:
 On Wed, May 16, 2012 at 04:23:05PM +0200, Olе Streicher wrote:
 Unpatching the sources *before* the build process was cleaned up makes
 no sense to me at all. Could you provide a use case for that?
 As was described in #649531:

   vcs clone repository with unpatched source
   cd repo
   ... tweak a little ...
   dpkg-buildpackage; # applies patches, builds, and unapplies patches
   vcs diff; # looks good?
   vcs commit

 This works only for the special case that build does not change any
 source file. Otherwise you would also commit the changed source files.

And it better not. There is no excuse for changing source files during
build. If you need to change a file then that means that file isn't
source anymore but generated. Try switching to out-of-tree builds if you
have something like that.

 Since for Debian you have only changes in the debian/ subdirectory, you
 may do the following:

 vcs clone repository with unpatched source
 cd repo
  tweak a little ...
 dpkg-buildpackage; # applies patches, builds, and unapplies patches
 vcs diff debian
 vcs commit debian

 this does not require unpatching, since it commits only the debian
 subdirectory, which is not affected by any patches.

 Therefore, I don't see that the workflow you mentioned is a use case
 that would require unpatching.

 [Quoting restored: Goswin wrote]
 What happens if you now call
 debuild patch
 to apply the patches in a 3.0 (quilt) package that has patch/unpatch
 targets?

 because it does *not* leave the sources in the same state.

 Yes, it does.  

 He wrote it himself: patch was meant as a target that *applies* the
 patches. Therefore, it does not leave the sources in the same state
 (since it applies the patches).

I ment: It leaves the source in the same state it found it other than
the side effects the called target(s) have themself.

 If you started with patches applied, then they will still be applied
 after calling dpkg-buildpackage.  If you didn't have patches
 applied, then dpkg-buildpackage will apply the patches, build and
 unapply the patches.

 We discussed the debuild patch option here which was introduced by
 Goswin. 

 Best

 Ole

My main point, which I didn't explicitly state, is this:

The way debuild/dpkg-buildpackage/dpkg-source currently behave allow
maintainers to modify the behaviour by adding something to
debian/rules. If the clean target needs the patches applied then
debian/rules can easily make sure that they are.

On the other hand if the clean target doesn't need the patches applied,
as is the case for 99.9% of all packages then applying them would be
wastefull.


Personaly I try to set up all my packages in such a way that clean or no
clean I can simply commit changes. That means that the build MUST not
modify any source files (which is simply evil to begin with) and that
you add a bunch of stuff to the ignore file (e.g. debian/files,
debian/pkg/, debian/*.debhelper.log, debian/*.substvars, stamp files,
...). That is something you do once and after that clean or no clean you
can simply commit. But that requires that patches are unapplied after
build (unapply-patches in debian/source/local-options to enforce
this). I usualy never call clean directly so the above problem can't
even arise.

MfG
Goswin


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8762btlniv.fsf@frosties.localnet