Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi josch, On Tue, 06 Aug 2019 20:11:07 +0200, Johannes Schauer wrote: > Quoting Stephen Kitt (2019-08-06 18:43:24) > > Could we avoid (some of) this by treating “source” as a new architecture? > > There would be a mostly do-nothing buildd which would repack the source > > package as a binary package in /usr/src/... > > The large number of binary packages needed with this method could of course > be reduced if only selected source packages get built that way. But such a > method already exists in the form of foo-source binary packages. > > Changing the build dependency syntax instead would have the advantage, that > we do not need more binary or source packages at all because all the > necessary logic would come from the resolvers themselves. Thanks for the detailed explanations, changing the build dependency syntax does indeed seem like a better approach. Regards, Stephen pgp4enki5g859.pgp Description: OpenPGP digital signature
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi, Quoting Stephen Kitt (2019-08-06 18:43:24) > > recently there was some discussion about this in #debian-apt. I don't have > > the logs of that discussion so others might want to expand on what I > > remember from back then. In no particular order plus my own thoughts. > > > > apt developers are in favour of such a feature but it would need > > implementation in dpkg first. This means that dpkg would have to also track > > "installed" (unpacked) source packages in /usr/src in a similar way of how > > it currently tracks installed binary packages. > > > > Tons of software that parses the Build-Depends field has to be patched. At > > least: dpkg, sbuild, apt, debhelper, cdbs, pbuilder, lintian, dose3, > > wanna-build, dak, devscripts, python-debian, libconfig-model-perl, augeas, > > haskell-debian, dh-exec, autopkgtest... > > Could we avoid (some of) this by treating “source” as a new architecture? > There would be a mostly do-nothing buildd which would repack the source > package as a binary package in /usr/src/... > > That would still mean we’d have to add meaningful arch-qualified > build-dependencies in all the affected tools, and it doesn’t help with the > transitive dependencies (although the binary package containing the source > could carry the appropriate dependencies), so perhaps it’s not such a great > idea. sure, it could work in theory. It would just be very ugly. Lets have a look at how this would look like: To get the raw sources in /usr/src you would generate around 31000 new binary packages of the "source" architecture. These would have to be Multi-Arch:foreign so that they can satisfy dependencies coming from source packages being built for any architecture. I don't see a way to express the Architecture field of source packages via this method, which is another downside. As you already point out, properly encoding build dependencies that way is another can of worms. For this to work, you would have to generate meta packages for every architecture we support for every source package. This is so that the host architecture gets properly forwarded down the dependency graph. This would mean another few hundred thousand more binary packages. The large number of binary packages needed with this method could of course be reduced if only selected source packages get built that way. But such a method already exists in the form of foo-source binary packages. Changing the build dependency syntax instead would have the advantage, that we do not need more binary or source packages at all because all the necessary logic would come from the resolvers themselves. Thanks! cheers, josch signature.asc Description: signature
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, 28 Jul 2019 16:33:21 +0200, Johannes Schauer wrote: > Quoting Mo Zhou (2019-07-28 16:03:43) > > On 2019-07-28 13:13, Ian Jackson wrote: > > > This is maybe not directly helpful to you right now, but: > > > > > > If we could build-depend on source packages, you could combine these > > > two ideas into something that might be less awful. > > More than once have I thought of "what if I can B-D on a source package". > > Is such demand common enough among developers? > > recently there was some discussion about this in #debian-apt. I don't have > the logs of that discussion so others might want to expand on what I > remember from back then. In no particular order plus my own thoughts. > > apt developers are in favour of such a feature but it would need > implementation in dpkg first. This means that dpkg would have to also track > "installed" (unpacked) source packages in /usr/src in a similar way of how > it currently tracks installed binary packages. > > Tons of software that parses the Build-Depends field has to be patched. At > least: dpkg, sbuild, apt, debhelper, cdbs, pbuilder, lintian, dose3, > wanna-build, dak, devscripts, python-debian, libconfig-model-perl, augeas, > haskell-debian, dh-exec, autopkgtest... Could we avoid (some of) this by treating “source” as a new architecture? There would be a mostly do-nothing buildd which would repack the source package as a binary package in /usr/src/... That would still mean we’d have to add meaningful arch-qualified build-dependencies in all the affected tools, and it doesn’t help with the transitive dependencies (although the binary package containing the source could carry the appropriate dependencies), so perhaps it’s not such a great idea. Regards, Stephen pgpv5VyQb9Dn8.pgp Description: OpenPGP digital signature
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On 28/07/2019 20:32, Paul Wise wrote: > On Sun, Jul 28, 2019 at 10:04 PM Mo Zhou wrote: > >> Is such demand common enough among developers? > > There are several ways this would be useful: > To replace all out-of-tree Linux kernel module -source packages, so > that dkms/etc doesn't need a binary package duplicating the source. So how / where exactly would you build modules? Modify dkms to use source packages? module-assistant again? -- Tzafrir
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
Johannes Schauer writes ("Re: B-D on src package? (was: Re: Challenge from
Julia's non-standard vendored openblas"64_""):
> We have to think about a good syntax for the Build-Depends field
> which is able to express a build dependency on source packages
> unpacked to /usr/src
Can I make a radical suggestion that the package should expect the
source to be in ../src (or similar) rather than in /usr/src ?
I appreciate we usually do this stuff in buildds but there are other
environments or situations where it would be much nicer if the source
code were somewhere that could be writeable by an ordinary user.
In particular if A src-build-depends B, usually that means that A
compiles B in a funny way. To do development work properly on A, it
is necessary to be constantly editing B. Now probably A has an ad-hoc
way to point to a different B but that is fragile and annoying.
I can see few downsides from treating
Build-Depends: src:foo:src
(or whatever) to mean "src:foo" unpacked in a directory ../src/foo
(or some suitable other prefix).
(It should not mean that "foo.dsc" et al are available anywhere - just
the unpacked version, as found from dpkg-source -x.)
Ian.
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, Jul 28, 2019 at 10:33 PM Johannes Schauer wrote: > If we choose the "src:" prefix to pick source instead of binary packages, then > it's important that we don't have any binary packages called "src" and no > source packages with a name equal to a valid debian architecture. Could we have a double suffix instead to avoid these issues? Build-Depends: foo:src:src -> src:foo unpacked in /usr/src/foo Build-Depends: foo:src -> b-d of src:foo for the current host architecture Build-Depends: foo:src:amd64-> b-d of src:foo for amd64 Build-Depends: foo:src:native -> b-d of src:foo for the current build architecture -- bye, pabs https://wiki.debian.org/PaulWise
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, Jul 28, 2019 at 10:04 PM Mo Zhou wrote: > Is such demand common enough among developers? There are several ways this would be useful: To replace all librust-*-dev and golang-*-dev packages (they contain source code). To replace all toolchain -source packages, so that cross-compiling toolchains can be built from separate source packages. To replace all out-of-tree Linux kernel module -source packages, so that dkms/etc doesn't need a binary package duplicating the source. Anywhere you want to build multiple independent build configurations of the same source code in different ways. -- bye, pabs https://wiki.debian.org/PaulWise
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
Mo Zhou 于 2019年7月28日周日 16:03写道： > On 2019-07-28 13:13, Ian Jackson wrote: > > This is maybe not directly helpful to you right now, but: > > > > If we could build-depend on source packages, you could combine these > > two ideas into something that might be less awful. > > More than once have I thought of "what if I can B-D on a source > package". > Is such demand common enough among developers? > Yes, for those static linked languages, at least for Go. // send from my mobile device
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
Hi,
Quoting Mo Zhou (2019-07-28 16:03:43)
> On 2019-07-28 13:13, Ian Jackson wrote:
> > This is maybe not directly helpful to you right now, but:
> >
> > If we could build-depend on source packages, you could combine these
> > two ideas into something that might be less awful.
> More than once have I thought of "what if I can B-D on a source package". Is
> such demand common enough among developers?
recently there was some discussion about this in #debian-apt. I don't have the
logs of that discussion so others might want to expand on what I remember from
back then. In no particular order plus my own thoughts.
apt developers are in favour of such a feature but it would need implementation
in dpkg first. This means that dpkg would have to also track "installed"
(unpacked) source packages in /usr/src in a similar way of how it currently
tracks installed binary packages.
Tons of software that parses the Build-Depends field has to be patched. At
least: dpkg, sbuild, apt, debhelper, cdbs, pbuilder, lintian, dose3,
wanna-build, dak, devscripts, python-debian, libconfig-model-perl, augeas,
haskell-debian, dh-exec, autopkgtest...
We have to think about a good syntax for the Build-Depends field which is able
to express a build dependency on source packages unpacked to /usr/src as well
as a build dependency on the build dependencies of a certain source package for
a given host architecture. Some initial thoughts:
Build-Depends: src:foo:src -> src:foo unpacked in /usr/src/foo
Build-Depends: src:foo -> b-d of src:foo for the current host
architecture
Build-Depends: src:foo:amd64-> b-d of src:foo for amd64
Build-Depends: src:foo:native -> b-d of src:foo for the current build
architecture
Here, ":src" is a new suffix next to :native, :any or :${arch} indicating the
"source architecture", meaning an unpacked source package. It is also open what
a B-D on foo:src (without the src: prefix) would mean. Maybe there is a more
elegant solution.
If we choose the "src:" prefix to pick source instead of binary packages, then
it's important that we don't have any binary packages called "src" and no
source packages with a name equal to a valid debian architecture.
I think it's important to separate a dependency on the source code of src:foo
and a dependency on the build dependencies of src:foo. Sometimes one needs only
parts of src:foo and if unpacked source and its build dependencies are always
installed together, then there is no way to get one without the other.
And there has to be a syntax for how to "install" these from the command line
using apt-get.
And there is the question of whether source packages of different versions
should be installable at the same time, maybe into /usr/src/foo-${version}.
Currently we have foo-source binary packages which install their content into
/usr/src but this requires such a package to be created and there is currently
no way to get the precise build dependencies on that package.
There is also the question about build profiles. Should it be possible to only
request the build dependencies of a source package with a certain set of build
profiles active?
Are other distributions doing something similar already?
Thanks!
cheers, josch
signature.asc
Description: signature
Re: B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On Sun, Jul 28, 2019 at 07:03:43AM -0700, Mo Zhou wrote: > > This is maybe not directly helpful to you right now, but: > > > > If we could build-depend on source packages, you could combine these > > two ideas into something that might be less awful. > > More than once have I thought of "what if I can B-D on a source > package". > Is such demand common enough among developers? `apt-file search -l /usr/src` may be of interest (after you skip all kernel-related packages). -- WBR, wRAR signature.asc Description: PGP signature
B-D on src package? (was: Re: Challenge from Julia's non-standard vendored openblas"64_"
On 2019-07-28 13:13, Ian Jackson wrote: > This is maybe not directly helpful to you right now, but: > > If we could build-depend on source packages, you could combine these > two ideas into something that might be less awful. More than once have I thought of "what if I can B-D on a source package". Is such demand common enough among developers? > Anyway, good luck.
