Re: Re: Split Packages files based on new section "buildlibs"

[Peter Green]

> The same applies to the GNOME/GTK stack, where Flatpak is the way to go
> for active development. libgtk-3-dev is really only for building Debian
> packages from their point of view, too.

Perhaps, but what matters is not upstream's point of view but Debian
user's point of view.

My perception is that when C/C++ users on debian want a library to compile
stuff against their first port of call is "apt-get install libwhatever-dev".
Building C/C++ libraries from upstream sources is considered a last resort.

Whereas for rust users, noone seems to build or distribute binary libraries
and the done thing is to use cargo which automatically downloads and builds
the libraries your project needs. You have to actively go out of your way to
make cargo use rust libraries from Debian rather than those from crates.io

Upstream gnome developers may use flatpak, but I think upstream gnome
developers are a small subset of the people building stuff against gtk.

Re: Split Packages files based on new section "buildlibs"

[Shengjing Zhu]

On Wed, Feb 17, 2021 at 9:33 PM Johannes Schauer Marin Rodrigues
 wrote:
[...]
> And then I run "cargo build". Every time I get a message like:
>
> error: no matching package named `foo` found
>
> I install librust-foo-dev until finally:
>
> Parent pid 535147, child pid 535148
> Child process initialized in 30.93 ms
>Compiling proc-macro2 v1.0.18
>Compiling unicode-xid v0.2.0
>Compiling syn v1.0.12
>Compiling dotenv v0.15.0 (/tmp/dotenv/dotenv)
>Compiling quote v1.0.7
>Compiling proc-macro-hack v0.5.9
>Compiling dotenv_codegen_implementation v0.15.0 
> (/tmp/dotenv/dotenv_codegen_implementation)
>Compiling dotenv_codegen v0.15.0 (/tmp/dotenv/dotenv_codegen)
> Finished dev [unoptimized + debuginfo] target(s) in 9.93s
>
> Parent is shutting down, bye...
>
> So how is this "very difficult"? The steps are the same as when I clone a
> Python upstream git repo and I get the message "ModuleNotFoundError: No module
> named 'foo'" -- I just install python3-foo and it will work. Same here with
> rust and the librust-foo-dev packages.

I do think it's "very difficult". You end up calculating dependency
trees by hand, instead of an automatic program, like cargo, pip.
How does it come for version satisfaction? For example a library
version installed in system path conflicts the software you are
developing.

-- 
Shengjing Zhu

Re: Split Packages files based on new section "buildlibs"

[Johannes Schauer Marin Rodrigues]

Quoting Andrej Shadura (2020-11-10 10:27:44)
> On Tue, 10 Nov 2020, at 08:28, Paul Wise wrote:
> > > The current proposal is to reduce the main Packages.xz files size by
> > > splitting[4] out all of the packages that are not intended for users,
> > > writing those into an own file. Those packages would have a section of
> > > "buildlibs", independent of their other properties.
>  
> > Should (almost?) everything in the existing libdevel section move to
> > the new buildlibs section?
> 
> I wouldn’t say so.
> 
> If I install, say, libftdi-dev, I expect to be able to do actual development
> with it, for Debian or not. In fact, installing libftdi-dev would be the
> first thing I do if I were to develop with the library.
> 
> On the contrary, if I’m going to do some development with, say, clap (Rust
> command-line arguments parser), I wouldn’t install librust-clap-dev; more
> than that, if I actually did, I’d be very difficult for me to actually use it
> to develop an app.

Aha. Would it?

I have the following in my ~/.cargo/config.toml:

[source.deb]
directory = "/usr/share/cargo/registry"

[source.crates-io]
replace-with = "deb"

[net]
offline = true

Then I clone some upstream git repo that uses clap:

git clone https://github.com/dotenv-rs/dotenv.git

And then I run "cargo build". Every time I get a message like:

error: no matching package named `foo` found

I install librust-foo-dev until finally:

Parent pid 535147, child pid 535148
Child process initialized in 30.93 ms
   Compiling proc-macro2 v1.0.18
   Compiling unicode-xid v0.2.0
   Compiling syn v1.0.12
   Compiling dotenv v0.15.0 (/tmp/dotenv/dotenv)
   Compiling quote v1.0.7
   Compiling proc-macro-hack v0.5.9
   Compiling dotenv_codegen_implementation v0.15.0 
(/tmp/dotenv/dotenv_codegen_implementation)
   Compiling dotenv_codegen v0.15.0 (/tmp/dotenv/dotenv_codegen)
Finished dev [unoptimized + debuginfo] target(s) in 9.93s

Parent is shutting down, bye...

So how is this "very difficult"? The steps are the same as when I clone a
Python upstream git repo and I get the message "ModuleNotFoundError: No module
named 'foo'" -- I just install python3-foo and it will work. Same here with
rust and the librust-foo-dev packages.

I do not deny that many upstreams will tell developers to use cargo, pip, go
get... whatever to manage their software. Personally, I rather avoid using
these package managers and use the packages provided by Debian instead. There
are real advantages over using the packages from Debian. Instead of relying on
another 3rd party repository where anybody can upload anything, I benefit from
the additional work put in by DDs to make sure that no garbage ends up in the
archive. Rather than the "fast-paced" development style that seems to be modern
these days, I prefer the stability and security that I get by sourcing all my
code and libraries from the Debian archive.  With Debian packages, I cannot
really get typosquatted, I know that all software is DFSG-free and I know that
I will receive security updates.

This is not an argument against splitting "main" into several archives. I'm no
big fan of this solution but maybe it should be done. What I want to make an
argument for in this email though is, that I do not believe that our packages
have no value outside compiling other Debian packages even for languages where
upstream prefers its users to use their own package manager. There are many
reasons to prefer the trusted Debian sources for quality, security and software
freedom and if any split is done, it should not be made in a way that makes it
harder for our users to install those packages. They have real value.

At this point let me also give a big thank you to all the nodejs, python, rust
etc package maintainers for taking on the tedious task of making Debian
packages for software that is already present in another repo. It's really
great stuff -- thanks a lot!

cheers, josch

signature.asc
Description: signature

Re: Re: Split Packages files based on new section "buildlibs"

[Stephan Verbücheln]

The same applies to the GNOME/GTK stack, where Flatpak is the way to go
for active development. libgtk-3-dev is really only for building Debian
packages from their point of view, too.

But at least GNOME has scheduled releases which enable Debian stable to
maintain it, while npm, pip, gems, cargo etc. are moving targets where
packages are changing constantly.

Regards

Re: Split Packages files based on new section "buildlibs"

[Fabrice BAUZAC-STEHLY]

Hello,

(I thought I had sent this mail already, but it looks I haven't)

One of the basic things we want is that users can get the source of
their packages, modify a little, and make a new package for their own
use.  So in particular we want to be able to "apt source" and get all
the sources, in their exact versions that were used when the package was
built (pinned).

A wish we can have in the case of libraries is to separate libraries and
their lifecycles, so that e.g. if I have 10 packages relying on libc
(dynamically), one security upgrade of libc will be immediately taken
into account by the 10 packages that use it.  So in that case it's nice
to separate the library from the binary.  But Rust and Go basically use
static linking, so whatever they produce will embed the libraries.  So
it looks like this wish does not apply in the case of Rust and Go.

So let's say that we do not packages libraries as binary packages: we do
binary packages of only the applications.  We will have one binary
package built from e.g.:

Application App 1.0
depends on Library Lib1 1.2
depends on Library Lib2 1.1
depends on Library Lib3 1.0

I'm not sure what is the mechanism used in Go or Rust to "pin" the
versions of Lib1, Lib2 and Lib3, but it is necessary that if a user does
"apt source App" the correct versions of the dependencies are checked
out.  And I guess most Application release tarballs do not pin them
themselves, so it would need to be pinned somewhere else, such as inside
the debian/ directory, or in an aggregate upstream tarball that would
contain all the sources.

If we choose to make an aggregate upstream tarball, it would contain all
the sources of all the dependency tree inside one single tarball:

App-aggregate-1.2.tar.gz:
Application App 1.0 (contents of the source tarball)
Library Lib1 1.2
Library Lib2 1.1
Library Lib3 1.0

App-aggregate would be versioned wholly instead of versioning the parts.
So if we want to bump e.g. Lib3 to 1.1, we would need to make a new
version of the upstream tarball.

The issue which this aggregate is that there will probably be lots of
duplicated source code.  For example, if some library Lib1 is very
popular and used by lots of applications, we will have its source code
embedded into the source packages of all these applications, and that
would be wasteful in terms of storage of the source packages.  Can we
find a way for source packages to indicate /several/ upstream tarballs,
so that each version of the Lib1 upstream tarball is stored only once?

App_1.0-1.dsc:
  Files:
   App-1.0.orig.tar.gz
   App-1.0-1.debian.tar.xz
   Lib1-1.2.orig.tar.gz
   Lib2-1.1.orig.tar.gz
   Lib3-1.0.orig.tar.gz

--
Fabrice BAUZAC-STEHLY
PGP 015AE9B25DCB0511D200A75DE5674DEA514C891D

Re: Split Packages files based on new section "buildlibs"

[Russ Allbery]

Matthias Klose  writes:
> On 11/11/20 8:37 PM, Russ Allbery wrote:

>> Rust and Go both vendor dependencies during their build.  Python isn't
>> really analogous; you *can* do something similar with virtualenvs, but
>> (a) Python doesn't really have a build the way that Rust and Go do
>> because one generally doesn't produce native code, and (b) the
>> virtualenv approach is optional and only one of many ways of assembling
>> Python applications, whereas there's largely only one way Rust and Go
>> binaries are built.

> No, a lot of python projects do vendoring like Rust and Go.  That's
> different than a virtualenv.  Usually you have a sub-package called
> vendor or _vendor in the package namespace.

I obviously have not done anywhere near as much Python packaging as you
have, so maybe I'm just ignorant about this.  I had thought that Python
packages that did this generally distributed those packages with their
releases, so this vendoring looks like the vendoring that C, some Perl,
etc. does.  That's a somewhat older problem for which we have
long-standing approaches, and usually the vendoring is small enough scale
that we can replace the vendored versions with packaged versions unless
they've been heavily forked.

That would be a little different than what Go or Rust do, where the
dependencies aren't distributed, but instead the build system does
something akin to constructing a new virtualenv (using pinned version
numbers) with each build (and then compiling it into a single native
executable).

But maybe I'm wrong and there are Python packages that do something like
pip install dependencies into a vendor directory during their build?

-- 
Russ Allbery (r...@debian.org)  

Re: Split Packages files based on new section "buildlibs"

[Matthias Klose]

On 11/11/20 8:37 PM, Russ Allbery wrote:
> Simon McVittie  writes:
>> My understanding is that Rust and Go code literally doesn't have
>> analogous built-in system-wide search paths for third-party libraries,
>> and when building Debian packages that contain Rust and Go code, we have
>> to invent them in a Debian-specific way.
> 
> Rust and Go both vendor dependencies during their build.  Python isn't
> really analogous; you *can* do something similar with virtualenvs, but (a)
> Python doesn't really have a build the way that Rust and Go do because one
> generally doesn't produce native code, and (b) the virtualenv approach is
> optional and only one of many ways of assembling Python applications,
> whereas there's largely only one way Rust and Go binaries are built.

No, a lot of python projects do vendoring like Rust and Go.  That's different
than a virtualenv.  Usually you have a sub-package called vendor or _vendor in
the package namespace.

Matthias

Re: Split Packages files based on new section "buildlibs"

[Pirate Praveen]

On Fri, Nov 13, 2020 at 19:28, Tomas Pospisek  
wrote:
This solution seems to be too trivial that nobody would have though 
of it, so what is it that I (and I guess many Debianers) are missing?

*t


Additionally these libraries change too fast and most of the time, 
there is no long term supported releases (sometimes not even any 
releases, people just use git commits). You will be stuck at a version 
used by another package or you will have to keep changing the library 
version (and your application) as other packages change these 
dependencies.

Re: Split Packages files based on new section "buildlibs"

[Tomas Pospisek]

On 13.11.20 20:51, Wolfgang Silbermayr wrote:

[...detailed explanation of why the buildlibs proposal for Rust is 
necessary...]


Thanks a lot for the explanation Wolfgang!
*t

Re: Split Packages files based on new section "buildlibs"

[Wolfgang Silbermayr]

On 11/13/20 7:28 PM, Tomas Pospisek wrote:
> Hi Antonio (and anybody else that understands the technical problem involved
> here),
> 
> I've been reading the whole thread and it seems to me that the reason, why
> Rust/Go build-time "libraries" need to be handled differently from all the
> other existing stuff in the world and that "no user ever wants to use" the
> Debian-provided build-time Rust/Go libraries has not been spelled out in
> plain, comprehensible english yet.
> 
> So since you seem to understand a bit about the technical problem involved
> here and I'd very much appreciate if you could spell it out. I think it would
> benefit the project as then everybody would be able to understand what this
> new section is about.

Hi Tomas,

The simple explanation is probably similar to go, that in the general case the
whole source code of the project at hand and all its dependencies is compiled
into a single static binary. Exceptions are e.g. libc et al which get linked
by default (if you don't build against musl), and linked libraries that are
used through FFI (foreign function interface), e.g. GTK.

> So let me ask a question that could maybe clear things up:
> 
> On 11.11.20 14:39, Antonio Terceiro wrote:
> 
>> In the Rust world there is no such thing as installing a library
>> globably. A crate that provides a library can only be used to build
>> other stuff, and is never made available globally. "cargo install" only
>> applies to creates that provide binaries:
>>
>> https://doc.rust-lang.org/cargo/commands/cargo-install.html
> 
> [I've read the cargo-install.html document in the past but not re-read it now]
> 
> So let's say user joe wants to code Go software that depends on Go's third
> party github.com/tazjin/kontemplate/templater package ("package" in Go's
> taxonomy not in Debian's!).
> 
> Then he'd `export GOPATH=~/src/go` and `go get github.com/tazjin/kontemplate`.
> Go would then `git clone` everything under 
> `~/src/go/src/github.com/tazjin/kontemplate/`.
> 
> So far so good and I think Rust has a similar mechanisms with cargo, right?

The mechanism is similar, as you say. In Rust we have the difference that
there is effectively one single platform where the ecosystem lives, crates.io.
When all I want is to install an application that was published there, I
simply use the `cargo install ` command, and it gets compiled and
installed into ~/.cargo/bin. Therefore the sources of the dependencies and the
binary crate are downloaded from crates.io into a cache directory that is
managed by cargo, compiled from there and then installed.

> Now given that alice wants to package joe's software. She'll do the above plus
> `go get github.com/joe/joes_app`. All will be under `~/src/go/src/github`.
> 
> The naive thing to do now would be to move `~/src/go/src/` to `/usr/lib/go`
> and package that as `go-tazjin-kontemplate-dev_0.1.deb` or similar.
> 
> Debian's automatic build process for "joes_app" would first install
> `go-tazjin-kontemplate-dev_0.1.deb`, then make a symlink from
> `~/src/go/src/github.com/tazjin` (or `~/.local/go` or whereever Go expects its
> stuff by default) to `/usr/lib/go/src/github.com/tazjin` and build and be 
> done.
> 
> A user wanting to develop software based on tazjin's stuff would do the same:
> `apt-get install go-tazjin-kontemplate-dev`, symlink, done.

When we package Rust library crates into librust-*-dev binary packages, the
source code is put into the package inside a subdirectory in
/usr/share/cargo/registry which can then be used during build instead of
fetching it from crates.io. In that regard it the procedure is very similar.
However, this is not a standard directory used by cargo, but instead we have
to inject it using the /usr/share/cargo/bin/cargo wrapper during build.

If you were to develop software in Rust, you'd simply add the dependency entry
into your Cargo.toml file and let the whole mechanism of cargo do the magic.
Everything else like using the library installed in a librust-*-dev package
would just complicate and slow down your development workflow. So these
packages are effectively useless for developers that want to write Rust
software, even if they use Debian or a derivative for that.

The additional thing we have in Rust, that doesn't seem to have a strict
equivalent in other languages (yet) is features. These can be provided by
library crates, and a crate depending on it can add a dependency to it. One
popular example that got mentioned frequently. A feature is somewhat a
combination of an entity in the Cargo.toml file, and conditional compilation
in the source.

So let's assume we have a library crate supporting different database
connection types, these would be enabled by features. If a program using that
library would like to use postgres, it would enable the postgres feature of
said library.

This gets mapped to Debian packages by either providing a
librust-*+feature-dev by the default package, or by creating a new empty
metapackage 

Re: Split Packages files based on new section "buildlibs"

[Tomas Pospisek]

Hi Antonio (and anybody else that understands the technical problem 
involved here),


I've been reading the whole thread and it seems to me that the reason, 
why Rust/Go build-time "libraries" need to be handled differently from 
all the other existing stuff in the world and that "no user ever wants 
to use" the Debian-provided build-time Rust/Go libraries has not been 
spelled out in plain, comprehensible english yet.


So since you seem to understand a bit about the technical problem 
involved here and I'd very much appreciate if you could spell it out. I 
think it would benefit the project as then everybody would be able to 
understand what this new section is about.


So let me ask a question that could maybe clear things up:

On 11.11.20 14:39, Antonio Terceiro wrote:


In the Rust world there is no such thing as installing a library
globably. A crate that provides a library can only be used to build
other stuff, and is never made available globally. "cargo install" only
applies to creates that provide binaries:

https://doc.rust-lang.org/cargo/commands/cargo-install.html


[I've read the cargo-install.html document in the past but not re-read 
it now]


So let's say user joe wants to code Go software that depends on Go's 
third party github.com/tazjin/kontemplate/templater package ("package" 
in Go's taxonomy not in Debian's!).


Then he'd `export GOPATH=~/src/go` and `go get 
github.com/tazjin/kontemplate`. Go would then `git clone` everything 
under  `~/src/go/src/github.com/tazjin/kontemplate/`.


So far so good and I think Rust has a similar mechanisms with cargo, right?

Now given that alice wants to package joe's software. She'll do the 
above plus `go get github.com/joe/joes_app`. All will be under 
`~/src/go/src/github`.


The naive thing to do now would be to move `~/src/go/src/` to 
`/usr/lib/go` and package that as `go-tazjin-kontemplate-dev_0.1.deb` or 
similar.


Debian's automatic build process for "joes_app" would first install 
`go-tazjin-kontemplate-dev_0.1.deb`, then make a symlink from 
`~/src/go/src/github.com/tazjin` (or `~/.local/go` or whereever Go 
expects its stuff by default) to `/usr/lib/go/src/github.com/tazjin` and 
build and be done.


A user wanting to develop software based on tazjin's stuff would do the 
same: `apt-get install go-tazjin-kontemplate-dev`, symlink, done.


This solution seems to be too trivial that nobody would have though of 
it, so what is it that I (and I guess many Debianers) are missing?

*t

Re: Split Packages files based on new section "buildlibs"

[Andrei POPESCU]

On Ma, 10 nov 20, 10:06:24, Paul Wise wrote:
> On Tue, Nov 10, 2020 at 9:28 AM Andrej Shadura wrote:
> 
> > Development packages for Rust and Go usually only ship source code.
> 
> This reminds me of the proposal for installable source packages that
> one could (Build-)Depend on. Seems like that proposal would also solve
> the issue with Go and Rust, as well as end the need for the -source
> binary package workaround used by GCC and other packages.

It could also allow better tracking of installed Build-Depends to enable 
an equivalent of 'autoremove' to do the right thing.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser


signature.asc
Description: PGP signature

Re: Split Packages files based on new section "buildlibs"

[Russ Allbery]

Simon McVittie  writes:

> I think perhaps the key thing here is that Python does *have* a
> reasonably well-defined system-wide search path for packages outside the
> Python core (/usr/lib/python3/dist-packages). Even if some projects
> prefer to use pip instead of dist-packages, they can't claim that
> dist-packages doesn't exist.

Even more specifically, I think the critical difference is that software
written in Python (or JavaScript, or Java) consumes the packaged libraries
at runtime.  If you have a Python application packaged for Debian, it uses
the Python libraries packaged for Debian when you run it.

This is *not* true for Go or Rust because the binaries are statically
linked.  At runtime, no Go or Rust libraries are used by the binary
because they've already been linked into the binary.  (My understanding is
that both support dynamic libraries to some extent but that they're
discouraged in both and not generally used.)

I think this is the critical difference and I'm not sure it's been
explicitly spelled out (although there have been a lot of messages in this
thread, so maybe I missed it).

Because of that, the packaged Rust and Go libraries are *only* useful for
building Rust and Go binaries, not for running them, and that's why it
matters that the Go and Rust build tools that Debian users would use don't
know how to, and don't want to, use installed versions of those libraries.
And thus this point:

> My understanding is that Rust and Go code literally doesn't have
> analogous built-in system-wide search paths for third-party libraries,
> and when building Debian packages that contain Rust and Go code, we have
> to invent them in a Debian-specific way.

Rust and Go both vendor dependencies during their build.  Python isn't
really analogous; you *can* do something similar with virtualenvs, but (a)
Python doesn't really have a build the way that Rust and Go do because one
generally doesn't produce native code, and (b) the virtualenv approach is
optional and only one of many ways of assembling Python applications,
whereas there's largely only one way Rust and Go binaries are built.

-- 
Russ Allbery (r...@debian.org)  

Re: Split Packages files based on new section "buildlibs"

[Simon McVittie]

On Wed, 11 Nov 2020 at 14:26:55 +0100, Matthias Klose wrote:
> If you ask some upstreams of Python based software, their recommendation would
> be to use pip, and probably conda (a cross OS distribution focusing on Python)
> to do upstream development.  If you ask casual users, you probably will get
> another answer.
> 
> Same thing probably for Java libraries. I don't know anybody who would do
> development using the Debian packaged libraries.

I think perhaps the key thing here is that Python does *have* a reasonably
well-defined system-wide search path for packages outside the Python
core (/usr/lib/python3/dist-packages). Even if some projects prefer to
use pip instead of dist-packages, they can't claim that dist-packages
doesn't exist.

Also analogous: just because some people use LD_LIBRARY_PATH-based
mechanisms like jhbuild or the Steam Runtime for their C/C++ libraries,
that doesn't invalidate the fact that the C compiler and runtime linker
are designed to have default search paths that contain more than just
libgcc and glibc, and can have (for example) a system copy of a
third-party library like zlib or GTK.

My understanding is that Rust and Go code literally doesn't have
analogous built-in system-wide search paths for third-party libraries,
and when building Debian packages that contain Rust and Go code, we have
to invent them in a Debian-specific way.

smcv

Re: Split Packages files based on new section "buildlibs"

[Geert Stappers]

On Wed, Nov 11, 2020 at 02:26:55PM +0100, Matthias Klose wrote:
> On 11/10/20 10:51 PM, Joerg Jaspert wrote:
> > On 15948 March 1977, Paul Wise wrote:
> > 
> >> Does this include the -dev packages for C/etc libraries?
> > 
> > No.
> > 
> >> I guess it also applies to Haskell and other statically-linked languages.
> >> https://wiki.debian.org/StaticLinking
> > 
> > StaticLinking itself is not enough. This is about languages where the actual
> > development in it is discouraged from doing with the debian packaged stuff.
> > Where you do not go "I need lib XY, i install libxy-perl/libxy-dev/whatever 
> > the
> > name" and hack around using it. But "Oh, i want to hack on foo, i go get
> > foo/cargo .../whateverthetool" and the debian package only ever comes in 
> > play if
> > you do build debian packages using it.
> 
> If you ask some upstreams of Python based software, their recommendation would
> be to use pip, and probably conda (a cross OS distribution focusing on Python)
> to do upstream development.  If you ask casual users, you probably will get
> another answer.
> 
> Same thing probably for Java libraries. I don't know anybody who would do
> development using the Debian packaged libraries.
> 
> > 
> >>> The current proposal is to reduce the main Packages.xz files size by
> >>> splitting[4] out all of the packages that are not intended for users,
> >>> writing those into an own file. Those packages would have a section of
> >>> "buildlibs", independent of their other properties.
> >> Should (almost?) everything in the existing libdevel section move to
> >> the new buildlibs section?
> > 
> > No, if so we would have split that section out.
> 
> Reducing the size of the index file is a technical issue inside Debian, and
> relating that to
> 
>   """
>   languages where the actual development in it is discouraged
>   from doing with the debian packaged stuff
>   """
> 
> seems to be wrong, as any upstream eco system providing their own environment
> for development and distribution would need to move to this section.  I don't
> think the reference to upstreams doesn't help with the definition of the new
> section.


The thing we should aim for, the thing I'm aiming for,
is that software developed in any programming language
can be distributed by Debian.

User point of view:   `apt install foo`

Debian policy p.o.v.  `apt-get source foo`


 
> Matthias
> 

Regards
Geert Stappers
-- 
Silence is hard to parse


signature.asc
Description: PGP signature

Re: Split Packages files based on new section "buildlibs"

[Antonio Terceiro]

On Tue, Nov 10, 2020 at 06:52:14PM -0500, Calum McConnell wrote:
> On Tue, 2020-11-10 at 10:07 +, Simon McVittie wrote:
> > On Tue, 10 Nov 2020 at 10:45:07 +0100, Johannes Schauer wrote:
> > > I'm confused. We are packaging libraries of language X but then those
> > > packages
> > > will not be used by people who write software for language X on
> > > Debian?
> > > Intuitively, should I ever start with Rust, I would've thought that I
> > > had to
> > > install librust-clap-dev if I want to write code using "clap". What
> > > else would
> > > I install?
> > 
> > The Rust community's expectation seems to be that you would install
> > cargo,
> > and use that to download and build the clap package directly from
> > upstream,
> > without apt/dpkg being involved at all.
> 
> I don't know that that means we should abandon efforts to integrate the
> debian packages into usable code.
> 
> Rubygems has a similar expectation: however, there is a package (rubygems-
> integration) that makes it work anyways.  With Ruby, the only (user-
> obvious) difference between installing via gem and via deb is the package
> version.

FWIW, that is not accurate. There is nothing in Rubygems that prevents
it from just working. That is orthogonal to what random people who
happen to use Ruby think of Debian providing Ruby packages.

What happened was that Rubygems was created *after* we already had quite
some Ruby code in Debian that was installed differently than what
Rubygems expected, and rubygems-integration was created to bridge the
two ways of installing code. The Ruby team is slowly migrating the
old-style packages to the Rubygems layout, and at some point most, if
not all, of rubygems-integration will be obsolete.

> What about Rust makes that impossible?

In the Rust world there is no such thing as installing a library
globably. A crate that provides a library can only be used to build
other stuff, and is never made available globally. "cargo install" only
applies to creates that provide binaries:

https://doc.rust-lang.org/cargo/commands/cargo-install.html


signature.asc
Description: PGP signature

Re: Split Packages files based on new section "buildlibs"

[Matthias Klose]

On 11/10/20 10:51 PM, Joerg Jaspert wrote:
> On 15948 March 1977, Paul Wise wrote:
> 
>> Does this include the -dev packages for C/etc libraries?
> 
> No.
> 
>> I guess it also applies to Haskell and other statically-linked languages.
>> https://wiki.debian.org/StaticLinking
> 
> StaticLinking itself is not enough. This is about languages where the actual
> development in it is discouraged from doing with the debian packaged stuff.
> Where you do not go "I need lib XY, i install libxy-perl/libxy-dev/whatever 
> the
> name" and hack around using it. But "Oh, i want to hack on foo, i go get
> foo/cargo .../whateverthetool" and the debian package only ever comes in play 
> if
> you do build debian packages using it.

If you ask some upstreams of Python based software, their recommendation would
be to use pip, and probably conda (a cross OS distribution focusing on Python)
to do upstream development.  If you ask casual users, you probably will get
another answer.

Same thing probably for Java libraries. I don't know anybody who would do
development using the Debian packaged libraries.

> 
>>> The current proposal is to reduce the main Packages.xz files size by
>>> splitting[4] out all of the packages that are not intended for users,
>>> writing those into an own file. Those packages would have a section of
>>> "buildlibs", independent of their other properties.
>> Should (almost?) everything in the existing libdevel section move to
>> the new buildlibs section?
> 
> No, if so we would have split that section out.

Reducing the size of the index file is a technical issue inside Debian, and
relating that to

  """
  languages where the actual development in it is discouraged
  from doing with the debian packaged stuff
  """

seems to be wrong, as any upstream eco system providing their own environment
for development and distribution would need to move to this section.  I don't
think the reference to upstreams doesn't help with the definition of the new
section.

Matthias

Re: Split Packages files based on new section "buildlibs"

[Joerg Jaspert]

On 15949 March 1977, Thomas Goirand wrote:

I'm sorry but I don't understand everything here. If we aren't going 
to

have new components like main/contrib/non-free, how will it work? We
main contain a new Packages.{gz,xz} file containing these? I'm 
guessing

that it's not just a modification to that file, because you wrote the
goal is to diminish its size...



From my original mail:


--8<---cut here---start->8---
The current proposal is to reduce the main Packages.xz files size by
splitting[4] out all of the packages that are not intended for users,
writing those into an own file.
--8<---cut here---end--->8---

The *exact* implementation is currently fluid, a Packages-buildlibs.xz 
and a kind of subcomponent-style for main both seem options.


--
bye, Joerg

Re: Split Packages files based on new section "buildlibs"

[Thomas Goirand]

On 11/9/20 11:36 PM, Joerg Jaspert wrote:
> [4] We first thought about an entire new archive, but that is much more
>    separate, creating a higher workload on maintaining it.
>    Additionally, it would create problems following the licenses of
>    packages. Then we thought about a new component besides
>    main/contrib/non-free, and while that works better, it still has
>    many negative side effects including requiring extra package
>    uploads, extra tracking for the release team and requiring multiple
>    components if we later decide that we need to support this for all
>    of main, contrib and non-free.

Hi Joerg,

First of all, thanks for working on this, and caring for the size of the
indices, which will greatly benefit our users.

I'm sorry but I don't understand everything here. If we aren't going to
have new components like main/contrib/non-free, how will it work? We
main contain a new Packages.{gz,xz} file containing these? I'm guessing
that it's not just a modification to that file, because you wrote the
goal is to diminish its size...

Cheers,

Thomas Goirand (zigo)

Re: Split Packages files based on new section "buildlibs"

[Joerg Jaspert]

On 15949 March 1977, Calum McConnell wrote:

The Rust community's expectation seems to be that you would install 
cargo,
and use that to download and build the clap package directly from 
upstream,

without apt/dpkg being involved at all.
I don't know that that means we should abandon efforts to integrate 
the

debian packages into usable code.


[...]


There might be something about how cargo works that makes this kind of
integration impossible, similar to how the Nix package manager doesn't
play well with others.  But I don't think we should just segregate 
Rust

libraries to a corner of the archive without at least considering it.


While I do think its kind of foolish to assume the rust people in Debian 
haven't thought if its feasible to do so (or not), the best way to find 
out is talking to them. I found them quite helpful and friendly, even 
when I told them that their way is not what we want in Debian. I mean, I 
block their most preferred way from (continuing to) enter Debian, the 
rest of the team tells them similar for a longish time, so if it would 
be easy to just flip something and be "more debian like", I do think it 
would have been used.


Of course, if you find the magic, all the better.

And if that way comes around in a year or three, the proposal here makes 
it *easy* (archive side) to "switch them back into the main packages 
file".


--
bye, Joerg

Re: Split Packages files based on new section "buildlibs"

[Calum McConnell]

On Tue, 2020-11-10 at 10:07 +, Simon McVittie wrote:
> On Tue, 10 Nov 2020 at 10:45:07 +0100, Johannes Schauer wrote:
> > I'm confused. We are packaging libraries of language X but then those
> > packages
> > will not be used by people who write software for language X on
> > Debian?
> > Intuitively, should I ever start with Rust, I would've thought that I
> > had to
> > install librust-clap-dev if I want to write code using "clap". What
> > else would
> > I install?
> 
> The Rust community's expectation seems to be that you would install
> cargo,
> and use that to download and build the clap package directly from
> upstream,
> without apt/dpkg being involved at all.

I don't know that that means we should abandon efforts to integrate the
debian packages into usable code.

Rubygems has a similar expectation: however, there is a package (rubygems-
integration) that makes it work anyways.  With Ruby, the only (user-
obvious) difference between installing via gem and via deb is the package
version.  What about Rust makes that impossible?

Pip packages are the same, but they are designed that way, so I don't
count it.  NPM packages aren't quite as well integrated (or maybe they
just update too fast), but they are still used.

There might be something about how cargo works that makes this kind of
integration impossible, similar to how the Nix package manager doesn't
play well with others.  But I don't think we should just segregate Rust
libraries to a corner of the archive without at least considering it.

Thanks,
Calum M.


signature.asc
Description: This is a digitally signed message part

Re: Split Packages files based on new section "buildlibs"

[Joerg Jaspert]

On 15948 March 1977, Paul Wise wrote:


Does this include the -dev packages for C/etc libraries?


No.

I guess it also applies to Haskell and other statically-linked 
languages.

https://wiki.debian.org/StaticLinking


StaticLinking itself is not enough. This is about languages where the 
actual development in it is discouraged from doing with the debian 
packaged stuff. Where you do not go "I need lib XY, i install 
libxy-perl/libxy-dev/whatever the name" and hack around using it. But 
"Oh, i want to hack on foo, i go get foo/cargo .../whateverthetool" and 
the debian package only ever comes in play if you do build debian 
packages using it.



The current proposal is to reduce the main Packages.xz files size by
splitting[4] out all of the packages that are not intended for users,
writing those into an own file. Those packages would have a section 
of

"buildlibs", independent of their other properties.

Should (almost?) everything in the existing libdevel section move to
the new buildlibs section?


No, if so we would have split that section out.

--
bye, Joerg

Re: Split Packages files based on new section "buildlibs"

[Simon McVittie]

On Tue, 10 Nov 2020 at 10:45:07 +0100, Johannes Schauer wrote:
> I'm confused. We are packaging libraries of language X but then those packages
> will not be used by people who write software for language X on Debian?
> Intuitively, should I ever start with Rust, I would've thought that I had to
> install librust-clap-dev if I want to write code using "clap". What else would
> I install?

The Rust community's expectation seems to be that you would install cargo,
and use that to download and build the clap package directly from upstream,
without apt/dpkg being involved at all.

You would only use librust-clap-dev if you were building *a Debian package*
that build-depends on librust-clap-dev (because we have a policy that our
archive must be self-contained, so downloading dependencies using cargo is
not allowed).

smcv

Re: Split Packages files based on new section "buildlibs"

[Paul Wise]

On Tue, Nov 10, 2020 at 9:28 AM Andrej Shadura wrote:

> Development packages for Rust and Go usually only ship source code.

This reminds me of the proposal for installable source packages that
one could (Build-)Depend on. Seems like that proposal would also solve
the issue with Go and Rust, as well as end the need for the -source
binary package workaround used by GCC and other packages.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Split Packages files based on new section "buildlibs"

[Johannes Schauer]

Hi,

Quoting Andrej Shadura (2020-11-10 10:27:44)
> On Tue, 10 Nov 2020, at 08:28, Paul Wise wrote:
> > > The current proposal is to reduce the main Packages.xz files size by
> > > splitting[4] out all of the packages that are not intended for users,
> > > writing those into an own file. Those packages would have a section of
> > > "buildlibs", independent of their other properties.
>  
> > Should (almost?) everything in the existing libdevel section move to
> > the new buildlibs section?
> 
> I wouldn’t say so.
> 
> If I install, say, libftdi-dev, I expect to be able to do actual development
> with it, for Debian or not. In fact, installing libftdi-dev would be the
> first thing I do if I were to develop with the library.
> 
> On the contrary, if I’m going to do some development with, say, clap (Rust
> command-line arguments parser), I wouldn’t install librust-clap-dev; more
> than that, if I actually did, I’d be very difficult for me to actually use it
> to develop an app.

I'm confused. We are packaging libraries of language X but then those packages
will not be used by people who write software for language X on Debian?
Intuitively, should I ever start with Rust, I would've thought that I had to
install librust-clap-dev if I want to write code using "clap". What else would
I install?

If I understand the goal of "buildlibs" correctly, then it seems like quite a
bit of a misnomer. The section will not focus on libraries to build stuff but
on packages that are only ever used for Debian-internal stuff, so things I
would not even need if I were to "build" software, right?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Split Packages files based on new section "buildlibs"

[Michael Hudson-Doyle]

On Tue, 10 Nov 2020 at 20:29, Paul Wise  wrote:

> On Mon, Nov 9, 2020 at 10:37 PM Joerg Jaspert wrote:
>
> > More and more packages are being uploaded into the Debian archive which
> > are only ever used for building packages. These are not only never
> > intended to be installed onto an end-user's system, they are even
> > actively discouraged from being used directly by a user. The two
> > currently most notable examples are packages used by the Go and Rust
> > programming languages and their ecosystem, but there well may be
> > others[1].
>
> Does this include the -dev packages for C/etc libraries?
>

No, those are useful for people writing C programs outside of packaging.


> I guess it also applies to Haskell and other statically-linked languages.
>
> https://wiki.debian.org/StaticLinking


It's not the static linking that's the issue, it's that go (and rust I
assume) packages do not install things on the default search path of the
compiler. I don't know whether haskell does or not. The -dev packages for C
libraries definitely do!

> The current proposal is to reduce the main Packages.xz files size by
> > splitting[4] out all of the packages that are not intended for users,
> > writing those into an own file. Those packages would have a section of
> > "buildlibs", independent of their other properties.
>
> Should (almost?) everything in the existing libdevel section move to
> the new buildlibs section?
>

I don't think so.

Cheers,
mwh

Re: Split Packages files based on new section "buildlibs"

[Paul Wise]

On Mon, Nov 9, 2020 at 10:37 PM Joerg Jaspert wrote:

> More and more packages are being uploaded into the Debian archive which
> are only ever used for building packages. These are not only never
> intended to be installed onto an end-user's system, they are even
> actively discouraged from being used directly by a user. The two
> currently most notable examples are packages used by the Go and Rust
> programming languages and their ecosystem, but there well may be
> others[1].

Does this include the -dev packages for C/etc libraries?

I guess it also applies to Haskell and other statically-linked languages.

https://wiki.debian.org/StaticLinking

> The current proposal is to reduce the main Packages.xz files size by
> splitting[4] out all of the packages that are not intended for users,
> writing those into an own file. Those packages would have a section of
> "buildlibs", independent of their other properties.

Should (almost?) everything in the existing libdevel section move to
the new buildlibs section?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Split Packages files based on new section "buildlibs"

[Joerg Jaspert]

Hi everybody,

Short Reason: Too many packages of no use to our users.

Longer reason: Many packages get added to Debian that are of no (direct)
use to our users. Each package adds metadata to the indices that needs
to be downloaded, processed by tools and also clutters up the whole
package list for no practical benefit. A split out packages file will
allow us to minimise the effect on users.

More and more packages are being uploaded into the Debian archive which
are only ever used for building packages. These are not only never
intended to be installed onto an end-user's system, they are even
actively discouraged from being used directly by a user. The two
currently most notable examples are packages used by the Go and Rust
programming languages and their ecosystem, but there well may be
others[1]. While we need their library packages to build the
applications that use them, they are entirely statically compiled and
none of the libraries will ever be installed on a normal user's system.

Moreover, the language ecosystem in Debian actively discourages users
from installing them for anything other than rebuilding a Debian
package. If you do general (non-Debian-specific) development using Go or
Rust on your machine, the expectation is that you will use the
language-specific tools to install your dependencies [2].

Currently however, all of those packages end up in the indices we
generate, which users have to download and package managers have to read
and deal with. Each of those packages therefore slightly increases the
size of these indices for little reason and while many users have access
to large bandwidth connections and fast CPUs, that is not the case for
many other users and does not benefit global warming.

For the Rust ecosystem, those sizes increase even more, as each of their
libraries can provide multiple features. For example, a TLS library can
link against GnuTLS or OpenSSL or some other random TLS implementation.
Such features may even be combined in various different ways, resulting
in an excess number of possible feature combinations for one Rust
package, called "crate". Those are "mapped" to the Debian package world
by creating something we call *feature packages*, with one such feature
package per feature or combination thereof (usually grouped by common
dependencies).

Those feature packages are empty packages, only containing a symlink for
their /usr/share/doc/… directory. Their size is smaller than the
metadata they will produce. Adding new features means one more trip
through the NEW queue each time such new binary packages are introduced.

The FTPTeam disagree with the feature-package solution[3], so currently
there is a workaround. By collapsing the features into the main library
package and declaring the features using the Provides header similar
functionality is achieved. However this doesn’t work in all situations,
for example:

   Tools can generate really long Provides: lines, with the current
   record being around 250kb. That's long enough that a tool (not dak
   itself) broke on it already. And those lines may grow larger in
   future.

   Some features may need different (build-)dependencies (say, GnuTLS
   vs OpenSSL), should those conflict with each other, you cannot
   combine them into one package and must fall back to the feature
   package solution.

   Generally, the workaround involves changing upstream's dependency
   structure in order to fit it into the aforementioned Debian
   constraints, and so of course this may not always play nicely with
   other packages that expect the unchanged upstream dependency
   structure. The feature-package solution is a 1-to-1 mapping.

There have been multiple discussions between the FTPTeam and the Rust
package maintainers. The FTPTeam does not want those feature packages in
the part of main downloaded by users and currently rejects them from
NEW, while the Rust maintainers see them as needed and the workaround as
just that. Both sides agree that this is not a productive and
sustainable solution and that we need to agree on something better.

The current proposal is to reduce the main Packages.xz files size by
splitting[4] out all of the packages that are not intended for users,
writing those into an own file. Those packages would have a section of
"buildlibs", independent of their other properties. That section should
only be activated on buildds and in situations that need
build-dependencies available (say, an archive rebuild, a user rebuilding
packages that need Build-Dependencies from there), but not by default
anywhere else. This section will allow feature packages and *may* even
let them bypass binary-NEW if they only add new feature (empty)
packages.

The exact way of how this gets implemented, both in dak and also apt, is
still being discussed between the ftpteam and the apt maintainers. We
have ideas from writing out section based packages files to presenting
it as a subcomponent to main, and we think we will have