Re: "dev" ports (=/= "devel" ports!)
On Thursday April 27 2017 11:37:41 Rainer Müller wrote: >No, this is not true at all. You can install a .deb or .rpm without >having a corresponding local index. In case of Debian, installing >packages is even fully separated in two tools (dpkg and apt). You're right, I forgot about standalone .debs. Regardless, I *think* that the apt system calls onto dpkg to handle the ultimate installation. >>> On installing, port would first check for a binary archive and if >>> it exists, it is downloaded and installed as usual. Only if it >>> does >> >> That's already what happens, no? > >No, not as far as I understand it. As you sketched it, you would first >have to download foo, which then contains the tarball for foo-dev. I meant the way MP works currently, not my -dev implementation. The latter indeed provides the install "media" for foo-dev in the tarball for foo. Which is nothing other than an easy implementation of keeping part of the port deactivated by default. >That means, if I am only installing foo, it will waste bandwidth and >disk space for foo-dev, even when I am never going to install foo-dev. >Isn't the idea of splitting to avoid that? That's one of the ideas, yes. But also a consideration that MacPorts has never really made. My main reason here was avoiding build conflicts without having to deactivate an entire port which can have far-fetching implications. On Linux for example I had to do an annoying dance deactivating port:gettext (and git, for some reason) in order to build Qt, which meant I couldn't use many of my installed ports. Now, I can just de- and re-activate gettext-dev when needed. >If your only goal is to activate files separately, but still ship all in >the same pre-compiled archive, why not modify the activate phase to >support that? That would sound a lot simpler. That would be another solution, but that would also mean modifying the depspec mechanism one way or another. >> advantage for the llvm/clang ports to use a build once, bundle twice >> approach too and there my current approach might be less >> appropriate. > >That is actually the better use case than creating separate -dev >subports in the way of Debian/Ubuntu. Depends on the hat you're wearing. From a developers' (but also distro maintainer's) point of view the possibilities provided by installing runtime and development payload separately are probably more useful than gain of disk space. And the advantages of creating multiple packages from a single build disappear for end-users when you can simply install binary builds. > >As soon as some dependent needs to be built from source, we would still >have to specify every single -dev subport in its dependencies... Yes, dependent ports will need to adapt their depspecs. I don't see this as something that ought to be used indiscriminately. A good example where this feature could (have been) useful is port:ffmpeg. There's still at least 1 port that doesn't work with ffmpeg 3 (VLC), which now provides a private ffmpeg 2.8 subport . With a -dev split it would have been straightforward to implement a more or less temporary port:ffmpeg2 ; port:ffmpeg2-dev would have conflicted with port:ffmpeg-dev, port:VLC would have had `depends_build-append port:ffmpeg2-dev ; depends_lib-append port:ffmpeg2`. Much easier and quicker to implement than the hoops I had to jump through to get the current implementation to work. R.
Re: "dev" ports (=/= "devel" ports!)
On 2017-04-26 19:11, René J.V. Bertin wrote: > On Wednesday April 26 2017 17:32:58 Rainer Müller wrote: > >> source, but skip the compilation with pre-compiled archives. These >> are not real packages, as they cannot be installed standalone >> without a source ports tree. > > Hmm? You mean the Portfiles? That's not really different from Debian > and presumably RPM-based systems, which also have a local database > that tells them which packages are available (and that usually allow > to install from source too). No, this is not true at all. You can install a .deb or .rpm without having a corresponding local index. In case of Debian, installing packages is even fully separated in two tools (dpkg and apt). > One fundamental difference with Debuntu and RPM packaging (AFAIK) is > that those always involve the binary package (.deb or .rpm file). > Even if you build a package from source that is part of a source > package generating 10 other packages you'll end up with all those > .deb or .rpm files somewhere, but install only that single package > plus its dependencies. That is it exactly. This is the way it should work. >> On installing, port would first check for a binary archive and if >> it exists, it is downloaded and installed as usual. Only if it >> does > > That's already what happens, no? No, not as far as I understand it. As you sketched it, you would first have to download foo, which then contains the tarball for foo-dev. That means, if I am only installing foo, it will waste bandwidth and disk space for foo-dev, even when I am never going to install foo-dev. Isn't the idea of splitting to avoid that? If your only goal is to activate files separately, but still ship all in the same pre-compiled archive, why not modify the activate phase to support that? That would sound a lot simpler. > Evidently there are other applications for split ports besides not > installing development content unless it's needed. There could be an > advantage for the llvm/clang ports to use a build once, bundle twice > approach too and there my current approach might be less > appropriate. That is actually the better use case than creating separate -dev subports in the way of Debian/Ubuntu. As soon as some dependent needs to be built from source, we would still have to specify every single -dev subport in its dependencies... Rainer
Re: "dev" ports (=/= "devel" ports!)
On Wednesday April 26 2017 17:32:58 Rainer Müller wrote:
> source, but skip the compilation with pre-compiled archives. These are
> not real packages, as they cannot be installed standalone without a
> source ports tree.
Hmm? You mean the Portfiles? That's not really different from Debian and
presumably RPM-based systems, which also have a local database that tells them
which packages are available (and that usually allow to install from source
too).
One fundamental difference with Debuntu and RPM packaging (AFAIK) is that those
always involve the binary package (.deb or .rpm file). Even if you build a
package from source that is part of a source package generating 10 other
packages you'll end up with all those .deb or .rpm files somewhere, but install
only that single package plus its dependencies.
> I do not understand how your provided Port Group is supposed to work,
> because 'create_devport_content_archive' is not called anywhere.
Well, that's because even for "dev" content it's a bit too port-specific what
qualifies so the ultimate archive creation step has to be made explicitly in
the Portfile. For instance in my Linux version of port:tiff :
{{{
post-destroot {
if {${subport} eq "${name}"} {
register_devport_standard_content
devport_content-append ${prefix}/share/doc/tiff/html
devport_content-append ${prefix}/share/man/man3
create_devport_content_archive
}
}
}}}
The register_devport_standard_content step could indeed be done in the
PortGroup, though that would mean that the port cannot modify things in that
standard content in its post-destroot .
> subport. This is a bad hack and is way too fragile, as de-/activate on
> the subport will inherently be broken, unless the main port is active.
>
Why? Installing from the custom tarball uses the same kind of install mechanism
that ports use that install content not needing any building: the file tree is
created under $destroot in the destroot step, which is then installed as usual.
Believe me I've tested the subport de/activation as that was the whole point of
the endeavour. Being able to avoid build conflicts by (temporarily) removing
just the files that could affect a build, not the runtime files.
That's also why I can remove the custom tarball once the -dev port is
installed: it isn't needed anymore (and can be regenerated by de/re-activating
the main port).
> The way this is done with other systems is to destroot once, then put
> files as specified into separate packages. Each resulting package can
> then be installed just as any other package.
The only difference here is that I'm not (yet) creating anything that looks
like official packages, but put install certain files in latent fashion.
Which reminds me that another way to achieve this particular goal would be to
move (hide) those files without archiving them, and then set up symlinks as
appropriate in the -dev port's destroot phase.
> The way this could work in MacPorts would be to have a new phase for
> packaging after destroot and create multiple binary archives from one
> Portfile. ...
> Since
> everything will need to be built anyway, we can install and register all
> packages in the local system, but only activate the requested subport.
Maybe we're thinking along the same lines, but what I'd do is modify the
existing phase that comes after the post-destroot. Currently that involves
installing ${subport} from everything that is found under $destroot . It should
be relatively straightforward to extend this to support multiple destroot
directories. To stick with the -dev port example, you'd
- associate port:foo-dev (${subport}==${name}-dev) with "${destroot}-dev"
- during the usual install, MacPorts now installs the $destroot dir as ${name}
(port:foo) and ${destroot}-dev as ${name}-dev (port:foo-dev).
- Of those two, only the one(s) of which an install or upgrade was requested
(or that were active before) are activated; the other(s) are not.
There may be side-effects here that I am overlooking and I don't know if it is
any simpler implementation-wise, but it does seem to fit in well with the
traditional MacPorts approach of just installing everything.
It's basically adding install labels that allow you to de/activate a port in
one or more sections. That part will probably indeed have to be done in Base.
> On installing, port would first check for a binary archive and
> if it exists, it is downloaded and installed as usual. Only if it does
That's already what happens, no?
> Instead of developing this in a port group, it seems much easier to me
> to do that in base, especially as this would become a new type of port
> besides the existing subports.
Or as I said, a new property that determines from what directory (sub)ports are
assembled and whether they build as standalone ports.
Evidently there are other applications for split ports besides not installing
development content unless it's needed. There
Re: "dev" ports (=/= "devel" ports!)
On 2017-04-26 14:15, René J.V. Bertin wrote: >> I really really don't think a PortGroup is the right way to do >> this. This should be done in base, with proper support of creating >> multiple packages from a single build and then optionally >> installing some of them, more like FreeBSD's ports does things. > > As often I use a PortGroup only because it's a much more convenient > way to develop and test new functionality, esp. the part that cannot > be automated reliably anyway (indicating which files and directories > have to go into the -dev port, or any number of "sub" ports). > > We don't know how many ports would be using the feature, so why spend > time and effort making possibly complex changes to Base until we know > that? In my opinion splits in the way they are done on Debian/Ubuntu or with RPM distros makes a lot of sense for binary-only distributions. Ports system work in a different way and usually do not implement such a feature. At the moment, MacPorts is kind of a mix. You can install from source, but skip the compilation with pre-compiled archives. These are not real packages, as they cannot be installed standalone without a source ports tree. I do not understand how your provided Port Group is supposed to work, because 'create_devport_content_archive' is not called anywhere. But if I get it right, you want the main port to ship tarballs that contain the files that will only be activated when installing the subport. This is a bad hack and is way too fragile, as de-/activate on the subport will inherently be broken, unless the main port is active. The way this is done with other systems is to destroot once, then put files as specified into separate packages. Each resulting package can then be installed just as any other package. The way this could work in MacPorts would be to have a new phase for packaging after destroot and create multiple binary archives from one Portfile. On installing, port would first check for a binary archive and if it exists, it is downloaded and installed as usual. Only if it does not exist, we need to do build/destroot/package the parent port. Since everything will need to be built anyway, we can install and register all packages in the local system, but only activate the requested subport. Instead of developing this in a port group, it seems much easier to me to do that in base, especially as this would become a new type of port besides the existing subports. For the syntax, I would recommend to look at the way RPM allows to define splits/sub-packages. > There's also the point that ports using this particular new feature > would impose users to upgrade if it's implemented in Base. I don't > know if that'd be a precedent, but wouldn't like it if it is. I'd > prefer to see Base as a kind of microkernel with a modular > architecture (but that's just me). New features have always been part of base. What you want to do is something base was not designed for, so the framework needs to be extended. Rainer
Re: "dev" ports (=/= "devel" ports!)
On Wednesday April 26 2017 13:32:47 Clemens Lang wrote: Hi, > I really really don't think a PortGroup is the right way to do this. This > should > be done in base, with proper support of creating multiple packages from a > single > build and then optionally installing some of them, more like FreeBSD's ports > does things. As often I use a PortGroup only because it's a much more convenient way to develop and test new functionality, esp. the part that cannot be automated reliably anyway (indicating which files and directories have to go into the -dev port, or any number of "sub" ports). We don't know how many ports would be using the feature, so why spend time and effort making possibly complex changes to Base until we know that? There's also the point that ports using this particular new feature would impose users to upgrade if it's implemented in Base. I don't know if that'd be a precedent, but wouldn't like it if it is. I'd prefer to see Base as a kind of microkernel with a modular architecture (but that's just me). R
Re: "dev" ports (=/= "devel" ports!)
On Apr 23 15:54:07, rjvber...@gmail.com wrote: > I'd like to draw some attention to a prototype implementation of a PortGroup > I wrote for creating "dev" ports, akin to Debian/Ubuntu's "-dev" packages: > https://trac.macports.org/ticket/52713 > I've been using this for a while now and find it particularly useful for > avoiding build conflicts. That kind of conflict usually arises when the > presence of certain files from a port A interferes with building port B. Most > often that concerns headerfiles but "linker interface" libraries can also be > at cause, or even pkgconfig/cmake files. Can you please describe the motivating situation in detail, i.e. what header files were conflicting with what? Also, I am not entirely clear on what you mean by "link libraries" (as opposed to "libraries"). I remember from my Debian days (long ago) how strange this felt. I have installed, say, libpng. So there is /usr/local/libpng.so and the programs that need png run hapilly. But I cannot write any program that uses the library, because? There is no png.h; I need to install a separate libpng-dev for that. Please don't. Or at least only in very specific cases that need it. Even if one library's header file(s) conflict with other library's header file (example?), the solution is imho not to not install the header files. Have one port, say 'png', that installs the library, the headers, and the manpages. Please don't go the Debian way where a modest workstation needs to have hundreds of packages installed. > Those are all files that are only required when using port "A" > as a dependency for *building* other software but not as a dependency > for running that dependent software. The keyword here is "only". It is not unusual to install a library (and its header files, and its pkgconfig files, and its documentation) because you want to develop software using that library. Please don't make it needlessly complicated by requiring more than one port to be installed. I admit there are packeges where a separate -doc might be in order (as a subport? or a variant? is there an agreement?) if the documentation is considerably bigger than the library itself, or is just a copy of a set of html pages to be found on the upstream's homepage. > I think that it should even be possible to use this kind of approach for > ports that cannot be built when an older version is already installed; such > ports could probably declare a build conflict with their own -dev port > (possibly even deactivate it in an appropriate stage). That's the kind of mess I'm talking about. "Declare a build conflict with you own -dev port." Please don't. Jan
"dev" ports (=/= "devel" ports!)
Hi,
I'd like to draw some attention to a prototype implementation of a PortGroup I
wrote for creating "dev" ports, akin to Debian/Ubuntu's "-dev" packages:
https://trac.macports.org/ticket/52713
I've been using this for a while now and find it particularly useful for
avoiding build conflicts. That kind of conflict usually arises when the
presence of certain files from a port A interferes with building port B. Most
often that concerns headerfiles but "linker interface" libraries can also be at
cause, or even pkgconfig/cmake files. Those are all files that are only
required when using port "A" as a dependency for *building* other software but
not as a dependency for running that dependent software.
I think that it should even be possible to use this kind of approach for ports
that cannot be built when an older version is already installed; such ports
could probably declare a build conflict with their own -dev port (possibly even
deactivate it in an appropriate stage).
The current implementation moves selected files into a "fake" port image in an
appropriate location during the main port's post-destroot; unpacks that tarball
during the -dev port's destroot and zeroes the "fake" port image during the
pre-activate because it is now unnecessary. Rather than being emptied that
tarball could be replaced with a symlink to the actual software image (but how
to get its name?). Or it could be completely removed if rev-upgrade can be told
not to complain about certain missing files.
A more elegant version would install that dev port in deactivated fashion when
the main port is installed. The basic implementation shouldn't be too difficult
(move the selected files into a second DESTDIR, e.g. ${destroot_dev}, use that
as the root for installing but not activating ${subport}-dev) but it might be
tricky to get right with upgrades etc.
My current simpler implementation via a PortGroup has the advantage that it can
help assess how much use this kind of feature would see, and thus how much
effort should be put into it.
R.
