Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 07/04/2015 01:24 PM, William Hubbs wrote: On Sat, Jul 04, 2015 at 12:43:37PM -0700, Zac Medico wrote: On 07/04/2015 12:32 PM, William Hubbs wrote: On Sat, Jul 04, 2015 at 12:19:28PM -0700, Zac Medico wrote: On 06/30/2015 03:08 PM, William Hubbs wrote: The source code is where the compatibility between versions of Go is, not the static objects, so what if, for third-party go packages, we skip installing the static objects? The only down side of this would be that there might be longer rebuilds if the packages have multiple consumers, but it gets rid of the static objects. What do you think? I'll give real example involving go-tools. The go-tools build requires go-net, which in turn requires go-text. If the go-net *.a files are installed, then it is possible to build go-tools against go-net without having go-text installed. If the go-net *.a files are not installed, then you will have to install go-text before you can build go-tools. It introduces an indirect build-time dependency between go-tools and go-text. Sure, but what I'm proposing is that we do not install any *.a files for Go software that is not part of dev-lang/go. Exactly the same type of situation can arise for packages that are not part of dev-lang/go. For example, if consul's static api.a library is not installed, then it will introduce indirect build-time dependencies for the consul-template package. Hmm, I haven't looked at either consul or consul-template yet, but I'm thinking that if you use golang-build.eclass to install everything and make sure GOPATH is set correctly, consul-template will pick up everything it needs. What am I missing? You need to recognize that build-time for package A is not the same as build-time for package B. When you build go-tools, you can't rely on the build-time dependencies of go-net being present. Likewise, you can't rely on the build-time dependencies of consul being present when consul-template is built. -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 07/04/2015 12:32 PM, William Hubbs wrote: On Sat, Jul 04, 2015 at 12:19:28PM -0700, Zac Medico wrote: On 06/30/2015 03:08 PM, William Hubbs wrote: The source code is where the compatibility between versions of Go is, not the static objects, so what if, for third-party go packages, we skip installing the static objects? The only down side of this would be that there might be longer rebuilds if the packages have multiple consumers, but it gets rid of the static objects. What do you think? I'll give real example involving go-tools. The go-tools build requires go-net, which in turn requires go-text. If the go-net *.a files are installed, then it is possible to build go-tools against go-net without having go-text installed. If the go-net *.a files are not installed, then you will have to install go-text before you can build go-tools. It introduces an indirect build-time dependency between go-tools and go-text. Sure, but what I'm proposing is that we do not install any *.a files for Go software that is not part of dev-lang/go. Exactly the same type of situation can arise for packages that are not part of dev-lang/go. For example, if consul's static api.a library is not installed, then it will introduce indirect build-time dependencies for the consul-template package. -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On Sat, Jul 04, 2015 at 12:43:37PM -0700, Zac Medico wrote: On 07/04/2015 12:32 PM, William Hubbs wrote: On Sat, Jul 04, 2015 at 12:19:28PM -0700, Zac Medico wrote: On 06/30/2015 03:08 PM, William Hubbs wrote: The source code is where the compatibility between versions of Go is, not the static objects, so what if, for third-party go packages, we skip installing the static objects? The only down side of this would be that there might be longer rebuilds if the packages have multiple consumers, but it gets rid of the static objects. What do you think? I'll give real example involving go-tools. The go-tools build requires go-net, which in turn requires go-text. If the go-net *.a files are installed, then it is possible to build go-tools against go-net without having go-text installed. If the go-net *.a files are not installed, then you will have to install go-text before you can build go-tools. It introduces an indirect build-time dependency between go-tools and go-text. Sure, but what I'm proposing is that we do not install any *.a files for Go software that is not part of dev-lang/go. Exactly the same type of situation can arise for packages that are not part of dev-lang/go. For example, if consul's static api.a library is not installed, then it will introduce indirect build-time dependencies for the consul-template package. Hmm, I haven't looked at either consul or consul-template yet, but I'm thinking that if you use golang-build.eclass to install everything and make sure GOPATH is set correctly, consul-template will pick up everything it needs. What am I missing? William signature.asc Description: Digital signature
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/30/2015 03:08 PM, William Hubbs wrote: The source code is where the compatibility between versions of Go is, not the static objects, so what if, for third-party go packages, we skip installing the static objects? The only down side of this would be that there might be longer rebuilds if the packages have multiple consumers, but it gets rid of the static objects. What do you think? I'll give real example involving go-tools. The go-tools build requires go-net, which in turn requires go-text. If the go-net *.a files are installed, then it is possible to build go-tools against go-net without having go-text installed. If the go-net *.a files are not installed, then you will have to install go-text before you can build go-tools. It introduces an indirect build-time dependency between go-tools and go-text. -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On Sat, Jul 04, 2015 at 12:19:28PM -0700, Zac Medico wrote: On 06/30/2015 03:08 PM, William Hubbs wrote: The source code is where the compatibility between versions of Go is, not the static objects, so what if, for third-party go packages, we skip installing the static objects? The only down side of this would be that there might be longer rebuilds if the packages have multiple consumers, but it gets rid of the static objects. What do you think? I'll give real example involving go-tools. The go-tools build requires go-net, which in turn requires go-text. If the go-net *.a files are installed, then it is possible to build go-tools against go-net without having go-text installed. If the go-net *.a files are not installed, then you will have to install go-text before you can build go-tools. It introduces an indirect build-time dependency between go-tools and go-text. Sure, but what I'm proposing is that we do not install any *.a files for Go software that is not part of dev-lang/go. William signature.asc Description: Digital signature
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 07/04/2015 01:33 PM, Zac Medico wrote: On 07/04/2015 01:24 PM, William Hubbs wrote: What am I missing? You need to recognize that build-time for package A is not the same as build-time for package B. When you build go-tools, you can't rely on the build-time dependencies of go-net being present. Likewise, you can't rely on the build-time dependencies of consul being present when consul-template is built. For example, you should be able to build go-net, unistall go-text, and then build go-tools without go-text installed. Likewise, you should be able to build consul, uninstall go-crypto, and then build consul-template without go-crypto installed. -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On Tue, Jun 30, 2015 at 04:48:29PM -0700, Zac Medico wrote: On 06/30/2015 03:08 PM, William Hubbs wrote: Thinking about this, there may be a third option. This would take a slight reworking of the golang-build.eclass, but that is easy to do, and it would possibly remove the subslot from the dependencies. The source code is where the compatibility between versions of Go is, not the static objects, so what if, for third-party go packages, we skip installing the static objects? If we did this with consul, for example, then the source code for all those libraries (that have no other consumers) would have to be installed in order to build consul-template against the consul's api library. It would be similar to a header dependency. This would necessitate the introduction of build-against dependencies [1], or equivalent virtuals (like virtual/podofo-build). How is this different from DEPEND=dev-go/podofo for example or DEPEND==dev-go/fodofo-0_pre? The only down side of this would be that there might be longer rebuilds if the packages have multiple consumers, but it gets rid of the static objects. What do you think? Considering the similarity to header dependencies, I don't know. The subslot thing seems slightly more appealing to me. I got the idea of not installing the objects from Debian's description of how they do this [1]; they do not mention installing the objects. Let me know what you think. William [1] http://pkg-go.alioth.debian.org/packaging.html signature.asc Description: Digital signature
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/30/2015 07:01 PM, William Hubbs wrote: On Tue, Jun 30, 2015 at 04:48:29PM -0700, Zac Medico wrote: On 06/30/2015 03:08 PM, William Hubbs wrote: Thinking about this, there may be a third option. This would take a slight reworking of the golang-build.eclass, but that is easy to do, and it would possibly remove the subslot from the dependencies. The source code is where the compatibility between versions of Go is, not the static objects, so what if, for third-party go packages, we skip installing the static objects? If we did this with consul, for example, then the source code for all those libraries (that have no other consumers) would have to be installed in order to build consul-template against the consul's api library. It would be similar to a header dependency. This would necessitate the introduction of build-against dependencies [1], or equivalent virtuals (like virtual/podofo-build). How is this different from DEPEND=dev-go/podofo for example or DEPEND==dev-go/fodofo-0_pre? The virtual/podofo-build package pulls in dev-libs/boost, since packages which build against podofo will fail to build unless the boost headers are installed. Since dev-libs/boost is not a run-time dependency of podofo, and it's not a direct build-time dependency of packages that build against podofo, we pull it in via virtual/podofo-build. If we install Go source files without the corresponding static libraries, they we create a similar situation to the above. For example, if consul doesn't install its static api library, then anything that wants to build against that library is going to need indirect dependencies installed in order to build that library. The indirect dependencies are not needed if there is an installed instance of consul's static api library. The only down side of this would be that there might be longer rebuilds if the packages have multiple consumers, but it gets rid of the static objects. What do you think? Considering the similarity to header dependencies, I don't know. The subslot thing seems slightly more appealing to me. I got the idea of not installing the objects from Debian's description of how they do this [1]; they do not mention installing the objects. Let me know what you think. William [1] http://pkg-go.alioth.debian.org/packaging.html As I understand it, debian does the equivalent of putting the Go dependencies in both DEPEND and RDEPEND. This means that users are forced to keep build-time dependencies around after they are no longer needed. -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 11:25 PM, Zac Medico wrote: Considering that Go binaries are statically linked, you'll end up with a bunch of Go libraries installed that you don't need during run-time. They'll eventually give this up, because everyone does when their language starts seeing serious use. I won't pretend that's a real argument though. Suppose ten years from now everything is written in Go. I have 500 statically linked Go packages on my system, all of whose dependencies were built and compiled-in at install time. Now someone finds a remote root vulnerability in the go-openssl library. I know some of the packages I have installed were built against it. What do I do? At least with the useless dev-go/go-openssl installed, I can use subslots to rebuild everything after an upgrade to the fixed version.
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
All, we have digressed a bit, so I want to bring the discussion back to what my main concerns are about this issue. 1. Should we bundle Go packages with Go software? If we do, except for the Go standard library which is part of dev-lang/go, do we need to bother with installing Go sources and packages at all? The down side of the whole bundling idea is that every consumer on someone's system could potentially have a different version of the Go package, which doesn't lend itself well to security concerns. This is why bundling is generally discouraged in Gentoo. Also, if we bundle, most of dev-go/* doesn't need to exist because these libraries would be bundled into and statically linked into the software that needs them. 2. How should we bundle? This is where my concern about consul and some other ebuilds comes in. The way the consul ebuild is written (putting the commit hashes of dependencies in SRC_URI) assumes that all of the dependencies will stay on github. This makes the ebuild far less flexable than go itself is. If we are going to bundle, I would rather have one tarball that includes all of the sources for consul and the dependent libraries dropped on the Gentoo mirrors. Such a tarball is very easy to create. Thoughts? William signature.asc Description: Digital signature
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/30/2015 08:49 AM, Michael Orlitzky wrote: On 06/29/2015 11:25 PM, Zac Medico wrote: Considering that Go binaries are statically linked, you'll end up with a bunch of Go libraries installed that you don't need during run-time. They'll eventually give this up, because everyone does when their language starts seeing serious use. I won't pretend that's a real argument though. Yeah, we'll see. We need to deal with the current version of reality though... Suppose ten years from now everything is written in Go. I have 500 statically linked Go packages on my system, all of whose dependencies were built and compiled-in at install time. Now someone finds a remote root vulnerability in the go-openssl library. I know some of the packages I have installed were built against it. What do I do? Use slot-operator := deps, together with the emerge --with-bdeps=y option. Then, if you bump the sub-slot of the go-openssl library, all of your go packages that have it in DEPEND with a slot-operator := dependency will be rebuilt automatically. At least with the useless dev-go/go-openssl installed, I can use subslots to rebuild everything after an upgrade to the fixed version. As I mentioned in my reply to William [1], we might invent a notion of having one ebuild execute another ebuild in order to install static dependencies into a temporary build directory. That way, static libraries would be built on-demand, and discarded as soon as possible. [1] https://archives.gentoo.org/gentoo-dev/message/4b150fe36bf9e0ba1eb29b1d695a3193 -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/30/2015 11:25 AM, Michael Orlitzky wrote: On 06/30/2015 02:12 PM, Zac Medico wrote: Suppose ten years from now everything is written in Go. I have 500 statically linked Go packages on my system, all of whose dependencies were built and compiled-in at install time. Now someone finds a remote root vulnerability in the go-openssl library. I know some of the packages I have installed were built against it. What do I do? Use slot-operator := deps, together with the emerge --with-bdeps=y option. Then, if you bump the sub-slot of the go-openssl library, all of your go packages that have it in DEPEND with a slot-operator := dependency will be rebuilt automatically. Right, and now what if go-openssl was built on-the-fly 500 times and there's no package for it? Yeah that's obviously sub-optimal, and it's the reason why I created the dev-go/* ebuilds. However, we may want to distinguish between libraries that would only have a single consumer and libraries that would have multiple consumers. Using the same rules regardless of the number of consumers is not necessarily optimal. -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/30/2015 11:12 AM, Zac Medico wrote: As I mentioned in my reply to William [1], we might invent a notion of having one ebuild execute another ebuild in order to install static dependencies into a temporary build directory. That way, static libraries would be built on-demand, and discarded as soon as possible. [1] https://archives.gentoo.org/gentoo-dev/message/4b150fe36bf9e0ba1eb29b1d695a3193 I should note that I'm not very fond of this idea. If the dependencies have separate ebuilds (like dev-go/*), then you can already use something like 'emerge --depclean --with-bdeps=n' to remove the static libraries that aren't needed at run-time. -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/30/2015 02:12 PM, Zac Medico wrote: Suppose ten years from now everything is written in Go. I have 500 statically linked Go packages on my system, all of whose dependencies were built and compiled-in at install time. Now someone finds a remote root vulnerability in the go-openssl library. I know some of the packages I have installed were built against it. What do I do? Use slot-operator := deps, together with the emerge --with-bdeps=y option. Then, if you bump the sub-slot of the go-openssl library, all of your go packages that have it in DEPEND with a slot-operator := dependency will be rebuilt automatically. Right, and now what if go-openssl was built on-the-fly 500 times and there's no package for it?
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/30/2015 08:35 AM, William Hubbs wrote: All, we have digressed a bit, so I want to bring the discussion back to what my main concerns are about this issue. 1. Should we bundle Go packages with Go software? If we do, except for the Go standard library which is part of dev-lang/go, do we need to bother with installing Go sources and packages at all? The down side of the whole bundling idea is that every consumer on someone's system could potentially have a different version of the Go package, which doesn't lend itself well to security concerns. This is why bundling is generally discouraged in Gentoo. Yes, as a general rule, bundling is sub-optimal. However, there are often exceptions to general rules like these, especially when there are competing concerns to contend with. Also, if we bundle, most of dev-go/* doesn't need to exist because these libraries would be bundled into and statically linked into the software that needs them. Some static libraries are commonly used enough that it might be reasonable to install them. Alternatively, we might invent a notion of having one ebuild execute another ebuild in order to install static dependencies into a temporary build directory. 2. How should we bundle? This is where my concern about consul and some other ebuilds comes in. The way the consul ebuild is written (putting the commit hashes of dependencies in SRC_URI) assumes that all of the dependencies will stay on github. This makes the ebuild far less flexable than go itself is. Agreed. However, there's no rule which says that we have to force all ebuilds to fit into common templates. If we are going to bundle, I would rather have one tarball that includes all of the sources for consul and the dependent libraries dropped on the Gentoo mirrors. Such a tarball is very easy to create. I would prefer to use separate tarballs for each dependency, preferably with the commit hash encoded in the tarball name. This makes the ebuild dependencies transparent in the sense that the commit hashes of the dependencies are readily available. The one big tarball is opaque rather than transparent, and it will have a tendency bloat the mirrors. By keeping the dependencies in separate tarballs, we can easily do a revbump that updates a subset of the dependencies, without having to re-pack everything into a big bloated tarball. -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On Tue, Jun 30, 2015 at 10:53:58AM -0700, Zac Medico wrote:
On 06/30/2015 08:35 AM, William Hubbs wrote:
All,
we have digressed a bit, so I want to bring the discussion back to what
my main concerns are about this issue.
1. Should we bundle Go packages with Go software?
If we do, except for the Go standard library which is part of
dev-lang/go, do we need to bother with installing Go sources and
packages at all?
The down side of the whole bundling idea is that every consumer
on someone's system could potentially have a different version of the
Go package, which doesn't lend itself well to security concerns.
This is why bundling is generally discouraged in Gentoo.
Yes, as a general rule, bundling is sub-optimal. However, there are
often exceptions to general rules like these, especially when there are
competing concerns to contend with.
I don't really see what the competing concerns are in this case.
Also, if we bundle, most of dev-go/* doesn't need to exist because these
libraries would be bundled into and statically linked into the software
that needs them.
Some static libraries are commonly used enough that it might be
reasonable to install them. Alternatively, we might invent a notion of
having one ebuild execute another ebuild in order to install static
dependencies into a temporary build directory.
Why do we need to worry about how many projects use a library? Upstream
has it as a library for good reason, so that multiple projects can use
it. If upstream installs it as a library, that's how we should install
it if we install it.
The problem I see with the argument about commonly used enough is the
vagueness of it. If we have two packages that use a library, it is
commonly used enough that it should be installed separately.
If we start out bundling libraries, especially libraries from different
upstreams than the package we are working on, that forces all go
maintainers to check all go ebuilds in the tree to see if multiple
bundling is going on and open bugs to create separate ebuilds for
libraries that were only used before by one package but now are used by
more than one.
2. How should we bundle?
This is where my concern about consul and some other ebuilds comes in.
The way the consul ebuild is written (putting the commit hashes of
dependencies in SRC_URI) assumes that all of the dependencies will stay
on github. This makes the ebuild far less flexable than go itself is.
Agreed. However, there's no rule which says that we have to force all
ebuilds to fit into common templates.
We do when they deal with common issues; that's the whole point of
language-based eclasses, e.g. ruby* perl* and python*.
If we are going to bundle, I would rather have one tarball that includes
all of the sources for consul and the dependent libraries dropped on the
Gentoo mirrors. Such a tarball is very easy to create.
I would prefer to use separate tarballs for each dependency, preferably
with the commit hash encoded in the tarball name. This makes the ebuild
dependencies transparent in the sense that the commit hashes of the
dependencies are readily available. The one big tarball is opaque
rather than transparent, and it will have a tendency bloat the mirrors.
By keeping the dependencies in separate tarballs, we can easily do a
revbump that updates a subset of the dependencies, without having to
re-pack everything into a big bloated tarball.
I can agree with part of this; one big tarball is less transparent than
multiple tarballs.
Another thing to consider is, with one big tarball, you can name the
extraction directory to match ${S}, which means there would be no need
for magic in the ebuilds to deal with putting extracted directories in
the right place.
All I'm saying is, if we are going to bundle, lets go all in and not
download multiple upstream packages in src_uri but put them in big
tarballs.
If we are not going to bundle, the best way to handle it is to not
bundle at all imo.
William
signature.asc
Description: Digital signature
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 30/06/2015 05:25, Zac Medico wrote: On 06/29/2015 07:24 PM, Michael Orlitzky wrote: On 06/29/2015 07:44 PM, Zac Medico wrote: Having faced the exact same problem I have to say I agree 100% with Zac. I'd like to say that Gentoo needs this kind of packages to stay actual and that our NOGO (yes that's an actual joke) on Go packages is not good for us nowdays. While it would certainly be possible to split out a number of separate ebuilds for Go libraries that are used *exclusively* by consul, what advantages would it have? Even in this limiting case, 1. You avoid pointless rebuilds. You rebuild the library (and probably the binary, for Go packages) when the library is upgraded rather than rebuilding everything whenever anything is updated. From my experience, Go packages don't take very long to build. +1, Go is not C, I have the same feeling 2. Security. If upstream treats the packages as separate, a user might hear that there's a security issue in libfoo but then run eix and see that he doesn't have libfoo installed (because it's bundled). That's a reasonable motivation. However, many of these libraries don't have any tags. So, you'll have to use the commit hashes if you want to test for vulnerabilities. In the case of the consul ebuild, the commit hashes of the libraries are available in the SRC_URI. I suppose that we could standardize a way to expose these. +1, there is no strong tagging on every upstream. Maybe that's another topic but handling git sub modules et al could be made easier while satisfying our QA (or maybe make some exceptions) 3. Chicken and egg problem. If the library only has one consumer and you keep it bundled with that consumer forever, then it will probably only ever have one consumer. If somebody wants to use it in an overlay or something he'd have to pull in the whole program. If a Go developer wants to use the libraries in question, then he'll probably use 'go get' to install them. I doubt the existence of an ebuild will have much relevance in people's decision to adopt a given Go library. 4. Ebuild complexity. Now you have to compile e.g. three packages in src_compile, install three packages in src_install, etc. The result is more complicated than building once, three times. In the case of the consul ebuild, all of the libraries are automatically built when the ebuild calls the emake. Even without a Makefile, Go makes it trivial to build the dependencies. Non live GIT ebuilds already make ebuilds more complex, this should indeed be enough. 5. One maintainer has to commit to maintaining all of the dependencies in addition to the program that he cares about. I guess that's a reasonable argument, depending on how much maintenance the dependencies require. Since there is no real Go support as such, this would be a pain ... Someone actually has to do the work to split out the libraries, so it may not be a clear-cut win in some cases. But it's nicer to have them split out should that happen by magic. Considering that Go binaries are statically linked, you'll end up with a bunch of Go libraries installed that you don't need during run-time. +1, this defeats Go's main advantage imho (not that I think it's smart, but it's the actual fact) Cheers
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/30/2015 01:30 PM, William Hubbs wrote: On Tue, Jun 30, 2015 at 10:53:58AM -0700, Zac Medico wrote: On 06/30/2015 08:35 AM, William Hubbs wrote: All, we have digressed a bit, so I want to bring the discussion back to what my main concerns are about this issue. 1. Should we bundle Go packages with Go software? If we do, except for the Go standard library which is part of dev-lang/go, do we need to bother with installing Go sources and packages at all? The down side of the whole bundling idea is that every consumer on someone's system could potentially have a different version of the Go package, which doesn't lend itself well to security concerns. This is why bundling is generally discouraged in Gentoo. Yes, as a general rule, bundling is sub-optimal. However, there are often exceptions to general rules like these, especially when there are competing concerns to contend with. I don't really see what the competing concerns are in this case. The competing concern is that un-bundling has some possibly undesirable consequences, mainly that it means we'll be installing static libraries that were only intended to be temporary build artifacts. It makes sense to install them if there are multiple consumers, otherwise it doesn't make much sense. Also, if we bundle, most of dev-go/* doesn't need to exist because these libraries would be bundled into and statically linked into the software that needs them. Some static libraries are commonly used enough that it might be reasonable to install them. Alternatively, we might invent a notion of having one ebuild execute another ebuild in order to install static dependencies into a temporary build directory. Why do we need to worry about how many projects use a library? If you want to clutter the tree with trivial ebuilds that only have a single consumer, then I guess that's fine. It's not clear to me that this is the best course of action, but I'm not going to try to stop you if that's what you want to do. Upstream has it as a library for good reason, so that multiple projects can use it. If upstream installs it as a library, that's how we should install it if we install it. I don't think that consul upstream installs it as a library. The last time that I checked upstream's build from source instructions, the consul binary was the only result of interest, so all of the temporary build artifacts could simply be discarded after the consul binary had been built. The problem I see with the argument about commonly used enough is the vagueness of it. If we have two packages that use a library, it is commonly used enough that it should be installed separately. As soon as you have at least two consumers, then you have a good reason to un-bundle a library. If there's only one consumer, then un-bundling becomes questionable. If we start out bundling libraries, especially libraries from different upstreams than the package we are working on, that forces all go maintainers to check all go ebuilds in the tree to see if multiple bundling is going on and open bugs to create separate ebuilds for libraries that were only used before by one package but now are used by more than one. Nobody is being forced to do anything. If a maintainer of a Go package than bundles libraries is doing a version bump, then it would be a good time for him to check if ebuilds have been created for any of those bundled dependencies, and un-bundle them at that point. 2. How should we bundle? This is where my concern about consul and some other ebuilds comes in. The way the consul ebuild is written (putting the commit hashes of dependencies in SRC_URI) assumes that all of the dependencies will stay on github. This makes the ebuild far less flexable than go itself is. Agreed. However, there's no rule which says that we have to force all ebuilds to fit into common templates. We do when they deal with common issues; that's the whole point of language-based eclasses, e.g. ruby* perl* and python*. There can always be outliers that don't fit your existing templates. If we are going to bundle, I would rather have one tarball that includes all of the sources for consul and the dependent libraries dropped on the Gentoo mirrors. Such a tarball is very easy to create. I would prefer to use separate tarballs for each dependency, preferably with the commit hash encoded in the tarball name. This makes the ebuild dependencies transparent in the sense that the commit hashes of the dependencies are readily available. The one big tarball is opaque rather than transparent, and it will have a tendency bloat the mirrors. By keeping the dependencies in separate tarballs, we can easily do a revbump that updates a subset of the dependencies, without having to re-pack everything into a big bloated tarball. I can agree with part of this; one big tarball is less transparent than multiple tarballs. Another thing to consider is, with one
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/30/2015 03:08 PM, William Hubbs wrote: On Tue, Jun 30, 2015 at 02:34:52PM -0700, Zac Medico wrote: On 06/30/2015 01:30 PM, William Hubbs wrote: I don't really see what the competing concerns are in this case. The competing concern is that un-bundling has some possibly undesirable consequences, mainly that it means we'll be installing static libraries that were only intended to be temporary build artifacts. It makes sense to install them if there are multiple consumers, otherwise it doesn't make much sense. Thinking about this, there may be a third option. This would take a slight reworking of the golang-build.eclass, but that is easy to do, and it would possibly remove the subslot from the dependencies. The source code is where the compatibility between versions of Go is, not the static objects, so what if, for third-party go packages, we skip installing the static objects? If we did this with consul, for example, then the source code for all those libraries (that have no other consumers) would have to be installed in order to build consul-template against the consul's api library. It would be similar to a header dependency. This would necessitate the introduction of build-against dependencies [1], or equivalent virtuals (like virtual/podofo-build). The only down side of this would be that there might be longer rebuilds if the packages have multiple consumers, but it gets rid of the static objects. What do you think? Considering the similarity to header dependencies, I don't know. The subslot thing seems slightly more appealing to me. [1] https://bugs.gentoo.org/show_bug.cgi?id=392239 -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On Tue, Jun 30, 2015 at 02:34:52PM -0700, Zac Medico wrote: On 06/30/2015 01:30 PM, William Hubbs wrote: On Tue, Jun 30, 2015 at 10:53:58AM -0700, Zac Medico wrote: On 06/30/2015 08:35 AM, William Hubbs wrote: All, we have digressed a bit, so I want to bring the discussion back to what my main concerns are about this issue. 1. Should we bundle Go packages with Go software? If we do, except for the Go standard library which is part of dev-lang/go, do we need to bother with installing Go sources and packages at all? The down side of the whole bundling idea is that every consumer on someone's system could potentially have a different version of the Go package, which doesn't lend itself well to security concerns. This is why bundling is generally discouraged in Gentoo. Yes, as a general rule, bundling is sub-optimal. However, there are often exceptions to general rules like these, especially when there are competing concerns to contend with. I don't really see what the competing concerns are in this case. The competing concern is that un-bundling has some possibly undesirable consequences, mainly that it means we'll be installing static libraries that were only intended to be temporary build artifacts. It makes sense to install them if there are multiple consumers, otherwise it doesn't make much sense. Thinking about this, there may be a third option. This would take a slight reworking of the golang-build.eclass, but that is easy to do, and it would possibly remove the subslot from the dependencies. The source code is where the compatibility between versions of Go is, not the static objects, so what if, for third-party go packages, we skip installing the static objects? The only down side of this would be that there might be longer rebuilds if the packages have multiple consumers, but it gets rid of the static objects. What do you think? William signature.asc Description: Digital signature
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 02:27 PM, William Hubbs wrote: All, we have several Go ebuilds in the tree that bundle multiple separate upstream sources. One example is app-admin/consul-0.5.2. My thought is that we shouldn't bundle like this, but we should figure out how to write ebuilds for the dependent packages as well. What do others think? Maybe we should take into account the number of consumers of said libraries? If there's only one consumer of a given library, then what's the advantage of splitting out a separate ebuild? Also, in our discussion, it may be useful to distinguish between bundling via one big tarball versus bundling via multiple tarballs in SRC_URI. -- Thanks, Zac
[gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
All, we have several Go ebuilds in the tree that bundle multiple separate upstream sources. One example is app-admin/consul-0.5.2. My thought is that we shouldn't bundle like this, but we should figure out how to write ebuilds for the dependent packages as well. What do others think? William signature.asc Description: Digital signature
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 05:27 PM, wirel...@tampabay.rr.com wrote: On 06/29/2015 05:50 PM, Zac Medico wrote: On 06/29/2015 02:27 PM, William Hubbs wrote: All, we have several Go ebuilds in the tree that bundle multiple separate upstream sources. One example is app-admin/consul-0.5.2. My thought is that we shouldn't bundle like this, but we should figure out how to write ebuilds for the dependent packages as well. What do others think? Maybe we should take into account the number of consumers of said libraries? If there's only one consumer of a given library, then what's the advantage of splitting out a separate ebuild? Also, in our discussion, it may be useful to distinguish between bundling via one big tarball versus bundling via multiple tarballs in SRC_URI. You have much to consider. Consul, like zookeeper (ultrabug overlay) is very useful for building clusters on (gentoo) linux. It would be very cool to split consul into a separate build. That way one can experiment with combining a wide variety of sys-cluster builds with other packages. While it would certainly be possible to split out a number of separate ebuilds for Go libraries that are used *exclusively* by consul, what advantages would it have? You mention a wide variety of sys-cluster builds, but I'm not sure what packages you're talking about. For example, are you aware of any other packages that use hashicorp's raft library [1]? Regardless of which way you go, it would be great to have some detail documents about the various (software) components if you stay with one large build. You can see all of the components (including github.com/hashicorp/raft) in the SRC_URI variable of the ebuild [2]. [1] https://github.com/hashicorp/raft [2] https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-admin/consul/consul-0.5.2.ebuild?view=markup -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 07:44 PM, Zac Medico wrote: While it would certainly be possible to split out a number of separate ebuilds for Go libraries that are used *exclusively* by consul, what advantages would it have? Even in this limiting case, 1. You avoid pointless rebuilds. You rebuild the library (and probably the binary, for Go packages) when the library is upgraded rather than rebuilding everything whenever anything is updated. 2. Security. If upstream treats the packages as separate, a user might hear that there's a security issue in libfoo but then run eix and see that he doesn't have libfoo installed (because it's bundled). 3. Chicken and egg problem. If the library only has one consumer and you keep it bundled with that consumer forever, then it will probably only ever have one consumer. If somebody wants to use it in an overlay or something he'd have to pull in the whole program. 4. Ebuild complexity. Now you have to compile e.g. three packages in src_compile, install three packages in src_install, etc. The result is more complicated than building once, three times. 5. One maintainer has to commit to maintaining all of the dependencies in addition to the program that he cares about. Someone actually has to do the work to split out the libraries, so it may not be a clear-cut win in some cases. But it's nicer to have them split out should that happen by magic.
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 05:50 PM, Zac Medico wrote: On 06/29/2015 02:27 PM, William Hubbs wrote: All, we have several Go ebuilds in the tree that bundle multiple separate upstream sources. One example is app-admin/consul-0.5.2. My thought is that we shouldn't bundle like this, but we should figure out how to write ebuilds for the dependent packages as well. What do others think? Maybe we should take into account the number of consumers of said libraries? If there's only one consumer of a given library, then what's the advantage of splitting out a separate ebuild? Also, in our discussion, it may be useful to distinguish between bundling via one big tarball versus bundling via multiple tarballs in SRC_URI. You have much to consider. Consul, like zookeeper (ultrabug overlay) is very useful for building clusters on (gentoo) linux. It would be very cool to split consul into a separate build. That way one can experiment with combining a wide variety of sys-cluster builds with other packages. Regardless of which way you go, it would be great to have some detail documents about the various (software) components if you stay with one large build. hth, James
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 07:08 PM, wirel...@tampabay.rr.com wrote: On 06/29/2015 06:50 PM, Zac Medico wrote: On 06/29/2015 05:27 PM, wirel...@tampabay.rr.com wrote: On 06/29/2015 05:50 PM, Zac Medico wrote: On 06/29/2015 02:27 PM, William Hubbs wrote: All, we have several Go ebuilds in the tree that bundle multiple separate upstream sources. One example is app-admin/consul-0.5.2. My thought is that we shouldn't bundle like this, but we should figure out how to write ebuilds for the dependent packages as well. What do others think? Maybe we should take into account the number of consumers of said libraries? If there's only one consumer of a given library, then what's the advantage of splitting out a separate ebuild? Also, in our discussion, it may be useful to distinguish between bundling via one big tarball versus bundling via multiple tarballs in SRC_URI. You have much to consider. Consul, like zookeeper (ultrabug overlay) is very useful for building clusters on (gentoo) linux. It would be very cool to split consul into a separate build. That way one can experiment with combining a wide variety of sys-cluster builds with other packages. While it would certainly be possible to split out a number of separate ebuilds for Go libraries that are used *exclusively* by consul, what advantages would it have? You mention a wide variety of sys-cluster builds, but I'm not sure what packages you're talking about. For example, are you aware of any other packages that use hashicorp's raft library [1]? First of all, I'm not sure why my nntp interface to gentoo-dev is not following the thread (sorry, I'm still working out how to use nntp to gentoo-dev). I'm not up on raft, although it looks very interesting [FSM] and all. I've been working on apache-mesos a bit. Consul is used frequently with mesos; here is one example [A]. My experience is that current clusters/clouds are mostly a unique mix of different software, consul being but one of many common components. Perhaps I did not have a sufficiently deep understanding of raft, Understanding raft is beyond the scope of this discussion. The question is, Do we know of any packages other than consul that consume the hashicorp/raft library? but my comment was meant to encourage a consul package for gentoo, We already have a consul package for gentoo, so there's no encouragement needed there. ;) I guess dependant on a raft package too. Are you sure about that, given that consul would be the only consumer of the hashicorp/raft library? Regardless of which way you go, it would be great to have some detail documents about the various (software) components if you stay with one large build. You can see all of the components (including github.com/hashicorp/raft) in the SRC_URI variable of the ebuild [2]. Yea, I need to read up on raft; it does look promising as it took mesos a while to become popular. Is raft as a separate ebuild useful; I'm not sure, but it does look interesting from what I've seen. It's not useful unless there are at least 2 ebuilds that can use it. Many projects within the cluster/cloud space have morphed, so raft has just as good a chance to diversify it's appeal and usefulness. Surely the convenience of the dev that maintains the package(s) is also keenly important. Unless you are writing an ebuild which uses the hashicorp/raft library, you really don't need an ebuild for it. So, maybe we should wait and see if the need arises. -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 06:50 PM, Zac Medico wrote: On 06/29/2015 05:27 PM, wirel...@tampabay.rr.com wrote: On 06/29/2015 05:50 PM, Zac Medico wrote: On 06/29/2015 02:27 PM, William Hubbs wrote: All, we have several Go ebuilds in the tree that bundle multiple separate upstream sources. One example is app-admin/consul-0.5.2. My thought is that we shouldn't bundle like this, but we should figure out how to write ebuilds for the dependent packages as well. What do others think? Maybe we should take into account the number of consumers of said libraries? If there's only one consumer of a given library, then what's the advantage of splitting out a separate ebuild? Also, in our discussion, it may be useful to distinguish between bundling via one big tarball versus bundling via multiple tarballs in SRC_URI. You have much to consider. Consul, like zookeeper (ultrabug overlay) is very useful for building clusters on (gentoo) linux. It would be very cool to split consul into a separate build. That way one can experiment with combining a wide variety of sys-cluster builds with other packages. While it would certainly be possible to split out a number of separate ebuilds for Go libraries that are used *exclusively* by consul, what advantages would it have? You mention a wide variety of sys-cluster builds, but I'm not sure what packages you're talking about. For example, are you aware of any other packages that use hashicorp's raft library [1]? First of all, I'm not sure why my nntp interface to gentoo-dev is not following the thread (sorry, I'm still working out how to use nntp to gentoo-dev). I'm not up on raft, although it looks very interesting [FSM] and all. I've been working on apache-mesos a bit. Consul is used frequently with mesos; here is one example [A]. My experience is that current clusters/clouds are mostly a unique mix of different software, consul being but one of many common components. Perhaps I did not have a sufficiently deep understanding of raft, but my comment was meant to encourage a consul package for gentoo, I guess dependant on a raft package too. Regardless of which way you go, it would be great to have some detail documents about the various (software) components if you stay with one large build. You can see all of the components (including github.com/hashicorp/raft) in the SRC_URI variable of the ebuild [2]. Yea, I need to read up on raft; it does look promising as it took mesos a while to become popular. Is raft as a separate ebuild useful; I'm not sure, but it does look interesting from what I've seen. Many projects within the cluster/cloud space have morphed, so raft has just as good a chance to diversify it's appeal and usefulness. Surely the convenience of the dev that maintains the package(s) is also keenly important. [1] https://github.com/hashicorp/raft [2] https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-admin/consul/consul-0.5.2.ebuild?view=markup [A] https://github.com/CiscoCloud/mesos-consul James
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 07:24 PM, Michael Orlitzky wrote: On 06/29/2015 07:44 PM, Zac Medico wrote: While it would certainly be possible to split out a number of separate ebuilds for Go libraries that are used *exclusively* by consul, what advantages would it have? Even in this limiting case, 1. You avoid pointless rebuilds. You rebuild the library (and probably the binary, for Go packages) when the library is upgraded rather than rebuilding everything whenever anything is updated. From my experience, Go packages don't take very long to build. 2. Security. If upstream treats the packages as separate, a user might hear that there's a security issue in libfoo but then run eix and see that he doesn't have libfoo installed (because it's bundled). That's a reasonable motivation. However, many of these libraries don't have any tags. So, you'll have to use the commit hashes if you want to test for vulnerabilities. In the case of the consul ebuild, the commit hashes of the libraries are available in the SRC_URI. I suppose that we could standardize a way to expose these. 3. Chicken and egg problem. If the library only has one consumer and you keep it bundled with that consumer forever, then it will probably only ever have one consumer. If somebody wants to use it in an overlay or something he'd have to pull in the whole program. If a Go developer wants to use the libraries in question, then he'll probably use 'go get' to install them. I doubt the existence of an ebuild will have much relevance in people's decision to adopt a given Go library. 4. Ebuild complexity. Now you have to compile e.g. three packages in src_compile, install three packages in src_install, etc. The result is more complicated than building once, three times. In the case of the consul ebuild, all of the libraries are automatically built when the ebuild calls the emake. Even without a Makefile, Go makes it trivial to build the dependencies. 5. One maintainer has to commit to maintaining all of the dependencies in addition to the program that he cares about. I guess that's a reasonable argument, depending on how much maintenance the dependencies require. Someone actually has to do the work to split out the libraries, so it may not be a clear-cut win in some cases. But it's nicer to have them split out should that happen by magic. Considering that Go binaries are statically linked, you'll end up with a bunch of Go libraries installed that you don't need during run-time. -- Thanks, Zac
