Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 07:24 PM, Michael Orlitzky wrote: > On 06/29/2015 07:44 PM, Zac Medico wrote: >> >> While it would certainly be possible to split out a number of separate >> ebuilds for Go libraries that are used *exclusively* by consul, what >> advantages would it have? > > Even in this limiting case, > > 1. You avoid pointless rebuilds. You rebuild the library (and > probably the binary, for Go packages) when the library is upgraded > rather than rebuilding everything whenever anything is updated. >From my experience, Go packages don't take very long to build. > 2. Security. If upstream treats the packages as separate, a user > might hear that there's a security issue in libfoo but then run > eix and see that he doesn't have libfoo installed (because it's > bundled). That's a reasonable motivation. However, many of these libraries don't have any tags. So, you'll have to use the commit hashes if you want to test for vulnerabilities. In the case of the consul ebuild, the commit hashes of the libraries are available in the SRC_URI. I suppose that we could standardize a way to expose these. > 3. Chicken and egg problem. If the library only has one consumer and > you keep it bundled with that consumer forever, then it will > probably only ever have one consumer. If somebody wants to use > it in an overlay or something he'd have to pull in the whole > program. If a Go developer wants to use the libraries in question, then he'll probably use 'go get' to install them. I doubt the existence of an ebuild will have much relevance in people's decision to adopt a given Go library. > 4. Ebuild complexity. Now you have to compile e.g. three packages in > src_compile, install three packages in src_install, etc. The result > is more complicated than building once, three times. In the case of the consul ebuild, all of the libraries are automatically built when the ebuild calls the emake. Even without a Makefile, Go makes it trivial to build the dependencies. > 5. One maintainer has to commit to maintaining all of the dependencies > in addition to the program that he cares about. I guess that's a reasonable argument, depending on how much maintenance the dependencies require. > Someone actually has to do the work to split out the libraries, so it > may not be a clear-cut win in some cases. But it's nicer to have them > split out should that happen by magic. Considering that Go binaries are statically linked, you'll end up with a bunch of Go libraries installed that you don't need during run-time. -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 07:08 PM, wirel...@tampabay.rr.com wrote: > On 06/29/2015 06:50 PM, Zac Medico wrote: >> On 06/29/2015 05:27 PM, wirel...@tampabay.rr.com wrote: >>> On 06/29/2015 05:50 PM, Zac Medico wrote: On 06/29/2015 02:27 PM, William Hubbs wrote: > All, > > we have several Go ebuilds in the tree that bundle multiple separate > upstream sources. One example is app-admin/consul-0.5.2. > > My thought is that we shouldn't bundle like this, but we should figure > out how to write ebuilds for the dependent packages as well. > > What do others think? Maybe we should take into account the number of consumers of said libraries? If there's only one consumer of a given library, then what's the advantage of splitting out a separate ebuild? Also, in our discussion, it may be useful to distinguish between bundling via "one big tarball" versus bundling via multiple tarballs in SRC_URI. >>> >>> You have much to consider. Consul, like zookeeper (ultrabug overlay) is >>> very useful for building clusters on (gentoo) linux. It would be very >>> cool to split consul into a separate build. That way one can experiment >>> with combining a wide variety of sys-cluster builds with other >>> packages. >> >> While it would certainly be possible to split out a number of separate >> ebuilds for Go libraries that are used *exclusively* by consul, what >> advantages would it have? You mention "a wide variety of sys-cluster >> builds," but I'm not sure what packages you're talking about. For >> example, are you aware of any other packages that use hashicorp's raft >> library [1]? > > First of all, I'm not sure why my nntp interface to gentoo-dev is not > following the thread (sorry, I'm still working out how to use nntp to > gentoo-dev). > > I'm not up on raft, although it looks very interesting [FSM] and all. > I've been working on apache-mesos a bit. Consul is used frequently > with mesos; here is one example [A]. My experience is that current > clusters/clouds are mostly a unique mix of different software, consul > being but one of many common components. Perhaps I did not have a > sufficiently deep understanding of raft, Understanding raft is beyond the scope of this discussion. The question is, "Do we know of any packages other than consul that consume the hashicorp/raft library?" > but my comment was meant to > encourage a consul package for gentoo, We already have a consul package for gentoo, so there's no encouragement needed there. ;) > I guess dependant on a raft package too. Are you sure about that, given that consul would be the only consumer of the hashicorp/raft library? > >>> Regardless of which way you go, it would be great to have some detail >>> documents about the various (software) components if you stay with one >>> large build. >> >> You can see all of the components (including github.com/hashicorp/raft) >> in the SRC_URI variable of the ebuild [2]. > > Yea, I need to read up on raft; it does look promising as it took mesos > a while to become popular. Is raft as a separate ebuild useful; I'm not > sure, but it does look interesting from what I've seen. It's not useful unless there are at least 2 ebuilds that can use it. > Many projects > within the cluster/cloud space have morphed, so raft has just as good a > chance to diversify it's appeal and usefulness. Surely the convenience > of the dev that maintains the package(s) is also keenly important. Unless you are writing an ebuild which uses the hashicorp/raft library, you really don't need an ebuild for it. So, maybe we should wait and see if the need arises. -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 07:44 PM, Zac Medico wrote: > > While it would certainly be possible to split out a number of separate > ebuilds for Go libraries that are used *exclusively* by consul, what > advantages would it have? Even in this limiting case, 1. You avoid pointless rebuilds. You rebuild the library (and probably the binary, for Go packages) when the library is upgraded rather than rebuilding everything whenever anything is updated. 2. Security. If upstream treats the packages as separate, a user might hear that there's a security issue in libfoo but then run eix and see that he doesn't have libfoo installed (because it's bundled). 3. Chicken and egg problem. If the library only has one consumer and you keep it bundled with that consumer forever, then it will probably only ever have one consumer. If somebody wants to use it in an overlay or something he'd have to pull in the whole program. 4. Ebuild complexity. Now you have to compile e.g. three packages in src_compile, install three packages in src_install, etc. The result is more complicated than building once, three times. 5. One maintainer has to commit to maintaining all of the dependencies in addition to the program that he cares about. Someone actually has to do the work to split out the libraries, so it may not be a clear-cut win in some cases. But it's nicer to have them split out should that happen by magic.
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 06:50 PM, Zac Medico wrote: On 06/29/2015 05:27 PM, wirel...@tampabay.rr.com wrote: On 06/29/2015 05:50 PM, Zac Medico wrote: On 06/29/2015 02:27 PM, William Hubbs wrote: All, we have several Go ebuilds in the tree that bundle multiple separate upstream sources. One example is app-admin/consul-0.5.2. My thought is that we shouldn't bundle like this, but we should figure out how to write ebuilds for the dependent packages as well. What do others think? Maybe we should take into account the number of consumers of said libraries? If there's only one consumer of a given library, then what's the advantage of splitting out a separate ebuild? Also, in our discussion, it may be useful to distinguish between bundling via "one big tarball" versus bundling via multiple tarballs in SRC_URI. You have much to consider. Consul, like zookeeper (ultrabug overlay) is very useful for building clusters on (gentoo) linux. It would be very cool to split consul into a separate build. That way one can experiment with combining a wide variety of sys-cluster builds with other packages. While it would certainly be possible to split out a number of separate ebuilds for Go libraries that are used *exclusively* by consul, what advantages would it have? You mention "a wide variety of sys-cluster builds," but I'm not sure what packages you're talking about. For example, are you aware of any other packages that use hashicorp's raft library [1]? First of all, I'm not sure why my nntp interface to gentoo-dev is not following the thread (sorry, I'm still working out how to use nntp to gentoo-dev). I'm not up on raft, although it looks very interesting [FSM] and all. I've been working on apache-mesos a bit. Consul is used frequently with mesos; here is one example [A]. My experience is that current clusters/clouds are mostly a unique mix of different software, consul being but one of many common components. Perhaps I did not have a sufficiently deep understanding of raft, but my comment was meant to encourage a consul package for gentoo, I guess dependant on a raft package too. Regardless of which way you go, it would be great to have some detail documents about the various (software) components if you stay with one large build. You can see all of the components (including github.com/hashicorp/raft) in the SRC_URI variable of the ebuild [2]. Yea, I need to read up on raft; it does look promising as it took mesos a while to become popular. Is raft as a separate ebuild useful; I'm not sure, but it does look interesting from what I've seen. Many projects within the cluster/cloud space have morphed, so raft has just as good a chance to diversify it's appeal and usefulness. Surely the convenience of the dev that maintains the package(s) is also keenly important. [1] https://github.com/hashicorp/raft [2] https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-admin/consul/consul-0.5.2.ebuild?view=markup [A] https://github.com/CiscoCloud/mesos-consul James
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 05:27 PM, wirel...@tampabay.rr.com wrote: > On 06/29/2015 05:50 PM, Zac Medico wrote: >> On 06/29/2015 02:27 PM, William Hubbs wrote: >>> All, >>> >>> we have several Go ebuilds in the tree that bundle multiple separate >>> upstream sources. One example is app-admin/consul-0.5.2. >>> >>> My thought is that we shouldn't bundle like this, but we should figure >>> out how to write ebuilds for the dependent packages as well. >>> >>> What do others think? >> >> Maybe we should take into account the number of consumers of said >> libraries? If there's only one consumer of a given library, then what's >> the advantage of splitting out a separate ebuild? Also, in our >> discussion, it may be useful to distinguish between bundling via "one >> big tarball" versus bundling via multiple tarballs in SRC_URI. > > You have much to consider. Consul, like zookeeper (ultrabug overlay) is > very useful for building clusters on (gentoo) linux. It would be very > cool to split consul into a separate build. That way one can experiment > with combining a wide variety of sys-cluster builds with other packages. While it would certainly be possible to split out a number of separate ebuilds for Go libraries that are used *exclusively* by consul, what advantages would it have? You mention "a wide variety of sys-cluster builds," but I'm not sure what packages you're talking about. For example, are you aware of any other packages that use hashicorp's raft library [1]? > Regardless of which way you go, it would be great to have some detail > documents about the various (software) components if you stay with one > large build. You can see all of the components (including github.com/hashicorp/raft) in the SRC_URI variable of the ebuild [2]. [1] https://github.com/hashicorp/raft [2] https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-admin/consul/consul-0.5.2.ebuild?view=markup -- Thanks, Zac
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 05:50 PM, Zac Medico wrote: On 06/29/2015 02:27 PM, William Hubbs wrote: All, we have several Go ebuilds in the tree that bundle multiple separate upstream sources. One example is app-admin/consul-0.5.2. My thought is that we shouldn't bundle like this, but we should figure out how to write ebuilds for the dependent packages as well. What do others think? Maybe we should take into account the number of consumers of said libraries? If there's only one consumer of a given library, then what's the advantage of splitting out a separate ebuild? Also, in our discussion, it may be useful to distinguish between bundling via "one big tarball" versus bundling via multiple tarballs in SRC_URI. You have much to consider. Consul, like zookeeper (ultrabug overlay) is very useful for building clusters on (gentoo) linux. It would be very cool to split consul into a separate build. That way one can experiment with combining a wide variety of sys-cluster builds with other packages. Regardless of which way you go, it would be great to have some detail documents about the various (software) components if you stay with one large build. hth, James
[gentoo-dev] Re: Herd/project cleanup
On Saturday 27 of June 2015 21:39:35 Johannes Huber wrote: Feel free to keep me there. (although my usefulness has lessened of late...) regards MM
Re: [gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
On 06/29/2015 02:27 PM, William Hubbs wrote: > All, > > we have several Go ebuilds in the tree that bundle multiple separate > upstream sources. One example is app-admin/consul-0.5.2. > > My thought is that we shouldn't bundle like this, but we should figure > out how to write ebuilds for the dependent packages as well. > > What do others think? Maybe we should take into account the number of consumers of said libraries? If there's only one consumer of a given library, then what's the advantage of splitting out a separate ebuild? Also, in our discussion, it may be useful to distinguish between bundling via "one big tarball" versus bundling via multiple tarballs in SRC_URI. -- Thanks, Zac
[gentoo-dev] rfc: Go ebuilds bundling multiple upstream sources
All, we have several Go ebuilds in the tree that bundle multiple separate upstream sources. One example is app-admin/consul-0.5.2. My thought is that we shouldn't bundle like this, but we should figure out how to write ebuilds for the dependent packages as well. What do others think? William signature.asc Description: Digital signature
