Re: [foreman-dev] Vendorizing or Building RPMs
The conversation has been open for 2 weeks now, and I appreciate all of the feedback and conversation. I am going to summarize the conversations as they stand and outline what I believe next steps are based on the feedback. RPMs I performed a quick tally of those that responded and essentially got the following counts on SCL vs. vendor. SCL: 4 Vendorizing: 2 Further, there appeared to be a lot of unanswered technical questions around how we maintain a vendorized stack with plugins that need to add dependencies. Based on this feedback, I believe the goal should be to create a new Rails SCL that we own and maintain. As for the plan, the how, of creating and maintaining this new SCL, I will start a new thread to discuss the plan and improvements we can make along the way. NPM Across the board, everyone appeared to be in favor of vendorizing our NPM modules to reduce the frequency of breakages and to allow the UI work that is on going across Foreman and plugins to continue at a rapid pace. I'll start a similar thread to outline and discuss the changes for this. Eric On Sun, Oct 29, 2017 at 5:43 PM, Ewoud Kohl van Wijngaarden < ew...@kohlvanwijngaarden.nl> wrote: > On Mon, Oct 23, 2017 at 09:52:39AM +0100, Greg Sutcliffe wrote: > >> On Mon, 2017-10-16 at 14:36 +0100, Greg Sutcliffe wrote: >> >>> >>> That said, I've not really been involved with the RPMs, so I'm unsure >>> if this causes a bigger headache for Yum users than Apt users. I'm >>> also unsure of the work required to create an SCL, but if it's non- >>> trivial then I'd be looking to CentOS to collaborate on a Rails SCL >>> for everyone to use - for just ourselves, then vendoring seems >>> easier. >>> >> >> I spoke to a few people I know about this, and it seems there's not >> much appetite for making new SCLs. We might be able to attract >> contributors once it's created, but I think we should assume the effort >> for creating/maintaining SCLs will fall on us initially. >> >> Do we have any conclusions on this thread? It's going to matter for >> 1.17 which is getting closer by the day. >> > > Personally I feel an updated RoR SCL is the way to go for 1.17 and to > prove that I'm willing to invest the time to make it a reality. After 1.16 > RC2 is out I'm going to spend time on it. > > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Eric D. Helms Red Hat Engineering -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
On Mon, Oct 23, 2017 at 09:52:39AM +0100, Greg Sutcliffe wrote: On Mon, 2017-10-16 at 14:36 +0100, Greg Sutcliffe wrote: That said, I've not really been involved with the RPMs, so I'm unsure if this causes a bigger headache for Yum users than Apt users. I'm also unsure of the work required to create an SCL, but if it's non- trivial then I'd be looking to CentOS to collaborate on a Rails SCL for everyone to use - for just ourselves, then vendoring seems easier. I spoke to a few people I know about this, and it seems there's not much appetite for making new SCLs. We might be able to attract contributors once it's created, but I think we should assume the effort for creating/maintaining SCLs will fall on us initially. Do we have any conclusions on this thread? It's going to matter for 1.17 which is getting closer by the day. Personally I feel an updated RoR SCL is the way to go for 1.17 and to prove that I'm willing to invest the time to make it a reality. After 1.16 RC2 is out I'm going to spend time on it. -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
On Mon, 2017-10-16 at 14:36 +0100, Greg Sutcliffe wrote: > > That said, I've not really been involved with the RPMs, so I'm unsure > if this causes a bigger headache for Yum users than Apt users. I'm > also unsure of the work required to create an SCL, but if it's non- > trivial then I'd be looking to CentOS to collaborate on a Rails SCL > for everyone to use - for just ourselves, then vendoring seems > easier. I spoke to a few people I know about this, and it seems there's not much appetite for making new SCLs. We might be able to attract contributors once it's created, but I think we should assume the effort for creating/maintaining SCLs will fall on us initially. Do we have any conclusions on this thread? It's going to matter for 1.17 which is getting closer by the day. Greg -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
On Wed, 2017-10-18 at 15:53 -0400, Eric D Helms wrote: > > How do the plugins isolate their dependent gems? i.e. is it a script > or build tool or just knowledge to grab the right gems and pop them > in? There's two forms. For gems only used by the plugin, we add them to the package as a list of deps - see [1] for an example. Where more than one gem needs a dep, we move it to a separate package that can be depended on, e.g. deface [2,3] Greg [1] https://github.com/theforeman/foreman-packaging/blob/deb/develop/pl ugins/ruby-foreman-graphite/debian/gem.list [2] https://github.com/theforeman/foreman-packaging/tree/deb/develop/pl ugins/ruby-foreman-deface [3] https://github.com/theforeman/foreman-packaging/blob/deb/develop/pl ugins/ruby-foreman-column-view/debian/control#L11 -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
On Sat, Oct 14, 2017 at 02:24:58PM -0400, Eric D Helms wrote: Note: This is specific to RPMs because as far as I know the Debian process is different and uses gems directly. Please correct me and contribute any information with respect to Debian that I miss. I believe the Nodejs part of the email does apply to Debian. This proposal is in two parts: gems and node packages. The general issue applies to both, the project has a need to be able to move and update dependencies more quickly given the speed at which platforms like Rails, React and Webpack update. Node/NPM Packages Today, we package nearly all nodejs packages that are required to either build the UI or runtime UI dependencies into packages. At current count, there are 78 nodejs packages listed in Koji as RPMs. Over the past few months, the move to React and updating UI within Foreman core and branching into plugins has led to multiple, and at times, compounding packaging and thus nightly breakages. One of those breakages lasted over a week with multiple developers spending many hours tracking down the issue. The speed at which these changes are made makes keeping up successful nightly builds difficult. The git repository gets well ahead of the packaging which leads to compounding issues. This has also been the cause of stable version release delays. Proposal: Deprecate nodejs packages in favor of foreman-assets or a new RPM foreman-node-modules that contains a source tarball of node_modules/ packaged into a simple RPM that is used as a build dependency. This tarball would be regenerated, and the package bumped, as dependency updates are needed. I'm not sure how wise this is because I'm not that familiar with NPM and webpack. After some experimenting I've grown to strongly dislike the entire JS ecosystem. Their ideas about package management differ strongly from mine. I do wonder how we'll handle plugins which have their own dependencies. Will they ship a foreman-PLUGIN-assets with their own (bundled) dependencies? What if there's a conflict? How will we monitor security issues? All the traditional issues that bundling brings along. In short: I'd like to avoid this but wonder if we'll be forced to because of their policies. One side note: it looks like our RPMs are growing a LOT by this and cause our mirrors to need a lot more disk space and bandwidth. While perhaps cheap, it is something to consider. node_modules is *big*. Ruby Gems Today we package all required rubygems as RPMs and utilize a thirdparty Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack (sclo-ror42). The team that manages the RHSCLs has already released Ruby 2.3 and RUby 2.4 SCLs and will continue to do so. However, there are no plans to release any further SCLs for the Rails stack. This means, in addition to our own dependencies, we need to build and manage the Rails stack for the versions we want. This adds more packaging burden on the RPM side. GIven this, we have a few options: 1) Build and manage our own SCLs for every version of Rails from here on out 2) Vendorize Rails and thirdparty gems into a one or more base RPMs (e.g. an RPM for rails stack and one for thirdparty dependencies) Option 2, to vendorize, is a deviation from our prior practices in the area of production deployment. Thus, I am reaching out to the community to get feedback. One interesting consideration for vendorizing is when containers are considered and having the ability to build them using 'bundle install' versus using RPM based installation. In theory vendorizing allows the community to move faster, keep up to date with dependencies more easily and reduces the burden on RPM packaging to keep nightly builds flowing. However, this does stray from the standard RPM based installation benefits typically associated to it. I'd strongly prefer 1) because I believe it leads to a higher quality. While it does have downsides, presently I don't think it's that much of an issue. I'm also willing to invest more time here to ensure it'll become smoother. -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
On Wed, Oct 18, 2017 at 3:01 PM, Michael Moll wrote: > hi, > > On Tue, Oct 17, 2017 at 03:42:50PM -0700, Dmitri Dolguikh wrote: > > >2) Vendorize Rails and thirdparty gems into a one or more base RPMs > > > (e.g. an RPM for rails stack and one for thirdparty dependencies) > > > > Not sure if this has already been considered: use bundler to package the > > app. In this case all dependencies, including rails will be cached > locally, > > and we could then wrap the result into an rpm. Not sure how plugins would > > fit into this though. > > I'm not 100% sure if you meant it like that, but basically this is what > we're doing in the current DEB approach. The "foreman" package is the > result of "bundle package", the subpackages drop in a file or two each > into bundler.d/ and a plugin package is its gem (and if needed, > dependent gems) to add into the current bundle and precompiled assets. > How do the plugins isolate their dependent gems? i.e. is it a script or build tool or just knowledge to grab the right gems and pop them in? > > Regards > -- > Michael Moll > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Eric D. Helms Red Hat Engineering -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
hi, On Tue, Oct 17, 2017 at 03:42:50PM -0700, Dmitri Dolguikh wrote: > >2) Vendorize Rails and thirdparty gems into a one or more base RPMs > > (e.g. an RPM for rails stack and one for thirdparty dependencies) > > Not sure if this has already been considered: use bundler to package the > app. In this case all dependencies, including rails will be cached locally, > and we could then wrap the result into an rpm. Not sure how plugins would > fit into this though. I'm not 100% sure if you meant it like that, but basically this is what we're doing in the current DEB approach. The "foreman" package is the result of "bundle package", the subpackages drop in a file or two each into bundler.d/ and a plugin package is its gem (and if needed, dependent gems) to add into the current bundle and precompiled assets. Regards -- Michael Moll -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
On Wed, Oct 18, 2017 at 1:46 AM, Eric D Helms wrote: > Plugins do present the most complicated part of this process given they > would each potentially be requiring a few independent gems separate from the > core stack and we would not want bloat. Seems we'd need either: > > 1) a single gem stack that plugins contribute to and is renegenrated with > all them enabled > 2) a method for plugins to generate the bundle install, and pick out just > the gems they need to contribute on top of the core stack The disadvantage of 2) comes, when there are two plugins, that both need the same dependency, that is not in the core and using both plugins can lead to version collisions (we've seen that in the deb packaging every now and then). Although, having multiple versions of gems usually is not that much of a big deal, as long as they are compatible with the plugins, it's seems a bit weird that the specific versions being used would depend on the specific plugins being installed. The disadvantage of the the single plugins gem stack is, that update of any dependency there means that the end user needs to re-download the whole bundle again. This is probably not issue for most of the dependencies, that we don't update that often (the part of the stack). But there are gems, such as Dynflow, that are more related to the code and it would be beneficial to have an ability to deliver new versions asynchronously. Thinking about it a bit more, we usually need the fast updates in nightlies, where it's probably not much of a deal for the end user, and having the gem stack rpm updated with every upstream release seems like it could work. There would be still possibility for any plugin, to bring in their own dependency, that would not be in the common stack, if needed, but the recommended would be to get into the Foreman ecosystem to get the deps into the common stack. So for now, 1) is winning for me. This might be also an entry-point for testing the plugins together. Another potential move could be tackling how the plugins are getting enable in the first place. If the bits and pieces for all the plugins would be available every time, than the act of installing a plugin would be just about enabling loading the corresponding gem (as opposed for need to installing it first). It would make it also easier to make the plugins discoverable: there is of course more work to be done for making that working, but just exploring what it might mean. It's also good to thing about how that could influence the containerization work. We could still build the containers at the end-user side, with a subset of plugins installed, but with the all-plugins bundle, we could distribute the images themselves. There would be additional requirement on the plugin developers to sync even more with the core releases, but I think it's a good thing. -- Ivan > > On Tue, Oct 17, 2017 at 6:42 PM, Dmitri Dolguikh > wrote: >> >> > Today we package all required rubygems as RPMs and utilize a thirdparty >> > Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack >> > (sclo-ror42). The team that manages the RHSCLs has already released Ruby >> > 2.3 and RUby 2.4 SCLs and will continue to do so. However, there are no >> > plans to release any further SCLs for the Rails stack. This means, in >> > addition to our own dependencies, we need to build and manage the Rails >> > stack for the versions we want. This adds more packaging burden on the >> > RPM >> > side. GIven this, we have a few options: >> > >> >1) Build and manage our own SCLs for every version of Rails from here >> > on >> > out >> >2) Vendorize Rails and thirdparty gems into a one or more base RPMs >> > (e.g. an RPM for rails stack and one for thirdparty dependencies) >> >> Not sure if this has already been considered: use bundler to package the >> app. In this case all dependencies, including rails will be cached locally, >> and we could then wrap the result into an rpm. Not sure how plugins would >> fit into this though. >> >> -d >> >> On Tue, Oct 17, 2017 at 11:17 AM, Bryan Kearney >> wrote: >>> >>> On 10/16/2017 04:36 AM, Michael Moll wrote: >>> > I don't really have stakes in RPMs, but I'd try to stick with option 1 >>> > and I even expect that there's some demand for Rails SCLs at least in >>> > the RHEL/CentOS world by other projects, too. >>> > >>> > Regards >>> >>> I would suggest the decision be made based on the assumption that the >>> foreman community owns all the work for options 1 and 2. I have no >>> special instights, but I would hate to assume we could package based on >>> another communities work. Assume this commuity owns it all, which way is >>> better/easier/quicker/less error prone. >>> >>> -- bk >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "foreman-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to foreman-dev+unsubscr...@googlegroups.com. >>> For mor
Re: [foreman-dev] Vendorizing or Building RPMs
Plugins do present the most complicated part of this process given they would each potentially be requiring a few independent gems separate from the core stack and we would not want bloat. Seems we'd need either: 1) a single gem stack that plugins contribute to and is renegenrated with all them enabled 2) a method for plugins to generate the bundle install, and pick out just the gems they need to contribute on top of the core stack On Tue, Oct 17, 2017 at 6:42 PM, Dmitri Dolguikh wrote: > > Today we package all required rubygems as RPMs and utilize a thirdparty > > Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack > > (sclo-ror42). The team that manages the RHSCLs has already released Ruby > > 2.3 and RUby 2.4 SCLs and will continue to do so. However, there are no > > plans to release any further SCLs for the Rails stack. This means, in > > addition to our own dependencies, we need to build and manage the Rails > > stack for the versions we want. This adds more packaging burden on the > RPM > > side. GIven this, we have a few options: > > > >1) Build and manage our own SCLs for every version of Rails from here > on > > out > >2) Vendorize Rails and thirdparty gems into a one or more base RPMs > > (e.g. an RPM for rails stack and one for thirdparty dependencies) > > Not sure if this has already been considered: use bundler to package the > app. In this case all dependencies, including rails will be cached locally, > and we could then wrap the result into an rpm. Not sure how plugins would > fit into this though. > > -d > > On Tue, Oct 17, 2017 at 11:17 AM, Bryan Kearney > wrote: > >> On 10/16/2017 04:36 AM, Michael Moll wrote: >> > I don't really have stakes in RPMs, but I'd try to stick with option 1 >> > and I even expect that there's some demand for Rails SCLs at least in >> > the RHEL/CentOS world by other projects, too. >> > >> > Regards >> >> I would suggest the decision be made based on the assumption that the >> foreman community owns all the work for options 1 and 2. I have no >> special instights, but I would hate to assume we could package based on >> another communities work. Assume this commuity owns it all, which way is >> better/easier/quicker/less error prone. >> >> -- bk >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Eric D. Helms Red Hat Engineering -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
> Today we package all required rubygems as RPMs and utilize a thirdparty > Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack > (sclo-ror42). The team that manages the RHSCLs has already released Ruby > 2.3 and RUby 2.4 SCLs and will continue to do so. However, there are no > plans to release any further SCLs for the Rails stack. This means, in > addition to our own dependencies, we need to build and manage the Rails > stack for the versions we want. This adds more packaging burden on the RPM > side. GIven this, we have a few options: > >1) Build and manage our own SCLs for every version of Rails from here on > out >2) Vendorize Rails and thirdparty gems into a one or more base RPMs > (e.g. an RPM for rails stack and one for thirdparty dependencies) Not sure if this has already been considered: use bundler to package the app. In this case all dependencies, including rails will be cached locally, and we could then wrap the result into an rpm. Not sure how plugins would fit into this though. -d On Tue, Oct 17, 2017 at 11:17 AM, Bryan Kearney wrote: > On 10/16/2017 04:36 AM, Michael Moll wrote: > > I don't really have stakes in RPMs, but I'd try to stick with option 1 > > and I even expect that there's some demand for Rails SCLs at least in > > the RHEL/CentOS world by other projects, too. > > > > Regards > > I would suggest the decision be made based on the assumption that the > foreman community owns all the work for options 1 and 2. I have no > special instights, but I would hate to assume we could package based on > another communities work. Assume this commuity owns it all, which way is > better/easier/quicker/less error prone. > > -- bk > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
On 10/16/2017 04:36 AM, Michael Moll wrote: > I don't really have stakes in RPMs, but I'd try to stick with option 1 > and I even expect that there's some demand for Rails SCLs at least in > the RHEL/CentOS world by other projects, too. > > Regards I would suggest the decision be made based on the assumption that the foreman community owns all the work for options 1 and 2. I have no special instights, but I would hate to assume we could package based on another communities work. Assume this commuity owns it all, which way is better/easier/quicker/less error prone. -- bk -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
I'm ok with the vendorizing approach, as long as it makes the release process more effective than it is right now. Could you expand more on how the core dependencies, plugins and plugin-dependencies work? Perhaps this is something we could align with the debian packaging as well to address some of the issues that we are facing. -- Ivan On Sat, Oct 14, 2017 at 8:24 PM, Eric D Helms wrote: > Note: This is specific to RPMs because as far as I know the Debian process > is different and uses gems directly. Please correct me and contribute any > information with respect to Debian that I miss. I believe the Nodejs part of > the email does apply to Debian. > > This proposal is in two parts: gems and node packages. The general issue > applies to both, the project has a need to be able to move and update > dependencies more quickly given the speed at which platforms like Rails, > React and Webpack update. > > > Node/NPM Packages > > Today, we package nearly all nodejs packages that are required to either > build the UI or runtime UI dependencies into packages. At current count, > there are 78 nodejs packages listed in Koji as RPMs. Over the past few > months, the move to React and updating UI within Foreman core and branching > into plugins has led to multiple, and at times, compounding packaging and > thus nightly breakages. One of those breakages lasted over a week with > multiple developers spending many hours tracking down the issue. The speed > at which these changes are made makes keeping up successful nightly builds > difficult. The git repository gets well ahead of the packaging which leads > to compounding issues. This has also been the cause of stable version > release delays. > > Proposal: > Deprecate nodejs packages in favor of foreman-assets or a new RPM > foreman-node-modules that contains a source tarball of node_modules/ > packaged into a simple RPM that is used as a build dependency. This tarball > would be regenerated, and the package bumped, as dependency updates are > needed. > > > Ruby Gems > > Today we package all required rubygems as RPMs and utilize a thirdparty > Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack > (sclo-ror42). The team that manages the RHSCLs has already released Ruby 2.3 > and RUby 2.4 SCLs and will continue to do so. However, there are no plans to > release any further SCLs for the Rails stack. This means, in addition to our > own dependencies, we need to build and manage the Rails stack for the > versions we want. This adds more packaging burden on the RPM side. GIven > this, we have a few options: > >1) Build and manage our own SCLs for every version of Rails from here on > out >2) Vendorize Rails and thirdparty gems into a one or more base RPMs (e.g. > an RPM for rails stack and one for thirdparty dependencies) > > Option 2, to vendorize, is a deviation from our prior practices in the area > of production deployment. Thus, I am reaching out to the community to get > feedback. One interesting consideration for vendorizing is when containers > are considered and having the ability to build them using 'bundle install' > versus using RPM based installation. > > In theory vendorizing allows the community to move faster, keep up to date > with dependencies more easily and reduces the burden on RPM packaging to > keep nightly builds flowing. However, this does stray from the standard RPM > based installation benefits typically associated to it. > > > Please consider carefully, and weigh in even if it is simply to +1 or -1 a > given idea. Deployment is a large part of our project and the more voices > weighing in the better. > > Thanks, > Eriic > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
I agree with Tomer here, ditto. On the other hand, the way how rubygems are being packaged makes package patches (RPM patch file sets) really hard (in short - it's a hack). Since I also touch packaging work here and there, I know how much work is to maintain full stack like Ruby on Rails. So I can imagine having RoR vendored completely, if and only if this goes into downstream without much changes (I see this as possible problem). There are advantages of this approach, we could easily ship more gems, e.g. development gems, without much effort. Every gem vendored would need to be reviewed (changelog and license) for sure, just like others. So I give Node vendoring +1 and both B1 and B2 proposals also +1 +1. LZ On Tue, Oct 17, 2017 at 9:56 AM, Tomer Brisker wrote: > Thanks for bringing up the discussion! > > As one who has been bitten multiple times by the issue of building RPMs for > node modules, a definite +1 (or more like +1000) from me to vendoring the > entire node_modules directory. > This will save a lot of time and effort on the part of maintainers that will > no longer have to wrestle with koji for every module, as well as reduce the > load on the builders since we won't be wasting cycles building RPMs for the > sole purpose of consuming them during our assets compilation. > > However, I'm not sure if this approach should also apply to ruby gems. In > general, the number of gems and the frequency in which they are updates are > both lower, and unlike the node RPMs they are actually used during runtime > and not only during build time, making sense for them to be separate RPMs > that can be in some cases even be upgraded without upgrading the core > packages (which again doesn't apply to node modules which will require a > recompile anyways). > > On Mon, Oct 16, 2017 at 4:50 PM, Michael Moll wrote: >> >> Hi, >> >> On Mon, Oct 16, 2017 at 02:36:27PM +0100, Greg Sutcliffe wrote: >> > > Ruby Gems >> > > >> > > Option 2, to vendorize, is a deviation from our prior practices in >> > > the area of production deployment. Thus, I am reaching out to the >> > > community to get feedback. One interesting consideration for >> > > vendorizing is when containers are considered and having the ability >> > > to build them using 'bundle install' versus using RPM based >> > > installation. >> > >> > Vendoring hasn't (to my knowledge) caused many issues for Debian users >> > (Michael?), and having consistent build processes makes it easier for >> > anyone to support users on different OSs. >> >> From a user's perspective it's not that bad, but IMHO the current >> approach does have some downsides regarding plugins and I'm unsure if a >> big plugin like Katello can be handled by the current DEB way of doing >> things. >> >> Regards >> -- >> Michael Moll >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > > > -- > Have a nice day, > Tomer Brisker > Red Hat Engineering > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
Thanks for bringing up the discussion! As one who has been bitten multiple times by the issue of building RPMs for node modules, a definite +1 (or more like +1000) from me to vendoring the entire node_modules directory. This will save a lot of time and effort on the part of maintainers that will no longer have to wrestle with koji for every module, as well as reduce the load on the builders since we won't be wasting cycles building RPMs for the sole purpose of consuming them during our assets compilation. However, I'm not sure if this approach should also apply to ruby gems. In general, the number of gems and the frequency in which they are updates are both lower, and unlike the node RPMs they are actually used during runtime and not only during build time, making sense for them to be separate RPMs that can be in some cases even be upgraded without upgrading the core packages (which again doesn't apply to node modules which will require a recompile anyways). On Mon, Oct 16, 2017 at 4:50 PM, Michael Moll wrote: > Hi, > > On Mon, Oct 16, 2017 at 02:36:27PM +0100, Greg Sutcliffe wrote: > > > Ruby Gems > > > > > > Option 2, to vendorize, is a deviation from our prior practices in > > > the area of production deployment. Thus, I am reaching out to the > > > community to get feedback. One interesting consideration for > > > vendorizing is when containers are considered and having the ability > > > to build them using 'bundle install' versus using RPM based > > > installation. > > > > Vendoring hasn't (to my knowledge) caused many issues for Debian users > > (Michael?), and having consistent build processes makes it easier for > > anyone to support users on different OSs. > > From a user's perspective it's not that bad, but IMHO the current > approach does have some downsides regarding plugins and I'm unsure if a > big plugin like Katello can be handled by the current DEB way of doing > things. > > Regards > -- > Michael Moll > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Have a nice day, Tomer Brisker Red Hat Engineering -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
Hi, On Mon, Oct 16, 2017 at 02:36:27PM +0100, Greg Sutcliffe wrote: > > Ruby Gems > > > > Option 2, to vendorize, is a deviation from our prior practices in > > the area of production deployment. Thus, I am reaching out to the > > community to get feedback. One interesting consideration for > > vendorizing is when containers are considered and having the ability > > to build them using 'bundle install' versus using RPM based > > installation. > > Vendoring hasn't (to my knowledge) caused many issues for Debian users > (Michael?), and having consistent build processes makes it easier for > anyone to support users on different OSs. >From a user's perspective it's not that bad, but IMHO the current approach does have some downsides regarding plugins and I'm unsure if a big plugin like Katello can be handled by the current DEB way of doing things. Regards -- Michael Moll -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
On Sat, 2017-10-14 at 14:24 -0400, Eric D Helms wrote: > Note: This is specific to RPMs because as far as I know the Debian > process is different and uses gems directly. Please correct me and > contribute any information with respect to Debian that I miss. I > believe the Nodejs part of the email does apply to Debian. As Michael says, I think we can mostly leave Debs out, at least for now. > Node/NPM Packages ... Proposal: > Deprecate nodejs packages in favor of foreman-assets or a new RPM > foreman-node-modules that contains a source tarball of node_modules/ > packaged into a simple RPM that is used as a build dependency. This > tarball would be regenerated, and the package bumped, as dependency > updates are needed. If we go for option 2 below, then vendoring an npm install at build time (as Michael said, we do this for Debian) feels like a better option. If we go for building our own SCL, then +1 for this. > Ruby Gems > > Option 2, to vendorize, is a deviation from our prior practices in > the area of production deployment. Thus, I am reaching out to the > community to get feedback. One interesting consideration for > vendorizing is when containers are considered and having the ability > to build them using 'bundle install' versus using RPM based > installation. Vendoring hasn't (to my knowledge) caused many issues for Debian users (Michael?), and having consistent build processes makes it easier for anyone to support users on different OSs. That said, I've not really been involved with the RPMs, so I'm unsure if this causes a bigger headache for Yum users than Apt users. I'm also unsure of the work required to create an SCL, but if it's non-trivial then I'd be looking to CentOS to collaborate on a Rails SCL for everyone to use - for just ourselves, then vendoring seems easier. Greg -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Vendorizing or Building RPMs
Hi, On Sat, Oct 14, 2017 at 02:24:58PM -0400, Eric D Helms wrote: > Node/NPM Packages > > [...] > > Proposal: > Deprecate nodejs packages in favor of foreman-assets or a new RPM > foreman-node-modules that contains a source tarball of node_modules/ > packaged into a simple RPM that is used as a build dependency. This tarball > would be regenerated, and the package bumped, as dependency updates are > needed. I do see a problem to apply this to Debian. There are "native extensions" (or whatever is the correct term in the Node world), used i.e. by node-sass that differ by architecture and Node API versionn. Gor Debian we're building on various architectures and also it's not 100% sure the same Node version is used (esp. it's often a different version than the one coming from EPEL). However, I don't see a problem with just leaving the Debian build process as (which is just a npm install at package build time). > Today we package all required rubygems as RPMs and utilize a thirdparty > Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack > (sclo-ror42). The team that manages the RHSCLs has already released Ruby > 2.3 and RUby 2.4 SCLs and will continue to do so. However, there are no > plans to release any further SCLs for the Rails stack. This means, in > addition to our own dependencies, we need to build and manage the Rails > stack for the versions we want. This adds more packaging burden on the RPM > side. GIven this, we have a few options: > >1) Build and manage our own SCLs for every version of Rails from here on > out >2) Vendorize Rails and thirdparty gems into a one or more base RPMs > (e.g. an RPM for rails stack and one for thirdparty dependencies) I don't really have stakes in RPMs, but I'd try to stick with option 1 and I even expect that there's some demand for Rails SCLs at least in the RHEL/CentOS world by other projects, too. Regards -- Michael Moll -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[foreman-dev] Vendorizing or Building RPMs
Note: This is specific to RPMs because as far as I know the Debian process is different and uses gems directly. Please correct me and contribute any information with respect to Debian that I miss. I believe the Nodejs part of the email does apply to Debian. This proposal is in two parts: gems and node packages. The general issue applies to both, the project has a need to be able to move and update dependencies more quickly given the speed at which platforms like Rails, React and Webpack update. Node/NPM Packages Today, we package nearly all nodejs packages that are required to either build the UI or runtime UI dependencies into packages. At current count, there are 78 nodejs packages listed in Koji as RPMs. Over the past few months, the move to React and updating UI within Foreman core and branching into plugins has led to multiple, and at times, compounding packaging and thus nightly breakages. One of those breakages lasted over a week with multiple developers spending many hours tracking down the issue. The speed at which these changes are made makes keeping up successful nightly builds difficult. The git repository gets well ahead of the packaging which leads to compounding issues. This has also been the cause of stable version release delays. Proposal: Deprecate nodejs packages in favor of foreman-assets or a new RPM foreman-node-modules that contains a source tarball of node_modules/ packaged into a simple RPM that is used as a build dependency. This tarball would be regenerated, and the package bumped, as dependency updates are needed. Ruby Gems Today we package all required rubygems as RPMs and utilize a thirdparty Software Collection (SCL) for both the Ruby (rh-ruby22) and Rails stack (sclo-ror42). The team that manages the RHSCLs has already released Ruby 2.3 and RUby 2.4 SCLs and will continue to do so. However, there are no plans to release any further SCLs for the Rails stack. This means, in addition to our own dependencies, we need to build and manage the Rails stack for the versions we want. This adds more packaging burden on the RPM side. GIven this, we have a few options: 1) Build and manage our own SCLs for every version of Rails from here on out 2) Vendorize Rails and thirdparty gems into a one or more base RPMs (e.g. an RPM for rails stack and one for thirdparty dependencies) Option 2, to vendorize, is a deviation from our prior practices in the area of production deployment. Thus, I am reaching out to the community to get feedback. One interesting consideration for vendorizing is when containers are considered and having the ability to build them using 'bundle install' versus using RPM based installation. In theory vendorizing allows the community to move faster, keep up to date with dependencies more easily and reduces the burden on RPM packaging to keep nightly builds flowing. However, this does stray from the standard RPM based installation benefits typically associated to it. Please consider carefully, and weigh in even if it is simply to +1 or -1 a given idea. Deployment is a large part of our project and the more voices weighing in the better. Thanks, Eriic -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
