On Thu, Jun 1, 2017 at 10:18 AM, Tomer Brisker <tbris...@redhat.com> wrote:
> Hello, > > While it is plugin maintainers responsibility to keep their plugins up to > date, one of the most common frustrations they face is "core changed and > broke something AGAIN". I wouldn't want to slow down core's already slow > merge process, but at the same time core contributors should try to > minimize breaking plugins as much as possible - for example, by deprecating > functions before removing them completely. We could make these tests > optional - i.e. not block merge if plugin tests fail, but just take > advantage of the tests to fix whatever is needed in the plugins sooner > rather then later. I know that in certain cases having katello test run on > every PR let us find and fix issues we wouldn't have known about otherwise. > Right now contributors (and reviewers) don't have any visibility to > whether a PR will cause problems to other plugins. If breaking is > unavoidable, at least having plugin tests run on PRs will allow giving a > "heads up" to plugin maintainers that they need to fix something, rather > then realizing after the next release that something broke. > During the past couple of months we had several cases of core PRs leading > to broken plugins, which could have been prevented or at least handled > better had we known that a certain change requires changes in plugins. > Agree! If I had to sum it up, developer happiness and innovation at speed (core and plugins) is key! One option we've mentioned before that goes directly to Tomer's point is a stable plugin API. We as a community both develop and tell other developers, come, build a plugin. We have a vibrant plugin community. We say plugins are central to the community and project yet CI/CD tells a different story. Do we have a defined contract for plugins? That would solve a lot of the issues IMO. If we had a supported interface (for ALL major use cases of extension, including the UI, since we do have a Foreman::Plugin today) then core could make changes at will as long as the interface passed testing. Any plugin using functionality not in the interface would be SOL but it would be explicitly stated and known. Once we get to installation, running in production mode, scenario testing things get dicier, less stable and with less testing. A good place to invest and bring in other communities (hint: Satlelite QE) to bolster. I'm writing a lot of words without action items which I typically try to avoid. I want to get our testing and matrices there and am working to bring about more time for myself and others to do so. We need to actively invest in this area, make it a priority, and a first class citizen of our community to make an impact. We have wiki pages on creating and releasing plugins. But maybe we need to step back and define what a plugin is to our community. I think it would also be a fruitful effort to examine the landscape of plugins and see how and where they are creating "interfaces" to Foreman. Eric > > As for which plugins to test - I think the best criteria would be to test > the 5 or 10 most downloaded plugins or most used plugins according to the > community survey. > > On Thu, Jun 1, 2017 at 4:50 PM, Lukas Zapletal <l...@redhat.com> wrote: > >> On Thu, Jun 1, 2017 at 2:55 PM, Eric D Helms <ericdhe...@gmail.com> >> wrote: >> > This sounds good in theory, but I'll refresh the history of why we did >> this. >> > A change would be made to Foreman core, that change would go in and then >> > propagate out. A developer would update their foreman checkout or the >> > Katello pipeline would run and break. More developers start updating >> local >> > checkouts and now everyone is broken. The Katello PR tests begin to >> fail and >> > all PRs are frozen. So now a significant portion of developers are all >> > trying to figure out what point in time to roll back to, who is going >> to fix >> > the breakage and how long it is going to take. 1-5 days later, the >> issue is >> > fixed and PRs are able to start back up as well as the nightly pipeline. >> > >> > How much developer frustration was created? How much time was lost and >> > wasted? If we want core to have more freedom of movement then core >> needs to >> > evolve to allow plugins more reliability on interfacing. >> >> Then let's don't block PRs, I propose nightly job with an email so we >> are at least aware that things do not work and we can at least wait >> with RC phase a little bit longer until things settle down, or even >> block doing another RC until we are all green. > > You lost me here. Blocking PRs is the only way that we reduced developer frustration. Not blocking means broken code, broken installs, broken developer environments. Say core merges something that removes a function Katello relied on. But we did not run Katello unit tests or block at all. Now a significant chunk of developers can't even boot their application. Meanwhile, we didn't block PRs due to failing tests, so more test start failing and getting lost in the sea of brokenness. Now we've got a compounding pile up. Ask any developer who has been on Katello bats duty (aka maintaining the nightly pipeline). Because of a lack of gating, there have been times where it takes a developer two weeks to get the pipeline back up and running because of compounding breakages. Two weeks lost. > >> > I am all for more testing, but what good does testing do if there are no >> > consequences? A nightly test that runs and either passes or fails is >> > informative and requires someone to be looking and willing to act on >> every >> > change. If the test has no consequence on deployment that likelihood >> drops >> > dramatically in my opinion. I think about this a lot and how it would be >> >> Of course, but we can add an item to the release process workflow to >> check if nightly unit tests are all green across give set of plugins. >> Current way of releasing Foreman is to do two-three RCs and then go >> green, plugin authors are trying to catch up later. So if you are >> beginner user and install Foreman today, Discovery won't work as >> expected. >> >> It does not need to be a jenkins job, does anybody have a script that >> performs unit test run of foreman+katello+some plugins? >> >> -- >> Later, >> Lukas @lzap Zapletal >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Have a nice day, > Tomer Brisker > Red Hat Engineering > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Eric D. Helms Red Hat Engineering -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
