Comments in-line On Fri, Jul 13, 2018 at 11:16 PM, Paul Belanger <[email protected]> wrote: > On Fri, Jul 13, 2018 at 09:27:23PM +0200, Matthieu Huin wrote: >> Hello, >> > Thanks for starting this discussion, I believe it to be important to the > future > relationship between SF and Zuul communities.
Indeed, we definitely strive to become model citizens in the communities. >> I would like to poke the collective brainpower about the way we ship >> upstream components to which we also contribute, namely zuul and >> nodepool, and see what we can do to improve if possible. >> >> As you probably all know, we choose to package these components as >> RPMs, which allows us to ship zuul and nodepool with extra patches >> that we call *tech previews*. These patches are always related to >> features we are either contributing upstream ourselves or closely >> follow, but that are not merged yet. The important point is also that >> we have some relative confidence that these patches will eventually >> make it into upstream. Unfortunately, "relative" and "eventually" must >> be taken with a grain of salt: >> > My main objection to this concept of 'Tech Preview' is this follows more the > development processes of OpenShift / Kubernetes and not OpenStack[1] (NOTE: > zuul > isn't actually an OpenStack project anymore, but governance likely will follow > the 4 opens of OpenStack). We don't want to be releasing things a head of > zuul, > I point to the issues between the k8s and openshift projects. > > The biggest red flag for me, is downstream Red Hat does not do this process > for > OSP (Red Hat product for OpenStack). Teams are committed to push code into > master first, then either make an release from it or in some cases backport > into > a downstream branch. I am 100% confident these teams had the same struggles we > have highlighted before and would like to us to may query some folk about how > they solved this issue. Because the latest version of OSP does not contain > downstream features outside of upstream master branches. > > By having SoftwareFactory do the opposite of this, it may mean new features > faster, but as you point out below, now means more technical debt and worst, > meaning SF is only people on the hook to support this 'tech release' version. > Give the size and workload of the current team, I assert we don't want to do > this just for this reason. By shipping something different then upstream, I > also > don't believe upstream will be happy with this. On the other hand, it is also an opportunity to demonstrate the added value or our contributions on a live example. Our deployments can be seen as a showcase for the tech preview features, which makes it easier to see why we'd like these considered for inclusion into master, and that they're functional as they are. Moreover, in the case of OCI support, this allowed people to experiment with Zuul without the need of an OpenStack cloud. This was understandably not a priority upstream at the time, but this was invaluable advertising for the project. > In fact, I encourage us to talk > the results of this discussion to the Zuul Discuss mailing list improving them > of this topic. This would be a great opportunity to work closer together as a > team, if we raised our concerns with them. This ML is public and the intent was to start the brainstorming on our side, then come to the community with ideas and possible solutions. >> 1. We usually cannot tell how long it'll take for our patches to land >> upstream. I don't have numbers to support it, but from memory It can >> vary between days (some bug fixes made it quickly into master >> upstream) to months (we've had OCI as tech preview for months, and >> it's still not merged upstream). Fabien or Tristan can certainly >> provide numbers on that point. >> > I took some time to run the numbers using reviewday[2] a tool we have in > openstack-infra, for today I've compared nova[3] with zuul[4] (using official > zuul projects[5]). As you can see, while does have a high openreview number, > it > is no where need nova. I also ran the numbers using tripleo[5] with zuul, and > again zuul does come out a head of tripleo here. Give the large number of > RedHat > developers to tripleo, I think it is fair dive more into here why a 'tech > preview' release of tripleo is not done. > > I can understand how it feels like zuul takes a long time for patches to land, > but I believe the fix here is to double down on our commitment upstream and > encourage more time for SF developers to be free to aid / assist with upstream > zuul development. This includes code reviews of other non related SF patches. We're getting there. You and Tristan are already well known and involved in the community, Fabien and I are starting to be where we can. But helping on reviews won't be enough to land our features faster if we can't also somehow weigh on the roadmap and priority efforts. As "outsiders" to upstream infra, even though we represent the RDO community in a way, how can we be taken into account? And how do we do that without looking like we're taking over? >> 2. We cannot guarantee that the feature in tech preview will land >> as-is. The upstream reviews are usually being discussed and can be >> subject to change, meaning users should not consider any of the tech >> preview features to be stable. >> > +1000 > > This extends with my comment above, and completely agree. This is a very large > concern as it means more work for only SF to deal with migration issues. > >> Looking at our distgit for Zuul, we currently ship 12 extra patches as >> tech preview (5 of which about to be merged or merged - thus the spec >> must be updated), and this is bound to increase if we keep >> contributing things faster than they can be reviewed and accepted >> upstream. It can become quite hard to maintain the patch chain as >> upstream evolves. We also face the very real risk that one of our use >> cases (and thus our upstream contributions) might contradict >> upstream's roadmap, leading to rejected patches: do we become a fork >> then ? Are we actually effectively a fork, providing a "Zuul that >> could be someday" but definitely not the current Zuul? >> > For me, the path forward here is staightforward, the patches that we have in > our > zuul RPM file, we keep, there is no point removing them now, as it will be > very > disruptive. We stop working on any new features that SF requires and spend the > effort working upstream to land these changes. This likely means, getting > approval from management to be allowed to help push on these efforts upstream. > > At the same time, we agree the SF workflow process will stop included these > patches in the RPM, and land everything upstream in master first. It isn't > enough to just submit the patch to review.o.o, we must be getting it merged > before including it into zuul RPM. >> Yet the tech preview system has obvious advantages that make it >> difficult to just drop this model, namely providing much needed >> features that make Zuul and Nodepool much more serious and versatile >> contenders in the world of CI, CD and code quality control - this is >> also why we believe our changes will eventually make it upstream. >> > This is important feedback for the zuul project, and something I think we > could > discuss upstream on the zuul discuss ML. If we as a team believe this is > important, then other zuul users upstream should too? Is there some form of user feedback system besides opening tickets on storyboard? Should we open a thread on the zuul discuss ML about what our users (and us as support engineers) need as we figure it out? >> I guess the question we need to answer is: why is it so hard or long >> to have upstream land the features we propose? And what can we do to >> change that? If we can improve on this, the need for patching will >> decrease until we can ship code as close as possible to master, or >> even tagged releases. >> >> What are your thoughts on this? >> > This is no limited to just zuul, I've seen this problem time and time again > for > other opensource projects. Patience is required, and we need to adjust our > expectations. If we want to influence change, we can and now is the time to do > so. However, we need to spend the time and energy doing so. This may mean > slowing down out feature development work to properly land what we have > already > done. During the retro, you mentioned that "[something I forgot but not very complex] could land relatively fast, three months from now". 90 days to land a simple feature is one of the main reasons we resort to this flawed "tech preview" system. I agree that doing more reviews would help, but we need a way to gently tug attention and upstream resources towards things we deem important. > > - Paul > > [1] https://governance.openstack.org/tc/reference/opens.html > [2] http://git.openstack.org/cgit/openstack-infra/reviewday > [3] http://paste.openstack.org/raw/725858/ > [4] http://paste.openstack.org/raw/725860/ > [5] https://git.zuul-ci.org/cgit _______________________________________________ Softwarefactory-dev mailing list [email protected] https://www.redhat.com/mailman/listinfo/softwarefactory-dev
