On Fri, Jul 13, 2018 at 09:27:23PM +0200, Matthieu Huin wrote: > Hello, > Thanks for starting this discussion, I believe it to be important to the future relationship between SF and Zuul communities.
> I would like to poke the collective brainpower about the way we ship > upstream components to which we also contribute, namely zuul and > nodepool, and see what we can do to improve if possible. > > As you probably all know, we choose to package these components as > RPMs, which allows us to ship zuul and nodepool with extra patches > that we call *tech previews*. These patches are always related to > features we are either contributing upstream ourselves or closely > follow, but that are not merged yet. The important point is also that > we have some relative confidence that these patches will eventually > make it into upstream. Unfortunately, "relative" and "eventually" must > be taken with a grain of salt: > My main objection to this concept of 'Tech Preview' is this follows more the development processes of OpenShift / Kubernetes and not OpenStack[1] (NOTE: zuul isn't actually an OpenStack project anymore, but governance likely will follow the 4 opens of OpenStack). We don't want to be releasing things a head of zuul, I point to the issues between the k8s and openshift projects. The biggest red flag for me, is downstream Red Hat does not do this process for OSP (Red Hat product for OpenStack). Teams are committed to push code into master first, then either make an release from it or in some cases backport into a downstream branch. I am 100% confident these teams had the same struggles we have highlighted before and would like to us to may query some folk about how they solved this issue. Because the latest version of OSP does not contain downstream features outside of upstream master branches. By having SoftwareFactory do the opposite of this, it may mean new features faster, but as you point out below, now means more technical debt and worst, meaning SF is only people on the hook to support this 'tech release' version. Give the size and workload of the current team, I assert we don't want to do this just for this reason. By shipping something different then upstream, I also don't believe upstream will be happy with this. In fact, I encourage us to talk the results of this discussion to the Zuul Discuss mailing list improving them of this topic. This would be a great opportunity to work closer together as a team, if we raised our concerns with them. > 1. We usually cannot tell how long it'll take for our patches to land > upstream. I don't have numbers to support it, but from memory It can > vary between days (some bug fixes made it quickly into master > upstream) to months (we've had OCI as tech preview for months, and > it's still not merged upstream). Fabien or Tristan can certainly > provide numbers on that point. > I took some time to run the numbers using reviewday[2] a tool we have in openstack-infra, for today I've compared nova[3] with zuul[4] (using official zuul projects[5]). As you can see, while does have a high openreview number, it is no where need nova. I also ran the numbers using tripleo[5] with zuul, and again zuul does come out a head of tripleo here. Give the large number of RedHat developers to tripleo, I think it is fair dive more into here why a 'tech preview' release of tripleo is not done. I can understand how it feels like zuul takes a long time for patches to land, but I believe the fix here is to double down on our commitment upstream and encourage more time for SF developers to be free to aid / assist with upstream zuul development. This includes code reviews of other non related SF patches. > 2. We cannot guarantee that the feature in tech preview will land > as-is. The upstream reviews are usually being discussed and can be > subject to change, meaning users should not consider any of the tech > preview features to be stable. > +1000 This extends with my comment above, and completely agree. This is a very large concern as it means more work for only SF to deal with migration issues. > Looking at our distgit for Zuul, we currently ship 12 extra patches as > tech preview (5 of which about to be merged or merged - thus the spec > must be updated), and this is bound to increase if we keep > contributing things faster than they can be reviewed and accepted > upstream. It can become quite hard to maintain the patch chain as > upstream evolves. We also face the very real risk that one of our use > cases (and thus our upstream contributions) might contradict > upstream's roadmap, leading to rejected patches: do we become a fork > then ? Are we actually effectively a fork, providing a "Zuul that > could be someday" but definitely not the current Zuul? > For me, the path forward here is staightforward, the patches that we have in our zuul RPM file, we keep, there is no point removing them now, as it will be very disruptive. We stop working on any new features that SF requires and spend the effort working upstream to land these changes. This likely means, getting approval from management to be allowed to help push on these efforts upstream. At the same time, we agree the SF workflow process will stop included these patches in the RPM, and land everything upstream in master first. It isn't enough to just submit the patch to review.o.o, we must be getting it merged before including it into zuul RPM. > Yet the tech preview system has obvious advantages that make it > difficult to just drop this model, namely providing much needed > features that make Zuul and Nodepool much more serious and versatile > contenders in the world of CI, CD and code quality control - this is > also why we believe our changes will eventually make it upstream. > This is important feedback for the zuul project, and something I think we could discuss upstream on the zuul discuss ML. If we as a team believe this is important, then other zuul users upstream should too? > I guess the question we need to answer is: why is it so hard or long > to have upstream land the features we propose? And what can we do to > change that? If we can improve on this, the need for patching will > decrease until we can ship code as close as possible to master, or > even tagged releases. > > What are your thoughts on this? > This is no limited to just zuul, I've seen this problem time and time again for other opensource projects. Patience is required, and we need to adjust our expectations. If we want to influence change, we can and now is the time to do so. However, we need to spend the time and energy doing so. This may mean slowing down out feature development work to properly land what we have already done. - Paul [1] https://governance.openstack.org/tc/reference/opens.html [2] http://git.openstack.org/cgit/openstack-infra/reviewday [3] http://paste.openstack.org/raw/725858/ [4] http://paste.openstack.org/raw/725860/ [5] https://git.zuul-ci.org/cgit _______________________________________________ Softwarefactory-dev mailing list [email protected] https://www.redhat.com/mailman/listinfo/softwarefactory-dev
