On Thu, Dec 15, 2016 at 12:22 PM, Sven Anderson <s...@redhat.com> wrote: > Hi all, > > while I was waiting again for the CI to be fixed and didn't want to > torture it with additional rechecks, I wanted to find out, how much of > our CI infrastructure we waste with rechecks. My assumption was that > every recheck is a waste of resources based on a false negative, because > it renders the previous build useless. So I wrote a small script[1] to > calculate how many rechecks are made on average per built patch-set. It > calculates the number of patch-sets of merged changes that CI was > testing (some patch-sets are not, because they were updated before CI > started testing), the number of rechecks issued on these patch-sets, and > a value "CI-factor", which is the factor by which the rechecks increased > the the CI runs, that is, without rechecks it would be 1, if every > tested patch-set would have exactly one recheck it would be 2.
I see 2 different topics here. # One is not related to $topic but still worth mentioning: "while I was waiting again for the CI to be fixed" This week has been tough, and many of us burnt our time to resolve different complex problems in TripleO CI, mostly related to external dependencies (qemu upgrade, centos 7.3 upgrade, tripleo-ci infra, etc). Resolving these problems is very challenging and you'll notice that only a few of us actually work on this task, while a lot of people continue to push their features "hoping" that it will pass CI sometimes and if not, well, we'll do 'recheck'. That is a way of working I would say. I personally can't continue to code if the project I'm working on has broken CI. In a previous experience, I've been working in a team where everyone stopped regular work when CI was broken and focus on fixing it. I'm not saying everyone should stop their tasks and help, but this "wait and see" comment doesn't actually help us to move forward. People need to get more involved in CI and be more helpful. I know it's difficult, but it's something anyone can learn, like you would learn how to write Python code for example. # The second one is about the actual $topic and your stats. Yes we have been thinking about a way to optimize the way we restart CI jobs and this is under discussion: https://review.openstack.org/#/c/411450/ As you can see, there is some pushback from Clark who is infra-core, so we might want to continue the discussion here and see how it goes. On the long-term, our goal is to have more consistency in the way we test TripleO and get more adoption in the tools we're developing for CI, so they are more consumable from anyone in our community. Also we hope to have more people involved when things are broken, and not always the same folks spending days and evenings to "extinguish fires". We are working hard on CI stabilization and consolidation with multinode scenarios and OVB improvements, but it takes time and iterations. Any help is highly welcome here. > The results were not as bad as my feeling, we are below 2 for most of > the projects I tested. :-) But still, on THT for instance we use 71% > more resources because of the false negatives. I made monthly > breakdowns, so you can see a positive trend at least. > > > Here the results: > > Project: tripleo-heat-templates > > month patches rechecks CI-factor > 1 221 102 1.46 > 2 282 300 2.06 > 3 588 567 1.96 > 4 220 253 2.15 > 5 333 242 1.73 > 6 459 325 1.71 > 7 612 390 1.64 > 8 694 442 1.64 > 9 717 440 1.61 > 10 474 316 1.67 > 11 358 189 1.53 > 12 168 80 1.48 > total 5126 3646 1.71 > > Project: tripleo-common > > month patches rechecks CI-factor > 1 73 29 1.4 > 2 59 48 1.81 > 3 92 101 2.1 > 4 17 19 2.12 > 5 47 27 1.57 > 6 83 46 1.55 > 7 66 26 1.39 > 8 209 102 1.49 > 9 261 129 1.49 > 10 110 51 1.46 > 11 121 47 1.39 > 12 40 19 1.48 > total 1178 644 1.55 > > Project: tripleo-puppet-elements > > month patches rechecks CI-factor > 1 24 9 1.38 > 2 9 20 3.22 > 3 7 16 3.29 > 4 9 24 3.67 > 5 14 17 2.21 > 6 17 33 2.94 > 7 12 16 2.33 > 8 15 21 2.4 > 9 10 14 2.4 > 10 12 5 1.42 > 11 34 25 1.74 > 12 10 13 2.3 > total 173 213 2.23 > > Project: puppet-tripleo > > month patches rechecks CI-factor > 1 29 23 1.79 > 2 36 68 2.89 > 3 40 44 2.1 > 4 68 74 2.09 > 5 129 43 1.33 > 6 265 206 1.78 > 7 235 118 1.5 > 8 193 130 1.67 > 9 147 123 1.84 > 10 233 159 1.68 > 11 137 86 1.63 > 12 20 5 1.25 > total 1532 1079 1.7 > > > [1] https://gist.github.com/ansiwen/e139cbf25bc243d30629e0157fc753ff > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Emilien Macchi __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
