Re: [openstack-dev] [nova] gerrit based statistics
On Wed, Oct 08, 2014 at 04:30:16PM -0700, Joe Gordon wrote: Recently there has been a lot of discussion around the development growing pains in nova. Instead of guessing about how bad some of the issues are, I tried to answer a few questions that may help us better understand the issues. What time period did you calculate these stats over ? One whole dev cycle ? Q: How many revisions does it take to merge a patch? Average: 6.76 revisions median: 4.0 revisions Q: How many rechecks/verifies does it take to merge a patch (ignoring rechecks where the same job failed before and after)? Average: 0.749 rechecks per patch revision median: 0.4285 rechecks per patch revision These feel really low to me, but I guess that is a consequence of aggregating over a long time period. For comparison here are the same results for tempest, which has a lot more gating tests: Average: 1.01591525738 median: 0.6 Q: How long does it take for a patch to get approved? Average: 28 days median: 11 days Q: How long does it take for a patch to get approved that touches 'nova/virt/'? Average: 34 days median: 18 days When looking at these numbers two things stick out out: * We successfully use recheck an awful lot. More then I expected * Patches that touch 'nova/virt' take about 20% more time to land or about 6 days. While that is definitely a difference, its smaller then I expected Dataset: last 800 patches in nova Code: https://github.com/jogo/gerrit-fun Ah ok, so I think what would be more informative is to caculate the stats on weekly blocks and graph it over 2 cycles. There are certainly times when the problems are much worse than others and a long term average will mask that to a large extent. It is the periods when it is particularly bad that are leading to the high frustration amongst contributors IMHO, so we must look at the worst cases. For example look at the stats I produced for patch review comments per core team reviewer: https://www.berrange.com/~dan/novacodeandspecsreviewrate.png The gap betweeen good weeks and bad weeks is enourmous. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] gerrit based statistics
On 10/08/2014 07:30 PM, Joe Gordon wrote: Recently there has been a lot of discussion around the development growing pains in nova. Instead of guessing about how bad some of the issues are, I tried to answer a few questions that may help us better understand the issues. Q: How many revisions does it take to merge a patch? Average: 6.76 revisions median: 4.0 revisions Q: How many rechecks/verifies does it take to merge a patch (ignoring rechecks where the same job failed before and after)? Average: 0.749 rechecks per patch revision median: 0.4285 rechecks per patch revision For comparison here are the same results for tempest, which has a lot more gating tests: Average: 1.01591525738 median: 0.6 Q: How long does it take for a patch to get approved? Average: 28 days median: 11 days Q: How long does it take for a patch to get approved that touches 'nova/virt/'? Average: 34 days median: 18 days When looking at these numbers two things stick out out: * We successfully use recheck an awful lot. More then I expected * Patches that touch 'nova/virt' take about 20% more time to land or about 6 days. While that is definitely a difference, its smaller then I expected Dataset: last 800 patches in nova Code: https://github.com/jogo/gerrit-fun Some related stats on open code reviews: http://russellbryant.net/openstack-stats/nova-openreviews.html I don't have historical data, which would be really useful. However, based on my memory and an old ML post [1], these numbers have tripled for Nova since mid 2013. [1] http://lists.openstack.org/pipermail/openstack-dev/2013-June/011043.html -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] gerrit based statistics
On 10/8/2014 9:21 PM, Joe Gordon wrote: On Wed, Oct 8, 2014 at 6:12 PM, Michael Still mi...@stillhq.com mailto:mi...@stillhq.com wrote: On Thu, Oct 9, 2014 at 11:58 AM, Joe Gordon joe.gord...@gmail.com mailto:joe.gord...@gmail.com wrote: On Wed, Oct 8, 2014 at 4:30 PM, Joe Gordon joe.gord...@gmail.com mailto:joe.gord...@gmail.com wrote: Recently there has been a lot of discussion around the development growing pains in nova. Instead of guessing about how bad some of the issues are, I tried to answer a few questions that may help us better understand the issues. Q: How many revisions does it take to merge a patch? Average: 6.76 revisions median: 4.0 revisions Q: How many rechecks/verifies does it take to merge a patch (ignoring rechecks where the same job failed before and after)? Average: 0.749 rechecks per patch revision median: 0.4285 rechecks per patch revision For comparison here are the same results for tempest, which has a lot more gating tests: Average: 1.01591525738 median: 0.6 Q: How long does it take for a patch to get approved? Average: 28 days median: 11 days Q: How long does it take for a patch to get approved that touches 'nova/virt/'? Average: 34 days median: 18 days To expand on these numbers, same results for last 6 months of commits: all of nova (1723 patches): Average: 28.8 median: 11.0 nova/virt (476 patches): Average: 34.5 I think it would be interesting to break this up by driver directory... Are there drivers which take longer to land code for than others? Like this? subtree: None (1724 patches): Average: 28.7 median: 11.0 subtree: nova/virt/ (476 patches): Average: 34.5 median: 18.0 subtree: nova/virt/hyperv/ (38 patches): Average: 46.8 median: 33.0 subtree: nova/virt/libvirt/ (224 patches): Average: 35.9 median: 18.0 subtree: nova/virt/xenapi/ (72 patches): Average: 39.5 median: 20.0 subtree: nova/virt/vmwareapi/ (134 patches): Average: 38.7 median: 26.0 When looking at these numbers two things stick out out: * We successfully use recheck an awful lot. More then I expected * Patches that touch 'nova/virt' take about 20% more time to land or about 6 days. While that is definitely a difference, its smaller then I expected Dataset: last 800 patches in nova Code:https://github.com/jogo/gerrit-fun ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Michael -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev That's the distribution, percentage-wise, I'd expect to see for the 4 in-tree virt drivers. -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] gerrit based statistics
Recently there has been a lot of discussion around the development growing pains in nova. Instead of guessing about how bad some of the issues are, I tried to answer a few questions that may help us better understand the issues. Q: How many revisions does it take to merge a patch? Average: 6.76 revisions median: 4.0 revisions Q: How many rechecks/verifies does it take to merge a patch (ignoring rechecks where the same job failed before and after)? Average: 0.749 rechecks per patch revision median: 0.4285 rechecks per patch revision For comparison here are the same results for tempest, which has a lot more gating tests: Average: 1.01591525738 median: 0.6 Q: How long does it take for a patch to get approved? Average: 28 days median: 11 days Q: How long does it take for a patch to get approved that touches 'nova/virt/'? Average: 34 days median: 18 days When looking at these numbers two things stick out out: * We successfully use recheck an awful lot. More then I expected * Patches that touch 'nova/virt' take about 20% more time to land or about 6 days. While that is definitely a difference, its smaller then I expected Dataset: last 800 patches in nova Code: https://github.com/jogo/gerrit-fun ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] gerrit based statistics
On Wed, Oct 8, 2014 at 4:30 PM, Joe Gordon joe.gord...@gmail.com wrote: Recently there has been a lot of discussion around the development growing pains in nova. Instead of guessing about how bad some of the issues are, I tried to answer a few questions that may help us better understand the issues. Q: How many revisions does it take to merge a patch? Average: 6.76 revisions median: 4.0 revisions Q: How many rechecks/verifies does it take to merge a patch (ignoring rechecks where the same job failed before and after)? Average: 0.749 rechecks per patch revision median: 0.4285 rechecks per patch revision For comparison here are the same results for tempest, which has a lot more gating tests: Average: 1.01591525738 median: 0.6 Q: How long does it take for a patch to get approved? Average: 28 days median: 11 days Q: How long does it take for a patch to get approved that touches 'nova/virt/'? Average: 34 days median: 18 days To expand on these numbers, same results for last 6 months of commits: all of nova (1723 patches): Average: 28.8 median: 11.0 nova/virt (476 patches): Average: 34.5 When looking at these numbers two things stick out out: * We successfully use recheck an awful lot. More then I expected * Patches that touch 'nova/virt' take about 20% more time to land or about 6 days. While that is definitely a difference, its smaller then I expected Dataset: last 800 patches in nova Code: https://github.com/jogo/gerrit-fun ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] gerrit based statistics
On Wed, Oct 8, 2014 at 6:12 PM, Michael Still mi...@stillhq.com wrote: On Thu, Oct 9, 2014 at 11:58 AM, Joe Gordon joe.gord...@gmail.com wrote: On Wed, Oct 8, 2014 at 4:30 PM, Joe Gordon joe.gord...@gmail.com wrote: Recently there has been a lot of discussion around the development growing pains in nova. Instead of guessing about how bad some of the issues are, I tried to answer a few questions that may help us better understand the issues. Q: How many revisions does it take to merge a patch? Average: 6.76 revisions median: 4.0 revisions Q: How many rechecks/verifies does it take to merge a patch (ignoring rechecks where the same job failed before and after)? Average: 0.749 rechecks per patch revision median: 0.4285 rechecks per patch revision For comparison here are the same results for tempest, which has a lot more gating tests: Average: 1.01591525738 median: 0.6 Q: How long does it take for a patch to get approved? Average: 28 days median: 11 days Q: How long does it take for a patch to get approved that touches 'nova/virt/'? Average: 34 days median: 18 days To expand on these numbers, same results for last 6 months of commits: all of nova (1723 patches): Average: 28.8 median: 11.0 nova/virt (476 patches): Average: 34.5 I think it would be interesting to break this up by driver directory... Are there drivers which take longer to land code for than others? Like this? subtree: None (1724 patches): Average: 28.7 median: 11.0 subtree: nova/virt/ (476 patches): Average: 34.5 median: 18.0 subtree: nova/virt/hyperv/ (38 patches): Average: 46.8 median: 33.0 subtree: nova/virt/libvirt/ (224 patches): Average: 35.9 median: 18.0 subtree: nova/virt/xenapi/ (72 patches): Average: 39.5 median: 20.0 subtree: nova/virt/vmwareapi/ (134 patches): Average: 38.7 median: 26.0 When looking at these numbers two things stick out out: * We successfully use recheck an awful lot. More then I expected * Patches that touch 'nova/virt' take about 20% more time to land or about 6 days. While that is definitely a difference, its smaller then I expected Dataset: last 800 patches in nova Code: https://github.com/jogo/gerrit-fun ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Michael -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
