Re: [openstack-dev] Hyper-V Meeting Minutes

[Joe Gordon]

Is there a plan to get Hyper-V CI working better? It looks like it is
failing significantly more frequently then Jenkins.

http://www.rcbops.com/gerrit/reports/nova-cireport.html


On Tue, Apr 15, 2014 at 9:25 AM, Peter Pouliot wrote:

>  Hi Everyone,
> Here are the minutes from today’s Hyper-V Meeting.
>
>  Minutes:
> http://eavesdrop.openstack.org/meetings/hyper_v/2014/hyper_v.2014-04-15-16.02.html
> Minutes (text):
> http://eavesdrop.openstack.org/meetings/hyper_v/2014/hyper_v.2014-04-15-16.02.txt
> Log:
> http://eavesdrop.openstack.org/meetings/hyper_v/2014/hyper_v.2014-04-15-16.02.log.html
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Russell Bryant]

On 10/16/2013 06:59 AM, Thierry Carrez wrote:
> Alessandro Pilotti wrote:
>> On Oct 16, 2013, at 13:19 , Thierry Carrez >> The other two alternatives are to accept the delays and work within Nova
>>> (slowly building the trust that will give you more autonomy), or ship it
>>> as a separate add-on that does not come with nova-core's signature on it.
>>
>> I never asked for a nova signature on it. My only requirerement is that
>> the project would be part of OpenStack and not an external project, even
>> if this means passing 2 releases in incubation on stackforge as long as
>> it can become part of the OpenStack core group of projects afterwards
>> (if it meets the required OpenStack criteria of course).
>>  https://wiki.openstack.org/wiki/Governance/NewProjects
> 
> That's a possible outcome of the second alternative I described above.
> The separate add-on could apply to the incubation track and potentially
> be made a part of the integrated release.

Yep, it's certainly a possible outcome.

You could ask the soon to be elected TC to give an opinion.  But
honestly, if I am on the TC, I would vote against it.  It doesn't make
any sense for Nova to include a bunch of drivers, but one driver be
separate but still an official project.

I think drivers need to be treated equally in this regard, and I think
the majority consensus is that it's best overall to have them in the tree.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[David Ripton]

On 10/16/2013 08:59 AM, Alessandro Pilotti wrote:


When somebody (especially a core reviewer) puts a -1 and a new patch is 
committed to address it,
I noticed that other reviewers wait for the guy that put the -1 to say 
something before +1/+2 it.

My feeling on this is that if somebody reviews a patch (positively or 
negatively) he/she should also
keep on with it (in a timely manner) until it is merged or clearly stating that 
there's no interest in reviewing it further.
This is especially true for core revs as other reviewers tend to be shy and 
avoid contradicting a core rev,
generating further delays.

What do you guys think?


Yeah, it's no fun when someone gives you a -1 then goes away.

But the people who do a lot of reviews do a lot of reviews, so they 
can't be immediately responsive to every change to every patch they've 
reviewed, or they'd never be able to do anything else.


The fundamental problem is that the ratio of patches to reviewers, and 
especially patches to core reviewers, is too high.  We either need 
people to submit fewer patches or do more reviewing.


I'm tempted to submit a patch to next-review to give priority to patches 
from authors who do a lot of reviews.  That would provide an incentive 
for everyone to review more.


--
David Ripton   Red Hat   drip...@redhat.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Daniel P. Berrange]

On Wed, Oct 16, 2013 at 06:51:50AM -0700, Dan Smith wrote:
> > +1 - I think we really want to have a strong preference for a stable 
> > api if we start separating parts out
> 
> So, as someone who is about to break the driver API all to hell over the
> next six months (er, I mean, make some significant changes), I can tell
> you that making it stable is the best way to kill velocity right now. We
> are a young project with a lot of work yet to do. Making the driver API
> stable at this point in the process, especially because just one driver
> wants to be out of tree, is going to be a huge problem.
> 
> > Otherwise we either end up with lots of pain in making
> > infrastructure changes or asymmetric gating which is to be avoided
> > wherever possible.
> 
> AFAICT, this is pain that would be experienced by the out-of-tree driver
> and pain which has been called out specifically by the authors as
> "better than the alternative".
> 
> Seriously, putting the brakes on the virt api right now because one
> driver wants to be out of tree is a huge problem. I fully support the
> hyper-v taking itself out-of-tree if it wants, but I don't think that
> means we can or should eject the others and move to a stable virt api.
> At least not anytime soon.

Agreed, it is way too premature to talk about the internal virt
API being declared even remotely stable. Personally I'd say it
should remain liable-to-change for the lifetime of the project,
because the ability to arbitrarily refactor internals of an app
is very valuable for ongoing maintenance IME.

We should be optimizing for what is best for the majority who
are doing their work collaboratively in-tree for OpenStack, not
a minority who wish to go their own way out of tree.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Christopher Yeoh]

On Thu, Oct 17, 2013 at 12:21 AM, Dan Smith  wrote:

> > +1 - I think we really want to have a strong preference for a stable
> > api if we start separating parts out
>
> So, as someone who is about to break the driver API all to hell over the
> next six months (er, I mean, make some significant changes), I can tell
> you that making it stable is the best way to kill velocity right now. We
> are a young project with a lot of work yet to do. Making the driver API
> stable at this point in the process, especially because just one driver
> wants to be out of tree, is going to be a huge problem.
>
>
Yes I agree. I just think if the internal API is not yet considered stable
its a sign we should
not be splitting the dependent bits out.


> > Otherwise we either end up with lots of pain in making
> > infrastructure changes or asymmetric gating which is to be avoided
> > wherever possible.
>
> AFAICT, this is pain that would be experienced by the out-of-tree driver
> and pain which has been called out specifically by the authors as
> "better than the alternative".
>
>
Yes, most of the pain will be felt by the out of tree driver. There may be
a small amount
on the nova side due to a reduction in the amount of immediate feedback if
a change breaks
something unexpectedly in a driver which is no longer integrated.

If a driver really wants to be out of tree then thats up to them, but it
doesn't mean we should
encourage or endorse it if its worse for the project overall.

Chris
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Dan Smith]

> +1 - I think we really want to have a strong preference for a stable 
> api if we start separating parts out

So, as someone who is about to break the driver API all to hell over the
next six months (er, I mean, make some significant changes), I can tell
you that making it stable is the best way to kill velocity right now. We
are a young project with a lot of work yet to do. Making the driver API
stable at this point in the process, especially because just one driver
wants to be out of tree, is going to be a huge problem.

> Otherwise we either end up with lots of pain in making
> infrastructure changes or asymmetric gating which is to be avoided
> wherever possible.

AFAICT, this is pain that would be experienced by the out-of-tree driver
and pain which has been called out specifically by the authors as
"better than the alternative".

Seriously, putting the brakes on the virt api right now because one
driver wants to be out of tree is a huge problem. I fully support the
hyper-v taking itself out-of-tree if it wants, but I don't think that
means we can or should eject the others and move to a stable virt api.
At least not anytime soon.

--Dan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Daniel P. Berrange]

On Wed, Oct 16, 2013 at 12:59:26PM +, Alessandro Pilotti wrote:
> 
> When somebody (especially a core reviewer) puts a -1 and a new patch is 
> committed to address it, 
> I noticed that other reviewers wait for the guy that put the -1 to say 
> something before +1/+2 it. 

I think that depends on the scope of the change the reviewer asked for. It is
normally easy for any other reviewer to identify whether the -1 was properly
addressed and as such there's no need to block on the original reviewer
adding +1. Any core reviewer should be capable of evaluating if a review
point was addressed. Only if the code change was fairly complicated and/or
controversial might it be worth blocking on the original reviewer. I tend to
take such a pragmatic approach when considering whether to wait for the original
reviewer to add a +1 or not.

> My feeling on this is that if somebody reviews a patch (positively or 
> negatively)
> he/she should also keep on with it (in a timely manner) until it is merged or
> clearly stating that there's no interest in reviewing it further. This is 
> especially
> true for core revs as other reviewers tend to be shy and avoid contradicting 
> a core
> rev, generating further delays. 

As above, I don't think we should block on waiting for original reviewers
to review followups, nor require them to, as it is an inefficient way
of working. Any core reviewer should be capable of reviewing any patch
at any stage of its life, unless it is a very controversial change. Forcing
reviewers to keep up with all versions of a patch will never work out in
practice whether we want it or not.

Non-core reviewers should be encouraged to speak up - by doing so it will
improve quality of reviewers and help us identify non-core reviewers who
are good candidates for promotion.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Alessandro Pilotti]

On Oct 16, 2013, at 15:16 , Sean Dague 
 wrote:

> On 10/16/2013 01:19 AM, Alessandro Pilotti wrote:
> 
>> 
>> Sean, you got "called out" in the meeting not because you asked to put a
>> refernce link to the specs which was perfectly reasonable, but because
>> after we did what you asked for in a timely manner, you didn't bother to
>> review the patch again until asked to please review it 6 days later!!!
>> 
>> This is a perfect example about why we need autonomy. We cannot leave a
>> patch starving in the review queue for a critical bug like that one!!
> 
> I -1ed the patch, you caught me on IRC and argued with me that the code 
> didn't need to change. You had my undivided attention there for 30 minutes on 
> this patch, but used the time to argue against change. So I moved on to other 
> things. Should I have gotten back around to my Nova review queue sooner, 
> sure. However once you made the fix I no longer had a -1 on the patch, so I 
> wasn't blocking it. And do I want to give up 30 minutes of my time every time 
> I try to review your patches because you'd rather argue than take feedback? 
> Not really. I still do it. But I'll admit, a patch author that gives me less 
> grief is a lot more fun to review. I'm only human in that regard.
> 

I beg for forgiveness for not obeying you on the spot and daring to discuss 
your -1, Master! ;-)

Jokes aside, this is actually bringing up another important point in the review 
system:

When somebody (especially a core reviewer) puts a -1 and a new patch is 
committed to address it, 
I noticed that other reviewers wait for the guy that put the -1 to say 
something before +1/+2 it. 

My feeling on this is that if somebody reviews a patch (positively or 
negatively) he/she should also
keep on with it (in a timely manner) until it is merged or clearly stating that 
there's no interest in reviewing it further.
This is especially true for core revs as other reviewers tend to be shy and 
avoid contradicting a core rev,
generating further delays. 

What do you guys think? 

Does it make sense to brainstorm constructively on a way to reduce the review 
lags? 
The review system itself is IMO already providing an excellent starting point, 
we just need to tweak it a bit. :-) 


> 14 days from bug filing to merge isn't starving - 
> (https://bugs.launchpad.net/nova/+bug/1233853). If it's such a critical bug, 
> how come it didn't expose until 4 weeks after feature freeze? If it was such 
> a critical bug how did it get past your internal review process and land in 
> tree in the first place? If it's such a critical bug why wasn't it brought up 
> at the weekly *nova* meeting?
> 

Because thanks to the gorgeus H3 phase we got all BPs merged together on the H3 
freeze deadline 
and only afterwards people had the opportunity to test that huge amoung of code 
and report bugs?

14 days is IMO a preposterously long wait when you have a dedicated team and a 
fix ready, 
but hey, it's a matter of perspective I guess. 

> I really feel like you continue down the path of special pleading, without 
> having used normal channels for things like this, which all exist. The nova 
> meeting is a great place to highlight reviews you feel are critical that need 
> eyes, and it happens every week on a regular schedule.

#OpenStack-Nova, ML and triaging bugs aren't normal channels?
For how it is going I should bring every single bug and patch to the meeting! 


> 
>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Sean Dague]

On 10/16/2013 01:45 AM, Vishvananda Ishaya wrote:

Hi Sean,

I'm going to top post because my response is general. I totally agree that we 
need people that understand the code base and we should encourage new people to 
be cross-functional. I guess my main issue is with how we get there. I believe 
in encouragment over punishment. In my mind giving people autonomy and control 
encourages them to contribute more.

In my opinion giving the driver developers control over their own code will 
lead to higher quality drivers. Yes, we risk integration issues, lack of test 
coverage, and buggy implementations, but it is my opinion that the increased 
velocity that the developers will enjoy will mean faster bug fixes and more 
opportunity to improve the drivers.


My experience reviewing a ton of driver code in grizzly makes me 
disagree 
(http://stackalytics.com/?release=grizzly&metric=marks&project_type=core&module=nova&company=&user_id=). 



Driver code tends to drift from the norm because the teams don't mix 
much with the rest of core. This makes it even harder to review their 
code because those teams aren't realizing what makes patches easy to 
review, by getting first hand experience reviewing lots of other 
people's code, and thinking to themselves "man, that was hard to wrap my 
head around, how would I do that better if it was my patch?".



I also think lowering the amount of code that nova-core has to keep an eye on 
will improve the review velocity of the rest of the code as well.


I think it would be at best a short term gain, completely offset by the 
fact that there are less eyes in the pool and I don't think would solve 
anything. If that were true the merge rate on smaller projects in 
OpenStack would far exceed Nova, and the numbers don't support that. My 
experience on a bunch of smaller trees that I've got +2 on are that 
review starvation actually hits them much worse.


So, again, it's about perspective.

Can we do better on review turn around? sure.

Would it be better if -core team members were spending more time on 
reviews? yes.


Would it be better if everyone spent time on reviews? yes.

Will driver teams getting real CI results posted back help? definitely.

Will an updated Gerrit that lets us do better dashboards so we don't 
loose reviews help? yes.


But OpenStack still moves crazy fast, and making process changes with 
sweeping future implications is something that needs to not be done 
lightly. And there are lots of other things to be done to make this 
better, which all kinds of people can help with.


-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Sean Dague]

On 10/16/2013 01:19 AM, Alessandro Pilotti wrote:



Sean, you got "called out" in the meeting not because you asked to put a
refernce link to the specs which was perfectly reasonable, but because
after we did what you asked for in a timely manner, you didn't bother to
review the patch again until asked to please review it 6 days later!!!

This is a perfect example about why we need autonomy. We cannot leave a
patch starving in the review queue for a critical bug like that one!!


I -1ed the patch, you caught me on IRC and argued with me that the code 
didn't need to change. You had my undivided attention there for 30 
minutes on this patch, but used the time to argue against change. So I 
moved on to other things. Should I have gotten back around to my Nova 
review queue sooner, sure. However once you made the fix I no longer had 
a -1 on the patch, so I wasn't blocking it. And do I want to give up 30 
minutes of my time every time I try to review your patches because you'd 
rather argue than take feedback? Not really. I still do it. But I'll 
admit, a patch author that gives me less grief is a lot more fun to 
review. I'm only human in that regard.


14 days from bug filing to merge isn't starving - 
(https://bugs.launchpad.net/nova/+bug/1233853). If it's such a critical 
bug, how come it didn't expose until 4 weeks after feature freeze? If it 
was such a critical bug how did it get past your internal review process 
and land in tree in the first place? If it's such a critical bug why 
wasn't it brought up at the weekly *nova* meeting?


I really feel like you continue down the path of special pleading, 
without having used normal channels for things like this, which all 
exist. The nova meeting is a great place to highlight reviews you feel 
are critical that need eyes, and it happens every week on a regular 
schedule.


-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Christopher Yeoh]

On Wed, Oct 16, 2013 at 8:49 PM, Thierry Carrez wrote:

> Sean Dague wrote:
> > The Linux kernel process works for a couple of reasons...
> >
> > 1) the subsystem maintainers have known each other for a solid decade
> > (i.e. 3x the lifespan of the OpenStack project), over a history of 10
> > years, of people doing the right things, you build trust in their
> judgment.
> >
> > *no one* in the Linux tree was given trust first, under the hope that it
> > would work out. They had to earn it, hard, by doing community work, and
> > not just playing in their corner of the world.
> >
> > 2) This
> > http://www.wired.com/wiredenterprise/2012/06/torvalds-nvidia-linux/ is
> > completely acceptable behavior. So when someone has bad code, they are
> > flamed to within an inch of their life, repeatedly, until they never
> > ever do that again. This is actually a time saving measure in code
> > review. It's a lot faster to just call people idiots then to help them
> > with line by line improvements in their code, 10, 20, 30, or 40
> > iterations in gerrit.
> >
> > We, as a community have decided, I think rightly, that #2 really isn't
> > in our culture. But you can't start cherry picking parts of the Linux
> > kernel community without considering how all the parts work together.
> > The good and the bad are part of why the whole system works.
>
> This is an extremely important point in that discussion.
>
> The Linux kernel model is built on a pyramidal model where Linus, in a
> PTL+Release Manager role, has the final ability to refuse whole sections
> of the code just because he doesn't like it.
>
> Over two decades, Linus built a solid trust relationship with most
> subsystem maintainers, so that he doesn't have to review every single
> patch for sanity. In those areas he has a set of people who consistently
> proved they would apply the same standards as he does. But for other
> less-trusted areas the pyramidal model is still working.
>
> I don't see how we could apply that to OpenStack as the trust
> relationships are far from being that advanced (think: not old enough),
> and I don't think we want to replicate the personified, pyramidal merge
> model to handle the less-trusted relationships in the mean time.
>
>
The other thing to note is that it isn't all sunshine and roses in linux
kernel development either. IMO the bar for trusted subsystem maintainers is
much much higher in linux kernel development than for core reviewer status
for openstack projects. Also patches can take a long time to get review
attention on linux-kernel with long gaps between feedback depending on how
busy the maintainer is. With gerrit I think we're actually very good at
keeping track of patches and they are much less likely to get completely
lost. We certainly have much better stats on how responsive we are to
proposed patches.

Chris
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Christopher Yeoh]

On Wed, Oct 16, 2013 at 6:49 PM, Robert Collins
wrote:

> On 16 October 2013 20:14, Alessandro Pilotti
>  wrote:>
> >
>
> > Drivers are IMO not part of the core of Nova, but completely separated
> and decoupled entities, which IMO should be treated that way. As a
> consequence, we frankly don't stricly feel as part of Nova, although some
> of us have a pretty strong understanding of how all the Nova pieces work.
>
> I don't have a particular view on whether they *should be* separate
> decoupled entities, but today I repeatedly hear concerns about the
> impact of treating 'internal APIs' as stable things. That's relevant
> because *if* nova drivers are to be separate decoupled entities, the
> APIs they use - and expose - have to be treated as stable things with
> graceful evolution, backwards compatibility etc. Doing anything else
> will lead to deployment issues and race conditions in the gate.
>
>
+1 - I think we really want to have a strong preference for a stable api if
we start separating parts out (and this has been the case in the past from
what I can see). Otherwise we either end up with lots of pain in making
infrastructure changes or asymmetric gating which is to be avoided wherever
possible.


> And by their effectiveness [this is more subjective:)]
>  - train more -core reviewers [essentially linear, very easy to predict]
>  - provide patches that are easier to review [many patches are good
> already, has a low upper bound on effectiveness]
>  - split the drivers out [won't help *at all* with changes required in
> core to support a driver feature]
>
>
I'd like to add to that, better tools (which will help both core and non
core reviewers). For example, rebase hell was mentioned in this thread. I
was in that a fair bit with the Nova v3 API changes where I'd have a long
series of dependent patches which would get fairly even review attention.
This sometimes had the unfortunate result that many in the series would end
up with a single +2. Not enough to merge, and the +2's would  get lost in
the inevitable rebase. Now perhaps as reviewers we should probably know
better to follow the dependency chain on reviews to review the changesets
with the least dependencies first, but we're only human and we don't always
remember to do that. So perhaps it'd be nice if gerrit or some other tool
showed changesets to review as a tree rather than a list. We might get more
changesets merged with the same number of reviews if the tools encouraged
the most efficient behaviour.

Another example is when you review a lot of patches the gerrit dashboard
doesn't seem to show all of the patches that you have reviewed. And I find
I get rather overwhelmed with the volume of email from gerrit with updates
of patches I've reviewed and so I find its not a great source of working
out what to review next. I'm sure I'm guilty of reviewing some patches and
then not getting back to them for a while because I've effectively lost
track of them (which is where an irc ping is appreciated). Perhaps better
tools could help here?

Chris
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Alessandro Pilotti]

Sorry guys about this, my OS X Mail client had no issues
in doing the proper indentation, so I never noticed it. Darn. 

I made a test with Daniel with a private email before spamming 
here for nothing. Hope it worked out here as well.

Thanks for the heads up.


On Oct 16, 2013, at 13:47 , "Daniel P. Berrange" 
 wrote:

> Alessandro, please fix your email program so that it does not send
> HTML email to the list, and correctly quotes text you are replying
> to with '> '. Your reply comes out looking like this which makes it
> impossible to see who wrote what:
> 
> On Wed, Oct 16, 2013 at 10:42:45AM +, Alessandro Pilotti wrote:
>> 
>> On Oct 16, 2013, at 13:19 , Thierry Carrez 
>> mailto:thie...@openstack.org>>
>> wrote:
>> 
>> Sean Dague wrote:
>> The Linux kernel process works for a couple of reasons...
>> 
>> 1) the subsystem maintainers have known each other for a solid decade
>> (i.e. 3x the lifespan of the OpenStack project), over a history of 10
>> years, of people doing the right things, you build trust in their judgment.
>> 
>> *no one* in the Linux tree was given trust first, under the hope that it
>> would work out. They had to earn it, hard, by doing community work, and
>> not just playing in their corner of the world.
>> 
>> 2) This
>> http://www.wired.com/wiredenterprise/2012/06/torvalds-nvidia-linux/ is
>> completely acceptable behavior. So when someone has bad code, they are
>> flamed to within an inch of their life, repeatedly, until they never
>> ever do that again. This is actually a time saving measure in code
>> review. It's a lot faster to just call people idiots then to help them
>> with line by line improvements in their code, 10, 20, 30, or 40
>> iterations in gerrit.
>> 
>> We, as a community have decided, I think rightly, that #2 really isn't
>> in our culture. But you can't start cherry picking parts of the Linux
>> kernel community without considering how all the parts work together.
>> The good and the bad are part of why the whole system works.
>> 
>> This is an extremely important point in that discussion.
>> 
>> The Linux kernel model is built on a pyramidal model where Linus, in a
>> PTL+Release Manager role, has the final ability to refuse whole sections
>> of the code just because he doesn't like it.
>> 
>> Over two decades, Linus built a solid trust relationship with most
>> subsystem maintainers, so that he doesn't have to review every single
>> patch for sanity. In those areas he has a set of people who consistently
>> proved they would apply the same standards as he does. But for other
>> less-trusted areas the pyramidal model is still working.
>> 
>> I don't see how we could apply that to OpenStack as the trust
>> relationships are far from being that advanced (think: not old enough),
>> and I don't think we want to replicate the personified, pyramidal merge
>> model to handle the less-trusted relationships in the mean time.
>> 
>> 
>> Younger projects at the bottom of the pyramid, especially kernel modules 
>> that we could consider equivant to drivers, IMO cannot be based on such a 
>> long trust relationship due to their age.
>> As an example, well, the Hyper-V linux kernel LIS modules fit pretty well :-)
>> 
>> You don't really want to develop the hyper-V driver in a private
>> subsystem branch all cycle, then at the very end have it rejected from
>> release by an empowered Russell or Thierry just because we think it's
>> not tested enough or we don't like the color it's been painted. This is
>> how the Linux kernel model works with untrusted subsystems -- by
>> subjecting your work to a final BDFL right to kill it at release time.
>> 
>> The other two alternatives are to accept the delays and work within Nova
>> (slowly building the trust that will give you more autonomy), or ship it
>> as a separate add-on that does not come with nova-core's signature on it.
>> 
>> 
>> I never asked for a nova signature on it. My only requirerement is that the 
>> project would be part of OpenStack and not an external project, even if this 
>> means passing 2 releases in incubation on stackforge as long as it can 
>> become part of the OpenStack core group of projects afterwards (if it meets 
>> the required OpenStack criteria of course).  
>> https://wiki.openstack.org/wiki/Governance/NewProjects
>> 
>> --
>> Thierry Carrez (ttx)
> 
> 
> Regards,
> Daniel
> -- 
> |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org  -o- http://virt-manager.org :|
> |: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http:

Re: [openstack-dev] Hyper-V meeting Minutes

[Thierry Carrez]

Daniel P. Berrange wrote:
> Alessandro, please fix your email program so that it does not send
> HTML email to the list, and correctly quotes text you are replying
> to with '> '.

+1 :)

Reference:
https://wiki.openstack.org/wiki/MailingListEtiquette

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Thierry Carrez]

Alessandro Pilotti wrote:
> On Oct 16, 2013, at 13:19 , Thierry Carrez > The other two alternatives are to accept the delays and work within Nova
>> (slowly building the trust that will give you more autonomy), or ship it
>> as a separate add-on that does not come with nova-core's signature on it.
> 
> I never asked for a nova signature on it. My only requirerement is that
> the project would be part of OpenStack and not an external project, even
> if this means passing 2 releases in incubation on stackforge as long as
> it can become part of the OpenStack core group of projects afterwards
> (if it meets the required OpenStack criteria of course).
>  https://wiki.openstack.org/wiki/Governance/NewProjects

That's a possible outcome of the second alternative I described above.
The separate add-on could apply to the incubation track and potentially
be made a part of the integrated release.

My rant was in answer to Vish's "adopt something more similar to the
linux model when dealing with subsystems" suggestion, where the
autonomous subsystem is still made part of "Nova" in the end, and
therefore carries nova-core's signature.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Daniel P. Berrange]

Alessandro, please fix your email program so that it does not send
HTML email to the list, and correctly quotes text you are replying
to with '> '. Your reply comes out looking like this which makes it
impossible to see who wrote what:

On Wed, Oct 16, 2013 at 10:42:45AM +, Alessandro Pilotti wrote:
> 
> On Oct 16, 2013, at 13:19 , Thierry Carrez 
> mailto:thie...@openstack.org>>
>  wrote:
> 
> Sean Dague wrote:
> The Linux kernel process works for a couple of reasons...
> 
> 1) the subsystem maintainers have known each other for a solid decade
> (i.e. 3x the lifespan of the OpenStack project), over a history of 10
> years, of people doing the right things, you build trust in their judgment.
> 
> *no one* in the Linux tree was given trust first, under the hope that it
> would work out. They had to earn it, hard, by doing community work, and
> not just playing in their corner of the world.
> 
> 2) This
> http://www.wired.com/wiredenterprise/2012/06/torvalds-nvidia-linux/ is
> completely acceptable behavior. So when someone has bad code, they are
> flamed to within an inch of their life, repeatedly, until they never
> ever do that again. This is actually a time saving measure in code
> review. It's a lot faster to just call people idiots then to help them
> with line by line improvements in their code, 10, 20, 30, or 40
> iterations in gerrit.
> 
> We, as a community have decided, I think rightly, that #2 really isn't
> in our culture. But you can't start cherry picking parts of the Linux
> kernel community without considering how all the parts work together.
> The good and the bad are part of why the whole system works.
> 
> This is an extremely important point in that discussion.
> 
> The Linux kernel model is built on a pyramidal model where Linus, in a
> PTL+Release Manager role, has the final ability to refuse whole sections
> of the code just because he doesn't like it.
> 
> Over two decades, Linus built a solid trust relationship with most
> subsystem maintainers, so that he doesn't have to review every single
> patch for sanity. In those areas he has a set of people who consistently
> proved they would apply the same standards as he does. But for other
> less-trusted areas the pyramidal model is still working.
> 
> I don't see how we could apply that to OpenStack as the trust
> relationships are far from being that advanced (think: not old enough),
> and I don't think we want to replicate the personified, pyramidal merge
> model to handle the less-trusted relationships in the mean time.
> 
> 
> Younger projects at the bottom of the pyramid, especially kernel modules that 
> we could consider equivant to drivers, IMO cannot be based on such a long 
> trust relationship due to their age.
> As an example, well, the Hyper-V linux kernel LIS modules fit pretty well :-)
> 
> You don't really want to develop the hyper-V driver in a private
> subsystem branch all cycle, then at the very end have it rejected from
> release by an empowered Russell or Thierry just because we think it's
> not tested enough or we don't like the color it's been painted. This is
> how the Linux kernel model works with untrusted subsystems -- by
> subjecting your work to a final BDFL right to kill it at release time.
> 
> The other two alternatives are to accept the delays and work within Nova
> (slowly building the trust that will give you more autonomy), or ship it
> as a separate add-on that does not come with nova-core's signature on it.
> 
> 
> I never asked for a nova signature on it. My only requirerement is that the 
> project would be part of OpenStack and not an external project, even if this 
> means passing 2 releases in incubation on stackforge as long as it can become 
> part of the OpenStack core group of projects afterwards (if it meets the 
> required OpenStack criteria of course).  
> https://wiki.openstack.org/wiki/Governance/NewProjects
> 
> --
> Thierry Carrez (ttx)


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Alessandro Pilotti]

On Oct 16, 2013, at 13:19 , Thierry Carrez 
mailto:thie...@openstack.org>>
 wrote:

Sean Dague wrote:
The Linux kernel process works for a couple of reasons...

1) the subsystem maintainers have known each other for a solid decade
(i.e. 3x the lifespan of the OpenStack project), over a history of 10
years, of people doing the right things, you build trust in their judgment.

*no one* in the Linux tree was given trust first, under the hope that it
would work out. They had to earn it, hard, by doing community work, and
not just playing in their corner of the world.

2) This
http://www.wired.com/wiredenterprise/2012/06/torvalds-nvidia-linux/ is
completely acceptable behavior. So when someone has bad code, they are
flamed to within an inch of their life, repeatedly, until they never
ever do that again. This is actually a time saving measure in code
review. It's a lot faster to just call people idiots then to help them
with line by line improvements in their code, 10, 20, 30, or 40
iterations in gerrit.

We, as a community have decided, I think rightly, that #2 really isn't
in our culture. But you can't start cherry picking parts of the Linux
kernel community without considering how all the parts work together.
The good and the bad are part of why the whole system works.

This is an extremely important point in that discussion.

The Linux kernel model is built on a pyramidal model where Linus, in a
PTL+Release Manager role, has the final ability to refuse whole sections
of the code just because he doesn't like it.

Over two decades, Linus built a solid trust relationship with most
subsystem maintainers, so that he doesn't have to review every single
patch for sanity. In those areas he has a set of people who consistently
proved they would apply the same standards as he does. But for other
less-trusted areas the pyramidal model is still working.

I don't see how we could apply that to OpenStack as the trust
relationships are far from being that advanced (think: not old enough),
and I don't think we want to replicate the personified, pyramidal merge
model to handle the less-trusted relationships in the mean time.


Younger projects at the bottom of the pyramid, especially kernel modules that 
we could consider equivant to drivers, IMO cannot be based on such a long trust 
relationship due to their age.
As an example, well, the Hyper-V linux kernel LIS modules fit pretty well :-)

You don't really want to develop the hyper-V driver in a private
subsystem branch all cycle, then at the very end have it rejected from
release by an empowered Russell or Thierry just because we think it's
not tested enough or we don't like the color it's been painted. This is
how the Linux kernel model works with untrusted subsystems -- by
subjecting your work to a final BDFL right to kill it at release time.

The other two alternatives are to accept the delays and work within Nova
(slowly building the trust that will give you more autonomy), or ship it
as a separate add-on that does not come with nova-core's signature on it.


I never asked for a nova signature on it. My only requirerement is that the 
project would be part of OpenStack and not an external project, even if this 
means passing 2 releases in incubation on stackforge as long as it can become 
part of the OpenStack core group of projects afterwards (if it meets the 
required OpenStack criteria of course).  
https://wiki.openstack.org/wiki/Governance/NewProjects

--
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Thierry Carrez]

Sean Dague wrote:
> The Linux kernel process works for a couple of reasons...
> 
> 1) the subsystem maintainers have known each other for a solid decade
> (i.e. 3x the lifespan of the OpenStack project), over a history of 10
> years, of people doing the right things, you build trust in their judgment.
> 
> *no one* in the Linux tree was given trust first, under the hope that it
> would work out. They had to earn it, hard, by doing community work, and
> not just playing in their corner of the world.
> 
> 2) This
> http://www.wired.com/wiredenterprise/2012/06/torvalds-nvidia-linux/ is
> completely acceptable behavior. So when someone has bad code, they are
> flamed to within an inch of their life, repeatedly, until they never
> ever do that again. This is actually a time saving measure in code
> review. It's a lot faster to just call people idiots then to help them
> with line by line improvements in their code, 10, 20, 30, or 40
> iterations in gerrit.
> 
> We, as a community have decided, I think rightly, that #2 really isn't
> in our culture. But you can't start cherry picking parts of the Linux
> kernel community without considering how all the parts work together.
> The good and the bad are part of why the whole system works.

This is an extremely important point in that discussion.

The Linux kernel model is built on a pyramidal model where Linus, in a
PTL+Release Manager role, has the final ability to refuse whole sections
of the code just because he doesn't like it.

Over two decades, Linus built a solid trust relationship with most
subsystem maintainers, so that he doesn't have to review every single
patch for sanity. In those areas he has a set of people who consistently
proved they would apply the same standards as he does. But for other
less-trusted areas the pyramidal model is still working.

I don't see how we could apply that to OpenStack as the trust
relationships are far from being that advanced (think: not old enough),
and I don't think we want to replicate the personified, pyramidal merge
model to handle the less-trusted relationships in the mean time.

You don't really want to develop the hyper-V driver in a private
subsystem branch all cycle, then at the very end have it rejected from
release by an empowered Russell or Thierry just because we think it's
not tested enough or we don't like the color it's been painted. This is
how the Linux kernel model works with untrusted subsystems -- by
subjecting your work to a final BDFL right to kill it at release time.

The other two alternatives are to accept the delays and work within Nova
(slowly building the trust that will give you more autonomy), or ship it
as a separate add-on that does not come with nova-core's signature on it.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Alessandro Pilotti]

On Oct 16, 2013, at 11:19 , Robert Collins 
mailto:robe...@robertcollins.net>>
 wrote:

On 16 October 2013 20:14, Alessandro Pilotti
mailto:apilo...@cloudbasesolutions.com>> 
wrote:>


Drivers are IMO not part of the core of Nova, but completely separated and 
decoupled entities, which IMO should be treated that way. As a consequence, we 
frankly don't stricly feel as part of Nova, although some of us have a pretty 
strong understanding of how all the Nova pieces work.

I don't have a particular view on whether they *should be* separate
decoupled entities, but today I repeatedly hear concerns about the
impact of treating 'internal APIs' as stable things. That's relevant
because *if* nova drivers are to be separate decoupled entities, the
APIs they use - and expose - have to be treated as stable things with
graceful evolution, backwards compatibility etc. Doing anything else
will lead to deployment issues and race conditions in the gate.

Obliging driver (or other decoupeld sub-components) developers to learn the 
entire Nova project before being able to contribute would just kill their 
effort before the start, resulting in a poorer ecosystem.

There are lots of different aspects of contribution: reviews (as
anybody), reviews (as core), code, bug assessment, design input,
translations, documentation etc. Nobody has said that you have to
learn everything before you contribute.

The /only item/ in that list that requires wide knowledge of the code
base is reviews (as core).

The difference between reviews (as anybody) and reviews (as core) is
that core is responsible for identifying things like duplicated
functionality, design and architectural issues - and for only
approving a patch when they are comfortable it doesn't introduce such
issues.

Core review is a bottleneck. When optimising systems with a bottleneck
there are only three things you can do to make the system work better:
- avoid the bottleneck
- increase capacity of the bottleneck
- make the bottleneck more efficient at the work it's given

Anything else will just end up backing up work behind the bottleneck
and make /no/ difference at all.

Your proposal (partition the code & reviewers by core/drivers) is an
'avoid the bottleneck' approach. It will remove some fraction of
reviews from the bottleneck - those that are per driver - at the cost
of losing /all/ of the feedback, architecture and design input that
the drivers currently benefit from, *and* removing from the nova core
any insight into the issues drivers are experiencing (because they
won't be reviewing driver code). Unless you have 'in-tree' and
'out-of-tree' drivers, and then one has to ask 'why are some drivers
out of tree'? The only answer for that so far is 'review latency is
hurting drivers'... but review latency is hurting *everyone*.

To increase bottleneck capacity we could ask -core reviewers to spend
more time reviewing. However that doesn't work because we're human -
we need time to do other things than think hard about other peoples
code. There is an upper bound on effective time spent reviewing by
reviewers. Or we can increase capacity of the bottleneck by training
up more -core reviewers. Thats pretty obvious :). However training up
more reviewers requires more reviewers - so the cost here is that
someone needs to volunteer that time.

The bottleneck - core reviewers - can get through more reviews when
the reviews are easy to do. From prior conversations here is my list:
- keep the change small. Lots of LOC == a hard review, which consumes
-core review time
- get the review reviewed by non-core as soon as you put it up - this
will catch many issues -core would and reduce re-work
- follow the style guides
- make your commit message really clear - there's a style guide for these too!

It seems to me that one can order the choices by their costs:
- provide patches that are easier to review [basically no cost: the
rules are already in place]
- train more -core reviewers [needs volunteer time]
- split the drivers out [lose coherency on tightly coupled things,
have to stabilise more interfaces, lose experienced input into driver
code]

And by their effectiveness [this is more subjective:)]
- train more -core reviewers [essentially linear, very easy to predict]
- provide patches that are easier to review [many patches are good
already, has a low upper bound on effectiveness]
- split the drivers out [won't help *at all* with changes required in
core to support a driver feature]

Finally, there is a key thing to note: As contributors to the project
scales, patch volume scales. As patch volume scales the pressure on
the bottleneck increases: we *have* to scale the -core review team [or
partition the code bases into two that will **both have a solid
reviewer community**].

Remember that every patch added requires *at minimum* 2 core reviews
[and I suspect in reality 3 core reviews - one to catch issues, then a
re-review, then a second and land]. This gives a pretty simple
equation fo

Re: [openstack-dev] Hyper-V meeting Minutes

[Robert Collins]

On 16 October 2013 20:14, Alessandro Pilotti
 wrote:>
>

> Drivers are IMO not part of the core of Nova, but completely separated and 
> decoupled entities, which IMO should be treated that way. As a consequence, 
> we frankly don't stricly feel as part of Nova, although some of us have a 
> pretty strong understanding of how all the Nova pieces work.

I don't have a particular view on whether they *should be* separate
decoupled entities, but today I repeatedly hear concerns about the
impact of treating 'internal APIs' as stable things. That's relevant
because *if* nova drivers are to be separate decoupled entities, the
APIs they use - and expose - have to be treated as stable things with
graceful evolution, backwards compatibility etc. Doing anything else
will lead to deployment issues and race conditions in the gate.

> Obliging driver (or other decoupeld sub-components) developers to learn the 
> entire Nova project before being able to contribute would just kill their 
> effort before the start, resulting in a poorer ecosystem.

There are lots of different aspects of contribution: reviews (as
anybody), reviews (as core), code, bug assessment, design input,
translations, documentation etc. Nobody has said that you have to
learn everything before you contribute.

The /only item/ in that list that requires wide knowledge of the code
base is reviews (as core).

The difference between reviews (as anybody) and reviews (as core) is
that core is responsible for identifying things like duplicated
functionality, design and architectural issues - and for only
approving a patch when they are comfortable it doesn't introduce such
issues.

Core review is a bottleneck. When optimising systems with a bottleneck
there are only three things you can do to make the system work better:
 - avoid the bottleneck
 - increase capacity of the bottleneck
 - make the bottleneck more efficient at the work it's given

Anything else will just end up backing up work behind the bottleneck
and make /no/ difference at all.

Your proposal (partition the code & reviewers by core/drivers) is an
'avoid the bottleneck' approach. It will remove some fraction of
reviews from the bottleneck - those that are per driver - at the cost
of losing /all/ of the feedback, architecture and design input that
the drivers currently benefit from, *and* removing from the nova core
any insight into the issues drivers are experiencing (because they
won't be reviewing driver code). Unless you have 'in-tree' and
'out-of-tree' drivers, and then one has to ask 'why are some drivers
out of tree'? The only answer for that so far is 'review latency is
hurting drivers'... but review latency is hurting *everyone*.

To increase bottleneck capacity we could ask -core reviewers to spend
more time reviewing. However that doesn't work because we're human -
we need time to do other things than think hard about other peoples
code. There is an upper bound on effective time spent reviewing by
reviewers. Or we can increase capacity of the bottleneck by training
up more -core reviewers. Thats pretty obvious :). However training up
more reviewers requires more reviewers - so the cost here is that
someone needs to volunteer that time.

The bottleneck - core reviewers - can get through more reviews when
the reviews are easy to do. From prior conversations here is my list:
 - keep the change small. Lots of LOC == a hard review, which consumes
-core review time
 - get the review reviewed by non-core as soon as you put it up - this
will catch many issues -core would and reduce re-work
 - follow the style guides
 - make your commit message really clear - there's a style guide for these too!

It seems to me that one can order the choices by their costs:
 - provide patches that are easier to review [basically no cost: the
rules are already in place]
 - train more -core reviewers [needs volunteer time]
 - split the drivers out [lose coherency on tightly coupled things,
have to stabilise more interfaces, lose experienced input into driver
code]

And by their effectiveness [this is more subjective:)]
 - train more -core reviewers [essentially linear, very easy to predict]
 - provide patches that are easier to review [many patches are good
already, has a low upper bound on effectiveness]
 - split the drivers out [won't help *at all* with changes required in
core to support a driver feature]

Finally, there is a key thing to note: As contributors to the project
scales, patch volume scales. As patch volume scales the pressure on
the bottleneck increases: we *have* to scale the -core review team [or
partition the code bases into two that will **both have a solid
reviewer community**].

Remember that every patch added requires *at minimum* 2 core reviews
[and I suspect in reality 3 core reviews - one to catch issues, then a
re-review, then a second and land]. This gives a pretty simple
equation for code contributors: if you upload a patch, to keep things
equitable, you need to do at least 3 review

Re: [openstack-dev] Hyper-V meeting Minutes

[Alessandro Pilotti]

On Oct 16, 2013, at 08:45 , Vishvananda Ishaya 
 wrote:

> Hi Sean,
> 
> I'm going to top post because my response is general. I totally agree that we 
> need people that understand the code base and we should encourage new people 
> to be cross-functional. I guess my main issue is with how we get there. I 
> believe in encouragment over punishment. In my mind giving people autonomy 
> and control encourages them to contribute more. 
> 

I couldn't agree more. By having more autonomy we would definitely be able to 
add more contributors to the team and people would definitely feel more 
productive. Nothing frustrates a developer more than having her/his code, 
meaning days or weeks of passionate hard work, sitting for long periods in a 
review queue and not knowing if it will be accepted or not.

Although this won't most probably affect the spirit of seasoned developers used 
to large projects politics and bureaucracy, it will IMO definitely kill the 
occasional contributor's effort or the junior's morale.

> In my opinion giving the driver developers control over their own code will 
> lead to higher quality drivers. Yes, we risk integration issues, lack of test 
> coverage, and buggy implementations, but it is my opinion that the increased 
> velocity that the developers will enjoy will mean faster bug fixes and more 
> opportunity to improve the drivers.
> 

Here's my opinion on the driver quality side which comes from personal 
experience and from what I observed in other drivers code taking our Hyper-V 
driver development history as an example:

*1st phase (latest days of Folsom)*

We started our contribution around July 2012 by patching the code that used to 
be in the tree before getting kicked out from Essex (extremely buggy and 
basically a collection of anti-patterns).
Since the above work has been done in 2 weeks, the Folsom code is obviously 
fitting in all the category that Vish points out, but it does not mean that we 
were not aware of it: 

> integration issues, lack of test coverage, and buggy implementations



*2nd phase (Grizzly)*

We refactored heavily the driver, getting rid almost entirely of the pre-Essex 
code, added the missing unit tests and new features to be on pair with the 
other main drivers.
With Grizzly we had a very low bug rate, excellent reviews from the users and 
in general the code stands out for its quality.


*3rd phase (Havana)*

Lots of new features and some minor refactoring. Sadly this process almost 
faltered due to the interaction and review issues with the Nova team that lead 
to the present discussions.
Some of the few bug fixes haven't even been reviewed in time for the Havana 
cut, we'll be force to tell distribution maintainers and users to cherry pick 
them from master or from our fork.


*4th phase (Icehouse, planned)*

The CI gate is the main focus plus blueprints already implemented for Havana 
that didn't make it and a few more new features.


During the process (especially during the first 2 cycles) we learned a lot 
about OpenStack and the Gerrit review system thanks in particular to people in 
the community that helped in getting quickly up to speed with all the details. 
I personally have to thank mainly Vish and Dan Smith for that. A separate 
project would simplify those steps for a new driver today, as it would go 
through stackforge incubation. 

Why do I report all those hystorical project details here? Because the Folsom 
messy code would have never passed today's review criteria, but if you look at 
how things came out in the end, we got an excellent product aligned with 
OpenStack's standards, lots of happy users and a fast expanding community. 

Drivers are IMO not part of the core of Nova, but completely separated and 
decoupled entities, which IMO should be treated that way. As a consequence, we 
frankly don't stricly feel as part of Nova, although some of us have a pretty 
strong understanding of how all the Nova pieces work.

Obliging driver (or other decoupeld sub-components) developers to learn the 
entire Nova project before being able to contribute would just kill their 
effort before the start, resulting in a poorer ecosystem.

My suggestion is to have separate projects for each driver, a versioned Nova 
driver interface contract and separate teams for each driver (completely 
independent from Nova), with new drivers going through an incubation period on 
stackforge like any other new OpenStack project. 


> I also think lowering the amount of code that nova-core has to keep an eye on 
> will improve the review velocity of the rest of the code as well.
> 
> Vish
> 
> On Oct 15, 2013, at 4:36 PM, Sean Dague  wrote:
> 
>> On 10/15/2013 04:54 PM, Vishvananda Ishaya wrote:
>>> Hi Everyone,
>>> 
>>> I've been following this conversation and weighing the different sides. 
>>> This is a tricky issue but I think it is important to decouple further and 
>>> extend our circle of trust.
>>> 
>>> When nova started it was very easy to do 

Re: [openstack-dev] Hyper-V meeting Minutes

[Alessandro Pilotti]

On Oct 16, 2013, at 05:48 , Dan Smith 
mailto:d...@danplanet.com>> wrote:

The last thing that OpenStack needs ANY more help with is velocity. I
mean, let's be serious - we land WAY more patches in a day than is
even close to sane.

Thanks for saying this -- it doesn't get said enough. I find it totally
amazing that we're merging 34 changes in a day (yesterday) which is like
170 per work week (just on nova). More amazing is that we're talking
about how to make it faster all the time. It's definitely the fastest
moving extremely complex thing I can recall working on.


Dan, nobody puts into discussion how good the work that you guys are doing is. 
You guys are doing an amazing job, that's unquestionable.

The problem is that the review team, in the way in which it is structured 
today, simply does not scale with the review load (which is quite funny in a 
project which is all about scalability :-)).

Here's a quick analysys over the 90 days Nova review stats published by 
Russell: http://russellbryant.net/openstack-stats/nova-reviewers-90.txt

Roughly 2/3 of the reviews are done by 20 people, with the top 10 getting close 
to 50%.
Let's say that we provide out of our sub-team 1 additional Nova core dev that 
will perform in the top 10, which averages 521 reviewes, 4,6% of the total.
This would reduce our stale review queue time by 5%, which is quite far from a 
practical improvement over the current mess IMO.

The picture changes if you put this additional resource to do reviews mostly on 
our sub-project code, but at that point I don't see the difference from having 
somebody with +2 rights on the driver sub-tree only.

This example applies to any sub-project of course, not only the Hyper-V driver.


(I hope that the table formatting will come out decently in the ML email, if 
not please find the data here: http://paste.openstack.org/show/48539/)

ReviewerCoreReviews % over totalPartials
russellbyes 888 7,84%
garyk   856 7,55%
jogoyes 475 4,19%
mikalstill  yes 450 3,97%
danms   yes 447 3,94%   23,55%  TOP 5
ndipanovyes 432 3,81%
klmitch yes 429 3,79%
cbehrensyes 360 3,18%
johngarbutt yes 351 3,10%
cyeoh-0 yes 327 2,89%   44,25%  TOP 10
markmc  yes 304 2,68%
alaski  yes 289 2,55%
mriedem 270 2,38%
cerberusyes 266 2,35%
dripton 261 2,30%
berrangeyes 251 2,21%
jhesketh250 2,21%
philip-day  250 2,21%
xuhj237 2,09%
belliottyes 212 1,87%   67,10%  TOP 20
guohliu 201 1,77%
boris-42170 1,50%
sdague  yes 164 1,45%
p-draigbradyyes 130 1,15%
vishvananda yes 123 1,09%
tracyajones 112 0,99%
JayLau  109 0,96%
hartsocks   108 0,95%
arosen  106 0,94%
dims-v  101 0,89%   78,79%  TOP 30




We MUST continue to be vigilent in getting people to care about more
than their specific part, or else this big complex mess is going to come
crashing down around us.

I totally agree.

--Dan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Vishvananda Ishaya]

Hi Sean,

I'm going to top post because my response is general. I totally agree that we 
need people that understand the code base and we should encourage new people to 
be cross-functional. I guess my main issue is with how we get there. I believe 
in encouragment over punishment. In my mind giving people autonomy and control 
encourages them to contribute more. 

In my opinion giving the driver developers control over their own code will 
lead to higher quality drivers. Yes, we risk integration issues, lack of test 
coverage, and buggy implementations, but it is my opinion that the increased 
velocity that the developers will enjoy will mean faster bug fixes and more 
opportunity to improve the drivers.

I also think lowering the amount of code that nova-core has to keep an eye on 
will improve the review velocity of the rest of the code as well.

Vish

On Oct 15, 2013, at 4:36 PM, Sean Dague  wrote:

> On 10/15/2013 04:54 PM, Vishvananda Ishaya wrote:
>> Hi Everyone,
>> 
>> I've been following this conversation and weighing the different sides. This 
>> is a tricky issue but I think it is important to decouple further and extend 
>> our circle of trust.
>> 
>> When nova started it was very easy to do feature development. As it has 
>> matured the pace has slowed. This is expected and necessary, but we 
>> periodically must make decoupling decisions or we will become mired in 
>> overhead. We did this already with cinder and neutron, and we have discussed 
>> doing this with virt drivers in the past.
>> 
>> We have a large number of people attempting to contribute to small sections 
>> of nova and getting frustrated with the process.  The perception of 
>> developers is much more important than the actual numbers here. If people 
>> are frustrated they are disincentivized to help and it hurts everyone. 
>> Suggesting that these contributors need to learn all of nova and help with 
>> the review queue is silly and makes us seem elitist. We should make it as 
>> easy as possible for new contributors to help.
>> 
>> I think our current model is breaking down at our current size and we need 
>> to adopt something more similar to the linux model when dealing with 
>> subsystems. The hyper-v team is the only one suggesting changes, but there 
>> have been similar concerns from the vmware team. I have no doubt that there 
>> are similar issues with the PowerVM, Xen, Docker, lxc and even kvm driver 
>> contributors.
> 
> The Linux kernel process works for a couple of reasons...
> 
> 1) the subsystem maintainers have known each other for a solid decade (i.e. 
> 3x the lifespan of the OpenStack project), over a history of 10 years, of 
> people doing the right things, you build trust in their judgment.
> 
> *no one* in the Linux tree was given trust first, under the hope that it 
> would work out. They had to earn it, hard, by doing community work, and not 
> just playing in their corner of the world.
> 
> 2) This http://www.wired.com/wiredenterprise/2012/06/torvalds-nvidia-linux/ 
> is completely acceptable behavior. So when someone has bad code, they are 
> flamed to within an inch of their life, repeatedly, until they never ever do 
> that again. This is actually a time saving measure in code review. It's a lot 
> faster to just call people idiots then to help them with line by line 
> improvements in their code, 10, 20, 30, or 40 iterations in gerrit.
> 
> We, as a community have decided, I think rightly, that #2 really isn't in our 
> culture. But you can't start cherry picking parts of the Linux kernel 
> community without considering how all the parts work together. The good and 
> the bad are part of why the whole system works.
> 
>> In my opinion, nova-core needs to be willing to trust the subsystem 
>> developers and let go of a little bit of control. I frankly don't see the 
>> drawbacks.
> 
> I actually see huge draw backs. Culture matters. Having people active and 
> willing to work on real core issues matter. The long term health of Nova 
> matters.
> 
> As the QA PTL I can tell you that when you look at Nova vs. Cinder vs. 
> Neutron, you'll see some very clear lines about how long it takes to get to 
> the bottom of a race condition, and how many deep races are in each of them. 
> I find this directly related to the stance each project has taken on whether 
> it's socially acceptable to only work on your own vendor code. Nova's 
> insistence up until this point that if you only play in your corner, you 
> don't get the same attention is important incentive for people to integrate 
> and work beyond just their boundaries. I think diluting this part of the 
> culture would be hugely detrimental to Nova.
> 
> Let's take an example that came up today, the compute_diagnostics API. This 
> is an area where we've left it completely to the virt drivers to vomit up a 
> random dictionary of the day for debugging reasons, and stamped it as an API. 
> With a model where we let virt driver authors go hide in a

Re: [openstack-dev] Hyper-V meeting Minutes

[Alessandro Pilotti]

On Oct 16, 2013, at 02:36 , Sean Dague mailto:s...@dague.net>> 
wrote:

On 10/15/2013 04:54 PM, Vishvananda Ishaya wrote:
Hi Everyone,

I've been following this conversation and weighing the different sides. This is 
a tricky issue but I think it is important to decouple further and extend our 
circle of trust.

When nova started it was very easy to do feature development. As it has matured 
the pace has slowed. This is expected and necessary, but we periodically must 
make decoupling decisions or we will become mired in overhead. We did this 
already with cinder and neutron, and we have discussed doing this with virt 
drivers in the past.

We have a large number of people attempting to contribute to small sections of 
nova and getting frustrated with the process.  The perception of developers is 
much more important than the actual numbers here. If people are frustrated they 
are disincentivized to help and it hurts everyone. Suggesting that these 
contributors need to learn all of nova and help with the review queue is silly 
and makes us seem elitist. We should make it as easy as possible for new 
contributors to help.

I think our current model is breaking down at our current size and we need to 
adopt something more similar to the linux model when dealing with subsystems. 
The hyper-v team is the only one suggesting changes, but there have been 
similar concerns from the vmware team. I have no doubt that there are similar 
issues with the PowerVM, Xen, Docker, lxc and even kvm driver contributors.

The Linux kernel process works for a couple of reasons...

1) the subsystem maintainers have known each other for a solid decade (i.e. 3x 
the lifespan of the OpenStack project), over a history of 10 years, of people 
doing the right things, you build trust in their judgment.

*no one* in the Linux tree was given trust first, under the hope that it would 
work out. They had to earn it, hard, by doing community work, and not just 
playing in their corner of the world.

2) This http://www.wired.com/wiredenterprise/2012/06/torvalds-nvidia-linux/ is 
completely acceptable behavior. So when someone has bad code, they are flamed 
to within an inch of their life, repeatedly, until they never ever do that 
again. This is actually a time saving measure in code review. It's a lot faster 
to just call people idiots then to help them with line by line improvements in 
their code, 10, 20, 30, or 40 iterations in gerrit.

We, as a community have decided, I think rightly, that #2 really isn't in our 
culture. But you can't start cherry picking parts of the Linux kernel community 
without considering how all the parts work together. The good and the bad are 
part of why the whole system works.

In my opinion, nova-core needs to be willing to trust the subsystem developers 
and let go of a little bit of control. I frankly don't see the drawbacks.

I actually see huge draw backs. Culture matters. Having people active and 
willing to work on real core issues matter. The long term health of Nova 
matters.

As the QA PTL I can tell you that when you look at Nova vs. Cinder vs. Neutron, 
you'll see some very clear lines about how long it takes to get to the bottom 
of a race condition, and how many deep races are in each of them. I find this 
directly related to the stance each project has taken on whether it's socially 
acceptable to only work on your own vendor code. Nova's insistence up until 
this point that if you only play in your corner, you don't get the same 
attention is important incentive for people to integrate and work beyond just 
their boundaries. I think diluting this part of the culture would be hugely 
detrimental to Nova.

Let's take an example that came up today, the compute_diagnostics API. This is 
an area where we've left it completely to the virt drivers to vomit up a random 
dictionary of the day for debugging reasons, and stamped it as an API. With a 
model where we let virt driver authors go hide in a corner, that's never going 
to become an API with any kind of contract, and given how much effort we've 
spent on ensuring RPC versioning and message formats, the idea that we are 
exposing a public rest endpoint that's randomly fluctuating data based on date 
and underlying implementation, is a bit saddening.

I'm leaning towards giving control of the subtree to the team as the best 
option because it is simple and works with our current QA system. 
Alternatively, we could split out the driver into a nova subproject (2 below) 
or we could allow them to have a separate branch and do a trusted merge of all 
changes at the end of the cycle (similar to the linux model).

I hope we can come to a solution to the summit that makes all of our 
contributors want to participate more. I believe that giving people more 
responsibility inspires them to participate more fully.

I would like nothing more than all our contributors to participate more. But 
more has to mean caring about not only your stuff.

I was ca

Re: [openstack-dev] Hyper-V meeting Minutes

[Dan Smith]

> The last thing that OpenStack needs ANY more help with is velocity. I
> mean, let's be serious - we land WAY more patches in a day than is
> even close to sane.

Thanks for saying this -- it doesn't get said enough. I find it totally
amazing that we're merging 34 changes in a day (yesterday) which is like
170 per work week (just on nova). More amazing is that we're talking
about how to make it faster all the time. It's definitely the fastest
moving extremely complex thing I can recall working on.

> We MUST continue to be vigilent in getting people to care about more
> than their specific part, or else this big complex mess is going to come
> crashing down around us.

I totally agree.

--Dan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Rochelle.Grober]

Monty Taylor wrote:



On 10/15/2013 08:36 PM, Sean Dague wrote:

> On 10/15/2013 04:54 PM, Vishvananda Ishaya wrote:

>> Hi Everyone,

>>

>> I've been following this conversation and weighing the different

>> sides. This is a tricky issue but I think it is important to decouple

>> further and extend our circle of trust.

>>

>> When nova started it was very easy to do feature development. As it

>> has matured the pace has slowed. This is expected and necessary, but

>> we periodically must make decoupling decisions or we will become mired

>> in overhead. We did this already with cinder and neutron, and we have

>> discussed doing this with virt drivers in the past.

>>

>> We have a large number of people attempting to contribute to small

>> sections of nova and getting frustrated with the process.  The

>> perception of developers is much more important than the actual

>> numbers here. If people are frustrated they are disincentivized to

>> help and it hurts everyone. Suggesting that these contributors need to

>> learn all of nova and help with the review queue is silly and makes us

>> seem elitist. We should make it as easy as possible for new

>> contributors to help.

>>

>> I think our current model is breaking down at our current size and we

>> need to adopt something more similar to the linux model when dealing

>> with subsystems. The hyper-v team is the only one suggesting changes,

>> but there have been similar concerns from the vmware team. I have no

>> doubt that there are similar issues with the PowerVM, Xen, Docker, lxc

>> and even kvm driver contributors.

>

> The Linux kernel process works for a couple of reasons...

>

> 1) the subsystem maintainers have known each other for a solid decade

> (i.e. 3x the lifespan of the OpenStack project), over a history of 10

> years, of people doing the right things, you build trust in their judgment.

>

> *no one* in the Linux tree was given trust first, under the hope that it

> would work out. They had to earn it, hard, by doing community work, and

> not just playing in their corner of the world.

>

> 2) This

> http://www.wired.com/wiredenterprise/2012/06/torvalds-nvidia-linux/ is

> completely acceptable behavior. So when someone has bad code, they are

> flamed to within an inch of their life, repeatedly, until they never

> ever do that again. This is actually a time saving measure in code

> review. It's a lot faster to just call people idiots then to help them

> with line by line improvements in their code, 10, 20, 30, or 40

> iterations in gerrit.

>

> We, as a community have decided, I think rightly, that #2 really isn't

> in our culture. But you can't start cherry picking parts of the Linux

> kernel community without considering how all the parts work together.

> The good and the bad are part of why the whole system works.

>

>> In my opinion, nova-core needs to be willing to trust the subsystem

>> developers and let go of a little bit of control. I frankly don't see

>> the drawbacks.

>

> I actually see huge draw backs. Culture matters. Having people active

> and willing to work on real core issues matter. The long term health of

> Nova matters.

>

> As the QA PTL I can tell you that when you look at Nova vs. Cinder vs.

> Neutron, you'll see some very clear lines about how long it takes to get

> to the bottom of a race condition, and how many deep races are in each

> of them. I find this directly related to the stance each project has

> taken on whether it's socially acceptable to only work on your own

> vendor code. Nova's insistence up until this point that if you only play

> in your corner, you don't get the same attention is important incentive

> for people to integrate and work beyond just their boundaries. I think

> diluting this part of the culture would be hugely detrimental to Nova.

>

> Let's take an example that came up today, the compute_diagnostics API.

> This is an area where we've left it completely to the virt drivers to

> vomit up a random dictionary of the day for debugging reasons, and

> stamped it as an API. With a model where we let virt driver authors go

> hide in a corner, that's never going to become an API with any kind of

> contract, and given how much effort we've spent on ensuring RPC

> versioning and message formats, the idea that we are exposing a public

> rest endpoint that's randomly fluctuating data based on date and

> underlying implementation, is a bit saddening.

>

>> I'm leaning towards giving control of the subtree to the team as the

>> best option because it is simple and works with our current QA system.

>> Alternatively, we could split out the driver into a nova subproject (2

>> below) or we could allow them to have a separate branch and do a

>> trusted merge of all changes at the end of the cycle (similar to the

>> linux model).

>>

>> I hope we can come to a solution to the summit that makes all of our

>> contributors want to participate

Re: [openstack-dev] Hyper-V meeting Minutes

[Monty Taylor]

On 10/15/2013 08:36 PM, Sean Dague wrote:
> On 10/15/2013 04:54 PM, Vishvananda Ishaya wrote:
>> Hi Everyone,
>>
>> I've been following this conversation and weighing the different
>> sides. This is a tricky issue but I think it is important to decouple
>> further and extend our circle of trust.
>>
>> When nova started it was very easy to do feature development. As it
>> has matured the pace has slowed. This is expected and necessary, but
>> we periodically must make decoupling decisions or we will become mired
>> in overhead. We did this already with cinder and neutron, and we have
>> discussed doing this with virt drivers in the past.
>>
>> We have a large number of people attempting to contribute to small
>> sections of nova and getting frustrated with the process.  The
>> perception of developers is much more important than the actual
>> numbers here. If people are frustrated they are disincentivized to
>> help and it hurts everyone. Suggesting that these contributors need to
>> learn all of nova and help with the review queue is silly and makes us
>> seem elitist. We should make it as easy as possible for new
>> contributors to help.
>>
>> I think our current model is breaking down at our current size and we
>> need to adopt something more similar to the linux model when dealing
>> with subsystems. The hyper-v team is the only one suggesting changes,
>> but there have been similar concerns from the vmware team. I have no
>> doubt that there are similar issues with the PowerVM, Xen, Docker, lxc
>> and even kvm driver contributors.
> 
> The Linux kernel process works for a couple of reasons...
> 
> 1) the subsystem maintainers have known each other for a solid decade
> (i.e. 3x the lifespan of the OpenStack project), over a history of 10
> years, of people doing the right things, you build trust in their judgment.
> 
> *no one* in the Linux tree was given trust first, under the hope that it
> would work out. They had to earn it, hard, by doing community work, and
> not just playing in their corner of the world.
> 
> 2) This
> http://www.wired.com/wiredenterprise/2012/06/torvalds-nvidia-linux/ is
> completely acceptable behavior. So when someone has bad code, they are
> flamed to within an inch of their life, repeatedly, until they never
> ever do that again. This is actually a time saving measure in code
> review. It's a lot faster to just call people idiots then to help them
> with line by line improvements in their code, 10, 20, 30, or 40
> iterations in gerrit.
> 
> We, as a community have decided, I think rightly, that #2 really isn't
> in our culture. But you can't start cherry picking parts of the Linux
> kernel community without considering how all the parts work together.
> The good and the bad are part of why the whole system works.
> 
>> In my opinion, nova-core needs to be willing to trust the subsystem
>> developers and let go of a little bit of control. I frankly don't see
>> the drawbacks.
> 
> I actually see huge draw backs. Culture matters. Having people active
> and willing to work on real core issues matter. The long term health of
> Nova matters.
> 
> As the QA PTL I can tell you that when you look at Nova vs. Cinder vs.
> Neutron, you'll see some very clear lines about how long it takes to get
> to the bottom of a race condition, and how many deep races are in each
> of them. I find this directly related to the stance each project has
> taken on whether it's socially acceptable to only work on your own
> vendor code. Nova's insistence up until this point that if you only play
> in your corner, you don't get the same attention is important incentive
> for people to integrate and work beyond just their boundaries. I think
> diluting this part of the culture would be hugely detrimental to Nova.
> 
> Let's take an example that came up today, the compute_diagnostics API.
> This is an area where we've left it completely to the virt drivers to
> vomit up a random dictionary of the day for debugging reasons, and
> stamped it as an API. With a model where we let virt driver authors go
> hide in a corner, that's never going to become an API with any kind of
> contract, and given how much effort we've spent on ensuring RPC
> versioning and message formats, the idea that we are exposing a public
> rest endpoint that's randomly fluctuating data based on date and
> underlying implementation, is a bit saddening.
> 
>> I'm leaning towards giving control of the subtree to the team as the
>> best option because it is simple and works with our current QA system.
>> Alternatively, we could split out the driver into a nova subproject (2
>> below) or we could allow them to have a separate branch and do a
>> trusted merge of all changes at the end of the cycle (similar to the
>> linux model).
>>
>> I hope we can come to a solution to the summit that makes all of our
>> contributors want to participate more. I believe that giving people
>> more responsibility inspires them to participate more fully.
> 
> 

Re: [openstack-dev] Hyper-V meeting Minutes

[Sean Dague]

On 10/15/2013 04:54 PM, Vishvananda Ishaya wrote:

Hi Everyone,

I've been following this conversation and weighing the different sides. This is 
a tricky issue but I think it is important to decouple further and extend our 
circle of trust.

When nova started it was very easy to do feature development. As it has matured 
the pace has slowed. This is expected and necessary, but we periodically must 
make decoupling decisions or we will become mired in overhead. We did this 
already with cinder and neutron, and we have discussed doing this with virt 
drivers in the past.

We have a large number of people attempting to contribute to small sections of 
nova and getting frustrated with the process.  The perception of developers is 
much more important than the actual numbers here. If people are frustrated they 
are disincentivized to help and it hurts everyone. Suggesting that these 
contributors need to learn all of nova and help with the review queue is silly 
and makes us seem elitist. We should make it as easy as possible for new 
contributors to help.

I think our current model is breaking down at our current size and we need to 
adopt something more similar to the linux model when dealing with subsystems. 
The hyper-v team is the only one suggesting changes, but there have been 
similar concerns from the vmware team. I have no doubt that there are similar 
issues with the PowerVM, Xen, Docker, lxc and even kvm driver contributors.


The Linux kernel process works for a couple of reasons...

1) the subsystem maintainers have known each other for a solid decade 
(i.e. 3x the lifespan of the OpenStack project), over a history of 10 
years, of people doing the right things, you build trust in their judgment.


*no one* in the Linux tree was given trust first, under the hope that it 
would work out. They had to earn it, hard, by doing community work, and 
not just playing in their corner of the world.


2) This 
http://www.wired.com/wiredenterprise/2012/06/torvalds-nvidia-linux/ is 
completely acceptable behavior. So when someone has bad code, they are 
flamed to within an inch of their life, repeatedly, until they never 
ever do that again. This is actually a time saving measure in code 
review. It's a lot faster to just call people idiots then to help them 
with line by line improvements in their code, 10, 20, 30, or 40 
iterations in gerrit.


We, as a community have decided, I think rightly, that #2 really isn't 
in our culture. But you can't start cherry picking parts of the Linux 
kernel community without considering how all the parts work together. 
The good and the bad are part of why the whole system works.



In my opinion, nova-core needs to be willing to trust the subsystem developers 
and let go of a little bit of control. I frankly don't see the drawbacks.


I actually see huge draw backs. Culture matters. Having people active 
and willing to work on real core issues matter. The long term health of 
Nova matters.


As the QA PTL I can tell you that when you look at Nova vs. Cinder vs. 
Neutron, you'll see some very clear lines about how long it takes to get 
to the bottom of a race condition, and how many deep races are in each 
of them. I find this directly related to the stance each project has 
taken on whether it's socially acceptable to only work on your own 
vendor code. Nova's insistence up until this point that if you only play 
in your corner, you don't get the same attention is important incentive 
for people to integrate and work beyond just their boundaries. I think 
diluting this part of the culture would be hugely detrimental to Nova.


Let's take an example that came up today, the compute_diagnostics API. 
This is an area where we've left it completely to the virt drivers to 
vomit up a random dictionary of the day for debugging reasons, and 
stamped it as an API. With a model where we let virt driver authors go 
hide in a corner, that's never going to become an API with any kind of 
contract, and given how much effort we've spent on ensuring RPC 
versioning and message formats, the idea that we are exposing a public 
rest endpoint that's randomly fluctuating data based on date and 
underlying implementation, is a bit saddening.



I'm leaning towards giving control of the subtree to the team as the best 
option because it is simple and works with our current QA system. 
Alternatively, we could split out the driver into a nova subproject (2 below) 
or we could allow them to have a separate branch and do a trusted merge of all 
changes at the end of the cycle (similar to the linux model).

I hope we can come to a solution to the summit that makes all of our 
contributors want to participate more. I believe that giving people more 
responsibility inspires them to participate more fully.


I would like nothing more than all our contributors to participate more. 
But more has to mean caring about not only your stuff.


I was called out today in the hyper-v meeting because I had the a

Re: [openstack-dev] Hyper-V meeting Minutes

[Ben Nemec]

On 2013-10-15 17:02, Robert Collins wrote:
On 16 October 2013 09:54, Vishvananda Ishaya  
wrote:

Hi Everyone,

I've been following this conversation and weighing the different 
sides. This is a tricky issue but I think it is important to decouple 
further and extend our circle of trust.


When nova started it was very easy to do feature development. As it 
has matured the pace has slowed. This is expected and necessary, but 
we periodically must make decoupling decisions or we will become mired 
in overhead. We did this already with cinder and neutron, and we have 
discussed doing this with virt drivers in the past.


We have a large number of people attempting to contribute to small 
sections of nova and getting frustrated with the process.  The 
perception of developers is much more important than the actual 
numbers here. If people are frustrated they are disincentivized to 
help and it hurts everyone. Suggesting that these contributors need to 
learn all of nova and help with the review queue is silly and makes us 
seem elitist. We should make it as easy as possible for new 
contributors to help.



So I agree that perception is a big and significant thing here, but...

I think the fundamental issue is review latency:
http://russellbryant.net/openstack-stats/nova-openreviews.html

Stats since the last revision without -1 or -2 (ignoring jenkins):

Average wait time: 6 days, 17 hours, 24 minutes
1rd quartile wait time: 0 days, 19 hours, 25 minutes
Median wait time: 3 days, 20 hours, 3 minutes
3rd quartile wait time: 7 days, 7 hours, 16 minutes

At the moment 25% of the time Nova reviews sit for over a week [btw we
probably want to not ignore Jenkins in that statistic, as I suspect
reviewers tend to ignore patches that Jenkins hated on].

Now, you may say 'hey, 7 days isn't terrible', but actually it is: if
a developer is producing (say) 2 patches a day, then after 7 calendar
days they may have as many as 10 patches in a vertical queue awaiting
review. Until the patch is reviewed for design-and-architectural
issues (ignoring style stuff which isn't disruptive), all the work
they are doing may need significant rework. Having to redo a week of
work is disruptive to the contributor.

The median wait time of 3 days would nearly halve the amount of rework
that can occur, which would be a significant benefit.

Nova has (over the last 30 days) -
http://russellbryant.net/openstack-stats/nova-reviewers-30.txt:
Total reviews: 3386
Total reviewers: 173

By my count 138 of the reviewers have done less than 20 reviews in
that 30 day period - thats less than one review a day. -
https://docs.google.com/spreadsheet/ccc?key=0AlLkXwa7a4bpdDNjd2gtTE1odjJRYjRVWjhhR2VKQVE&usp=sharing

So, 520 reviews from that 138 folk, but if they did 1 per weekday,
that would be 2760 reviews, bringing the total to 3386+2760-520=5626
reviews - or nearly *double* the review bandwidth.

Now, that won't get you more cores overnight, but that sustained
effort learning about the codebase will bring significant knowledge to
a wide set of folk - and thats what's needed to become core, and
increase the approval bandwidth in the team. I don't know exactly what
russell looks for in managing the set of -core, but surely having
enough knowledge is part of it.

Now, I've nothing against having reviewers specialise in a part of the
code base, and making that official - but I think it must be paired
with still requiring substantial knowledge of the projects code and
architecture: just because something is changing in e.g.
nova/virt/disk/api.py doesn't make it irrelevant to specific virt
drivers, and the right way to use something in the rest of the code
base is also relevant to virt drivers, and knowing *whats there to
reuse* is also important.

So, I guess my proposal is, make a restricted +2 category of reviewer:
 - social agreement to +2 only in enumerated areas
 - still need widespread knowledge of nova's code
 - best demonstrated by sustained regular reviews of other changes
 - but granted after a shorter incubation period
 - migrates to full in time
 - privilege and responsibility lost by the same criteria as all other 
reviewers


Of course, if this has been tried, fine - but AFAICT 'contribute
equally' hasn't been tried yet.


FWIW, this is similar to how things work in Oslo, except that the 
restricted +2 is done through the MAINTAINERS file rather than actual 
granting of +2 authority.  So Oslo still requires a regular core 
reviewer to approve, but it only requires one because a +1 from a 
maintainer qualifies as a +2 (e.g. maintainer +1 and core +2 -> 
approved).  For Nova that might not be quite as simple as just giving +2 
to driver maintainers, but it has the advantage of keeping the people 
with an overall view of the project in the loop.


Details on the Oslo system can be found here: 
https://github.com/openstack/oslo-incubator/blob/master/MAINTAINERS#L17


-Ben

___
OpenStack-dev mailing lis

Re: [openstack-dev] Hyper-V meeting Minutes

[Robert Collins]

On 16 October 2013 09:54, Vishvananda Ishaya  wrote:
> Hi Everyone,
>
> I've been following this conversation and weighing the different sides. This 
> is a tricky issue but I think it is important to decouple further and extend 
> our circle of trust.
>
> When nova started it was very easy to do feature development. As it has 
> matured the pace has slowed. This is expected and necessary, but we 
> periodically must make decoupling decisions or we will become mired in 
> overhead. We did this already with cinder and neutron, and we have discussed 
> doing this with virt drivers in the past.
>
> We have a large number of people attempting to contribute to small sections 
> of nova and getting frustrated with the process.  The perception of 
> developers is much more important than the actual numbers here. If people are 
> frustrated they are disincentivized to help and it hurts everyone. Suggesting 
> that these contributors need to learn all of nova and help with the review 
> queue is silly and makes us seem elitist. We should make it as easy as 
> possible for new contributors to help.


So I agree that perception is a big and significant thing here, but...

I think the fundamental issue is review latency:
http://russellbryant.net/openstack-stats/nova-openreviews.html

Stats since the last revision without -1 or -2 (ignoring jenkins):

Average wait time: 6 days, 17 hours, 24 minutes
1rd quartile wait time: 0 days, 19 hours, 25 minutes
Median wait time: 3 days, 20 hours, 3 minutes
3rd quartile wait time: 7 days, 7 hours, 16 minutes

At the moment 25% of the time Nova reviews sit for over a week [btw we
probably want to not ignore Jenkins in that statistic, as I suspect
reviewers tend to ignore patches that Jenkins hated on].

Now, you may say 'hey, 7 days isn't terrible', but actually it is: if
a developer is producing (say) 2 patches a day, then after 7 calendar
days they may have as many as 10 patches in a vertical queue awaiting
review. Until the patch is reviewed for design-and-architectural
issues (ignoring style stuff which isn't disruptive), all the work
they are doing may need significant rework. Having to redo a week of
work is disruptive to the contributor.

The median wait time of 3 days would nearly halve the amount of rework
that can occur, which would be a significant benefit.

Nova has (over the last 30 days) -
http://russellbryant.net/openstack-stats/nova-reviewers-30.txt:
Total reviews: 3386
Total reviewers: 173

By my count 138 of the reviewers have done less than 20 reviews in
that 30 day period - thats less than one review a day. -
https://docs.google.com/spreadsheet/ccc?key=0AlLkXwa7a4bpdDNjd2gtTE1odjJRYjRVWjhhR2VKQVE&usp=sharing

So, 520 reviews from that 138 folk, but if they did 1 per weekday,
that would be 2760 reviews, bringing the total to 3386+2760-520=5626
reviews - or nearly *double* the review bandwidth.

Now, that won't get you more cores overnight, but that sustained
effort learning about the codebase will bring significant knowledge to
a wide set of folk - and thats what's needed to become core, and
increase the approval bandwidth in the team. I don't know exactly what
russell looks for in managing the set of -core, but surely having
enough knowledge is part of it.

Now, I've nothing against having reviewers specialise in a part of the
code base, and making that official - but I think it must be paired
with still requiring substantial knowledge of the projects code and
architecture: just because something is changing in e.g.
nova/virt/disk/api.py doesn't make it irrelevant to specific virt
drivers, and the right way to use something in the rest of the code
base is also relevant to virt drivers, and knowing *whats there to
reuse* is also important.

So, I guess my proposal is, make a restricted +2 category of reviewer:
 - social agreement to +2 only in enumerated areas
 - still need widespread knowledge of nova's code
 - best demonstrated by sustained regular reviews of other changes
 - but granted after a shorter incubation period
 - migrates to full in time
 - privilege and responsibility lost by the same criteria as all other reviewers

Of course, if this has been tried, fine - but AFAICT 'contribute
equally' hasn't been tried yet.

-Rob



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Rochelle.Grober]

From: Vishvananda Ishaya [mailto:vishvana...@gmail.com]


Hi Everyone,



I've been following this conversation and weighing the different sides. This is 
a tricky issue but I think it is important to decouple further and extend our 
circle of trust.



When nova started it was very easy to do feature development. As it has matured 
the pace has slowed. This is expected and necessary, but we periodically must 
make decoupling decisions or we will become mired in overhead. We did this 
already with cinder and neutron, and we have discussed doing this with virt 
drivers in the past.

+1



We have a large number of people attempting to contribute to small sections of 
nova and getting frustrated with the process.  The perception of developers is 
much more important than the actual numbers here. If people are frustrated they 
are disincentivized to help and it hurts everyone. Suggesting that these 
contributors need to learn all of nova and help with the review queue is silly 
and makes us seem elitist. We should make it as easy as possible for new 
contributors to help.



Agreed.  I suspect the driver developers have more in common with each other, 
even across the different hypervisors, than they have with core Nova 
developers.  If Virt Drivers become their own project, the core reviewers will 
come from the driver community.  Core reviewers will not necessarily be 
reviewing the drivers from their team, but will understand the cogs and gears 
much better.  Plus, this puts all the driver folks on equal footing and 
incentivizes them to review each others' submissions.



I think our current model is breaking down at our current size and we need to 
adopt something more similar to the linux model when dealing with subsystems. 
The hyper-v team is the only one suggesting changes, but there have been 
similar concerns from the vmware team. I have no doubt that there are similar 
issues with the PowerVM, Xen, Docker, lxc and even kvm driver contributors.



Definitely heard the issue from the VMWare guys earlier in the summer.



In my opinion, nova-core needs to be willing to trust the subsystem developers 
and let go of a little bit of control. I frankly don't see the drawbacks.



I'm leaning towards giving control of the subtree to the team as the best 
option because it is simple and works with our current QA system. 
Alternatively, we could split out the driver into a nova subproject (2 below) 
or we could allow them to have a separate branch and do a trusted merge of all 
changes at the end of the cycle (similar to the linux model).



Option 1 or 2.  The last option would be much too painful for everyone and 
could cause all sorts of issues at crunch time.



I hope we can come to a solution to the summit that makes all of our 
contributors want to participate more. I believe that giving people more 
responsibility inspires them to participate more fully.



+1



--Rocky



Vish



On Oct 15, 2013, at 11:24 AM, Russell Bryant  wrote:



> On 10/15/2013 12:52 PM, Peter Pouliot wrote:

>> Hi Everyone,

>>

>>

>> Here are the minutes from today's hyper-v meeting.

>>

>>

>>

>> Minutes:

>> http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-10-15-16.03.html

>>

>> Minutes (text):

>> http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-10-15-16.03.txt

>>

>> Log:

>> http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-10-15-16.03.log.html

>

> I read over the meeting notes and felt it was worth continuing the

> discussion about the home of this driver.  I feel like we're not that

> far from a conclusion, so we don't necessarily have to wait a few weeks

> to talk about it.

>

> In the meeting, the following options were metioned:

>

> 16:16:51  1) we move the code somewhere else in a separate

> repo (e.g.: cloudbase/nova)

> 16:17:26  2) we move the code somewhere else in a separate

> repo in OpenStack (e.g.: openstack/nova-driver-hyperv)

> 16:17:50  errata: on 1) it was meant to be:

> cloudbase/nova-driver-hyperv

> 16:18:48  3) we find a solution in which we get +2 rights

> in our subtree: nova/virt/hyperv and nova/tests/virt/hyperv

>

> I've thought about this quite a bit, and I no longer feel that #2 is an

> option on the table.

>

> #3 is possible, but it's not automatic.  It would happen the same way

> anyone else gets on the core team (through participation and gaining

> trust).  Staying in the tree, and eventually having someone with hyper-v

> expertise on nova-core is the ideal outcome here, IMO.

>

> #1 is certainly an option, and if that's what you want to do, I would

> support that.  Honestly, after reading the full meeting log, it really

> sounds like this is what you want.  You really do want the full control

> that you get with having it be your own project, and that's fine.  I

> feel that there are downsides too, but it's your call.  If you'd like to

> go this route, just let me know so we can coordinate, and we can

Re: [openstack-dev] Hyper-V meeting Minutes

[Vishvananda Ishaya]

Hi Everyone,

I've been following this conversation and weighing the different sides. This is 
a tricky issue but I think it is important to decouple further and extend our 
circle of trust.

When nova started it was very easy to do feature development. As it has matured 
the pace has slowed. This is expected and necessary, but we periodically must 
make decoupling decisions or we will become mired in overhead. We did this 
already with cinder and neutron, and we have discussed doing this with virt 
drivers in the past.

We have a large number of people attempting to contribute to small sections of 
nova and getting frustrated with the process.  The perception of developers is 
much more important than the actual numbers here. If people are frustrated they 
are disincentivized to help and it hurts everyone. Suggesting that these 
contributors need to learn all of nova and help with the review queue is silly 
and makes us seem elitist. We should make it as easy as possible for new 
contributors to help.

I think our current model is breaking down at our current size and we need to 
adopt something more similar to the linux model when dealing with subsystems. 
The hyper-v team is the only one suggesting changes, but there have been 
similar concerns from the vmware team. I have no doubt that there are similar 
issues with the PowerVM, Xen, Docker, lxc and even kvm driver contributors.

In my opinion, nova-core needs to be willing to trust the subsystem developers 
and let go of a little bit of control. I frankly don't see the drawbacks.

I'm leaning towards giving control of the subtree to the team as the best 
option because it is simple and works with our current QA system. 
Alternatively, we could split out the driver into a nova subproject (2 below) 
or we could allow them to have a separate branch and do a trusted merge of all 
changes at the end of the cycle (similar to the linux model).

I hope we can come to a solution to the summit that makes all of our 
contributors want to participate more. I believe that giving people more 
responsibility inspires them to participate more fully.

Vish

On Oct 15, 2013, at 11:24 AM, Russell Bryant  wrote:

> On 10/15/2013 12:52 PM, Peter Pouliot wrote:
>> Hi Everyone,
>> 
>> 
>> Here are the minutes from today’s hyper-v meeting.
>> 
>> 
>> 
>> Minutes:   
>> http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-10-15-16.03.html
>> 
>> Minutes (text):
>> http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-10-15-16.03.txt
>> 
>> Log:   
>> http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-10-15-16.03.log.html
> 
> I read over the meeting notes and felt it was worth continuing the
> discussion about the home of this driver.  I feel like we're not that
> far from a conclusion, so we don't necessarily have to wait a few weeks
> to talk about it.
> 
> In the meeting, the following options were metioned:
> 
> 16:16:51  1) we move the code somewhere else in a separate
> repo (e.g.: cloudbase/nova)
> 16:17:26  2) we move the code somewhere else in a separate
> repo in OpenStack (e.g.: openstack/nova-driver-hyperv)
> 16:17:50  errata: on 1) it was meant to be:
> cloudbase/nova-driver-hyperv
> 16:18:48  3) we find a solution in which we get +2 rights
> in our subtree: nova/virt/hyperv and nova/tests/virt/hyperv
> 
> I've thought about this quite a bit, and I no longer feel that #2 is an
> option on the table.
> 
> #3 is possible, but it's not automatic.  It would happen the same way
> anyone else gets on the core team (through participation and gaining
> trust).  Staying in the tree, and eventually having someone with hyper-v
> expertise on nova-core is the ideal outcome here, IMO.
> 
> #1 is certainly an option, and if that's what you want to do, I would
> support that.  Honestly, after reading the full meeting log, it really
> sounds like this is what you want.  You really do want the full control
> that you get with having it be your own project, and that's fine.  I
> feel that there are downsides too, but it's your call.  If you'd like to
> go this route, just let me know so we can coordinate, and we can remove
> the driver from the nova tree.
> 
> -- 
> Russell Bryant
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V meeting Minutes

[Russell Bryant]

On 10/15/2013 12:52 PM, Peter Pouliot wrote:
> Hi Everyone,
> 
> 
> Here are the minutes from today’s hyper-v meeting.
> 
>  
> 
> Minutes:   
> http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-10-15-16.03.html
> 
> Minutes (text):
> http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-10-15-16.03.txt
> 
> Log:   
> http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-10-15-16.03.log.html

I read over the meeting notes and felt it was worth continuing the
discussion about the home of this driver.  I feel like we're not that
far from a conclusion, so we don't necessarily have to wait a few weeks
to talk about it.

In the meeting, the following options were metioned:

16:16:51  1) we move the code somewhere else in a separate
repo (e.g.: cloudbase/nova)
16:17:26  2) we move the code somewhere else in a separate
repo in OpenStack (e.g.: openstack/nova-driver-hyperv)
16:17:50  errata: on 1) it was meant to be:
cloudbase/nova-driver-hyperv
16:18:48  3) we find a solution in which we get +2 rights
in our subtree: nova/virt/hyperv and nova/tests/virt/hyperv

I've thought about this quite a bit, and I no longer feel that #2 is an
option on the table.

#3 is possible, but it's not automatic.  It would happen the same way
anyone else gets on the core team (through participation and gaining
trust).  Staying in the tree, and eventually having someone with hyper-v
expertise on nova-core is the ideal outcome here, IMO.

#1 is certainly an option, and if that's what you want to do, I would
support that.  Honestly, after reading the full meeting log, it really
sounds like this is what you want.  You really do want the full control
that you get with having it be your own project, and that's fine.  I
feel that there are downsides too, but it's your call.  If you'd like to
go this route, just let me know so we can coordinate, and we can remove
the driver from the nova tree.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Hyper-V Meeting minutes

[Peter Pouliot]

Thank you.  I appreciate you handling it.

P

Sent from my Verizon Wireless 4G LTE Smartphone



 Original message 
From: Alessandro Pilotti 
Date: 08/27/2013 12:40 PM (GMT-05:00)
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] Hyper-V Meeting minutes


Today's Hyper-V meeting minutes:

Minutes: 
http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-08-27-16.06.html
Minutes (text):  
http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-08-27-16.06.txt
Log: 
http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-08-27-16.06.log.html


Thanks,

Alessandro
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev