Matt Riedemann wrote:
On 12/1/2015 2:07 PM, Joshua Harlow wrote:
So whats needed to build out that 'framework'?
Is it just dispatching provision requests to nova, and then seeing when
the quota becomes out of sync, and then backtracking through logs and
notifications (+- other) to then figure out the root cause?
Or is that framework some kind of local functional testing built-in to
nova itself, something that can work without probing via the REST
endpoints...?
IMHO it'd be nice to have something like rally scenarios that probe the
quota (via REST or not) but with post-validation that can be done on the
post scenario run and/or post request run. The post-validation ensures
the quota is as expected, using some locally computed 'expected' quota
that nova should also have locally computed, if those two computed
'expected' quotas are different, then backtracking/log analysis... needs
to occur to figure out when the computed values started to diverge (and
to figure out why they diverged); rinse and repeat that and u probably
have a pretty powerful validation framework...
Vilobh Meshram wrote:
I am highly supportive for the idea of Nova Quota sub-team, for
something as complex as Quota, as it helps to move quickly on reviews
and changes.
Agree with John, a test framework to test quotas will be helpful and can
be one of the first task the Nova Quota sub team can focus on as that
will lay the foundation for whether the bugs mentioned here
http://bit.ly/1Pbr8YL are valid or not.
Having worked in the area of Quotas for a while now by introducing
features like Cinder Nested Quota Driver [1] [2] I strongly feel that
something like a Nova Quota sub-team will definitely help. Mentioning
about Cinder Quota driver since it was accepted in Mitaka design summit
that Nova Nested Quota Driver[3] would like to pursue the route taken by
Cinder. Since Nested quota is a one part of Quota subsystem and working
in small team helped to iterate quickly for Nested Quota
patches[4][5][6][7] so IMHO forming a Nova quota subteam will help.
Melanie,
If you can share the details of the bug that Joe mentioned, to reproduce
quota bugs locally, it would be helpful.
-Vilobh (irc: vilobhmm)
[1] Code : https://review.openstack.org/#/c/205369/
[2] Blueprint :
https://blueprints.launchpad.net/cinder/+spec/cinder-nested-quota-driver
[3] Nova Nested Quota Spec : https://review.openstack.org/#/c/209969/
[4] https://review.openstack.org/#/c/242568/
[5] https://review.openstack.org/#/c/242500/
[6] https://review.openstack.org/#/c/242514/
[7] https://review.openstack.org/#/c/242626/
On Mon, Nov 30, 2015 at 10:59 AM, melanie witt <melwi...@gmail.com
<mailto:melwi...@gmail.com>> wrote:
On Nov 26, 2015, at 9:36, John Garbutt <j...@johngarbutt.com
<mailto:j...@johngarbutt.com>> wrote:
> A suggestion in the past, that I like, is creating a nova
functional
> test that stress tests the quota code.
>
> Hopefully that will be able to help reproduce the error.
> That should help prove if any proposed fix actually works.
+1, I think it's wise to get some data on the current state of
quotas before choosing a redesign. IIRC, Joe Gordon described a test
scenario he used to use to reproduce quota bugs locally, in one of
the launchpad bugs. If we could automate something like that, we
could use it to demonstrate how quotas currently behave during
parallel requests and try things like disabling reservations. I also
like the idea of being able to verify the effects of proposed fixes.
-melanie (irc: melwitt)
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
I seem to remember jogo teasing out quota bugs by randomly restarting
the nova-compute service, I'm not sure how easy that is to recreate in a
functional test setup though. We do have functional tests in tree that
run a nova-compute service though with a backing database (sqlite), so
it seems we could kick off some API request, restart the compute service
immediately, and then check quotas once the compute service is back up.
It's using the fake virt driver though so everything is going to be very
fast and that might make finding races difficult.
Cool, I have once upon a time created,
https://github.com/harlowja/quietus#quietus which can offer controlled
(aka, repeatable restarts and such); people are free to mess around with
that. This 'framework' likely also needs something like a
`LocalQuotaEngine` that calculates its own expected quota that can then
be compared to what the nova QuotaEngine believes it computed (and
divergence here means things are going badly).
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
