Public bug reported:
Due to the fact that loopingcall.py uses time.time for recording wall-
clock time which is not guaranteed to be monotonic, if a time drift to
the future occurs, and then gets corrected, all the timers will get
blocked until the actual time reaches the moment of the original drift.
This can be pretty bad if the interval is not insignificant - in Nova's
case - all services uses FixedIntervalLoopingCall for it's heartbeat
periodic tasks - if a drift is on the order of magnitude of several
hours, no heartbeats will happen.
DynamicLoopingCall is affected by this as well but because it relies on
eventlet which would also use a non-monotonic time.time function for
it's internal timers.
Solving this will require looping calls to start using a monotonic timer
(for python 2.7 there is a monotonic package).
Also all services that want to use timers and avoid this issue should
doe something like
import monotonic
hub = eventlet.get_hub()
hub.clock = monotonic.monotonic
immediately after calling eventlet.monkey_patch()
** Affects: nova
Importance: Undecided
Status: New
** Affects: oslo-incubator
Importance: Undecided
Status: New
** Also affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1450438
Title:
loopingcall: if a time drift to the future occurs, all timers will be
blocked
Status in OpenStack Compute (Nova):
New
Status in The Oslo library incubator:
New
Bug description:
Due to the fact that loopingcall.py uses time.time for recording wall-
clock time which is not guaranteed to be monotonic, if a time drift to
the future occurs, and then gets corrected, all the timers will get
blocked until the actual time reaches the moment of the original
drift.
This can be pretty bad if the interval is not insignificant - in
Nova's case - all services uses FixedIntervalLoopingCall for it's
heartbeat periodic tasks - if a drift is on the order of magnitude of
several hours, no heartbeats will happen.
DynamicLoopingCall is affected by this as well but because it relies
on eventlet which would also use a non-monotonic time.time function
for it's internal timers.
Solving this will require looping calls to start using a monotonic
timer (for python 2.7 there is a monotonic package).
Also all services that want to use timers and avoid this issue should
doe something like
import monotonic
hub = eventlet.get_hub()
hub.clock = monotonic.monotonic
immediately after calling eventlet.monkey_patch()
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1450438/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
