We still hit the issue, though the trace is a bit different now:
http://logs.openstack.org/38/284738/69/check/gate-neutron-dsvm-
fullstack-ubuntu-xenial/2e022c5/logs/syslog.txt.gz
Apr 28 17:24:20 ubuntu-xenial-rax-ord-8648308
neutron-keepalived-state-change[21615]: 2017-04-28 17:24:20.423 21615 CRITICAL
neutron [-] AssertionError: do not call blocking functions from the mainloop
2017-04-28 17:24:20.423 21615 ERROR neutron Traceback (most recent call
last):
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/.tox/dsvm-fullstack/bin/neutron-keepalived-state-change",
line 10, in <module>
2017-04-28 17:24:20.423 21615 ERROR neutron sys.exit(main())
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/neutron/cmd/keepalived_state_change.py", line 19, in
main
2017-04-28 17:24:20.423 21615 ERROR neutron
keepalived_state_change.main()
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/neutron/agent/l3/keepalived_state_change.py", line 156,
in main
2017-04-28 17:24:20.423 21615 ERROR neutron
cfg.CONF.monitor_cidr).start()
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/neutron/agent/linux/daemon.py", line 253, in start
2017-04-28 17:24:20.423 21615 ERROR neutron self.run()
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/neutron/agent/l3/keepalived_state_change.py", line 69,
in run
2017-04-28 17:24:20.423 21615 ERROR neutron for iterable in
self.monitor:
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/neutron/agent/linux/async_process.py", line 261, in
_iter_queue
2017-04-28 17:24:20.423 21615 ERROR neutron yield
queue.get(block=block)
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/queue.py",
line 313, in get
2017-04-28 17:24:20.423 21615 ERROR neutron return waiter.wait()
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/queue.py",
line 141, in wait
2017-04-28 17:24:20.423 21615 ERROR neutron return get_hub().switch()
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/hubs/hub.py",
line 294, in switch
2017-04-28 17:24:20.423 21615 ERROR neutron return
self.greenlet.switch()
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/hubs/hub.py",
line 346, in run
2017-04-28 17:24:20.423 21615 ERROR neutron self.wait(sleep_time)
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/hubs/poll.py",
line 85, in wait
2017-04-28 17:24:20.423 21615 ERROR neutron presult =
self.do_poll(seconds)
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/hubs/epolls.py",
line 62, in do_poll
2017-04-28 17:24:20.423 21615 ERROR neutron return
self.poll.poll(seconds)
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/neutron/agent/l3/keepalived_state_change.py", line 133,
in handle_sigterm
2017-04-28 17:24:20.423 21615 ERROR neutron self._kill_monitor()
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/neutron/agent/l3/keepalived_state_change.py", line 130,
in _kill_monitor
2017-04-28 17:24:20.423 21615 ERROR neutron run_as_root=True)
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 225, in kill_process
2017-04-28 17:24:20.423 21615 ERROR neutron execute(['kill', '-%d' %
signal, pid], run_as_root=run_as_root)
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 131, in execute
2017-04-28 17:24:20.423 21615 ERROR neutron _stdout, _stderr =
obj.communicate(_process_input)
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/usr/lib/python2.7/subprocess.py", line 800, in communicate
2017-04-28 17:24:20.423 21615 ERROR neutron return
self._communicate(input)
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/usr/lib/python2.7/subprocess.py", line 1419, in _communicate
2017-04-28 17:24:20.423 21615 ERROR neutron stdout, stderr =
self._communicate_with_select(input)
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/usr/lib/python2.7/subprocess.py", line 1520, in _communicate_with_select
2017-04-28 17:24:20.423 21615 ERROR neutron rlist, wlist, xlist =
select.select(read_set, write_set, [])
2017-04-28 17:24:20.423 21615 ERROR neutron File
"/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/green/select.py",
line 40, in select
2017-04-28 17:24:20.423 21615 ERROR neutron assert hub.greenlet is
not current, 'do not call blocking functions from the mainloop'
2017-04-28 17:24:20.423 21615 ERROR neutron AssertionError: do not call
blocking functions from the mainloop
2017-04-28 17:24:20.423 21615 ERROR neutron
We probably need to rework how we handle the signal, not calling
execute() from inside the handler.
** Changed in: neutron
Status: Fix Released => Confirmed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1680183
Title:
neutron-keepalived-state-change fails with "AssertionError: do not
call blocking functions from the mainloop"
Status in neutron:
Confirmed
Bug description:
17:39:17.802 6173 CRITICAL neutron [-] AssertionError: do not call blocking
functions from the mainloop
17:39:17.802 6173 ERROR neutron Traceback (most recent call last):
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/.tox/dsvm-functional/bin/neutron-keepalived-state-change",
line 10, in <module>
17:39:17.802 6173 ERROR neutron sys.exit(main())
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/neutron/cmd/keepalived_state_change.py", line 19, in main
17:39:17.802 6173 ERROR neutron keepalived_state_change.main()
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/neutron/agent/l3/keepalived_state_change.py", line 157, in
main
17:39:17.802 6173 ERROR neutron cfg.CONF.monitor_cidr).start()
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/neutron/agent/linux/daemon.py", line 249, in start
17:39:17.802 6173 ERROR neutron self.run()
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/neutron/agent/l3/keepalived_state_change.py", line 70, in
run
17:39:17.802 6173 ERROR neutron for iterable in self.monitor:
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/neutron/agent/linux/async_process.py", line 256, in
_iter_queue
17:39:17.802 6173 ERROR neutron yield queue.get(block=block)
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/eventlet/queue.py",
line 313, in get
17:39:17.802 6173 ERROR neutron return waiter.wait()
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/eventlet/queue.py",
line 141, in wait
17:39:17.802 6173 ERROR neutron return get_hub().switch()
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/eventlet/hubs/hub.py",
line 294, in switch
17:39:17.802 6173 ERROR neutron return self.greenlet.switch()
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/eventlet/hubs/hub.py",
line 346, in run
17:39:17.802 6173 ERROR neutron self.wait(sleep_time)
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/eventlet/hubs/poll.py",
line 85, in wait
17:39:17.802 6173 ERROR neutron presult = self.do_poll(seconds)
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/eventlet/hubs/epolls.py",
line 62, in do_poll
17:39:17.802 6173 ERROR neutron return self.poll.poll(seconds)
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/neutron/agent/l3/keepalived_state_change.py", line 134, in
handle_sigterm
17:39:17.802 6173 ERROR neutron self._kill_monitor()
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/neutron/agent/l3/keepalived_state_change.py", line 131, in
_kill_monitor
17:39:17.802 6173 ERROR neutron run_as_root=True)
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/neutron/agent/linux/utils.py", line 221, in kill_process
17:39:17.802 6173 ERROR neutron execute(['kill', '-%d' % signal, pid],
run_as_root=run_as_root)
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/neutron/agent/linux/utils.py", line 155, in execute
17:39:17.802 6173 ERROR neutron greenthread.sleep(0)
17:39:17.802 6173 ERROR neutron File
"/opt/stack/neutron/.tox/dsvm-functional/lib/python2.7/site-packages/eventlet/greenthread.py",
line 31, in sleep
17:39:17.802 6173 ERROR neutron assert hub.greenlet is not current, 'do
not call blocking functions from the mainloop'
17:39:17.802 6173 ERROR neutron AssertionError: do not call blocking
functions from the mainloop
17:39:17.802 6173 ERROR neutron
This is what I see when running fullstack l3ha tests, once I enable
syslog logging for the helper process.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1680183/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
