Public bug reported: This traceback below was found in the neutron api server log. It happened about one minute after the rabbitmq server was restarted after the msgq service was out. Afterwards, noticed that the api server is no longer responding to requests from python-neutronclient. However, the api-server continues to respond to RPC calls from L3 and DHCP agents even after the traceback was reported.
To get the api server to fully functional again, I have to stop and restart the neutron api server. There are multiple workers for the api server. The traceback: 2014-05-28 08:46:16.977 16091 CRITICAL quantum [-] [Errno 24] Too many open files 2014-05-28 08:46:16.977 16091 TRACE quantum Traceback (most recent call last): 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/bin/quantum-server", line 27, in <module> 2014-05-28 08:46:16.977 16091 TRACE quantum server() 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/server/__init__.py", line 38, in main 2014-05-28 08:46:16.977 16091 TRACE quantum quantum_service = service.serve_wsgi(service.QuantumApiService) 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/service.py", line 102, in serve_wsgi 2014-05-28 08:46:16.977 16091 TRACE quantum service.start() 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/service.py", line 69, in start 2014-05-28 08:46:16.977 16091 TRACE quantum self.wsgi_app = _run_wsgi(self.app_name) 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/service.py", line 113, in _run_wsgi 2014-05-28 08:46:16.977 16091 TRACE quantum server.start(app, cfg.CONF.bind_port, cfg.CONF.bind_host, workers=cfg.CONF.workers) 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 206, in start 2014-05-28 08:46:16.977 16091 TRACE quantum self.run_child() 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 265, in run_child 2014-05-28 08:46:16.977 16091 TRACE quantum self._run(self._application, self._socket) 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 277, in _run 2014-05-28 08:46:16.977 16091 TRACE quantum log=logging.WritableLogger(logger)) 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 655, in server 2014-05-28 08:46:16.977 16091 TRACE quantum client_socket = sock.accept() 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 154, in accept 2014-05-28 08:46:16.977 16091 TRACE quantum res = socket_accept(fd) 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 52, in socket_accept 2014-05-28 08:46:16.977 16091 TRACE quantum return descriptor.accept() 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/socket.py", line 202, in accept 2014-05-28 08:46:16.977 16091 TRACE quantum sock, addr = self._sock.accept() 2014-05-28 08:46:16.977 16091 TRACE quantum error: [Errno 24] Too many open files 2014-05-28 08:46:16.977 16091 TRACE quantum ** Affects: neutron Importance: Undecided Status: New ** Tags: api files many neutron open too -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1331877 Title: Neutron api server stopped working due to Errno 24 Status in OpenStack Neutron (virtual network service): New Bug description: This traceback below was found in the neutron api server log. It happened about one minute after the rabbitmq server was restarted after the msgq service was out. Afterwards, noticed that the api server is no longer responding to requests from python-neutronclient. However, the api-server continues to respond to RPC calls from L3 and DHCP agents even after the traceback was reported. To get the api server to fully functional again, I have to stop and restart the neutron api server. There are multiple workers for the api server. The traceback: 2014-05-28 08:46:16.977 16091 CRITICAL quantum [-] [Errno 24] Too many open files 2014-05-28 08:46:16.977 16091 TRACE quantum Traceback (most recent call last): 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/bin/quantum-server", line 27, in <module> 2014-05-28 08:46:16.977 16091 TRACE quantum server() 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/server/__init__.py", line 38, in main 2014-05-28 08:46:16.977 16091 TRACE quantum quantum_service = service.serve_wsgi(service.QuantumApiService) 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/service.py", line 102, in serve_wsgi 2014-05-28 08:46:16.977 16091 TRACE quantum service.start() 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/service.py", line 69, in start 2014-05-28 08:46:16.977 16091 TRACE quantum self.wsgi_app = _run_wsgi(self.app_name) 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/service.py", line 113, in _run_wsgi 2014-05-28 08:46:16.977 16091 TRACE quantum server.start(app, cfg.CONF.bind_port, cfg.CONF.bind_host, workers=cfg.CONF.workers) 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 206, in start 2014-05-28 08:46:16.977 16091 TRACE quantum self.run_child() 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 265, in run_child 2014-05-28 08:46:16.977 16091 TRACE quantum self._run(self._application, self._socket) 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/quantum/wsgi.py", line 277, in _run 2014-05-28 08:46:16.977 16091 TRACE quantum log=logging.WritableLogger(logger)) 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 655, in server 2014-05-28 08:46:16.977 16091 TRACE quantum client_socket = sock.accept() 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 154, in accept 2014-05-28 08:46:16.977 16091 TRACE quantum res = socket_accept(fd) 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 52, in socket_accept 2014-05-28 08:46:16.977 16091 TRACE quantum return descriptor.accept() 2014-05-28 08:46:16.977 16091 TRACE quantum File "/usr/lib/python2.7/socket.py", line 202, in accept 2014-05-28 08:46:16.977 16091 TRACE quantum sock, addr = self._sock.accept() 2014-05-28 08:46:16.977 16091 TRACE quantum error: [Errno 24] Too many open files 2014-05-28 08:46:16.977 16091 TRACE quantum To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1331877/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp