[issue44036] asyncio SSL server can be DOSed, event loop gets blocked: busy loops and uses 100% CPU

[Andrew Svetlov]

Andrew Svetlov  added the comment:

Could you check Python 3.11?
It has a new asyncio SSL implementation rewritten from scratch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44036] asyncio SSL server can be DOSed, event loop gets blocked: busy loops and uses 100% CPU

[Christian Heimes]

Change by Christian Heimes :


--
assignee: christian.heimes -> 
components:  -SSL

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44036] asyncio SSL server can be DOSed, event loop gets blocked: busy loops and uses 100% CPU

[Neil Booth]

Change by Neil Booth :


--
nosy: +kyuupichan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44036] asyncio SSL server can be DOSed, event loop gets blocked: busy loops and uses 100% CPU

[ghost43]

New submission from ghost43 :

This is about a potential DOS vector that can get an asyncio server serving SSL 
connections to enter a busy loop and hang. To recover the server (python 
process) needs to be restarted.

See downstream report at https://github.com/spesmilo/electrumx/issues/92

ElectrumX is server software that serves JSON-RPC requests either directly over 
plaintext TCP or over SSL/TLS (no HTTP involved). The server code is written 
using asyncio. The most popular client is also written using python+asyncio. 
However, there are other server implementations (e.g. in Rust), and other 
clients (e.g. in javascript, scala, java). The servers are part of a federated 
network and peer with each other.

In the last 1+ year, server operators have started reporting that the python 
process (of ElectrumX) sporadically hangs: it uses 100% CPU and stops serving 
clients or accepting new connections. The only way we could find to recover is 
restarting the server (the python process); waiting for even days did not help, 
python was stuck. The hang seemed to mostly happen on busy servers that serve 
several thousands of concurrent sessions, and even on those only happened e.g. 
once a month. So the hang was difficult to reproduce.

Nevertheless we observed that the hang only happens if it is the asyncio server 
that handles SSL, i.e. if the server operator put nginx in front of the python 
process and handles SSL termination in nginx, they would be unaffected. One 
server operator whose server at one point hanged multiple times a day confirmed 
this, and reported that nginx subsequently started logging lines like this:
```
2021/01/11 02:28:30 [crit] 21#21: *437620 SSL_shutdown() failed (SSL: 
error:14094123:SSL routines:ssl3_read_bytes:application data after close 
notify) while proxying connection, client: REDACTED, server: 0.0.0.0:50002, 
upstream: "127.0.0.1:50213", bytes from/to client:81/205, bytes from/to 
upstream:205/0
```

Over these last months, the hang has been reproduced on many different python 
versions by different people, e.g. 3.7.1, 3.7.5, 3.8.5, 3.9.1, 3.9.4.

A few days ago, many servers hanged almost simultaneously, and when restarted, 
they would soon hang again at most a few hours later. Presumably someone either 
flooded the network with buggy clients, or it might have been an attack. 
Anyway, this allowed me to look into this better. I set up gdb and waited. This 
was on ubuntu 20.04 with python 3.8.5 and openssl=1.1.1f-1ubuntu2.3.

It seems the event loop thread is stuck in a busy loop.
The deepest common stack frame looks to be at
https://github.com/python/cpython/blob/v3.8.5/Lib/asyncio/sslproto.py#L675

```
(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 535, in feed_appdata

  File "/usr/lib/python3.8/asyncio/sslproto.py", line 675, in 
_process_write_backlog
ssldata, offset = self._sslpipe.feed_appdata(data, offset)
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 599, in _write_appdata
self._process_write_backlog()
  File "/usr/lib/python3.8/asyncio/sslproto.py", line 387, in write
self._ssl_protocol._write_appdata(data)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/rawsocket.py", line 118, 
in write
self._asyncio_transport.write(framed_message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 153, 
in _send_message
await self.transport.write(message)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/session.py", line 496, 
in _throttled_request
await self._send_message(message)
  
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 2627, in _run_once
--Type  for more, q to quit, c to continue without paging--
  File "/usr/lib/python3.8/asyncio/base_events.py", line 826, in run_forever
None, getaddr_func, host, port, family, type, proto, flags)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in 
run_until_complete
self.run_forever()
  File "/usr/lib/python3.8/asyncio/runners.py", line 299, in run
  File "/usr/local/bin/electrumx_server", line 35, in main
asyncio.run(controller.run())
  File "/usr/local/bin/electrumx_server", line 43, in 
main()
```
```
(gdb) py-list
 530except (SystemExit, KeyboardInterrupt):
 531raise
 532except BaseException as e:
 533self._fatal_error(e, 'SSL error in data received')
 534return
>535
 536for chunk in ssldata:
 537self._transport.write(chunk)
 538
 539for chunk in appdata:
 540if chunk:
```
```
(gdb) py-locals
self = <_SSLPipe(_context=, 
_server_side=True, _server_hostname=None, _state='WRAPPED', 
_incoming=<_ssl.MemoryBIO at remote 0x7f09cc0925b0>, _outgoing=<_ssl.MemoryBIO 
at remote 0x7f09b919c430>, _sslobj=) at remote 0x7f09db428910>,