[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

STINNER Victor added the comment: Thank you for fixing the regression Jason R. Coombs ;-) -- ___ Python tracker ___ ___

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

miss-islington added the comment: New changeset 0064d561b8e44f7a991188d7c5016c165bc89326 by Miss Islington (bot) in branch '3.8': [3.8] bpo-37193: Remove thread objects which finished process its request (GH-23127) (GH-24749)

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

miss-islington added the comment: New changeset 0e76157b0ca70bd38157fed56680a7752a52668d by Miss Islington (bot) in branch '3.9': [3.9] bpo-37193: Remove thread objects which finished process its request (GH-23127) (GH-24750)

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by miss-islington : -- pull_requests: +23520 pull_request: https://github.com/python/cpython/pull/24750 ___ Python tracker ___

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by miss-islington : -- pull_requests: +23519 pull_request: https://github.com/python/cpython/pull/24749 ___ Python tracker ___

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by miss-islington : -- pull_requests: +22874 pull_request: https://github.com/python/cpython/pull/24033 ___ Python tracker ___

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by miss-islington : -- pull_requests: +22873 pull_request: https://github.com/python/cpython/pull/24032 ___ Python tracker ___

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by Pablo Galindo Salgado : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Pablo Galindo Salgado added the comment: New changeset b5711c940f70af89f2b4cf081a3fcd83924f3ae7 by Jason R. Coombs in branch 'master': bpo-37193: Remove thread objects which finished process its request (GH-23127)

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Jason R. Coombs added the comment: I filed issue42263 to capture the underlying cause of the memory leak that led to the buildbot failures and the rollback. -- ___ Python tracker

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by Jason R. Coombs : -- pull_requests: +22043 pull_request: https://github.com/python/cpython/pull/23127 ___ Python tracker ___

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Pablo Galindo Salgado added the comment: New changeset aca67da4fe68d5420401ac1782203d302875eb27 by Jason R. Coombs in branch 'master': Revert "bpo-37193: remove thread objects which finished process its request (GH-13893)" (GH-23107)

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by Jason R. Coombs : -- pull_requests: +22024 pull_request: https://github.com/python/cpython/pull/23107 ___ Python tracker ___

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Jason R. Coombs added the comment: I recommend a rollback. I’ll try to get to it later today. -- ___ Python tracker ___ ___

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

STINNER Victor added the comment: I rejected the backport to 3.8 and 3.9 since the change causes a regression on master. -- ___ Python tracker ___

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

STINNER Victor added the comment: The change fixing a leak in socketserver introduces a leak in socketserver :-) $ ./python -m test test_socketserver -u all -m test.test_socketserver.SocketServerTest.test_ThreadingTCPServer -R 3:3 0:00:00 load avg: 0.95 Run tests sequentially 0:00:00 load

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Pablo Galindo Salgado added the comment: Commit c41559021213cfc9dc62a83fc63306b3bdc3e64b has introduced reference leaks: -- Ran 202 tests in 21.654s OK (skipped=1) .. test_logging leaked [20, 20, 20] references, sum=60

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by miss-islington : -- pull_requests: +22006 pull_request: https://github.com/python/cpython/pull/23089 ___ Python tracker ___

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by miss-islington : -- pull_requests: +22005 pull_request: https://github.com/python/cpython/pull/23088 ___ Python tracker ___

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Jason R. Coombs added the comment: New changeset c41559021213cfc9dc62a83fc63306b3bdc3e64b by MARUYAMA Norihiro in branch 'master': bpo-37193: remove thread objects which finished process its request (GH-13893) https://github.com/python/cpython/commit/c41559021213cfc9dc62a83fc63306b3bdc3e64b

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by miss-islington : -- nosy: +miss-islington nosy_count: 6.0 -> 7.0 pull_requests: +22004 pull_request: https://github.com/python/cpython/pull/23087 ___ Python tracker

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Ned Deily added the comment: > Any chance this can get into Python 3.7? Perhaps but there's a lot that needs to be done yet. Like any bugfix, it needs to be reviewed, merged to master, and get some buildbot exposure first before it is backported anywhere. --

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Jason R. Coombs added the comment: Thanks for the notice Ned. I've revived the PR and addressed all the comments from Victor. Any chance this can get into Python 3.7? -- ___ Python tracker

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by Gregory P. Smith : -- versions: +Python 3.10, Python 3.8, Python 3.9 ___ Python tracker ___ ___ Python-bugs-list mailing

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Ned Deily added the comment: I note this is marked as a 3.7regression and still open. Since the cutoff for the final 3.7 bugfix mode release is in a few days, I'm assuming this means that 3.7 users will have to live with this regression. If you feel that is a problem, speak up now.

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Martin Panter added the comment: FTR I have been trialling a patched Python 3.7 based on Maru's changes (revision 6ac217c) + review suggestions, and it has reduced the size of the leak (hit 1 GB over a couple days, vs only 60 MB increase over three days). The remaining leak could be

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Martin Panter added the comment: Another workaround might be to set the new "block_on_close" flag (Issue 33540) to False on the server subclass or instance. Victor: Replying to "What do I think of also using a weakref?", I assume you mean

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Jason R. Coombs added the comment: This issue was also [reported in prometheus client](https://salsa.debian.org/debian/python-prometheus-client/commit/5aa256d8aab3b81604b855dc03f260342fc391fb) where the workaround was to use daemon threads. --

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by Jason R. Coombs : -- nosy: +jaraco ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

STINNER Victor added the comment: I marked bpo-37389 as a duplicate of this issue: """ msg346410 - (view) Author: Daniel W Forsyth (d...@dataforge.on.ca) Date: 2019-06-24 14:53 After putting a basic ThreadingUDPServer under load (500 messages per/second) I noticed that after

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

STINNER Victor added the comment: Martin Panter: In addition to PR 13893 change, what do you think of also using a weakref? It might make the code even more reliable if something goes wrong. -- ___ Python tracker

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

STINNER Victor added the comment: PR 13893 with an additional lock sounds like a reasonable solution. The code should be skipped if the thread is a daemon thread. -- ___ Python tracker

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Martin Panter added the comment: Looking at the code, this would be caused by Issue 31233. I expect 3.7+ is affected. 3.6 has similar code, but the leaking looks to be disabled by default. 2.7 doesn't collect a "_threads" list at all. Looks like Victor was aware of the leak when he changed

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Wei Li added the comment: I got the same problem when uing the ThreadingTCPServer. I think adding "self._threads = list(filter(lambda x: x.is_alive(), self._threads))" at the last line in process_request method is a potential way to fix the bug -- nosy: +Wei Li

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

Change by Norihiro Maruyama : -- keywords: +patch pull_requests: +13767 stage: -> patch review pull_request: https://github.com/python/cpython/pull/13893 ___ Python tracker

[issue37193] Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn

New submission from Norihiro Maruyama : UDP/TCPServer with socketserver.ThreadingMixin class (also ThreadingTCPServer and ThreadingUDPServer class) seems to be memory leak while running the server. https://docs.python.org/3/library/socketserver.html#socketserver.ThreadingMixIn My code which