[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[Martin Panter]

Martin Panter  added the comment:

FYI the change here to remember all the thread objects ever created looks to be 
the cause of the memory leak reported in Issue 37193

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor  added the comment:

The bug has been fixed in Python 3.7.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor  added the comment:

I created bpo-33540: "socketserver: Add an opt-in option to get Python 3.6 
behaviour on server_close()". I set the release blocker priority on this new 
issue.

--
priority: deferred blocker -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor  added the comment:


New changeset db8189bb8db609ca3993dec31fb95fdbe134469a by Victor Stinner in 
branch 'master':
bpo-31233, bpo-31151: Document socketserver changes (#5417)
https://github.com/python/cpython/commit/db8189bb8db609ca3993dec31fb95fdbe134469a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +5251

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[Ned Deily]

Ned Deily  added the comment:

If we're going to leave this open until 3.7.0b2, can you at least add a brief 
warning in the "Porting section" for b1 that the issue is unresolved?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor  added the comment:

> What's the status of this?

The status is bad. The commit b8f4163da30e16c7cd58fe04f4b17e38d53cd57e is 
backward incompatible, it's not documented in the "Porting the Python 3.7" 
section of What's New in Python 3.7, and it's not possible to opt-in for the 
previous behaviour.


> Feature freeze is here.

IMHO it's ok to post-pone this issue to beta2.

IMHO it's better to wait until all threads complete on server_close(), and it's 
possible to workaround the issue using daemon_threads = True which can be seen 
as the option to opt-in for the previous behaviour.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[Ned Deily]

Ned Deily  added the comment:

What's the status of this?  Feature freeze is here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor  added the comment:

Klon666 added 
https://github.com/python/cpython/commit/96a6cbc5b0aee267f47e5efb50fba183b52cd8c6
 repository. But this URL is unrelated. It looks like a bot spamming the bug 
tracker and I don't see how to remove the URL...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[Berker Peksag]

Change by Berker Peksag :


--
components:  -Build
nosy:  -Tu madre
type: security -> resource usage

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[Klon666]

Change by Klon666 :


--
components: +Build
nosy: +Tu madre
type: resource usage -> security

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[Klon666]

Change by Klon666 :


--
hgrepos: +376

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor  added the comment:

The current code leaks memory since it never clears threads which completed. We 
need a cleanup function similar to ForkingMixIn.collect_children() which is 
called by handle_timeout() and service_actions().

We can check if a thread is alive: thread.is_alive(), to decide to remove it or 
not.

Moreover, maybe we should keep a list of weak references?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[Ned Deily]

Changes by Ned Deily :


--
priority: release blocker -> deferred blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor added the comment:


New changeset b8f4163da30e16c7cd58fe04f4b17e38d53cd57e by Victor Stinner in 
branch 'master':
bpo-31233: socketserver.ThreadingMixIn.server_close() (#3523)
https://github.com/python/cpython/commit/b8f4163da30e16c7cd58fe04f4b17e38d53cd57e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor added the comment:


New changeset 97d7e65dfed1d42d40d9bc2f630af56240555f02 by Victor Stinner in 
branch 'master':
bpo-30830: logging.config.listen() calls server_close() (#3524)
https://github.com/python/cpython/commit/97d7e65dfed1d42d40d9bc2f630af56240555f02


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor added the comment:

Again, PR 3523 is a temporary fix to unblock the situation.

For a proper fix, I have a question: Should server_close() calls 
self.close_request(request) on thread requests?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +3524

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor added the comment:

Attached PR 3523 is a temporary solution until we agree how to handle these 
threads and child processes in socketserver. I wrote the PR to fix "dangling 
threads" warnings to fix random buildbot failures.

The PR fixes warnings but also reenable test_logging tests which are currently 
skipped. These tests were skipped by bpo-30830 to prevent random buildbot 
failures.

See also the thread on the python-dev mailing list to discuss about bpo-31151 
"socketserver.ForkingMixIn.server_close() leaks zombie processes":
https://mail.python.org/pipermail/python-dev/2017-August/148826.html

I tag this issue as release blocker as a remainder that we have to agree how to 
handle threads/processes before Python 3.7 feature freeze.

--
nosy: +ned.deily
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

Changes by STINNER Victor :


--
keywords: +patch
pull_requests: +3521
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor added the comment:

Multiple test_logging tests have been skipped until this issue is fixed: see 
bpo-30830 and commit 6966960468327c958b03391f71f24986bd697307.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor added the comment:

Another recent example while testing bpo-31234:

test_ThreadingUDPServer (test.test_socketserver.SocketServerTest) ...
(...)
done
Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor added the comment:


New changeset 6966960468327c958b03391f71f24986bd697307 by Victor Stinner in 
branch 'master':
bpo-30830: test_logging uses threading_setup/cleanup (#3137)
https://github.com/python/cpython/commit/6966960468327c958b03391f71f24986bd697307


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

STINNER Victor added the comment:

Oh, test_socketserver just failed with ENV_CHANGED on AMD64 Windows7 SP1 3.x:

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/865/steps/test/logs/stdio

test_TCPServer (test.test_socketserver.SocketServerTest) ... creating server
(...)
waiting for server
done

Warning -- threading_cleanup() failed to cleanup -1 threads after 5 sec (count: 
0, dangling: 1)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[Antoine Pitrou]

Antoine Pitrou added the comment:

The difference is that letting a thread run doesn't create a zombie thread, so 
I don't think the issue is really similar.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31233] socketserver.ThreadingMixIn leaks running threads after server_close()

[STINNER Victor]

New submission from STINNER Victor:

test_logging has a race condition: sometimes, it leaks dangling threads on 
FreeBSD: bpo-30830.

I identified the root issue: socketserver.ThreadingMixIn spawns threads without 
waiting for their completion in server_close(). This issue is very similar to 
socketserver.ForkingMixIn which leaks child processes and so create zombie 
processes. See bpo-31151.

--
components: Library (Lib)
messages: 300514
nosy: haypo
priority: normal
severity: normal
status: open
title: socketserver.ThreadingMixIn leaks running threads after server_close()
type: resource usage
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com