[issue19530] cross thread shutdown of UDP socket exhibits unexpected behavior
Charles-François Natali added the comment: mpb added the comment: Someone wrote a kernel patch based on my bug report. http://www.spinics.net/lists/netdev/msg257653.html It's just a patch to avoid returning garbage in the address. But AFAICT, recvfrom() returning 0 is enough to know that the socket was shut down. But two things to keep in mind: - it'll only work on connected datagram sockets - even then, I'm not sure it's supported by POSIX: I can't think of any spec specifying the behavior in case of cross-thread shutdown (and close won't unblock for example). Also, I think HP-UX doesn't wake up the waiting thread in that situation. So I'd still advise you to either use a timeout or a select(). Cheers, -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19530] cross thread shutdown of UDP socket exhibits unexpected behavior
mpb added the comment: It's just a patch to avoid returning garbage in the address. Right, which is why I pursued the point. recvfrom should not return ambiguous data (the ambiguity being between shutdown and receiving a zero length message). It is now possible to distinguish the two by looking at the src_addr. (Arguably this could have been done before, but garbage in src_addr is not a reliable indicator, IMO.) But AFAICT, recvfrom() returning 0 is enough to know that the socket was shut down. My example code clearly shows a zero length UPD message being sent and received prior to shutdown. I admit, sending a zero length UDP message is probably pretty rare, but it is allowed and it does work. And it makes more sense than returning garbage in src_addr. But two things to keep in mind: - it'll only work on connected datagram sockets What will only work on connected datagram sockets? Shutdown *already* works (ie, wakes up blocked threads) on non-connected datagram sockets on Linux. Shutdown does wake them up (it just happens to return an error *after* waking them up). So... the only reason to connect the UDP socket (prior to calling shutdown) is to avoid the error (or, in Python, to avoid the raised Exception). - even then, I'm not sure it's supported by POSIX: I can't think of any spec specifying the behavior in case of cross-thread shutdown (and close won't unblock for example). Also, I think HP-UX doesn't wake up the waiting thread in that situation. Do you consider the POSIX specifications to be robust when it comes to threading? It would not surprise me if there are other threading related grey areas in POSIX. So I'd still advise you to either use a timeout or a select(). My application only needs to run on Linux. If I cared about portability, I might well do something else. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19530] cross thread shutdown of UDP socket exhibits unexpected behavior
mpb added the comment: Someone wrote a kernel patch based on my bug report. http://www.spinics.net/lists/netdev/msg257653.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19530] cross thread shutdown of UDP socket exhibits unexpected behavior
mpb added the comment: Connecting a UDP socket doesn't established a duplex connection like in TCP: Stream and duplex are orthogonal concepts. I still contend that connected UDP sockets are a duplex communication channel (under every definition of duplex I have read). The Linux connect manpage and the behavior of the Linux connect and shutdown system calls agree with me. (So does the OpenBSD shutdown manpage.) But we agree that this is not a Python issue (unless Python wants to improve its documentation to explicitly mention the benefits of cross thread shutdowns of TCP sockets). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19530] cross thread shutdown of UDP socket exhibits unexpected behavior
Charles-François Natali added the comment: After some research... Which is normal, since UDP sockets aren't connected. But UDP sockets can be connected! No, they can't. Connecting a UDP socket doesn't established a duplex connection like in TCP: it's just a shortand for not having to repeat the destination address upon every sendto()/sendmsg(). FYI, the FreeBSD (and OpenBSD) shutdown manpages anticipate calling shutdown on DGRAM sockets. And the Linux connect manpage discusses connecting DGRAM sockets. And since shutdown() is designed for duplex connection, it doesn't really make much sense. It might very well work when you passe SHUT_RD because it can be interpreted as triggering an EOF, but I wouldn't rely on this. Here is the updated Python code. I do expect to try to report this upstream. (Also, I now have C/pthreads code, if you want to see it. As expected, C behaves identically.) So you see it's not a Python bug. It's really not a bug at all, but if you want to report this upstream, have fun :-). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19530] cross thread shutdown of UDP socket exhibits unexpected behavior
mpb added the comment:
After some research...
Which is normal, since UDP sockets aren't connected.
But UDP sockets can be connected!
If I connect the UDP sockets, then shutdown succeeds (no exception is raised),
but recvfrom still appears to succeed, returning a zero length message with a
bogus address family, IP address and port. (Bogus even if I set them to zero
before the call!)
FYI, the FreeBSD (and OpenBSD) shutdown manpages anticipate calling shutdown on
DGRAM sockets. And the Linux connect manpage discusses connecting DGRAM
sockets.
Here is the updated Python code. I do expect to try to report this upstream.
(Also, I now have C/pthreads code, if you want to see it. As expected, C
behaves identically.)
import socket, threading, time
fd_0 = socket.socket (socket.AF_INET, socket.SOCK_DGRAM)
fd_0.bind(('localhost', 8000))
fd_0.connect (('localhost', 8001))
fd_1 = socket.socket (socket.AF_INET, socket.SOCK_DGRAM)
fd_1.bind(('localhost', 8001))
fd_1.connect (('localhost', 8000))
def thread_main ():
for i in range (3) :
# print ('recvfrom blocking ...')
recv, remote_addr = fd_0.recvfrom (1024)
print ('recvfrom %s %s' % (recv, remote_addr))
def main ():
fd_1.send (b'test')
fd_1.send (b'')
fd_0.shutdown (socket.SHUT_RD)
thread = threading.Thread ( target = thread_main )
thread.start ()
time.sleep (0.5)
main ()
thread.join ()
print ('exiting')
And the code outputs:
recvfrom b'test' ('127.0.0.1', 8001)
recvfrom b'' ('127.0.0.1', 8001)
recvfrom b'' (36100, b'\xe4\xc6\xf0^7\xe2\x85\xf8\x07\xc1\x04\x8d\xe4\xc6')
exiting
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19530
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19530] cross thread shutdown of UDP socket exhibits unexpected behavior
New submission from mpb:
I have a multi-threaded application.
A background thread is blocked, having called recvfrom on a UDP socket.
The main thread wants to cause the background thread to unblock.
With TCP sockets, I can achieve this by calling:
sock.shutdown (socket.SHUT_RD)
When I try this with a UDP socket, the thread calling shutdown raises an OS
Error (transport end point not connected).
The blocked thread does unblock (which is helpful), but recvform appears to
return successfully, returning a zero length byte string, and a bogus address!
(This is the opposite of the TCP case, where the blocked thread raises the
exception, and the call to shutdown succeeds.)
In contrast, sock.close does not cause the blocked thread to unblock. (This is
the same for both TCP and UDP sockets.)
I suspect Python is just exposing the underlying C behavior of shutdown and
recvfrom. I'd test it in C, but I'm not fluent in writing multi-threaded code
in C.
It would be nice if the recvfrom thread could raise some kind of exception,
rather than appearing to return successfully. It might also be worth reporting
this bug upstream (where ever upstream is for recvfrom). I'm running Python
3.3.1 on Linux.
See also this similar bug.
http://bugs.python.org/issue8831
The Python socket docs could mention that to unblock a reading thread, sockets
should be shutdown, not closed. This might be implied in the current docs, but
it could be made explicit. See:
http://docs.python.org/3/library/socket.html#socket.socket.close
For example, the following sentence could be appended to the Note at the above
link. Note: (...) Specifically, in multi-threaded programming, if a thread
is blocked performing a read or write on a socket, calling shutdown from
another thread will unblock the blocked thread. Unblocking via shutdown seems
to work with TCP sockets, but may result in strange behavior with UDP sockets.
Here is sample Python code that demonstrates the behavior.
import socket, threading, time
sock = socket.socket (socket.AF_INET, socket.SOCK_DGRAM)
sock.bind (('localhost', 8000))
def recvfrom ():
for i in range (2) :
print ('recvfrom blocking ...')
recv, remote_addr = sock.recvfrom (1024)
print ('recvfrom %s %s' % (recv, remote_addr))
thread = threading.Thread ( target = recvfrom )
thread.start ()
time.sleep (0.5)
sock2 = socket.socket (socket.AF_INET, socket.SOCK_DGRAM)
sock2.sendto (b'test', ('localhost', 8000))
time.sleep (0.5)
try: sock.shutdown (socket.SHUT_RD)
except OSError as exc : print ('shutdown os error %s' % str (exc))
sock.close ()
thread.join ()
print ('exiting')
And here is the output of the above code:
recvfrom blocking ...
recvfrom b'test' ('127.0.0.1', 48671)
recvfrom blocking ...
shutdown os error [Errno 107] Transport endpoint is not connected
recvfrom b'' (59308, b'\xaa\xe5\xec\xde3\xe6\x82\x02\x00\x00\xa8\xe7\xaa\xe5')
exiting
--
components: IO
messages: 202457
nosy: mpb
priority: normal
severity: normal
status: open
title: cross thread shutdown of UDP socket exhibits unexpected behavior
type: behavior
versions: Python 3.3
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19530
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19530] cross thread shutdown of UDP socket exhibits unexpected behavior
Charles-François Natali added the comment: When I try this with a UDP socket, the thread calling shutdown raises an OS Error (transport end point not connected). Which is normal, since UDP sockets aren't connected. In contrast, sock.close does not cause the blocked thread to unblock. (This is the same for both TCP and UDP sockets.) Which is normal, since you're not supposed to do this. I suspect Python is just exposing the underlying C behavior of shutdown and recvfrom. I'd test it in C, but I'm not fluent in writing multi-threaded code in C. You'd get exactly the same behavior. It would be nice if the recvfrom thread could raise some kind of exception, rather than appearing to return successfully. It might also be worth reporting this bug upstream (where ever upstream is for recvfrom). I'm running Python 3.3.1 on Linux. This isn't a bug: you're not using using the BSD socket API correctly. You can try reporting this bug upstream (i.e. to the kernel mailing list): it'll be an interesting experience :-) The Python socket docs could mention that to unblock a reading thread, sockets should be shutdown, not closed. This might be implied in the current docs, but it could be made explicit. See: If we start documenting any possible misuse of our exposed API, the documentation will get *really* large :-) Really, the problem is simply that you're not using the socket API as you should. Iy you want do unblock your thread doing a recvfrom(), you have several options: - send a datagram to the socket address from another thread - use a timeout on the socket, and periodically check a termination flag - use select()/poll() to multiplex between this socket and the read-end of a pipe: when you want to shutdown, simply write some data to the pipe: this will wake up select()/poll(), and you'll know your thread can exit Closing as invalid. -- nosy: +neologix resolution: - invalid stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
