[issue23285] PEP 475 - EINTR handling

[Berker Peksag]

Changes by Berker Peksag :


Removed file: http://bugs.python.org/file43674/b3b439d7dd40.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[mostafa shahdadi]

Changes by mostafa shahdadi :


Added file: http://bugs.python.org/file43674/b3b439d7dd40.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[STINNER Victor]

Changes by STINNER Victor :


--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[STINNER Victor]

STINNER Victor added the comment:

> For the record, it seems test_eintr sometimes left zombie processes in my 
> machine where I run reference leak tests every night. I didn't investigate 
> and just disabled the tests.
>
> (I'll add that that machine is hosted on an OpenVZ-based VPS, so perhaps 
> there are issues with the old patched kernel and whatnot?)

On my Fedora 21 (on a physical PC, not virtualized), I ran "./python -m test -R 
3:3: test_eintr" 3 times. After that, I didn't see any zombi Python process. If 
I cannot reproduce the issue, I cannot fix it. I bet that it's related to 
OpenVZ.

IMO this issue can be closed. It already has a long history. I prefer to open 
new issues. See the issue #23648 "PEP 475 meta issue" which lists all issues 
related to the PEP 475. I opened new issues for each module which didn't handle 
completly the PEP 475 yet.

By the way: great job Charles-François! I like your changeset, it works well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Antoine Pitrou]

Antoine Pitrou added the comment:

(I'll add that that machine is hosted on an OpenVZ-based VPS, so perhaps there 
are issues with the old patched kernel and whatnot?)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Antoine Pitrou]

Antoine Pitrou added the comment:

For the record, it seems test_eintr sometimes left zombie processes in my 
machine where I run reference leak tests every night. I didn't investigate and 
just disabled the tests.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[STINNER Victor]

STINNER Victor added the comment:

test_setitimer.patch: hack unittest.TestCase.run() to inject signals every 1 
ms, send the first signal in 100 ms.

This patch helped me to find that socket.connect() doesn't handle EINTR yet: 
see issue #23618 "PEP 475: handle EINTR in the socket module".

--
Added file: http://bugs.python.org/file38403/setitimer.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Roundup Robot]

Roundup Robot added the comment:

New changeset cad6eac598ec by Victor Stinner in branch 'default':
Issue #23285: Fix handling of EINTR in fileio.c
https://hg.python.org/cpython/rev/cad6eac598ec

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[STINNER Victor]

STINNER Victor added the comment:

> Would be nice to have a test for it;

There are already some tests on EINTR in test_file_eintr and test_subprocess 
(test_communicate_eintr()).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Charles-François Natali]

Charles-François Natali added the comment:

@Victor: please commit.
Would be nice to have a test for it;

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[STINNER Victor]

STINNER Victor added the comment:

Note: I found the bug while working on a patch for #22181.

My test is this shell script:

$ while true; do ./python -c 'import os, signal; 
signal.setitimer(signal.ITIMER_REAL, 0.001, 0.0001); 
signal.signal(signal.SIGALRM, lambda *args: print(".", end="")); 
print("urandom"); x=os.urandom(5000); print("ok"); signal.alarm(0)'; if [ $? 
-ne 142 -a $? -ne 0 ]; then break; fi done  

The test calls print() in a signal handler which can likely create a reentrant 
call, which is forbidden. But Python handles this case, it's fine.

After removing all debug prints and reverting the fix on fileio.c, the test 
doesn't crash with the assertion error anymore. Before, an assertion failed 
because fileio_write() returned Py_None with an exception set.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[STINNER Victor]

STINNER Victor added the comment:

The change on Modules/_io/fileio.c is wrong: functions may return None with an 
exception set. It is wrong because a function must return a result with no 
exception set, or NULL and an exception set.

Attached patch fixes this issue.

--
Added file: http://bugs.python.org/file38309/fileio.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Roundup Robot]

Roundup Robot added the comment:

New changeset 000bbdf0ea76 by Ned Deily in branch 'default':
Issue #23285: Install new test directory.
https://hg.python.org/cpython/rev/000bbdf0ea76

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Martin Panter]

Changes by Martin Panter :


--
nosy: +vadmium

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Roundup Robot]

Roundup Robot added the comment:

New changeset 5b63010be19e by Charles-François Natali in branch 'default':
Issue #23285: PEP 475 -- Retry system calls failing with EINTR.
https://hg.python.org/cpython/rev/5b63010be19e

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Charles-François Natali]

Charles-François Natali added the comment:

> Antoine Pitrou added the comment:
>
>> Would it be possible to push the latest patch right now
>
> It's ok for me. Please watch the buildbots :)

Cool, I'll push on Friday evening or Saturday.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Antoine Pitrou]

Antoine Pitrou added the comment:

> Would it be possible to push the latest patch right now

It's ok for me. Please watch the buildbots :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[STINNER Victor]

STINNER Victor added the comment:

Would it be possible to push the latest patch right now, and fix remaining 
issues (if there are known issues?), before Python 3.4 alpha 1? According to 
the PEP 478, the alpha 1 is scheduled for this sunday (February 8, 2015).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Antoine Pitrou]

Antoine Pitrou added the comment:

PEP is now updated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[STINNER Victor]

STINNER Victor added the comment:

Charles-François Natali added the comment:
> I just realized I didn't retry upon EINTR for open(): eintr-4.diff
> adds this, along with tests (using a fifo).
>
> Also, I added comments explaining why we don't retry upon close() (see
> e.g. http://lwn.net/Articles/576478/ and
> http://linux.derkeiler.com/Mailing-Lists/Kernel/2005-09/3000.html).

I didn't read these articles yet, but I will. IMO the PEP must be
updated if we don't retry in close(), since close() is used as an
example where "EINTR is unexpected"!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Charles-François Natali]

Charles-François Natali added the comment:

I just realized I didn't retry upon EINTR for open(): eintr-4.diff
adds this, along with tests (using a fifo).

Also, I added comments explaining why we don't retry upon close() (see
e.g. http://lwn.net/Articles/576478/ and
http://linux.derkeiler.com/Mailing-Lists/Kernel/2005-09/3000.html).

--
Added file: http://bugs.python.org/file37935/eintr-4.diff

___
Python tracker 

___diff -r 7e6340e67618 Lib/_pyio.py
--- a/Lib/_pyio.py  Fri Jan 30 00:16:31 2015 +0100
+++ b/Lib/_pyio.py  Fri Jan 30 22:08:27 2015 +
@@ -1006,10 +1006,7 @@
 current_size = 0
 while True:
 # Read until EOF or until read() would block.
-try:
-chunk = self.raw.read()
-except InterruptedError:
-continue
+chunk = self.raw.read()
 if chunk in empty_values:
 nodata_val = chunk
 break
@@ -1028,10 +1025,7 @@
 chunks = [buf[pos:]]
 wanted = max(self.buffer_size, n)
 while avail < n:
-try:
-chunk = self.raw.read(wanted)
-except InterruptedError:
-continue
+chunk = self.raw.read(wanted)
 if chunk in empty_values:
 nodata_val = chunk
 break
@@ -1060,12 +1054,7 @@
 have = len(self._read_buf) - self._read_pos
 if have < want or have <= 0:
 to_read = self.buffer_size - have
-while True:
-try:
-current = self.raw.read(to_read)
-except InterruptedError:
-continue
-break
+current = self.raw.read(to_read)
 if current:
 self._read_buf = self._read_buf[self._read_pos:] + current
 self._read_pos = 0
@@ -1214,8 +1203,6 @@
 while self._write_buf:
 try:
 n = self.raw.write(self._write_buf)
-except InterruptedError:
-continue
 except BlockingIOError:
 raise RuntimeError("self.raw should implement RawIOBase: it "
"should not raise BlockingIOError")
diff -r 7e6340e67618 Lib/distutils/spawn.py
--- a/Lib/distutils/spawn.pyFri Jan 30 00:16:31 2015 +0100
+++ b/Lib/distutils/spawn.pyFri Jan 30 22:08:27 2015 +
@@ -137,9 +137,6 @@
 try:
 pid, status = os.waitpid(pid, 0)
 except OSError as exc:
-import errno
-if exc.errno == errno.EINTR:
-continue
 if not DEBUG:
 cmd = executable
 raise DistutilsExecError(
diff -r 7e6340e67618 Lib/multiprocessing/connection.py
--- a/Lib/multiprocessing/connection.py Fri Jan 30 00:16:31 2015 +0100
+++ b/Lib/multiprocessing/connection.py Fri Jan 30 22:08:27 2015 +
@@ -365,10 +365,7 @@
 def _send(self, buf, write=_write):
 remaining = len(buf)
 while True:
-try:
-n = write(self._handle, buf)
-except InterruptedError:
-continue
+n = write(self._handle, buf)
 remaining -= n
 if remaining == 0:
 break
@@ -379,10 +376,7 @@
 handle = self._handle
 remaining = size
 while remaining > 0:
-try:
-chunk = read(handle, remaining)
-except InterruptedError:
-continue
+chunk = read(handle, remaining)
 n = len(chunk)
 if n == 0:
 if remaining == size:
@@ -595,13 +589,7 @@
 self._unlink = None
 
 def accept(self):
-while True:
-try:
-s, self._last_accepted = self._socket.accept()
-except InterruptedError:
-pass
-else:
-break
+s, self._last_accepted = self._socket.accept()
 s.setblocking(True)
 return Connection(s.detach())
 
diff -r 7e6340e67618 Lib/multiprocessing/forkserver.py
--- a/Lib/multiprocessing/forkserver.py Fri Jan 30 00:16:31 2015 +0100
+++ b/Lib/multiprocessing/forkserver.py Fri Jan 30 22:08:27 2015 +
@@ -188,8 +188,6 @@
 finally:
 os._exit(code)
 
-except InterruptedError:
-pass
 except OSError as e:
 if e.errno != errno.ECONNABORTED:
 raise
@@ -230,13 +228,7 @@
 data = b''
 length = UNSIGNED_STRUCT.size
 while len(data) < length:
-while True:
-try:
-s = os.read(fd, length - len(data))
-except InterruptedError:
- 

[issue23285] PEP 475 - EINTR handling

[Charles-François Natali]

Charles-François Natali added the comment:

> With eintr-2.diff, fast!:

Victory \°/.

> Instrumented test_send, 3 socket.send calls, many socket.recv_into calls:

Yep, that's expected.
I think we should keep the default socket buffer size: it increases
the test coverage, and it's probably not worth trying to increase it
to make the test a bit faster, especially since it spends so much time
sleeping.

Antoine, I'm now happy with the patch, so we'll be waiting for your
decision on the PEP with Victor :-).

--
Added file: http://bugs.python.org/file37910/eintr-3.diff

___
Python tracker 

___diff -r fe0fddd6fd21 Lib/_pyio.py
--- a/Lib/_pyio.py  Sun Jan 18 11:17:39 2015 +0200
+++ b/Lib/_pyio.py  Thu Jan 29 22:20:01 2015 +
@@ -1006,10 +1006,7 @@
 current_size = 0
 while True:
 # Read until EOF or until read() would block.
-try:
-chunk = self.raw.read()
-except InterruptedError:
-continue
+chunk = self.raw.read()
 if chunk in empty_values:
 nodata_val = chunk
 break
@@ -1028,10 +1025,7 @@
 chunks = [buf[pos:]]
 wanted = max(self.buffer_size, n)
 while avail < n:
-try:
-chunk = self.raw.read(wanted)
-except InterruptedError:
-continue
+chunk = self.raw.read(wanted)
 if chunk in empty_values:
 nodata_val = chunk
 break
@@ -1060,12 +1054,7 @@
 have = len(self._read_buf) - self._read_pos
 if have < want or have <= 0:
 to_read = self.buffer_size - have
-while True:
-try:
-current = self.raw.read(to_read)
-except InterruptedError:
-continue
-break
+current = self.raw.read(to_read)
 if current:
 self._read_buf = self._read_buf[self._read_pos:] + current
 self._read_pos = 0
@@ -1214,8 +1203,6 @@
 while self._write_buf:
 try:
 n = self.raw.write(self._write_buf)
-except InterruptedError:
-continue
 except BlockingIOError:
 raise RuntimeError("self.raw should implement RawIOBase: it "
"should not raise BlockingIOError")
diff -r fe0fddd6fd21 Lib/distutils/spawn.py
--- a/Lib/distutils/spawn.pySun Jan 18 11:17:39 2015 +0200
+++ b/Lib/distutils/spawn.pyThu Jan 29 22:20:01 2015 +
@@ -137,9 +137,6 @@
 try:
 pid, status = os.waitpid(pid, 0)
 except OSError as exc:
-import errno
-if exc.errno == errno.EINTR:
-continue
 if not DEBUG:
 cmd = executable
 raise DistutilsExecError(
diff -r fe0fddd6fd21 Lib/multiprocessing/connection.py
--- a/Lib/multiprocessing/connection.py Sun Jan 18 11:17:39 2015 +0200
+++ b/Lib/multiprocessing/connection.py Thu Jan 29 22:20:01 2015 +
@@ -365,10 +365,7 @@
 def _send(self, buf, write=_write):
 remaining = len(buf)
 while True:
-try:
-n = write(self._handle, buf)
-except InterruptedError:
-continue
+n = write(self._handle, buf)
 remaining -= n
 if remaining == 0:
 break
@@ -379,10 +376,7 @@
 handle = self._handle
 remaining = size
 while remaining > 0:
-try:
-chunk = read(handle, remaining)
-except InterruptedError:
-continue
+chunk = read(handle, remaining)
 n = len(chunk)
 if n == 0:
 if remaining == size:
@@ -595,13 +589,7 @@
 self._unlink = None
 
 def accept(self):
-while True:
-try:
-s, self._last_accepted = self._socket.accept()
-except InterruptedError:
-pass
-else:
-break
+s, self._last_accepted = self._socket.accept()
 s.setblocking(True)
 return Connection(s.detach())
 
diff -r fe0fddd6fd21 Lib/multiprocessing/forkserver.py
--- a/Lib/multiprocessing/forkserver.py Sun Jan 18 11:17:39 2015 +0200
+++ b/Lib/multiprocessing/forkserver.py Thu Jan 29 22:20:01 2015 +
@@ -188,8 +188,6 @@
 finally:
 os._exit(code)
 
-except InterruptedError:
-pass
 except OSError as e:
 if e.errno != errno.ECONNABORTED:
 raise
@@ -230,13 +228,7 @@
 data = b''
 length = UNSIGNED_STRUCT.size
 

[issue23285] PEP 475 - EINTR handling

[Antoine Pitrou]

Antoine Pitrou added the comment:

Victor, do you think there's anything left to do?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
stage:  -> patch review
versions: +Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Ned Deily]

Ned Deily added the comment:

With eintr-2.diff, fast!:

$ time ./python 
~/Projects/PyDev/active/dev/3x/source/Lib/test/eintrdata/eintr_tester.py
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... ok
test_sendall (__main__.SocketEINTRTest) ... ok
test_sendmsg (__main__.SocketEINTRTest) ... ok

--
Ran 12 tests in 7.652s

OK

real0m7.959s
user0m2.573s
sys 0m1.604s

Instrumented test_send, 3 socket.send calls, many socket.recv_into calls:

test_send (__main__.SocketEINTRTest) ... rd SO_RCVBUF default was 8192, wr 
SO_SNDBUF default was 8192
len(data) = 16777215
 
 sent = 8192, total written = 8192
 sent = 4440064, total written = 4448256
 sent = 12328959, total written = 16777215
 
 received = 8192, total read = 8192
 received = 8192, total read = 16384
 received = 8192, total read = 24576
 [...]
 received = 8192, total read = 16760832
 received = 8192, total read = 16769024
 received = 8191, total read = 16777215

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Charles-François Natali]

Charles-François Natali added the comment:

> eintr-1.diff doesn't seem to make any significant difference from eintr.diff 
> on my system.  It's still pegging a CPU at 100% and takes 7 minutes wall time 
> to complete.

Alright, enough played: the patch attached uses a memoryview and
socket.recv_into() to remove all memory allocations: if this doesn't
fix your problem, I don't know what's going on...

--
Added file: http://bugs.python.org/file37909/eintr-2.diff

___
Python tracker 

___diff -r fe0fddd6fd21 Lib/_pyio.py
--- a/Lib/_pyio.py  Sun Jan 18 11:17:39 2015 +0200
+++ b/Lib/_pyio.py  Thu Jan 29 19:53:06 2015 +
@@ -1006,10 +1006,7 @@
 current_size = 0
 while True:
 # Read until EOF or until read() would block.
-try:
-chunk = self.raw.read()
-except InterruptedError:
-continue
+chunk = self.raw.read()
 if chunk in empty_values:
 nodata_val = chunk
 break
@@ -1028,10 +1025,7 @@
 chunks = [buf[pos:]]
 wanted = max(self.buffer_size, n)
 while avail < n:
-try:
-chunk = self.raw.read(wanted)
-except InterruptedError:
-continue
+chunk = self.raw.read(wanted)
 if chunk in empty_values:
 nodata_val = chunk
 break
@@ -1060,12 +1054,7 @@
 have = len(self._read_buf) - self._read_pos
 if have < want or have <= 0:
 to_read = self.buffer_size - have
-while True:
-try:
-current = self.raw.read(to_read)
-except InterruptedError:
-continue
-break
+current = self.raw.read(to_read)
 if current:
 self._read_buf = self._read_buf[self._read_pos:] + current
 self._read_pos = 0
@@ -1214,8 +1203,6 @@
 while self._write_buf:
 try:
 n = self.raw.write(self._write_buf)
-except InterruptedError:
-continue
 except BlockingIOError:
 raise RuntimeError("self.raw should implement RawIOBase: it "
"should not raise BlockingIOError")
diff -r fe0fddd6fd21 Lib/distutils/spawn.py
--- a/Lib/distutils/spawn.pySun Jan 18 11:17:39 2015 +0200
+++ b/Lib/distutils/spawn.pyThu Jan 29 19:53:06 2015 +
@@ -137,9 +137,6 @@
 try:
 pid, status = os.waitpid(pid, 0)
 except OSError as exc:
-import errno
-if exc.errno == errno.EINTR:
-continue
 if not DEBUG:
 cmd = executable
 raise DistutilsExecError(
diff -r fe0fddd6fd21 Lib/multiprocessing/connection.py
--- a/Lib/multiprocessing/connection.py Sun Jan 18 11:17:39 2015 +0200
+++ b/Lib/multiprocessing/connection.py Thu Jan 29 19:53:06 2015 +
@@ -365,10 +365,7 @@
 def _send(self, buf, write=_write):
 remaining = len(buf)
 while True:
-try:
-n = write(self._handle, buf)
-except InterruptedError:
-continue
+n = write(self._handle, buf)
 remaining -= n
 if remaining == 0:
 break
@@ -379,10 +376,7 @@
 handle = self._handle
 remaining = size
 while remaining > 0:
-try:
-chunk = read(handle, remaining)
-except InterruptedError:
-continue
+chunk = read(handle, remaining)
 n = len(chunk)
 if n == 0:
 if remaining == size:
@@ -595,13 +589,7 @@
 self._unlink = None
 
 def accept(self):
-while True:
-try:
-s, self._last_accepted = self._socket.accept()
-except InterruptedError:
-pass
-else:
-break
+s, self._last_accepted = self._socket.accept()
 s.setblocking(True)
 return Connection(s.detach())
 
diff -r fe0fddd6fd21 Lib/multiprocessing/forkserver.py
--- a/Lib/multiprocessing/forkserver.py Sun Jan 18 11:17:39 2015 +0200
+++ b/Lib/multiprocessing/forkserver.py Thu Jan 29 19:53:06 2015 +
@@ -188,8 +188,6 @@
 finally:
 os._exit(code)
 
-except InterruptedError:
-pass
 except OSError as e:
 if e.errno != errno.ECONNABORTED:
 raise
@@ -230,13 +228,7 @@
 data = b''
 length = UNSIGNED_STRUCT.size
 while len(data) < length:
-while True:
-try:
-s = os.read(fd, len

[issue23285] PEP 475 - EINTR handling

[Charles-François Natali]

Charles-François Natali added the comment:

OK, it turns out the culprit was repeated calls to BytesIO.getvalue(),
which forced large allocations upon reception of every message.
The patch attached fixes this (without changing the socket buffer size).

--
Added file: http://bugs.python.org/file37905/eintr-1.diff

___
Python tracker 

___diff -r fe0fddd6fd21 Lib/_pyio.py
--- a/Lib/_pyio.py  Sun Jan 18 11:17:39 2015 +0200
+++ b/Lib/_pyio.py  Thu Jan 29 07:59:47 2015 +
@@ -1006,10 +1006,7 @@
 current_size = 0
 while True:
 # Read until EOF or until read() would block.
-try:
-chunk = self.raw.read()
-except InterruptedError:
-continue
+chunk = self.raw.read()
 if chunk in empty_values:
 nodata_val = chunk
 break
@@ -1028,10 +1025,7 @@
 chunks = [buf[pos:]]
 wanted = max(self.buffer_size, n)
 while avail < n:
-try:
-chunk = self.raw.read(wanted)
-except InterruptedError:
-continue
+chunk = self.raw.read(wanted)
 if chunk in empty_values:
 nodata_val = chunk
 break
@@ -1060,12 +1054,7 @@
 have = len(self._read_buf) - self._read_pos
 if have < want or have <= 0:
 to_read = self.buffer_size - have
-while True:
-try:
-current = self.raw.read(to_read)
-except InterruptedError:
-continue
-break
+current = self.raw.read(to_read)
 if current:
 self._read_buf = self._read_buf[self._read_pos:] + current
 self._read_pos = 0
@@ -1214,8 +1203,6 @@
 while self._write_buf:
 try:
 n = self.raw.write(self._write_buf)
-except InterruptedError:
-continue
 except BlockingIOError:
 raise RuntimeError("self.raw should implement RawIOBase: it "
"should not raise BlockingIOError")
diff -r fe0fddd6fd21 Lib/distutils/spawn.py
--- a/Lib/distutils/spawn.pySun Jan 18 11:17:39 2015 +0200
+++ b/Lib/distutils/spawn.pyThu Jan 29 07:59:47 2015 +
@@ -137,9 +137,6 @@
 try:
 pid, status = os.waitpid(pid, 0)
 except OSError as exc:
-import errno
-if exc.errno == errno.EINTR:
-continue
 if not DEBUG:
 cmd = executable
 raise DistutilsExecError(
diff -r fe0fddd6fd21 Lib/multiprocessing/connection.py
--- a/Lib/multiprocessing/connection.py Sun Jan 18 11:17:39 2015 +0200
+++ b/Lib/multiprocessing/connection.py Thu Jan 29 07:59:47 2015 +
@@ -365,10 +365,7 @@
 def _send(self, buf, write=_write):
 remaining = len(buf)
 while True:
-try:
-n = write(self._handle, buf)
-except InterruptedError:
-continue
+n = write(self._handle, buf)
 remaining -= n
 if remaining == 0:
 break
@@ -379,10 +376,7 @@
 handle = self._handle
 remaining = size
 while remaining > 0:
-try:
-chunk = read(handle, remaining)
-except InterruptedError:
-continue
+chunk = read(handle, remaining)
 n = len(chunk)
 if n == 0:
 if remaining == size:
@@ -595,13 +589,7 @@
 self._unlink = None
 
 def accept(self):
-while True:
-try:
-s, self._last_accepted = self._socket.accept()
-except InterruptedError:
-pass
-else:
-break
+s, self._last_accepted = self._socket.accept()
 s.setblocking(True)
 return Connection(s.detach())
 
diff -r fe0fddd6fd21 Lib/multiprocessing/forkserver.py
--- a/Lib/multiprocessing/forkserver.py Sun Jan 18 11:17:39 2015 +0200
+++ b/Lib/multiprocessing/forkserver.py Thu Jan 29 07:59:47 2015 +
@@ -188,8 +188,6 @@
 finally:
 os._exit(code)
 
-except InterruptedError:
-pass
 except OSError as e:
 if e.errno != errno.ECONNABORTED:
 raise
@@ -230,13 +228,7 @@
 data = b''
 length = UNSIGNED_STRUCT.size
 while len(data) < length:
-while True:
-try:
-s = os.read(fd, length - len(data))
-except InterruptedError:
-pass
-else:
-break
+s = os.read(fd, length - 

[issue23285] PEP 475 - EINTR handling

[Ned Deily]

Ned Deily added the comment:

eintr-1.diff doesn't seem to make any significant difference from eintr.diff on 
my system.  It's still pegging a CPU at 100% and takes 7 minutes wall time to 
complete.

$ time ./python 
~/Projects/PyDev/active/dev/3x/source/Lib/test/eintrdata/eintr_tester.py
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... ok
test_sendall (__main__.SocketEINTRTest) ... ok
test_sendmsg (__main__.SocketEINTRTest) ... ok

--
Ran 12 tests in 439.966s

OK

real7m20.276s
user5m7.575s
sys 2m9.846s

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Charles-François Natali]

Charles-François Natali added the comment:

> Charles-François Natali added the comment:
>
> Hmmm...
> Basically, with a much smaller socket buffer, we get much more context
> switches, which increases drastically the test runtime.
> But I must admit I'm still really surprised by the time it takes on
> OS-X: with a SOCK_MAX_SIZE of 16MB and a socket buffer size of 8kb,
> that's 2000 calls to send with context switches between the sender and
> receiver - and thie overhead should be much less on a two core
> machine.

OK, actually the receiver is completely CPU-bound, because of memory
allocation for the socket.recv() buffer I'll play with recv_into() and
profile a bit more.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Charles-François Natali]

Charles-François Natali added the comment:

> It turns out the times are not important; the hangup is the default size of 
> the socket buffers on OS X and possibly BSD in general.  In my case, the send 
> and receive buffers are 8192, which explains why the chunks written are so 
> small.

Hmmm...
Basically, with a much smaller socket buffer, we get much more context
switches, which increases drastically the test runtime.
But I must admit I'm still really surprised by the time it takes on
OS-X: with a SOCK_MAX_SIZE of 16MB and a socket buffer size of 8kb,
that's 2000 calls to send with context switches between the sender and
receiver - and thie overhead should be much less on a two core
machine.

I'll try to increase the socket buffer size and see what the buildbots
do: most of them will probably cap it at around 100k, but that'll
hopefully be enough.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Ned Deily]

Ned Deily added the comment:

It turns out the times are not important; the hangup is the default size of the 
socket buffers on OS X and possibly BSD in general.  In my case, the send and 
receive buffers are 8192, which explains why the chunks written are so small.  
I somewhat arbitrarily changed the sizes of the buffers in _test_send with:

rd.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, support.SOCK_MAX_SIZE // 3)
wr.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, support.SOCK_MAX_SIZE // 3)

The results were:

test_send (__main__.SocketEINTRTest) ... rd SO_RCVBUF default was 8192, wr 
SO_SNDBUF default was 8192
len(data) = 50331651
 sent = 5592405, written = 5592405
 chunk = 5592405, total read = 5592405
 chunk = 5592405, total read = 11184810
 chunk = 5592405, total read = 16777215
 sent = 16777215, written = 22369620
 chunk = 5592405, total read = 22369620
 chunk = 5592405, total read = 27962025
 chunk = 5592405, total read = 33554430
 chunk = 5592405, total read = 39146835
 sent = 22369620, written = 44739240
 chunk = 5592405, total read = 44739240
 sent = 5592411, written = 50331651
 chunk = 5592405, total read = 50331645
 chunk = 6, total read = 50331651
ok
test_sendall (__main__.SocketEINTRTest) ... rd SO_RCVBUF default was 8192, wr 
SO_SNDBUF default was 8192
len(data) = 50331651
 chunk = 5592405, total read = 5592405
 chunk = 5592405, total read = 11184810
 chunk = 5592405, total read = 16777215
 chunk = 5592405, total read = 22369620
 chunk = 5592405, total read = 27962025
 chunk = 5592405, total read = 33554430
 chunk = 5592405, total read = 39146835
 chunk = 5592405, total read = 44739240
 sent = None, written = 50331651
 chunk = 5592405, total read = 50331645
 chunk = 6, total read = 50331651
ok
test_sendmsg (__main__.SocketEINTRTest) ... rd SO_RCVBUF default was 8192, wr 
SO_SNDBUF default was 8192
len(data) = 50331651
 sent = 5592405, written = 5592405
 chunk = 5592405, total read = 5592405
 chunk = 5592405, total read = 11184810
 chunk = 5592405, total read = 16777215
 chunk = 5592405, total read = 22369620
 chunk = 5592405, total read = 27962025
 sent = 27962025, written = 33554430
 chunk = 5592405, total read = 33554430
 chunk = 5592405, total read = 39146835
 chunk = 5592405, total read = 44739240
 sent = 16777221, written = 50331651
 chunk = 5592405, total read = 50331645
 chunk = 6, total read = 50331651
ok

I dunno if a value that large will work in all environments, so the code to 
call setsockopt might need to be smarter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Charles-François Natali]

Charles-François Natali added the comment:

> I added a few prints to the send and receive loops of _test_send.  When 
> running on a reasonably current Debian testing Linux:

Thanks, that's what I was suspecting, but I really don't understand
why 200ms isn't enough for a socket write to actually do something:
maybe OS-X and *BSD schedulers are large timeslice...

Could you try by increasing signal_period to e.g. 0.5, and sleep_time to 1?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Ned Deily]

Ned Deily added the comment:

I added a few prints to the send and receive loops of _test_send.  When running 
on a reasonably current Debian testing Linux:

./python Lib/test/eintrdata/eintr_tester.py
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... len(data) = 50331651
 sent = 183360, written = 183360
 sent = 50148291, written = 50331651
 chunk = 50331651, total read = 50331651
ok
test_sendall (__main__.SocketEINTRTest) ... len(data) = 50331651
 chunk = 49323840, total read = 49323840
 sent = None, written = 50331651
 chunk = 1007811, total read = 50331651
ok
test_sendmsg (__main__.SocketEINTRTest) ... len(data) = 50331651
 sent = 183360, written = 183360
 sent = 50148291, written = 50331651
 chunk = 50331651, total read = 50331651
ok

--
Ran 12 tests in 10.139s

OK

When run on OS X (10.10.1):

test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... len(data) = 50331651
 sent = 8192, written = 8192
 chunk = 8192, total read = 8192
 chunk = 8192, total read = 16384
 sent = 16384, written = 24576
 chunk = 8192, total read = 24576
 chunk = 8192, total read = 32768
 chunk = 8192, total read = 40960
 chunk = 8192, total read = 49152
 sent = 32768, written = 57344
 chunk = 8192, total read = 57344
 chunk = 8192, total read = 65536
 chunk = 8192, total read = 73728
 chunk = 8192, total read = 81920
 chunk = 8192, total read = 90112
 sent = 40960, written = 98304
 chunk = 8192, total read = 98304
 chunk = 8192, total read = 106496
 chunk = 8192, total read = 114688
 chunk = 8192, total read = 122880
 sent = 32768, written = 131072
 chunk = 8192, total read = 131072
 chunk = 8192, total read = 139264
 chunk = 8192, total read = 147456
 chunk = 8192, total read = 155648
 chunk = 8192, total read = 163840
 sent = 40960, written = 172032
 chunk = 8192, total read = 172032
 chunk = 8192, total read = 180224
 chunk = 8192, total read = 188416
 chunk = 8192, total read = 196608
 sent = 32768, written = 204800
 chunk = 8192, total read = 204800
 chunk = 8192, total read = 212992
 chunk = 8192, total read = 221184
 chunk = 8192, total read = 229376
 chunk = 8192, total read = 237568
 sent = 40960, written = 245760
 chunk = 8192, total read = 245760
 chunk = 8192, total read = 253952
 chunk = 8192, total read = 262144
 chunk = 8192, total read = 270336
 sent = 32768, written = 278528
 chunk = 8192, total read = 278528
 chunk = 8192, total read = 286720
 chunk = 8192, total read = 294912
 chunk = 8192, total read = 303104
 chunk = 8192, total read = 311296
 sent = 40960, written = 319488
[... and so on]

When run standalone, the tests do eventually finish without error but take a 
*long* time to do so.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Charles-François Natali]

Charles-François Natali added the comment:

> The review diff is weird: it seems it contains changes that aren't 
> EINTR-related (see e.g. argparse.rst).

Here's a manually generated diff.

--
Added file: http://bugs.python.org/file37802/eintr.diff

___
Python tracker 

___diff -r fe0fddd6fd21 Lib/_pyio.py
--- a/Lib/_pyio.py  Sun Jan 18 11:17:39 2015 +0200
+++ b/Lib/_pyio.py  Wed Jan 21 05:55:58 2015 +
@@ -1006,10 +1006,7 @@
 current_size = 0
 while True:
 # Read until EOF or until read() would block.
-try:
-chunk = self.raw.read()
-except InterruptedError:
-continue
+chunk = self.raw.read()
 if chunk in empty_values:
 nodata_val = chunk
 break
@@ -1028,10 +1025,7 @@
 chunks = [buf[pos:]]
 wanted = max(self.buffer_size, n)
 while avail < n:
-try:
-chunk = self.raw.read(wanted)
-except InterruptedError:
-continue
+chunk = self.raw.read(wanted)
 if chunk in empty_values:
 nodata_val = chunk
 break
@@ -1060,12 +1054,7 @@
 have = len(self._read_buf) - self._read_pos
 if have < want or have <= 0:
 to_read = self.buffer_size - have
-while True:
-try:
-current = self.raw.read(to_read)
-except InterruptedError:
-continue
-break
+current = self.raw.read(to_read)
 if current:
 self._read_buf = self._read_buf[self._read_pos:] + current
 self._read_pos = 0
@@ -1214,8 +1203,6 @@
 while self._write_buf:
 try:
 n = self.raw.write(self._write_buf)
-except InterruptedError:
-continue
 except BlockingIOError:
 raise RuntimeError("self.raw should implement RawIOBase: it "
"should not raise BlockingIOError")
diff -r fe0fddd6fd21 Lib/distutils/spawn.py
--- a/Lib/distutils/spawn.pySun Jan 18 11:17:39 2015 +0200
+++ b/Lib/distutils/spawn.pyWed Jan 21 05:55:58 2015 +
@@ -137,9 +137,6 @@
 try:
 pid, status = os.waitpid(pid, 0)
 except OSError as exc:
-import errno
-if exc.errno == errno.EINTR:
-continue
 if not DEBUG:
 cmd = executable
 raise DistutilsExecError(
diff -r fe0fddd6fd21 Lib/multiprocessing/connection.py
--- a/Lib/multiprocessing/connection.py Sun Jan 18 11:17:39 2015 +0200
+++ b/Lib/multiprocessing/connection.py Wed Jan 21 05:55:58 2015 +
@@ -365,10 +365,7 @@
 def _send(self, buf, write=_write):
 remaining = len(buf)
 while True:
-try:
-n = write(self._handle, buf)
-except InterruptedError:
-continue
+n = write(self._handle, buf)
 remaining -= n
 if remaining == 0:
 break
@@ -379,10 +376,7 @@
 handle = self._handle
 remaining = size
 while remaining > 0:
-try:
-chunk = read(handle, remaining)
-except InterruptedError:
-continue
+chunk = read(handle, remaining)
 n = len(chunk)
 if n == 0:
 if remaining == size:
@@ -595,13 +589,7 @@
 self._unlink = None
 
 def accept(self):
-while True:
-try:
-s, self._last_accepted = self._socket.accept()
-except InterruptedError:
-pass
-else:
-break
+s, self._last_accepted = self._socket.accept()
 s.setblocking(True)
 return Connection(s.detach())
 
diff -r fe0fddd6fd21 Lib/multiprocessing/forkserver.py
--- a/Lib/multiprocessing/forkserver.py Sun Jan 18 11:17:39 2015 +0200
+++ b/Lib/multiprocessing/forkserver.py Wed Jan 21 05:55:58 2015 +
@@ -188,8 +188,6 @@
 finally:
 os._exit(code)
 
-except InterruptedError:
-pass
 except OSError as e:
 if e.errno != errno.ECONNABORTED:
 raise
@@ -230,13 +228,7 @@
 data = b''
 length = UNSIGNED_STRUCT.size
 while len(data) < length:
-while True:
-try:
-s = os.read(fd, length - len(data))
-except InterruptedError:
-pass
-else:
-break
+s = os.read(fd, length - len(data))
 if not s:
 raise EOFError('unexp

[issue23285] PEP 475 - EINTR handling

[Ned Deily]

Ned Deily added the comment:

(It may be several days before I can spend much time on it but I will take a 
look.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Antoine Pitrou]

Antoine Pitrou added the comment:

The review diff is weird: it seems it contains changes that aren't 
EINTR-related (see e.g. argparse.rst).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Antoine Pitrou]

Antoine Pitrou added the comment:

Perhaps Ned can help on the OS X side of things.

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23285] PEP 475 - EINTR handling

[Charles-François Natali]

Changes by Charles-François Natali :


--
title: PEP 475 - EINTR hanndling -> PEP 475 - EINTR handling

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com