[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[STINNER Victor]

STINNER Victor  added the comment:

> In any case it looks like self-pipe sock's buffer was overflown because 
> call_soon_threadsafe was called too many times, and loop._read_from_self 
> couldn't empty the buffer promptly.  Then, at some point, _write_to_self 
> failed with an IOError.

I fixed the issue. Thanks for your bug report ;-)

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[miss-islington]

miss-islington  added the comment:


New changeset c9f26714d511a338ba2fdd926e3dc62636f31815 by Miss Islington (bot) 
in branch '3.7':
bpo-23846: Fix ProactorEventLoop._write_to_self() (GH-11566)
https://github.com/python/cpython/commit/c9f26714d511a338ba2fdd926e3dc62636f31815


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[miss-islington]

Change by miss-islington :


--
pull_requests: +11230, 11231, 11232

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[STINNER Victor]

STINNER Victor  added the comment:


New changeset c9f872b0bdce5888f1879fa74e098bf4a05430c5 by Victor Stinner in 
branch 'master':
bpo-23846: Fix ProactorEventLoop._write_to_self() (GH-11566)
https://github.com/python/cpython/commit/c9f872b0bdce5888f1879fa74e098bf4a05430c5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[miss-islington]

Change by miss-islington :


--
pull_requests: +11230, 11231

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[miss-islington]

Change by miss-islington :


--
pull_requests: +11230

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch, patch, patch
pull_requests: +11227, 11228, 11229
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +11227
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch, patch
pull_requests: +11227, 11228
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[Yury Selivanov]

Yury Selivanov added the comment:

> Why are you using 2 threads?

That's a good question.

In any case it looks like self-pipe sock's buffer was overflown because 
call_soon_threadsafe was called too many times, and loop._read_from_self 
couldn't empty the buffer promptly.  Then, at some point, _write_to_self failed 
with an IOError.

It looks like this was fixed for the selector loop, but not for proactor:

selector_events.py:

def _write_to_self(self):
csock = self._csock
if csock is not None:
try:
csock.send(b'\0')
except OSError:
if self._debug:
logger.debug("Fail to write a null byte into the "
 "self-pipe socket",
 exc_info=True)

proactor_events.py:

def _write_to_self(self):
self._csock.send(b'\0')

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[Guido van Rossum]

Guido van Rossum added the comment:

Why are you using 2 threads?

--Guido (mobile)
On Dec 22, 2015 02:03, "Joseph Gordon"  wrote:

>
> Changes by Joseph Gordon :
>
>
> --
> nosy: +josephgordon
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[Joseph Gordon]

Changes by Joseph Gordon :


--
nosy: +josephgordon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[SeungHyun.Hwang]

New submission from SeungHyun.Hwang:

I tested on python3.4.2, windows 7

1. Only proactorEvent issued.
2. ThreadPoolExecutor has many workers (in the attached example file, worker 
count 20,000)
3. The loop does run_in_executor more call than worker count (in the attached 
example file, 40,000 calls)
4. After some random seconds, raise BlockingIOError

BlockingIOError: [WinError 10035] A non-blocking socket operation could not be 
completed immediately.
exception calling callback for Future at 0x1ab89ef0 state=finished returned 
NoneType
Traceback (most recent call last):
  File c:\Python342\Lib\concurrent\futures\_base.py, line 297, in 
_invoke_callbacks
callback(self)
  File c:\Python342\Lib\asyncio\futures.py, line 410, in lambda
new_future._copy_state, fut))
  File c:\Python342\Lib\asyncio\base_events.py, line 403, in 
call_soon_threadsafe
self._write_to_self()
  File c:\Python342\Lib\asyncio\proactor_events.py, line 449, in 
_write_to_self
self._csock.send(b'\0')


I guess that proactor's _write_to_self method misses exception handle.

proactor_events.py
def _write_to_self(self):
self._csock.send(b'\0')


selector_events.py
def _write_to_self(self):
# This may be called from a different thread, possibly after
# _close_self_pipe() has been called or even while it is
# running.  Guard for self._csock being None or closed.  When
# a socket is closed, send() raises OSError (with errno set to
# EBADF, but let's not rely on the exact error code).
csock = self._csock
if csock is not None:
try:
csock.send(b'\0')
except OSError:
if self._debug:
logger.debug(Fail to write a null byte into the 
 self-pipe socket,
 exc_info=True)


Ps: It's my first publication. Hope you understand my poor comment..

--
components: asyncio
files: example_thread_executor.py
messages: 239872
nosy: gvanrossum, haypo, kernel0, yselivanov
priority: normal
severity: normal
status: open
title: asyncio : ProactorEventLoop raised BlockingIOError when 
ThreadPoolExecutor has many workers
type: crash
versions: Python 3.4
Added file: http://bugs.python.org/file38801/example_thread_executor.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23846
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23846] asyncio : ProactorEventLoop raised BlockingIOError when ThreadPoolExecutor has many workers

[SeungHyun.Hwang]

Changes by SeungHyun.Hwang thewhiteap...@gmail.com:


--
type: crash - behavior

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23846
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com