[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-10-21 Thread miss-islington 


miss-islington  added the comment:


New changeset 216c040bb1fdf73e78d67ab82a43563d7593f874 by Miss Islington (bot) 
in branch '3.9':
bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)
https://github.com/python/cpython/commit/216c040bb1fdf73e78d67ab82a43563d7593f874


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-10-21 Thread Gregory P. Smith 


Change by Gregory P. Smith :


--
resolution:  -> fixed
stage: patch review -> commit review
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-10-21 Thread miss-islington 


miss-islington  added the comment:


New changeset 8f6aa48cb2dc827a2cb76e35e91bf02d099875c5 by Miss Islington (bot) 
in branch '3.10':
bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)
https://github.com/python/cpython/commit/8f6aa48cb2dc827a2cb76e35e91bf02d099875c5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-10-21 Thread miss-islington 


Change by miss-islington :


--
message_count: 10.0 -> 11.0
nosy: +miss-islington
nosy_count: 6.0 -> 7.0
pull_requests: +27420
pull_request: https://github.com/python/cpython/pull/29144

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-10-21 Thread Gregory P. Smith 


Gregory P. Smith  added the comment:


New changeset 843b890334ca30cf6af27dffe29cecd06b49f7d9 by Gregory P. Smith in 
branch 'main':
bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)
https://github.com/python/cpython/commit/843b890334ca30cf6af27dffe29cecd06b49f7d9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-10-21 Thread miss-islington 


Change by miss-islington :


--
pull_requests: +27421
pull_request: https://github.com/python/cpython/pull/29145

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-10-21 Thread Gregory P. Smith 


Change by Gregory P. Smith :


--
keywords: +patch
pull_requests: +27414
pull_request: https://github.com/python/cpython/pull/29127

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-10-21 Thread Gregory P. Smith 


Gregory P. Smith  added the comment:

macOS often has an unreasonably low default max number of file descriptors.  
This is easy to raise from user code, it's a soft limit.

https://github.com/python/cpython/pull/29127

is out to try that from regrtest itself.

--
assignee:  -> gregory.p.smith
components: +macOS
nosy: +gregory.p.smith, ronaldoussoren
stage:  -> patch review
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-09-10 Thread STINNER Victor 


STINNER Victor  added the comment:

FYI the test is still failing with the same error on the billenstein-macos 
builder worker, x86-64 macOS 3.10:
https://buildbot.python.org/all/#/builders/681/builds/370

"OSError: [Errno 24] Too many open files"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-09-01 Thread Łukasz Langa 

Łukasz Langa  added the comment:

Before you do it, check the defaults with:

$ launchctl limit maxfiles

I'm curious.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-09-01 Thread Łukasz Langa 

Łukasz Langa  added the comment:

Try setting the following:

$ sudo sysctl -w kern.maxfiles=65536
$ sudo sysctl -w kern.maxfilesperproc=65536
$ sudo ulimit -n 65536

--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-09-01 Thread STINNER Victor 


STINNER Victor  added the comment:

> x86-64 macOS 3.10: https://buildbot.python.org/all/#/builders/681/builds/345

On this machine, RLIMIT_NOFILE soft limit is 256.

test.pythoninfo:

resource.RLIMIT_NOFILE: (256, 9223372036854775807)
platform.platform: macOS-11.5.2-x86_64-i386-64bit
socket.hostname: mattb-mbp2

All limits:

resource.RLIMIT_AS: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_CORE: (0, 9223372036854775807)
resource.RLIMIT_CPU: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_DATA: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_FSIZE: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_MEMLOCK: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_NOFILE: (256, 9223372036854775807)
resource.RLIMIT_NPROC: (2784, 4176)
resource.RLIMIT_RSS: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_STACK: (67104768, 67104768)
resource.pagesize: 4096

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-09-01 Thread STINNER Victor 


STINNER Victor  added the comment:

x86-64 macOS 3.10:
https://buildbot.python.org/all/#/builders/681/builds/345

==
FAIL: test_multiprocessing_pool_circular_import 
(test.test_importlib.test_threaded_import.ThreadedImportTests)
--
Traceback (most recent call last):
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py",
 line 258, in test_multiprocessing_pool_circular_import
script_helper.assert_python_ok(fn)
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py",
 line 160, in assert_python_ok
return _assert_python(True, *args, **env_vars)
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py",
 line 145, in _assert_python
res.fail(cmd_line)
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py",
 line 72, in fail
raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: 
['/Users/buildbot/buildarea/3.10.billenstein-macos/build/python.exe', '-X', 
'faulthandler', '-I', 
'/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py']

stdout:
---

---

stderr:
---
Traceback (most recent call last):
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py",
 line 9, in t
with multiprocessing.Pool(1):
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/context.py",
 line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py",
 line 212, in __init__
self._repopulate_pool()
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py",
 line 303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py",
 line 326, in _repopulate_pool_static
w.start()
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/process.py",
 line 121, in start
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/context.py",
 line 284, in _Popen
return Popen(process_obj)
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py",
 line 32, in __init__
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_fork.py",
 line 19, in __init__
self._launch(process_obj)
  File 
"/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py",
 line 53, in _launch
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224:
 UserWarning: resource_tracker: There appear to be 120 leaked semaphore objects 
to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237:
 UserWarning: resource_tracker: '/mp-klnfwvgv': [Errno 2] No such file or 
directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237:
 UserWarning: resource_tracker: '/mp-phgypx9f': [Errno 2] No such file or 
directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
---

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-04-28 Thread Liel Fridman 


Liel Fridman  added the comment:

I have the same error when running the tests on my machine (Same macOS version).

--
nosy: +lielfr

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-03-22 Thread Ned Deily 


Ned Deily  added the comment:

FWIW, I don't recall ever seeing a "too many open files" failure for 
test_importlib on any of my macOS systems.

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-03-22 Thread STINNER Victor 


STINNER Victor  added the comment:

test_importlib only fails *randomly* on x86-64 macOS 3.x.

For example, the build 970 is a success (but orange): "Tests result: FAILURE 
then SUCCESS". test_importlib failed with too many open files, and then passed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43592] test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()

2021-03-22 Thread STINNER Victor 


New submission from STINNER Victor :

x86-64 macOS 3.x:
https://buildbot.python.org/all/#/builders/366/builds/969

Build triggered by the commit 88d9983b561cd59e5f186d98227de0c1a022b498 which 
changes PyImport_Import().


The buildbot is running on macOS 10.15.7 (Darwin Kernel Version 19.6.0) with a 
limit of 256 file descriptors.


The latest successful build is build 968, whereas the the RLIMIT_NOFILE 
resource soft limit was also set to 256:
https://buildbot.python.org/all/#/builders/366/builds/968


test.pythoninfo:

* os.uname: posix.uname_result(sysname='Darwin', nodename='mattb-mbp2', 
release='19.6.0', version='Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 
PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64', machine='x86_64')
* sysconfig[HOST_GNU_TYPE]: x86_64-apple-darwin19.6.0
* platform.platform: macOS-10.15.7-x86_64-i386-64bit
* resource.RLIMIT_NOFILE: (256, 9223372036854775807)


FAIL: test_multiprocessing_pool_circular_import 
(test.test_importlib.test_threaded_import.ThreadedImportTests)
--
Traceback (most recent call last):
  File 
"/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py",
 line 258, in test_multiprocessing_pool_circular_import
script_helper.assert_python_ok(fn)
  File 
"/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py",
 line 160, in assert_python_ok
return _assert_python(True, *args, **env_vars)
  File 
"/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py",
 line 145, in _assert_python
res.fail(cmd_line)
  File 
"/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py",
 line 72, in fail
raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: 
['/Users/buildbot/buildarea/3.x.billenstein-macos/build/python.exe', '-X', 
'faulthandler', '-I', 
'/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py']

stdout:
---

---

stderr:
---
Traceback (most recent call last):
  File 
"/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py",
 line 9, in t
  File 
"/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py",
 line 119, in Pool
  File 
"/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py",
 line 196, in __init__
  File 
"/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py",
 line 113, in SimpleQueue
  File 
"/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/queues.py",
 line 341, in __init__
self._reader, self._writer = connection.Pipe(duplex=False)
  File 
"/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/connection.py",
 line 532, in Pipe
fd1, fd2 = os.pipe()
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224:
 UserWarning: resource_tracker: There appear to be 110 leaked semaphore objects 
to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237:
 UserWarning: resource_tracker: '/mp-56i6i4ap': [Errno 2] No such file or 
directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237:
 UserWarning: resource_tracker: '/mp-0au5otkl': [Errno 2] No such file or 
directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237:
 UserWarning: resource_tracker: '/mp-vcv0xwbi': [Errno 2] No such file or 
directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237:
 UserWarning: resource_tracker: '/mp-vxfb4ks9': [Errno 2] No such file or 
directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237:
 UserWarning: resource_tracker: '/mp-5e2_0z1f': [Errno 2] No such file or 
directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237:
 UserWarning: resource_tracker: '/mp-6vsgax4k': [Errno 2] No such file or 
directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237:
 UserWarning: resource_tracker: '/mp-mq51g4b_': [Errno 2] No such file or 
directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))