[issue35242] multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished

[Sergei Zobov]

New submission from Sergei Zobov :

In case if the process target function put something unpickable in the
`multiprocessing.Queue` and finish before `ForkingPickler`'s exception is 
handled we get two weird situations:
* Internal queue semaphore stay acquired and never will be released, so we get 
`qsize()` always returns 1(or more if more then one process) even if there is 
nothing in queue's buffer and `empty()` returns `True`.
* An exception's traceback will be silently suppressed.

To reproduce this behavior I wrote a test in cpython repo (be aware sometime it 
pass sometime no):
https://github.com/szobov/cpython/commit/95524f10a7b6510e9ab4647223cc67af85ebff86

I suppose it happens because of this:

* The function `util._exit_function()` calls every time process target function 
finished. Inside this function the global (for current process) flag 
`utils._exiting` marks as `True`:
https://github.com/python/cpython/blob/master/Lib/multiprocessing/process.py#L300
https://github.com/python/cpython/blob/master/Lib/multiprocessing/util.py#L294-L295
* When exception while pickling object happens in `Queue._feed` function we are 
ignore it because of the clause `is_exiting()` that checks the flag 
`utils._exiting` described above.
https://github.com/python/cpython/blob/master/Lib/multiprocessing/queues.py#L257-L259

So we have unreleased semaphore and ignored exceptions.
I think it's quite disappointing for user because they have broken queue and 
nothing about why it happened.

I'm not quite sure that it's a bag but I found nothing in documentation about 
it.

--
messages: 329889
nosy: szobov
priority: normal
severity: normal
status: open
title: multiprocessing.Queue in an inconsistent state and a traceback silently 
suppressed if put an unpickable object and process's target function is finished
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35240] Travis CI: xvfb-run: error: Xvfb failed to start

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Found a relevant issue but 3 years old : 
https://github.com/travis-ci/travis-ci/issues/4387#issuecomment-125635027

Doc PR : https://github.com/travis-ci/docs-travis-ci-com/pull/311 . The docs 
recommend usage of -a to select a free port.

--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35241] test_embed fails on MacOS buildbots

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
stage:  -> needs patch
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35241] test_embed fails on MacOS buildbots

[Pablo Galindo Salgado]

New submission from Pablo Galindo Salgado :

https://buildbot.python.org/all/#/builders/147/builds/430/steps/4/logs/stdio
https://buildbot.python.org/all/#/builders/109/builds/789/steps/4/logs/stdio


==
FAIL: test_init_default_config (test.test_embed.InitConfigTests)
--
Traceback (most recent call last):
  File 
"/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_embed.py",
 line 398, in test_init_default_config
self.check_config("init_default_config", {}, {})
  File 
"/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_embed.py",
 line 391, in check_config
self.assertEqual(config['global_config'], expected_global)
AssertionError: {'Py_FileSystemDefaultEncoding': 'utf-8', 'Py_[467 chars]': 0} 
!= {'Py_BytesWarningFlag': 0, 'Py_DebugFlag': 0, [467 chars]': 0}
  {'Py_BytesWarningFlag': 0,
   'Py_DebugFlag': 0,
   'Py_DontWriteBytecodeFlag': 0,
   'Py_FileSystemDefaultEncodeErrors': 'surrogateescape',
   'Py_FileSystemDefaultEncoding': 'utf-8',
   'Py_FrozenFlag': 0,
-  'Py_HasFileSystemDefaultEncoding': 1,
? ^
+  'Py_HasFileSystemDefaultEncoding': 0,
? ^
   'Py_HashRandomizationFlag': 1,
   'Py_IgnoreEnvironmentFlag': 0,
   'Py_InspectFlag': 0,
   'Py_InteractiveFlag': 0,
   'Py_IsolatedFlag': 0,
   'Py_NoSiteFlag': 0,
   'Py_NoUserSiteDirectory': 0,
   'Py_OptimizeFlag': 0,
   'Py_QuietFlag': 0,
   'Py_UTF8Mode': 0,
   'Py_UnbufferedStdioFlag': 0,
   'Py_VerboseFlag': 0}
==
FAIL: test_init_dev_mode (test.test_embed.InitConfigTests)
--
Traceback (most recent call last):
  File 
"/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_embed.py",
 line 479, in test_init_dev_mode
self.check_config("init_dev_mode", core_config, {})
  File 
"/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_embed.py",
 line 391, in check_config
self.assertEqual(config['global_config'], expected_global)
AssertionError: {'Py_FileSystemDefaultEncoding': 'utf-8', 'Py_[467 chars]': 0} 
!= {'Py_BytesWarningFlag': 0, 'Py_DebugFlag': 0, [467 chars]': 0}
  {'Py_BytesWarningFlag': 0,
   'Py_DebugFlag': 0,
   'Py_DontWriteBytecodeFlag': 0,
   'Py_FileSystemDefaultEncodeErrors': 'surrogateescape',
   'Py_FileSystemDefaultEncoding': 'utf-8',
   'Py_FrozenFlag': 0,
-  'Py_HasFileSystemDefaultEncoding': 1,
? ^
+  'Py_HasFileSystemDefaultEncoding': 0,
? ^
   'Py_HashRandomizationFlag': 1,
   'Py_IgnoreEnvironmentFlag': 0,
   'Py_InspectFlag': 0,
   'Py_InteractiveFlag': 0,
   'Py_IsolatedFlag': 0,
   'Py_NoSiteFlag': 0,
   'Py_NoUserSiteDirectory': 0,
   'Py_OptimizeFlag': 0,
   'Py_QuietFlag': 0,
   'Py_UTF8Mode': 0,
   'Py_UnbufferedStdioFlag': 0,
   'Py_VerboseFlag': 0}
==
FAIL: test_init_isolated (test.test_embed.InitConfigTests)
--
Traceback (most recent call last):
  File 
"/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_embed.py",
 line 489, in test_init_isolated
self.check_config("init_isolated", core_config, global_config)
  File 
"/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_embed.py",
 line 391, in check_config
self.assertEqual(config['global_config'], expected_global)
AssertionError: {'Py_FileSystemDefaultEncoding': 'utf-8', 'Py_[467 chars]': 1} 
!= {'Py_BytesWarningFlag': 0, 'Py_DebugFlag': 0, [467 chars]': 1}
  {'Py_BytesWarningFlag': 0,
   'Py_DebugFlag': 0,
   'Py_DontWriteBytecodeFlag': 0,
   'Py_FileSystemDefaultEncodeErrors': 'surrogateescape',
   'Py_FileSystemDefaultEncoding': 'utf-8',
   'Py_FrozenFlag': 0,
-  'Py_HasFileSystemDefaultEncoding': 1,
? ^
+  'Py_HasFileSystemDefaultEncoding': 0,
? ^
   'Py_HashRandomizationFlag': 1,
   'Py_IgnoreEnvironmentFlag': 1,
   'Py_InspectFlag': 0,
   'Py_InteractiveFlag': 0,
   'Py_IsolatedFlag': 1,
   'Py_NoSiteFlag': 0,
   'Py_NoUserSiteDirectory': 1,
   'Py_OptimizeFlag': 0,
   'Py_QuietFlag': 0,
   'Py_UTF8Mode': 0,
   'Py_UnbufferedStdioFlag': 0,
   'Py_VerboseFlag': 0}
--
Ran 15 tests in 6.518s
FAILED (failures=3)
1 test failed again:
test_embed

--
components: Tests
messages: 329887
nosy: pablogsal, vstinner
priority: normal
severity: normal
status: open
title: test_embed fails on MacOS buildbots
versions: Python 3.7

___
Python tracker 

___
___

[issue35233] _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers

[STINNER Victor]

Change by STINNER Victor :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35233] _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 88cbea4c6ff4987ce31f4fe6f73c2d04a3d37829 by Victor Stinner in 
branch '3.7':
bpo-35233: Fix _PyMainInterpreterConfig_Copy() (GH-10537)
https://github.com/python/cpython/commit/88cbea4c6ff4987ce31f4fe6f73c2d04a3d37829


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

A few other things I don't understand:

* Why does setting OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES only seem to work 
when it's set in the shell before the parent process executes?  AFAICT, it does 
*not* work if you set that in os.environ in the parent process before the 
os.fork().

* Why does it only crash on the first invocation of our app?  Does getproxies() 
cache the results somehow?  There's too much internal application code in the 
way to know if we're doing something that prevents getproxies() from getting 
called in subsequent calls.

* I can't seem to produce a smaller test case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35214] Get the test suite passing with clang Memory Sanitizer enabled

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +9789

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35240] Travis CI: xvfb-run: error: Xvfb failed to start

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

xvfb is a utility that allows you to run a command in a virtual X server 
environment. I assume that is needed for the doctests that use the X server. I 
assume is related to PR9806

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35240] Travis CI: xvfb-run: error: Xvfb failed to start

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +matrixise

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35240] Travis CI: xvfb-run: error: Xvfb failed to start

[STINNER Victor]

STINNER Victor  added the comment:

Extract of .travis.yml:

$XVFB_RUN make buildbottest TESTOPTS="-j4 -uall,-cpu"
if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then
  $XVFB_RUN make PYTHON=../python SPHINXOPTS="-q -W -j4" -C Doc/ venv 
doctest
fi

I'm not sure what is xvfb, nor why it's needed to run doctests...

--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34784] Heap-allocated StructSequences

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

PR9665 solves also Issue28709

--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35240] Travis CI: xvfb-run: error: Xvfb failed to start

[STINNER Victor]

New submission from STINNER Victor :

Travis CI fails randomly with "xvfb-run: error: Xvfb failed to start". Example:
https://travis-ci.org/python/cpython/jobs/454776965

whereas the full test suite succeeded.

--
messages: 329881
nosy: vstinner
priority: normal
severity: normal
status: open
title: Travis CI: xvfb-run: error: Xvfb failed to start

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

Hoo boy.  I'm not sure I have the full picture, but things are starting to come 
into focus.  After much debugging, I've narrowed down at least one crash to 
urllib.request.getproxies().  On macOS (darwin), this ends up calling 
_scproxy.get_proxies() which calls into the SystemConfiguration framework.  
I'll bet dollars to donuts that that calls into the ObjC runtime.  Thus it is 
unsafe to call between fork and exec.  This certainly seems to be the case even 
if the environment variable is set.

The problem is that I think requests.post() probably also ends up in here 
somehow (still untraced), because by removing our call to 
urllib.requests.getproxies(), we just crash later on when requests.post() is 
called.

I don't know what, if anything can be done in Python, except perhaps to 
document that anything that calls into the ObjC runtime between fork and exec 
can potentially crash the subprocess.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35239] _PySys_EndInit() doesn't copy main interpreter configuration

[STINNER Victor]

STINNER Victor  added the comment:

Attached PR 10532 modifies _PySys_EndInit() to copy lists and dictionaries from 
the config.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35233] _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9788

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Python crashes on macOS after fork with no exec

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
title: Pytho crashes on macOS after fork with no exec -> Python crashes on 
macOS after fork with no exec

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34784] Heap-allocated StructSequences

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +9787

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35233] _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 35c28d562ec7249f2a6aef8e326eadac130a1656 by Victor Stinner in 
branch '3.7':
[3.7] bpo-35233: Rewrite test_embed.InitConfigTests (GH-10524) (GH-10529)
https://github.com/python/cpython/commit/35c28d562ec7249f2a6aef8e326eadac130a1656


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23734] zipimport should not check pyc timestamps against zipped py files

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Correct, this issue was filed before we had the new pyc options (which came out 
of our September core dev. sprint in 2016 iirc).

I'll just close it, the way forward for people using zip files on modern 
CPython interpreters is to explicitly use the new unchecked feature.

--
resolution:  -> out of date
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31354] Fixing a bug related to LTO only build

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Is it an actual problem to compile extension modules with -flto?  (I realize as 
an extension module there isn't a huge benefit to the concept unless it happens 
to have multiple source files or link against a non-shared library)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35214] Get the test suite passing with clang Memory Sanitizer enabled

[miss-islington]

miss-islington  added the comment:


New changeset 9fbcb1402efab4e287f25145a69ba14c9c6dbce9 by Miss Islington (bot) 
in branch '3.7':
[3.7] bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506) 
(GH-10522)
https://github.com/python/cpython/commit/9fbcb1402efab4e287f25145a69ba14c9c6dbce9


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35239] _PySys_EndInit() doesn't copy main interpreter configuration

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +9786
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35233] _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9785

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16438] Numeric operator predecence confusing

[Mariatta Wijaya]

Mariatta Wijaya  added the comment:

I'm actually thinking perhaps the "see :ref:`operator-summary`" clause should 
be separate and under Notes below the table, as opposed to current PR:

```
All numeric types (except complex) support the following operations (for 
priorities of
the operations, see :ref:`operator-summary`)
```

reads oddly to me.

--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35233] _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 7ddd56f4d835c6107b20a0b4233185bb59270142 by Victor Stinner in 
branch 'master':
bpo-35233: Rewrite test_embed.InitConfigTests (GH-10524)
https://github.com/python/cpython/commit/7ddd56f4d835c6107b20a0b4233185bb59270142


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35239] _PySys_EndInit() doesn't copy main interpreter configuration

[STINNER Victor]

New submission from STINNER Victor :

Extract of _PySys_EndInit():

SET_SYS_FROM_STRING_BORROW("path", config->module_search_path);

sys.path is initialized from _PyMainInterpreterConfig.module_search_path 
object: the list of strings is not copied. As a consequence, when sys.path is 
modified, _PyMainInterpreterConfig is modified as well. For example, "import 
site" modifies sys.path.

I dislike this behavior. I prefer to see _PyMainInterpreterConfig as 
"constant": a snapshot of the configuration used to startup Python.

A side effect is that Py_NewInterpreter() copies the modified configuration, 
whereas starting from the "original" configuration. For the specific case of 
Py_NewInterpreter(), I'm not sure of what is the expected behavior... 
Py_NewInterpreter() imports the "site" module again, so it should reapply the 
same sys.path change.

Note: I found this bug while working on better tests for global, core and main 
configurations: https://github.com/python/cpython/pull/10524

--
components: Interpreter Core
messages: 329872
nosy: eric.snow, ncoghlan, vstinner
priority: normal
severity: normal
status: open
title: _PySys_EndInit() doesn't copy main interpreter configuration
versions: Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35233] _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9784

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] Pytho crashes on macOS after fork with no exec

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
title: macOS crashes after fork with no exec -> Pytho crashes on macOS after 
fork with no exec

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] macOS crashes after fork with no exec

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
title: High Sierra hang when using multi-processing -> macOS crashes after fork 
with no exec

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] High Sierra hang when using multi-processing

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

issue35219 is where I've run into this problem.  I'm still trying to figure out 
all the details in my own case, but I can confirm that setting the environment 
variable does not always help.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33725] High Sierra hang when using multi-processing

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
versions: +Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35214] Get the test suite passing with clang Memory Sanitizer enabled

[miss-islington]

miss-islington  added the comment:


New changeset fdc485a5dff8508328b4f26265d0af1ba5004597 by Miss Islington (bot) 
in branch '3.6':
bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506)
https://github.com/python/cpython/commit/fdc485a5dff8508328b4f26265d0af1ba5004597


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35219] macOS 10.14 Mojave crashes in multiprocessing

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

I am going to dupe this to 33725 after all.

--
resolution:  -> duplicate
superseder:  -> High Sierra hang when using multi-processing

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

[Oscar Esteban]

New submission from Oscar Esteban :

## Context

We are developers of nipype (https://github.com/nipype) which is a workflow 
engine for neuroimaging software. We are experiencing problems that gave rise 
to the addition of ``os.posix_spawn`` to Python 3.8, and particularly, this - 
https://bugs.python.org/issue20104#msg222570

Our software runs command line subprocesses that can be quite memory-hungry and 
in some cases, in the order of tens of thousands processes. Therefore, we 
frequently see the OOM killing some of the processes.

## Status

We have successfully leveraged the ``forkserver`` context (in addition to a low 
number of `maxtasksperchild`) of multiprocessing to ease the load. However, the 
fork_exec memory allocation is still problematic on systems that do not allow 
overcommitting virtual memory. Waiting for os.posix_spawn to be rolled out 
might not be an option for us, as the problem is hitting badly right now.

## Proposed solution

I'd like to page experts on Lib/multiprocessing and Lib/subprocess to give 
their opinions about the following: is it possible to write an extension to 
`multiprocessing.util.Popen` such that it has the API of `subprocess.Popen` but 
the fork happens via the forkserver?

My naive intuition is that we would need to create a new type of Process, make 
sure that it then calls os.exec*e() --possibly around here 
https://github.com/python/cpython/blob/f966e5397ed8f5c42c185223fc9b4d750a678d02/Lib/multiprocessing/popen_forkserver.py#L51--,
 and finally handle communication with the subprocess.

Please let me know if that is even possible.

--
components: Library (Lib)
messages: 329868
nosy: oesteban
priority: normal
severity: normal
status: open
title: Alleviate memory reservation of fork_exec in subprocess.Popen via 
forkserver
type: enhancement
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35214] Get the test suite passing with clang Memory Sanitizer enabled

[miss-islington]

Change by miss-islington :


--
pull_requests: +9783

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35214] Get the test suite passing with clang Memory Sanitizer enabled

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset 746b2d35ea47005054ed774fecaed64fab803d7d by Gregory P. Smith in 
branch 'master':
bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506)
https://github.com/python/cpython/commit/746b2d35ea47005054ed774fecaed64fab803d7d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35214] Get the test suite passing with clang Memory Sanitizer enabled

[miss-islington]

Change by miss-islington :


--
pull_requests: +9782

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35235] Access violation on alloc in Windows x86-64 python, pymalloc_alloc

[STINNER Victor]

STINNER Victor  added the comment:

Try to reproduce your bug using PYTHONMALLOC=debug environment variable:
https://pythondev.readthedocs.io/debug_tools.html#memory-debugger

On Python 3.7, you can use "py -3.7 -X dev program.py" (-X dev option) to 
enable PYTHONMALLOC=debug.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35237] Allow Path instances in sys.path ?

[Ammar Askar]

Ammar Askar  added the comment:

Looks like there's an issue already open for this.

--
nosy: +ammar2
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> add support for path-like objects in sys.path

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35237] Allow Path instances in sys.path ?

[Thierry Parmentelat]

New submission from Thierry Parmentelat :

Hi; this is my first entry in this repo

I ran today into a rather frustrating experience, that I'd to avoid it for 
others as far as possible

In a nutshell I had a sphinx stub that read:

import sys
sys.path.append("..")

import mymodule


and because I was switching from os.path to pathlib.Path, I rather stupidly 
changed it into this

import sys
sys.path.append(Path("..").resolve())

import mymodule


---
When trying to run this new code, the message I got was ModuleNotFoundError
So I tried to troubleshoot the situation, and inserted something like

for x in sys.path:
print(x)

and so the proper path showed up, and that's the point where I started banging 
my head against the screen
Granted, I was doing too many changes at the same time, and it's entirely my 
fault if it took me ages really to figure it out eventually.

---
Still, my proposal would be to

*) either accept Path instances in sys.path, or
*) somehow send a warning message stating that the Path instance is totally 
ignored; at least, I haven't found the corresponding piece of code yet, but as 
far as I am concerned it's truly as if this Path instance had been completely 
ignored when executing the import statement
*) or at the very least add a mention about this possible issue in the import 
documentation


I understand it was a stupid mistake, but with Path becoming THE right way to 
deal with paths, I think it is safe to predict that I won't be the only one 
making it.

As I said I am a complete newbie with cpython's codebase and devel practices, 
so I'll warmly welcome all inputs on this one.

thanks

--
components: Library (Lib)
messages: 329864
nosy: thierry.parmentelat
priority: normal
severity: normal
status: open
title: Allow Path instances in sys.path ?
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16438] Numeric operator predecence confusing

[Anjali Malik]

Change by Anjali Malik :


--
pull_requests: +9781
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Glenn Linderman]

Glenn Linderman  added the comment:

That's interesting, Pierre, I hadn't really read the RFC carefully, to realize 
that many of the "missing" variables from Apache are HTTP headers, and that 
section 4.1.18 tell how to convert HTTP headers to meta variables.

The code in server.py 3.6 (Sorry, I should check the master branch) picks 
specific HTTP_ headers to include, rather than including them all per the 
rules. Doing the latter would go a long way toward being more compatible with 
Apache. I don't know if Rémi got his NGINX list from source code (looks like 
it) and if maybe NGINX also defines meta variables from the HTTP_ headers, that 
are not listed in the header file he seems to be quoting.

Unless the code has already been improved for Python 3.7, I think there is 
still some work to do to make server.py conform even to the RFC, if not be 
compatible with Apache.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35236] urllib.request.urlopen throws on some valid FTP files

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch
pull_requests: +9779
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35236] urllib.request.urlopen throws on some valid FTP files

[Ian Liu Rodrigues]

New submission from Ian Liu Rodrigues :

Some FTP clients will not allow changing to a directory if the path does not 
ends with a slash. For example, try out this in a public FTP:

from ftplib import FTP
ftp = FTP('ftp.unicamp.br')
ftp.login()
ftp.cwd('pub/libreoffice') # throws error
ftp.cwd('pub/libreoffice/') # OK

The problem is urllib.request doesn't include the trailing slash, thus throwing 
an error. This behavior also happens with the command line ftp client.

I think this happens because the libreoffice directory is a symlink, and this 
can be a FTP server specific behavior.

--
components: Library (Lib)
messages: 329862
nosy: Ian Liu Rodrigues
priority: normal
severity: normal
status: open
title: urllib.request.urlopen throws on some valid FTP files
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35223] Pathlib incorrectly merges strings.

[Eryk Sun]

Eryk Sun  added the comment:

os.path.join and pathlib are working as designed and documented. Similarly in 
POSIX we have the following:

>>> p = os.path.join("/123/345", "/", "folder///filename.bin")
>>> print(p)
/folder///filename.bin
>>> print(os.path.normpath(p))
/folder/filename.bin

The difference is that in Windows there's no root ("/") filesystem, but instead 
a set of DOS devices (e.g. "C:", "CON:") and UNC shares (e.g. 
r"\\server\share"), so the Windows implementation of join() uses the drive from 
the already-joined components in order to resolve a rooted component.

--
nosy: +eryksun
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35233] _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 00b137c72f90fbc39a6cd7e48b37c58d19977180 by Victor Stinner in 
branch 'master':
bpo-35233: Fix _PyMainInterpreterConfig_Copy() (GH-10519)
https://github.com/python/cpython/commit/00b137c72f90fbc39a6cd7e48b37c58d19977180


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35235] Access violation on alloc in Windows x86-64 python, pymalloc_alloc

[Mark Inderhees]

Mark Inderhees  added the comment:

Actually, I have now seen this in full installed python. I have updated the 
title.

--
title: Access violation on alloc in Windows x86-64 embeddable python, 
pymalloc_alloc -> Access violation on alloc in Windows x86-64 python, 
pymalloc_alloc

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35232] Add `module`/`qualname` arguments to make_dataclass for picklability

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35222] email.utils.formataddr is not exactly the reverse of email.utils.parseaddr

[R. David Murray]

R. David Murray  added the comment:

Because the RFCs are defined only for ascii.  Non-ascii in RFC 2822 addresses 
is an RFC violation.  In python2 non-ascii would usually round-trip through 
these functions, but again that was an accident.

If you'd like to propose a doc clarification that would be fine, but the 
clarification would be that behavior on strings containing non-ascii is 
undefined.

Note that these functions are considered soft-deprecated...they are in modules 
that are in the "Legacy API" section of the email docs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35235] Access violation on alloc in Windows x86-64 embeddable python, pymalloc_alloc

[Mark Inderhees]

New submission from Mark Inderhees :

I'm seeing an AV in python allocation logic when running my scripts. This only 
happens when using embedded python. It does not reproduce when using installed 
python. This is on Windows x86-64, I'm running Windows 10. I've seen this on 
3.6.5 and 3.7.1.

Here's the 3.7.1 stack on failure:

0:004> kpn
 # Child-SP  RetAddr   Call Site
00 00d1`7439d650 7ff8`e8303db6 python37!pymalloc_alloc(void ** ptr_p = 
0x00d1`7439d6c0, unsigned int64 nbytes = )+0x4b 
[c:\_work\5\s\objects\obmalloc.c @ 1398] 
01 00d1`7439d680 7ff8`e82fb064 python37!_PyObject_Malloc(void * ctx = 
0x`, unsigned int64 nbytes = 0x1c)+0x16 
[c:\_work\5\s\objects\obmalloc.c @ 1555] 
02 (Inline Function) ` python37!PyObject_Malloc+0x1c 
[c:\_work\5\s\objects\obmalloc.c @ 616] 
03 00d1`7439d6b0 7ff8`e82fb18c python37!_PyLong_New(int64 size = 
0n1)+0x40 [c:\_work\5\s\objects\longobject.c @ 210] 
04 00d1`7439d6e0 7ff8`e83cbf28 python37!PyLong_FromLongLong(int64 ival 
= )+0x4c [c:\_work\5\s\objects\longobject.c @ 1102] 
05 (Inline Function) ` python37!_PyLong_FromTime_t+0xc 
[c:\_work\5\s\python\pytime.c @ 88] 
06 00d1`7439d710 7ff8`e83cb86e python37!tmtotuple(struct tm * p = 
0x00d1`7439d768, char * zone = 0x00d1`7439d7c0 "Pacific Standard Time", 
int64 gmtoff = 0n-28800)+0xe0 [c:\_work\5\s\modules\timemodule.c @ 426] 
07 00d1`7439d740 7ff8`e8307e18 python37!time_localtime(struct _object * 
self = 0x029f`3753bbe0, struct _object * args = )+0x96 [c:\_work\5\s\modules\timemodule.c @ 521] 
08 00d1`7439d850 7ff8`e8308737 
python37!_PyMethodDef_RawFastCallKeywords(struct PyMethodDef * method = 
0x`, struct _object * self = 0x029f`371b7778, struct 
_object ** args = 0x029f`3754b3c8, int64 nargs = 0n1, struct _object * 
kwnames = 0x`)+0x3a8 [c:\_work\5\s\objects\call.c @ 694] 
09 (Inline Function) ` 
python37!_PyCFunction_FastCallKeywords+0x22 [c:\_work\5\s\objects\call.c @ 730] 
0a 00d1`7439d8d0 7ff8`e83091d3 python37!call_function(struct _object 
*** pp_stack = 0x00d1`7439da28, int64 oparg = , 
struct _object * kwnames = 0x`)+0x3a7 
[c:\_work\5\s\python\ceval.c @ 4554] 
0b 00d1`7439d990 7ff8`e82f1390 python37!_PyEval_EvalFrameDefault(struct 
_frame * f = 0x029f`3754b220, int throwflag = 0n928230480)+0x913 
[c:\_work\5\s\python\ceval.c @ 3095] 
0c (Inline Function) ` python37!PyEval_EvalFrameEx+0x17 
[c:\_work\5\s\python\ceval.c @ 547] 
0d 00d1`7439dad0 7ff8`e83086a2 python37!_PyEval_EvalCodeWithName(struct 
_object * _co = 0x029f`37415a50, struct _object * globals = , struct _object * locals = , struct 
_object ** args = 0x`2a88, int64 argcount = 0n3, struct _object ** 
kwnames = 0x`, struct _object ** kwargs = 0x029f`3753de28, 
int64 kwcount = 0n0, int kwstep = 0n1, struct _object ** defs = 
0x029f`3743fe60, int64 defcount = 0n1, struct _object * kwdefs = 
0x`, struct _object * closure = 0x`, struct 
_object * name = 0x029f`3744f470, struct _object * qualname = 
0x029f`37441930)+0x1a0 [c:\_work\5\s\python\ceval.c @ 3930] 
0e (Inline Function) ` 
python37!_PyFunction_FastCallKeywords+0x274 [c:\_work\5\s\objects\call.c @ 433] 
0f 00d1`7439db80 7ff8`e8308d29 python37!call_function(struct _object 
*** pp_stack = 0x00d1`7439dcd8, int64 oparg = , 
struct _object * kwnames = 0x`)+0x312 
[c:\_work\5\s\python\ceval.c @ 4607] 


--
messages: 329857
nosy: markind
priority: normal
severity: normal
status: open
title: Access violation on alloc in Windows x86-64 embeddable python, 
pymalloc_alloc

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35216] misleading error message from shutil.copy()

[Eryk Sun]

Eryk Sun  added the comment:

The EINVAL error in Windows also needs improvement, but I don't what can be 
done after the fact. If there's a trailing slash when opening or creating a 
regular file, the NtCreateFile system call returns STATUS_OBJECT_INVALID_NAME. 
The Windows API maps this to ERROR_INVALID_NAME (123), which the CRT in turn 
maps to EINVAL (22). This error is too generic to handle. Even if the name ends 
in a slash, the error could be due to some other invalid character in the path 
(e.g. a common mistake is a '\t' or '\n' in a string literal). 

The problem could be addressed beforehand in shutil.copy by manually raising an 
exception if isdir() is false and the name has a trailing slash.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35195] [Windows] Python 3.7 initializes LC_CTYPE locale at startup, causing performance issue on msvcrt isdigit()

[Dragoljub]

Dragoljub  added the comment:

This is the default LC_CTYPE locale type I see on Windows10 and Python 3.7.1 vs 
3.5.2: Same: 'C'


'3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)]'

import _locale
_locale.setlocale(_locale.LC_CTYPE, None)

'C'


'3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AMD64)]'

import _locale
_locale.setlocale(_locale.LC_CTYPE, None)

'C'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35195] [Windows] Python 3.7 initializes LC_CTYPE locale at startup, causing performance issue on msvcrt isdigit()

[Dragoljub]

Dragoljub  added the comment:

On Python 3.7.1 and Windows 10:

I attempted locale.setlocale(locale.LC_ALL, "POSIX") --> Errors Out
---
Error Traceback (most recent call last)
 in 
> 1 locale.setlocale(locale.LC_ALL, "POSIX")

D:\Python37\lib\locale.py in setlocale(category, locale)
602 # convert to string
603 locale = normalize(_build_localename(locale))
--> 604 return _setlocale(category, locale)
605 
606 def resetlocale(category=LC_ALL):

Error: unsupported locale setting


I was able to set the loacle to "C" but that does not improve the parsing 
performance.

locale.setlocale(locale.LC_ALL, "C") --> returns 'C'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35234] ssl module falls over with internationalized domain names

[Christian Heimes]

Christian Heimes  added the comment:

This is a duplicate of #28414. I decided to not fix Python 3.6 because I had to 
break an existing API to fix the problem.

And no, it didn't get fixed by accident. Nathaniel and I poured a lot of time 
and effort into untangling this mess. The fix is unrelated to the new 
verification code.

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> pending
superseder:  -> SSL match_hostname fails for internationalized domain names

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35234] ssl module falls over with internationalized domain names

[Michael Casadevall]

New submission from Michael Casadevall :

Test case attached.

In Python 3.6, ssl tries to validate the hostname on its own, but fails to 
convert the SSL certificates hostname from IDNA back to UTF-8 and mismatches. 
Python 3.7 and master are unaffected since this got fixed by accident when 
validation was changed to depend on OpenSSL alone and not do it in python 
though the underlying match_hostname function is still bugged.

--
assignee: christian.heimes
components: SSL
files: ssl_test.py
messages: 329852
nosy: christian.heimes, mcasadevall
priority: normal
severity: normal
status: open
title: ssl module falls over with internationalized domain names
type: behavior
versions: Python 3.6
Added file: https://bugs.python.org/file47932/ssl_test.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10486] http.server doesn't set all CGI environment variables

[Pierre Quentel]

Pierre Quentel  added the comment:

The QUERY_STRING value is always set by the code at lines 1135-1137 of 
http.server:

for k in ('QUERY_STRING', 'REMOTE_HOST', 'CONTENT_LENGTH',
  'HTTP_USER_AGENT', 'HTTP_COOKIE', 'HTTP_REFERER'):
env.setdefault(k, "")

The RFC for CGI has not evolved since 2004, probably because the technology is 
stable, and also because other, more efficient protocols have been defined to 
avoid the "CGI overhead" (FastCGI for instance).

I think that http.server should only implement the "meta-variables" defined in 
RFC 3875:
- AUTH_TYPE  CONTENT_LENGTH CONTENT_TYPE  GATEWAY_INTERFACE PATH_INFO  
PATH_TRANSLATED QUERY_STRING  REMOTE_ADDR REMOTE_HOST  REMOTE_IDENT REMOTE_USER 
 REQUEST_METHOD SCRIPT_NAME  SERVER_NAME SERVER_PORT  SERVER_PROTOCOL 
SERVER_SOFTWARE. Some of these must always be set (eg QUERY_STRING, 
REQUEST_METHOD, SERVER_NAME...) but for other ones, there are conditions (for 
instance for CONTENT_LENGTH: "The server MUST set this meta-variable if and 
only if the request is accompanied by a message-body entity")
- "protocol-specific meta variables" : for HTTP, variables determined by the 
HTTP request headers such as HTTP_COOKIE (cf section 4.1.18.  Protocol-Specific 
Meta-Variables)

Other meta variables are probably beyond the scope of a module in the standard 
library.

In short, in my opinion the issue can be closed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29564] ResourceWarning: suggest to enable tracemalloc in the message

[STINNER Victor]

STINNER Victor  added the comment:


New changeset ae8878176d6dc1878d1e792ac4308ca1920b103b by Victor Stinner in 
branch '3.6':
bpo-29564:_PyMem_DumpTraceback() suggests enabling tracemalloc (GH-10510) 
(GH-10518)
https://github.com/python/cpython/commit/ae8878176d6dc1878d1e792ac4308ca1920b103b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35233] _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +9778
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35233] _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers

[STINNER Victor]

New submission from STINNER Victor :

_PyMainInterpreterConfig_Copy() doesn't copy the install_signal_handlers 
attribute and so config->install_signal_handlers is always -1.

Bug spotted by Francis Hart:
https://github.com/python/cpython/pull/10516

Attached PR fix the issue.

--
components: Interpreter Core
messages: 329849
nosy: vstinner
priority: normal
severity: normal
status: open
title: _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers
versions: Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29564] ResourceWarning: suggest to enable tracemalloc in the message

[STINNER Victor]

STINNER Victor  added the comment:


New changeset d57a3e5d03efdda6247c27bd682872edf399d88b by Victor Stinner in 
branch '3.7':
bpo-29564:_PyMem_DumpTraceback() suggests enabling tracemalloc (GH-10510) 
(GH-10517)
https://github.com/python/cpython/commit/d57a3e5d03efdda6247c27bd682872edf399d88b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35232] Add `module`/`qualname` arguments to make_dataclass for picklability

[Antony Lee]

New submission from Antony Lee :

Currently, dataclasses created by make_dataclass are not picklable, because 
their __module__ is set to "types".  It seems that this would be easily fixed 
by letting make_dataclass gain a `module` and a `qualname` kwarg, similarly to 
the Enum functional form 
(https://docs.python.org/3/library/enum.html#functional-api).

--
components: Library (Lib)
messages: 329847
nosy: Antony.Lee
priority: normal
severity: normal
status: open
title: Add `module`/`qualname` arguments to make_dataclass for picklability
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33930] Segfault with deep recursion into object().__dir__

[Gus Goulart]

Gus Goulart  added the comment:

Sorry, I forgot to include the trashcan API. I'll double check and update the 
issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29564] ResourceWarning: suggest to enable tracemalloc in the message

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9777

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33196] SEGV in mp.synchronize.Lock.__repr__ in spawn'ed proc if ctx mismatched

[Gus Goulart]

Gus Goulart  added the comment:

I've tested on Fedora 29 server and also failed to reproduce the error.

--
Added file: https://bugs.python.org/file47931/testing_on_fedora.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29564] ResourceWarning: suggest to enable tracemalloc in the message

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9776

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31354] Fixing a bug related to LTO only build

[Charalampos Stratakis]

Charalampos Stratakis  added the comment:

This change exports -flto in cflags.

You can check it with python3-config --cflags after this commit.

Which means that every c extension which will use those cflags will also 
utilize -flto which wasn't happening before, thus I'd say it's a regression.

--
nosy: +cstratak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29564] ResourceWarning: suggest to enable tracemalloc in the message

[miss-islington]

Change by miss-islington :


--
pull_requests: +9774

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29564] ResourceWarning: suggest to enable tracemalloc in the message

[miss-islington]

Change by miss-islington :


--
pull_requests: +9775

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29564] ResourceWarning: suggest to enable tracemalloc in the message

[STINNER Victor]

STINNER Victor  added the comment:


New changeset f966e5397ed8f5c42c185223fc9b4d750a678d02 by Victor Stinner in 
branch 'master':
bpo-29564:_PyMem_DumpTraceback() suggests enabling tracemalloc (GH-10510)
https://github.com/python/cpython/commit/f966e5397ed8f5c42c185223fc9b4d750a678d02


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35231] make install may not call ldconfig on GNU/Linux when using --enable-shared

[pmpp]

Change by pmpp :


--
title: make install may not call ldconfig on GNU/Linux -> make install may not 
call ldconfig on GNU/Linux when using --enable-shared

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35231] make install may not call ldconfig on GNU/Linux

[pmpp]

Change by pmpp :


--
title: make install may not call ldconfig on posix -> make install may not call 
ldconfig on GNU/Linux

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35059] Convert Py_INCREF() and PyObject_INIT() to inlined functions

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-35230: "Remove _Py_REF_DEBUG_COMMA".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35229] Deprecate _PyObject_GC_TRACK() in Python 3.6

[STINNER Victor]

Change by STINNER Victor :


--
components: +Interpreter Core
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35230] Remove _Py_REF_DEBUG_COMMA

[STINNER Victor]

STINNER Victor  added the comment:

This change is related to bpo-35059.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35231] make install may not call ldconfig on posix

[pmpp]

New submission from pmpp :

observed on ubuntu trusty x64 with release 3.7.1

at least when switching from "m" abi to "dm" abi sometimes libpython is not 
resolved in /usr/local/lib so maybe add "ldconfig" as a preventive mesure.

--
components: Installation
messages: 329840
nosy: pmpp
priority: normal
severity: normal
status: open
title: make install may not call ldconfig on posix
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32170] Contrary to documentation, ZipFile.extract does not extract timestamps or other metadata

[Sascha Silbe]

Change by Sascha Silbe :


--
nosy: +sascha_silbe

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29708] support reproducible Python builds

[Sascha Silbe]

Change by Sascha Silbe :


--
nosy: +sascha_silbe

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34033] distutils is not reproducible

[Sascha Silbe]

Change by Sascha Silbe :


--
nosy: +sascha_silbe

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33930] Segfault with deep recursion into object().__dir__

[Gus Goulart]

Change by Gus Goulart :


--
nosy: +taleinat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33930] Segfault with deep recursion into object().__dir__

[Gus Goulart]

Gus Goulart  added the comment:

ronaldoussoren, I have tested (Debian 9) substituting 
"_PyObject_GC_UNTRACK(m);" to "PyObject_GC_UnTrack(m);", and although the tests 
pass, I'm can still see the segfault error.

diff --git a/Objects/methodobject.c b/Objects/methodobject.c
index cfea8cf410..cb8e279c7d 100644
--- a/Objects/methodobject.c
+++ b/Objects/methodobject.c
@@ -84,7 +84,7 @@ PyCFunction_GetFlags(PyObject *op)
 static void
 meth_dealloc(PyCFunctionObject *m)
 {
-_PyObject_GC_UNTRACK(m);
+PyObject_GC_UnTrack(m);
 if (m->m_weakreflist != NULL) {
 PyObject_ClearWeakRefs((PyObject*) m);
 }

--
nosy: +gus.goulart

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17972] inspect module docs omits many functions

[Lysandros Nikolaou]

Lysandros Nikolaou  added the comment:

...but then we would have to include them in the docs... should be *but then we 
would have to include all the public functions in the docs*.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17972] inspect module docs omits many functions

[Lysandros Nikolaou]

Lysandros Nikolaou  added the comment:

I think this issue should be discussed once more. I am not sure it's worth 
defining __all__, but then we would have to include them in the docs. If you 
think __all__ should be defined instead, then I could go through the trouble of 
doing so. Could a core developer maybe offer their opinion here?

--
nosy: +lys.nikolaou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35229] Deprecate _PyObject_GC_TRACK() in Python 3.6

[miss-islington]

miss-islington  added the comment:


New changeset 60a6baef4005a32fb4cf45e15942ea18e59ecf19 by Miss Islington (bot) 
in branch '3.6':
bpo-35229: Deprecate _PyObject_GC_TRACK() (GH-10511)
https://github.com/python/cpython/commit/60a6baef4005a32fb4cf45e15942ea18e59ecf19


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35230] Remove _Py_REF_DEBUG_COMMA

[INADA Naoki]

Change by INADA Naoki :


--
keywords: +patch
pull_requests: +9773
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35230] Remove _Py_REF_DEBUG_COMMA

[INADA Naoki]

INADA Naoki  added the comment:

I meant 2aaf0c1.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34784] Heap-allocated StructSequences

[Petr Viktorin]

Petr Viktorin  added the comment:


New changeset 474eedfb3d1b6fecbd749f36bf4a987cf4a00b44 by Petr Viktorin (Eddie 
Elizondo) in branch 'master':
bpo-34784: Fix PyStructSequence_NewType with heap-allocated StructSequence 
(GH-9665)
https://github.com/python/cpython/commit/474eedfb3d1b6fecbd749f36bf4a987cf4a00b44


--
nosy: +petr.viktorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35230] Remove _Py_REF_DEBUG_COMMA

[INADA Naoki]

New submission from INADA Naoki :

After 2aaf0c, _Py_REF_DEBUG_COMMA in Include/object.h is used only in 
dictobject.c.  Let's remove it.

--
messages: 329833
nosy: inada.naoki
priority: normal
severity: normal
status: open
title: Remove _Py_REF_DEBUG_COMMA

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35229] Deprecate _PyObject_GC_TRACK() in Python 3.6

[miss-islington]

Change by miss-islington :


--
pull_requests: +9772

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35229] Deprecate _PyObject_GC_TRACK() in Python 3.6

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 55b4771d5d18d5ff7c095aa1ab750fe0c8bb5e67 by Victor Stinner in 
branch '3.7':
bpo-35229: Deprecate _PyObject_GC_TRACK() (GH-10511)
https://github.com/python/cpython/commit/55b4771d5d18d5ff7c095aa1ab750fe0c8bb5e67


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35229] Deprecate _PyObject_GC_TRACK() in Python 3.6

[STINNER Victor]

STINNER Victor  added the comment:

> These macros have been broken in Python 3.7 by Eric Snow's work on moving 
> internal APIs to Include/internal/ (...)

Oh, I forgot to mention that IMHO it's perfectly fine :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35229] Deprecate _PyObject_GC_TRACK() in Python 3.6

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +9771
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35081] Move internal headers to Include/internal/

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 1a6be91e6fd65ce9cb88c193db7e92ec6076 by Victor Stinner in 
branch 'master':
bpo-35081: Make some _PyGC macros internal (GH-10507)
https://github.com/python/cpython/commit/1a6be91e6fd65ce9cb88c193db7e92ec6076


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35229] Deprecate _PyObject_GC_TRACK() in Python 3.6

[STINNER Victor]

New submission from STINNER Victor :

The _PyObject_GC_TRACK() and _PyObject_GC_UNTRACK() macros are documented:

* https://docs.python.org/dev/c-api/gcsupport.html#c._PyObject_GC_TRACK
* https://docs.python.org/dev/c-api/gcsupport.html#c._PyObject_GC_UNTRACK

The documentation says "It should not be used for extension modules."

These macros have been broken in Python 3.7 by Eric Snow's work on moving 
internal APIs to Include/internal/: _PyObject_GC_TRACK() access 
_PyRuntime.gc.generation0, whereas accessing _PyRuntime requires to include 
Include/internal/pycore_pystate.h (header previously called 
Include/internal/pystate.h in Python 3.7) which is not installed by "make 
install".

My PR 10507 moved these macros to Include/internal/ to clarify that they must 
not be used outside CPython internals.

I suggest to deprecate them in Python 3.6: just add a note in Python 3.6 and 
3.7 documentation.

--
messages: 329829
nosy: vstinner
priority: normal
severity: normal
status: open
title: Deprecate _PyObject_GC_TRACK() in Python 3.6
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35195] [Windows] Python 3.7 initializes LC_CTYPE locale at startup, causing performance issue on msvcrt isdigit()

[STINNER Victor]

Change by STINNER Victor :


--
title: Pandas read_csv() is 3.5X Slower on Python 3.7.1 vs Python 3.6.7 & 3.5.2 
On Windows 10 -> [Windows] Python 3.7 initializes LC_CTYPE locale at startup, 
causing performance issue on msvcrt isdigit()

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35195] Pandas read_csv() is 3.5X Slower on Python 3.7.1 vs Python 3.6.7 & 3.5.2 On Windows 10

[STINNER Victor]

STINNER Victor  added the comment:

> Yes, that slows down Python 3.7.0a3 to the 3.7.0a4 level.

Ok, so something calls setlocale(LC_CTYPE, "") or setlocale(LC_ALL, "") in 
Python 3.7.0. I'm not interested to dig the Git history. It doesn't really 
matter at this point.

Can you try to get the current LC_CTYPE locale on Python 3.6 or 3.7.0a3? 
Example:

$ python3 -c 'import _locale; print(_locale.setlocale(_locale.LC_CTYPE, None))'
fr_FR.utf8

Or using the locale module (it should give the same result):

$ python3 -c 'import locale; print(locale.setlocale(locale.LC_CTYPE, None))'
fr_FR.utf8

Can also also try on 3.7.0a4 and newer (ex: Python 3.7.1): 
locale.setlocale(locale.LC_ALL, "C"), to see it does workaround your 
performance issue? I don't recall if "C" or "POSIX" locales are supported on 
Windows.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35226] mock.call equality surprisingly broken

[Michael Foord]

Michael Foord  added the comment:

Parents comparing upwards sounds like the right (and simple) fix. Not breaking 
the tuple tests would be good. I'm happy for Chris to produce the patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29564] ResourceWarning: suggest to enable tracemalloc in the message

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +9770

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35226] mock.call equality surprisingly broken

[Chris Withers]

Chris Withers  added the comment:

I'm chatting with Michael already (on Facebook of all places), and I use this 
feature heavily and deeply, being one of the people who originally requested it.

I'm both happy and capable of seeing this through, so no need for anyone else 
to dive in :-)

Hopefully Michael will follow up when he gets a chance...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29564] ResourceWarning: suggest to enable tracemalloc in the message

[miss-islington]

Change by miss-islington :


--
pull_requests: +9769

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com