[issue40280] Consider supporting emscripten/webassembly as a build target

[Simon Biggs]

New submission from Simon Biggs :

Since asm.js came on the scene, and now Web Assembly people have created 
CPython patches to support building CPython with emscripten. See:

* https://github.com/PeachPy/EmCPython -- Python 2.7
* https://github.com/dgym/cpython-emscripten/tree/master/3.5.2/patches -- 
Python 3.5.2
* https://github.com/iodide-project/pyodide/tree/master/cpython/patches -- 
Python 3.7.4

To ease the compiling of CPython with emscripten it would be helpful if patches 
that achieved these ends for the compiling to Web Assembly with emscripten were 
built into the upstream source repository itself.

If web assembly were to became a supported compilation target of the upstream 
CPython repository this would significantly reduce the friction of allowing 
CPython, and the latest CPython, to become a language readily usable within the 
browser.

Cheers,
Simon

--
components: Build
messages: 366369
nosy: Simon Biggs
priority: normal
severity: normal
status: open
title: Consider supporting emscripten/webassembly as a build target
type: enhancement
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40257] Improve the use of __doc__ in pydoc

[Vedran Čačić]

Vedran Čačić  added the comment:

Ok, I get what you're saying. But if someone writes

class B(A):
  # no docstring at all
  ...

help(B)

they'll still get other elements of current help? Particularly, "Methods 
inherited from A" (with their docstrings)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40257] Improve the use of __doc__ in pydoc

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Yes, of course. And if it overrides some methods, but do not specify doctrings 
for new methods, they will be inherited from the parent class.

class A:
"""Base class"""
def foo(self): """Some docstring"""
def bar(self): """Other docstring"""

class B(A):
def foo(self): pass

help(B)

Help on class B in module __main__:

class B(A)
 |  Method resolution order:
 |  B
 |  A
 |  builtins.object
 |  
 |  Methods defined here:
 |  
 |  foo(self)
 |  Some docstring
 |  
 |  --
 |  Methods inherited from A:
 |  
 |  bar(self)
 |  Other docstring
 |  
...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

[Torsten Landschoff]

Torsten Landschoff  added the comment:

I intended to file a bug because of the shutdown problem but found out there is 
this ticket already.

In our application we sometimes run into this problem that you can't exit 
because of a hanging TCP connection used inside a ThreadPoolExecutor task. 

executor.shutdown(wait=False) does not help - the process still hangs.

Another problem was inside our tests were a bug caused a hanging task inside a 
thread pool executor and the test runner (pytest) would not terminate because 
of this.

I actually went so far to drop the thread reference inside the 
concurrent.futures.thread module. :-(

A documented way to drop a hanging executor thread would be very much 
appreciated.

--
nosy: +torsten

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39943] Meta: Clean up various issues in C internals

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Yes, it is a consequence of PR 19345. But it looks like a compiler bug. It 
complains about implicit conversion from `const void **` to `void *` in 
memcpy() and PyMem_Del().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29255] selects.KqueueSelector behaves incorrectly when no fds are registered

[Russell Davis]

Change by Russell Davis :


--
keywords: +patch
pull_requests: +18860
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19508

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25680] Selector.select() hangs when there is nothing to select

[Russell Davis]

Change by Russell Davis :


--
keywords: +patch
pull_requests: +18861
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19508

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40280] Consider supporting emscripten/webassembly as a build target

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Do you want to provide a pull request?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40280] Consider supporting emscripten/webassembly as a build target

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40280] Consider supporting emscripten/webassembly as a build target

[pmp-p]

pmp-p  added the comment:

you can add 
* https://github.com/pmp-p/pydk/tree/master/sources.em/Python-3.8.0b4.patchset 
-- Python 3.8.x 

(wasm not asm.js, clang-10+ required)

demo https://pmp-p.github.io/python-next/test.html

CPython can already run in the browser with very little patching, but major 
issues are :
 
 - asyncify'ing the whole wasm VM to have pre-emption over cPython's one to 
prevent blocking I/O slows down things *a lot* (10x)
=> (very?) bad user experience.

 - the size of vm + stdlib  ~ 30 MiB and wasm compilation time.
=> bad user experience on first load or slow connexion.

 - the lack of threading in wasm MinimumViableProduct specification (but this 
is the browser standard for now), that leads to rewrite bits of stdlib ( like 
eg asyncio module )
=> adding more maintenance burden on stdlib (!)


i tested them all and my personnal opinion is : I can see no use case that 
would favour "stock" cPython wasm versus a blazing fast MicroPytho (or pycopy) 
wasm flavour or supercharged full stack pyodide.

--
nosy: +pmpp

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40257] Improve the use of __doc__ in pydoc

[Vedran Čačić]

Vedran Čačić  added the comment:

Then I'm fine with it. Thanks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[hai shi]

Change by hai shi :


--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40271] Allow shell like paths in

[Gavin D'souza]

Gavin D'souza  added the comment:

@serhiy.storchaka That makes perfect sense. Could we do something to add a 
parameter perhaps, to evaluate literal paths to not bread existing code? 
although this isn't "needed" but it'd be neat to handle this internally

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40268] Reorganize pycore_pystate.h header

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 4a3fe0835310643193ea45529ab0fb45c5f8f2fd by Victor Stinner in 
branch 'master':
bpo-40268: Include explicitly pycore_interp.h (GH-19505)
https://github.com/python/cpython/commit/4a3fe0835310643193ea45529ab0fb45c5f8f2fd


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40280] Consider supporting emscripten/webassembly as a build target

[Simon Biggs]

Simon Biggs  added the comment:

Hi pmp-p and Serhiy,

I'd be more than happy to attempt a pull request, but I imagine a change
such as this needs to be discussed first, trying not to "rush to make a
patch" (https://www.youtube.com/watch?v=voXVTjwnn-U&feature=youtu.be&t=2546).
Also, I doubt I will do a good job of it... but I am more than happy to try.

A note regarding "supercharged full stack pyodide", potentially without
efforts such as upstreaming into CPython and emscripten the relevant
patches, that supercharged full stack may just unfortunately stagnate. See
https://github.com/iodide-project/pyodide/issues/635#issuecomment-613408912

With respect to blocking when running Python as WASM, I have found running
the WebAssembly CPython within a webworker and signalling data back and
forth causes there to be no UI issues. It ends up being quite a neat set
up. Main down side right now however is the set up is currently going
stale, hence me believing reaching out like this is in the best interests
of Python going forward.

Cheers,
Simon

On Tue, 14 Apr 2020 at 18:58, pmp-p  wrote:

>
> pmp-p  added the comment:
>
> you can add
> *
> https://github.com/pmp-p/pydk/tree/master/sources.em/Python-3.8.0b4.patchset
> -- Python 3.8.x
>
> (wasm not asm.js, clang-10+ required)
>
> demo https://pmp-p.github.io/python-next/test.html
>
> CPython can already run in the browser with very little patching, but
> major issues are :
>
>  - asyncify'ing the whole wasm VM to have pre-emption over cPython's one
> to prevent blocking I/O slows down things *a lot* (10x)
> => (very?) bad user experience.
>
>  - the size of vm + stdlib  ~ 30 MiB and wasm compilation time.
> => bad user experience on first load or slow connexion.
>
>  - the lack of threading in wasm MinimumViableProduct specification (but
> this is the browser standard for now), that leads to rewrite bits of stdlib
> ( like eg asyncio module )
> => adding more maintenance burden on stdlib (!)
>
>
> i tested them all and my personnal opinion is : I can see no use case that
> would favour "stock" cPython wasm versus a blazing fast MicroPytho (or
> pycopy) wasm flavour or supercharged full stack pyodide.
>
> --
> nosy: +pmpp
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40244] AIX: build: _PyObject_GC_TRACK Asstertion failure

[Michael Felt]

Michael Felt  added the comment:

With the print statements - it does not crash:

 ./python -E -S -m sysconfig --generate-posix-vars ; if test $?
-ne 0 ; then  echo "generate-posix-vars failed" ;  rm -f
./pybuilddir.txt ;  exit 1 ;  fi
Objects/genobject.c:122:537318120
Objects/genobject.c:126:0
Objects/genobject.c:131:0
Objects/genobject.c:135:537318120
Objects/genobject.c:122:805942488
Objects/genobject.c:126:0
Objects/genobject.c:131:0
Objects/genobject.c:135:537318120
...

diff --git a/Objects/genobject.c b/Objects/genobject.c
index d3455f8..b8287a3 100644
--- a/Objects/genobject.c
+++ b/Objects/genobject.c
@@ -117,14 +117,23 @@ exc_state_clear(_PyErr_StackItem *exc_state)
 static void
 gen_dealloc(PyGenObject *gen)
 {
+#include 
 PyObject *self = (PyObject *) gen;
+    fprintf(stderr,"%s:%d:%ld\n", __FILE__,__LINE__,
_Py_AS_GC(self)->_gc_next);
+    fflush(stderr);

 _PyObject_GC_UNTRACK(gen);
+    fprintf(stderr,"%s:%d:%ld\n", __FILE__,__LINE__,
_Py_AS_GC(self)->_gc_next);
+    fflush(stderr);

 if (gen->gi_weakreflist != NULL)
 PyObject_ClearWeakRefs(self);
+    fprintf(stderr,"%s:%d:%ld\n", __FILE__,__LINE__,
_Py_AS_GC(self)->_gc_next);
+    fflush(stderr);

 _PyObject_GC_TRACK(self);
+    fprintf(stderr,"%s:%d:%ld\n", __FILE__,__LINE__,
_Py_AS_GC(self)->_gc_next);
+    fflush(stderr);

 if (PyObject_CallFinalizerFromDealloc(self))
 return; /* resurrected.  :( */
$

On 13/04/2020 23:19, Batuhan Taskaya wrote:
> Batuhan Taskaya  added the comment:
>
> @Michael.Felt can you just insert some prints between these 3 to see what is 
> going on (and re-compile)
>
> static void
> gen_dealloc(PyGenObject *gen)
> {
> PyObject *self = (PyObject *) gen;
> 
> _PyObject_GC_UNTRACK(gen);
> 
> if (gen->gi_weakreflist != NULL)
> PyObject_ClearWeakRefs(self);
> 
> _PyObject_GC_TRACK(self);
>
>
> something like this should work: printf("%ld\n", _Py_AS_GC(self)->_gc_next);
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40244] AIX: build: _PyObject_GC_TRACK Asstertion failure

[Michael Felt]

Michael Felt  added the comment:

Also tried this:

 ./python -E -S -m sysconfig --generate-posix-vars ; if test $?
-ne 0 ; then  echo "generate-posix-vars failed" ;  rm -f
./pybuilddir.txt ;  exit 1 ;  fi
Objects/genobject.c:127: _PyObject_GC_TRACK: Assertion "!(((PyGC_Head
*)(op)-1)->_gc_next != 0)" failed: object already tracked by the garbage
collector
Enable tracemalloc to get the memory block allocation traceback

object address  : 30084150
object refcount : 0
object type : 20013718
object type name: generator
object repr : 

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: core initialized

Current thread 0x0001 (most recent call first):
  File "", line 1593 in _setup
  File "", line 1634 in _install
  File "", line 1189 in
_install_external_importers
/bin/sh: 17302202 IOT/Abort trap(coredump)
make: 1254-004 The error code from the last command is 134.

Stop.
$ git diff
diff --git a/Objects/genobject.c b/Objects/genobject.c
index d3455f8..e783636 100644
--- a/Objects/genobject.c
+++ b/Objects/genobject.c
@@ -119,7 +119,7 @@ gen_dealloc(PyGenObject *gen)
 {
 PyObject *self = (PyObject *) gen;

-    _PyObject_GC_UNTRACK(gen);
+    _PyObject_GC_UNTRACK(self);

 if (gen->gi_weakreflist != NULL)
 PyObject_ClearWeakRefs(self);
$

On 14/04/2020 00:18, Batuhan Taskaya wrote:
> Batuhan Taskaya  added the comment:
>
> By the noticed something that looks kind of weird, not sure if it is 
> intentional or not but shouldn't this be self instead of gen
>
> static void
> gen_dealloc(PyGenObject *gen)
> {
> PyObject *self = (PyObject *) gen;
> 
> _PyObject_GC_UNTRACK(gen);
> 
> if (gen->gi_weakreflist != NULL)
> PyObject_ClearWeakRefs(self);
> _PyObject_GC_TRACK(self);
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40268] Reorganize pycore_pystate.h header

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18862
pull_request: https://github.com/python/cpython/pull/19509

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40244] AIX: build: _PyObject_GC_TRACK Asstertion failure

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

> With the print statements - it does not crash:

I think this isn't directly relevant with prints but about re-compiling? (just 
guessing). Because I experienced when I compile python for the first time on a 
clean AIX environment, all extension modules failed to build. When I recompiled 
(with keeping all artifacts from previous build) some of them successfully got 
compiled. When I try to compile again, most of them successfully compiled. I'm 
sorry but I don't know why this happens or how to solve it. We can always 
revert that change but I guess that isn't the real problem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40244] AIX: build: _PyObject_GC_TRACK Asstertion failure

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

> Also tried this:

Thanks for it, that was just something that I slightly suspected but expected 
to see it crash.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40281] Add pathlib.PurePath.as_uri()

[Antoine Pitrou]

New submission from Antoine Pitrou :

"file" scheme URIs are normalized:
https://tools.ietf.org/html/rfc8089

Therefore it would be possible to provide a PurePath method that returns a 
"file" scheme URI corresponding to the given path.

--
components: Library (Lib)
messages: 366384
nosy: pitrou
priority: normal
severity: normal
status: open
title: Add pathlib.PurePath.as_uri()
type: enhancement
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40281] Add pathlib.PurePath.as_uri()

[Antoine Pitrou]

Change by Antoine Pitrou :


--
nosy: +barneygale

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40268] Reorganize pycore_pystate.h header

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 81a7be3fa22c983209cc0ffb3537b92b0370f83c by Victor Stinner in 
branch 'master':
bpo-40268: Rename _PyInterpreterState_GET_UNSAFE() (GH-19509)
https://github.com/python/cpython/commit/81a7be3fa22c983209cc0ffb3537b92b0370f83c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40221] Use new _at_fork_reinit() lock method in multiprocessing

[Dong-hee Na]

Dong-hee Na  added the comment:


New changeset e1945307d36849f8be8937144cf3dd6ebab6274c by Dong-hee Na in branch 
'master':
bpo-40221: Update multiprocessing to use _at_fork_reinit (GH-19477)
https://github.com/python/cpython/commit/e1945307d36849f8be8937144cf3dd6ebab6274c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Steve Dower]

Steve Dower  added the comment:

>> This isn't actually about removing immortal objects, but removing *mutable*
>> objects that would be shared between subinterpreters. Making some objects
>> immortal, such as interned strings or stateless singletons, would actually
>> benefit this work, as they could then be safely shared between
>> subinterpreters.
> 
> From what I understood, we simply cannot share any object (mutable or not)
> between two subinterpreters. Otherwise, we will need to put a lock on all
> Py_INCREF/Py_DECREF operation which would kill performances of running
> multiple interpreters in parallel. Join bpo-39511 discussion.

That thread consists of everyone else telling you what I've just told you. Not 
sure that me telling you as well is going to help ;)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39511] [subinterpreters] Per-interpreter singletons (None, True, False, etc.)

[Steve Dower]

Change by Steve Dower :


--
nosy: +steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40268] Reorganize pycore_pystate.h header

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18863
pull_request: https://github.com/python/cpython/pull/19510

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39947] [C API] Make the PyThreadState structure opaque (move it to the internal C API)

[STINNER Victor]

Change by STINNER Victor :


--
title: Make the PyThreadState structure opaque (move it to the internal C API) 
-> [C API] Make the PyThreadState structure opaque (move it to the internal C 
API)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40281] Add pathlib.PurePath.as_uri()

[Dong-hee Na]

Dong-hee Na  added the comment:

+1 to add this :)

--
nosy: +corona10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> I run the pyperformance test suite with PGO + LTO + full cpu isolation in the 
> speed.python.org machine and these were the results:

Be mindful that synthetic benchmarks are probably a gentle case for branch 
prediction.  Real-world performance on irregular workloads might be worse still 
(or perhaps better, who knows :-)).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Antoine Pitrou]

Antoine Pitrou  added the comment:

As a separate discussion, I would be interested to know whether the original 
use case (Eddie's) could be satisfied differently.  It probably doesn't belong 
to this issue, though.

(Eddie, if you want to discuss this, feel free to e-mail me privately. I think 
Davin Potts - co-maintainer of multiprocessing who recently added some 
facilities around shared memory - might be interested as well.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39943] Meta: Clean up various issues in C internals

[Andy Lester]

Andy Lester  added the comment:

I remember coming across a similar error from GCC about casting from a const 
double pointer to a single pointer void and it said (I believe) something about 
having to have each cast having to be valid.  I think it was implying something 
like that if you have 

const void **p

you have to cast that as

(void *)(const void *)p

I will see if I can find that message and/or I can find out more about this 
cast problem in the Windows compiler.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40221] Use new _at_fork_reinit() lock method in multiprocessing

[Dong-hee Na]

Change by Dong-hee Na :


--
pull_requests: +18864
pull_request: https://github.com/python/cpython/pull/19511

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40282] random.getrandbits(0) should succeed

[Antoine Pitrou]

New submission from Antoine Pitrou :

When creating variable-sized random binary strings with random.getrandbits(), 
you currently have to special case when the number of bytes is 0, because 
otherwise getrandbits() raises:

  ValueError: number of bits must be greater than zero

It seems like it wouldn't hurt to simply return 0 in that case.

The actual snippet looks something like:

  random.getrandombits(nbytes * 8).to_bytes(nbytes, 'little')

--
messages: 366392
nosy: mark.dickinson, pitrou, rhettinger, steven.daprano
priority: normal
severity: normal
status: open
title: random.getrandbits(0) should succeed
type: enhancement
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40282] random.getrandbits(0) should succeed

[STINNER Victor]

STINNER Victor  added the comment:

How random would be the 0 returned by getrandbits(0)? :-)

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40282] random.getrandbits(0) should succeed

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I think you know the answer to your question ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40268] Reorganize pycore_pystate.h header

[STINNER Victor]

STINNER Victor  added the comment:


New changeset e5014be0497d06d78343623588a80f491a6f7b74 by Victor Stinner in 
branch 'master':
bpo-40268: Remove a few pycore_pystate.h includes (GH-19510)
https://github.com/python/cpython/commit/e5014be0497d06d78343623588a80f491a6f7b74


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39522] AST Unparser with unicode kinded constants

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
keywords: +patch
pull_requests: +18865
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19512

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40273] mappingproxy isn't reversible

[Zackery Spytz]

Change by Zackery Spytz :


--
keywords: +patch
nosy: +ZackerySpytz
nosy_count: 2.0 -> 3.0
pull_requests: +18866
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19513

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40268] Reorganize pycore_pystate.h header

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18867
pull_request: https://github.com/python/cpython/pull/19515

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40232] PyOS_AfterFork_Child() should use _PyThread_at_fork_reinit()

[Dong-hee Na]

Dong-hee Na  added the comment:


New changeset 62f75fe3dd138f72303814d27183aa469eefcca6 by Dong-hee Na in branch 
'master':
bpo-40232: Update PyOS_AfterFork_Child() to use _PyThread_at_fork_reinit() 
(GH-19450)
https://github.com/python/cpython/commit/62f75fe3dd138f72303814d27183aa469eefcca6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40232] PyOS_AfterFork_Child() should use _PyThread_at_fork_reinit()

[Dong-hee Na]

Dong-hee Na  added the comment:

This issue is now solved.

Thanks, Victor for the suggestion! :)

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40268] Reorganize pycore_pystate.h header

[STINNER Victor]

STINNER Victor  added the comment:


New changeset e560f90602870601945ea7a4f7770827608817d2 by Victor Stinner in 
branch 'master':
bpo-40268: Move struct _gc_runtime_state to pycore_gc.h (GH-19515)
https://github.com/python/cpython/commit/e560f90602870601945ea7a4f7770827608817d2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40221] Use new _at_fork_reinit() lock method in multiprocessing

[Dong-hee Na]

Dong-hee Na  added the comment:


New changeset a5900ecf9f22e65bef489633692e9ea6941379c5 by Dong-hee Na in branch 
'master':
bpo-40221: Update multiprocessing to use _at_fork_reinit (GH-19511)
https://github.com/python/cpython/commit/a5900ecf9f22e65bef489633692e9ea6941379c5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40221] Use new _at_fork_reinit() lock method in multiprocessing

[Dong-hee Na]

Dong-hee Na  added the comment:

Looks like we update all the methods which are related to _after_fork
under Lib/multiprocessing.

I am now closing this issue.
Thanks for the suggestion, Victor :)

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40283] Documentation of turtle.circle()

[guchao]

New submission from guchao :

refer to circle() in the url: 
https://docs.python.org/2/library/turtle.html#turtle.circle

[current]
... Draw the arc in counterclockwise direction if radius is positive, otherwise 
in clockwise direction. 

[suggestion]
... Draw the arc in counterclockwise direction if extent is positive, otherwise 
in clockwise direction. 

[explanation]
It should be "extent" , instead of "radius", that affects the direction.

--
assignee: docs@python
components: Documentation
messages: 366401
nosy: docs@python, guchao
priority: normal
severity: normal
status: open
title: Documentation of turtle.circle()
type: resource usage

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Carl Meyer]

Carl Meyer  added the comment:

> Anything that is touched by the immortal object will be leaked. This can also 
> happen in obscure ways if reference cycles are created.

I think this is simply expected behavior if you choose to create immortal 
objects, and not really an issue. How could you have an immortal object that 
doesn't keep its strong references alive?

> this does not fully cover all cases as objects that become tracked by the GC 
> after they are modified (for instance, dicts and tuples that only contain 
> immutable objects). Those objects will still participate in reference 
> counting after they start to be tracked.

I think the last sentence here is not quite right. An immortalized object will 
never start participating in reference counting again after it is immortalized.

There are two cases. If at the time of calling `immortalize_heap()` you have a 
non-GC-tracked object that is also not reachable from any GC-tracked container, 
then it will not be immortalized at all, so will be unaffected. This is a side 
effect of the PR using the GC to find objects to immortalize.

If the non-GC-tracked object is reachable from a GC-tracked object (I believe 
this is by far the more common case), then it will be immortalized. If it later 
becomes GC-tracked, it will start participating in GC (but the immortal bit 
causes it to appear to the GC to have a very high reference count, so GC will 
never collect it or any cycle it is part of), but that will not cause it to 
start participating in reference counting again.

> if immortal objects are handed to extension modules compiled with the other 
> version of the macros, the reference count can be corrupted

I think the word "corrupted" makes this sound worse than it is in practice. 
What happens is just that the object is still effectively immortal (because the 
immortal bit is a very high bit), but the copy-on-write benefit is lost for the 
objects touched by old extensions.

> 1.17x slower on logging_silent or unpickle_pure_python is a very expensive 
> price

Agreed. It seems the only way this makes sense is under an ifdef and off by 
default. CPython does a lot of that for debug features; this might be the first 
case of doing it for a performance feature?

> I would be more interested by an experiment to move ob_refcnt outside 
> PyObject to solve the Copy-on-Write issue

It would certainly be interesting to see results of such an experiment. We 
haven't tried that for refcounts, but in the work that led to `gc.freeze()` we 
did try relocating the GC header to a side location. We abandoned that because 
the memory overhead of adding a single indirection pointer to every PyObject 
was too large to even consider the option further. I suspect that this memory 
overhead issue and/or likely cache locality problems will make moving refcounts 
outside PyObject look much worse for performance than this immortal-instances 
patch does.

--
nosy: +carljm

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Carl Meyer]

Carl Meyer  added the comment:

> An immortalized object will never start participating in reference counting 
> again after it is immortalized.

Well, "passed to an extension compiled with no-immortal headers" is an 
exception to this.

But for the "not GC tracked but later becomes GC tracked" case, it will not 
re-enter reference counting, only the GC.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Carl Meyer]

Carl Meyer  added the comment:

> This may break the garbage collector algorithm that relies on the balance 
> between strong references between objects and its reference count to do the 
> calculation of the isolated cycles.

I don't think it really breaks anything. What happens is that the immortal 
object appears to the GC to have a very large reference count, even after 
adjusting for within-cycle references. So cycles including an immortal object 
are always kept alive, which is exactly the behavior one should expect from an 
immortal object.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40244] AIX: build: _PyObject_GC_TRACK Asstertion failure

[Michael Felt]

Michael Felt  added the comment:

On 14/04/2020 14:54, Batuhan Taskaya wrote:
> Batuhan Taskaya  added the comment:
>
>> With the print statements - it does not crash:
> I think this isn't directly relevant with prints but about re-compiling? 
> (just guessing).

I only recompiled the one .c file. With that one file re-compiled -
wqith fprintf statements it succeeds, restore the original .c file (git
checkout -- Objects/whatever.c; make - it fails.

Tomorrow I'll search for the option(s) needed to get (complete) assembly
code listing and try to see (and understand) the difference between what
xlc-v13 and xlc-v16 makes. And, what I shall also test - is to recompile
only this one file using xlc-v13 and see if the make then proceeds normally.

>  Because I experienced when I compile python for the first time on a clean 
> AIX environment, all extension modules failed to build.

I only see this happen (on occasion) when I use make -j4 (or greater) -
and I have seen it happen to a lessor extent with -j2. On the subsequent
passes, whatever it is that setup.py (guessing) really needs is now
available - and the modules build as expected.

This is also why, for the last 4 years I have used my own personal
server - where I control everything (mainly NO other party OSS packaged
software and their artifacts).

>  When I recompiled (with keeping all artifacts from previous build) some of 
> them successfully got compiled. When I try to compile again, most of them 
> successfully compiled. I'm sorry but I don't know why this happens or how to 
> solve it.
why - I do not understand the finer details either, but my guess is that
it is related to linking. I am nearly "amazed" - yet happy - that the
PPC64 AIX 3.X compile succeeds - but acknowledge the 22K+ lines of
warnings is related to the over parallelization of the linking.
> We can always revert that change but I guess that isn't the real problem.

No. I do not think it is the real problem either. And I do not know
compiler behavior well enough. Actually, considering the setting is
still -O0 (aka no optimization) I am surprised it has any effect. if I
understood correctly "no return" is intended to help the optimizer make
"informed" decisions.

As Victor commented earlier - very much looking like a compiler bug.
That said, still do not know what to say/write to software support as a
complaint.

>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Dino Viehland]

Dino Viehland  added the comment:

I think there's other cases of performance related features being hidden under 
an ifdef.  Computed gotos show up that way, although probably more because it's 
a compiler extension that's not supported everywhere.  Pymalloc is also very 
similar in that it implies an ABI change as well.

I wonder if it would be worth it to introduce an ABI flag for this as well?  On 
the one hand is it a slightly different contract, on the other hand using 
extensions that don't support the immortalization actually work just fine.

--
nosy: +dino.viehland

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> I think this is simply expected behavior if you choose to create immortal 
> objects, and not really an issue. How could you have an immortal object that 
> doesn't keep its strong references alive?

I think I didn't express myself very well: I am saying that I perceive this as 
a problem because when you immortalize the heap at a given time you will also 
immortalize any strong reference that these objects made afterwards. The 
immortal property will be "infecting" other objects as strong references are 
created.

> I think the last sentence here is not quite right. An immortalized object 
> will never start participating in reference counting again after it is 
> immortalized.

Yeah, but the objects I mention will not be immortalized. These are objects 
that were not tracked when the immortalization is done and are not reachable 
from tracked objects at the moment and become tracked afterwards (for instance, 
some dictionary that only had immutable objects when the immortalization was 
done). And this is always very impacting: a single object that participates in 
refcounting will copy an entire page of memory. Although I agree this is a rare 
thing is a possible thing that can happen.

> I think the word "corrupted" makes this sound worse than it is in practice. 
> What happens is just that the object is still effectively immortal (because 
> the immortal bit is a very high bit), but the copy-on-write benefit is lost 
> for the objects touched by old extensions.

Yeah, I agree that corrupted may sound a bit more dramatic than it should but I 
think this is still a problem because when it happens (as mentioned before) it 
affects entire pages and not the single object that we consider.

> But for the "not GC tracked but later becomes GC tracked" case, it will not 
> re-enter reference counting, only the GC.

But if said objects (isolated and untracked before and now tracked) acquire 
strong references to immortal objects, those objects will be visited when the 
gc starts calculating the isolated cycles and that requires a balanced 
reference count to work.

--
nosy:  -dino.viehland

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Steve Dower]

Steve Dower  added the comment:

There hasn't been much said about which kind of objects would be immortal. 
Could this be a creation-time option? Or a type object (similar to a no-op 
tp_dealloc)?

If it's something that is known when the object is created, then some of the 
"infection" concern can be reduced - but if you wanted to arbitrarily make 
arbitrary objects immortal dynamically then it's a different matter.

Another benefit of knowing at creation time is that immortal objects could be 
allocated to a different page (or potentially even to read-only shared memory, 
if that was appropriate).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Steve Dower]

Change by Steve Dower :


--
nosy: +dino.viehland

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37531] Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 67b8a1f0f0f78ec38b8626fa9f5b2f5a55c17e15 by Victor Stinner in 
branch '3.8':
[3.8] Update libregrtest from master (GH-19516)
https://github.com/python/cpython/commit/67b8a1f0f0f78ec38b8626fa9f5b2f5a55c17e15


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37957] Allow regrtest to receive a file with test (and subtests) to ignore

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 67b8a1f0f0f78ec38b8626fa9f5b2f5a55c17e15 by Victor Stinner in 
branch '3.8':
[3.8] Update libregrtest from master (GH-19516)
https://github.com/python/cpython/commit/67b8a1f0f0f78ec38b8626fa9f5b2f5a55c17e15


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38502] regrtest: use process groups

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 67b8a1f0f0f78ec38b8626fa9f5b2f5a55c17e15 by Victor Stinner in 
branch '3.8':
[3.8] Update libregrtest from master (GH-19516)
https://github.com/python/cpython/commit/67b8a1f0f0f78ec38b8626fa9f5b2f5a55c17e15


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> There hasn't been much said about which kind of objects would be immortal.

The current patch makes *all objects that are tracked by the gc* immortal, 
independently of their type (unless I am missing something).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> all objects that are tracked by the gc

also, all the objects that are reachable from objects tracked by the gc.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40260] modulefinder traceback regression starting on Windows

[Barry Alan Scott]

Barry Alan Scott  added the comment:

io.open_code() used in the patch.

--
components:  -Windows

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

[Antoine Pitrou]

Change by Antoine Pitrou :


--
nosy: +aeros

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I don't think there's much ThreadPoolExecutor can do.  If you drop the 
references to the threads, they still exist and they still be waited upon at 
interpreter exit.

The solution is for you to avoid having hanging threads.  In the particular 
case of TCP connections, I'd recommend using a dedicated framework such as 
asyncio (or Twisted, Tornado, etc.) instead of home-baked networking code.

Also, note that Python sockets have a feature called *timeouts*.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Marking everything as immortal/eternal after you've loaded your common code & 
data but before you do your forking is thenorm for this kind of serving system.

You already presumably have a defined lifetime for the processes so they'll 
likely be set to die within N hours or days.  Memory leaks of too many things 
persisting is a non-issue in this kind of system.

The alternative of trying to pick and choose exactly what (and anything they 
reference) sticks around is a maintenance nightmare exercise in futility.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40170] [C API] Make PyTypeObject structure an opaque structure in the public C API

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 675d9a3d7afc767a2818c84da7ba4bf4181dcf26 by Hai Shi in branch 
'master':
bpo-40170: Convert PyObject_IS_GC() macro to a function (GH-19464)
https://github.com/python/cpython/commit/675d9a3d7afc767a2818c84da7ba4bf4181dcf26


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32033] The pwd module implementation incorrectly sets some attributes to None

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 96515e9f6785328c52ebc5d4ce60e0087a9adc2d by Zackery Spytz in 
branch 'master':
bpo-32033: Fix test_pwd failures on Android (GH-19502)
https://github.com/python/cpython/commit/96515e9f6785328c52ebc5d4ce60e0087a9adc2d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32033] The pwd module implementation incorrectly sets some attributes to None

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 4.0 -> 5.0
pull_requests: +18868
pull_request: https://github.com/python/cpython/pull/19518

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

[Hrvoje Nikšić]

Hrvoje Nikšić  added the comment:

> I don't think there's much ThreadPoolExecutor can do.  If you drop the 
> references to the threads, they still exist and they still be waited upon at 
> interpreter exit.

ThreadPoolExecutor introduces additional waiting of its own, and it is this 
wait the PR adds an option to disable.

> The solution is for you to avoid having hanging threads.

In some cases that is not possible. For example, the file IO on network file 
systems can take an arbitrary amount of time, and there is no way to implement 
a timeout. DNS lookups have been notorious for not supporting timeouts. Many 
third-party libraries, such as database drivers, still don't support timeouts.

This is a real issue that bit many users in production systems, it's not about 
a toy program using "home-baked networking code".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32033] The pwd module implementation incorrectly sets some attributes to None

[miss-islington]

Change by miss-islington :


--
pull_requests: +18869
pull_request: https://github.com/python/cpython/pull/19519

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32033] The pwd module implementation incorrectly sets some attributes to None

[STINNER Victor]

STINNER Victor  added the comment:

It's now fixed in master and backports to 3.7 and 3.8 will be merged as soon as 
the CI pass.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I think there's a misunderstanding: "wait_at_exit" will make the *executor* 
forget about the threads, but Python itself still knows about them, and it 
waits for them to end at interpreter shutdown.

These threads were daemon threads in 3.8, so your patch indeed works there, but 
we've made them non-daemon in 3.9, for two reasons:
1) daemon threads are fragile and can crash the interpreter at shutdown
2) they are not supported on subinterpreters

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> But if said objects (isolated and untracked before and now tracked) acquire 
> strong references to immortal objects, those objects will be visited when the 
> gc starts calculating the isolated cycles and that requires a balanced 
> reference count to work.

I was thinking about this a bit more and because the refcount bit is too high, 
the GC algorithm will be "fine" in the sense that won't break. It will just 
treat immortal objects as referenced from outside any cycle isolate (because 
the refcount hopefully will never reach 0), which is consistent with the fact 
that they are in the permanent generation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Carl Meyer]

Carl Meyer  added the comment:

I think the concerns about "perfect" behavior in corner cases are in general 
irrelevant here.

In the scenarios where this optimization matters, there is no quantitative 
change that occurs at 100% coverage. Preventing 99% of CoW is 99% as good as 
preventing 100% :) So the fact that a few objects here and there in special 
cases could still trigger CoW just doesn't matter; it's still a massive 
improvement over the status quo. (That said, I wouldn't _mind_ improving the 
coverage, e.g. if you can suggest a better way to find all heap objects instead 
of using the GC.)

And similarly, gps is right that the concern that immortal objects can keep 
other objects alive (even via references added after immortalization) is a 
non-issue in practice. There really is no other behavior one could prefer or 
expect instead.

> if said objects (isolated and untracked before and now tracked) acquire 
> strong references to immortal objects, those objects will be visited when the 
> gc starts calculating the isolated cycles and that requires a balanced 
> reference count to work.

I'm not sure what you mean here by "balanced ref count" or by "work" :) What 
will happen anytime an immortal object gets into the GC, for any reason, is 
that the GC will "subtract" cyclic references and see that the immortal object 
still has a large refcount even after that adjustment, and so it will keep the 
immortal object and any cycle it is part of alive. This behavior is correct and 
should be fully expected; nothing breaks. It doesn't matter at all to the GC 
that this large refcount is "fictional," and it doesn't break the GC algorithm, 
it results only in the desired behavior of maintaining immortality of immortal 
objects.

It is perhaps slightly weird that this behavior falls out of the immortal bit 
being a high bit rather than being more explicit. I did do some experimentation 
with trying to explicitly prevent immortal instances from ever entering GC, but 
it turned out to be hard to do that in an efficient way. And motivation to do 
it is low, because there's nothing wrong with the behavior in the existing PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

[Hrvoje Nikšić]

Hrvoje Nikšić  added the comment:

Thanks for the clarification, I didn't know about the change to non-daemon 
threads.

I still think this patch is useful, and won't harm general use because it is 
opt-in, just like daemon threads themselves. I suggest to update the PR to 
specify non-waiting pool at the point of creation rather than in shutdown(). 
(That will also make it more acceptable for the same option not being supported 
for process pools - it is ok for constructor signatures to differ.)

If there is interest, someone should take over the PR, as I have changed jobs 
and no longer have time to actively pursue this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32033] The pwd module implementation incorrectly sets some attributes to None

[miss-islington]

miss-islington  added the comment:


New changeset 1e1dbdf23f7a18f53a3257badc3541973831f2c4 by Miss Islington (bot) 
in branch '3.8':
bpo-32033: Fix test_pwd failures on Android (GH-19502)
https://github.com/python/cpython/commit/1e1dbdf23f7a18f53a3257badc3541973831f2c4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32033] The pwd module implementation incorrectly sets some attributes to None

[miss-islington]

miss-islington  added the comment:


New changeset 8821200d85657ef3bbec78dcb43694449c05e896 by Miss Islington (bot) 
in branch '3.7':
bpo-32033: Fix test_pwd failures on Android (GH-19502)
https://github.com/python/cpython/commit/8821200d85657ef3bbec78dcb43694449c05e896


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40260] modulefinder traceback regression starting on Windows

[Steve Dower]

Steve Dower  added the comment:

Thanks, Barry!

I tweaked your NEWS entry a little, so once CI completes I'll merge and 
backport.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[Miro Hrončok]

Change by Miro Hrončok :


--
nosy: +hroncok
nosy_count: 17.0 -> 18.0
pull_requests: +18870
pull_request: https://github.com/python/cpython/pull/19520

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40255] Fixing Copy on Writes from reference counting

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> I'm not sure what you mean here by "balanced ref count" or by "work" :) What 
> will happen anytime an immortal object gets into the GC, for any reason, is 
> that the GC will "subtract" cyclic references and see that the immortal 
> object still has a large refcount even after that adjustment, and so it will 
> keep the immortal object and any cycle it is part of alive. This behavior is 
> correct and should be fully expected; nothing breaks. It doesn't matter at 
> all to the GC that this large refcount is "fictional," and it doesn't break 
> the GC algorithm, it results only in the desired behavior of maintaining 
> immortality of immortal objects.


Yep, that is right. I think there was a race condition between my previous 
message and yours :)

I think what was confusing me in this line of reasoning is that I was not 
taking into account that the immortal bit is a very high one, making the 
refcount gigantic. I was treating it mentally like a flag without factoring the 
implications of such big reference count.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40282] random.getrandbits(0) should succeed

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Seconded.

And I wish to add the getrandbytes() method.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40267] Error message differs when an expression is in an fstring

[Lysandros Nikolaou]

Change by Lysandros Nikolaou :


--
keywords: +patch
pull_requests: +18871
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19521

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40282] random.getrandbits(0) should succeed

[Mark Dickinson]

Mark Dickinson  added the comment:

This was discussed previously in #37000.

I agree that `getrandbits(0)` should succeed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

[Kyle Stanley]

Kyle Stanley  added the comment:

> I don't think there's much ThreadPoolExecutor can do.  If you drop the 
> references to the threads, they still exist and they still be waited upon at 
> interpreter exit.
> 
> The solution is for you to avoid having hanging threads.  In the particular 
> case of TCP connections, I'd recommend using a dedicated framework such as 
> asyncio (or Twisted, Tornado, etc.) instead of home-baked networking code.

As far as I'm aware, I don't think there's a _safe_ way to force a hanging 
thread to immediately exit without causing resource-related problems (an unsafe 
way would be something like releasing the internal thread state lock). But in 
general, I think that hanging threads indicate a fundamental issue in the 
implementation of the function the thread is targeting. For any operation that 
can potentially stall or run indefinitely, there should be some form of fail 
safe or timeout in place to break out of it. So I agree with Antoine that it's 
ultimately the responsibility of the thread itself to not hang. There's not 
much of anything that ThreadPoolExecutor can safely do to resolve a hanging 
thread.

> Also, note that Python sockets have a feature called *timeouts*.

There's also of course timeouts implemented at a higher level in many 
networking frameworks, if the developer doesn't want to do so at the socket 
level. For example, see asyncio.wait_for(): 
https://docs.python.org/3/library/asyncio-task.html#asyncio.wait_for.

> These threads were daemon threads in 3.8, so your patch indeed works there, 
> but we've made them non-daemon in 3.9, for two reasons:
1) daemon threads are fragile and can crash the interpreter at shutdown
2) they are not supported on subinterpreters

Just to briefly clarify on (2), Victor recently opened a PR to revert daemon 
threads being entirely unsupported in subinterpreters: PR-19456. From reading 
over the attached bpo issue, it looks like the plan is to deprecate daemon 
threads in subinterpreters, but to not immediately drop support because users 
of mod_wsgi and various monitoring tools were reliant upon it (through the 
C-API).

It seems like the current plan is for it to undergo a deprecation process. 
Either way though, I still think the change to make executors no longer reliant 
upon daemon threads was a significant stability improvement and will mean we 
don't have to make the change later when they are fully unsupported in 
subinterpreters.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40260] modulefinder traceback regression starting on Windows

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 7.0 -> 8.0
pull_requests: +18872
pull_request: https://github.com/python/cpython/pull/19522

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40260] modulefinder traceback regression starting on Windows

[Steve Dower]

Steve Dower  added the comment:


New changeset d42e5820631cd66ee1eab8f610d4b58f3dfdd81c by Barry in branch 
'master':
bpo-40260: Update modulefinder to use io.open_code() and respect coding 
comments (GH-19488)
https://github.com/python/cpython/commit/d42e5820631cd66ee1eab8f610d4b58f3dfdd81c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23082] pathlib relative_to() can give confusing error message

[Steve Dower]

Steve Dower  added the comment:

I agree it's worth improving the error message (and probably the docs too).

It's never made clear that relative_to only looks deeper (won't ever generate 
leading ".." parts) - the closest hint in the docs is that os.path.relpath is 
different (and that isn't even in the relative_to() section).


Tagging this easy/newcomer friendly. If you'd like to work on it, just post 
here - I'm happy to help get it merged.

--
keywords: +easy, newcomer friendly
nosy: +steve.dower
versions: +Python 3.8, Python 3.9 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39522] AST Unparser with unicode kinded constants

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
pull_requests: +18873
pull_request: https://github.com/python/cpython/pull/19523

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40260] modulefinder traceback regression starting on Windows

[miss-islington]

miss-islington  added the comment:


New changeset 59047fab0ef37f583c9e7c3a48d67792fd10ff91 by Miss Islington (bot) 
in branch '3.8':
bpo-40260: Update modulefinder to use io.open_code() and respect coding 
comments (GH-19488)
https://github.com/python/cpython/commit/59047fab0ef37f583c9e7c3a48d67792fd10ff91


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40260] modulefinder traceback regression starting on Windows

[Steve Dower]

Change by Steve Dower :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40249] __import__ doesn't honour globals

[Brett Cannon]

Brett Cannon  added the comment:

How would you propose changing the wording found at 
https://docs.python.org/3/library/functions.html?highlight=__import__#__import__?

--
nosy: +brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[miss-islington]

Change by miss-islington :


--
pull_requests: +18874
pull_request: https://github.com/python/cpython/pull/19524

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9216] FIPS support for hashlib

[miss-islington]

miss-islington  added the comment:


New changeset 4c0a31fb08407ba043688ad1c21102dd4cb99146 by Miro Hrončok in 
branch 'master':
bpo-9216: Nobody expects the geohashing FIPS inquisition (GH-19520)
https://github.com/python/cpython/commit/4c0a31fb08407ba043688ad1c21102dd4cb99146


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

[Kyle Stanley]

Kyle Stanley  added the comment:

> ThreadPoolExecutor introduces additional waiting of its own, and it is this 
> wait the PR adds an option to disable.

[next post]

> Thanks for the clarification, I didn't know about the change to non-daemon 
> threads.

> I still think this patch is useful, and won't harm general use because it is 
> opt-in, just like daemon threads themselves. I suggest to update the PR to 
> specify non-waiting pool at the point of creation rather than in shutdown(). 
> (That will also make it more acceptable for the same option not being 
> supported for process pools - it is ok for constructor signatures to differ.)

Yes, but if you simply make the ThreadPoolExecutor forget about the thread, 
then all that it does is divert the wait to occur at a later time. In this 
case, it would get stuck at `threading._shutdown()` (where all non-daemon 
threads are joined) instead of `executor.shutdown()` since they no longer use 
daemon threads.

The only way I could see this to work as intended without making any changes to 
threading would be to optionally use daemon threads and avoid joining the 
threads in `executor.shutdown()` if `wait_at_exit` is set to False in the 
constructor for the executor. But at the least, users would have to be made 
aware in the documentation that this could lead to significant resource 
finalization issues, and potential incompatibility with subinterpreters. 

Otherwise, they may end up accidentally shooting themselves in the foot (which 
we certainly want to avoid). I'm also somewhat concerned that it may end up 
getting used to cover up actual bugs. 

IMO, it also would require some fairly extensive testing to make sure it works 
as intended, which the patch currently lacks. So in order to implement this 
properly, it would require a significant amount of additional time investment 
beyond what has been put into the existing patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40249] __import__ doesn't honour globals

[Stefan Seefeld]

Stefan Seefeld  added the comment:

I'm not entirely sure, but have to admit that the sentence

"The function imports the module name, potentially using the given globals and 
locals to determine how to interpret the name in a package context."

is a bit obscure. What does "determine how to interpret the name" actually mean 
? Is the algorithm described anywhere in detail ? In that case, a simple 
reference might be enough.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36780] Interpreter exit blocks waiting for futures of shut-down ThreadPoolExecutors

[Hrvoje Nikšić]

Hrvoje Nikšić  added the comment:

> The only way I could see this to work as intended without making any changes 
> to threading would be to optionally use daemon threads and avoid joining the 
> threads in `executor.shutdown()` if `wait_at_exit` is set to False in the 
> constructor for the executor.

Agreed, and that is precisely what I suggested in my previous comment. The 
attached PR already deals with the executor-specific part of the wait, but it 
would be straightforward to have it also affect the executor to create daemon 
threads, and the flag moved to the constructor.

> IMO, it also would require some fairly extensive testing to make sure it 
> works as intended, which the patch currently lacks.

It is quite easy to check that a hanging thread (emulated by a simple sleep) is 
not joined by the executor with the appropriate flag set.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40282] random.getrandbits(0) should succeed

[Raymond Hettinger]

Raymond Hettinger  added the comment:

+0 for having getrandbits(0) return 0.  Conceptually, it is reasonable.  
Practically, it is a bit inconvenient because the ValueError may have to be 
moved upstream to the _randbelow() methods.

-1 for getrandbytes().  That is feature creep and no user has requested it.  
Also, the name leads to a confusing API with getrandbits() returning arbitrary 
sized python ints and getrandbytes() returning bytes.  Lastly, it mostly 
duplicates functionality already supplied by secrets.token_bytes().  If you 
really want this, open another tracker issue and don't derail the issue at hand.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39522] AST Unparser with unicode kinded constants

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
nosy: +pablogsal
nosy_count: 1.0 -> 2.0
pull_requests: +18875
pull_request: https://github.com/python/cpython/pull/19525

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39522] AST Unparser with unicode kinded constants

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 43aeefa41915e4d3b0e68bbd4268c1c378a72dce by Batuhan Taşkaya in 
branch 'master':
bpo-39522: Use _PyUnicodeWriter_WriteStr instead of PyUnicode_AS_DATA (GH-19523)
https://github.com/python/cpython/commit/43aeefa41915e4d3b0e68bbd4268c1c378a72dce


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39522] AST Unparser with unicode kinded constants

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 33986465bde2a2188537c4ef6cdb6055e348f31f by Pablo Galindo in 
branch 'master':
bpo-39522: Always initialise kind attribute in constant ast nodes (GH-19525)
https://github.com/python/cpython/commit/33986465bde2a2188537c4ef6cdb6055e348f31f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39522] AST Unparser with unicode kinded constants

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40282] random.getrandbits(0) should succeed

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I do not want to open an issue if I know that the idea will be rejected.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com