[issue32561] Add API to io objects for non-blocking reads/writes

[Xavier G. Domingo]

Change by Xavier G. Domingo :


--
nosy: +xgdomingo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32561] Add API to io objects for non-blocking reads/writes

[Nathaniel Smith]

Nathaniel Smith  added the comment:

> BufferedIOBase is an abstract class and, despite the name, doesn’t 
> necessitate a buffer or cache

Right, sorry, skimmed too fast.

> In Issue 32475 there is a proposal to add a “getbuffn” method returning the 
> amount of unread pending data in a reader object. Perhaps that would be 
> enough for reading.

Ideally we would be able to do buffer-only reads through all the of the 
different read methods (read, readline, readinto, ...), and ideally we would be 
able to do it given objects at different points in the IO stack – so a 
buffer-only read on TextWrapper wrapped around a BufferedRandom wrapped around 
a FileIO, should propagate the buffer-only-ness all the way down the stack. I 
don't think getbuffn is enough to solve that? Or at least I don't see how in 
any simple way.

Also, the immediate thing that spurred me to file this issue was learning that 
Linux has just added a non-blocking file read syscall. It would be pretty neat 
if we could expose that. If we had a way to propagate this down, then it could 
just be FileIO's implementation of the buffer-only flag.

But yeah, actually doing that is complicated given the need to continue 
supporting existing implementations of these interfaces.

Here's a straw man proposal: add a unbuffered_supported flag to the abstract IO 
interfaces. If missing or false, you can't do unbuffered reads/writes. If 
present and True, then you can pass a new unbuffered=True kw-only argument to 
their read/write calls.

When (for example) TextWrapper.read needs to call its wrapped object's .read, 
it does a check like:

  if unbuffered:
  # This call is unbuffered, so we're only allowed to call
  # unbuffered methods.
  if not getattr(self.wrapped, "unbuffered_supported", False):
  # lower level doesn't support this, can't be done
  raise ...
  else:
  self.wrapped.read(..., unbuffered=True)
  else:
  # We're a regular call, so we can make a regular call
  self.wrapped.read(...)

(I'm intentionally using "unbuffered" here to distinguish from regular POSIX 
"non-blocking", which is an API that's conceptually very similar but totally 
distinct in implementation. Especially since it's also possible to use the io 
stack with sockets/pipes in non-blocking mode.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31368] Support asynchronous, non-blocking buffered reads (RWF_NONBLOCK)

[Nitish]

Change by Nitish :


--
nosy: +nitishch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29723] 3.6.1rc1 adds the current directory to sys.path when running a subdirectory's __main__.py; previous versions did not

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests:  -585

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32551] Zipfile & directory execution in 3.5.4 adds the current directory to sys.path

[Nick Coghlan]

Nick Coghlan  added the comment:

I've updated the issue title to reduce the need to have read bpo-29723 first to 
understand it.

I've also filed https://github.com/pantsbuild/pex/issues/440 essentially asking 
the pex folks to check if they're affected.

--
stage:  -> commit review
title: 3.5.4 has a regression that was fixed in 3.6.1rc1 -> Zipfile & directory 
execution in 3.5.4 adds the current directory to sys.path

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32551] 3.5.4 has a regression that was fixed in 3.6.1rc1

[Nick Coghlan]

Nick Coghlan  added the comment:

PR submitted for 3.5. Since the problem was in a full release this time (rather 
than a pre-release the way it was for 3.6), I've reclassified it as a security 
bug, since it means some previously safe operations (where no user-writable 
directory would end up on sys.path even without the "-I" switch) are 
technically unsafe.

There's a fair combination of factors required for it to actually cause a 
problem though:

- it only comes up if you're *not* using isolated mode (so the "workaround" is 
"Use isolated mode when running privileged processes", which is something we 
recommend doing anyway)
- it only comes up for directory or zipfile execution, *not* normal script 
execution
- you have to be executing an admin-controlled directory or zipfile, rather 
than one the user has write access to (otherwise there's no new opportunity for 
code injection introduced)
- for actual privilege escalation to take place as a result of the code 
injection, you have to be running the command with elevated privileges

That said, I suspect exactly the above may happen when using PEX files 
(https://github.com/pantsbuild/pex), since I can't find any reference in their 
code to forcing the use of isolated mode in the underlying interpreter.

--
keywords:  -patch
nosy: +christian.heimes, encukou
stage: patch review -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32551] 3.5.4 has a regression that was fixed in 3.6.1rc1

[Nick Coghlan]

Change by Nick Coghlan :


--
keywords: +patch
pull_requests: +5050
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29319] Embedded 3.6.0 distribution cannot run pyz files

[Nick Coghlan]

Change by Nick Coghlan :


--
pull_requests: +5051

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32563] -Werror=declaration-after-statement expat build failure on Python 3.5

[Nick Coghlan]

Nick Coghlan  added the comment:

Details of the compilation failure:

```
building 'pyexpat' extension
gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 
-Wall -Wstrict-prototypes -Werror=declaration-after-statement 
-DHAVE_EXPAT_CONFIG_H=1 -DXML_POOR_ENTROPY=1 -DUSE_PYEXPAT_CAPI 
-I/home/ncoghlan/devel/py35/Modules/expat -I./Include -I. -I/usr/local/include 
-I/home/ncoghlan/devel/py35/Include -I/home/ncoghlan/devel/py35 -c 
/home/ncoghlan/devel/py35/Modules/pyexpat.c -o 
build/temp.linux-x86_64-3.5/home/ncoghlan/devel/py35/Modules/pyexpat.o
gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 
-Wall -Wstrict-prototypes -Werror=declaration-after-statement 
-DHAVE_EXPAT_CONFIG_H=1 -DXML_POOR_ENTROPY=1 -DUSE_PYEXPAT_CAPI 
-I/home/ncoghlan/devel/py35/Modules/expat -I./Include -I. -I/usr/local/include 
-I/home/ncoghlan/devel/py35/Include -I/home/ncoghlan/devel/py35 -c 
/home/ncoghlan/devel/py35/Modules/expat/xmlparse.c -o 
build/temp.linux-x86_64-3.5/home/ncoghlan/devel/py35/Modules/expat/xmlparse.o
gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 
-Wall -Wstrict-prototypes -Werror=declaration-after-statement 
-DHAVE_EXPAT_CONFIG_H=1 -DXML_POOR_ENTROPY=1 -DUSE_PYEXPAT_CAPI 
-I/home/ncoghlan/devel/py35/Modules/expat -I./Include -I. -I/usr/local/include 
-I/home/ncoghlan/devel/py35/Include -I/home/ncoghlan/devel/py35 -c 
/home/ncoghlan/devel/py35/Modules/expat/xmlrole.c -o 
build/temp.linux-x86_64-3.5/home/ncoghlan/devel/py35/Modules/expat/xmlrole.o
gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 
-Wall -Wstrict-prototypes -Werror=declaration-after-statement 
-DHAVE_EXPAT_CONFIG_H=1 -DXML_POOR_ENTROPY=1 -DUSE_PYEXPAT_CAPI 
-I/home/ncoghlan/devel/py35/Modules/expat -I./Include -I. -I/usr/local/include 
-I/home/ncoghlan/devel/py35/Include -I/home/ncoghlan/devel/py35 -c 
/home/ncoghlan/devel/py35/Modules/expat/xmltok.c -o 
build/temp.linux-x86_64-3.5/home/ncoghlan/devel/py35/Modules/expat/xmltok.o
/home/ncoghlan/devel/py35/Modules/expat/xmltok.c: In function ‘utf8_toUtf8’:
/home/ncoghlan/devel/py35/Modules/expat/xmltok.c:408:3: error: ISO C90 forbids 
mixed declarations and code [-Werror=declaration-after-statement]
   const char * const fromLimBefore = fromLim;
   ^
/home/ncoghlan/devel/py35/Modules/expat/xmltok.c:414:3: error: ISO C90 forbids 
mixed declarations and code [-Werror=declaration-after-statement]
   const ptrdiff_t bytesToCopy = fromLim - *fromP;
   ^
cc1: some warnings being treated as errors
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32563] -Werror=declaration-after-statement expat build failure on Python 3.5

[Nick Coghlan]

New submission from Nick Coghlan :

A fresh Python 3.5 checkout failed to build the embedded expat XML library for 
me, due to the configure check that adds "-Werror=declaration-after-statement" 
to the compiler flags.

I'm guessing this was introduced via the expat code changes brought in for 
https://github.com/python/cpython/commit/f2492bb6aae061aea47e21fc7e56b7ab9bfdf543

That configure check is gone in 3.6 and 3.7, but it's still there in 3.5.

Locally, I've worked around the problem by removing the configure check, but 
I'm not sure that's the right answer, as I believe we still promise that 3.5 
will build under C89.

--
messages: 310037
nosy: larry, ncoghlan, vstinner
priority: low
severity: normal
stage: needs patch
status: open
title: -Werror=declaration-after-statement expat build failure on Python 3.5
type: compile error

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32561] Add API to io objects for non-blocking reads/writes

[Martin Panter]

Change by Martin Panter :


--
dependencies: +Add ability to query number of buffered bytes available on 
buffered I/O

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32561] Add API to io objects for non-blocking reads/writes

[Martin Panter]

Martin Panter  added the comment:

BufferedIOBase is an abstract class and, despite the name, doesn’t necessitate 
a buffer or cache. Adding methods and properties might break compatibility with 
third-party implementations, or get ugly with optional methods and multiple 
versions of the API. It seems like it would be better to extend the concrete 
APIs: io.BufferedReader, BufferedWriter and/or FileIO.

In Issue 32475 there is a proposal to add a “getbuffn” method returning the 
amount of unread pending data in a reader object. Perhaps that would be enough 
for reading.

I would support an similar API for BufferedWriter etc. Perhaps a property 
called “available_space”. You could check that and decide whether to do a 
direct non-blocking “write”, or launch a blocking “write” in the background.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32475] Add ability to query number of buffered bytes available on buffered I/O

[Martin Panter]

Martin Panter  added the comment:

If I remember the implementation of “peek” right, it may do what you want. But 
the documentation doesn’t guarantee much about its behaviour; see Issue 5811.

Anyway, I agree that a “getbuffn” method (or property) would be nice. (Perhaps 
with a better name!) But please don’t add it to the abstract APIs like 
BufferedIOBase. It could break compatibility with third-party implementations, 
or make the API complicated with little benefit. Just extend the concrete APIs 
like BufferedReader.

Two other use cases where the “peek” implementation won’t help, but “getbuffn” 
would:

1. Issue 32561: Decide whether a non-blocking “read” call is possible, or if a 
background read (e.g. of a regular “disk” file) should be started instead.

2. Get the pending unread data before it is lost by calling 
”BufferedReader.detach”.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32562] Support fspath protocol in AF_UNIX sockaddr resolution

[Nathaniel Smith]

New submission from Nathaniel Smith :

In 3.6, trying to connect to a AF_UNIX socket using a pathlib.Path object 
doesn't work:

Python 3.6.4 (default, Dec 27 2017, 13:02:49) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket, pathlib
>>> s = socket.socket(family=socket.AF_UNIX)
>>> s.connect(pathlib.Path("asdf"))
Traceback (most recent call last):
  File "", line 1, in 
TypeError: a bytes-like object is required, not 'PosixPath'

It would be good to fix this to use the fspath protocol.

--
messages: 310034
nosy: brett.cannon, njs
priority: normal
severity: normal
status: open
title: Support fspath protocol in AF_UNIX sockaddr resolution
versions: Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31368] Support asynchronous, non-blocking buffered reads (RWF_NONBLOCK)

[Nathaniel Smith]

Nathaniel Smith  added the comment:

Whoa, cool.

bpo-32561 is complementary to this. (They both make sense on their own, but 
they're even better together.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32561] Add API to io objects for non-blocking reads/writes

[Nathaniel Smith]

New submission from Nathaniel Smith :

Background: Doing I/O to files on disk has a hugely bimodal latency. If the I/O 
happens to be in or going to cache (either user-space cache, like in 
io.BufferedIOBase, or the OS's page cache), then the operation returns 
instantly (~1 µs) without blocking. OTOH if the I/O isn't cached (for reads) or 
cacheable (for writes), then the operation may block for 10 ms or more.

This creates a problem for async programs that want to do disk I/O. You have to 
use a thread pool for reads/writes, because sometimes they block for a long 
time, and you want to let your event loop keep doing other useful work while 
it's waiting. But dispatching to a thread pool adds a lot of overhead (~100 
µs), so you'd really rather not do it for operations that can be serviced 
directly through cache. For uncached operations a thread gives a 100x speedup, 
but for cached operations it's a 100x slowdown, and -- this is the kicker -- 
there's no way to predict which ahead of time.

But, io.BufferedIOBase at least knows when it can satisfy a request directly 
from its buffer without issuing any syscalls. And in Linux 4.14, it's even 
possible to issue a non-blocking read to the kernel that will only succeed if 
the data is immediately available in page cache (bpo-31368).

So, it would be very nice if there were some way to ask a Python file object to 
do a "nonblocking read/write", which either succeeds immediately or else raises 
an error. The intended usage pattern would be:

async def read(self, *args):
try:
self._fileobj.read(*args, nonblock=True)
except BlockingIOError: # maybe?
return await run_in_worker_thread(self._fileobj.read, *args)

It would *really* help for this to be in the Python core, because right now the 
convenient way to do non-blocking disk I/O is to re-use the existing Python I/O 
stack, with worker threads. (This is how both aiofiles and trio's async file 
support work. I think maybe curio's too.) But to implement this feature 
ourselves, we'd have to first reimplement the whole I/O stack, because the 
important caching information, and choice of what syscall to use, are hidden 
inside.

--
components: IO
messages: 310032
nosy: benjamin.peterson, njs, stutzbach
priority: normal
severity: normal
status: open
title: Add API to io objects for non-blocking reads/writes
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32560] inherit the py launcher's STARTUPINFO

[Steve Dower]

Steve Dower  added the comment:

This sounds like a good idea to me - feel free to contribute your PR.

--
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31368] Support asynchronous, non-blocking buffered reads (RWF_NONBLOCK)

[Antoine Pitrou]

Antoine Pitrou  added the comment:

In any case, I think this would take the form of new low-level calls in the 
posix module, i.e. add thin wrappers around the new system / libc calls.

--
nosy: +njs, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

[STINNER Victor]

STINNER Victor  added the comment:

> New changeset 7ed7aead9503102d2ed316175f198104e0cd674c by Victor Stinner in 
> branch 'master':
> bpo-29240: Fix locale encodings in UTF-8 Mode (#5170)

Oh, this change broke test_nonascii() of test_readline() on FreeBSD.

Previsously, readline used ASCII/surrogateescape encoding for the POSIX locale. 
Now, mbstowcs() / wcstombs() is called directly, with the surrogateescape error 
handler.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32553] venv says to use python3 which does not exist in 3.6.4

[Eryk Sun]

Eryk Sun  added the comment:

`PATHEXT` is irrelevant here, so that should be removed. As to the py launcher, 
it's optional and not always installed for all users, though that's the 
default. I'd prefer `python` that's found in PATH as the first example. Then 
add another example with `py -X.Y` to show how to create virtual environments 
for other versions.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32551] 3.5.4 has a regression that was fixed in 3.6.1rc1

[Brett Cannon]

Brett Cannon  added the comment:

Nick makes this sound like it really should land in 3.5.5, so marking as a 
release blocker for now.

--
nosy: +brett.cannon
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14976] queue.Queue() is not reentrant, so signals and GC can cause deadlocks

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Ok, there has been enough reviews in the last four months :-) This is now 
merged.

--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14976] queue.Queue() is not reentrant, so signals and GC can cause deadlocks

[Antoine Pitrou]

Antoine Pitrou  added the comment:


New changeset 94e1696d04c65e19ea52e5c8664079c9d9aa0e54 by Antoine Pitrou in 
branch 'master':
bpo-14976: Reentrant simple queue (#3346)
https://github.com/python/cpython/commit/94e1696d04c65e19ea52e5c8664079c9d9aa0e54


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32553] venv says to use python3 which does not exist in 3.6.4

[Paul Moore]

Paul Moore  added the comment:

That's why I suggest using the launcher rather than absolute paths - there are 
far too many variations to enumerate with absolue paths (you didn't mention 
per-user Python 3.6 installations, for example).

If you have a suggested wording that you feel would explain things better, feel 
free to propose it. What's there at the moment seems relatively OK to me (aside 
from a minor tweak to use the launcher) so I'm not sure I could modify it in a 
way that would address your concerns.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32534] Speed-up list.insert

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Perhaps the patch is interfering weirdly with PGO?

> Should I run the benchmark without a PGO build (i.e without 
> --enable-optimizations)?

That would help clear things up, IMHO.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32534] Speed-up list.insert

[Jeethu Rao]

Jeethu Rao  added the comment:

I rebased my branch off of master and rebuilt it, and also rebuilt the baseline 
from master. Both versions were configured with --with-lto and 
--enable-optimizations. The benchmark numbers are rather different this 
time[1]. pidigits is slower, but nbody is still inexplicably faster. Should I 
run the benchmark without a PGO build (i.e without --enable-optimizations)?

Also, I've attached the output from the pyperformance run to the aforementioned 
gist.

[1]: https://gist.github.com/jeethu/dc0811d415dd6d1a1621761e43842f88

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32555] Encoding issues with the locale encoding

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b92c159efada05b3a5ff9d0dbce3fcb2334631f6 by Victor Stinner in 
branch '3.6':
[3.6] bpo-32555: Fix locale encodings (#5193)
https://github.com/python/cpython/commit/b92c159efada05b3a5ff9d0dbce3fcb2334631f6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 5f959c4f9eca404b8bc4bc6348fed27c4b907b89 by Victor Stinner in 
branch '3.6':
[3.6] bpo-31900: Fix localeconv() encoding for LC_NUMERIC (#4174) (#5192)
https://github.com/python/cpython/commit/5f959c4f9eca404b8bc4bc6348fed27c4b907b89


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32554] random.seed(tuple) uses the randomized hash function and so is not reproductible

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Maybe deprecate using a hash?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32539] os.listdir(...) on deep path on windows in python2.7 fails with errno 123

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Thank you Anthony for your patch!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32539] os.listdir(...) on deep path on windows in python2.7 fails with errno 123

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 27f32e938ff51fd5d90a29abbbabc6b81d156f33 by Serhiy Storchaka 
(Anthony Sottile) in branch '2.7':
bpo-32539: Fix OSError for os.listdir() for extended-length paths on Windows 
(#5169)
https://github.com/python/cpython/commit/27f32e938ff51fd5d90a29abbbabc6b81d156f33


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32560] inherit the py launcher's STARTUPINFO

[Eryk Sun]

New submission from Eryk Sun :

I've occasionally seen requests to find the path of the LNK shortcut that was 
used to run a script -- for whatever reason. This can be reliably supported in 
most cases by calling WinAPI GetStartupInfo. If the flag STARTF_TITLEISLINKNAME 
is set, then the lpTitle field is the path to the LNK file. (This is how the 
console host process knows to use the shortcut file for its properties instead 
of the registry.)

However, if .py scripts are associated with the py launcher, the STARTUPINFO 
that has the required information is in the launcher process instead of the 
python.exe process. Since to some extent the launcher is acting as a proxy for 
python.exe, I think it should initialize Python's STARTUPINFO from a copy of 
its own values as returned by GetStartupInfo. This solves the dwFlags and 
lpTitle problem.

Additionally, the C runtime spawn family of functions supports inheriting file 
descriptors by passing them in the lpReserved2 field of STARTUPINFO. Currently 
this doesn't work when spawning Python via the launcher, since it calls 
CreateProcess instead of a spawn function. By inheriting the launcher's 
STARTUPINFO, this feature is restored as well. 

For example, in `run_child` in PC/launcher.c, I replaced `si.cb = sizeof(si)` 
with GetStartupInfoW(). Then I tested passing inherited file descriptors 
through the launcher as follows:

grandparent:

>>> import os
>>> os.pipe()
(3, 4)
>>> os.set_inheritable(4, True)
>>> os.spawnl(os.P_WAIT, 'amd64/py_d.exe', 'py_d')

grandchild (py_d.exe => python.exe):

Python 3.7.0a4 (v3.7.0a4:07c9d85, Jan  9 2018, 07:07:02)
[MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.write(4, b'spam')
4
>>> ^Z

grandparent:

0
>>> os.read(3, 4)
b'spam'

--
components: Windows
keywords: easy (C)
messages: 310016
nosy: eryksun, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
stage: needs patch
status: open
title: inherit the py launcher's STARTUPINFO
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32553] venv says to use python3 which does not exist in 3.6.4

[Paul Watson]

Paul Watson  added the comment:

Ok, I do find later on where it says to do something different on Windows. 
However, the first "Note" box in "28.3. venv" specifies using "python3". It 
does not say that this is for non-Windows systems. This should be changed.

Also, the path specified, "c:\Python35\python", would not be the default 
location for a Windows install. "C:\Program Files\Python36\python" or 
"C:\Program Files (x86)\Python36\python" should be specified.

I detest that it is this complex and verbose, but it is. Glossing over it does 
not help the user.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14976] queue.Queue() is not reentrant, so signals and GC can cause deadlocks

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Hi all,

The PR has been ready for quite some time now.  Raymond posted some review 
comments back in September, which I addressed by making the requested changes.

If someone wants to add their comments, now is the time. Otherwise I plan to 
merge the PR sometimes soon, so that it gets in before 3.7 beta.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32146] multiprocessing freeze_support needed outside win32

[bbayles]

Change by bbayles :


--
keywords: +patch
pull_requests: +5049
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15115] Duplicated Content-Transfer-Encoding header when applying email.encoders

[R. David Murray]

R. David Murray  added the comment:

I believe so.  For python3 I think it should only apply to the legacy API docs 
(you would use set_content (directly or indirectly) in python3, not 
set_payload).  I've updated the versions.

--
versions: +Python 3.6, Python 3.7 -Python 3.2, Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29708] support reproducible Python builds

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

On Jan 15, 2018, at 11:31, Brett Cannon  wrote:
> 
> 1. SOURCE_DATE_EPOCH acts as an environment variable flag to forcibly 
> generate hash-based .pyc files with the check_source bit set in py_compile 
> and compileall
> 2. SOURCE_DATE_EPOCH is used to specifically set the timestamp in .pyc files 
> in py_compile and compileall

I’d suggest that if SDE is set to an integer, that is used as the timestamp.  
If it’s set to a special symbol (e.g. ‘hash’) then the hash is used.  I’m not 
volunteering to write the code though. :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32516] Add a shared library mechanism for win32

[Brett Cannon]

Brett Cannon  added the comment:

To answer Steve's question, what you would want is a finder which recognized 
the directory of the package so as to return a special loader just for that 
package (basically __path__ is sent through the normal import mechanism and so 
you would want something on sys.path_hooks which knew how to get an appropriate 
finder which would return the loader you want).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29708] support reproducible Python builds

[Brett Cannon]

Brett Cannon  added the comment:

Bernhard's idea of SOURCE_DATE_EPOCH being an implicit envvar to forcibly 
switch on hash-based .pyc files in py_compile is intriguing. I assume this 
would force the check_source bit to be set? Or since SOURCE_DATE_EPOCH should 
only be used in build scenarios would you want UNCHECKED_HASH?

As the core dev who seems the most engaged and willing to commit this, I'm 
willing to make the final decision on this and commit the final PR. I see the 
options of getting this into 3.7 as the following:

1. SOURCE_DATE_EPOCH acts as an environment variable flag to forcibly generate 
hash-based .pyc files with the check_source bit set in py_compile and compileall
2. SOURCE_DATE_EPOCH is used to specifically set the timestamp in .pyc files in 
py_compile and compileall

That's it. No clamping, no changing how timestamp-based .pyc files are 
invalidated, no touching source files, etc.

If this is going to make it into Python 3.7 then a decision must be made by 
Friday, Jan 19, so have your opinions on those two options in before then (and 
in the case of the hash-based solution, would you expect CHECKED_HASH or 
UNCHECKED_HASH?). At that point I will make a decision and Bernhard can either 
update his PR or I can create a new one forked from his(I leave that up to 
Bernhard based on the decision I'll make on/by Friday).

--
assignee:  -> brett.cannon
versions:  -Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26163] FAIL: test_hash_effectiveness (test.test_set.TestFrozenSet)

[Raymond Hettinger]

Change by Raymond Hettinger :


--
pull_requests: +5048

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32554] random.seed(tuple) uses the randomized hash function and so is not reproductible

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> This is very confusing, I hope you can fix the behavior, not the doc string.

I'll fix the docstring to make it more specific.

We really don't want to use hash(obj) because it produces too few bits of 
entropy.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32554] random.seed(tuple) uses the randomized hash function and so is not reproductible

[Raymond Hettinger]

Change by Raymond Hettinger :


--
Removed message: https://bugs.python.org/msg310006

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26163] FAIL: test_hash_effectiveness (test.test_set.TestFrozenSet)

[Raymond Hettinger]

Raymond Hettinger  added the comment:

I'm getting a nice improvement in dispersion statistics by shuffling in higher 
bits right at the end:

 /* Disperse patterns arising in nested frozensets */
  +  hash ^= (hash >> 11) ^ (~hash >> 25);
 hash = hash * 69069U + 907133923UL;

Results for range() check:

 range   range
baseline  new
  1st percentile 35.06%  40.63%
  1st decile 48.03%  51.34%
  mean   61.47%  63.24%  
  median 63.24%  65.58% 

Test code for the letter_range() test:

 letter  letter
baseline  new
  1st percentile 39.59%  40.14%
  1st decile 50.90%  51.07%
  mean   63.02%  63.04%  
  median 65.21%  65.23% 


def letter_range(n):
return string.ascii_letters[:n]

def powerset(s):
for i in range(len(s)+1):
yield from map(frozenset, itertools.combinations(s, i))

# range() check
for i in range(1):
for n in range(5, 19):
t = 2 ** n
mask = t - 1
u = len({h & mask for h in map(hash, powerset(range(i, i+n)))})
print(u/t*100)

# letter_range() check needs to be restarted (reseeded on every run)
for n in range(5, 19):
t = 2 ** n
mask = t - 1
u = len({h & mask for h in map(hash, powerset(letter_range(n)))})
print(u/t)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26163] FAIL: test_hash_effectiveness (test.test_set.TestFrozenSet)

[Raymond Hettinger]

Change by Raymond Hettinger :


--
Removed message: https://bugs.python.org/msg310007

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26163] FAIL: test_hash_effectiveness (test.test_set.TestFrozenSet)

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Messages (3)
msg309956 - (view)  Author: Johnny Dude (JohnnyD)   Date: 2018-01-15 01:08
When using a tuple that include a string the results are not consistent when 
invoking a new interpreter or process.

For example executing the following on a linux machine will yield different 
results:
python3.6 -c 'import random; random.seed(("a", 1)); print(random.random())"

Please note that the doc string of random.seed states: "Initialize internal 
state from hashable object."

Python documentation does not. 
(https://docs.python.org/3.6/library/random.html#random.seed)

This is very confusing, I hope you can fix the behavior, not the doc string.
msg309957 - (view)  Author: STINNER Victor (vstinner) * (Python committer)  
Date: 2018-01-15 01:13
random.seed(str) uses:

if version == 2 and isinstance(a, (str, bytes, bytearray)):
if isinstance(a, str):
a = a.encode()
a += _sha512(a).digest()
a = int.from_bytes(a, 'big')

Whereas for other types, random.seed(obj) uses hash(obj), and hash is 
randomized by default in Python 3.

Yeah, the random.seed() documentation should describe the implementation and 
explain that hash(obj) is used and that the hash function is randomized by 
default:
https://docs.python.org/dev/library/random.html#random.seed
msg310006 - (view)  Author: Raymond Hettinger (rhettinger) * (Python 
committer) Date: 2018-01-15 10:41
I'm getting a nice improvement in dispersion statistics by shuffling in higher 
bits right at the end:

 /* Disperse patterns arising in nested frozensets */
  +  hash ^= (hash >> 11) ^ (~hash >> 25);
 hash = hash * 69069U + 907133923UL;

Results for range() check:

 range   range
baseline  new
  1st percentile 35.06%  40.63%
  1st decile 48.03%  51.34%
  mean   61.47%  63.24%  
  median 63.24%  65.58% 

Test code for the letter_range() test:

 letter  letter
baseline  new
  1st percentile 39.59%  40.14%
  1st decile 50.90%  51.07%
  mean   63.02%  63.04%  
  median 65.21%  65.23% 


def letter_range(n):
return string.ascii_letters[:n]

def powerset(s):
for i in range(len(s)+1):
yield from map(frozenset, itertools.combinations(s, i))

# range() check
for i in range(1):
for n in range(5, 19):
t = 2 ** n
mask = t - 1
u = len({h & mask for h in map(hash, powerset(range(i, i+n)))})
print(u/t*100)

# letter_range() check needs to be restarted (reseeded on every run)
for n in range(5, 19):
t = 2 ** n
mask = t - 1
u = len({h & mask for h in map(hash, powerset(letter_range(n)))})
print(u/t)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32554] random.seed(tuple) uses the randomized hash function and so is not reproductible

[Raymond Hettinger]

Raymond Hettinger  added the comment:

I'm getting a nice improvement in dispersion statistics by shuffling in higher 
bits right at the end:

 /* Disperse patterns arising in nested frozensets */
  +  hash ^= (hash >> 11) ^ (~hash >> 25);
 hash = hash * 69069U + 907133923UL;

Results for range() check:

 range   range
baseline  new
  1st percentile 35.06%  40.63%
  1st decile 48.03%  51.34%
  mean   61.47%  63.24%  
  median 63.24%  65.58% 

Test code for the letter_range() test:

 letter  letter
baseline  new
  1st percentile 39.59%  40.14%
  1st decile 50.90%  51.07%
  mean   63.02%  63.04%  
  median 65.21%  65.23% 


def letter_range(n):
return string.ascii_letters[:n]

def powerset(s):
for i in range(len(s)+1):
yield from map(frozenset, itertools.combinations(s, i))

# range() check
for i in range(1):
for n in range(5, 19):
t = 2 ** n
mask = t - 1
u = len({h & mask for h in map(hash, powerset(range(i, i+n)))})
print(u/t*100)

# letter_range() check needs to be restarted (reseeded on every run)
for n in range(5, 19):
t = 2 ** n
mask = t - 1
u = len({h & mask for h in map(hash, powerset(letter_range(n)))})
print(u/t)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32346] Speed up slot lookup for class creation

[Antoine Pitrou]

Change by Antoine Pitrou :


--
status: pending -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32554] random.seed(tuple) uses the randomized hash function and so is not reproductible

[Raymond Hettinger]

Change by Raymond Hettinger :


--
assignee: docs@python -> rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32559] logging - public function to get level from name

[Varun Agrawal]

New submission from Varun Agrawal :

Currently, the logging module has a function `getLevelName` and directly 
accessing the `_nameToLevel` dict is poor practice bordering on non-pythonic.

It would be nice to have a simple method like `getNameLevel` in the module to 
retrieve the logger level given the name. This can then be used in things such 
as argument parsers so someone can pass in the logging level name and the 
logging level can be appropriately set.

--
components: Library (Lib)
messages: 310005
nosy: varunagrawal
priority: normal
severity: normal
status: open
title: logging - public function to get level from name
type: enhancement
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32553] venv says to use python3 which does not exist in 3.6.4

[Paul Moore]

Paul Moore  added the comment:

Actually, for Windows, the docs don't recommend "python3", they say:

"""
On Windows, invoke the venv command as follows:

c:\>c:\Python35\python -m venv c:\path\to\myenv
"""

This should probably be altered to say "py -m venv" (or maybe "py -3 -m venv", 
but I don't think that's really necessary), as using the launcher is less tied 
to having your PATH set, or using an explicit path.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31368] Support asynchronous, non-blocking buffered reads (RWF_NONBLOCK)

[Ned Deily]

Ned Deily  added the comment:

It seems to me there are two issues here:

1. What are the pros and cons of adding support for this new mode, e.g. are 
there any potential gotchas?

2. Assuming the result of 1 is we should go ahead, then omeone needs to 
generate a pull request including configure checks as necessary, doc changes 
and tests.

It's getting late in the game for inclusion in 3.7 with only 2 weeks left until 
feature freeze.

--
components: +Library (Lib)
nosy: +ned.deily
stage:  -> needs patch
title: RWF_NONBLOCK -> Support asynchronous, non-blocking buffered reads 
(RWF_NONBLOCK)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32553] venv says to use python3 which does not exist in 3.6.4

[Ned Deily]

Ned Deily  added the comment:

Thanks! Pinging the Windows team.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32553] venv says to use python3 which does not exist in 3.6.4

[Ned Deily]

Change by Ned Deily :


--
components: +Windows
nosy: +paul.moore, steve.dower, tim.golden, zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32553] venv says to use python3 which does not exist in 3.6.4

[Paul Watson]

Paul Watson  added the comment:

I am running 3.6.4 from a download on pythong.org yesterday.

Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit 
(AMD64)] on win32

There is a python3.dll, but no python3 executable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32553] venv says to use python3 which does not exist in 3.6.4

[Ned Deily]

Ned Deily  added the comment:

Unfortunately, it may depend on what environment and/or from which distributor 
you obtain Python 3.6 as to whether there is a "python3" link installed.  On 
Unix-like systems, the default Python installation built from source ("make 
install") does install both "python3.6" and "python3" links in the configured 
"bin" directory.  But that may not be true everywhere.  On what environment are 
you running Python 3.6 and from where was it installed?

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11191] test_search_cpp error on AIX (with xlc)

[Michael Felt]

Michael Felt  added the comment:

a) Is this normal?

root@x065:[/data/prj/python/git/python3-3.7]./python -m unittest 
test.test_distutils

--
Ran 0 tests in 0.000s

OK

In short - at the start I had:

./python Lib/test/test_distutils.py 
...
ERROR: test_run (distutils.tests.test_build_clib.BuildCLibTestCase)
ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
ERROR: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase)
ERROR: test_build_ext (distutils.tests.test_build_ext.ParallelBuildExtTestCase)
ERROR: test_get_outputs 
(distutils.tests.test_build_ext.ParallelBuildExtTestCase)
ERROR: test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase)
ERROR: test_record_extensions (distutils.tests.test_install.InstallTestCase)

That is now down to:
ERROR: test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase)

There were two issues for search_cpp:
a) xlc -E needs -C to include comments in the output - this should be fixed
b) xlc -E does not accept -o as an argument - output must be to stdout, and 
sadly I am not clever enough to fix that. Help appreciated.

As is: the error was:

ERROR: test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase)
--
Traceback (most recent call last):
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/unixccompiler.py", 
line 107, in preprocess
self.spawn(pp_args)
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/ccompiler.py", line 
909, in spawn
spawn(cmd, dry_run=self.dry_run)
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/spawn.py", line 36, 
in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/spawn.py", line 159, 
in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'xlc_r' failed with exit status 40

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File 
"/data/prj/python/git/xlc-python3-3.7/Lib/distutils/tests/test_config_cmd.py", 
line 49, in test_search_cpp
match = cmd.search_cpp(pattern='xxx', body='/* xxx */')
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/command/config.py", 
line 201, in search_cpp
src, out = self._preprocess(body, headers, include_dirs, lang)
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/command/config.py", 
line 124, in _preprocess
self.compiler.preprocess(src, out, include_dirs=include_dirs)
  File "/data/prj/python/git/xlc-python3-3.7/Lib/distutils/unixccompiler.py", 
line 109, in preprocess
raise CompileError(msg)
distutils.errors.CompileError: command 'xlc_r' failed with exit status 40

and it is now:
==
ERROR: test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase)
--
Traceback (most recent call last):
  File 
"/data/prj/python/git/python3-3.7/Lib/distutils/tests/test_config_cmd.py", line 
49, in test_search_cpp
match = cmd.search_cpp(pattern='xxx', body='/* xxx */')
  File "/data/prj/python/git/python3-3.7/Lib/distutils/command/config.py", line 
206, in search_cpp
file = open(out)
FileNotFoundError: [Errno 2] No such file or directory: '_configtest.i'

*
The code I do not know how to fix is:

File: Lib/distutils/command/config.py
def _preprocess(self, body, headers, include_dirs, lang):
src = self._gen_temp_sourcefile(body, headers, lang)
out = "_configtest.i"
self.temp_files.extend([src, out])
self.compiler.preprocess(src, out, include_dirs=include_dirs)
return (src, out)

See https://github.com/aixtools/cpython/tree/bpo-11191 for my changes

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32555] Encoding issues with the locale encoding

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +5047
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32558] Socketserver documentation : error in server example

[Tristan Colombo]

Tristan Colombo  added the comment:

Running on Debian Etch
Ok, my python3 was broken and links to python3.5.3
All my apologizes

--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32558] Socketserver documentation : error in server example

[Ned Deily]

Ned Deily  added the comment:

In what environment are you trying this?  The example works for me (on a 
current macOS) system.

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32558] Socketserver documentation : error in server example

[Tristan Colombo]

Tristan Colombo  added the comment:

Correction of my previous correction :
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)

'with' statement can not be used...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32558] Socketserver documentation : error in server example

[Tristan Colombo]

New submission from Tristan Colombo :

In the documentation, at page :
- https://docs.python.org/3.6/library/socketserver.html
- Section 21.21.4.1

In the server side code :
if __name__ == "__main__":
HOST, PORT = "localhost", 

# Create the server, binding to localhost on port 
with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server:
...

We got an error line 24 : 
with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server:
AttributeError: __exit__

socketserver.TCPServer object has no __exit__() method and can not be used in 
'with' statement.

Proposed correction :
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
with server:
...

--
assignee: docs@python
components: Documentation
messages: 309995
nosy: docs@python, tcolombo
priority: normal
severity: normal
status: open
title: Socketserver documentation : error in server example
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32555] Encoding issues with the locale encoding

[STINNER Victor]

STINNER Victor  added the comment:

Example of bug on FreeBSD 11:

haypo@freebsd$ LC_ALL=C ./python -c 'import locale, os; 
locale.setlocale(locale.LC_ALL, "fr_FR.ISO8859-1"); 
print(ascii(os.strerror(2)))'

'Fichier ou r\udce9pertoire inexistant'

Expected result:

haypo@freebsd$ LC_ALL=fr_FR.ISO8859-1 ./python -c 'import locale, os; 
locale.setlocale(locale.LC_ALL, ""); print(ascii(os.strerror(2)))'

'Fichier ou r\xe9pertoire inexistant'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +5046

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

STINNER Victor  added the comment:

lc_numeric.py contains a typo, used fixed lc_numeric2.py instead to test my PR 
5191 which fixes decimal.Decimal.

--
Added file: https://bugs.python.org/file47386/lc_numeric2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +5045

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32557] allow shutil.disk_usage to take a file path

[Eryk Sun]

New submission from Eryk Sun :

Issue 26330 was resolved by documenting that shutil.disk_usage requires a 
directory. However, the shutil module is in a position to harmonize 
cross-platform behavior in ways that aren't normally possible or recommended in 
the low-level os module. To that end, the Windows implementation could retry 
calling nt._getdiskusage on the resolved parent directory in case of 
NotADirectoryError. For example:

def disk_usage(path):
try:
total, free = nt._getdiskusage(path)
except NotADirectoryError:
path = os.path.dirname(nt._getfinalpathname(path))
total, free = nt._getdiskusage(path)
used = total - free
return _ntuple_diskusage(total, used, free)

Alternatively, this could be addressed in the implementation of 
nt._getdiskusage itself.

--
components: Library (Lib), Windows
messages: 309992
nosy: eryksun, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
stage: needs patch
status: open
title: allow shutil.disk_usage to take a file path
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32544] Speed up hasattr(o, name) and getattr(o, name, default)

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I considered the idea of implementing _PyObject_GetAttrWithoutError() for 
issue31572, but dropped it because the code simplification was too small. But 
if this has performance benefits I'm +1 to implementing it. I'll update my 
still open PRs to use this API.

--
components: +Interpreter Core
nosy: +serhiy.storchaka
type:  -> performance
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32556] support bytes paths in nt _getdiskusage, _getvolumepathname, and _getfinalpathname

[Eryk Sun]

New submission from Eryk Sun :

In issue 26330 it was noted that nt._getdiskusage doesn't accept bytes paths. 
The same applies to nt._getvolumepathname and nt._getfinalpathname. Callers of 
these functions shouldn't have to manually implement PEP 529 UTF-8 support to 
handle input and output bytes paths. 

On the other hand, nt._getfullpathname works with a bytes path since it takes a 
`path_t` argument instead of `unicode` or `Py_UNICODE`. The above 3 functions 
could be rewritten to use `path_t` for the input path, and then encode the 
output path via PyUnicode_EncodeFSDefault if path->narrow is set.

--
components: Library (Lib), Unicode, Windows
messages: 309990
nosy: eryksun, ezio.melotti, paul.moore, steve.dower, tim.golden, vstinner, 
zach.ware
priority: normal
severity: normal
stage: needs patch
status: open
title: support bytes paths in nt _getdiskusage, _getvolumepathname, and 
_getfinalpathname
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

STINNER Victor  added the comment:


New changeset cb064fc2321ce8673fe365e9ef60445a27657f54 by Victor Stinner in 
branch 'master':
bpo-31900: Fix localeconv() encoding for LC_NUMERIC (#4174)
https://github.com/python/cpython/commit/cb064fc2321ce8673fe365e9ef60445a27657f54


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

STINNER Victor  added the comment:

On macOS 10.13.2, I failed to find any non-ASCII decimal_point or thousands_sep 
in localeconv(). I wrote a script to find all non-ASCII data in all locales:
https://github.com/vstinner/misc/blob/master/python/all_locales.py

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

STINNER Victor  added the comment:

Test on Linux (Fedora 27, glibc 2.26):

locale.setlocale(locale.LC_ALL, "fr_FR")
locale.setlocale(locale.LC_NUMERIC, "es_MX.utf8")

It works as expected, result:

decimal_point: '.'
thousands_sep: '\u2009'

Python 3.6 returns mojibake:

decimal_point: '.'
thousands_sep: '\xe2\x80\x89'

Python 2.7 raw strings, thousands_sep = b'\xE2\x80\x89'.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

STINNER Victor  added the comment:

I tested localeconv() with PR 4174 on FreeBSD:
--
locale.setlocale(locale.LC_ALL, "C")
locale.setlocale(locale.LC_NUMERIC, "ar_SA.UTF-8")
--

It works as expected, result:
--
decimal_point: '\u066b'
thousands_sep: '\u066c'
--

Compare it to Python 3.6 which returns mojibake, it seems like bytes are 
decoded from Latin1:
--
decimal_point: '\xd9\xab'
thousands_sep: '\xd9\xac'
--

Raw byte strings, Python 2.7:

* decimal_point: b'\xd9\xab'
* thousands_sep: b'\xd9\xac'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26330] shutil.disk_usage() on Windows can't properly handle unicode

[Mariatta Wijaya]

Mariatta Wijaya  added the comment:

Thanks!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26330] shutil.disk_usage() on Windows can't properly handle unicode

[Mariatta Wijaya]

Mariatta Wijaya  added the comment:


New changeset fb8569e36f2629654d5bc9c7ba05978edce408f4 by Mariatta (Miss 
Islington (bot)) in branch '3.6':
bpo-26330: Update shutil.disk_usage() documentation (GH-5184) (GH-5188)
https://github.com/python/cpython/commit/fb8569e36f2629654d5bc9c7ba05978edce408f4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32534] Speed-up list.insert

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I'm quite surprised so many benchmarks would be speeded up so significantly by 
a list.insert() optimization (why a 27% speedup on computing digits of pi, or a 
33% speedup on a N-body simulation?).  Are you sure the two executables are 
similarly compiled?

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11191] test_search_cpp error on AIX (with xlc)

[Mark Lawrence]

Change by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11191] test_search_cpp error on AIX (with xlc)

[Michael Felt]

Change by Michael Felt :


--
versions: +Python 3.7 -Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11191] test_search_cpp error on AIX (with xlc)

[Michael Felt]

Michael Felt  added the comment:

After even more years - I see the same test failing, just a bit different.

And, others

in short:

./python Lib/test/test_distutils.py
...
Ran 245 tests in 10.337s

FAILED (errors=7, skipped=31)

I have managed to get this to:

FAILED (errors=1, skipped=37)

And this is still:

ERROR: test_search_cpp (distutils.tests.test_config_cmd.ConfigTestCase)

So, moving forward now.

--
nosy: +Michael.Felt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

Sounds like a good compromise :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

STINNER Victor  added the comment:

> I would not consider this a bug in Python, but rather in the locale settings 
> passed to setlocale().

Past 10 years, I repeated to every single user I met that "Python 3 is right, 
your system setup is wrong". But that's a waste of time. People continue to 
associate Python3 and Unicode to annoying bugs, because they don't understand 
how locales work.

Instead of having to repeat to each user that "hum, maybe your config is 
wrong", I prefer to support this non convential setup and work as expected ("it 
just works"). With my latest implementation, setlocale() is only done when 
LC_CTYPE and LC_NUMERIC are different, which is the corner case which 
"shouldn't occur in practice".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32513] dataclasses: make it easier to use user-supplied special methods

[Eric V. Smith]

Eric V. Smith  added the comment:

See #32546 for the hash=None problem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

Indeed. The major problem with all libc locale functions is that they are not 
thread safe. The GIL does help a bit protecting against corrupted data, though.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

STINNER Victor  added the comment:

The technical issue here is that the libc has no "stateless" function to 
process bytes and text with one specific locale. All functions rely on the 
*current* locales. To decode byte strings, we use mbstowcs(), and this function 
relies on the current LC_CTYPE locale, whereas decimal_point and thousands_sep 
should be decoded from the current LC_NUMERIC locale.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

Ok, it seems that the C setlocale() itself does not follow the conventions set 
forth for environment variables:

http://pubs.opengroup.org/onlinepubs/7908799/xsh/setlocale.html

(see the example at the bottom)

So the behavior shown by Python's setlocale() is fine.

However, that still doesn't magically make this work:

locale.setlocale(locale.LC_ALL, 'C.UTF-8')
locale.setlocale(locale.LC_NUMERIC, 'fr_FR.ISO8859-1')

If LC_NUMERIC uses a different encoding than LC_ALL, there's really no surprise 
in having numeric formatting fail. localeconv() will output the set encoding 
for the numeric string conversion and Python will decode this using the locale 
encoding set by LC_ALL. If those two are different, you run into problems.

I would not consider this a bug in Python, but rather in the locale settings 
passed to setlocale().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32513] dataclasses: make it easier to use user-supplied special methods

[Eric V. Smith]

Eric V. Smith  added the comment:

Another case to consider is that if a class defines __eq__, Python will 
automatically set hash=False before the decorator gets involved. NOte to self: 
be sure to add a test case for this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

STINNER Victor  added the comment:

Example of Fedora 27 and Python 3.6:

vstinner@apu$ env -i LC_NUMERIC=uk_UA.koi8u python3 -c 'import locale; 
print(locale.setlocale(locale.LC_ALL, "")); 
print(locale.getpreferredencoding(), 
ascii(locale.localeconv()["thousands_sep"]))'
LC_CTYPE=C.UTF-8;LC_NUMERIC=uk_UA.koi8u;LC_TIME=C;LC_COLLATE=C;LC_MONETARY=C;LC_MESSAGES=C;LC_PAPER=C;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=C;LC_IDENTIFICATION=C
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib64/python3.6/locale.py", line 110, in localeconv
d = _localeconv()
UnicodeDecodeError: 'locale' codec can't decode byte 0x9a in position 0: 
Invalid or incomplete multibyte or wide character

"env -i" starts Python in an empty environment. It seems like LC_CTYPE defaults 
to C.UTF-8 in this case.

* LC_CTYPE = C.UTF-8, encoding = UTF-8
* LC_NUMERIC = uk_UA.koi8u, encoding = KOI8-U


With my PR, it works:

vstinner@apu$ env -i LC_NUMERIC=uk_UA.koi8u ./python -c 'import locale; 
print(locale.setlocale(locale.LC_ALL, "")); 
print(locale.getpreferredencoding(), 
ascii(locale.localeconv()["thousands_sep"]))'
LC_CTYPE=C.UTF-8;LC_NUMERIC=uk_UA.koi8u;LC_TIME=C;LC_COLLATE=C;LC_MONETARY=C;LC_MESSAGES=C;LC_PAPER=C;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=C;LC_IDENTIFICATION=C
UTF-8 '\xa0'

=> thousands_sep byte string b'\x9A' is decoded as the Uniode string '\xa0'.


vstinner@apu$ env -i LC_NUMERIC=uk_UA.koi8u ./python -c 'import locale; 
locale.setlocale(locale.LC_ALL, ""); print(ascii(f"{1234:n}"))'
'1\xa0234'

=> the number is properly formatted


vstinner@apu$ env -i LC_NUMERIC=uk_UA.koi8u ./python -c 'import locale; 
locale.setlocale(locale.LC_ALL, ""); print(f"{1234:n}")'
1 234

It's possible to display the result using print().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

STINNER Victor  added the comment:

Marc-Andre Lemburg: "If you first set LC_ALL and then one of the other 
categories such as LC_NUMERIC, locale C functions will still use the LC_ALL 
setting for everything. LC_NUMERIC does not override the LC_ALL setting."

The root of this issue is 
https://bugzilla.redhat.com/show_bug.cgi?id=1484497#c0:

Petr Viktorin reproducer scripts uses Python locale.setlocale(), not 
environment variables:
https://gist.github.com/encukou/70b3d3f9ef3e29ac1dbc23a5f7bd6431
---
locale.setlocale(locale.LC_ALL, 'C.UTF-8')
locale.setlocale(locale.LC_NUMERIC, 'fr_FR.ISO8859-1')
---

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29708] support reproducible Python builds

[Bernhard M. Wiedemann]

Bernhard M. Wiedemann  added the comment:

I think, there is no single nice and clean solution with time-based .pyc files, 
but to get a whole distribution to build reproducibly, there are two other ways:

1) if the SOURCE_DATE_EPOCH environment variable is set,
make hash-based .pyc files the default.

2) instead of storing .py mtime in the .pyc header, use the .pyc's filesystem 
mtime value - also making it more available to users.
Not sure if this would have side-effects or cause regressions.

on the side-issue: IMHO checking exact mtimes is the right thing to do, because 
sometimes users will copy back old .py files and expect mismatching .pyc files 
to not be used.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[Stefan Krah]

Stefan Krah  added the comment:

On Mon, Jan 15, 2018 at 12:37:28PM +, Marc-Andre Lemburg wrote:
> If you first set LC_ALL and then one of the other categories such as 
> LC_NUMERIC, locale C functions will still use the LC_ALL setting for 
> everything. LC_NUMERIC does not override the LC_ALL setting.

I have the exact same questions as Marc-Andre.  This is one of the reasons why I
blocked the _decimal change.  I don't fully understand the role of the new 
glibc,
since #7442 has existed for ages -- and it is a open question whether it is a 
bug
or not.

Both views are reasonable IMO.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

I just wanted to note that the description and title may cause a wrong 
interpretation of what should happen:

If you first set LC_ALL and then one of the other categories such as 
LC_NUMERIC, locale C functions will still use the LC_ALL setting for 
everything. LC_NUMERIC does not override the LC_ALL setting.

I tested this on OpenSUSE and get the same wrong results. Apparently, 
locale.localeconv() does not respect the above order. That's a bug.

I'm not sure whether the OP's quoted behavior is a bug, though, since if the 
locale encoding is not UTF-8, you cannot really expect using UTF-8 numeric 
separators to output correctly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

STINNER Victor  added the comment:

> Please confirm the bug without having LC_ALL or LANG set.

lc_numeric.py uses:

  locale.setlocale(locale.LC_ALL, "fr_FR")

Are you talking about that? What is the problem with this configuration?

I'm sure that there is a bug :-) You aren't able to reproduce it? What is your 
operating system?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32516] Add a shared library mechanism for win32

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

>From experience with doing something similar in egenix-pyopenssl, I recommend 
>putting the DLLs into the same directory as the PYD file on Windows. If you 
>want to be extra safe, you can explicitly load the DLL, but normally this is 
>not needed.

On Linux and other OSes, it's best to dlopen() to explicitly load the lib, 
since rpath and OS search paths are not always reliable.

--
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32521] NIS module fails to build due to the removal of interfaces related to Sun RPC from glibc.

[Christian Heimes]

Christian Heimes  added the comment:

It's not just the headers. The shared library is now /usr/lib64/nsl/libnsl.so.2 
instead of /usr/lib64/libnsl.so.1.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32521] NIS module fails to build due to the removal of interfaces related to Sun RPC from glibc.

[Christian Heimes]

Change by Christian Heimes :


--
pull_requests: +5044

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

Just FYI: LC_ALL has precedence over all other more specific LC_* settings:

http://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html
http://man7.org/linux/man-pages/man7/locale.7.html

Please confirm the bug without having LC_ALL or LANG set. Thanks.

--
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32555] Encoding issues with the locale encoding

[STINNER Victor]

STINNER Victor  added the comment:

Another issue: _Py_DecodeUTF8Ex() creates surrogate pairs with 16-bit wchar_t 
(on Windows), whereas input bytes should be escaped. I'm quite sure that it's a 
bug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32555] Encoding issues with the locale encoding

[STINNER Victor]

STINNER Victor  added the comment:

I'm not sure that locale.bindtextdomain() uses the right encoding neither. I 
propose the following fix:

diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c
index 324b694b83..1de17d3620 100644
--- a/Modules/_localemodule.c
+++ b/Modules/_localemodule.c
@@ -555,7 +555,7 @@ PyIntl_bindtextdomain(PyObject* self,PyObject*args)
 PyErr_SetFromErrno(PyExc_OSError);
 return NULL;
 }
-result = PyUnicode_DecodeLocale(current_dirname, NULL);
+result = PyUnicode_DecodeFSDefault(current_dirname);
 Py_XDECREF(dirname_bytes);
 return result;
 }

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32555] Encoding issues with the locale encoding

[STINNER Victor]

New submission from STINNER Victor :

Python 3.6 doesn't use the right encoding in os.strerror(), time.stftime(), 
locale.localeconv(), time.tzname, etc. on macOS, FreeBSD and other platforms.

See my fix locale encodings in bpo-29240: commit 
7ed7aead9503102d2ed316175f198104e0cd674c, and test_all_locales.py attached to 
bpo-29240.

See also the bpo-31900 for locale.localeconv() encoding issue when LC_NUMERIC 
encoding is different than the LC_CTYPE encoding.

--
components: Unicode
messages: 309963
nosy: ezio.melotti, vstinner
priority: normal
severity: normal
status: open
title: Encoding issues with the locale encoding
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29240] PEP 540: Add a new UTF-8 mode

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +5043

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31900] localeconv() should decode numeric fields from LC_NUMERIC encoding, not from LC_CTYPE encoding

[STINNER Victor]

STINNER Victor  added the comment:

Oops lc_numeric.py contains a typo:

d = decimal.Decimal(1234)
print("Decimal.__format__: %a" % f"{i:n}")

=> it should be f"{d:n}"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com