[issue35559] Optimize base64.b16decode to use compiled regex

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

How this affects the import time (use -X importtime)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35469] [2.7] time.asctime() regression

[Antti Haapala]

Antti Haapala  added the comment:

C11 specifies the format used by asctime as being exactly 

"%.3s %.3s%3d %.2d:%.2d:%.2d %d\n",

which matches the *new* output with space padding, less the newline.

As always, Microsoft got it wrong: 
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/asctime-wasctime?view=vs-2017
 - even if deliberately saying 1-31 instead of 01-31 in the table.

--
nosy: +ztane

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35559] Optimize base64.b16decode to use compiled regex

[Stefan Behnel]

Stefan Behnel  added the comment:

One regex related code pattern that I generally like is to assign bound methods 
to good names and use those. In this case, I would write

_has_non_base16_digits = re.compile(b'[^0-9A-F]').search
...
if _has_non_base16_digits(s):
raise ...

--
nosy: +scoder

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35559] Optimize base64.b16decode to use compiled regex

[Karthikeyan Singaravelan]

New submission from Karthikeyan Singaravelan :

I came across this as a result of issue35557 and thought to make a new issue to 
keep the discussion separate. Currently the b16decode function uses a regex 
with re.search that can be compiled at the module level as a static variable to 
give up to 30% improvement when executed on Python 3.7. I am proposing a PR for 
this change since it looks safe to me.

$ python3 -m perf compare_to default.json optimized.json --table
++-+--+
| Benchmark  | default | optimized|
++=+==+
| b16decode  | 2.97 us | 2.03 us: 1.46x faster (-32%) |
++-+--+
| b16decode_casefold | 3.18 us | 2.19 us: 1.45x faster (-31%) |
++-+--+

Benchmark script : 

import perf
import re
import binascii
import base64

_B16DECODE_PAT = re.compile(b'[^0-9A-F]')

def b16decode_re_compiled_search(s, casefold=False):
s = base64._bytes_from_decode_data(s)
if casefold:
s = s.upper()
if _B16DECODE_PAT.search(s):
raise binascii.Error('Non-base16 digit found')
return binascii.unhexlify(s)

if __name__ == "__main__":
hex_data = 
"806903d098eb50957b1b376385f233bb3a5d54f54191c8536aefee21fc9ba3ca"
hex_data_upper = hex_data.upper()

assert base64.b16decode(hex_data_upper) == 
b16decode_re_compiled_search(hex_data_upper)
assert base64.b16decode(hex_data, casefold=True) == 
b16decode_re_compiled_search(hex_data, casefold=True)

runner = perf.Runner()
if True: # toggle to False for default.json
runner.timeit(name="b16decode",
  stmt="b16decode_re_compiled_search(hex_data_upper)",
  setup="from __main__ import b16decode_re_compiled_search, 
hex_data, hex_data_upper")
runner.timeit(name="b16decode_casefold",
  stmt="b16decode_re_compiled_search(hex_data, 
casefold=True)",
  setup="from __main__ import b16decode_re_compiled_search, 
hex_data, hex_data_upper")
else:
runner.timeit(name="b16decode",
  stmt="base64.b16decode(hex_data_upper)",
  setup="from __main__ import hex_data, hex_data_upper; 
import base64")
runner.timeit(name="b16decode_casefold",
  stmt="base64.b16decode(hex_data, casefold=True)",
  setup="from __main__ import hex_data, hex_data_upper; 
import base64")

--
assignee: xtreak
components: Library (Lib)
messages: 332330
nosy: djhoulihan, serhiy.storchaka, xtreak
priority: normal
severity: normal
status: open
title: Optimize base64.b16decode to use compiled regex
type: performance
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35557] Allow lowercase hexadecimal characters in base64.b16decode()

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

This is compatibility breaking change. Furthermore, it changes the purposed 
behavior that was from the initial version 
(4c904d1bf71d01bc22fbb123493f975050560e9c).

Since currently there is an option which allows to accept lowercase hexadecimal 
characters, I do not see a need in this change. You can also use 
bytes.fromhex().

--
nosy: +barry, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35555] IDLE: Gray out Code Context on non-editor windows

[Terry J. Reedy]

Terry J. Reedy  added the comment:

PR 11282 should have been listed above.

--
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22703] Idle Code Context menu entrie(s)

[Terry J. Reedy]

Change by Terry J. Reedy :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22703] Idle Code Context menu entrie(s)

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Both pr-11214 and the pr-11286 (3.7 backport, not listed below) have been 
merged.

Tal: I am pretty sure the Options menu should continue to work on Macs, but a 
test to verify would be helpful.

--
nosy: +taleinat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35557] Allow lowercase hexadecimal characters in base64.b16decode()

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Thanks for the report. A couple of points as below : 

* This changes the interface of the function by removing a parameter. Thus it 
will break compatibility with Python 2 and also earlier versions of Python 3. 
Removing a parameter in the signature has to go through a deprecation cycle if 
this is going to be accepted.
* Please don't use time.time and mean for benchmarks that can be misleading. 
There are modules like timeit and perf (https://pypi.org/project/perf/) that 
are more reliable.

I looked for some more inefficiencies and I can see re.search for every run. 
Perhaps re.compile can be used to store the compiled regex at module level and 
then to match against the string. This makes the function 25% faster without 
changing the interface. In case casefold=False then an extra call to make the 
string upper case is avoided giving some more benefit.

With re.search inside the function

$ python3.7 -m perf timeit -s 'import base64; 
hex_data="806903d098eb50957b1b376385f233bb3a5d54f54191c8536aefee21fc9ba3ca"' 
'base64.b16decode(hex_data, casefold=True)'
.
Mean +- std dev: 3.08 us +- 0.22 us
$ python3.7 -m perf timeit -s 'import base64; 
hex_data="806903d098eb50957b1b376385f233bb3a5d54f54191c8536aefee21fc9ba3ca".upper()'
 'base64.b16decode(hex_data)'
.
Mean +- std dev: 2.93 us +- 0.20 us

With the regex compiled to a variable at the module level

$ python3.7 -m perf timeit -s 'import base64; 
hex_data="806903d098eb50957b1b376385f233bb3a5d54f54191c8536aefee21fc9ba3ca"' 
'base64.b16decode(hex_data, casefold=True)'
.
Mean +- std dev: 2.08 us +- 0.15 us
$ python3.7 -m perf timeit -s 'import base64; 
hex_data="806903d098eb50957b1b376385f233bb3a5d54f54191c8536aefee21fc9ba3ca".upper()'
 'base64.b16decode(hex_data)'
.
Mean +- std dev: 1.98 us +- 0.17 us


Since this is a comparison of fixed set of elements I tried using a set of 
elements and any to short-circuit but it seems to be slower

$ python3.7 -m perf timeit -s 'import base64; 
hex_data="806903d098eb50957b1b376385f233bb3a5d54f54191c8536aefee21fc9ba3ca"' 
'base64.b16decode(hex_data, casefold=True)'
.
Mean +- std dev: 8.21 us +- 0.66 us


I am opening a PR to use the compiled regex at the module level since I see it 
as a net win of 25-30% without any interface change or test case changes 
required.

--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35549] Add globbing to unicodedata.lookup

[Roman Inflianskas]

Roman Inflianskas  added the comment:

I like your proposal with globbing, steven.daprano.

I updated the title.

--
title: Add partial_match: bool = False argument to unicodedata.lookup -> Add 
globbing to unicodedata.lookup

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32409] venv activate.bat is UTF-8 encoded but uses current console codepage

[Vinay Sajip]

Vinay Sajip  added the comment:

See also bpo-35558.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35558] venv: running activate.bat gives ' parameter format not correct - 65001'

[Vinay Sajip]

Vinay Sajip  added the comment:

This seems related to bpo-32409.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35530] Counter-intuitive logging API

[Vinay Sajip]

Vinay Sajip  added the comment:

> If this behaviour can't be changed for backwards compatibility reasons, then 
> so be it.

It can't be changed for this reason.

> But I think it would be disingenuous to claim it's not a design flaw.

Do you think *I'm* being disingenuous, Mark? I don't mean to be. As well as 
developers from Enthought, who write large, well-tested libraries, logging also 
has to satisfy the newbie who writes a one-off script that probably won't even 
be tested via a test runner. For that use case, logging.debug() and friends 
call basicConfig() under the hood so that simple usage is as easy as possible. 
So the "accidental" configuring of logging only occurs if nothing has 
previously been configured and logging.debug() or similar is called. This is 
documented, as has been mentioned. 

Libraries are not supposed to configure logging except using a NullHandler, and 
this is also documented, but there are numerous authors of libraries that 
ignore that advice. If this causes problems to users of those libraries, that 
is down to the authors of those libraries rather than logging itself, and 
issues should be logged against those libraries. I maintain that calling 
basicConfig() in logging.XXX() is not accidental or a design flaw that crept in 
somehow, but was considered specifically in the context of logging in simple 
cases. This is documented in the PEP:

https://www.python.org/dev/peps/pep-0282/#id30

This PEP was the basis for discussion on python-dev before the logging package 
was accepted into Python.

What guidelines does Enthought provide to its developers for configuring Python 
logging and its usage in libraries and unit tests? If this is public, I would 
be interested to see it. Are there any specific third-party libraries you have 
come across which Enthought uses regularly/repeatedly, which misconfigure 
logging in the way you describe? I would also be curious to know about those.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35485] Mac: tkinter windows turn black while resized

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Raymond, as Ned explains above, there seem to be no really good options for the 
upcoming releases on Mac.  The *ed paragraph below has my best guess at to 
whether a Mac-IDLE user should install or ignore.
---

Basic options A, to hold up the release (indefinitely) on all platforms, has 
precedents, but usually because of bug(s) affecting more than one platform, and 
usually AFAIK for at most a week.  We have already closed 3.6 to routine 
maintenance and merged multiple patches into the future 3.7.3.  Withholding the 
current set of bug fixes does no one any good except possibly for people using 
resizable tkinter windows on Mac.

Option B, to hold up the release on only one platform is unprecedented as far 
as I know.  Option C, to release with one more known bug than we would like is 
relatively normal.  Given the indefiniteness of a fix, I agree with C.

AS for which version of C, I agree that option 2 above seems least bad.  The 
main issue with option 1, staying with 8.6.8, is how it performs (or rather, 
fails to perform)  on the new macOS Mohave.  Before opening this issue, I did 
fairly extensive testing of IDLE 3.7.2rc1 on Mohave and this was the only 
problem I encountered.

* As far as IDLE goes, what's new in 3.7.2/3.6.8 are mostly doc changes that 
are already online.  A user who has not upgraded to Mohave could consider 
staying with the existing releases. I suspect that a user running Mohave will 
be happier with the new releases with the tcl/tk upgrade. 

I thought of augmenting the the interactive splash notice, but this would give 
too much attention and emphasis to only 1 of many known but unfixed bugs, and 1 
that only affects resizable tkinter windows on Mac.

Perhaps the hopefully temporary Mac installers should be labeled internally and 
on the splash screen as ...rc2 even if temporarily included on the 'final' 
download page with the final releases for other platforms.  I believe some 
people would object is two slightly different releases were called '3.6.8' and 
'3.7.2'.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35558] venv: running activate.bat gives ' parameter format not correct - 65001'

[Ned Deily]

Change by Ned Deily :


--
nosy: +vinay.sajip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35558] venv: running activate.bat gives ' parameter format not correct - 65001'

[Nils Lindemann]

New submission from Nils Lindemann :

Windows 7, Python 3.7.1:260ec2c36a

after doing

C:\python\python -m venv C:\myvenv

and then

C:\>myvenv\Scripts\activate.bat

it prints

parameter format not correct - 65001

However, it activates the venv - the prompt shows

(myvenv) C:\>

and

C:\myvenv\Scripts;

gets prepended to PATH.

When i outcomment

for /f "tokens=2 delims=:" %%a in ('"%SystemRoot%\System32\chcp.com"') do (
set "_OLD_CODEPAGE=%%a"
)

in the activate.bat then the message wont show up.

related: 
https://stackoverflow.com/questions/51358202/python-3-7-activate-venv-error-parameter-format-not-correct-65001-windows

--
messages: 332320
nosy: Nils-Hero
priority: normal
severity: normal
status: open
title: venv: running activate.bat gives ' parameter format not correct - 65001'
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35557] Allow lowercase hexadecimal characters in base64.b16decode()

[Dylan Houlihan]

Change by Dylan Houlihan :


--
keywords: +patch
pull_requests: +10512
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35557] Allow lowercase hexadecimal characters in base64.b16decode()

[Dylan Houlihan]

New submission from Dylan Houlihan :

Currently, the `base64` method `b16decode` does not decode a hexadecimal string 
with lowercase characters by default. To do so requires passing `casefold=True` 
as the second argument. I propose a change to the `b16decode` method to allow 
it to accept hexadecimal strings containing lowercase characters without 
requiring the `casefold` argument.

The revision itself is straightforward. We simply have to amend the regular 
expression to match the lowercase characters a-f in addition to A-F. Likewise 
the corresponding tests in Lib/base64.py also need to be changed to account for 
the lack of a second argument. Therefore there are two files total which need 
to be refactored.

In my view, there are several compelling reasons for this change:

1. There is a nontrivial performance improvement. I've already made the changes 
on my own test branch[1] and I see a mean decoding performance improvement of 
approximately 9.4% (tested by taking the average of 1,000,000 hexadecimal 
string encodings). The testing details are included in a file attached to this 
issue.

2. Hexadecimal strings are case insensitive, i.e. 8DEF is equivalent to 8def. 
This is the particularly motivating reason why I've written the patch - there 
have been many times when I've been momentarily confounded by a hexadecimal 
string that won't decode only to realize I'm yet again passing in a lowercase 
string.

3. The behavior of the underlying method in `binascii`, `unhexlify`, accepts 
both uppercase and lowercase characters by default without requiring a second 
parameter. From the perspective of code hygiene and language consistency, I 
think it's both more practical and more elegant for the language to behave in 
the same, predictable manner (particularly because `base64.py` simply calls 
`binascii.c` under the hood). Additionally, the `binascii` method `hexlify` 
actually outputs strings in lowercase encoding, meaning that any use of both 
`binascii` and `base64` in the same application will have to consistently do a 
`casefold` conversion if output from `binascii.hexlify` is fed back as input to 
`base64.b16decode` for some reason.

There are two arguments against this patch, as far as I can see it:

1. In the relevant IETF reference documentation (RFC3548[2], referenced 
directly in the `b16decode` docstring; and RFC4648[3] with supersedes it), 
under Security Considerations the author Simon Josefsson claims that there 
exists a potential side channel security issue intrinsic to accepting case 
insensitive hexadecimal strings in a decoding function. While I'm not 
dismissing this out of hand, I personally do not find the claimed vulnerability 
compelling, and Josefsson does not clarify a real world attack scenario or 
threat model. I think it's important we challenge this assumption in light of 
the potential nontrivial improvements to both language consistency and 
performance. I would be very interested in hearing a real threat model here 
that would practically exist outside of a very contrived scenario. Moreover if 
this is such a security issue, why is the behavior already evident in 
`binascii.unhexlify`?

2. The other reason may be that there's simply no reason to make such a change. 
An argument can be put forward that a developer won't frequently have to deal 
with this issue because the opposite method, `b16encode`, produces hexadecimal 
strings with uppercase characters. However, in my experience hexadecimal 
strings with lowercase characters are extremely common in situations where 
developers haven't produced the strings themselves in the language.

As I mentioned, I have already written the changes on my own patch branch. I'll 
open a pull request once this issue has been created and reference the issue in 
the pull request on GitHub.

References:

1. https://github.com/djhoulihan/cpython/tree/base64_case_sensitivity

2. https://tools.ietf.org/html/rfc3548

3. https://tools.ietf.org/html/rfc4648

--
components: Library (Lib)
files: testing_data.txt
messages: 332319
nosy: djhoulihan
priority: normal
severity: normal
status: open
title: Allow lowercase hexadecimal characters in base64.b16decode()
type: performance
versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file48013/testing_data.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11566] hypot define in pyconfig.h clashes with g++'s cmath

[INADA Naoki]

Change by INADA Naoki :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed
versions: +Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35549] Add partial_match: bool = False argument to unicodedata.lookup

[Steven D'Aprano]

Steven D'Aprano  added the comment:

Here's my implementation:

from unicodedata import name
from unicodedata import lookup as _lookup
from fnmatch import translate
from re import compile, I

_NAMES = None

def getnames():
global _NAMES
if _NAMES is None:
_NAMES = []
for i in range(0x11):
s = name(chr(i), '')
if s:
_NAMES.append(s)
return _NAMES

def lookup(name_or_glob):
if any(c in name_or_glob for c in '*?['):
match = compile(translate(name_or_glob), flags=I).match
return [name for name in getnames() if match(name)]
else:
return _lookup(name_or_glob)




The major limitation of my implementation is that it doesn't match name aliases 
or sequences.

http://www.unicode.org/Public/11.0.0/ucd/NameAliases.txt
http://www.unicode.org/Public/11.0.0/ucd/NamedSequences.txt

For example:

lookup('TAMIL SYLLABLE TAA?')  # NamedSequence

ought to return ['தா'] but doesn't.

Parts of the Unicode documentation uses the convention that canonical names are 
in UPPERCASE, aliases are lowercase, and sequences are in Mixed Case. and I 
think that we should follow that convention:

http://www.unicode.org/charts/aboutcharindex.html

That makes it easy to see what is the canonical name and what isn't.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35549] Add partial_match: bool = False argument to unicodedata.lookup

[Steven D'Aprano]

Steven D'Aprano  added the comment:

I love the idea, but dislike the proposed interface.

As a general rule of thumb, Guido dislikes "constant bool parameters", where 
you pass a literal True or False to a parameter to a function to change its 
behaviour. Obviously this is not a hard rule, there are functions in the stdlib 
that do this, but like Guido I think we should avoid them in general.

Instead, I think we should allow the name to include globbing symbols * ? etc. 
(I think full blown re syntax is overkill.) I have an implementation which I 
use:

lookup(name) -> single character # the current behaviour

lookup(name_with_glob_symbols) -> list of characters

For example lookup('latin * Z') returns:

['LATIN CAPITAL LETTER Z', 'LATIN SMALL LETTER Z', 'LATIN CAPITAL LETTER D WITH 
SMALL LETTER Z', 'LATIN LETTER SMALL CAPITAL Z', 'LATIN CAPITAL LETTER 
VISIGOTHIC Z', 'LATIN SMALL LETTER VISIGOTHIC Z']


A straight substring match takes at worst twelve extra characters:

lookup('*' + name + '*')

and only two if the name is a literal:

lookup('*spam*')

This is less than `partial_match=True` (18 characters) and more flexible and 
powerful. There's no ambiguity between the two styles of call because the 
globbing symbols * ? and [] are never legal in Unicode names. See section 4.8 of

http://www.unicode.org/versions/Unicode11.0.0/ch04.pdf

--
nosy: +steven.daprano

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35548] memoryview needlessly (?) requires represented object to be hashable

[Ilya Kulakov]

Ilya Kulakov  added the comment:

True, but perhaps it's too strict to require both memoryview and the 
represented object to be immutable?

The logic is as follows: 

Every object in Python can be seen as a view of some outside data (in memory, 
on disk etc.). And while Python's runtime may attempt to guarantee immutability 
of its objects, it cannot guarantee immutability of the outside data. Therefore 
a hashable object is an object immutable from the perspective of Python's 
runtime.
Memory views connect Python objects with outside data. They can be marked as 
immutable by Python's runtime. Therefore it should be sufficient to be hashable 
regardless of volatility of outside data.

In your example the immutability of the ndarray is indeed bypassed via a view 
created beforehand. But that's implementation detail and may actually be a flaw 
(i.e. being unable to propagate the change). The documentation [1] somewhat 
warns your that this behavior might change:

> However, **currently**, locking a base object does not lock any views that 
> already reference it, so under that circumstance it is possible to alter the 
> contents of a locked array via a previously created writeable view onto it.

1: 
https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.ndarray.flags.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35387] Dialogs on IDLE are accompanied by a small black window

[Ned Deily]

Ned Deily  added the comment:

Using the current tip of Tk core-8-6-branch, I also now see the problem of the 
extra blank (not necessarily black) window appearing (also it looks like a 
blank window at least sometimes briefly appears and disappears when IDLE 
launches).  This problem hasn't been seen with the release tarballs for 8.6.8 
or 8.6.9.1.  But there are other, serious problems seen with the current tip of 
Tk core-8-6-branch.  See the discussion in Issue35485 for more information and 
for followup discussion.  For the moment, I'm going to close this issue as a 
duplicate of Issue35485 so we don't have the discussion spread out over 
multiple issues.  If necessary, we can update and/or re-open this one.

--
priority: normal -> deferred blocker
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> Mac: tkinter windows turn black while resized
versions: +Python 2.7, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35556] See if frozen modules can use relative imports

[Brett Cannon]

New submission from Brett Cannon :

https://gregoryszorc.com/blog/2018/12/18/distributing-standalone-python-applications/
 claims it doesn't work.

--
components: Library (Lib)
messages: 332314
nosy: brett.cannon
priority: low
severity: normal
stage: test needed
status: open
title: See if frozen modules can use relative imports
type: behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35485] Mac: tkinter windows turn black while resized

[Ned Deily]

Ned Deily  added the comment:

At the moment, I am holding the 3.7.2 and 3.6.8 final releases for this and 
another unrelated regression.  I don't want to hold up these releases for all 
platforms just for issues related to use of Tk, tkinter, and/or IDLE on macOS.  
There doesn't seem to be any one really good option, though, pending real fixes 
or workarounds.  The options I see are:

1. revert to Tcl/Tk 8.6.8 as was used with 3.7.1 and 3.6.7.  At this point, I 
don't remember all of the problems seen with that version of Tk, though I 
believe one was the scrolling problem described in Issue34370.

2. continue with Tk 8.6.9.1 as shipped with 3.7.2rc1 and 3.6.8rc1.  As far as a 
I know at the moment, the only regression there is the problem originally 
described in this issue, that is, windows temporarily turning black while being 
resized.

3. use the tip of Tk core-8-6-branch despite the problems described above.

4. look at reverting to Tcl/Tk 8.5.x and hope it works on the latest releases 
of macOS.  (Note that the macOS Installers and the Tcl/Tk they use are built on 
macOS 10.9 and 10.6 depending on the installer variant.)

5. hold the release for Tk and/or tkinter and/or IDLE fixes

None of those are really good options.  I would rule out option 3 based on my 
quick tests. I don't think going to 8.5.x is a viable option at this point, 
either, and certainly not desirable.  I really don't want to hold the release 
for a permanent fix or fixes unless they were likely to be available in the 
next 24 hours or so, since this is strictly a python.org macOS issue (although 
other distributors of Python and Tk on macOS will likely run into these 
problems).

That leaves either option 1 or 2. Of the 2, at this point, I lean towards 
option 2 assuming no other regressions are noted, it wouldn't be the end of the 
world to live for awhile with the temporary black window while resizing.  It's 
ugly but probably not as ugly as the scrolling problems that would remain with 
option 1.

In any case, if we do go ahead and release with one of these compromises, I 
would document it in the release notes and would also push to release updated 
installers for these releases as soon as fixes are available, with the proviso 
that the updated installers vary from the original release installers only in 
Tk code and any necessary tkinter or IDLE change(s) to correct these issues, 
i.e. no changes in any other components in the release.

Any other opinions??

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33610] IDLE: Make multiple improvements to CodeContext

[Cheryl Sabella]

Cheryl Sabella  added the comment:

For M3 - #3.

--
dependencies: +IDLE: Gray out Code Context on non-editor windows

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35555] IDLE: Gray out Code Context on non-editor windows

[Cheryl Sabella]

New submission from Cheryl Sabella :

M3 from #33610.

Gray out menu entry when not applicable.

--
assignee: terry.reedy
components: IDLE
messages: 332311
nosy: cheryl.sabella, terry.reedy
priority: normal
severity: normal
status: open
title: IDLE: Gray out Code Context on non-editor windows
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35485] Mac: tkinter windows turn black while resized

[Ned Deily]

Ned Deily  added the comment:

Based on the discussion here and in Issue35387, I built and tested macOS 
installers for 3.7.2rc1+ and 3.6.8rc1+ using the current tip of the Tk 
core-8-6-branch (https://core.tcl.tk/tk/info/deb6ecc05e4202e3); I continued to 
build Tcl from the Tcl 8.6.9 release tarball.  As a reminder, for 3.7.2rc1 and 
3.6.8rc1, I used the Tk 8.6.9.1 release tarball with Tcl 8.6.9.

With the current Tk tip, the problem described in this issue seems to have been 
fixed: Tk windows did not turn black while they were being resized.

However, new regressions appeared.  First, the problem described in Issue35387 
is now visible, that is, a separate blank window appears when selecting either 
"About IDLE" or "Preferences" from the application cascade in the application's 
menu bar at the top of the screen.  (Note that when launching IDLE.app (by 
double-clicking on the IDLE icon, for example), that menu cascade is normally 
labeled "IDLE" but when launching IDLE from the command line, for example, with 
"python -m idledlib" the menu cascade will usually be labeled "Python" instead 
of "IDLE"; this is normal behavior.)  These days, it is also possible to bring 
up the IDLE's Preferences window by using the Options -> Configure IDLE menu.  
In that case, the spurious blank window does *not* appear.  If I recall 
correctly, both the About menu item and the Preferences menu items (but not the 
Options menu items) are special menu items automatically provided by Tk and/or 
macOS and that IDLE has code to customize, so that may be a
  good lead.

Second, I happened to notice that, when running a Python program with a syntax 
error from an IDLE edit window, the syntax error pop-down message appears 
twice.  A sequence to reproduce:
!. Open a new edit window (cmd-N).
2. Type: x = $1
3. Save to a file (cmd-S).
4. Run the program (F5).
5. The expected "Invalid syntax" panel pulls down on the edit window.
6. Click the OK button in the panel to dismiss.
7. The panel disappears but is immediately replaced by a second and identical 
"Invalid syntax" panel. (This is new and incorrect behavior).
8. Again click the OK button in the panel to dismiss.  This time the panel goes 
away and does not reappear.

Third, running Python test_idle causes a segfault.  From the traceback 
displayed, it appears that the failing test is calling idlelib/configdialog.py 
so I wouldn't be surprised if this failure is related to the first issue above, 
i.e. the appearance of the blank window when Preferences is chosen.  To 
reproduce, from a Terminal command line:

/path/to/python -m test -w -uall -j3 test_idle

where /path/to/python might be /usr/local/bin/python3.6 (or 3.7)

To make it easier to reproduce the problems, I've made test installers built 
with the Tk core-8-6-branch temporarily available on python.org.  These are 
similar to but not exactly the same as what is planned for 3.7.2 final and 
3.6.8 final.

https://www.python.org/ftp/python/devtest/python-3.7.2rc1+-macosx10.9.pkg
https://www.python.org/ftp/python/devtest/python-3.6.8rc1+-macosx10.9.pkg

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31503] Enhance dir(module) to be informed by __all__ by updating module.__dir__

[Cheryl Sabella]

Cheryl Sabella  added the comment:

The OP commented on the PR that a feature close enough to the original request 
was being implemented in PEP562, so I will close this issue with that one as a 
superseder.

--
nosy: +cheryl.sabella
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> Implement PEP 562: module __getattr__ and __dir__

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35554] Test

[Ernest W. Durbin III]

Change by Ernest W. Durbin III :


--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35554] Test

[Ernest W. Durbin III]

Ernest W. Durbin III  added the comment:

Test Reply

On December 21, 2018 at 6:13:30 PM, Ernest W. Durbin III 
(rep...@bugs.python.org) wrote:

New submission from Ernest W. Durbin III : 

Testing mailgateway 

-- 
messages: 332307 
nosy: EWDurbin 
priority: normal 
severity: normal 
status: open 
title: Test 

___ 
Python tracker  
 
___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35554] Test

[Ernest W. Durbin III]

New submission from Ernest W. Durbin III :

Testing mailgateway

--
messages: 332307
nosy: EWDurbin
priority: normal
severity: normal
status: open
title: Test

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24928] mock.patch.dict spoils order of items in collections.OrderedDict

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Thanks Mario, I will convert the unit test as a PR before closing the issue 
since I feel the test is a good one for inclusion and can help if dict order 
guarantee is changed in future. I will raise a backport PR to cabal's mock repo 
where the fix with test can be merged with reference to this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35547] email.parser / email.policy does not correctly handle multiple RFC2047 encoded-word tokens across RFC5322 folded headers

[Martijn Pieters]

Martijn Pieters  added the comment:

While RFC2047 clearly states that an encoder MUST not split multi-byte 
encodings in the middle of a character (section 5, "Each 'encoded-word' MUST 
represent an integral number of characters. A multi-octet character may not be 
split across adjacent 'encoded-word's.), it also states that to fit length 
restrictions, CRLF SPACE is used as a delimiter between encoded words (section 
2, "If it is desirable to encode more text than will fit in an 'encoded-word' 
of 75 characters, multiple 'encoded-word's (separated by CRLF SPACE) may be 
used."). In section 6.2 it states

   When displaying a particular header field that contains multiple
   'encoded-word's, any 'linear-white-space' that separates a pair of
   adjacent 'encoded-word's is ignored.  (This is to allow the use of
   multiple 'encoded-word's to represent long strings of unencoded text,
   without having to separate 'encoded-word's where spaces occur in the
   unencoded text.)

(linear-white-space is the RFC822 term for foldable whitespace).

The parser is leaving spaces between two encoded-word tokens in place, where it 
must remove them instead. And it is doing so correctly for unstructured 
headers, just not in get_bare_quoted_string, get_atom and get_dot_atom.

Then there is Postel's law (*be liberal in what you accept from others*), and 
the email package already applies that principle to RFC2047 elsewhere; RFC2047 
also states that "An 'encoded-word' MUST NOT appear within a 'quoted-string'." 
yet email._header_value_parser's handling of quoted-string will process EW 
sections.

--
title: email.parser / email.policy does correctly handle multiple RFC2047 
encoded-word tokens across RFC5322 folded headers -> email.parser / 
email.policy does not correctly handle multiple RFC2047 encoded-word tokens 
across RFC5322 folded headers

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35552] Do not read memory past the specified limit in PyUnicode_FromFormat() and PyBytes_FromFormat()

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +10510
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35552] Do not read memory past the specified limit in PyUnicode_FromFormat() and PyBytes_FromFormat()

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

Format characters %s and %V in PyUnicode_FromFormat() and %s 
PyBytes_FromFormat() allow to limit the number of bytes read from the argument. 
For example PyUnicode_FromFormat("must be string, not '%.50s'", 
obj->ob_type->tp_name) will use not more than 50 bytes from 
obj->ob_type->tp_name for creating a message.

But while the number of bytes used for creating the resulting Unicode or bytes 
object is limited, the current implementation can read past this limit. It uses 
strlen() for searching the first null byte, and bounds the result to the 
specified limit. If the input is not null terminated, this can cause a crash.

The proposed PR makes the code never reading past the specified limit.

--
components: Interpreter Core
messages: 332289
nosy: serhiy.storchaka
priority: normal
severity: normal
status: open
title: Do not read memory past the specified limit in PyUnicode_FromFormat() 
and PyBytes_FromFormat()
type: crash
versions: Python 2.7, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35545] asyncio.base_events.create_connection doesn't handle scoped IPv6 addresses

[twisteroid ambassador]

twisteroid ambassador  added the comment:

Also I believe it's a good idea to change the arguments of _ensure_resolved() 
from (address, *, ...) to (host, port, *, ...), and go through all its usages, 
making sure we're not mixing host + port with address tuples everywhere in 
asyncio.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35545] asyncio.base_events.create_connection doesn't handle scoped IPv6 addresses

[twisteroid ambassador]

twisteroid ambassador  added the comment:

I think the root cause of this bug is a bit of confusion.

The "customer-facing" asyncio API, create_connection(), takes two arguments: 
host and port. The lower-level API that actually deal with connecting sockets, 
socket.connect() and loop.sock_connect(), takes one argument: the address 
tuple. These are *not the same thing*, despite an IPv4 address tuple having two 
elements (host, port), and must not be mixed.

_ensure_resolved() is the function responsible for turning host + port into an 
address tuple, and it does the right thing, turning 
host="fe80::1%lo",port=12345 into ('fe80::1', 12345, 0, 1) correctly. The 
mistake is taking the address tuple and passing it through _ensure_resolved() 
again, since that's not the correct input type for it: the only correct input 
type is host + port.

So I think the appropriate fix for this bug is to make sure _ensure_resolved is 
only called once. In particular, BaseSelectorEventLoop.sock_connect() 
https://github.com/python/cpython/blob/3bc0ebab17bf5a2c29d2214743c82034f82e6573/Lib/asyncio/selector_events.py#L458
 should not call _ensure_resolved(). It might be a good idea to add some 
comments clarifying that sock_connect() takes an address tuple argument, not 
host + port, and likewise for sock_connect() on each event loop implementation.

--
nosy: +twisteroid ambassador

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24928] mock.patch.dict spoils order of items in collections.OrderedDict

[Mario Corchero]

Mario Corchero  added the comment:

I would suggest applying the fix with the latest version in mind to keep the 
codebase clean. If you want to backport it to previous versions of Python you 
can do it manually through a PR targetting the right branch.
I think the process is to set up a label and then use 
https://github.com/python/core-workflow/tree/master/cherry_picker as I'd expect 
the tests to fail to apply the patch "manually.

Alternative 1: Might be easier is to send a patch that works in all version and 
another one that "modernises" it to python3.7. Basically, write all tests with 
OrderedDict and then just shoot an extra PR that converts that to a plain dict.

Alternative 2: This is only a problem on versions on versions < 3.6, isn't it? 
If so you might want to close this issue or just have a PR that targets the 3.5 
if worth backporting and/or PyPI's. (This is my conservative mind as you know I 
usually push for changing as less as possible hehe).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35550] Some define guards for Solaris are wrong

[Jakub Kulik]

Change by Jakub Kulik :


--
keywords: +patch
pull_requests: +10509
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35551] Encoding and alias issues

[BLKSerene]

New submission from BLKSerene :

There're some minor issues about encodings supported by Python.
1. "tis260" is the alias for "tactis", where "tis260" might be a typo, which 
should be tis620. And "tactis" is not a supported encoding by Python (and I 
can't find any information about this encoding on Google).
2. "mac_latin2" and "mac_centeuro" refer to the same encoding (the decoding 
tables are identical), but they are provided as two encodings in different 
names ("maccentraleurope" is an alias for "mac_latin2", but "mac_centeuro" 
isn't).
3. The same problem for "latin_1" and "iso8859_1" ("iso_8859_1" is an alias for 
"latin_1", but "iso8859_1" isn't).

--
components: Unicode
messages: 332285
nosy: blkserene, ezio.melotti, vstinner
priority: normal
severity: normal
status: open
title: Encoding and alias issues
type: enhancement
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35550] Some define guards for Solaris are wrong

[Jakub Kulik]

New submission from Jakub Kulik :

Python source code uses on several places ifdef sun or defined(sun) without the 
underscores, which is not standard compliant and shouldn't be used. Our recent 
Solaris python build ended up skipping these sections resulting in some obvious 
problems.

Defines should check for __sun instead.

(link: 
http://nadeausoftware.com/articles/2012/01/c_c_tip_how_use_compiler_predefined_macros_detect_operating_system#Solaris)

--
components: Build
messages: 332284
nosy: kulikjak
priority: normal
severity: normal
status: open
title: Some define guards for Solaris are wrong
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35549] Add partial_match: bool = False argument to unicodedata.lookup

[Roman Inflianskas]

New submission from Roman Inflianskas :

I propose to add partial_match: bool = False argument to unicodedata.lookup so 
that the programmer could search Unicode symbols using partial_names.

--
components: Unicode
messages: 332283
nosy: ezio.melotti, rominf, vstinner
priority: normal
severity: normal
status: open
title: Add partial_match: bool = False argument to unicodedata.lookup
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com