[Python-Dev] Re: Help with the grail browser

[Martin Panter]

On 30/05/2021, Prakhar Sehgal  wrote:
> The latest version of Grail is 0.6 which was released in 1999 was made for
> Python 1 but I want to make one compatible with Python 3. But I need help
> for that. Currently me and Guido van Rossum are working on this. Any help
> will be welcomed. This is the link for the GitHub repo:
> https://github.com/PrakharPythonProgrammer/Grail-0.6

Quite a while ago I got Grail working with Python 3. That was in the
era of 3.3; there might be more deprecated modules since (I'm thinking
the "formatter" module at least). I'm pretty sure I put all the
important changes on the "py3" branch on my Git Hub repository, at
. It may be useful
to you.

-Martin.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RQTC4E3Q3I6PHXCQOX23AV6K2FTQ47EG/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-Dev] git history conundrum

[Martin Panter]

On Sat, 27 Apr 2019 at 19:07, Chris Withers  wrote:
> Right, so I've merged up to 15f44ab043, what comes next?
>
> $ git log --oneline  --no-merges 15f44ab043.. -- Lib/unittest/mock.py
> Lib/unittest/test/testmock/ | tail -n 3

This Git command line means list all the revisions except 15f44ab043
and those leading up to it.

> 161a4dd495 Issue #28919: Simplify _copy_func_details() in unittest.mock
> ac5084b6c7 Fixes issue28380: unittest.mock Mock autospec functions now
> properly support assert_called, assert_not_called, and assert_called_once.
> 0be894b2f6 Issue #27895:  Spelling fixes (Contributed by Ville Skyttä).
>
> Okay, no idea why 0be894b2f6 is there, appears to be a totally identical
> commit to 15f44ab043

Git revision 15f44ab043 is the original spelling fixes, which were
pushed to the Mercurial “default” branch (= Git master) for Python 3.6
by Raymond. Revision 0be894b2f6 is my backport to the 3.5 branch, done
about a week later. The backport is probably a subset of the original,
rather than identical (e.g. the datetime.rst change was not applicable
to 3.5).

The convention at the time was to keep the 3.5 branch merged into
Default (Master). That is why my 3.5 backport appears in your history
of Master.

> so let's skip it:
>
> $ git log --oneline  --no-merges 0be894b2f6.. -- Lib/unittest/mock.py
> Lib/unittest/test/testmock/ | tail -n 3
> 161a4dd495 Issue #28919: Simplify _copy_func_details() in unittest.mock
> ac5084b6c7 Fixes issue28380: unittest.mock Mock autospec functions now
> properly support assert_called, assert_not_called, and assert_called_once.
> 15f44ab043 Issue #27895:  Spelling fixes (Contributed by Ville Skyttä).
>
> Wat?! Why is 15f44ab043 showing up again?!

Because you are asked for all the revisions except my backport and its
ancestors. As far as Git is concerned, the original spelling fixes are
not an ancestor of my backport.

I don’t have a copy of the Git repository to try, but I suggest the
following command is what you want:

git log --oneline  --no-merges HEAD ^15f44ab043 ^0be894b2f6 --
Lib/unittest/mock.py Lib/unittest/test/testmock/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] bpo-34203: FAQ now recommends python 3.x over 2.x (GH-9796)

[Martin Panter]

On 12/10/2018, Eric V. Smith  wrote:
> On 10/12/2018 5:17 AM, Tal Einat wrote:
>
>>   The latest stable releases can always be found on the `Python download page
>> -`_.  There are two recommended 
>> production-ready
>> -versions at this point in time, because at the moment there are two 
>> branches of
>> -stable releases: 2.x and 3.x.  Python 3.x may be less useful than 2.x, since
>> -currently there is more third party software available for Python 2 than for
>> -Python 3.  Python 2 code will generally not run unchanged in Python 3.
>> +`_.  There are two production-ready 
>> version
>> +of Python: 2.x and 3.x, but the recommended one at this times is Python 3.x.
>
> This should be "time", not "times". I'd fix it, but I'm unsure if this
> is being backported or not, and I don't want to mess up any merges
> before they're done.

Or just remove “at this time[s]”; it doesn’t add much.

Also, “two . . . version” should be changed back to plural: “two . . .
versions”.

> I do think this should backported to 3.7, 3.6, and 2.7.
>
>> +Although Python 2.x is still widely used, `it will not be
>> +maintained after January 1, 2020
>> `_.
>> +Python 2.x was known for having more third-party libraries available,
>> however,
>> +by the time of this writing, most of the widely used libraries support
>> Python 3.x,
>
> Should probably be "at the time of this writing".
>
>> +and some are even dropping the Python 2.x support.
>
> And this would read better as "are even dropping support for Python 2.x".
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is object the most base type? (bpo-20285)

[Martin Panter]

> On Fri, Feb 02, 2018 at 01:53:00AM -0500, Terry Reedy wrote:
>> >>> object.__doc__
>> 'The most base type'
> [...]
>> I have suggested
>> "The superclass for all Python classes."
>> "The starting base class of all types and classes other than itself."
>>
>> I intended to pick the second, but Serhiy Storchake wants more opinions.

On 2 February 2018 at 07:25, Steven D'Aprano  wrote:
> "the base class of the class heirarchy"
>
> "the root of the class heirarchy"

The RST documentation

says

Object is a base for all classes.

which is better than “the most base type” IMO, although I would write
_the_ base rather than _a_ base; there is only one object class. In my
patch for the RST documentation in
 I proposed

This is the ultimate base class of all other classes.

which I still think is okay (though other parts of my patch doesn’t
seem very good).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Evil reference cycles caused Exception.__traceback__

[Martin Panter]

On 18 September 2017 at 09:31, Victor Stinner  wrote:
> Last years, I fixed many reference cycles in various parts of the
> Python 3 standard library. Sometimes, it takes years to become aware
> of the reference cycle and finally fix it.
>
> For example, recently, I worked on fixing all "dangling threads"
> leaked by tests of the Python test suite, and I found and fixed many
> reference cycles which probably existed since Python 3 was created
> (forked from Python 2)

My instinct is to suggest to make the “dangling threads” test tolerate
Thread objects that are no longer running. Last time I looked, it
tests a list of weak references to Thread objects. But maybe it should
check “Thread.is_alive” or use “threading.enumerate”.

> . . .
>
> Ideally, CPython 3.x should never create reference cycles. Removing
> Exception.__traceback__ is the obvious "fix" for the issue. But I
> expect that slowly, a lot of code started to rely on the attribute,
> maybe even for good reasons :-)
>
> A more practical solution would be to log a warning. Maybe the garbage
> collector can emit a warning if it detects an exception part of a
> reference cycle? Or maybe detect frames?

The “gc” module can do some of this already. In the past (when __del__
used to prevent garbage collection) I used DEBUG_SAVEALL to and
graphed the result to figure out where code was creating reference
cycles. Or maybe if you turned on DEBUG_COLLECTABLE and filtered the
objects printed out, or played with the new “gc.callbacks” API.

> If the GC cannot do it, maybe we might use a debug thread (enabled
> manually) which checks manually if an exception is part of a reference
> cycle using gc.get_objects(): check if an exception remains alive
> longer than X seconds? I had the same idea for asyncio, to detect
> reference cycles or if a task is never "awaited", but I never
> implemented the idea.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] socketserver ForkingMixin waiting for child processes

[Martin Panter]

> On Fri, Aug 11, 2017 at 6:46 AM Victor Stinner 
> wrote:
>> => http://bugs.python.org/issue31151
>>
>> I changed the code to call waitpid() in blocking mode on each child
>> process on server_close(), to ensure that all children completed when
>> on server close:
>>
>> https://github.com/python/cpython/commit/aa8ec34ad52bb3b274ce91169e1bc4a598655049
>>
>> After pushing my change, I'm not sure anymore if it's a good idea.
>> There is a risk that server_close() blocks if a child is stuck on a
>> socket recv() or send() for some reasons.

I agree that this could be an unwanted change in behaviour. For
example, a web browser could be holding a HTTP 1.1 persistent
connection open.

>> Should we relax the code by waiting a few seconds (problem: hardcoded
>> timeouts are always a bad idea), or *terminate* processes (SIGKILL on
>> UNIX) if they don't complete fast enough?

It does seem reasonable to have an option to clean up background
forks, whether by blocking, or terminating them immediately.

On 11 August 2017 at 22:34, Ryan Smith-Roberts  wrote:
> Since ThreadingMixIn also leaks threads,
> server_close() could grow a timeout flag (following the socket module
> timeout convention) and maybe a terminate boolean. ThreadingMixIn could then
> also be fixed.

You could do a blocking join on each background thread. But I suspect
there is no perfect way to terminate the threads without waiting.
Using ThreadingMixIn.daemon_threads and exiting the interpreter might
have to be good enough.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Escaping docs markup in NEWS entries?

[Martin Panter]

On 1 May 2017 at 06:37, Nick Coghlan  wrote:
> Hi folks,
>
> I'm trying to write a NEWS entry that explains that the
> ":func:`bytes`" cross-references have changed to refer to the type
> descriptions by default (matching other builtin container types), so
> you now need to use ``:ref:`func-bytes`" to refer to the old target in
> the list of builtin functions (if you really want that for some
> reason).
>
> Unfortunately, my first two attempts both cause warnings in "make
> suspicious" with the following output:

What is the full output? Usually it includes instructions to add false
positives to Doc/tools/susp-ignored.csv; maybe that is all you have to
do?

> 
> WARNING: [whatsnew/changelog:986] ":func" found in "\:func\:\`bytes\`"
> WARNING: [whatsnew/changelog:986] "`" found in "\:func\:\`bytes\`"
> WARNING: [whatsnew/changelog:986] ":func" found in "\:func\:\`bytearray\`"
> WARNING: [whatsnew/changelog:986] "`" found in "\:func\:\`bytearray\`"
> WARNING: [whatsnew/changelog:986] ":ref" found in "\:ref\:\`func-bytes\`"
> WARNING: [whatsnew/changelog:986] "`" found in "\:ref\:\`func-bytes\`"
> WARNING: [whatsnew/changelog:986] ":ref" found in "\:ref\:\`func-bytes\`"
> WARNING: [whatsnew/changelog:986] "`" found in "\:ref\:\`func-bytes\`"
> 
>
> My first attempt just escaped the nested backticks:
>
> - Issue #30052: the link targets for ``:func:\`bytes\``` and
>   ``:func:\`bytearray\``` are now their respective type definitions, rather
>   than the corresponding builtin function entries. Use ``:ref:\`func-bytes\```
>   and ``:ref:\`func-bytes\``` to reference the latter.
>
> My second attempt escaped the colons as well:
>
> - Issue #30052: the link targets for ``\:func\:\`bytes\``` and
>   ``\:func\:\`bytearray\``` are now their respective type definitions, rather
>   than the corresponding builtin function entries. Use 
> ``\:ref\:\`func-bytes\```
>   and ``\:ref\:\`func-bytes\``` to reference the latter.
>
> My fallback plan is to just include the unescaped markup in the NEWS
> entry (rather than trying to make it readable even in rendered form),
> but I figured I'd ask for advice here first.

I thought the NEWS file was mainly regarded as plain text, so it would
be important to avoid ugly RST markup like backslashes if possible.

I am no RST expert, but perhaps it would be clearer to a human RST
parser if you added a space among the last three backticks, where the
underscore is in ``:func:`bytes`_``.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

[Martin Panter]

On 31 March 2017 at 05:22, Nathaniel Smith  wrote:
>>> On Mon, Mar 20, 2017 at 11:30 AM, Oleg Nesterov  wrote:
>>> > [Aborting "yield" in a "for" loop leaves a sub-generator open, but 
>>> > aborting "yield from" cleans up the sub-generator]
>
> In any case the short answer to your original question is that PEP 342
> says that generator finalization calls the generator's close() method,
> which throws a GeneratorExit into the generator, and PEP 380 says that
> as a special case, when a GeneratorExit is thrown into a yield from,
> then this is propagated by calling .close() on the yielded-from
> iterator (if such a method exists) and then re-raised in the original
> generator.

I think the Python documentation could be improved regarding this.
When I wrote the documentation for coroutine methods
,
I included details about the "close" and "throw" methods delegating to
inner iterators. I thought I was going to propose similar updates to
the generator documentation
,
but it seems I never got around to it. (In the mean time,

was added, to which this may also be relevant, but that is too
complicated for me.)

There is a parallel with another annoyance with Python generator
cleanup: . There are two ways you
can require generators to be used. With a simple generator, you can
partially iterate it and then throw it away without any special
cleaning up. But with more complex generators that "own" expensive
resources, it would be nice to produce a ResourceWarning I you forget
to clean them up.

With the "for / yield" case, the sub-generator is not cleaned up, so
if a resource-intensive sub-generator has to be cleaned up you have to
do it yourself. With "yield from", the cleanup is implicit and
unavoidable, which means you can't use it if you want keep the
sub-generator alive for later. But the automatic cleanup may be useful
in other cases.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Martin Panter]

On 30 March 2017 at 15:31, Nick Coghlan  wrote:
> On 29 March 2017 at 02:18, Paul Moore  wrote:
>> On 28 March 2017 at 12:24, Miro Hrončok  wrote:
>>> I'd like some clarification on what ABI compatibility we can expect.
>>>  * Should the ABI be stable across patch releases (so calling
>>> PySlice_AdjustIndices from an existing macro would be a bug)?
>>>  * Should the ABI be forward-compatible within a minor release (so modules
>>> built for 3.6.0 should be usable with 3.6.1, but not vice versa)?
>>>  * Or should we expect the ABI to change even across patch releases?
>>
>> Given that binary wheels are built against a specific minor version
>> (3.6, 3.5, ...) I would expect the ABI to be consistent over a minor
>> release. That would fit with my expectations of the compatibility
>> guarantees on patch releases.
>>
>> So I from what you describe, I'd consider this as a bug. Certainly, if
>> someone built a C extension as a wheel using Python 3.6.1, it would be
>> tagged as compatible with cp36, and pip would happily use it when
>> installing to a Python 3.6.0 system, where it would fail.
>
> Right, this is the main problem - while "build against the X.Y.0
> headers" is useful advice, it's not something we've ever explicitly
> stated, and it's not something we can reasonably expect all providers
> of pre-built binary modules to do.
>
> Instead, it makes sense to explicitly strengthen the ABI guarantees
> within CPython maintenance releases, and add some automated testing to
> avoid accidental changes and oversights (similar to the pending test
> to ensure magic number stability for cached bytecode files)

There's a website that has nice reports of ABI compatibility of
various packages and might useful for testing. It shows up the added
PySlice_AdjustIndices function in 3.6.1, along with PySlice_Unpack
(and some changes to internal names, so probably not such a problem).
https://abi-laboratory.pro/tracker/compat_report/python/3.6.0/3.6.1/496a4/abi_compat_report.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue with _thread.interrupt_main (29926)

[Martin Panter]

On 29 March 2017 at 00:40, Terry Reedy  wrote:
> [. . .] Eryk Sun suggested a patch for Windows, (and
> the possibility of using pthread_kill).  Can you possibly do one for *nix?
> This is out of my ballpark, but the bug (relative to console behavior) is a
> nuisance.

I'll try to find time, but no promises.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue with _thread.interrupt_main (29926)

[Martin Panter]

On 28 March 2017 at 03:11, Steven D'Aprano  wrote:
> On Mon, Mar 27, 2017 at 10:33:44PM -0400, Terry Reedy wrote:
>> https://bugs.python.org/issue29926 was opened as an IDLE issue, which
>> means that most watching the new issues list would ignore it.  But I
>> think it is an issue with _thread.interrupt_main (which IDLE calls in
>> respond to ^C) not interrupting time.sleep(n) in main thread*.  I tested
>> on Windows, don't know yet about OP.  Since there is no Expert's Index
>> listing for _thread (or threading), I am asking here for someone who
>> knows anything to take a look.
>>
>> *
>> >>> time.sleep(10)
>> 
>> 
>> <... remainder of 10 seconds pass>
>> KeyboardInterrupt
>
>
> I get similar behaviour under Linux. I don't have the debug print, but
> the KeyboardInterrupt doesn't interrupt the sleep until the 10 seconds
> are up.

Looking at the implementation, _thread.interrupt_main just calls
PyErr_SetInterrupt. It doesn’t appear to send a signal. I played with
“strace” and couldn’t see any evidence of a signal. I guess it just
sets a flag that will be polled. To actually interrupt the “sleep”
call, you might need to use “pthread_kill” or similar (at least on
Unix).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Regarding writing tests for module tabnanny

[Martin Panter]

On 13 March 2017 at 11:56, Jaysinh Shukla  wrote:
> Respected Members,
>
> I identified the standard module 'tabnanny' is having 16.66% of code
> coverage (Source:
> https://codecov.io/gh/python/cpython/src/master/Lib/tabnanny.py). I am
> interested to write tests for this module. Before starting, I would like to
> get help from any core developer on this.
>
>
> 1. Is community expecting to have tests for this module?
> 2. Any module specific guidelines?
>
> I am waiting for green signal from any core developer. Thanks!

Try the people involved in the existing patches at
 and
.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory)

[Martin Panter]

On 24 February 2017 at 07:51, Benjamin Peterson  wrote:
> As for this, particular issue, we should determine if there's a tracker
> issue yet and continue discussion there.

That would be .
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] adding threaded tests to the test suite

[Martin Panter]

> Le dim. 22 janv. 2017 à 21:04, Ethan Furman  a écrit :
>> Question:  I need to add a threaded test to the enum test module [1] -- is
>> there anything extra I
>> need to worry about besides the test itself?  Setting or resetting or
>> using a tool library, etc?
>>
>> threads = []
>> for i in range(8):
>> threads.append(threading.Thread(target=cycle_enum))
>> for t in threads:
>> t.start()
>> for t in threads:
>> t.join()

On 22 January 2017 at 20:17, Victor Stinner  wrote:
> There is @support.reap_thread which can help.

As I understand, @reap_threads basically does a join() on each
background thread, with a total timeout of 1 s. So since your test is
unlikely to fail between starting threads and joining them, I don’t
think you need to use @reap_threads.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Update the porting HOWTO

[Martin Panter]

On 17 December 2016 at 20:39, brett.cannon  wrote:
> https://hg.python.org/cpython/rev/287d4290b1b4
> changeset:   105714:287d4290b1b4
> branch:  2.7
> parent:  105677:eb02db65e148
> user:Brett Cannon 
> date:Sat Dec 17 12:38:54 2016 -0800
> summary:
>   Update the porting HOWTO
>
> diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst
> --- a/Doc/howto/pyporting.rst
> +++ b/Doc/howto/pyporting.rst
> . . .
>  Have good test coverage
>  ---
>
> @@ -106,10 +107,11 @@
>  thumb is that if you want to be confident enough in your test suite that any
>  failures that appear after having tools rewrite your code are actual bugs in 
> the
>  tools and not in your code. If you want a number to aim for, try to get over 
> 80%
> -coverage (and don't feel bad if you can't easily get past 90%). If you
> +coverage (and don't feel bad if you can't easily get passed 90%). If you
>  don't already have a tool to measure test coverage then coverage.py_ is
>  recommended.

Hi Brett, why did you make the above change (get past → get passed)?
To me, “get past 90%” means achieving over 90%, but “you can’t get
passed 90%” would mean that 90% cannot be given (passed) to you. The
original made more sense. Another option would be “get over 90%”,
consistent with the previous sentence.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Implementation difference of audioop.lin2lin in Python2 and Python3

[Martin Panter]

On 13 December 2016 at 13:37, MRAB  wrote:
> On 2016-12-13 11:31, KH Luke Kim wrote:
>>
>> Hello,
>> recently there had been some issues in audioread and librosa that 3-byte
>> samples can be loaded in Python 3 but 2.
>>
>> The documentation says that the audioop.lin2lin function in Python 3
>> support 1-, 2-, 3-, 4-byte samples but only 1-, 2-, 4-byte samples in
>> Python 2.
>>
>> I wonder why 3-byte support is not implemented in Python 2. If there is
>> any previous thread or history regarding this issue, could you refer it
>> to me?
>>
> The Python docs say that support for 3-byte (24-bit) samples was added in
> Python 3.4, so anyone using a version before that one is out of luck!

The 3.4 reference leads you to What’s New, which leads to discussion
in the bug tracker:
https://docs.python.org/3/whatsnew/3.4.html#audioop
https://bugs.python.org/issue12866
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Revert unintended merge

[Martin Panter]

On 3 December 2016 at 22:15, Steve Dower  wrote:
> On 03Dec2016 1312, Serhiy Storchaka wrote:
>>
>> On 03.12.16 22:13, steve.dower wrote:
>>>
>>> https://hg.python.org/cpython/rev/a60767015bed
>>> changeset:   105436:a60767015bed
>>> user:Steve Dower 
>>> date:Sat Dec 03 12:12:23 2016 -0800
>>> summary:
>>>   Revert unintended merge
>>
>>
>> I suppose it should be reverted in the 3.6 branch too.
>
>
> Maybe, but I didn't make the change in the 3.6 branch, so I have no idea
> whether it is meant to be there or not. But it wasn't part of the change I
> made, so I didn't want to merge it forward.

This change comes from Matthias (Doko), and was originally only in the
3.5 branch:
https://hg.python.org/cpython/rev/14c80065c36e

I presume it was left unmerged until there is a branch for 3.6.1 to
merge into. So Steve, when you did your first 3.5 → 3.6 merge, you
merged Doko’s change into 3.6:

https://hg.python.org/cpython/rev/5171bd86a36f

> Someone who actually understands the implications of changing these files
> (config.guess, config.sub) can figure it out :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New problem accessing tracker with Firefox 50.0

[Martin Panter]

On 18/11/2016, Terry Reedy  wrote:
> Starting today, while reviewing the "Summary of Python tracker issues",
> I get the following for about half the clicks.
>
> """
> Secure Connection Failed
>
> An error occurred during a connection to bugs.python.org. A PKCS #11
> module returned CKR_DEVICE_ERROR, indicating that a problem has occurred
> with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR
>
>  The page you are trying to view cannot be shown because the
> authenticity of the received data could not be verified.
>  Please contact the website owners to inform them of this problem.
>
> [   Try Again   ]
> """
>
> So far, the Try Again button has always worked.
>
> I believe I upgraded Firefox (to 50.0) just last night, or possibly the
> night before.

Yesterday I started seeing this behaviour too. I have Firefox 43 and
haven’t updated any relevant libraries recently, so I assumed it was
the server’s fault :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (merge 3.5 -> default): Null merge.

[Martin Panter]

On 29 September 2016 at 04:26, Zachary Ware
 wrote:
> On Wed, Sep 28, 2016 at 10:04 PM, Terry Reedy  wrote:
>> On 9/28/2016 9:57 PM, terry.reedy wrote:
>>> https://hg.python.org/cpython/rev/02eb35b79af0
>>
>>
>> (2nd try) I mistakenly null merged from 3.5 to default.
>> Should a now do a proper null merge from 3.5 to 3.6 to default?
>> Should I revert this null merge?

FYI I committed some merges (04060fa4428d and ae0c983d3c65) which
should have fixed this all up.

> You aren't the only one who's missed 3.6 since it was branched :).  If
> there are changes in 3.5 that should not be in 3.6, you should go
> ahead and do a null merge from 3.5 -> 3.6 -> default.  If the changes
> in 3.5 are already in 3.6, I'd just leave it as is; it will clear up
> when somebody next merges something.

In this case, my automatic merge process gave conflicts and spooky
“ambiguous merge” warnings, so in this case I think it was good to
deal with it sooner rather than later.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (merge 3.5 -> default): Null merge.

[Martin Panter]

On 29 September 2016 at 03:04, Terry Reedy  wrote:
> On 9/28/2016 9:57 PM, terry.reedy wrote:
>> https://hg.python.org/cpython/rev/02eb35b79af0
>
> (2nd try) I mistakenly null merged from 3.5 to default.
> Should a now do a proper null merge from 3.5 to 3.6 to default?

Yes, I think 3.5 needs to be merged into 3.6, and the result needs to
be merged into default. I guess they are null merges because the
entries were already present in 3.6b1.

> Should I revert this null merge?

I don’t think there is much point in reverting a null merge, if there
are no actual changes to revert.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Fixes test_getargs2 to get the buildbots working again.

[Martin Panter]

On 12 September 2016 at 02:48, Steve Dower  wrote:
>>   Fixes test_getargs2 to get the buildbots working again.
>
> I'm not sure this is the fix we want to keep here, but it was sufficient to
> get the test going and unblock all the buildbots.
>
> I'm not entirely sure when the break appeared (essentially we seem to not be
> copying *args into a new tuple), but I'd guess it's to do with the fast
> calling improvements.

That seems to be everyone else’s guess too. See
https://bugs.python.org/issue28086 (bug about this failure)
https://bugs.python.org/issue27213 (bisected cause)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (some) C99 added to PEP 7

[Martin Panter]

Thank you very much Benjamin.

On 7 September 2016 at 17:56, Benjamin Peterson  wrote:
> To conclude our discussion about using C99 features, I've updated PEP 7
> to allow the following features:
> - Standard integer types in  and 

Perhaps PEP 7 should clarify if the optional types like uint32_t are
allowed, or only C99 mandatory types like uint_fast32_t etc. I think
more people will be familiar with the fixed-width uint32_t etc. I know
they are mandatory in Posix, and presumably also Windows, so they may
be okay.

> - ``static inline`` functions
> - designated initializers
> - intermingled declarations
> - booleans
>
> I've been adding examples of these to 3.6 over the last few days to make
> sure the buildbots will like it.
>
> https://github.com/python/peps/commit/b6efe6e06fa70e8933440da26474a804fb3edb6e
>
> Enjoy.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Martin Panter]

On 5 September 2016 at 21:40, eryk sun  wrote:
> On Mon, Sep 5, 2016 at 7:54 PM, Steve Dower  wrote:
>> On 05Sep2016 1234, eryk sun wrote:
>>> It would probably be simpler to use UTF-16 in the main pipeline and
>>> implement Martin's suggestion to mix in a UTF-8 buffer. The UTF-16
>>> buffer could be renamed as "wbuffer", for expert use. However, if
>>> you're fully committed to transcoding in the raw layer, I'm certain
>>> that these problems can be addressed with small buffers and using
>>> Python's codec machinery for a flexible mix of "surrogatepass" and
>>> "replace" error handling.
>>
>> I don't think it actually makes things simpler. Having two buffers is
>> generally a bad idea unless they are perfectly synced, which would be
>> impossible here without data corruption (if you read half a utf-8 character
>> sequence and then read the wide buffer, do you get that character or not?).
>
> Martin's idea, as I understand it, is a UTF-8 buffer that reads from
> and writes to the text wrapper.

Yes, that was basically it. Though I had only thought as far as simple
encodings like ASCII, where one byte corresponds to one character. I
wonder if you really need UTF-8 support. Are the encoding values
currently encountered for Windows consoles all single-byte encodings
or are they more complicated?

> It necessarily consumes at least one
> character and buffers it to allow reading per byte. Likewise for
> writing, it buffers bytes until it can write a character to the text
> wrapper. ISTM, it has to look for incomplete lead-continuation byte
> sequences at the tail end, to hold them until the sequence is
> complete, at which time it either decodes to a valid character or the
> U+FFFD replacement character.

This buffering behaviour would be necessary for a multi-byte encodings
like UTF-8.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Where are the list and array.array implementations in CPython source?

[Martin Panter]

On 5 September 2016 at 23:45, Jonathan Goble  wrote:
> I'd like to study the CPython implementations of lists and array.array
> instances for a personal project of mine, but I've very unfamiliar
> with the Python source code as it pertains to internals like this.
> Which files would I need to look at to do this,

Built-in objects are usually in the Objects/ directory, with a
corresponding include file in the Include/ directory:
https://hg.python.org/cpython/file/default/Objects/listobject.c
https://hg.python.org/cpython/file/default/Include/listobject.h

Modules implemented in C are usually in the Modules/ directory:
https://hg.python.org/cpython/file/default/Modules/arraymodule.c

> and are there a few
> particular functions/structures I should pay attention to? I'm just
> looking for a brief pointer in the right direction here, not a full
> explanation of how it works -- I'll get that from studying the source
> code. :-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Martin Panter]

On 5 September 2016 at 09:10, Paul Moore  wrote:
> On 5 September 2016 at 06:54, Steve Dower  wrote:
>> +Using the raw object with small buffers
>> +---
>> +
>> +Code that uses the raw IO object and attempts to read less than four 
>> characters
>> +will now receive an error. Because it's possible that any single character 
>> may
>> +require up to four bytes when represented in utf-8, requests must fail.
>
> I'm very concerned about this statement. It's clearly not true that
> the request *must* fail, as reading 1 byte from a UTF-8 enabled Linux
> console stream currently works (at least I believe it does). And there
> is code in the wild that works by doing a test that "there's input
> available" (using kbhit on Windows and select on Unix) and then doing
> read(1) to ensure a non-blocking read (the pyinvoke code I referenced
> earlier). If we're going to break this behaviour, I'd argue that we
> need to provide a working alternative.
>
> At a minimum, can the PEP include a recommended cross-platform means
> of implementing a non-blocking read from standard input, to replace
> the current approach? (If the recommendation is to read a larger
> 4-byte buffer and manage the process of retaining unused bytes
> yourself, then that's quite a major change to at least the code I'm
> thinking of in invoke, and I'm not sure read(4) guarantees that it
> *won't* block if only 1 byte is available without blocking...)

FWIW, on Linux and Unix in general, if select() or similar indicates
that some read data is available, calling raw read() with any buffer
size should return at least one byte, whatever is available, without
blocking. If the user has only typed one byte, read(4) would return
that one byte immediately.

But if you are using a BufferedReader (stdin.buffer rather than
stdin.buffer.raw), then this guarantee is off and read(4) will block
until it gets 4 bytes, or until EOF.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Martin Panter]

On 1 September 2016 at 23:28, Random832  wrote:
> On Thu, Sep 1, 2016, at 18:28, Steve Dower wrote:
>> This is a raw (bytes) IO class that requires text to be passed encoded
>> with utf-8, which will be decoded to utf-16-le and passed to the Windows 
>> APIs.
>> Similarly, bytes read from the class will be provided by the operating
>> system as utf-16-le and converted into utf-8 when returned to Python.
>
> What happens if a character is broken across a buffer boundary? e.g. if
> someone tries to read or write one byte at a time (you can't do a
> partial read of zero bytes, there's no way to distinguish that from an
> EOF.)
>
> Is there going to be a higher-level text I/O class that bypasses the
> UTF-8 encoding step when the underlying bytes stream is a console? What
> if we did that but left the encoding as mbcs? I.e. the console is text
> stream that can magically handle characters that aren't representable in
> its encoding. Note that if anything does os.read/write to the console's
> file descriptors, they're gonna get MBCS and there's nothing we can do
> about it.

Maybe it is too complicated and impractical, but I have imagined that
the sys.stdin/stdout/stderr could be custom TextIOBase objects. They
would not be wrappers or do much encoding (other than maybe newline
encoding). To solve the compatibility problems with code that uses
stdout.buffer or whatever, you could add a custom “buffer” object,
something like my AsciiBufferMixin class:
https://gist.github.com/vadmium/d1b07d771fbf4347683c005c40991c02

Just putting this idea out there, but maybe Steve’s UTF-8 encoding
solution is good enough.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Martin Panter]

On 1 September 2016 at 19:36, Ethan Furman  wrote:
> Deprecation of current "zero-initialised sequence" behaviour without removal
> 
>
> Currently, the ``bytes`` and ``bytearray`` constructors accept an integer
> argument and interpret it as meaning to create a zero-initialised sequence
> of the given size::
>
> >>> bytes(3)
> b'\x00\x00\x00'
> >>> bytearray(3)
> bytearray(b'\x00\x00\x00')
>
> This PEP proposes to deprecate that behaviour in Python 3.6, but to leave
> it in place for at least as long as Python 2.7 is supported, possibly
> indefinitely.

Can you clarify what “deprecate” means? Just add a note in the
documentation, or make calls trigger a DeprecationWarning as well?
Having bytearray(n) trigger a DeprecationWarning would be a minor
annoyance for code being compatible with Python 2 and 3, since
bytearray(n) is supported in Python 2.

> Addition of "getbyte" method to retrieve a single byte
> --
>
> This PEP proposes that ``bytes`` and ``bytearray`` gain the method
> ``getbyte``
> which will always return ``bytes``::

Should getbyte() handle negative indexes? E.g. getbyte(-1) returning
the last byte.

> Open Questions
> ==
>
> Do we add ``iterbytes`` to ``memoryview``, or modify
> ``memoryview.cast()`` to accept ``'s'`` as a single-byte interpretation?  Or
> do we ignore memory for now and add it later?

Apparently memoryview.cast('s') comes from Nick Coghlan:
.
However, since 3.5 (https://bugs.python.org/issue15944) you can call
cast("c") on most memoryviews, which I think already does what you
want:

>>> tuple(memoryview(b"ABC").cast("c"))
(b'A', b'B', b'C')
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Martin Panter]

> Le samedi 3 septembre 2016, Random832  a écrit :
>> On Fri, Sep 2, 2016, at 19:44, Ethan Furman wrote:
>> > The problem with only having `bchr` is that it doesn't help with
>> > `bytearray`;
>>
>> What is the use case for bytearray.fromord? Even in the rare case
>> someone needs it, why not bytearray(bchr(...))?

On 3 September 2016 at 08:47, Victor Stinner  wrote:
> Yes, this was my point: I don't think that we need a bytearray method to
> create a mutable string from a single byte.

I agree with the above. Having an easy way to turn an int into a bytes
object is good. But I think the built-in bchr() function on its own is
enough. Just like we have bytes object literals, but the closest we
have for a bytearray literal is bytearray(b". . .").
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Martin Panter]

On 2 September 2016 at 17:54, Koos Zevenhoven  wrote:
> On Thu, Sep 1, 2016 at 10:36 PM, Ethan Furman  wrote:
>> * Deprecate passing single integer values to ``bytes`` and ``bytearray``
>> * Add ``bytes.fromsize`` and ``bytearray.fromsize`` alternative
>> constructors
>> * Add ``bytes.fromord`` and ``bytearray.fromord`` alternative constructors
>> * Add ``bytes.getbyte`` and ``bytearray.getbyte`` byte retrieval methods
>> * Add ``bytes.iterbytes`` and ``bytearray.iterbytes`` alternative
>> iterators
>
> I wonder if from_something with an underscore is more consistent (according
> to a quick search perhaps yes).

That would not be too inconsistent with the sister constructor bytes.fromhex().
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Lib/http/client.py: could it return an OSError with the current response?

[Martin Panter]

On 30 August 2016 at 13:41, Ivo Bellin Salarin
 wrote:
> While using requests to tunnel a request via a proxy requiring user
> authentication, I have seen that httplib
> (https://hg.python.org/cpython/file/3.5/Lib/http/client.py#l831) raises the
> message returned by the proxy, along with its status code (407) without
> including the proxy response. This one could be very interesting to the
> consumer, since it could contain some useful headers (like the supported
> authentication schemes).

Here are some existing bug threads which may be relevant:

https://bugs.python.org/issue7291 (urllib.request support for handling
tunnel authentication)
https://bugs.python.org/issue24964 (get tunnel response header fields
in http.client)

> Would it be possible to change the http/client.py behavior in order to raise
> an exception including the whole response?

That would be one way, and might be good enough for getting a
Proxy-Authenticate value. Although there might be other ways that
also:

* Allow reading the body (e.g. HTML page) of the error response. IMO
an exception instance is the wrong place for this;
urllib.error.HTTPError is a bad example.
* Allow the tunnel response fields to be obtained even when the
request was successful

> If you don't see any problem with my proposal, how can I propose a pull
> request? :-)

Perhaps you can use one of the patches at one of the above bug reports
as a starting point. What you want seems to be a prerequisite for
Issue 7291 (urllib.request), so maybe we can discuss it there. Or open
a new bug to focus on the http.client-only aspect.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] socket.setsockopt() with optval=NULL

[Martin Panter]

On 21 August 2016 at 12:37, Christian Heimes  wrote:
> the socket.setsockopt(level, optname, value) method has two calling
> variants. When it is called with a buffer-like object as value, it calls
> the C API function setsockopt() with optval=buffer.buf and
> optlen=buffer.len. When value is an integer, setsockopt() packs it as
> int32 and sends it with optlen=4.
>
> ---
> # example.py
> import socket
> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,
> b'\x00\x00\x00\x00')
> sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> ---
>
> $ strace -e setsockopt ./python example.py
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
>
>
> For AF_ALG (Linux Kernel crypto) I need a way to call the C API function
> setsockopt() with optval=NULL and optlen as any arbitrary number. I have
> been playing with multiple ideas. So far I liked the idea of
> value=(None, int) most.
>
> setsockopt(socket.SOL_ALG, socket.ALG_SET_AEAD_AUTHSIZE, (None, taglen))

Would this new functionality be open-ended? What would happen if you did

sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, (None, 4))

On Linux, this seems to fail with errno = EFAULT, which would be fine.
But this does not seem to be specified by Posix, and I imagine other
platforms might crash. On the other hand, these sort of holes are
already present in the socket module. E.g. with MSG_TRUNC
 you can apparently get Python to
copy from unallocated memory.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Method signatures in the datetime module documentation

[Martin Panter]

On 4 August 2016 at 00:24, Alexander Belopolsky
 wrote:
> On Fri, Jul 29, 2016 at 12:55 PM, Alexander Belopolsky
>  wrote:
>>
>> How should I present the signature of the new replace method in the
>> documentation?
>
> Having not seeing any replies, let me make a proposal:
>
> datetime.replace(hour=self.hour, minute=self.minute, second=self.second,
> microsecond=self.microsecond, tzinfo=self.tzinfo, *, fold=self.fold)
>
> while not a valid method signature, this seems to correctly reflect
> what replace() does.

Does that mean replace() accepts most of its arguments as either
positional or keyword arguments? IMO when the documentation uses the
“name=default” notation, it implies keyword argument support, while
the [name] notation implies only supporting a positional argument. If
replace() actually supported these keywords all along, then I am happy
with this proposal.

There are a few bug reports that may be relevant:

https://bugs.python.org/issue13386: notation where there is no simple
default value
https://bugs.python.org/issue23738: notation for defaults for
positional-only parameters
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] stuck issue 26826

[Martin Panter]

On 03/08/2016, Marcos Dione  wrote:
> Hi pythonistas. A couple of moths ago I opened an issue in the bug
> tracker for adding a new syscall to the os module. It's based on new
> developments in the Linux kernel. Here's the link:
>
> https://bugs.python.org/issue26826

To give more context, this is about adding support for Linux’s
copy_file_range() system call.

> After two months and a half I managed to create a nice patch with
> unit tests and documentation (yay!), but then the issue went cold. I
> would like to know how to proceed to get the issue going again.

I thought the main problem remaining was getting concensus about
adding it, since a couple people mentioned waiting for glibc support.
I don’t have much personal opinion on this, but FWIW I don’t see much
disadvantage in adding it now.

I did mean to look over your latest patch, but that has drifted
towards the bottom of my list of things to do :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should we fix these errors?

[Martin Panter]

FYI there is also a bug tracker report about this:
https://bugs.python.org/issue27587

On 23 July 2016 at 13:22, Christian Heimes  wrote:
> On 2016-07-22 16:36, Guido van Rossum wrote:
>> Somebody did some research and found some bugs in CPython (IIUC). The
>> published some questionable fragments. If there's a volunteer we could
>> probably easily fix these. (I know we already have occasional Coverity
>> scans and there are other tools too (anybody try lgtm yet?) But this
>> seems honest research (also Python leaves Ruby in the dust :-):
>>
>> http://www.viva64.com/en/b/0414/
>
> I had a closer look at the report. About half of the bugs, maybe more
> are not in the C code of CPython but in OpenSSL code. I really mean
> OpenSSL code, not _ssl.c and _hashopenssl.c. It's safe to assume that
> they forgot to exclude external dependencies.
>
> The issues in ASN1_PRINTABLE_type() [N2], BN_mask_bits() [N4 bn_lib.c,
> digest.c, evp_enc.c], dh_cms_set_peerkey() [N5, dh_ameth.c] and
> cms_env_set_version() [N6, cms_env.c] are all OpenSSL issues and should
> be reported to OpenSSL.
>
> Guido, did the company contact you or do you have Pavel Belikov's email
> address?

Perhaps you can contact him via the email address at
.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fun with ExitStack

[Martin Panter]

On 18 July 2016 at 23:40, Barry Warsaw  wrote:
> I was trying to debug a problem in some work code and I ran into some
> interesting oddities with contextlib.ExitStack and other context managers in
> Python 3.5.
>
> This program creates a temporary directory, and I wanted to give it a --keep
> flag to not automatically delete the tempdir at program exit.  I was using an
> ExitStack to manage a bunch of resources, and the temporary directory is the
> first thing pushed into the ExitStack.  At that point in the program, I check
> the value of --keep and if it's set, I use ExitStack.pop_all() to clear the
> stack, and thus, presumably, prevent the temporary directory from being
> deleted.
>
> There's this relevant quote in the contextlib documentation:
>
> """
> Each instance [of an ExitStack] maintains a stack of registered callbacks that
> are called in reverse order when the instance is closed (either explicitly or
> implicitly at the end of a with statement). Note that callbacks are not
> invoked implicitly when the context stack instance is garbage collected.
> """
>
> However if I didn't save the reference to the pop_all'd ExitStack, the tempdir
> would be immediately deleted.  If I did save a reference to the pop_all'd
> ExitStack, the tempdir would live until the saved reference went out of scope
> and got refcounted away.
>
> As best I can tell this happens because TemporaryDirectory.__init__() creates
> a weakref finalizer which ends up calling the _cleanup() function.  Although
> it's rather difficult to trace, it does appear that when the ExitStack is
> gc'd, this finalizer gets triggered (via weakref.detach()), thus causing the
> _cleanup() method to be called and the tmpdir to get deleted.

This seems to be missing from the documentation, but when you delete a
TemporaryDirectory instance without using a context manager or
cleanup(), it complains, and then cleans up anyway:

>>> d = TemporaryDirectory()
>>> del d
/usr/lib/python3.5/tempfile.py:797: ResourceWarning: Implicitly
cleaning up 
  _warnings.warn(warn_message, ResourceWarning)

Perhaps you will have to use the lower-level mkdtemp() function
instead if you want the option of making the “temporary” directory
long-lived.

> I "fix" this by
> doing:
>
> def __init__(self):
> tmpdir = TemporaryDirectory()
> self._tmpdir = (tmpdir.name if keep
> else self.resources.enter_context(tmpdir))
>
> There must be more to the story because when __init__() exits in the --keep
> case, tmpdir should have gotten refcounted away and the directory deleted, but
> it doesn't.  I haven't dug down deep enough to figure that out.
>
> Now, while I was debugging that behavior, I ran across more interesting bits.
> I put this in a file to drive some tests:
>
> --snip snip-
> with ExitStack() as resources:
> print('enter context')
> tmpdir = resources.enter_context(X())
> resources.pop_all()
> print('exit context')
> --snip snip-
>
> Let's say X is:
>
> class X:
> def __enter__(self):
> print('enter Foo')
> return self
>
> def __exit__(self, *args, **kws):
> print('exit Foo')
> return False
>
> the output is:
>
> enter context
> enter Foo
> exit context
>
> So far so good.  A fairly standard context manager class doesn't get its
> __exit__() called even when the program exits.  Let's try this:
>
> @contextmanager
> def X():
> print('enter bar')
> yield
> print('exit bar')
>
> still good:
>
> enter context
> enter bar
> exit context
>
> Let's modify X a little bit to be a more common idiom:
>
> @contextmanager
> def X():
> print('enter foo')
> try:
> yield
> finally:
> print('exit foo')
>
> enter context
> enter foo
> exit foo
> exit context
>
> Ah, the try-finally changes the behavior!  There's probably some documentation
> somewhere that defines how a generator gets finalized, and that triggers the
> finally clause, whereas in the previous example, nothing after the yield gets
> run.  I just can't find that anything that would describe the observed
> behavior.

I suspect the documentation doesn’t spell everything out, but my
understanding is that garbage collection of a generator instance
effectively calls its close() method, triggering any “finally” and
__exit__() handlers.

IMO, in some cases if a generator would execute these handlers and
gets garbage collected, it is a programming error because the
programmer should have explicitly called generator.close(). In these
cases, it would be nice to emit a ResourceWarning, just like
forgetting to close a file, or delete your temporay directory above.

But maybe there are other cases where there is no valid reason to emit
a warning. I have hesitated in suggesting this change in the past, but
I don’t remember why. One reason is it might be an annoyance with code
that also wants to handle non-generator iterators that don’t have a
close() method. In a 

Re: [Python-Dev] [Python-checkins] devguide: Star self as idlelib expert. Mark other 2 as inactive.

[Martin Panter]

On 12 July 2016 at 04:14, Martin Panter <vadmium...@gmail.com> wrote:
> On 12 July 2016 at 03:47, terry.reedy <python-check...@python.org> wrote:
>> https://hg.python.org/devguide/rev/cc1c0dd798e7
>
> Terry it looks like you accidentally added Christian back (undoing
> <https://hg.python.org/devguide/rev/cc1c0dd798e7>)

Sorry, you can ignore that, I see you already reverted it :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] devguide: Star self as idlelib expert. Mark other 2 as inactive.

[Martin Panter]

On 12 July 2016 at 03:47, terry.reedy  wrote:
> https://hg.python.org/devguide/rev/cc1c0dd798e7

Terry it looks like you accidentally added Christian back (undoing
)

> -xml.parsers.expat
> -xml.sax
> +xml.parsers.expat christian.heimes
> +xml.sax   christian.heimes
>  xml.sax.handler
>  xml.sax.saxutils
>  xml.sax.xmlreader
> @@ -319,7 +319,7 @@
>  bytecodebenjamin.peterson, pitrou, georg.brandl, yselivanov
>  context managersncoghlan
>  coverity scan   christian.heimes, brett.cannon, twouters
> -cryptographygregory.p.smith, dstufft
> +cryptographychristian.heimes, gregory.p.smith, dstufft
>  data formatsmark.dickinson, georg.brandl
>  databaselemburg
>  devguidencoghlan, eric.araujo, ezio.melotti, willingc
> @@ -341,7 +341,7 @@
>  georg.brandl
>  str.format  eric.smith
>  testing michael.foord, pitrou, ezio.melotti
> -test coverage   giampaolo.rodola
> +test coverage   giampaolo.rodola, christian.heimes
>  threads pitrou
>  time and dates  lemburg, belopolsky
>  unicode lemburg, ezio.melotti, haypo, benjamin.peterson, pitrou
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Martin Panter]

> On 06/09/2016 08:52 AM, Guido van Rossum wrote:
> That leaves direct calls to os.urandom(). I don't think this should block
> either.

On 9 June 2016 at 22:22, Larry Hastings  wrote:
> Then it's you and me against the rest of the world ;-)
>
>
> Okay, it's decided: os.urandom() must be changed for 3.5.2 to never block on
> a getrandom() call.  It's permissible to take advantage of
> getrandom(GRND_NONBLOCK), but if it returns EAGAIN we must read from
> /dev/urandom.

So assuming this is the “final” decision, where to from here? I think
the latest change by Colm and committed by Victor already implements
this decision:

https://hg.python.org/cpython/rev/9de508dc4837

Getrandom() is still called, but if it would block, we fall back to
trying the less-secure Linux /dev/urandom, or fail if /dev/urandom is
missing. The Python hash seed is still set using this code. And
os.urandom() calls this code. Random.seed() and SystemRandom still use
os.urandom(), as documented.

So I suggest we close the original mega bug thread
 as fixed. Unless people think
they can change Larry or Guido’s mind, we should focus further
discussion on any changes for 3.6.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: Minor API improvements to bytes, bytearray, and memoryview

[Martin Panter]

On 7 June 2016 at 21:56, Nick Coghlan  wrote:
> On 7 June 2016 at 14:33, Paul Sokolovsky  wrote:
>> Ethan Furman  wrote:
>>> Deprecation of current "zero-initialised sequence" behaviour
>>> 
>>>
>>> Currently, the ``bytes`` and ``bytearray`` constructors accept an
>>> integer argument and interpret it as meaning to create a
>>> zero-initialised sequence of the given size::
>>>
>>>  >>> bytes(3)
>>>  b'\x00\x00\x00'
>>>  >>> bytearray(3)
>>>  bytearray(b'\x00\x00\x00')
>>>
>>> This PEP proposes to deprecate that behaviour in Python 3.6, and
>>> remove it entirely in Python 3.7.
>>
>> Why the desire to break applications of thousands and thousands of
>> people?
>
> Same argument as any deprecation: to make existing and future defects
> easier to find or easier to debug.
>
> That said, this is the main part I was referring to in the other
> thread when I mentioned some of the constructor changes were
> potentially controversial and probably not worth the hassle - it's the
> only one with the potential to break currently working code, while the
> others are just a matter of choosing suitable names.

An argument against deprecating bytearray(n) in particular is that
this is supported in Python 2. I think I have (ab)used this fact to
work around the problem with bytes(n) in Python 2 & 3 compatible code.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C99

[Martin Panter]

On 4 June 2016 at 06:11, Benjamin Peterson  wrote:
> PEP 7 requires CPython to use C code conforming to the venerable C89
> standard. Traditionally, we've been stuck with C89 due to poor C support
> in MSVC. However, MSVC 2013 and 2015 implement the key features of C99.
> C99 does not offer anything earth-shattering; here are the features I
> think we'd find most interesting:
> - Variable declarations can be on any line: removes possibly the most
> annoying limitation of C89.
> - Inline functions: We can make Py_DECREF and Py_INCREF inline functions
> rather than unpleasant macros.
> - C++-style line comments: Not an killer feature but commonly used.
> - Booleans

My most-missed C99 feature would be designated initializers. Does MSVC
support them? It might allow you to do away with those giant pasted
slot tables, and just write the slots you need:

PyTypeObject PyUnicodeIter_Type = {
PyVarObject_HEAD_INIT(_Type, 0)
.tp_name = "str_iterator",
.tp_basicsize = sizeof(unicodeiterobject),
.tp_dealloc = unicodeiter_dealloc,
.tp_getattro = PyObject_GenericGetAttr,
.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
.tp_traverse = unicodeiter_traverse,
.tp_iter = PyObject_SelfIter,
.tp_iternext = unicodeiter_next,
.tp_methods = unicodeiter_methods,
};

> So, what say you to updating PEP 7 to allow C99 features for Python 3.6
> (in so much as GCC and MSVC support them)?

Sounds good for features that are well-supported by compilers that
people use. (Are there other compilers used than just GCC and MSVC?)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ctypes compatibility with 2.3

[Martin Panter]

On 12 May 2016 at 01:05, Brett Cannon  wrote:
>
>
> On Wed, 11 May 2016 at 09:28 Brett Cannon  wrote:
>>
>> On Wed, 11 May 2016 at 09:07 Thomas Heller  wrote:
>>>
>>> Am 11.05.2016 um 18:04 schrieb Brett Cannon:
>>> >
>>> >
>>> > On Wed, 11 May 2016 at 04:35 Thomas Heller >> > > wrote:
>>> > For me it is totally ok to lift this restriction.
>>> >
>>> >
>>> > Great! I'll also update PEP 291.
>>>
>>> Cool.  While you're at it, the compatibility restriction for
>>> modulefinder could also be lifted.
>>
>>
>> Will do.
>
>
> PEP 291 no longer lists any restrictions on ctypes or modulefinder.

Thanks everyone for your responses. I will look at removing the
notices in the code when I get a chance. That would probably involve
reverting

https://hg.python.org/cpython/rev/381a72ab5fb8
And also the modulefinder.py comment

There are also these commits that could be backported
https://hg.python.org/cpython/rev/0980034adaa7 (ctypes)
https://hg.python.org/cpython/diff/627db59031be/Lib/modulefinder.py
but it might be safer just to leave the compatibility code there,
perhaps with a clarifying comment.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] ctypes compatibility with 2.3

[Martin Panter]

I am working on , to fix shell
injection problems with ctypes.util.find_library(). The proposal for
Python 3 is to change os.popen(shell-script) calls to use
subprocess.Popen().

However the Python 2.7 version of the module has a comment which says
“This file should be kept compatible with Python 2.3, see PEP 291.”
Looking at , it is not
clear why we have to maintain this compatibility. My best guess is
that there may be an external ctypes package that people want(ed) to
keep compatible with 2.3, and also keep synchronized with 2.7.

I would like to lift this restriction to at least 2.4, because that is
when the “subprocess” module was added. I notice that there is already
code that relies on the list.sort(key=...) feature, which was added in
2.4. Ideally I would prefer to drop the restriction and only require
2.7 compatibility. Would either of these options be a problem?

If it is a problem, I think it is still possible to avoid the shell by
passing a sequence of program arguments to os.popen(). But I prefer
not to do this, because the function is marked as deprecated, and the
code would be substantially different to Python 3.

-Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tracker Etiquette

[Martin Panter]

On 7 May 2016 at 23:23, Senthil Kumaran  wrote:
>
> On Sat, May 7, 2016 at 4:17 PM, MRAB  wrote:
>>
>> I think you shouldn't delete them. It would be better just to say that
>> you've changed your mind and explain why.
>
>
> I support this. Please leave your new comments correcting previous one and
> support your current stance. I think, it is alright to make revisions in
> comments.

Taking a middle ground between this and what Brett said, I would say
it is okay to delete stuff if it is fresh and hasn’t been replied to
(but maybe leave a message if it is not obvious why you delete it). On
the other hand, avoid deleting stuff if it would harm the overall
thread.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Convert int() to size_t in Python/C

[Martin Panter]

On 29 April 2016 at 18:11, Marcos Dione  wrote:
> On Fri, Apr 29, 2016 at 12:18:46PM -0400, Random832 wrote:
>> On Fri, Apr 29, 2016, at 10:45, Marcos Dione wrote:
>> > One possible solution hat was suggested to me in the #python IRC
>> > channel was to use that, then test if the resulting value is negative,
>> > and adjust accordingly, but I wonder if there is a cleaner, more general
>> > solution (for instance, what if the type was something else, like loff_t,
>> > although for that one in particular there *is* a convertion
>> > function/macro).
>>
>> In principle, you could just use PyLong_AsUnsignedLong (or LongLong),
>> and raise OverflowError manually if the value happens to be out of
>> size_t's range. (99% sure that on every linux platform unsigned long is
>> the same size as size_t.
>>
>> But it's not like it'd be the first function in OS to call a system call
>> that takes a size_t. Read just uses Py_ssize_t. Write uses the buffer
>> protocol, which uses Py_ssize_t. How concerned are you really about the
>> lost range here? What does the system call return (its return type is
>> ssize_t) if it writes more than SSIZE_MAX bytes? (This shouldn't be hard
>> to test, just try copying a >2GB file on a 32-bit system)

I would probably just use Py_ssize_t, since that is what the return
value is. Otherwise, a large positive count input could return a
negative value, which would be inconsistent, and could be mistaken as
an error.

> It's a very good point, but I don't have any 32 bits systems around
> with a kernel-4.5. I'll try to figure it out and/or ask in the kernel ML.

Maybe you can compile a 32-bit program and run it on a 64-bit computer
(gcc -m32).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Convert int() to size_t in Python/C

[Martin Panter]

On 29 April 2016 at 18:25, Random832  wrote:
> On Fri, Apr 29, 2016, at 14:11, Marcos Dione wrote:
>> These are not output parameters, even if they're pointers. they'r
>> using the NULL pointer to signal that the current offsets should not be
>> touched, to differentiate from a offset of 0. Something that in Python we
>> would use None.
>
> That's not actually true according to the documentation. (And if it
> were, they could simply use -1 rather than a null pointer)
> . . .
>*  If off_in is not NULL, then off_in must point to a buffer that
>   specifies the starting offset where bytes from fd_in will be
>   read.
>   The file offset of fd_in is not changed, >>>but off_in is
>   adjusted
>   appropriately.<<<

Linux’s sendfile() syscall takes a similar offset parameter that may
be updated, but Python’s os.sendfile() wrapper does not return the
updated offset. Do you think we need to return the updated offsets for
copy_file_range()?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Not receiving bug tracker emails

[Martin Panter]

On 14 April 2016 at 08:51, Serhiy Storchaka  wrote:
> On 13.04.16 07:39, Terry Reedy wrote:
>>
>> On 4/4/2016 5:05 PM, Terry Reedy wrote:
>>
>> Since a few days, I am getting bug tracker emails again, in my Inbox.  I
>> just got a Rietveld review in the Inbox and I believe it went there
>> directly instead of first to Junk.  Thank you to whoever made the
>> improvements.
>
>
> AFAIK David just disabled IPv6 support.
>
> Most bug tracker emails still went in the Spam folder. I have a filter for
> Roundap emails, but there is no any mark that I can use for filtering
> Rietveld emails.

FWIW I set up the following filter in Gmail for Rietveld reviews:

Matches: http://bugs.python.org/review
Do this: Never send it to Spam

I suspect it helps, but occasionally I think stuff still goes to spam.
(Just don’t tell this secret rule to actual spammers :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Most 3.x buildbots are green again, please don't break them and watch them!

[Martin Panter]

On 14 April 2016 at 09:15, Serhiy Storchaka  wrote:
> On 13.04.16 14:40, Victor Stinner wrote:
>> By the way, I don't understand why "AMD64 OpenIndiana 3.x" is
>> considered as stable since it's failing with multiple issues since
>> many months and nobody is working on these failures. I suggest to move
>> this buildbot back to the unstable category.
>
> I think the main cause is the lack of memory in this buildbot. I tried to
> minimize memory consumption and leaks, but some leaks are left, and they
> provoke other tests failures, and additional resource leaks. Would be nice
> to add a feature for running every test in separate subprocess. This will
> isolate the effect of failed tests.

Last time I looked into the Open Indiana buildbot, I concluded that
the biggest problem was Python using fork() to spawn subprocesses. I
understand that OS does not do “memory overcommitment” like Linux
does, so every time you fork, the OS has to double the amount of
memory that is reserved. It is ironic, but running each test using the
current subprocess module (which uses fork) would probably make the
problem worse.

I suspect using posix_spawn() if possible would help a lot. But this
was rejected in  for not being
flexible enough and making maintainence too complicated.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Not receiving bug tracker emails

[Martin Panter]

On 30 March 2016 at 13:30, R. David Murray  wrote:
> Anyone know how to find out what changed from Google's POV?  As far as
> we know nothing changed at the bugs end, but it is certainly possible
> that something did change in the hosting infrastructure without our
> knowledge.  Knowing what is setting google off would help track it down,
> if so...or perhaps something changed at the google end, in which case we
> *really* need to know what.

My only guess is that Google decided to get stricter regarding
something mentioned in
, maybe
something in its sending guidelines. Perhaps to do with IPv6 DNS
.

FYI I am now working around the problem for myself by pointing my
bugs.python.org account at a Yahoo email address, and setting up Yahoo
to forward all emails to my G Mail address.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Not receiving bug tracker emails

[Martin Panter]

For the last ~36 hours I have stopped receiving emails for messages
posted in the bug tracker. Is anyone else having this problem? Has
anything changed recently?

I have had it set to send to my gmail.com address since the beginning.
At the moment the last bug message email is
 with “Date: Mon, 28 Mar
2016 12:19:49 +”. I have checked spam and they are not going
there.

Earlier this year I had to set up a rule to avoid lots of tracker
emails suddenly going to spam. I suspect there was something about the
emails that Google doesn’t like (though I don’t understand the
technical details). Maybe this has recently gotten worse at the Google
end?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Martin Panter]

On 15 March 2016 at 08:04, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 15 March 2016 at 15:15, Martin Panter <vadmium...@gmail.com> wrote:
>> _freeze_importlib.o: _freeze_importlib.c Makefile
>>
>> _freeze_importlib: _freeze_importlib.o [. . .]
>> $(LINKCC) [. . .]
>>
>> importlib_external.h: _bootstrap_external.py _freeze_importlib
>> _freeze_importlib _bootstrap_external.py importlib_external.h
>>
>> importlib.h: _bootstrap.py _freeze_importlib
>> _freeze_importlib _bootstrap.py importlib.h
>
> Ah, I understand now - the fundamental problem is with a checked in
> file depending on a non-checked-in file, so if you clean out all the
> native build artifacts when cross-compiling, the makefile will attempt
> to create target versions of all the helper utilities (pgen,
> _freeze_importlib, argument clinic, etc).
>
> Would it help to have a "make bootstrap" target that touched all the
> checked in generated files with build dependencies on non-checked-in
> files, but only after first touching the expected locations of the
> built binaries they depend on?

That sounds similar to “make touch”, with a couple differences. One
trouble I forsee is the conflict with shared prerequisites. E.g. “make
bootstrap” would have to create some dummy object files as
prerequisites of the pgen program, but we would first have build
others e.g. Parser/acceler.o properly for the main Python library. It
all feels way too complicated to me. The Python build system is
complicated enough as it is.

Maybe it is simplest to just add something in the spirit of Xavier’s
suggested patch. This would mean that we keep the generated files
checked in (to help with Windows and cross compiled builds), we keep
the current rules that force normal makefile builds to blindly
regenerate the files, but we add some flag or configure.ac check to
disable this regeneration if desired.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Martin Panter]

On 15 March 2016 at 04:32, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 15 March 2016 at 10:49, Martin Panter <vadmium...@gmail.com> wrote:
>> One more idea I am considering is to decouple the makefile rules from
>> the main build. So to update the generated files you would have to run
>> a separate command like “make graminit” or “make frozen”. The normal
>> build would never regenerate them; although perhaps it could still
>> result in an error or warning if they appear out of date.
>
> Some of them used to work that way and it's an incredible PITA when
> you actually *are* working on one of the affected bits of the
> interpreter - working on those parts is rare, so if there are special
> cases to remember, you can pretty much guarantee we'll have forgotten
> them by the time we work on that piece again.

Perhaps if we wrapped them all up in a common “make regenerate”
target, so it is only one special case to remember? Maybe you could
include other stuff like “make clinic” in that as well. Or you could
include the special commands in the warning messages.

> However, it would be worth reviewing the explicit dependencies on
> "Makefile" and see whether they could be replaced by dependencies on
> Makefile.pre.in instead. I'm confident that will work just fine for
> the importlib bootstrapping, and I suspect it will work for the other
> pregenerated-and-checked-in files as well.

The problem is not the reference to Makefile. The graminit files do
not depend on Makefile. The bigger problem is that the checked-in
files depend on compiled programs. This is a summary of the current
rules for importlib:

_freeze_importlib.o: _freeze_importlib.c Makefile

_freeze_importlib: _freeze_importlib.o [. . .]
$(LINKCC) [. . .]

importlib_external.h: _bootstrap_external.py _freeze_importlib
_freeze_importlib _bootstrap_external.py importlib_external.h

importlib.h: _bootstrap.py _freeze_importlib
_freeze_importlib _bootstrap.py importlib.h

So importlib.h depends on the _freeze_importlib compiled program (and
only indirectly on Makefile). The makefile says we have to compile
_freeze_importlib before checking if importlib.h is up to date.

Gnu Make has order-only prerequisites
<https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html>,
which it seems we could abuse to do most of what we want. But (1) I’m
not sure if we can restrict ourselves to Gnu Make, and (2) it is a
horrible hack and would always compile _freeze_importlib even if it is
never run.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Martin Panter]

On 14 March 2016 at 13:26, R. David Murray <rdmur...@bitdance.com> wrote:
> On Mon, 14 Mar 2016 03:04:08 -, "Gregory P. Smith" <g...@krypto.org> 
> wrote:
>> On Sun, Mar 13, 2016 at 7:41 PM Martin Panter <vadmium...@gmail.com> wrote:
>> > Include/graminit.h
>> > Python/graminit.c
>> > Python/importlib.h
>> > Python/importlib_external.h
>> >
>> > A question for other Python developers: Why are these generated files
>> > stored in the repository? [. . .]
>>
>> They should not be regenerated every build, if they are, that seems like a
>> bug in the makefile to me (or else the timestamp checks that make does vs
>> how your code checkout happened).

The reason the current Python 3 build regenerates some files, is
because of the makefile prerequisites. For example, Include/graminit.h
currently depends on Parser/pgen, which needs to be compiled for the
native build host.

>> Having them checked in is convenient for
>> cross builds as it is one less thing that needs a build-host-arch build.
>
> [. . .]
> And yes, checking in these platform-independent artifacts is very
> intentional: less to build, fewer external dependencies in the build
> process...you don't need to *have* python to *build* python, which you
> would have to if they were not checked in.

Okay so it sounds like the generated files (more listed in .hgtouch)
have to stay. Reasons given:

* Some need Python to generate them (bootstrap problem)
* Relied on by Windows build system
* General convenience (less build steps, less prerequisites, less
things to go wrong)

One more idea I am considering is to decouple the makefile rules from
the main build. So to update the generated files you would have to run
a separate command like “make graminit” or “make frozen”. The normal
build would never regenerate them; although perhaps it could still
result in an error or warning if they appear out of date.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Martin Panter]

On 13 March 2016 at 01:13, Russell Keith-Magee <russ...@keith-magee.com> wrote:
> The patches that I've uploaded to Issue23670 [1] show a full cross-platform
> [1] http://bugs.python.org/issue23670
> build process. After you apply that patch, the iOS directory contains a
> meta-Makefile that manages the build process.

Thanks very much for pointing that out. This has helped me understand
a lot more things. Only now do I realize that the four files generated
by pgen and _freeze_importlib are actually already committed into the
Mercurial repository:

Include/graminit.h
Python/graminit.c
Python/importlib.h
Python/importlib_external.h

A question for other Python developers: Why are these generated files
stored in the repository? The graminit ones seem to have been there
since forever (1990). It seems the importlib ones were there due to a
bootstrapping problem, but now that is solved. Antoine
<https://bugs.python.org/issue14928#msg163048> said he kept them in
the repository on purpose, but I want to know why.

If we ignore the cross compiling use case, would there be any other
consequences of removing these generated files from the repository?
E.g. would it affect the Windows build process?

I have two possible solutions in mind: either remove the generated
files from the repository and always build them, or keep them but do
not automatically regenerate them every build. Since they are
generated files, not source files, I would prefer to remove them, but
I want to know the consequences first.

> On Sat, Mar 12, 2016 at 8:48 AM, Martin Panter <vadmium...@gmail.com> wrote:
>> On 11 March 2016 at 23:16, Russell Keith-Magee <russ...@keith-magee.com>
>> wrote:
>> >
>> > On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter <vadmium...@gmail.com>
>> > wrote:
>> >> I don't understand. After I run Make, it looks like I get working
>> >> executables leftover at Programs/_freeze_importlib and Parser/pgen. Do
>> >> you mean to install these programs with "make install" or something?
>> >
>> >
>> > Making them part of the installable artefacts would be one option, but
>> > they
>> > don't have to be installed, just preserved.
>>
>> What commands are you running that cause them to not be preserved at
>> the end of the build?
>
>
> I don't know - this is where I hit the end of my understanding of the build
> process. All I know for certain is that 3.4.2 doesn't have this problem;
> 3.5.1 does, and Issue22359 (fixed in [3]) is the source of the problem. A
> subsequent fix [4] introduced an additional problem with _freeze_importlib.
>
> [3] https://hg.python.org/cpython/rev/565b96093ec8
> [4] https://hg.python.org/cpython/rev/02e3bf65b2f8

After my realization about the generated files, I think I can solve
this one. Before the changes you identified, the build process
probably thought the generated files were up to date, so it didn't
need to cross-compile pgen or _freeze_importlib. If the generated file
timestamps were out of date (e.g. depending on the order they are
checked out or extracted), the first native build stage would have
fixed them up.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Martin Panter]

On 11 March 2016 at 23:16, Russell Keith-Magee <russ...@keith-magee.com> wrote:
>
> On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter <vadmium...@gmail.com> wrote:
>> make clean  # Work around confusion with existing in-source build
>> mkdir native
>> (cd native/ && ../configure)
>> make -C native/ Parser/pgen
>> mkdir mingw
>> (cd mingw/ && ../configure --host=i486-mingw32 --build=x86)
>> make -C mingw/ PGEN=../native/Parser/pgen
>>
>> Actually it was not as smooth as the above commands, because pgen
>> tends to get overwritten with a cross-compiled version. Perhaps we
>> could add a PGEN_FOR_BUILD override, like HOSTPGEN in the patch used
>> at
>> <https://wayback.archive.org/web/20160131224915/http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html>.
>>
> That might fix the pgen problem,  but _freeze_importlib still remains. I
> suppose the same thing might be possible for _freeze_importlib as well…

Yes. I never got up to it failing in my experiments, but I think I
would propose a FREEZE_IMPORTLIB override variable for that too.

>> Would it be more correct to say instead that in 3.4 you did a separate
>> native build step, precompiling pgen and _freeze_importlib for the
>> native build host? Then you hoped that the child Make was _not_
>> invoked in the cross-compilation stage and your precompiled
>> executables would not be rebuilt?
>
>
> Yes - as far as I can make out (with my admittedly hazy understanding), that
> appears to be what is going on. Although it’s not that I “hoped” the build
> wouldn’t happen on the second pass - it was the behavior that was previously
> relied, and on was altered.

Do you have a copy/patch/link/etc to the actual commands that you
relied on? It’s hard to guess exactly what you were doing that broke
without this information.

>> > As best as I can work out, the solution is to:
>> >
>> > (1) Include the parser generator and _freeze_importlib as part of the
>> > artefacts of local platform. That way, you could use the version of pgen
>> > and
>> > _freeze_importlib that was compiled as part of the local platform build.
>> > At
>> > present, pgen and _freeze_importlib are used during the build process,
>> > but
>> > aren’t preserved at the end of the build; or
>>
>> I don’t understand. After I run Make, it looks like I get working
>> executables leftover at Programs/_freeze_importlib and Parser/pgen. Do
>> you mean to install these programs with “make install” or something?
>
>
> Making them part of the installable artefacts would be one option, but they
> don’t have to be installed, just preserved.

What commands are you running that cause them to not be preserved at
the end of the build?

> For example, as a nasty hack, I’ve been able to use this approach to get the
> build working for 3.5. After the native build, I copy _freeze_importlib to a
> “safe” location. I then copy it back into place prior to the target build.
> It works, but it’s in no way suitable for a final build.
>
>>
>> > (2) Include some concept of the “local compiler” in the build process,
>> > which
>> > can be used to compile pgen and _freeze_importlib; or
>>
>> On the surface solution (2) sounds like the ideal fix. But I guess the
>> local native compiler might also require a separate set of CPPFLAGS,
>> pyconfig.h settings etc. In other words it is sounding like a whole
>> separate “configure” run. I am thinking it might be simplest to just
>> require this native “configure” run to be done manually.
>
>
> That run is going to happen anyway, since you have to compile and build for
> the native platform.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Martin Panter]

Hi Russell. Sorry for the minor ~1 month delay in replying :)

I have been doing some experimenting to see what is involved in
cross-compiling Python (Native host = Linux, target = Windows via
mingw and some patches). So I have a slightly better understanding of
the problem than before.

On 16 February 2016 at 01:41, Russell Keith-Magee
 wrote:
> In order to build for a host platform, you have to compile for a local
> platform first - for example, to compile an iOS ARM64 binary, you have to
> compile for OS X x86_64 first. This gives you a local platform version of
> Python you can use when building the iOS version.
>
> Early in the Makefile, the variable PYTHON_FOR_BUILD is set. This points at
> the CPU-local version of Python that can be invoked, which is used for
> module builds, and for compiling the standard library source code. This is
> set by —host and —build flags to configure, plus the use of CC and LDFLAGS
> environment variables to point at the compiler and libraries for the
> platform you’re compiling for, and a PATH variable that provides the local
> platform’s version of Python.

So far I haven’t succeeded with my Min GW cross build and am
temporarily giving up due to incompatibilities. But my attempts looked
a bit like this:

make clean  # Work around confusion with existing in-source build
mkdir native
(cd native/ && ../configure)
make -C native/ Parser/pgen
mkdir mingw
(cd mingw/ && ../configure --host=i486-mingw32 --build=x86)
make -C mingw/ PGEN=../native/Parser/pgen

Actually it was not as smooth as the above commands, because pgen
tends to get overwritten with a cross-compiled version. Perhaps we
could add a PGEN_FOR_BUILD override, like HOSTPGEN in the patch used
at 
.

> There are two places where special handling is required: the compilation and
> execution of the parser generator, and _freeze_importlib. In both cases, the
> tool needs to be compiled for the local platform, and then executed.
> Historically (i.e., Py3.4 and earlier), this has been done by spawning a
> child MAKE to compile the tool; this runs the compilation phase with the
> local CPU environment, before returning to the master makefile and executing
> the tool. By spawning the child MAKE, you get a “clean” environment, so the
> tool is built natively. However, as I understand it, it causes problems with
> parallel builds due to race conditions on build rules. The change in
> Python3.5 simplified the rule so that child MAKE calls weren’t used, but
> that means that pgen and _freeze_importlib are compiled for ARM64, so they
> won’t run on the local platform.

You suggest that the child Make command happened to compile pgen etc
natively, rather than with the cross compiler. But my understanding is
that when you invoke $(MAKE), all the environment variables, configure
settings, etc, including the cross compiler, would be inherited by the
child.

Would it be more correct to say instead that in 3.4 you did a separate
native build step, precompiling pgen and _freeze_importlib for the
native build host? Then you hoped that the child Make was _not_
invoked in the cross-compilation stage and your precompiled
executables would not be rebuilt?

> As best as I can work out, the solution is to:
>
> (1) Include the parser generator and _freeze_importlib as part of the
> artefacts of local platform. That way, you could use the version of pgen and
> _freeze_importlib that was compiled as part of the local platform build. At
> present, pgen and _freeze_importlib are used during the build process, but
> aren’t preserved at the end of the build; or

I don’t understand. After I run Make, it looks like I get working
executables leftover at Programs/_freeze_importlib and Parser/pgen. Do
you mean to install these programs with “make install” or something?

> (2) Include some concept of the “local compiler” in the build process, which
> can be used to compile pgen and _freeze_importlib; or

On the surface solution (2) sounds like the ideal fix. But I guess the
local native compiler might also require a separate set of CPPFLAGS,
pyconfig.h settings etc. In other words it is sounding like a whole
separate “configure” run. I am thinking it might be simplest to just
require this native “configure” run to be done manually.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Martin Panter]

On 15 February 2016 at 08:24, Russell Keith-Magee
 wrote:
> Hi all,
>
> I’ve been working on developing Python builds for mobile platforms, and I’m
> looking for some help resolving a bug in Python’s build system.
>
> The problem affects cross-platform builds - builds where you are compiling
> python for a CPU architecture other than the one on the machine that is
> doing the compilation. This requirement stems from supporting mobile
> platforms (iOS, Android etc) where you compile on your laptop, then ship the
> compiled binary to the device.
>
> In the Python 3.5 dev cycle, Issue 22359 [1] was addressed, fixing parallel
> builds. However, as a side effect, this patch broke (as far as I can tell)
> *all* cross platform builds. This was reported in issue 22625 [2].
>
> Since that time, the problem has gotten slightly worse; the addition of
> changeset 95566 [3] and 95854 [4] has cemented the problem. I’ve been able
> to hack together a fix that enables me to get a set of binaries, but the
> patch is essentially reverting 22359, and making some (very dubious)
> assumptions about the order in which things are built.
>
> Autoconf et al aren’t my strong suit; I was hoping someone might be able to
> help me resolve this issue.

Would you mind answering my question in
? In particular, how did
cross-compiling previously work before these changes. AFAIK Python
builds a preliminary Python executable which is executed on the host
to complete the final build. So how do you differentiate between host
and target compilers etc?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 515: Underscores in Numeric Literals

[Martin Panter]

On 11 February 2016 at 11:12, Chris Angelico  wrote:
> On Thu, Feb 11, 2016 at 7:22 PM, Georg Brandl  wrote:

The following extensions are open for discussion:

>> * Allowing underscores in string arguments to the ``Decimal`` constructor.  
>> It
>>   could be argued that these are akin to literals, since there is no Decimal
>>   literal available (yet).
>>
>> * Allowing underscores in string arguments to ``int()`` with base argument 0,
>>   ``float()`` and ``complex()``.
>
> I'm -0.5 on both of these, with the caveat that if either gets done,
> both should be. Decimal() shouldn't be different from int() just
> because there's currently no way to express a Decimal literal; if
> Python 3.7 introduces such a literal, there'd be this weird rule
> difference that has to be maintained for backward compatibility, and
> has no justification left.

I would be weakly in favour of all relevant constructors being updated
to match the new syntax. The main reason is just consistency, and that
the documentation already kind of guarantees that the literal syntax
is supported (definitely for int and float; for complex it is too
vague).

To be consistent, the following minor extensions of the syntax should
be allowed, which are not legal Python literals: int("0_001"),
int("J_00", 20), float("0_001"), complex("0_001").

Maybe also with non-ASCII digits. However I tried writing Arabic-Indic
digits (U+0600 etc) and my web browser split the number apart when I
inserted an underscore. Maybe a right-to-left thing. But using
Devangari digits U+0966, U+0967: int("१_०००") (= 1_000). Non-ASCII
digits are apparently intentionally supported, but not documented:
.

> (As a side point, I would be fully in favour of Decimal literals. I'd
> also be in favour of something like "from __future__ import
> fraction_literals" so 1/2 would evaluate to Fraction(1,2) rather than
> 0.5. Hence I'm inclined *not* to support underscores in Decimal().)

Seems more like an argument to have the support in Decimal()
consistent with float() etc, i.e. all or nothing.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 515: Underscores in Numeric Literals

[Martin Panter]

On 12 February 2016 at 00:16, Steven D'Aprano  wrote:
> On Thu, Feb 11, 2016 at 06:03:34PM +, Brett Cannon wrote:
>> On Thu, 11 Feb 2016 at 02:13 Steven D'Aprano  wrote:
>>
>> > On Wed, Feb 10, 2016 at 08:41:27PM -0800, Andrew Barnert wrote:
>> >
>> > > And honestly, are you really claiming that in your opinion, "123_456_"
>> > > is worse than all of their other examples, like "1_23__4"?
>> >
>> > Yes I am, because 123_456_ looks like you've forgotten to finish typing
>> > the last group of digits, while 1_23__4 merely looks like you have no
>> > taste.
>> >
>>
>> OK, but the keyword in your sentence is "taste".
>
> I disagree. The key *idea* in my sentence is that the trailing
> underscore looks like a programming error. In my opinion, avoiding that
> impression is important enough to make trailing underscores a syntax
> error.
>
> I've seen a few people vote +1 for things like 123_j and 1.23_e99, but I
> haven't seen anyone in favour of trailing underscores. Does anyone think
> there is a good case for allowing trailing underscores?
>
>
>> If we update PEP 8 for our
>> needs to say "Numerical literals should not have multiple underscores in a
>> row or have a trailing underscore" then this is taken care of. We get a
>> dead-simple rule for when underscores can be used, the implementation is
>> simple, and we get to have more tasteful usage in the stdlib w/o forcing
>> our tastes upon everyone or complicating the rules or implementation.
>
> I think this is a misrepresentation of the alternative. As I see it, we
> have two alternatives:
>
> - one or more underscores can appear AFTER the base specifier or any digit;
+1

> - one or more underscores can appear BETWEEN two digits.
-0

Having underscores between digits is the main usage, but I don’t see
much harm in the more liberal version, unless it that makes the
specification or implementation too complex. Allowing stuff like
0x_100, 4.7_e3, and 1_j seems of slightly more benefit IMO than
disallowing 1_000_.

> To describe the second alternative as "complicating the rules" is, I
> think, grossly unfair. And if Serhiy's proposal is correct, the
> implementation is also no more complicated:
>
> # underscores after digits
> octinteger: "0" ("o" | "O") "_"* octdigit (octdigit | "_")*
> hexinteger: "0" ("x" | "X") "_"* hexdigit (hexdigit | "_")*
> bininteger: "0" ("b" | "B") "_"* bindigit (bindigit | "_")*
>
> # underscores between digits
> octinteger: "0" ("o" | "O") octdigit (["_"] octdigit)*
> hexinteger: "0" ("x" | "X") hexdigit (["_"] hexdigit)*
> bininteger: "0" ("b" | "B") bindigit (["_"] bindigit)*
>
>
> The idea that the second alternative "forc[es] our tastes on everyone"
> while the first does not is bogus. The first alternative also prohibits
> things which are a matter of taste:
>
> # prohibited in both alternatives
> 0_xDEADBEEF
> 0._1234
> 1.2e_99
> -_1

This one is already a valid variable identifier name.

> 1j_
>
>
> I think that there is broad agreement that:
>
> - the basic idea is sound
> - leading underscores followed by digits are currently legal
>   identifiers and this will not change
+1 to both
> - underscores should not follow the sign - +
> - underscores should not follow the decimal point .
> - underscores should not follow the exponent e|E
No strong opinion on these from me
> - underscores will not be permitted inside the exponent (even if
>   it is harmless, it's silly to write 1.2e9_9)
-0, it seems like a needless inconsistency, unless it somehow hurts
the implementation
> - underscores should not follow the complex suffix j
No opinion

> and only minor disagreement about:
>
> - whether or not underscores will be allowed after the base
>   specifier 0x 0o 0b
+0

> - whether or not underscores will be allowed before the decimal
>   point, exponent and complex suffix.
No opinion about directly before decimal point; +0 before exponent or
imaginary (complex) suffix.

> Can we have a show of hands, in favour or against the above two? And
> then perhaps Guido can rule on this one way or the other and we can get
> back to arguing about more important matters? :-)
>
> In case it isn't obvious, I prefer to say No to allowing underscores
> after the base specifier, or before the decimal point, exponent and
> complex suffix.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 515: Underscores in Numeric Literals

[Martin Panter]

I have occasionally wondered about this missing feature.

On 10 February 2016 at 22:20, Georg Brandl  wrote:
> Abstract and Rationale
> ==
>
> This PEP proposes to extend Python's syntax so that underscores can be used in
> integral and floating-point number literals.

This should extend complex or imaginary literals like 10_000j for consistency.

> Specification
> =
>
> * Trailing underscores are not allowed, because they look confusing and don't
>   contribute much to readability.
> * No underscore allowed after a sign in an exponent (``1e-_5``), because
>   underscores can also not be used after the signs in front of the number
>   (``-1e5``).
> [. . .]
>
> The production list for integer literals would therefore look like this::
>
>integer: decimalinteger | octinteger | hexinteger | bininteger
>decimalinteger: nonzerodigit [decimalrest] | "0" [("0" | "_")* "0"]
>nonzerodigit: "1"..."9"
>decimalrest: (digit | "_")* digit
>digit: "0"..."9"
>octinteger: "0" ("o" | "O") (octdigit | "_")* octdigit
>hexinteger: "0" ("x" | "X") (hexdigit | "_")* hexdigit
>bininteger: "0" ("b" | "B") (bindigit | "_")* bindigit
>octdigit: "0"..."7"
>hexdigit: digit | "a"..."f" | "A"..."F"
>bindigit: "0" | "1"
>
> For floating-point literals::
>
>floatnumber: pointfloat | exponentfloat
>pointfloat: [intpart] fraction | intpart "."
>exponentfloat: (intpart | pointfloat) exponent
>intpart: digit (digit | "_")*

This allows trailing underscores such as 1_.2, 1.2_, 1.2_e-5. Your
bullet point above suggests at least some of these are not desired.

>fraction: "." intpart
>exponent: ("e" | "E") "_"* ["+" | "-"] digit [decimalrest]

This allows underscores in the exponent (1e-5_0), contradicting the
other bullet point.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: merge

[Martin Panter]

On 2 February 2016 at 05:21, raymond.hettinger
 wrote:
> https://hg.python.org/cpython/rev/0731f097157b
> changeset:   100142:0731f097157b
> parent:  100140:c7f1acdd8be1
> user:Raymond Hettinger 
> date:Mon Feb 01 21:21:19 2016 -0800
> summary:
>   merge
>
> files:
>   Doc/library/collections.rst  |   4 ++--
>   Lib/test/test_deque.py   |  23 ---
>   Modules/_collectionsmodule.c |   7 ++-
>   3 files changed, 16 insertions(+), 18 deletions(-)

This wasn’t actually a merge (there is only one parent). Hopefully I
fixed it up with . But
it looks like the original NEWS entry didn’t get merged in your
earlier merge , so
there was nothing for me to merge the NEWS changes into in the default
branch.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot timing out - test suite failure - test_socket issue with UDP6?

[Martin Panter]

> After digging through test_socket.py for over an hour (the MRO for
> RecvmsgUDP6Test is enormous!!), I've boiled the issue down to this:
>
> import socket
> MSG = b'asdf qwer zxcv'
> serv = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
> serv.bind(("::1", 0))
> cli = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
> cli.bind(("::1", 0))
> cli.sendto(MSG, serv.getsockname())
> print(serv.recvmsg(len(MSG) - 3, 0, socket.MSG_PEEK))
> print(serv.recvmsg(len(MSG), 0, socket.MSG_PEEK))
> print(serv.recvmsg(len(MSG)))
>
> On my main system, this produces three lines of output: the first has
> truncated text, the second has full text, and the third also has full
> text. This proves that MSG_PEEK is working correctly. On the buildbot,
> though, the first one stalls out. Commenting that line out produces
> correct results - peek the full data, then read it, and all is well.
>
> Any idea why partial read on a datagram socket would sometimes stall?

I think it would stall if there is no data to receive. Maybe check the
return value of sendto(), to ensure it is sending the whole message.

Attached is a C program which should do the equivalent of your
boiled-down Python script, in case that helps:

$ gcc -Wall peek-udp6.c -o peek-udp6
$ ./peek-udp6
Bytes sent: 14
Received [asdf qwer z]
Received [asdf qwer zxcv]
Received [asdf qwer zxcv]

Other things that come to mind are to see if there is anything odd
about the buildbot’s Linux kernel and glibc versions. Maybe run the
Python script under “strace” to see if anything strange is going on.
/*
$ gcc -Wall peek-udp6.c -o peek-udp6
$ ./peek-udp6
Bytes sent: 14
Received [asdf qwer z]
Received [asdf qwer zxcv]
Received [asdf qwer zxcv]
*/

#include 
#include 
#include 
#include 
#include 

static const char MSG[] = "asdf qwer zxcv";
static const size_t MSG_LEN = sizeof MSG - 1;

int main() {
int serv = socket(AF_INET6, SOCK_DGRAM | SOCK_CLOEXEC, 0);
if(serv < 0) {
perror("serv socket()");
return 1;
}

struct sockaddr_in6 bind_addr;
memset(_addr, 0, sizeof bind_addr);
bind_addr.sin6_family = AF_INET6;
bind_addr.sin6_port = htons(0);
bind_addr.sin6_addr = (struct in6_addr)IN6ADDR_LOOPBACK_INIT;
if(bind(serv, (struct sockaddr *)_addr, sizeof bind_addr) < 0) {
perror("bind(serv)");
return 1;
}

int cli = socket(AF_INET6, SOCK_DGRAM | SOCK_CLOEXEC, 0);
if(cli < 0) {
perror("cli socket()");
return 1;
}

if(bind(cli, (struct sockaddr *)_addr, sizeof bind_addr) < 0) {
perror("bind(cli)");
return 1;
}

struct sockaddr_in6 addr;
socklen_t addr_len = sizeof addr;
if(getsockname(serv, (struct sockaddr *), _len) < 0) {
perror("getsockname()");
return 1;
}
ssize_t size;
size = sendto(cli, MSG, MSG_LEN, 0, (struct sockaddr *), addr_len);
if(size < 0) {
perror("sendto()");
return 1;
}
printf("Bytes sent: %zi\n", size);

char buffer[MSG_LEN];
struct msghdr message = {
.msg_name = ,
.msg_namelen = sizeof addr,
.msg_iov = (struct iovec [1]){
{.iov_base = buffer, .iov_len = MSG_LEN - 3},
},
.msg_iovlen = 1,
};
size = recvmsg(serv, , MSG_PEEK);
if(size < 0) {
perror("short recvmsg(MSG_PEEK)");
return 1;
}
printf("Received [%.*s]\n", (int)size, buffer);

message.msg_iov[0].iov_len = MSG_LEN;
size = recvmsg(serv, , MSG_PEEK);
if(size < 0) {
perror("full recvmsg(MSG_PEEK)");
return 1;
}
printf("Received [%.*s]\n", (int)size, buffer);

size = recvmsg(serv, , 0);
if(size < 0) {
perror("full recvmsg(0)");
return 1;
}
printf("Received [%.*s]\n", (int)size, buffer);

return 0;
}
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update PEP 7 to require curly braces in C

[Martin Panter]

On 19 January 2016 at 20:12, Brett Cannon  wrote:
> Here is a proposed update:
>
> diff -r 633f51d10a67 pep-0007.txt
> --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800
> +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800
> @@ -75,9 +75,9 @@
>}
>
>  * Code structure: one space between keywords like ``if``, ``for`` and
> -  the following left paren; no spaces inside the paren; braces may be
> -  omitted where C permits but when present, they should be formatted
> -  as shown::
> +  the following left paren; no spaces inside the paren; braces are
> +  strongly preferred but may be omitted where C permits, and they
> +  should be formatted as shown::
>
>if (mro != NULL) {
>...

This change seems to be accidentally smuggled in, in the guise of a
PEP 512 update :)

My view is I prefer always using curly brackets in my own code. It is
easier to add printf() debugging without making mistakes. I support
“strongly preferring” them in the style guide, which is as much as a
style guide can do anyway.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Modifying the self-signed.pythontest.net certificate

[Martin Panter]

In order to fix the SSL test suite
, I would like to modify the
certificate used by https://self-signed.pythontest.net. So far I have
a patch ready for the pythontestdotnet repository, but I want to know
if I can just push to that repository, or if other steps are required.

Judging by , Georg Brandl was
involved in setting the server up. So far I haven’t heard anything
from him on the SSL tests bug, so I am asking here in case somebody
else can help.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Modifying the self-signed.pythontest.net certificate

[Martin Panter]

On 14 January 2016 at 05:34, Benjamin Peterson <benja...@python.org> wrote:
> On Wed, Jan 13, 2016, at 13:15, Martin Panter wrote:
>> In order to fix the SSL test suite
>> <https://bugs.python.org/issue25940>, I would like to modify the
>> certificate used by https://self-signed.pythontest.net. So far I have
>> a patch ready for the pythontestdotnet repository, but I want to know
>> if I can just push to that repository, or if other steps are required.
>
> It should suffice to update that repo. I suppose the server might have
> to be kicked, too, as it's a cert change. Ping this thread if that
> doesn't work.

It seems to have worked, thanks. Not immediately, but after waiting
~20 minutes the certificate is now updated. I am working on an update
for all branches 3.2+ and 2.7; I expect test_httplib will be broken
until I fix it.

BTW <https://self-signed.pythontest.net/> is giving me a 403 Forbidden
response, but it was doing this before I pushed my change, so I think
it’s not my fault :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GitHub migration planning has started

[Martin Panter]

On 11 January 2016 at 03:52, Terry Reedy  wrote:
> On 1/10/2016 12:43 PM, Brett Cannon wrote:
>> If you want to help with the transition, then feel free to join the
>> core-workflow mailing list where all the discussions on the details of
>> the migration are occurring (including the PEP I'm starting to write to
>> outline the steps we will be taking):
>> https://mail.python.org/mailman/listinfo/core-workflow
>
>
> Is there a gmane mirror, or do you think this is too limited (and temporary)
> for that?

I guess this is it: 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bitwise operations for bytes and bytearray

[Martin Panter]

On 10 January 2016 at 00:17, Blake Griffith  wrote:
> A little update, I got ^, &, and | working for bytearrays. You can view the
> diff here:
> https://github.com/python/cpython/compare/master...cowlicks:bitwise-bytes?expand=1
>
> How does it look?

I left some comments against your commits on Git Hub. Unfortunately
they seem to be made in the base Python repository, rather than your
fork, which I did not expect.

> Joe, is this how I should allocate the arrays? Am I freeing them properly?
> Am I checking the input enough?
>
> After some feedback, I'll probably add bitshifting and negation. Then work
> on bytes objects.
>
> Does this warrant a pep?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] bitwise operations for bytes and bytearray

[Martin Panter]

On 7 January 2016 at 22:26, Blake Griffith  wrote:
> I'm interested in adding the functionality to do something like:
>
  b'a' ^ b'b'
> b'\x03'
>
>
> Instead of the good ol' TypeError.
>
> I think both bytes and bytearray should support all the bitwise operations.

There is a bug open about adding this kind of functionality:
.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Branches in which to fix the SSL tests

[Martin Panter]

Currently some SSL tests in the test suite are broken by a recent
certificate change at https://svn.python.org/; see
 for the bug report. The tests are
broken when the test suite is run with the “-unetwork” option enabled,
and most of the buildbots appear to be affected. (In 3.6 the tests
have temporarily been disabled as a workaround.) I have a simple patch
that subsitutes the old root certificate for the new which I would
like to commit, but I’m not sure which branches to apply it to, or
even which branches are open to normal maintainence and bug fixes.

According to Larry
,
3.4.4 was the last bug fix release for 3.4, so I assumed the 3.4
branch should now be in security-fixes-only mode. However this branch
still seems to get a lot of non-security action, for example the most
recent bunch of changes were some work on the provisional “pathlib”
module. So firstly I would like some clarification on the status of
3.4 and what its future is.

Secondly, I would normally say a fix for the test suite isn’t really
appropriate for the older security branches. But in the bug report,
Koobs specifically requested this be fixed in 3.4 and possibly earlier
branches as well. What do others think about this?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New poll about a macro for safe reference replacing

[Martin Panter]

On 16/12/2015, Serhiy Storchaka  wrote:
> Here are names gained the largest numbers of votes plus names proposed
> during polling.
>
> 1. Py_SETREF
+0. I can live with it, but SET sounds like a complement to CLEAR, or
that it ignores the old value.

> 2. Py_DECREF_REPLACE
+0.5

> 3. Py_REPLACE
+1. Fairly obvious what it does.

> 4. Py_SET_POINTER
-1
> 5. Py_SET_ATTR
-1 ** -1. What’s the attribute name?

> 6. Py_REPLACE_REF
+0.5

Ryan’s Py_RESET: -1, it sounds too much like CLEAR
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (3.4): asyncio.docs: Fix versionadded

[Martin Panter]

On 18 November 2015 at 17:45, yury.selivanov  wrote:
> https://hg.python.org/cpython/rev/b34c42e46e7b
> changeset:   99204:b34c42e46e7b
> branch:  3.4
> parent:  99201:89d66f912671
> user:Yury Selivanov 
> date:Wed Nov 18 12:44:31 2015 -0500
> summary:
>   asyncio.docs: Fix versionadded
>
> files:
>   Doc/library/asyncio-task.rst |  2 +-
>   1 files changed, 1 insertions(+), 1 deletions(-)
>
>
> diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst
> --- a/Doc/library/asyncio-task.rst
> +++ b/Doc/library/asyncio-task.rst
> @@ -699,4 +699,4 @@
>Unlike the functions above, :func:`run_coroutine_threadsafe` requires 
> the
>*loop* argument to be passed explicitely.
>
> -   .. versionadded:: 3.4.4
> +   .. versionadded:: 3.4.4, 3.5.1

This doesn’t get marked up properly; have a look at
.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should PEP 498 specify if rf'...' is valid?

[Martin Panter]

On 26 October 2015 at 19:43, MRAB  wrote:
> On 2015-10-26 18:45, Sven R. Kunze wrote:
>>
>> On 26.10.2015 16:22, Ethan Furman wrote:
>>>
>>> On 10/23/2015 08:20 AM, Nick Coghlan wrote:

 My own objection isn't to allowing "fR" or "fbR", it's to
 allowing the uppercase "F".

 I also don't understand why we can't say "if 'f' is part of a
 string prefix, it must be first".
>>>
>>>
>>> Sometimes order matters, and sometimes it does not.  If the order
>>> does not have an impact on the final code, it does not matter, and
>>> making us have to remember an order that does not matter is a
>>> waste.
>>
>>
>> Order that matters? You must be kidding. That would turn different
>> types of string extremely hard to understand because semantics
>> differ.
>>
>> That is, btw., one reason, why I favor a fixed order (alphabetically
>> or something). Easy to remember and no way to misinterpret it.
>>
> In Python 2, how often have you seen prefix "ur" rather than "ru"?
>
> I always used "ur".
>
> How often is alphabetical order used in the prefixes?
>
> If the order isn't alphabetical, then it's going to be some order
> that's harder to remember, so I agree with Ethan here.

In Python 2, ru". . ." is a SyntaxError, despite R coming before U in
the alphabet. And rb". . ." is also a SyntaxError, but in Python 3 it
was made legal.

I don’t see much point restricting the order of rf". . ." versus fr".
. .". Neither flag is particularly more important than the other, and
even if one were, should that one be at the front, or at the end
closer to the string?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] type(obj) vs. obj.__class__

[Martin Panter]

On 18 October 2015 at 05:55, Steven D'Aprano  wrote:
> On Sat, Oct 17, 2015 at 03:45:19PM -0600, Eric Snow wrote:
>> So, would it make sense to establish some concrete guidelines about
>> when to use type(obj) vs. obj.__class__?  If so, what would those be?
>> It may also be helpful to enumerate use cases for "type(obj) is not
>> obj.__class__".
>
> I for one would like to see a definitive explanation for when they are
> different, and when you should use one or the other. The only
> obvious example I've seen is the RingBuffer from the Python Cookbook:
>
> http://code.activestate.com/recipes/68429-ring-buffer/

It looks like this example just assigns to the existing __class__
attribute, to switch to a different class. I haven’t seen this ability
mentioned in the documentation, but I suspect it is meant to be
supported. However assigning to __class__ like that should
automatically update the type() return value, so type(ring_buffer) ==
ring_buffer.__class__ is still maintained.

Perhaps some of this confusion comes from Python 2. I don’t know the
details, but I know in Python 2, type() can do something different, so
you have to use __class__ directly if you want to be compatible with
Python 2 classes. But in Python 3 code I prefer using direct function
calls like type() to “special attributes” like __class__ where
practical.

The documentation says that __*__ names are reserved for Python and
its built-in library, rather than user code. So user code that creates
a class attribute or property called __class__ is asking for trouble
IMO, and we shouldn’t spend much effort accommodating such cases.

For __repr__() I would use type(), which seems to agree with what
object.__repr__() uses.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Committing a bug fix

[Martin Panter]

On 28/09/2015, Terry Reedy  wrote:
> You neglected to forward merge a 3.4-3.4 merge and a 3.5-3.5 merge.  I
> failed to notice the first and added a commit, and then backed it out,
> when I could not merge forward due to merge conflicts with your commits.
>   Perhaps you just need to null-merge the 3.4 commits into 3.5, but I do
> not really know how to make the repository usable again.

Hopefully I have fixed this for you. My technique is below. I first
tried with Mercurial 2.7, but updated to 3.5 which apparently grew
some support for “criss-cross” merges, and this helped a lot with the
second merge.

hg update 3.5
hg merge 3.4  # NEWS and Argument Clinic files conflict
hg revert -r . Misc/NEWS Modules/clinic/_datetimemodule.c.h
python Tools/clinic/clinic.py --make
# Verify both Alexander and Serhiy’s entries already in NEWS
hg diff -p  # No file changes to 3.5 branch
hg resolve -m
hg commit -m "Merge from 3.4 into 3.5; no file changes"
hg update default
hg merge 3.5  # No conflicts; 3 × “ambiguous merge - picked m action”
hg diff -p  # Duplicate NEWS entry
hg revert -r . Misc/NEWS
hg diff -p  # No changes
hg commit -m "Merge from 3.5; no file changes"

Summary of what I think was going on: Serhiy made some changes quite a
few hours ago to do with readline.set_completer_delims(). Meanwhile,
Alexander made some tzinfo.fromutc() commits based on revisions before
Serhiy’s changes. A couple comments on Alexander’s commits:

ff68705c56a8: Initial 3.4 patch has a trailing space in NEWS
c706c062f545: Initial merge into 3.5 removes this trailing space
cbcf82f92c25: Looks like a second version of the original patch
(ff68705c56a8), except without the extra space. I guess you got
confused, but it would have been better to not push this commit :)

Also @Terry, not sure on your workflow, but you might have been able
to avoid pushing your 3.4 commits. You might have been able to remove
it locally by using “hg strip” or deleting your local repository. Or
just leave it locally until the other branches got sorted out. Notice
that the 3.4 branch still has your backout commit at the top.

For what it’s worth, I have more faith in Git for merging stuff like
this smoothly. But today I learnt that the newer Mercurial version
does behave a bit better than it used to.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Committing a bug fix

[Martin Panter]

On 28 September 2015 at 22:31, Terry Reedy  wrote:
> . . . it may have merged in the wrong place, such as putting a things into the
> 3.5.1 section of 3.6 NEWS, which should not be there.

Since you mentioned this and I also noticed this, can I ask why the
3.5.1 section is there, if it isn’t meant to be? There are also other
sections, going back to 3.4.0. They have been there for a while. Is it
just that nobody removed it when it was inherited from 3.5 and 3.4?
When I have merged bug fixes into the default 3.6 branch I have
sometimes let the NEWS entries be automatically be merged into the
3.5.1 section, without updating the 3.6.0 section, after I noticed
other people doing this.

Perhaps we should get rid of the old pre-3.6 sections (perhaps after
adding any missing entries to the 3.6 section)? Otherwise I imagine
some release manager might come along and chop them all out without
adding anything to the 3.6 section.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com