Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Mark Lawrence via Python-Dev]

On 12/09/2016 23:25, Gregory P. Smith wrote:

On Mon, Sep 12, 2016 at 10:25 AM INADA Naoki wrote:


So fundamental question is: Is it to so bad thing that some people
write code depending on CPython and PyPy implementation?


Yes.  See below.

I think cross-interpreter libraries can use OrederedDict correctly
when they should use it. (They may run test on micropython, Jython
and IronPython).


The problem is that libraries which could otherwise be cross-VM
compatible are not because they depend upon an implementation detail. So
it becomes an additional porting burden on people trying to use the
library on another VM that could've been avoided if we required people
to be explicit about their needs.

BUT...

At this point I think coding up an example patch against beta1 offering
a choice of disordered iteration capability that does not increase
memory or iteration overhead in any significant way is needed.

The problem is... I don't know how to express this as an API. Which
sinks my whole though process and tables the idea.



"tables the idea"  has the US meaning of close it down, not the UK 
meaning of open it up? :)


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Tim Delaney]

On 13 September 2016 at 10:28, Brett Cannon  wrote:

>
>> I'd like to add one more documented constraint - that dict literals
>> maintain definition order (so long as the dict is not further modified).
>> This allows defining a dict literal and then passing it as **kwargs.
>>
>
> That would require all dictionaries keep their insertion order which we
> are explicitly not doing (at least yet). If you look at the PEPs that are
> asking for definition order they specify an "ordered mapping", not a dict.
> Making dict literals do this means dict literals become "order mapping
> literals" which isn't what they are; they are dict literals. I don't think
> we should extend this guarantee to literals any more than any other
> dictionary.
>

I'm not sure I agree with you, but I'm not going to argue too strongly
either (it can always be revisited later). I will note that a conforming
implementation could be that the result of evaluating a dict literal is a
frozen ordered dict which transparently changes to be a mutable dict as
required. There could well be performance and/or memory benefits from such
a dict implementation.

Personally I expect all Python 3.6 implementations will have
order-preserving dict as that's the easiest way to achieve the existing
guarantees. And that enough code will come to depend on an order-preserving
dict that eventually the decision will be made to retrospectively guarantee
the semantics.

Tim Delaney
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Brett Cannon]

On Mon, 12 Sep 2016 at 16:52 Tim Delaney 
wrote:

> On 10 September 2016 at 03:17, Guido van Rossum  wrote:
>
>> I've been asked about this. Here's my opinion on the letter of the law in
>> 3.6:
>>
>> - keyword args are ordered
>> - the namespace passed to a metaclass is ordered by definition order
>> - ditto for the class __dict__
>>
>> A compliant implementation may ensure the above three requirements
>> either by making all dicts ordered, or by providing a custom dict
>> subclass (e.g. OrderedDict) in those three cases.
>>
>
> I'd like to add one more documented constraint - that dict literals
> maintain definition order (so long as the dict is not further modified).
> This allows defining a dict literal and then passing it as **kwargs.
>

That would require all dictionaries keep their insertion order which we are
explicitly not doing (at least yet). If you look at the PEPs that are
asking for definition order they specify an "ordered mapping", not a dict.
Making dict literals do this means dict literals become "order mapping
literals" which isn't what they are; they are dict literals. I don't think
we should extend this guarantee to literals any more than any other
dictionary.


>
> Hmm - again, there's no mention of dict literals in the PEPs. I'm assuming
> that dict literals will preserve their definition order with the new
> implementation, but is that a valid assumption? Guess I can test it now
> 3.6.0b1 is out.
>

They will as an implementation detail, not because the language spec
requires it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 what's new

[Ned Deily]

On Sep 12, 2016, at 19:57, Benjamin Peterson  wrote:
> Thank you.

Ditto!  Many thanks, Yury!

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 3.6.0 Beta Phase Development

[Ned Deily]

Wow!  What a busy and productive couple of weeks it has been leading up to 
3.6.0b1 and feature code freeze!  Congratulations and thanks to all of you 
who've contributed to the amazing number of PEPs, features, bug fixes, and doc 
changes that have gone into 3.6.0b1!  Now that feature development for 3.6 is 
over, the challenge is to put the finishing touches on the features and 
documentation, squash bugs, and test test test.  The next preview release will 
be 3.6.0b2 scheduled for 2016-10-03.

In the cpython repo, there is now a 3.6 branch.  Starting now, all changes for 
3.6.0 should get pushed to the 3.6 branch and then merged to default for 3.7.  
New features nay continue to be pushed to the default branch for release in 
3.7; no new features are now permitted in 3.6 (unless you have contacted me and 
we have agreed on an extension).  Bug fixes appropriate for 3.5.x should get 
pushed to the 3.5 branch and then merged to 3.6 and then to default.  I've 
updated the Developer's Guide to reflect the now current workflow.  Let me know 
if you find any bugs in it.  Likewise, please contact me if you have any 
questions about the workflow or about whether a change is appropriate for 3.6 
beta.

To recap:

2016-09-12 3.6 branch open for 3.6.0; 3.7.0 feature development begins

2016-09-12 to 2016-12-04: 3.6.0 beta phase (no new features)
- push code for 3.6.0 (bug/regression/doc fixes) to the new 3.6 branch
- push code for new features to the default branch for release in 3.7

2016-10-03: 3.6.0 beta 2

2016-12-04 3.6.0 release candidate 1 (3.6.0 code freeze)

2016-12-16 3.6.0 release (3.6.0rc1 plus, if necessary, any dire emergency fixes)

2018-06 3.7.0 release (3.6.0 release + 18 months, details TBD)


Thank you all again for your great efforts so far on 3.6!

--Ned

http://cpython-devguide.readthedocs.io/en/latest/

https://www.python.org/dev/peps/pep-0494/

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 what's new

[Benjamin Peterson]

Thank you.


On Mon, Sep 12, 2016, at 16:21, Yury Selivanov wrote:
> Hi,
>
> Elvis and I authored What's New in Python 3.5.  We'd like to volunteer
> to do the same for 3.6.  If there are no objections, we can make the
> first editing pass in a couple of weeks.
>
> Yury
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 what's new

[Guido van Rossum]

No objection!

On Mon, Sep 12, 2016 at 4:21 PM, Yury Selivanov  wrote:
> Hi,
>
> Elvis and I authored What's New in Python 3.5.  We'd like to volunteer to do
> the same for 3.6.  If there are no objections, we can make the first editing
> pass in a couple of weeks.
>
> Yury
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Tim Delaney]

On 10 September 2016 at 03:17, Guido van Rossum  wrote:

> I've been asked about this. Here's my opinion on the letter of the law in
> 3.6:
>
> - keyword args are ordered
> - the namespace passed to a metaclass is ordered by definition order
> - ditto for the class __dict__
>
> A compliant implementation may ensure the above three requirements
> either by making all dicts ordered, or by providing a custom dict
> subclass (e.g. OrderedDict) in those three cases.
>

I'd like to add one more documented constraint - that dict literals
maintain definition order (so long as the dict is not further modified).
This allows defining a dict literal and then passing it as **kwargs.

Hmm - again, there's no mention of dict literals in the PEPs. I'm assuming
that dict literals will preserve their definition order with the new
implementation, but is that a valid assumption? Guess I can test it now
3.6.0b1 is out.

Tim Delaney
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASE] Python 3.6.0b1 is now available

[Ned Deily]

On behalf of the Python development community and the Python 3.6 release
team, I'm happy to announce the availability of Python 3.6.0b1. 3.6.0b1
is the first of four planned beta releases of Python 3.6, the next major
release of Python, and marks the end of the feature development phase
for 3.6.

Among the new major new features in Python 3.6 are:

* PEP 468 - Preserving the order of **kwargs in a function
* PEP 487 - Simpler customization of class creation
* PEP 495 - Local Time Disambiguation
* PEP 498 - Literal String Formatting
* PEP 506 - Adding A Secrets Module To The Standard Library
* PEP 509 - Add a private version to dict
* PEP 515 - Underscores in Numeric Literals
* PEP 519 - Adding a file system path protocol
* PEP 520 - Preserving Class Attribute Definition Order
* PEP 523 - Adding a frame evaluation API to CPython
* PEP 524 - Make os.urandom() blocking on Linux (during system startup)
* PEP 525 - Asynchronous Generators (provisional)
* PEP 526 - Syntax for Variable Annotations (provisional)
* PEP 528 - Change Windows console encoding to UTF-8 (provisional)
* PEP 529 - Change Windows filesystem encoding to UTF-8 (provisional)
* PEP 530 - Asynchronous Comprehensions

Please see "What’s New In Python 3.6" for more information:

https://docs.python.org/3.6/whatsnew/3.6.html

You can find Python 3.6.0b1 here:

https://www.python.org/downloads/release/python-360b1/

Beta releases are intended to give the wider community the opportunity
to test new features and bug fixes and to prepare their projects to
support the new feature release. We strongly encourage maintainers of
third-party Python projects to test with 3.6 during the beta phase and
report issues found to bugs.python.org as soon as possible. While the
release is feature complete entering the beta phase, it is possible that
features may be modified or, in rare cases, deleted up until the start
of the release candidate phase (2016-12-05). Our goal is have no changes
after rc1. To achieve that, it will be extremely important to get as
much exposure for 3.6 as possible during the beta phase. Please keep in
mind that this is a preview release and its use is not recommended for
production environments

The next planned release of Python 3.6 will be 3.6.0b2, currently
scheduled for 2016-10-03. More information about the release schedule
can be found here:

https://www.python.org/dev/peps/pep-0494/

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python 3.6 what's new

[Yury Selivanov]

Hi,

Elvis and I authored What's New in Python 3.5.  We'd like to volunteer 
to do the same for 3.6.  If there are no objections, we can make the 
first editing pass in a couple of weeks.


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Eric Snow]

On Mon, Sep 12, 2016 at 4:46 PM, Ethan Furman  wrote:
> Does anyone have a short explanation of the interaction between hash
> randomization and this new always ordered dict?  Why doesn't one make the
> other useless?

Before 3.6, dict iteration was based on the hash table, which varies
based on the hash seed.  The compact dict implementation separates the
hash table from the keys table (which preserves insertion order), and
iterates over the keys table.  So the hash table uses the same hash
randomization as before, but it no longer impacts iteration.

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Gregory P. Smith]

On Mon, Sep 12, 2016 at 3:57 PM Brett Cannon  wrote:

> On Mon, 12 Sep 2016 at 15:46 Ethan Furman  wrote:
>
> On 09/12/2016 09:27 AM, Gregory P. Smith wrote:
>
> > For the regular dict (non kwargs or namespace __dict__) use case I would
> actually like to /see disorder preserved during iteration/.
> >
> > If we don't, we will eventually to find ourselves in a similar state we
> were in pre hash-randomization:
>
> Does anyone have a short explanation of the interaction between hash
> randomization and this new always ordered dict?  Why doesn't one make the
> other useless?
>
>
> There is still a hash table that stores a pointer into an array that
> stores the keys/values that are kept in an ordered array. So that
> first-level hash table still uses hash randomization.
>

More specifically: If the goal of hash randomization is to reduce DDOS hash
table stuffing attacks, that is still true. The hashing is randomized.

Dict ordering may actually _help_ DDOS protection. It no longer leaks
information potentially revealing details about the hash seed via the
iteration order.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Brett Cannon]

On Mon, 12 Sep 2016 at 15:46 Ethan Furman  wrote:

> On 09/12/2016 09:27 AM, Gregory P. Smith wrote:
>
> > For the regular dict (non kwargs or namespace __dict__) use case I would
> actually like to /see disorder preserved during iteration/.
> >
> > If we don't, we will eventually to find ourselves in a similar state we
> were in pre hash-randomization:
>
> Does anyone have a short explanation of the interaction between hash
> randomization and this new always ordered dict?  Why doesn't one make the
> other useless?
>

There is still a hash table that stores a pointer into an array that stores
the keys/values that are kept in an ordered array. So that first-level hash
table still uses hash randomization.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Ethan Furman]

On 09/12/2016 09:27 AM, Gregory P. Smith wrote:


For the regular dict (non kwargs or namespace __dict__) use case I would 
actually like to /see disorder preserved during iteration/.

If we don't, we will eventually to find ourselves in a similar state we were in 
pre hash-randomization:


Does anyone have a short explanation of the interaction between hash 
randomization and this new always ordered dict?  Why doesn't one make the other 
useless?

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Gregory P. Smith]

On Mon, Sep 12, 2016 at 9:51 AM Chris Angelico  wrote:

> On Tue, Sep 13, 2016 at 2:27 AM, Gregory P. Smith  wrote:
> > Disorder for this purpose need not be a random shuffle (overkill). It
> just
> > needs to be regularly inconsistent. A simple thing to do on top of 3.6's
> new
> > dict implementation would be to pick a random starting point within the
> > order array rather than offset 0 to start iteration from. That small
> change
> > would be sufficient to guarantee that code depending on order must ask
> for
> > order. It could even allow us to get people ready for iteration within
> the
> > same process to become unstable.
>
> Don't forget that .items(), .keys(), and .values() are all
> synchronized, so you'd probably have to pick an offset at dict
> creation and run with it forever after.
>

Indeed. We could "cheat" and match existing 2.7 and 3.5 behavior by using
the hash randomization seed to determine a "consistent within the life of a
process" dict iteration order randomization without storing anything per
dict. That has the added bonus/drawback (POV) of allowing people to fix a
specific behavior via the existing environment variable as they already
expect.

But given my previous message deciding trying to implement disordered
iteration by default in some cases is infeasible, it's moot. :)

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Gregory P. Smith]

On Mon, Sep 12, 2016 at 10:25 AM INADA Naoki  wrote:

>
> So fundamental question is: Is it to so bad thing that some people
> write code depending on CPython and PyPy implementation?
>

Yes.  See below.

I think cross-interpreter libraries can use OrederedDict correctly
> when they should use it. (They may run test on micropython, Jython and
> IronPython).
>

The problem is that libraries which could otherwise be cross-VM compatible
are not because they depend upon an implementation detail. So it becomes an
additional porting burden on people trying to use the library on another VM
that could've been avoided if we required people to be explicit about their
needs.

BUT...

At this point I think coding up an example patch against beta1 offering a
choice of disordered iteration capability that does not increase memory or
iteration overhead in any significant way is needed.

The problem is... I don't know how to express this as an API. Which sinks
my whole though process and tables the idea.

A parameter to .items(), .keys() and .values() is undesirable as it isn't
backwards compatible [meaning it'll never be used] and .keys() needs to
match __iter__ which can't have one anyways. A parameter on dict
construction is similarly infeasible.

Requiring the use of an orderdict like type in order to get the behavior is
undesirable. Effectively I'm asking for some boolean state in each dict as
to if it should iterate in order or not and a way to expose that to pure
Python code in a way that namespace dicts iterate in order by default and
others do not unless explicitly configured to do so.

oh well.  end thought process on my end.  it was good while it lasted.

Thanks for the compact and ordered dicts! People will love them. :)

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[INADA Naoki]

> From what I understood, Python 3.6 dict got two *different* changes:
>
> * modify the dict structure to use two tables instead of only one: an
> "index" table (the hash table) and a second key/value table
> * tune the dict implementation to only append to the key/value table
>
> The second change depends on the first change.
>
> When a key is deleted, the entry is marked as DUMMY. When we add a new
> item, DUMMY entries are skipped and we only append at the end of the
> key/value table. Sometimes, the key/value table is compacted to free
> memory: all DUMMY entries are removed.

Minor correction:

Put dummy key in *hash* table.  The purpose of the dummy key is same to
previous dict implementation.
The entry where deleted is filled with NULL.


> It would be possible to add a flag to allow to reuse DUMMY entries,
> which means loosing the order. The order would only be lost when we
> add the first item after we removed at least one entry (when the first
> DUMMY entry is reused).

Reusing NULL entry is possible, like original compact dict idea by Raymond.

But we should rebuild hash table before it is filled by dummy keys.
Otherwise, hash lookup may be very slow, or never stop.
Sparseness of hash table is very important.

Compaction in current implementation is not only for packing key-value entries,
but also removing dummy keys from hash table.


>
> The OrderedDict would set the flag to preserve the order.
>
> So technically, it is possible. The question is more what should be
> the "default" dict :-) Ordered or not? :-)

Even if dict don't preserve insertion order when deletion, people may depend
"preserving insertion order unless deletion".

So fundamental question is: Is it to so bad thing that some people
write code depending
on CPython and PyPy implementation?

I think cross-interpreter libraries can use OrederedDict correctly
when they should use it.
(They may run test on micropython, Jython and IronPython).

And I think there are many use cases that "keeping insertion order is
not required, but it's
very nice if it is nearly zero cost.".

For example, when logging with JSON lines,

log.write(json.dumps( { "msg": "hello", "foo": foo, "bar" bar } ))

Stable key order may be not required, but it makes the log more readable.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Tim Peters]

[Guido]
> Wouldn't attempting to reuse DUMMY entries be expensive? You'd have to
> search forward in the array. Just keeping a count of DUMMY entries and
> compacting when there are too many seems better somehow.

I haven't looked at the code, but presumably one of the members of a
DUMMY key/value struct could be (ab)used to hold the index of "the
next" DUMMY (i.e., treating DUMMYs as a stack implemented by a
singly-linked list).  In which case no search is needed, but the dict
would need a word to hold the index of the DUMMY stack top (or, e.g.,
-1 when no DUMMY exists) - or dedicate "the first" key/value slot to
holding the stack top - or ...

It's just code, so it can do anything ;-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[INADA Naoki]

On Tue, Sep 13, 2016 at 1:35 AM, Guido van Rossum  wrote:
> Couldn't we use the order in the actual hash table (which IIUC now
> contains just indexes into the ordered vector of key/value/hash
> structs)? That would probably simulate the pre-3.6 order quite
> effectively.

Maybe, it can.
But current implementation may be faster on iteration, thanks to
hardware prefetch of CPU.
When sizeof(entry) is 24 (amd64), only 2.66... entries can be on cache line.


> But we'd have to add a new API to reveal the order (in effect just
> what Nick wanted). How much of the OrderedDict can be implemented just
> by adding new methods (IOW without changing the data structure)?

Current data structure uses fixed capacity, mostly append only array
for entries.

To implement `OrderedDict.move_to_end(last=False)`, OrderedDict should be
implement more hack. (e.g. use the array as ring.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Guido van Rossum]

Wouldn't attempting to reuse DUMMY entries be expensive? You'd have to
search forward in the array. Just keeping a count of DUMMY entries and
compacting when there are too many seems better somehow.

On Mon, Sep 12, 2016 at 10:00 AM, Victor Stinner
 wrote:
> 2016-09-12 18:35 GMT+02:00 Guido van Rossum :
>> Couldn't we use the order in the actual hash table (which IIUC now
>> contains just indexes into the ordered vector of key/value/hash
>> structs)? That would probably simulate the pre-3.6 order quite
>> effectively.
>
> From what I understood, Python 3.6 dict got two *different* changes:
>
> * modify the dict structure to use two tables instead of only one: an
> "index" table (the hash table) and a second key/value table
> * tune the dict implementation to only append to the key/value table
>
> The second change depends on the first change.
>
> When a key is deleted, the entry is marked as DUMMY. When we add a new
> item, DUMMY entries are skipped and we only append at the end of the
> key/value table. Sometimes, the key/value table is compacted to free
> memory: all DUMMY entries are removed.
>
> It would be possible to add a flag to allow to reuse DUMMY entries,
> which means loosing the order. The order would only be lost when we
> add the first item after we removed at least one entry (when the first
> DUMMY entry is reused).
>
> The OrderedDict would set the flag to preserve the order.
>
> So technically, it is possible. The question is more what should be
> the "default" dict :-) Ordered or not? :-)
>
> Victor



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Victor Stinner]

2016-09-12 18:35 GMT+02:00 Guido van Rossum :
> Couldn't we use the order in the actual hash table (which IIUC now
> contains just indexes into the ordered vector of key/value/hash
> structs)? That would probably simulate the pre-3.6 order quite
> effectively.

>From what I understood, Python 3.6 dict got two *different* changes:

* modify the dict structure to use two tables instead of only one: an
"index" table (the hash table) and a second key/value table
* tune the dict implementation to only append to the key/value table

The second change depends on the first change.

When a key is deleted, the entry is marked as DUMMY. When we add a new
item, DUMMY entries are skipped and we only append at the end of the
key/value table. Sometimes, the key/value table is compacted to free
memory: all DUMMY entries are removed.

It would be possible to add a flag to allow to reuse DUMMY entries,
which means loosing the order. The order would only be lost when we
add the first item after we removed at least one entry (when the first
DUMMY entry is reused).

The OrderedDict would set the flag to preserve the order.

So technically, it is possible. The question is more what should be
the "default" dict :-) Ordered or not? :-)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: remove all private C functions from the Python C API?

[Yury Selivanov]

Some of the functions we have are really intended to be used *only* by 
the interpreter itself.  For those it would be cool to have them in 
private headers (AFAIK we already do this, see dict-common.h for 
instance).  Other than that, I think that using the underscore 
convention is fine.



Yury


On 2016-09-11 4:37 AM, Victor Stinner wrote:

Hi,

Currently, Python has 3 C API:

* python core API
* regular API: subset of the core API
* stable API (ABI?), the Py_LIMITED_API thing: subset of the regular API

For practical purpose, all functions are declared in Include/*.h.
Basically, Python exposes "everything". There are private functions
which are exported using PyAPI_FUNC(), whereas they should only be
used inside Python "core". Technically, I'm not sure that we can get
ride of PyAPI_FUNC() because the stdlib also has extensions which use
a few private functions.

For Python 3.7, I propose that we move all these private functions in
separated header files, maybe Include/private/ or Include/core/, and
not export them as part of the "regular API".

The risk is that too many C extensions rely on all these tiny
"private" functions. Maybe for performance. I don't know.

What do you think?

See also the issue #26900, "Exclude the private API from the stable API":
http://bugs.python.org/issue26900

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Chris Angelico]

On Tue, Sep 13, 2016 at 2:27 AM, Gregory P. Smith  wrote:
> Disorder for this purpose need not be a random shuffle (overkill). It just
> needs to be regularly inconsistent. A simple thing to do on top of 3.6's new
> dict implementation would be to pick a random starting point within the
> order array rather than offset 0 to start iteration from. That small change
> would be sufficient to guarantee that code depending on order must ask for
> order. It could even allow us to get people ready for iteration within the
> same process to become unstable.

Don't forget that .items(), .keys(), and .values() are all
synchronized, so you'd probably have to pick an offset at dict
creation and run with it forever after.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Guido van Rossum]

Couldn't we use the order in the actual hash table (which IIUC now
contains just indexes into the ordered vector of key/value/hash
structs)? That would probably simulate the pre-3.6 order quite
effectively.

But we'd have to add a new API to reveal the order (in effect just
what Nick wanted). How much of the OrderedDict can be implemented just
by adding new methods (IOW without changing the data structure)?

On Mon, Sep 12, 2016 at 9:27 AM, Gregory P. Smith  wrote:
> For the regular dict (non kwargs or namespace __dict__) use case I would
> actually like to see disorder preserved during iteration.
>
> If we don't, we will eventually to find ourselves in a similar state we were
> in pre hash-randomization:
>  (1) Over time, code will come to depend on the order for no good reason.
> Especially true of tests. This greatly increases the engineering barrier
> when trying to move a codebase between Python versions or Python VMs.
>
> The underlying implementation is free to preserve order (as it now does,
> great work!) but I think the behavior of iteration when an ordered type was
> not explicitly requested or ordered iteration was not explicitly requested
> should be perturbed in order to maintain long term code health.
>
> Disorder for this purpose need not be a random shuffle (overkill). It just
> needs to be regularly inconsistent. A simple thing to do on top of 3.6's new
> dict implementation would be to pick a random starting point within the
> order array rather than offset 0 to start iteration from. That small change
> would be sufficient to guarantee that code depending on order must ask for
> order. It could even allow us to get people ready for iteration within the
> same process to become unstable.
>
> Maybe I worry too much. Having slogged through fixing problems to enable
> hash randomization on a code base of tens of millions of lines in 2012...
> there is a lot of value in enforcing disorder where none is intended to be
> guaranteed. Explicit is better than implicit.
>
> -gps
>
> On Mon, Sep 12, 2016 at 5:37 AM Victor Stinner 
> wrote:
>>
>> 2016-09-12 13:50 GMT+02:00 Antoine Pitrou :
>> > Besides, I don't think it has been proven that the compact-and-ordered
>> > dict implementation is actually *faster* than the legacy one.
>>
>> Python 3.6 dict is slower than Python 3.5 dict, at least for a simple
>> lookup:
>> http://bugs.python.org/issue27350#msg275581
>>
>> But its memory usage is 25% smaller.
>>
>> I'm curious about the performance of the "compaction" needed after
>> adding too many dummy entries (and to preserve insertion order), but I
>> don't know how to benchmark this :-) Maybe add/remove many new keys? I
>> expect bad performance on the compaction, but maybe not as bad as the
>> "hash DoS".
>>
>> For regular Python code, I don't expect compaction to be a common
>> operation, since it's rare to remove attributes. It's more common to
>> modify attributes value, than to remove them and later add new
>> attributes.
>>
>> Victor
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Gregory P. Smith]

For the regular dict (non kwargs or namespace __dict__) use case I would
actually like to *see disorder preserved during iteration*.

If we don't, we will eventually to find ourselves in a similar state we
were in pre hash-randomization:
 (1) Over time, code will come to depend on the order for no good reason.
Especially true of tests. This greatly increases the engineering barrier
when trying to move a codebase between Python versions or Python VMs.

The underlying implementation is free to preserve order (as it now does,
great work!) but I think the behavior of iteration when an ordered type was
not explicitly requested or ordered iteration was not explicitly requested
should be perturbed in order to maintain long term code health.

Disorder for this purpose need not be a random shuffle (overkill). It just
needs to be regularly inconsistent. A simple thing to do on top of 3.6's
new dict implementation would be to pick a random starting point within the
order array rather than offset 0 to start iteration from. That small change
would be sufficient to guarantee that code depending on order must ask for
order. It could even allow us to get people ready for iteration within the
same process to become unstable.

Maybe I worry too much. Having slogged through fixing problems to enable
hash randomization on a code base of tens of millions of lines in 2012...
there is a lot of value in enforcing disorder where none is intended to be
guaranteed. Explicit is better than implicit.

-gps

On Mon, Sep 12, 2016 at 5:37 AM Victor Stinner 
wrote:

> 2016-09-12 13:50 GMT+02:00 Antoine Pitrou :
> > Besides, I don't think it has been proven that the compact-and-ordered
> > dict implementation is actually *faster* than the legacy one.
>
> Python 3.6 dict is slower than Python 3.5 dict, at least for a simple
> lookup:
> http://bugs.python.org/issue27350#msg275581
>
> But its memory usage is 25% smaller.
>
> I'm curious about the performance of the "compaction" needed after
> adding too many dummy entries (and to preserve insertion order), but I
> don't know how to benchmark this :-) Maybe add/remove many new keys? I
> expect bad performance on the compaction, but maybe not as bad as the
> "hash DoS".
>
> For regular Python code, I don't expect compaction to be a common
> operation, since it's rare to remove attributes. It's more common to
> modify attributes value, than to remove them and later add new
> attributes.
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Victor Stinner]

2016-09-12 13:50 GMT+02:00 Antoine Pitrou :
> Besides, I don't think it has been proven that the compact-and-ordered
> dict implementation is actually *faster* than the legacy one.

Python 3.6 dict is slower than Python 3.5 dict, at least for a simple lookup:
http://bugs.python.org/issue27350#msg275581

But its memory usage is 25% smaller.

I'm curious about the performance of the "compaction" needed after
adding too many dummy entries (and to preserve insertion order), but I
don't know how to benchmark this :-) Maybe add/remove many new keys? I
expect bad performance on the compaction, but maybe not as bad as the
"hash DoS".

For regular Python code, I don't expect compaction to be a common
operation, since it's rare to remove attributes. It's more common to
modify attributes value, than to remove them and later add new
attributes.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's make the SSL module sane

[Antoine Pitrou]

On Sat, 10 Sep 2016 20:23:13 +0200
Christian Heimes  wrote:
> 
> It's a bit too clever and tricky for my taste. I prefer 'explicit is
> better than implicit' for trust anchors. My main concern are secure
> default settings. A SSLContext should be secure w/o further settings in
> order to prevent developers to shoot themselves in the knee.
> 
> Missing root certs are not a direct security issue with CERT_REQUIRED.
> The connection will simply fail. I'd rather improve the error message
> than to auto-load certs.

Agreed with all this.  You don't want to have "magic" behaviour in a
security-oriented module.  Let people configure their contexts
explicitly.

As a reminder, people who don't want to configure TLS themselves should
use an intermediate layer instead, such as ssl.create_default_context()
or an application protocol implementation (httplib, etc.).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: remove all private C functions from the Python C API?

[Antoine Pitrou]

On Sun, 11 Sep 2016 04:37:58 -0400
Victor Stinner  wrote:
> 
> For Python 3.7, I propose that we move all these private functions in
> separated header files, maybe Include/private/ or Include/core/, and
> not export them as part of the "regular API".

-1 from me.  There are reasons to rely on private stuff when necessary.

As long as private APIs are underscore-prefixed, people know what they
are risking by using them.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Let's make the SSL module sane

[Antoine Pitrou]

On Sat, 10 Sep 2016 16:22:57 +0200
Christian Heimes  wrote:
> 
> For 3.6 I like to make the SSL more sane and more secure by default.
> Yes, I'm a bit late but all my proposals are implemented, documented,
> partly tested and  existing tests are passing.

I don't have time nor motivation to review most of them, but I trust
you that the implementations are sane :-)

> First I like to deprecated some old APIs and favor of SSLCotext.

This has always been the plan (to me), so a big +1.

> The patch
> also deprecates certfile, keyfile an similar arguments in network
> protocol libraries.

+1.

> I also considered to make cert validation enabled by default for all
> protocol in 3.6, Victor has rising some concerns.

I assume you mean "in client mode". I think that sounds fine nowadays.
If people haven't configured a set of trusted CAs properly, this should
error out immediately, so they would notice it quickly IMHO.

(in other words, +0.5)

> How about we change
> the behavior in 3.7 and just add a warning to 3.6?

As you (or others) prefer :-)

> Next up SSLContext default configuration. A bare SSLContext comes with
> insecure default settings. I'd like to make SSLContext(PROTOCOL_SSLv23)
> secure bu default. Changelog: The context is created with more secure
> default values. The options OP_NO_COMPRESSION,
> OP_CIPHER_SERVER_PREFERENCE, OP_SINGLE_DH_USE, OP_SINGLE_ECDH_USE,
> OP_NO_SSLv2 (except for PROTOCOL_SSLv2), and OP_NO_SSLv3 (except for
> PROTOCOL_SSLv3) are set by default.
> The initial cipher suite list
> contains only HIGH ciphers, no NULL ciphers and MD5 ciphers (except for
> PROTOCOL_SSLv2).

+1 to all this from me.  The ship has sailed on most of this stuff
already.

> Finally (and this is the biggest) I like to change how the protocols
> work. OpenSSL 1.1.0 has deprecated all version specific protocols. Soon
> OpenSSL will only support auto-negotiation (formerly known as
> PROTOCOL_SSLv23). My patch #26470 added PROTOCOL_TLS as alias for
> PROTOCOL_SSLv23. If the last idea is accepted I will remove PROTOCOL_TLS
> again. It hasn't been released yet. Instead I'm going to add
> PROTOCOL_TLS_CLIENT and PROTOCOL_TLS_SERVER (see
> https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_new.html
> TLS_server_method(), TLS_client_method()). PROTOCOL_TLS_CLIENT is like
> PROTOCOL_SSLv23 but only supports client-side sockets and
> PROTOCOL_TLS_SERVER just server-side sockets. In my experience we can't
> have a SSLContext with sensible and secure settings for client and
> server at the same time. Hostname checking and cert validation is only
> sensible for client-side sockets.

This sounds reasonable.  No strong opinion from me but +0.5 as well.

> Starting in 3.8 (or 3.7?) there will be only PROTOCOL_TLS_CLIENT and
> PROTOCOL_TLS_SERVER.

You *may* provide the old constants for compatibility, though (meaning
"PROTOCOL_TLS", roughly).

> How will my proposals change TLS/SSL code?
> 
> Application must create a SSLContext object. Applications are
> recommended to keep the context around to benefit from session reusage
> and reduce overload of cert parsing.

(well, most applications are advised to use an intermediate layer such
as httplib ;-))

> I hope this mail makes sense.

It does to me!

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Antoine Pitrou]

On Fri, 9 Sep 2016 14:01:08 -0500
David Mertz  wrote:
> It seems unlikely, but not inconceivable, that someday in the future
> someone will implement a dictionary that is faster than current versions
> but at the cost of losing inherent ordering.

I agree with this.  Since ordering is a constraint, in abstracto it is
quite understandable that relaxing a constraint may enable more
efficient algorithms or implementations.

Besides, I don't think it has been proven that the compact-and-ordered
dict implementation is actually *faster* than the legacy one.  It is
more compact, which can matter in some contexts (memory-heavy workloads
with lots of objects, perhaps), but not necessarily others.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #27999: Make "global after use" a SyntaxError, and ditto for nonlocal.

[Christian Heimes]

On 2016-09-12 12:29, Ivan Levkivskyi wrote:
> On 12 September 2016 at 12:24, Christian Heimes  > wrote:
> 
> The code looks suspicious. Can you please
> provide a patch that makes it more obvious, e.g. either by using if /
> else if / else or a comment?
> 
> 
> Sure, I will open an issue with a patch and will add you to nosy (cannot
> do this *right* now, sorry).

Don't worry, it's not relevant for the beta release. My request is
purely cosmetic to make the code a bit easier to understand. :)

Christian

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #27999: Make "global after use" a SyntaxError, and ditto for nonlocal.

[Ivan Levkivskyi]

On 12 September 2016 at 12:24, Christian Heimes 
wrote:

> The code looks suspicious. Can you please
> provide a patch that makes it more obvious, e.g. either by using if /
> else if / else or a comment?


Sure, I will open an issue with a patch and will add you to nosy (cannot do
this *right* now, sorry).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Use HTTP in testPythonOrg

[Victor Stinner]

I just noticed a failure on a recent Windows build:
http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/11620/steps/test/logs/stdio

"urllib.error.URLError: "

So I guess that the change is to restrict the unit test on parsing the
robot failed and not test the SSL module.

Am I right?

Victor


2016-09-11 14:58 GMT+02:00 Eric V. Smith :
> Hi, Berker.
>
> Could you add a comment to the test on why this should use http? I can see
> this bouncing back and forth between http and https, as people clean an up
> all http usages to be https.
>
> Thanks.
> Eric.
>
> On 9/11/2016 8:46 AM, berker.peksag wrote:
>>
>> https://hg.python.org/cpython/rev/bc085b7e8fd8
>> changeset:   103634:bc085b7e8fd8
>> user:Berker Peksag 
>> date:Sun Sep 11 15:46:47 2016 +0300
>> summary:
>>   Use HTTP in testPythonOrg
>>
>> files:
>>   Lib/test/test_robotparser.py |  2 +-
>>   1 files changed, 1 insertions(+), 1 deletions(-)
>>
>>
>> diff --git a/Lib/test/test_robotparser.py b/Lib/test/test_robotparser.py
>> --- a/Lib/test/test_robotparser.py
>> +++ b/Lib/test/test_robotparser.py
>> @@ -276,7 +276,7 @@
>>  support.requires('network')
>>  with support.transient_internet('www.python.org'):
>>  parser = urllib.robotparser.RobotFileParser(
>> -"https://www.python.org/robots.txt";)
>> +"http://www.python.org/robots.txt";)
>>  parser.read()
>>  self.assertTrue(
>>  parser.can_fetch("*",
>> "http://www.python.org/robots.txt";))
>>
>>
>>
>> ___
>> Python-checkins mailing list
>> python-check...@python.org
>> https://mail.python.org/mailman/listinfo/python-checkins
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #27999: Make "global after use" a SyntaxError, and ditto for nonlocal.

[Christian Heimes]

On 2016-09-12 11:46, Ivan Levkivskyi wrote:
> Christian,
> 
> When I wrote this, my intention was like: cur & DEF_LOCAL is a "more
> serious" error, so that if both errors are made in the same statement:
> def f():
> x: int = 5
> global x
> 
> "SyntaxError: global after assignment" will be reported. The same logic
> applies to nonlocal.

Hi Ivan,

thanks for your explanation. The code looks suspicious. Can you please
provide a patch that makes it more obvious, e.g. either by using if /
else if / else or a comment?

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #27999: Make "global after use" a SyntaxError, and ditto for nonlocal.

[Ivan Levkivskyi]

Christian,

When I wrote this, my intention was like: cur & DEF_LOCAL is a "more
serious" error, so that if both errors are made in the same statement:
def f():
x: int = 5
global x

"SyntaxError: global after assignment" will be reported. The same logic
applies to nonlocal.

--
Ivan


On 12 September 2016 at 11:37, Christian Heimes 
wrote:

> On 2016-09-09 18:53, guido.van.rossum wrote:
> > https://hg.python.org/cpython/rev/804b71d43c85
> > changeset:   103415:804b71d43c85
> > user:Guido van Rossum 
> > date:Fri Sep 09 09:36:26 2016 -0700
> > summary:
> >   Issue #27999: Make "global after use" a SyntaxError, and ditto for
> nonlocal.
> > Patch by Ivan Levkivskyi.
> >
> > files:
> >   Doc/reference/simple_stmts.rst |5 +-
> >   Lib/test/test_syntax.py|   18 +++-
> >   Misc/NEWS  |3 +
> >   Python/symtable.c  |  104 +++-
> >   4 files changed, 59 insertions(+), 71 deletions(-)
> >
>
> [...]
>
> > @@ -1337,31 +1313,23 @@
> >  long cur = symtable_lookup(st, name);
> >  if (cur < 0)
> >  VISIT_QUIT(st, 0);
> > -if (cur & DEF_ANNOT) {
> > -PyErr_Format(PyExc_SyntaxError,
> > - "annotated name '%U' can't be nonlocal",
> > - name);
> > +if (cur & (DEF_LOCAL | USE | DEF_ANNOT)) {
> > +char* msg;
> > +if (cur & DEF_ANNOT) {
> > +msg = NONLOCAL_ANNOT;
> > +}
> > +if (cur & DEF_LOCAL) {
> > +msg = NONLOCAL_AFTER_ASSIGN;
> > +}
> > +else {
> > +msg = NONLOCAL_AFTER_USE;
> > +}
> > +PyErr_Format(PyExc_SyntaxError, msg, name);
>
> Hi Guido,
>
> did you mean if / else if / else here? It's not completely clear if the
> code means to set msg a second time if both cur & DEF_ANNOT and cur &
> DEF_LOCAL are true.
>
> Christian
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> levkivskyi%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #27999: Make "global after use" a SyntaxError, and ditto for nonlocal.

[Christian Heimes]

On 2016-09-09 18:53, guido.van.rossum wrote:
> https://hg.python.org/cpython/rev/804b71d43c85
> changeset:   103415:804b71d43c85
> user:Guido van Rossum 
> date:Fri Sep 09 09:36:26 2016 -0700
> summary:
>   Issue #27999: Make "global after use" a SyntaxError, and ditto for nonlocal.
> Patch by Ivan Levkivskyi.
> 
> files:
>   Doc/reference/simple_stmts.rst |5 +-
>   Lib/test/test_syntax.py|   18 +++-
>   Misc/NEWS  |3 +
>   Python/symtable.c  |  104 +++-
>   4 files changed, 59 insertions(+), 71 deletions(-)
> 

[...]

> @@ -1337,31 +1313,23 @@
>  long cur = symtable_lookup(st, name);
>  if (cur < 0)
>  VISIT_QUIT(st, 0);
> -if (cur & DEF_ANNOT) {
> -PyErr_Format(PyExc_SyntaxError,
> - "annotated name '%U' can't be nonlocal",
> - name);
> +if (cur & (DEF_LOCAL | USE | DEF_ANNOT)) {
> +char* msg;
> +if (cur & DEF_ANNOT) {
> +msg = NONLOCAL_ANNOT;
> +}
> +if (cur & DEF_LOCAL) {
> +msg = NONLOCAL_AFTER_ASSIGN;
> +}
> +else {
> +msg = NONLOCAL_AFTER_USE;
> +}
> +PyErr_Format(PyExc_SyntaxError, msg, name);

Hi Guido,

did you mean if / else if / else here? It's not completely clear if the
code means to set msg a second time if both cur & DEF_ANNOT and cur &
DEF_LOCAL are true.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: remove all private C functions from the Python C API?

[Serhiy Storchaka]

On 12.09.16 11:41, Victor Stinner wrote:

2016-09-12 8:23 GMT+02:00 Benjamin Peterson :

That seems like a good idea in abstract. However, the boundaries will
have to be delineated. Many functions beginning _Py are effectively part
of the public API even for "well-behaved" 3rd-party extensions


Oh ok, that's also what I expected.

So we should be very careful. Maybe we can experiment building a few
major C extensions like numpy to find such issues?


I think would be nice to create a test extension that uses *all* stable 
functions, build it with old Python versions and test if it works with 
new Python versions.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: remove all private C functions from the Python C API?

[Victor Stinner]

2016-09-12 8:23 GMT+02:00 Benjamin Peterson :
> That seems like a good idea in abstract. However, the boundaries will
> have to be delineated. Many functions beginning _Py are effectively part
> of the public API even for "well-behaved" 3rd-party extensions

Oh ok, that's also what I expected.

So we should be very careful. Maybe we can experiment building a few
major C extensions like numpy to find such issues?

I already know that some C extensions have to access low-level
internals, like debuggers or profilers. Maybe we need to add something
to allow these extensions being compiled with the "private API"?


> because they are used by magic macros. For example, _Py_Dealloc is used by 
> Py_DECREF.

I suggest to make _Py_Dealloc() public, but explain in its
documentation that you should not use it directly :-)


In some cases, we should define a function for the public API/ABI, but
use a macro for the Python core. We already do that in some cases.
Example:
---
PyAPI_FUNC(PyThreadState *) PyThreadState_Get(void);

#ifdef Py_BUILD_CORE
PyAPI_DATA(_Py_atomic_address) _PyThreadState_Current;
#  define PyThreadState_GET() \
 ((PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current))
#else
#  define PyThreadState_GET() PyThreadState_Get()
#endif
---

For Py_DECREF, I prefer to keep a macro because this one is
performance critical.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com