Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-21 Thread Victor Stinner
2016-09-20 21:01 GMT+02:00 Maciej Fijalkowski : > How about we just make timeit show average and not disable the GC then > (two of the complaints that will not change the execution time)? Thanks for the reminder. The first part of my plan was to write a new module to experiment changes. This part

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-21 Thread Dima Tisnek
I guess what `lru_cache` needs is atomic push-pop: on hit: pop(this) + push_back(this) on miss: pop_front() + push_back(this) I reckon, if flat array is lazy (i.e. can grow larger than no. of keys), then *amortised* push-pop performance is not hard to achieve. Overall, it sounds more like heap q

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-20 Thread Maciej Fijalkowski
On Thu, Sep 15, 2016 at 1:27 PM, Paul Moore wrote: > On 15 September 2016 at 10:43, Raymond Hettinger > wrote: >> Something like this will reveal the true and massive improvement in >> iteration speed: >> >> $ ./python.exe -m timeit -s "d=dict.fromkeys(map(str,range(10**6)))" >> "list(d)"

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-20 Thread Eric Snow
On Tue, Sep 20, 2016 at 5:11 AM, INADA Naoki wrote: > But both of OrderedDict and lru_cache improvements can't be in 3.6 > since 3.6 is beta now. > I'll try it after 3.6rc1. When you do, make sure you keep in mind the performance constraints of *all* the OrderedDict methods. The constraints are

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-20 Thread INADA Naoki
On Tue, Sep 20, 2016 at 7:02 PM, INADA Naoki wrote: > On Tue, Sep 20, 2016 at 6:56 PM, Dima Tisnek wrote: >> Totally random thought: >> >> Can lru_cache be simplified to use an ordered dict instead of dict + >> linked list? >> > > I think so. > See also: http://bugs.python.org/issue28199#msg27693

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-20 Thread Dima Tisnek
Totally random thought: Can lru_cache be simplified to use an ordered dict instead of dict + linked list? On 15 September 2016 at 20:30, Serhiy Storchaka wrote: > On 15.09.16 19:13, Antoine Pitrou wrote: >> >> Since this micro-benchmark creates the keys in order just before >> filling the dict w

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Serhiy Storchaka
On 15.09.16 19:13, Antoine Pitrou wrote: Since this micro-benchmark creates the keys in order just before filling the dict with them, randomizing the insertion order destroys the temporal locality of object header accesses when iterating over the dict keys. *This* looks like the right explanation

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Antoine Pitrou
On Thu, 15 Sep 2016 18:13:54 +0200 Antoine Pitrou wrote: > > This also shows that a micro-benchmark that merely looks ok can actually > be a terrible proxy of actual performance. ... unless all your dicts have their key objects nicely arranged sequentially in heap memory, of course. Regards An

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Antoine Pitrou
On Thu, 15 Sep 2016 08:02:10 -0700 Raymond Hettinger wrote: > > Eric is correct on this one. The consecutive hashes make a huge difference > for Python 3.5. While there is a table full table scan, the check for NULL > entries becomes a predictable branch when all the keys are in consecutive

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Ethan Furman
On 09/15/2016 08:02 AM, Raymond Hettinger wrote: Eric is correct on this one. The consecutive hashes make a huge difference for Python 3.5. While there is a table full table scan, the check for NULL entries becomes a predictable branch when all the keys are in consecutive positions. Ther

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Raymond Hettinger
[Eric] >> My understanding is that the all-int-keys case is an outlier. This is due >> to how ints hash, resulting in fewer collisions and a mostly >> insertion-ordered hash table. Consequently, I'd expect the above >> microbenchmark to give roughly the same result between 3.5 and 3.6, which >>

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Antoine Pitrou
On Thu, 15 Sep 2016 07:08:50 -0600 Eric Snow wrote: > On Sep 15, 2016 06:06, "Serhiy Storchaka" wrote: > > Python 3.5: 10 loops, best of 3: 33.5 msec per loop > > Python 3.6: 10 loops, best of 3: 37.5 msec per loop > > > > These results look surprisingly and inexplicably to me. I expected that

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Eric Snow
On Sep 15, 2016 06:06, "Serhiy Storchaka" wrote: > Python 3.5: 10 loops, best of 3: 33.5 msec per loop > Python 3.6: 10 loops, best of 3: 37.5 msec per loop > > These results look surprisingly and inexplicably to me. I expected that even if there is some performance regression in the lookup or mod

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Serhiy Storchaka
On 15.09.16 12:43, Raymond Hettinger wrote: On Sep 14, 2016, at 11:31 PM, Serhiy Storchaka wrote: Note that this is made at the expense of the 20% slowing down an iteration. $ ./python -m timeit -s "d = dict.fromkeys(range(10**6))" -- "list(d)" Python 3.5: 66.1 msec per loop Python 3.6: 82.5 m

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Serhiy Storchaka
On 15.09.16 11:57, Victor Stinner wrote: Stop! Please stop using timeit, it's lying! * You must not use the minimum but average or median * You must run a microbenchmark in multiple processes to test different randomized hash functions and different memory layouts In short: you should use m

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Serhiy Storchaka
On 15.09.16 11:02, INADA Naoki wrote: Are two Pythons built with same options? Both are built from clean checkout with default options (hg update -C 3.x; ./configure; make -s). The only difference is -std=c99 and additional warnings in 3.6: Python 3.5: gcc -pthread -c -Wno-unused-result -Ws

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Paul Moore
On 15 September 2016 at 10:43, Raymond Hettinger wrote: > Something like this will reveal the true and massive improvement in iteration > speed: > > $ ./python.exe -m timeit -s "d=dict.fromkeys(map(str,range(10**6)))" > "list(d)" >py -3.5 -m timeit -s "d=dict.fromkeys(map(str,range(10**6))

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Victor Stinner
2016-09-15 11:29 GMT+02:00 Antoine Pitrou : > That sounds irrelevant. LTO+PGO improves performance, it does > nothing for benchmarking per se. In the past, I had bad surprised when running benchmarks without PGO: https://haypo.github.io/journey-to-stable-benchmark-deadcode.html I don't recall if

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Raymond Hettinger
> On Sep 14, 2016, at 11:31 PM, Serhiy Storchaka wrote: > > Note that this is made at the expense of the 20% slowing down an iteration. > > $ ./python -m timeit -s "d = dict.fromkeys(range(10**6))" -- "list(d)" > Python 3.5: 66.1 msec per loop > Python 3.6: 82.5 msec per loop A range of consec

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Antoine Pitrou
On Thu, 15 Sep 2016 10:57:07 +0200 Victor Stinner wrote: > > > Both Python is built without neither `--with-optimizations` or `make > > profile-opt`. > > That's bad :-) For most reliable benchmarks, it's better to use > LTO+PGO compilation. That sounds irrelevant. LTO+PGO improves performance

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Paul Moore
On 15 September 2016 at 09:57, Victor Stinner wrote: > 2016-09-15 10:02 GMT+02:00 INADA Naoki : >> In my environ: >> >> ~/local/python-master/bin/python3 -m timeit -s "d = >> dict.fromkeys(range(10**6))" 'list(d)' > > Stop! Please stop using timeit, it's lying! > > * You must not use the minim

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread INADA Naoki
On Thu, Sep 15, 2016 at 5:57 PM Victor Stinner wrote: > 2016-09-15 10:02 GMT+02:00 INADA Naoki : > > In my environ: > > > > ~/local/python-master/bin/python3 -m timeit -s "d = > > dict.fromkeys(range(10**6))" 'list(d)' > > Stop! Please stop using timeit, it's lying! > > * You must not use the

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Victor Stinner
2016-09-15 10:02 GMT+02:00 INADA Naoki : > In my environ: > > ~/local/python-master/bin/python3 -m timeit -s "d = > dict.fromkeys(range(10**6))" 'list(d)' Stop! Please stop using timeit, it's lying! * You must not use the minimum but average or median * You must run a microbenchmark in multip

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Paul Moore
On 15 September 2016 at 07:31, Serhiy Storchaka wrote: > Note that this is made at the expense of the 20% slowing down an iteration. > > $ ./python -m timeit -s "d = dict.fromkeys(range(10**6))" -- "list(d)" > Python 3.5: 66.1 msec per loop > Python 3.6: 82.5 msec per loop On my Windows 7 PC with

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread INADA Naoki
> > > Note that this is made at the expense of the 20% slowing down an iteration. > > $ ./python -m timeit -s "d = dict.fromkeys(range(10**6))" -- "list(d)" > Python 3.5: 66.1 msec per loop > Python 3.6: 82.5 msec per loop > > Are two Pythons built with same options? In my environ: ~/local/python

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-15 Thread Victor Stinner
2016-09-15 8:31 GMT+02:00 Serhiy Storchaka : > Note that this is made at the expense of the 20% slowing down an iteration. > > $ ./python -m timeit -s "d = dict.fromkeys(range(10**6))" -- "list(d)" > Python 3.5: 66.1 msec per loop > Python 3.6: 82.5 msec per loop > > Fortunately the cost of the loo

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-14 Thread Serhiy Storchaka
On 08.09.16 23:22, Victor Stinner wrote: I pushed INADA Naoki's implementation of the "compact dict". The hash table now stores indices pointing to a new second table which contains keys and values: it adds one new level of indirection. The table of indices is "compact": use 1, 2, 4 or 8 bytes pe

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-14 Thread Raymond Hettinger
> On Sep 14, 2016, at 3:50 PM, Eric Snow wrote: > >> >> Then, I'll do same to sets. > > Unless I've misunderstood, Raymond was opposed to making a similar > change to set. That's right. Here are a few thoughts on the subject before people starting running wild. * For the compact dict, the

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-14 Thread Tim Delaney
On 15 September 2016 at 05:02, Terry Reedy wrote: > > We already have compact mutable collection types that can be kept > insert-ordered if one chooses -- lists and collections.deque -- and they > are not limited to hashables. Before sets were added, either lists or > dicts with None values were

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-14 Thread Eric Snow
On Wed, Sep 14, 2016 at 7:33 AM, INADA Naoki wrote: > I'll improve OrderedDict after dict in 3.6 is stable enough. +1 and if it's done carefully we could even utilize the pure Python OrderedDict and get rid of odictobject.c (and fold dict-common.h back into dictobject.c). We'd need to leave the

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-14 Thread Terry Reedy
On 9/14/2016 9:33 AM, INADA Naoki wrote: I mean using a compact representation, if not an ordered one. I have no particular usecase in mind. As far as I understand the compact implementation, sets can do it just as well. The original discussion proposed trying to implement it for sets first. L

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-14 Thread INADA Naoki
> > I mean using a compact representation, if not an ordered one. > > I have no particular usecase in mind. As far as I understand the compact > implementation, sets can do it just as well. The original discussion > proposed trying to implement it for sets first. > > Like dict, they would (probably

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-14 Thread Franklin? Lee
On Sep 14, 2016 8:29 AM, "Paul Moore" wrote: > > On 14 September 2016 at 13:18, Franklin? Lee > wrote: > > On Sep 9, 2016 1:35 AM, "Benjamin Peterson" wrote: > >> On Thu, Sep 8, 2016, at 22:33, Tim Delaney wrote: > >> > Are sets also ordered by default now? None of the PEPs appear to mention > >

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-14 Thread Paul Moore
On 14 September 2016 at 13:18, Franklin? Lee wrote: > On Sep 9, 2016 1:35 AM, "Benjamin Peterson" wrote: >> On Thu, Sep 8, 2016, at 22:33, Tim Delaney wrote: >> > Are sets also ordered by default now? None of the PEPs appear to mention >> > it. >> >> No. > > Is there anyone working to move sets i

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-14 Thread Franklin? Lee
On Sep 9, 2016 1:35 AM, "Benjamin Peterson" wrote: > On Thu, Sep 8, 2016, at 22:33, Tim Delaney wrote: > > Are sets also ordered by default now? None of the PEPs appear to mention > > it. > > No. Is there anyone working to move sets in the same direction for 3.6? _

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-13 Thread MRAB
On 2016-09-14 00:42, Greg Ewing wrote: MRAB wrote: On 2016-09-13 07:57, Mark Lawrence via Python-Dev wrote: "tables the idea" has the US meaning of close it down, not the UK meaning of open it up? :) A better phrase would've been "shelves the idea". There's even a module in Python called "s

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-13 Thread Greg Ewing
MRAB wrote: On 2016-09-13 07:57, Mark Lawrence via Python-Dev wrote: "tables the idea" has the US meaning of close it down, not the UK meaning of open it up? :) A better phrase would've been "shelves the idea". There's even a module in Python called "shelve", which makes it Pythonic. :-)

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-13 Thread Sven R. Kunze
On 13.09.2016 20:21, Tres Seaver wrote: *Lots* of library authors have to straddle Python versions: consumers of those libraries only get to pick and choose when their code is at the "leaf" of the dependency tree (the application). Maybe, I didn't express myself well but this was not my intende

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-13 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/13/2016 02:11 PM, Sven R. Kunze wrote: > On 13.09.2016 19:59, MRAB wrote: >> The recommended way of dealing with features across different >> versions of Python is to check for them and see if they raise >> NameError or whatever, but I wonder if

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-13 Thread Sven R. Kunze
On 13.09.2016 19:59, MRAB wrote: The recommended way of dealing with features across different versions of Python is to check for them and see if they raise NameError or whatever, but I wonder if there would be any benefit to recording such things somewhere, e.g. sys.features['ordered_args'] re

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-13 Thread MRAB
On 2016-09-13 11:44, Nick Coghlan wrote: On 13 September 2016 at 08:25, Gregory P. Smith wrote: At this point I think coding up an example patch against beta1 offering a choice of disordered iteration capability that does not increase memory or iteration overhead in any significant way is neede

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-13 Thread MRAB
On 2016-09-13 07:57, Mark Lawrence via Python-Dev wrote: On 12/09/2016 23:25, Gregory P. Smith wrote: [snip] The problem is... I don't know how to express this as an API. Which sinks my whole though process and tables the idea. "tables the idea" has the US meaning of close it down, not the

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-13 Thread Nick Coghlan
On 13 September 2016 at 08:25, Gregory P. Smith wrote: > At this point I think coding up an example patch against beta1 offering a > choice of disordered iteration capability that does not increase memory or > iteration overhead in any significant way is needed. > > The problem is... I don't know

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-13 Thread Nick Coghlan
On 13 September 2016 at 12:37, Tim Delaney wrote: > Personally I expect all Python 3.6 implementations will have > order-preserving dict as that's the easiest way to achieve the existing > guarantees. Not all Python 3 implementation will be able to afford the memory hit that comes from doing that

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Mark Lawrence via Python-Dev
On 12/09/2016 23:25, Gregory P. Smith wrote: On Mon, Sep 12, 2016 at 10:25 AM INADA Naoki wrote: So fundamental question is: Is it to so bad thing that some people write code depending on CPython and PyPy implementation? Yes. See below. I think cross-interpreter libraries can us

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Tim Delaney
On 13 September 2016 at 10:28, Brett Cannon wrote: > >> I'd like to add one more documented constraint - that dict literals >> maintain definition order (so long as the dict is not further modified). >> This allows defining a dict literal and then passing it as **kwargs. >> > > That would require

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Brett Cannon
On Mon, 12 Sep 2016 at 16:52 Tim Delaney wrote: > On 10 September 2016 at 03:17, Guido van Rossum wrote: > >> I've been asked about this. Here's my opinion on the letter of the law in >> 3.6: >> >> - keyword args are ordered >> - the namespace passed to a metaclass is ordered by definition order

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Tim Delaney
On 10 September 2016 at 03:17, Guido van Rossum wrote: > I've been asked about this. Here's my opinion on the letter of the law in > 3.6: > > - keyword args are ordered > - the namespace passed to a metaclass is ordered by definition order > - ditto for the class __dict__ > > A compliant implemen

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Eric Snow
On Mon, Sep 12, 2016 at 4:46 PM, Ethan Furman wrote: > Does anyone have a short explanation of the interaction between hash > randomization and this new always ordered dict? Why doesn't one make the > other useless? Before 3.6, dict iteration was based on the hash table, which varies based on th

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Gregory P. Smith
On Mon, Sep 12, 2016 at 3:57 PM Brett Cannon wrote: > On Mon, 12 Sep 2016 at 15:46 Ethan Furman wrote: > > On 09/12/2016 09:27 AM, Gregory P. Smith wrote: > > > For the regular dict (non kwargs or namespace __dict__) use case I would > actually like to /see disorder preserved during iteration/.

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Brett Cannon
On Mon, 12 Sep 2016 at 15:46 Ethan Furman wrote: > On 09/12/2016 09:27 AM, Gregory P. Smith wrote: > > > For the regular dict (non kwargs or namespace __dict__) use case I would > actually like to /see disorder preserved during iteration/. > > > > If we don't, we will eventually to find ourselves

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Ethan Furman
On 09/12/2016 09:27 AM, Gregory P. Smith wrote: For the regular dict (non kwargs or namespace __dict__) use case I would actually like to /see disorder preserved during iteration/. If we don't, we will eventually to find ourselves in a similar state we were in pre hash-randomization: Does a

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Gregory P. Smith
On Mon, Sep 12, 2016 at 9:51 AM Chris Angelico wrote: > On Tue, Sep 13, 2016 at 2:27 AM, Gregory P. Smith wrote: > > Disorder for this purpose need not be a random shuffle (overkill). It > just > > needs to be regularly inconsistent. A simple thing to do on top of 3.6's > new > > dict implementa

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Gregory P. Smith
On Mon, Sep 12, 2016 at 10:25 AM INADA Naoki wrote: > > So fundamental question is: Is it to so bad thing that some people > write code depending on CPython and PyPy implementation? > Yes. See below. I think cross-interpreter libraries can use OrederedDict correctly > when they should use it.

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread INADA Naoki
> From what I understood, Python 3.6 dict got two *different* changes: > > * modify the dict structure to use two tables instead of only one: an > "index" table (the hash table) and a second key/value table > * tune the dict implementation to only append to the key/value table > > The second change

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Tim Peters
[Guido] > Wouldn't attempting to reuse DUMMY entries be expensive? You'd have to > search forward in the array. Just keeping a count of DUMMY entries and > compacting when there are too many seems better somehow. I haven't looked at the code, but presumably one of the members of a DUMMY key/value

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread INADA Naoki
On Tue, Sep 13, 2016 at 1:35 AM, Guido van Rossum wrote: > Couldn't we use the order in the actual hash table (which IIUC now > contains just indexes into the ordered vector of key/value/hash > structs)? That would probably simulate the pre-3.6 order quite > effectively. Maybe, it can. But curren

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Guido van Rossum
Wouldn't attempting to reuse DUMMY entries be expensive? You'd have to search forward in the array. Just keeping a count of DUMMY entries and compacting when there are too many seems better somehow. On Mon, Sep 12, 2016 at 10:00 AM, Victor Stinner wrote: > 2016-09-12 18:35 GMT+02:00 Guido van Ros

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Victor Stinner
2016-09-12 18:35 GMT+02:00 Guido van Rossum : > Couldn't we use the order in the actual hash table (which IIUC now > contains just indexes into the ordered vector of key/value/hash > structs)? That would probably simulate the pre-3.6 order quite > effectively. >From what I understood, Python 3.6 d

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Chris Angelico
On Tue, Sep 13, 2016 at 2:27 AM, Gregory P. Smith wrote: > Disorder for this purpose need not be a random shuffle (overkill). It just > needs to be regularly inconsistent. A simple thing to do on top of 3.6's new > dict implementation would be to pick a random starting point within the > order arr

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Guido van Rossum
Couldn't we use the order in the actual hash table (which IIUC now contains just indexes into the ordered vector of key/value/hash structs)? That would probably simulate the pre-3.6 order quite effectively. But we'd have to add a new API to reveal the order (in effect just what Nick wanted). How m

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Gregory P. Smith
For the regular dict (non kwargs or namespace __dict__) use case I would actually like to *see disorder preserved during iteration*. If we don't, we will eventually to find ourselves in a similar state we were in pre hash-randomization: (1) Over time, code will come to depend on the order for no

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Victor Stinner
2016-09-12 13:50 GMT+02:00 Antoine Pitrou : > Besides, I don't think it has been proven that the compact-and-ordered > dict implementation is actually *faster* than the legacy one. Python 3.6 dict is slower than Python 3.5 dict, at least for a simple lookup: http://bugs.python.org/issue27350#msg27

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-12 Thread Antoine Pitrou
On Fri, 9 Sep 2016 14:01:08 -0500 David Mertz wrote: > It seems unlikely, but not inconceivable, that someday in the future > someone will implement a dictionary that is faster than current versions > but at the cost of losing inherent ordering. I agree with this. Since ordering is a constraint,

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-11 Thread Sven R. Kunze
On 11.09.2016 01:41, Nathaniel Smith wrote: I feel like I'm missing something here... by this reasoning, we should *never* change the language spec when new features are added. E.g. if people use async/await in 3.5 then their code won't be compatible with 3.4, but async/await are still part of th

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-11 Thread Chris Angelico
On Sun, Sep 11, 2016 at 6:42 PM, Victor Stinner wrote: > 2016-09-10 23:24 GMT-04:00 Nick Coghlan : >> To conform with the updated language spec, implementations just need >> to use collections.OrderedDict in 3 places: >> >> (...) >> - storage type for passing kwargs to functions > > I'm not sure a

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-11 Thread Victor Stinner
2016-09-10 23:24 GMT-04:00 Nick Coghlan : > To conform with the updated language spec, implementations just need > to use collections.OrderedDict in 3 places: > > (...) > - storage type for passing kwargs to functions I'm not sure about the "just need" for this one, especially if you care of perfo

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-10 Thread Nick Coghlan
On 11 September 2016 at 09:41, Nathaniel Smith wrote: > On Fri, Sep 9, 2016 at 11:39 AM, Barry Warsaw wrote: >> On Sep 09, 2016, at 01:08 PM, Elvis Pranskevichus wrote: >> >>>Are there any downsides to explicitly specifying that all dicts are ordered? >>>People will inevitably start relying on th

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-10 Thread Nathaniel Smith
On Fri, Sep 9, 2016 at 11:39 AM, Barry Warsaw wrote: > On Sep 09, 2016, at 01:08 PM, Elvis Pranskevichus wrote: > >>Are there any downsides to explicitly specifying that all dicts are ordered? >>People will inevitably start relying on this behaviour, and this will >>essentially become the *de-fact

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-09 Thread Elvis Pranskevichus
On Friday, September 9, 2016 2:01:08 PM EDT David Mertz wrote: > It feels best to me only to promise order in specific cases like kwargs, > but say nothing (even in 3.6 or 3.7) about the requirement for how dict > itself is implemented. On Saturday, September 10, 2016 6:39:27 AM EDT Barry Warsaw w

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-09 Thread David Mertz
It seems unlikely, but not inconceivable, that someday in the future someone will implement a dictionary that is faster than current versions but at the cost of losing inherent ordering. It feels best to me only to promise order in specific cases like kwargs, but say nothing (even in 3.6 or 3.7) a

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-09 Thread Barry Warsaw
On Sep 09, 2016, at 01:08 PM, Elvis Pranskevichus wrote: >Are there any downsides to explicitly specifying that all dicts are ordered? >People will inevitably start relying on this behaviour, and this will >essentially become the *de-facto* spec, so alternative Python implementations >will hav

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-09 Thread Brett Cannon
On Fri, 9 Sep 2016 at 10:28 Victor Stinner wrote: > 2016-09-09 10:17 GMT-07:00 Guido van Rossum : > > - keyword args are ordered > > - the namespace passed to a metaclass is ordered by definition order > > - ditto for the class __dict__ > > Maybe we should define exactly "ordered" somewhere the l

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-09 Thread Victor Stinner
2016-09-09 10:17 GMT-07:00 Guido van Rossum : > - keyword args are ordered > - the namespace passed to a metaclass is ordered by definition order > - ditto for the class __dict__ Maybe we should define exactly "ordered" somewhere the language reference: https://docs.python.org/dev/reference/index.

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-09 Thread Guido van Rossum
I've been asked about this. Here's my opinion on the letter of the law in 3.6: - keyword args are ordered - the namespace passed to a metaclass is ordered by definition order - ditto for the class __dict__ A compliant implementation may ensure the above three requirements either by making all dic

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-09 Thread Elvis Pranskevichus
On Friday, September 9, 2016 4:52:21 PM EDT Brett Cannon wrote: > On Fri, 9 Sep 2016 at 01:58 Antoine Pitrou wrote: > > On Thu, 8 Sep 2016 14:20:53 -0700 > > > > Victor Stinner wrote: > > > 2016-09-08 13:36 GMT-07:00 Guido van Rossum : > > > > IIUC there's one small thing we might still want to

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-09 Thread Brett Cannon
On Fri, 9 Sep 2016 at 01:58 Antoine Pitrou wrote: > On Thu, 8 Sep 2016 14:20:53 -0700 > Victor Stinner wrote: > > 2016-09-08 13:36 GMT-07:00 Guido van Rossum : > > > IIUC there's one small thing we might still want to change somewhere > > > after 3.6b1 but before 3.6rc1: the order is not preserv

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-09 Thread Maciej Fijalkowski
On Fri, Sep 9, 2016 at 10:55 AM, Antoine Pitrou wrote: > On Thu, 8 Sep 2016 14:20:53 -0700 > Victor Stinner wrote: >> 2016-09-08 13:36 GMT-07:00 Guido van Rossum : >> > IIUC there's one small thing we might still want to change somewhere >> > after 3.6b1 but before 3.6rc1: the order is not preser

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-09 Thread Antoine Pitrou
On Thu, 8 Sep 2016 14:20:53 -0700 Victor Stinner wrote: > 2016-09-08 13:36 GMT-07:00 Guido van Rossum : > > IIUC there's one small thing we might still want to change somewhere > > after 3.6b1 but before 3.6rc1: the order is not preserved when you > > delete some keys and then add some other keys.

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-08 Thread Tim Delaney
On 9 September 2016 at 15:34, Benjamin Peterson wrote: > On Thu, Sep 8, 2016, at 22:33, Tim Delaney wrote: > > On 9 September 2016 at 07:45, Chris Angelico wrote: > > > > > On Fri, Sep 9, 2016 at 6:22 AM, Victor Stinner < > victor.stin...@gmail.com> > > > wrote: > > > > A nice "side effect" of c

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-08 Thread Tim Delaney
On 9 September 2016 at 07:45, Chris Angelico wrote: > On Fri, Sep 9, 2016 at 6:22 AM, Victor Stinner > wrote: > > A nice "side effect" of compact dict is that the dictionary now > > preserves the insertion order. It means that keyword arguments can now > > be iterated by their creation order: >

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-08 Thread Benjamin Peterson
On Thu, Sep 8, 2016, at 22:33, Tim Delaney wrote: > On 9 September 2016 at 07:45, Chris Angelico wrote: > > > On Fri, Sep 9, 2016 at 6:22 AM, Victor Stinner > > wrote: > > > A nice "side effect" of compact dict is that the dictionary now > > > preserves the insertion order. It means that keywo

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-08 Thread Chris Angelico
On Fri, Sep 9, 2016 at 6:22 AM, Victor Stinner wrote: > A nice "side effect" of compact dict is that the dictionary now > preserves the insertion order. It means that keyword arguments can now > be iterated by their creation order: > This is pretty sweet! Of course, there are going to be 1172 com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-08 Thread Guido van Rossum
On Thu, Sep 8, 2016 at 1:36 PM, Guido van Rossum wrote: > IIUC there's one small thing we might still want to change somewhere > after 3.6b1 but before 3.6rc1: the order is not preserved when you > delete some keys and then add some other keys. Apparently PyPy has > come up with a clever solution

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-08 Thread Victor Stinner
2016-09-08 13:36 GMT-07:00 Guido van Rossum : > IIUC there's one small thing we might still want to change somewhere > after 3.6b1 but before 3.6rc1: the order is not preserved when you > delete some keys and then add some other keys. Apparently PyPy has > come up with a clever solution for this, a

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-08 Thread Guido van Rossum
Thanks Victor for the review and commit, and thanks Naoki for your truly amazing implementation work!! I've also accepted Eric's PEP 468. IIUC there's one small thing we might still want to change somewhere after 3.6b1 but before 3.6rc1: the order is not preserved when you delete some keys and th

[Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

2016-09-08 Thread Victor Stinner
Hi, I pushed INADA Naoki's implementation of the "compact dict". The hash table now stores indices pointing to a new second table which contains keys and values: it adds one new level of indirection. The table of indices is "compact": use 1, 2, 4 or 8 bytes per indice depending on the size of the