[Python-Dev] Compact ordered dict is not ordered for split table. (was: PEP XXX: Compact ordered dict
I'm sorry, but I hadn't realized which compact ordered dict is
not ordered for split table.
For example:
>>> class A:
... ...
...
>>> a = A()
>>> b = A()
>>> a.a = 1
>>> a.b = 2
>>> b.b = 3
>>> b.a = 4
>>> a.__dict__.items()
dict_items([('a', 1), ('b', 2)])
>>> b.__dict__.items()
dict_items([('a', 4), ('b', 3)])
This doesn't affects to **kwargs and class namespace.
But if we change the language spec to dict preserves insertion order,
this should be addressed.
On Tue, Jun 21, 2016 at 2:02 PM, INADA Naoki wrote:
> On Tue, Jun 21, 2016 at 12:17 PM, Oleg Broytman wrote:
>> Hi!
>>
>> On Tue, Jun 21, 2016 at 11:14:39AM +0900, INADA Naoki
>> wrote:
>>> Here is my draft, but I haven't
>>> posted it yet since
>>> my English is much worse than C.
>>> https://www.dropbox.com/s/s85n9b2309k03cq/pep-compact-dict.txt?dl=0
>>
>>It's good enough for a start (if a PEP is needed at all). If you push
>> it to Github I'm sure they will come with pull requests.
>>
>> Oleg.
>
> Thank you for reading my draft.
>
>> (if a PEP is needed at all)
>
> I don't think so. My PEP is not for changing Python Language,
> just describe implementation detail.
>
> Python 3.5 has new OrderedDict implemented in C without PEP.
> My patch is relatively small than it. And the idea has been well known.
>
> --
> INADA Naoki
--
INADA Naoki
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487: Simpler customization of class creation
On Mon, Jun 20, 2016 at 12:31 PM, Nikita Nemkin wrote: > Right. Ordered by default is a very serious implementation constraint. > It's only superior in a sense that it completely subsumes/obsoletes > PEP 520. Just to be clear, PEP 520 is more than just OrderedDict-by-default. In fact, the key point is preserving the definition order, which the PEP now reflects better. Raymond's compact dict would only provide the ordered-by-default part and does nothing to persist the definition order like the PEP specifies. -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487: Simpler customization of class creation
On Tue, Jun 21, 2016 at 10:12 AM, Eric Snow wrote: > On Mon, Jun 20, 2016 at 12:31 PM, Nikita Nemkin wrote: > > Right. Ordered by default is a very serious implementation constraint. > > It's only superior in a sense that it completely subsumes/obsoletes > > PEP 520. > > Just to be clear, PEP 520 is more than just OrderedDict-by-default. > In fact, the key point is preserving the definition order, which the > PEP now reflects better. Raymond's compact dict would only provide > the ordered-by-default part and does nothing to persist the definition > order like the PEP specifies. > Judging from Inada's message there seems to be some confusion about how well the compact dict preserves order (personally I think if it doesn't guarantee order after deletions it's pretty useless). Assuming it preserves order across deletions/compactions (like IIUC OrderedDict does) isn't that good enough for any of the use cases considered? It would require a delete+insert to change an item's order. If we had had these semantics in the language from the start, there would have been plenty uses of this order, and I suspect nobody would have considered asking for __definition_order__. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487: Simpler customization of class creation
> On Jun 21, 2016, at 10:18 AM, Guido van Rossum wrote:
>
> Judging from Inada's message there seems to be some confusion about how well
> the compact dict preserves order (personally I think if it doesn't guarantee
> order after deletions it's pretty useless).
Inada should follow PyPy's implementation of the compact dict which does
preserve order after deletions (see below).
My original proof-of-concept code didn't have that feature; instead, it was
aimed at saving space and speeding-up iteration. The key ordering was just a
by-product. Additional logic was needed to preserve order for interleaved
insertions and deletions.
Raymond
---(PyPy test of order
preservation)-
'Demonstrate PyPy preserves order across repeated insertions and deletions'
from random import randrange
import string
s = list(string.letters)
d = dict.fromkeys(s)
n = len(s)
for _ in range(1):
i = randrange(n)
c = s.pop(i);s.append(c)
d.pop(c);d[c] = None
assert d.keys() == s
---(PyPy session showing order
preservation)--
$ pypy
Python 2.7.10 (c09c19272c990a0611b17569a0085ad1ab00c8ff, Jun 13 2016, 03:59:08)
[PyPy 5.3.0 with GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on
darwin
Type "help", "copyright", "credits" or "license" for more information.
d = dict(raymond='red', rachel='blue', matthew='yellow')
del d['rachel']
d['cindy'] = 'green'
d['rachel'] = 'azure'
d
{'raymond': 'red', 'matthew': 'yellow', 'cindy': 'green', 'rachel': 'azure'}
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] When to use EOFError?
There is a design question. If you read file in some format or with some protocol, and the data is ended unexpectedly, when to use general EOFError exception and when to use format/protocol specific exception? For example when load truncated pickle data, an unpickler can raise EOFError, UnpicklingError, ValueError or AttributeError. It is possible to avoid ValueError or AttributeError, but what exception should be raised instead, EOFError or UnpicklingError? Maybe convert all EOFError to UnpicklingError? Or all UnpicklingError caused by unexpectedly ended input to EOFError? Or raise EOFError if the input is ended after completed opcode, and UnpicklingError if it contains truncated opcode? http://bugs.python.org/issue25761 ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487: Simpler customization of class creation
On 21 June 2016 at 10:18, Guido van Rossum wrote: > On Tue, Jun 21, 2016 at 10:12 AM, Eric Snow > wrote: >> >> On Mon, Jun 20, 2016 at 12:31 PM, Nikita Nemkin wrote: >> > Right. Ordered by default is a very serious implementation constraint. >> > It's only superior in a sense that it completely subsumes/obsoletes >> > PEP 520. >> >> Just to be clear, PEP 520 is more than just OrderedDict-by-default. >> In fact, the key point is preserving the definition order, which the >> PEP now reflects better. Raymond's compact dict would only provide >> the ordered-by-default part and does nothing to persist the definition >> order like the PEP specifies. > > > Judging from Inada's message there seems to be some confusion about how well > the compact dict preserves order (personally I think if it doesn't guarantee > order after deletions it's pretty useless). > > Assuming it preserves order across deletions/compactions (like IIUC > OrderedDict does) isn't that good enough for any of the use cases > considered? It would require a delete+insert to change an item's order. If > we had had these semantics in the language from the start, there would have > been plenty uses of this order, and I suspect nobody would have considered > asking for __definition_order__. RIght, if *tp_dict itself* on type objects is guaranteed to be order-preserviing, then we don't need to do anything except perhaps provide a helper method or descriptor on type that automatically filters out the dunder-attributes, and spell out the type dict population order for: - class statements (universal) - types.new_class (universal) - calling type() directly (universal) - PyType_Ready (CPython-specific) - PyType_FromSpec (CPython-specific) Something that isn't currently defined in PEP 520, and probably should be regardless of whether the final implementation is an order preserving tp_dict or a new __definition_order__ attribute, is where descriptors implicitly defined via __slots__ will appear relative to other attributes. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] frame evaluation API PEP
On 20 June 2016 at 13:32, Dino Viehland via Python-Dev wrote: > It doesn’t help with the issue of potentially multiple consumers of that field > that has been brought up before but I’m not sure how concerned we should be > about that scenario anyway. Brett's comparison with sys.settrace seems relevant here - we don't allow multiple trace hooks at once, which means if you want more than one active at once, either they need to cooperate with each other, or you need to install a meta-tracehook to manage them somehow. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] When to use EOFError?
When loading truncated data with pickle, I expect a pickle error, not a generic ValueError nor EOFError. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Review of PEP 520: Ordered Class Definition Namespace
On 20 June 2016 at 19:11, Eric Snow wrote: > FWIW, regarding repercussions, I do not expect any other potential > future feature will subsume the functionality of PEP 520. The closest > thing would be if cls.__dict__ became ordered. However, that would > intersect with __definition_order__ only at first. Furthermore, > cls.__dict__ would only ever be able to make vague promises about any > relationship with the definiton order. The point of > __definiton_order__ is to provide the one obvious place to get a > specific bit of information about a class. It occurs to me that a settable __definition_order__ provides a benefit that an ordered tp_dict doesn't: to get the "right" definition order in something like Cython or dynamic type creation, you don't need to carefully craft the order in which attributes are defined, you just need to set __definition_order__ appropriately. It also means that the "include dunder-attributes or not" decision is easy to override, regardless of what we set as the default. By contrast, if the *only* ordering information is cls.__dict__.keys(), then there's no way for a type implementor to hide implementation details. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487: Simpler customization of class creation
On Wed, Jun 22, 2016 at 2:50 AM, Raymond Hettinger
wrote:
>
>> On Jun 21, 2016, at 10:18 AM, Guido van Rossum wrote:
>>
>> Judging from Inada's message there seems to be some confusion about how well
>> the compact dict preserves order (personally I think if it doesn't guarantee
>> order after deletions it's pretty useless).
>
> Inada should follow PyPy's implementation of the compact dict which does
> preserve order after deletions (see below).
I follow it, for most cases.
When my compact dict doesn't preserve order is using PEP 412 Key sharing dict.
>>> class A:
... ...
...
>>> a = A()
>>> b = A() # a and b shares same keys, and have each values
>>> a.a = 1
>>> a.b = 2 # The order in shared key is (a, b)
>>> b.b = 3
>>> b.a = 4
>>> a.__dict__.items()
dict_items([('a', 1), ('b', 2)])
>>> b.__dict__.items()
dict_items([('a', 4), ('b', 3)])
It's possible to split keys when the insertion order is not strictly same.
But it decrease efficiency of key sharing dict.
If key sharing dict is effective only such a very strict cases,
I feel __slots__ can be used for it.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Review of PEP 520: Ordered Class Definition Namespace
On Tue, Jun 21, 2016 at 3:21 PM, Nick Coghlan wrote: > It occurs to me that a settable __definition_order__ provides a > benefit that an ordered tp_dict doesn't: to get the "right" definition > order in something like Cython or dynamic type creation, you don't > need to carefully craft the order in which attributes are defined, you > just need to set __definition_order__ appropriately. > > It also means that the "include dunder-attributes or not" decision is > easy to override, regardless of what we set as the default. > > By contrast, if the *only* ordering information is > cls.__dict__.keys(), then there's no way for a type implementor to > hide implementation details. Good point. I'll make a note of this in the PEP. -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487: Simpler customization of class creation
On Tue, Jun 21, 2016 at 11:18 AM, Guido van Rossum wrote: > If we had had these semantics in the language from the start, there would have > been plenty uses of this order, and I suspect nobody would have considered > asking for __definition_order__. True. The key thing that __definition_order__ provides is an explicit relationship with the class definition. Since we have the opportunity to capture that now, I think we should take it, regardless of the type of the class definition namespace or even of cls.__dict__. For me the strong association with the order in the class definition is worth having. -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487: Simpler customization of class creation
On Tue, Jun 21, 2016 at 3:01 PM, Nick Coghlan wrote: > RIght, if *tp_dict itself* on type objects is guaranteed to be > order-preserviing, then we don't need to do anything except perhaps > provide a helper method or descriptor on type that automatically > filters out the dunder-attributes, and spell out the type dict > population order for: > > - class statements (universal) > - types.new_class (universal) > - calling type() directly (universal) > - PyType_Ready (CPython-specific) > - PyType_FromSpec (CPython-specific) The problem I have with this is that it still doesn't give any strong relationship with the class definition. Certainly in most cases it will amount to the same thing. However, there is no way to know if cls.__dict__ represents the class definition or not. You also lose knowing whether or not a class came from a definition (or acts as though it did). Finally, __definition_order__ makes the relationship with the definition order clear, whereas cls.__dict__ does not. Instead of being an obvious tool, with cls.__dict__ that relationship would be tucked away where only a few folks with deep knowledge of Python would be in a position to take advantage. > > Something that isn't currently defined in PEP 520, and probably should > be regardless of whether the final implementation is an order > preserving tp_dict or a new __definition_order__ attribute, is where > descriptors implicitly defined via __slots__ will appear relative to > other attributes. I'll add that. -eric ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compact ordered dict is not ordered for split table. (was: PEP XXX: Compact ordered dict
There are three options I can think.
1) Revert key-shared dict (PEP412).
pros: Removing key-shared dict makes dict implementation simple.
cons: In some applications, PEP 412 is far more compact than compact
ordered dict. (Note: Using __slots__ may help such situation).
2) Don't make "keeping insertion order" is Python Language Spec.
pros: Best efficiency
cons: Different behavior between normal dict and instance.__dict__ may
confuse people.
3) More strict rule for key sharing dict.
My idea is:
* Increasing number of entries (inserting new key) can be possible
only if refcnt of keys == 1.
* Inserting new item (with existing key) into dict is allowed only when
insertion position == number of items in the dict (PyDictObject.ma_used).
pros: We can have "dict keeping insertion order".
cons: Can't use key-sharing dict for many cases. Small and harmless
change may cause
sudden memory usage increase. (__slots__ is more predicable).
On Wed, Jun 22, 2016 at 12:10 AM, INADA Naoki wrote:
> I'm sorry, but I hadn't realized which compact ordered dict is
> not ordered for split table.
>
> For example:
class A:
> ... ...
> ...
a = A()
b = A()
a.a = 1
a.b = 2
b.b = 3
b.a = 4
a.__dict__.items()
> dict_items([('a', 1), ('b', 2)])
b.__dict__.items()
> dict_items([('a', 4), ('b', 3)])
>
>
> This doesn't affects to **kwargs and class namespace.
>
> But if we change the language spec to dict preserves insertion order,
> this should be addressed.
>
>
> On Tue, Jun 21, 2016 at 2:02 PM, INADA Naoki wrote:
>> On Tue, Jun 21, 2016 at 12:17 PM, Oleg Broytman wrote:
>>> Hi!
>>>
>>> On Tue, Jun 21, 2016 at 11:14:39AM +0900, INADA Naoki
>>> wrote:
Here is my draft, but I haven't
posted it yet since
my English is much worse than C.
https://www.dropbox.com/s/s85n9b2309k03cq/pep-compact-dict.txt?dl=0
>>>
>>>It's good enough for a start (if a PEP is needed at all). If you push
>>> it to Github I'm sure they will come with pull requests.
>>>
>>> Oleg.
>>
>> Thank you for reading my draft.
>>
>>> (if a PEP is needed at all)
>>
>> I don't think so. My PEP is not for changing Python Language,
>> just describe implementation detail.
>>
>> Python 3.5 has new OrderedDict implemented in C without PEP.
>> My patch is relatively small than it. And the idea has been well known.
>>
>> --
>> INADA Naoki
>
>
>
> --
> INADA Naoki
--
INADA Naoki
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487: Simpler customization of class creation
Nick Coghlan wrote: Something that isn't currently defined in PEP 520 ... is where descriptors implicitly defined via __slots__ will appear relative to other attributes. In the place where the __slots__ attribute appears? -- Greg ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
