[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Inada Naoki added the comment: New changeset 4f74052b455a54ac736f38973693aeea2ec14116 by Inada Naoki in branch 'main': bpo-40116: dict: Add regression test for iteration order. (GH-31550) https://github.com/python/cpython/commit/4f74052b455a54ac736f38973693aeea2ec14116 --

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Inada Naoki added the comment: PyDict_Keys(), PyDict_Values(), and PyDict_Items() don't respect insertion order too. -- ___ Python tracker ___

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Change by Inada Naoki : -- pull_requests: +29671 pull_request: https://github.com/python/cpython/pull/31550 ___ Python tracker ___

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Inada Naoki added the comment: I found regression caused by GH-28520. ``` class C: def __init__(self, n): if n: self.a = 1 self.b = 2 self.c = 3 else: self.c = 1 self.b = 2 self.a = 3 o1 = C(True) o2

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Change by Mark Shannon : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Mark Shannon added the comment: New changeset a7252f88d3fa33036bdd6036b8c97bc785ed6f17 by Mark Shannon in branch 'main': bpo-40116: Add insertion order bit-vector to dict values to allow dicts to share keys more freely. (GH-28520)

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Serhiy Storchaka added the comment: There are several common idioms which do not work well with shared dictionaries. 1. Class attributes as defaults. If most instances of the class have the default value for some attribute, it can be set as the class attribute. It saves memory because most

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Mark Shannon added the comment: Raymond, Only split dicts need the extra field. Classes where many instances do not have exactly the same set of attributes may be more common than you think. There are many reasons why some attributes may be added conditionally. PR 28520 actually makes the

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Marc-Andre Lemburg added the comment: On 22.09.2021 21:02, Raymond Hettinger wrote: >> The language specification says that the dicts maintain insertion >> order, but the wording implies that this only to explicit >> dictionaries, not instance attribute or other namespace dicts. > > That is

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Raymond Hettinger added the comment: > Overall, I expect the improved sharing to more than > compensate for the disadvantages. I expect the opposite. This makes all dicts pay a price (in space, initialization time, and complexity) for a micro-optimization of an uncommon case (the normal

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Change by Mark Shannon : -- keywords: +patch pull_requests: +26911 stage: test needed -> patch review pull_request: https://github.com/python/cpython/pull/28520 ___ Python tracker

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Mark Shannon added the comment: This can be mitigated, if not entirely fixed, by storing an ordering bit vector in the values. This way all instances of the class SometimesShared in the example above can share the keys. The keys might be ("optional", "attr") For any instances with only

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Change by Mark Shannon : -- nosy: +rhettinger ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Mark Shannon added the comment: You can't end up with a growing set of keys. The problem is to do with the *order* of insertion, not the number of keys. -- ___ Python tracker

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Mark Shannon added the comment: Just to clarify. class AlwaysShared: opt = DEFAULT def __init__(self, attr, optional=None): self.attr = attr if optional: self.opt = optional class SometimesShared: opt = DEFAULT def __init__(self, attr, optional=None):

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Serhiy Storchaka added the comment: I think the current behavior is a guard against such pitfall. If you allow to add new keys when not all other keys are set, you can end with sharing growing set of keys. -- ___ Python tracker

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Mark Shannon added the comment: Indeed it shouldn't. How is that relevant? -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

Serhiy Storchaka added the comment: But the following class should not lead to unlimited memory consumption when create new instances: class C: count = 0 def __init__(self): count = self.__class__.count self.__class__.count = count + 1 setattr(self,

[issue40116] Regression in memory use of shared key dictionaries for "compact dicts"

New submission from Mark Shannon : The current implementation of dicts prevents keys from being shared when the order of attribute differs from the first instance created. This can potentially use a considerably larger amount of memory than expected. Consider the class: class C: opt =