> Date: Wed, 10 Jun 2015 21:33:44 +1200
> From: robe...@robertcollins.net
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [all][python3] use of six.iteritems()
>
> On 10 June 2015 at 17:22, gordon chung <g...@live.ca> wrote:
> > maybe the suggestion should be "don't blindly apply six.iteritems or items"
> > rather than don't apply iteritems at all. admittedly, it's a massive
> > eyesore, but it's a very real use case that some projects deal with large
> > data results and to enforce the latter policy can have negative effects[1].
> > one "million item dictionary" might be negligible but in a multi-user,
> > multi-* environment that can have a significant impact on the amount memory
> > required to store everything.
>
> > [1] disclaimer: i have no real world results but i assume memory management
> > was the reason for the switch in logic from py2 to py3
>
> I wouldn't make that assumption.
>
> And no, memory isn't an issue. If you have a million item dict,
> ignoring the internal overheads, the dict needs 1 million object
> pointers. The size of a list with those pointers in it is 1M (pointer
> size in bytes). E.g. 4M or 8M. Nothing to worry about given the
> footprint of such a program :)
iiuc, items() (in py2) will create a copy of the dictionary in memory to be
processed. this is useful for cases such as concurrency where you want to
ensure consistency but doing a quick test i noticed a massive spike in memory
usage between items() and iteritems.
'for i in dict(enumerate(range(1000000))).items(): pass' consumes significantly
more memory than 'for i in dict(enumerate(range(1000000))).iteritems(): pass'.
on my system, the difference in memory consumption was double when using
items() vs iteritems() and the cpu util was significantly more as well... let
me know if there's anything that stands out as inaccurate.
unless there's something wrong with my ignorant testing above, i think it's
something projects should consider when mass applying any iteritems/items patch.
cheers,gord
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
