[issue23103] ipaddress should be Flyweight
New submission from Seth Bromberger: ipaddress.ip_address instances should be flyweight. There's really no reason to make them mutable: a = ipaddress.ip_address(10.1.2.3) b = ipaddress.ip_address(10.1.2.3) id(a) 140066533772368 id(b) 140066504622544 Especially with IPv6 and large numbers of addresses, it would be helpful not to have separate instances with the same underlying properties. -- components: Library (Lib) messages: 233038 nosy: Seth.Bromberger priority: normal severity: normal status: open title: ipaddress should be Flyweight type: enhancement versions: Python 3.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Seth Bromberger added the comment: Confirmed on 3.4.0; likely exists in previous versions as well. -- versions: +Python 3.4 -Python 3.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Changes by Berker Peksag berker.pek...@gmail.com: -- nosy: +ncoghlan, pmoody ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23058] argparse silently ignores arguments
Changaco added the comment: honcho has been affected by this: https://github.com/nickstenning/honcho/issues/122 Working around the problem is possible but not very pretty: https://github.com/nickstenning/honcho/pull/121 -- nosy: +Changaco ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23058 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9351] argparse set_defaults on subcommands should override top level set_defaults
Changaco added the comment: The broken patch made it into the 2.7.9 release, causing argparse to silently ignore arguments, cf http://bugs.python.org/issue23058 -- nosy: +Changaco ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23104] [Windows x86-64] Incorrect function call
New submission from Андрей Парамонов:
To reproduce:
0) Compile attached testlib.c
1) Run the following code:
from __future__ import print_function
from __future__ import unicode_literals
from ctypes import *
testlib = windll.LoadLibrary('testlib')
testfun = testlib.test
class objid(Structure):
_fields_ = [('bytes', c_ubyte*16)]
print('Calling...')
testfun(objid(), c_wchar_p('test'))
print('Done.')
---
It gives different output for different versions of Python and processor
architectures:
c:\python27\python test.py
Calling...
test
Done.
c:\python34\python test.py
Calling...
test
Done.
c:\python27-64\python test.py
Calling...
test
Done.
c:\python34-64\python test.py
Calling...
Done.
It appears that Python 3.4 on Windows x86-64 generates incorrect function call
code.
--
components: ctypes
files: testlib.c
messages: 233042
nosy: Андрей.Парамонов
priority: normal
severity: normal
status: open
title: [Windows x86-64] Incorrect function call
versions: Python 3.4
Added file: http://bugs.python.org/file37534/testlib.c
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23104
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Josh Rosenberg added the comment: What is your proposal? WeakValueDictionary mapping raw bytes object to single instance of ipaddress that is queried from ipaddress's __new__? No built-in has quite that extensive a level of caching and aggressive deduplication to my knowledge. -- nosy: +josh.r ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Seth Bromberger added the comment: What is your proposal? WeakValueDictionary mapping raw bytes object to single instance of ipaddress that is queried from ipaddress's __new__? No built-in has quite that extensive a level of caching and aggressive deduplication to my knowledge. I was thinking along those lines, but since v4 addresses are just UInt32 and v6 are UInt128, somehow mapping just the address information if we can do that via weakref dicts. Right now these objects can't really be used to represent addresses parsed from logs with tens of millions of entries generated by a couple thousand machines. It would be nice to have a memory-efficient way of doing that without having to duplicate the concept in a third-party package. There are a few other changes I'd like to see/make as well - this package is too focused on external functions for manipulating/creating addresses, when standard object/method structure would serve it better and be more intuitive. There is also no discernible need to modify an IP address' value. They should be thought of as immutables to the extent we can enforce that policy. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23102] distutils: tip-toe around quirks owing to setuptools monkey-patching Extension
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23102 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22669] Test_venv fails when _ctypes is not available.
Donald Stufft added the comment: This should be fixed now as of https://hg.python.org/cpython/rev/651e1862dbed, https://hg.python.org/cpython/rev/651e1862dbed, and https://hg.python.org/cpython/rev/9f60d024e586. -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22669 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21793] httplib client/server status refactor
Roundup Robot added the comment: New changeset edf669b13482 by Serhiy Storchaka in branch 'default': Issue #21793: Added http.HTTPStatus enums (i.e. HTTPStatus.OK, https://hg.python.org/cpython/rev/edf669b13482 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21793 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21793] httplib client/server status refactor
Serhiy Storchaka added the comment: Thank you for your contribution Demian. -- resolution: - fixed stage: commit review - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21793 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23105] os.O_SHLOCK and os.O_EXLOCK are not available on Linux
New submission from Sworddragon: From the documentation: os.O_DSYNC os.O_RSYNC os.O_SYNC os.O_NDELAY os.O_NONBLOCK os.O_NOCTTY os.O_SHLOCK os.O_EXLOCK os.O_CLOEXEC These constants are only available on Unix. But os.O_SHLOCK and os.O_EXLOCK are not available on my system (Linux 3.16.7-ckt1 x86_64): sworddragon@ubuntu:~$ python3 Python 3.4.2 (default, Dec 4 2014, 09:34:20) [GCC 4.9.2] on linux Type help, copyright, credits or license for more information. import os os.O_SHLOCK Traceback (most recent call last): File stdin, line 1, in module AttributeError: 'module' object has no attribute 'O_SHLOCK' os.O_EXLOCK Traceback (most recent call last): File stdin, line 1, in module AttributeError: 'module' object has no attribute 'O_EXLOCK' exit() -- messages: 233048 nosy: Sworddragon priority: normal severity: normal status: open title: os.O_SHLOCK and os.O_EXLOCK are not available on Linux versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23105 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23105] os.O_SHLOCK and os.O_EXLOCK are not available on Linux
Changes by Sworddragon sworddrag...@aol.com: -- type: - behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23105 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23105] os.O_SHLOCK and os.O_EXLOCK are not available on Linux
Changes by Sworddragon sworddrag...@aol.com: -- components: +Library (Lib) ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23105 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Serhiy Storchaka added the comment: IP address instances already are immutable and flyweight. There are no mutating methods. And a number of address attributes (packed, is_loopback, etc) are calculated on fly from integer representation. But IP address objects can be lighter. 1) Use __slots__. 2) Every instance has the _version attribute. Why this is not class attribute? 3) Memory consumption can be even less if IP addresses would int subclasses. But this changes the API (in particular adds the __index__ method) and I doubt that we should do this. -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Changes by R. David Murray rdmur...@bitdance.com: -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23105] os.O_SHLOCK and os.O_EXLOCK are not available on Linux
R. David Murray added the comment: The documentation says Some of them are not available on all platforms. For descriptions of their availability and use, consult the open(2) manual page on Unix or the MSDN on Windows.. The important bit there being consult the open(2) manual page on Unix for availability. That said, since the options are grouped and labeled with per-platform availability...but with a slightly different meaning of the word platform...I think the docs could be clarified. Do you have a suggestion for rewording that sentence to make it clearer? -- assignee: - docs@python components: +Documentation -Library (Lib) nosy: +docs@python, r.david.murray versions: +Python 2.7, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23105 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23105] os.O_SHLOCK and os.O_EXLOCK are not available on Linux
Sworddragon added the comment: I have missed the first part of the documentation and am not sure if something needs really to be changed. But if you think so maybe comments like These constants are only available on Unix. could be extended by the word commonly like These constants are commonly only available on Unix.. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23105 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Seth Bromberger added the comment:
1) However:
b = ipaddress.IPv4Address('1.2.3.4')
a = ipaddress.IPv4Address('1.2.3.4')
id(a)
4428380928
id(b)
4428381768
a == b
True
b._ip += 6
id(b)
4428381768
b
IPv4Address('1.2.3.10')
2) Isn’t _version already a class attribute? It’s set in _Basev4/Basev6.
3) Memory consumption seems particularly wasteful in the current
implementation: it takes 56 bytes (via sys.getsizeof) to store what amounts to
4 bytes of data.
I thought about subclassing Int as well, and I agree it’s the wrong approach.
There are too many int methods that don’t make sense in the context of IP
addresses.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23103
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23043] doctest ignores from __future__ import print_function
Julien Palard added the comment: Works for me in 2.7.8: $ python --version Python 2.7.8 # cat /tmp/test.py #!/usr/bin/env python from __future__ import print_function def toto(): print (42, 43) 42 43 return 42 $ python -m doctest -v /tmp/test.py Trying: print (42, 43) Expecting: 42 43 ok 1 items had no tests: test 1 items passed all tests: 1 tests in test.toto 1 tests in 2 items. 1 passed and 0 failed. Test passed. -- nosy: +Julien.Palard ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23043 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23043] doctest ignores from __future__ import print_function
Demian Brecht added the comment: @Julien.Palard: There's a subtle difference between your test and the issue as written. Your test lives within a module and therefore executes testmodule (see https://hg.python.org/cpython/file/9f60d024e586/Lib/doctest.py#l1819) whereas the issue reported uses testfile (see https://hg.python.org/cpython/file/9f60d024e586/Lib/doctest.py#l1923). I believe the issue is that the __future__ import doesn't make it into compile (https://hg.python.org/cpython/file/9f60d024e586/Lib/doctest.py#l1314). I've been able to confirm the issue on 2.7 and that it's been resolved in 3.5. Unfortunately, I haven't had time to dig into this any further. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23043 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23043] doctest ignores from __future__ import print_function
Demian Brecht added the comment: it's been resolved in 3.5 Sorry, that statement can be a little misleading, possibly indicating that something may have changed in the doctest globals handling. It was resolved in 3.5 because print is no longer a statement so this ambiguous behaviour resolved by the print_function import no longer exists. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23043 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
R. David Murray added the comment: If ipaddress objects are already immutable, it seems to me that having a registry that makes it so there is only one instance per ip value isn't a crazy idea. I've written such code in other contexts and it's not complicated. We have caching of identifier strings and common integer constants for a reason...this seems a logical extension of that policy in the realm of a specific library data type. The opposite argument is that it may be better left up to the application that has to handle lots of ips to do the caching according to what it knows to be an optimum pattern. So consider me +0 on the idea. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Seth Bromberger added the comment: The opposite argument is that it may be better left up to the application that has to handle lots of ips to do the caching according to what it knows to be an optimum pattern. I'd agree with you wholeheartedly if ipaddress wasn't part of stdlib, but the issue is that people generally gravitate to using stdlib over other packages when given the choice, and having something that behaves reasonably* in stdlib makes sense wherever possible. *in this case, reasonably means, I think, I shouldn't have to worry that there's 1300% overhead for using IPv4Address() over a native 32-bit representation of an IP address. This matters little when you're just creating a few of these, but best practices regarding reuse might persuade folks to use these classes for all sorts of things. Doing things sanely at scale by default doesn't necessarily preclude further optimization if some developer thinks it's warranted. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22350] nntplib file write failure causes exception from QUIT command
Changes by Jakub Wilk jw...@jwilk.net: -- nosy: +jwilk ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22350 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22669] Test_venv fails when _ctypes is not available.
Terry J. Reedy added the comment: test_venv passed on Open Indiana in the 14:10-14:22 buildbot runs triggered by the patch. It failed with a different error on a later default run, but that looked like a different issue, if permanent. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22669 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22669] Test_venv fails when _ctypes is not available.
Donald Stufft added the comment: I'm not sure how to get a link to that, can you link it and I can see if I think it's a permanent error or not? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22669 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22669] Test_venv fails when _ctypes is not available.
Terry J. Reedy added the comment: http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/9012/steps/test/logs/stdio Start with buildbot.python.org, select set of (all/stable) buildbots for version. 'Build ' like gives commit(s) that triggered build. Test stdio link, as above, gives test results. Above was triggered by Serhiy's commit after yours. No module named pip.__main__; -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22669 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Raymond Hettinger added the comment: Adding __slots__ is reasonable. Adding caching to fold together repeated addresses is dubious (it comes with its own space and time cost but won't benefit all users) and, as David said, the decision on how to handle this is up to the user. Also, it is rather late in the game to be challenging the design of the module. IIRC, this code or some variant of it already had heavy use prior to landing in the standard library (it also took into account what was learned with various other ipaddr modules). The module docs say that the library was designed for speed and makes no mention of optimizing for space. For the most part, once an API is deployed, users can be expected to exercise every feature whether intended or not. Right now, you can add attributes to two otherwise indistinct ipaddresses. The OP proposes to break that code. Right now, if a user has concerns about memory use due to duplicate address instances, they can easily created their own interning scheme (a = intern_dict.setdefault(a, a)). If the OP get his wish, then those users will be worse off (due to double interning). In other words, this ship has already sailed. -- nosy: +rhettinger ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Josh Rosenberg added the comment: So, just to be clear, checking the implementation (as of 3.4): 1. All the ipaddress classes leave __slots__ unset. So the overhead is actually higher than 56 bytes per instance; the __dict__ for an IPv4Address (ignoring the actual keys and values in the dict, just looking at the size of the dict itself), on Python 3.4, 64 bit Windows, is 56 + 288 = 344 bytes. Based on the size of the dict, I'm guessing some instance attributes aren't being set eagerly in __init__, so it's not getting the reduced memory usage from shared key dictionaries either. 2. It looks like as of 3.4, _version and _max_prefixlen were both instance attributes; the current code base seems to have made _max_prefixlen a class attribute, but left _version as an instance attribute (anything set on self is an instance attribute; doesn't matter if it's set in __init__ of a base class) 3. Even if we switch to using __slots__, remove support for weak references and user added attributes, and store nothing but the raw address encoded as an int, you're never going to get true flyweight objects in CPython. The lowest instance cost you can get for a class defined in Python (on a system with 64 bit pointers) inheriting from object, is 40 bytes, plus 8 bytes per slot (plus the actual cost of the int object, which is another 16 bytes for an IPv4 address on a 64 bit OS). If you want it lighter-weight than that, either you need to inherit from another built-in that it even lighter-weight, or you need to implement it in C. If you built IPv4Address on top of int for instance, you could get the instance cost down to 16 bytes. Downside to inheriting from int is that you'll interact with many other objects as if you were an int, which can cause a lot of unexpected behaviors (as others have noted). Basically, there is no concept of flyweight object in Python. Aside from implementing it in C or inheriting from int, you're going to be using an absolute minimum (on 64 bit build) of 48 bytes for the basic object structure, plus another 16 for the int itself, 64 bytes total, or about 16x the real data being stored. Using a WeakValueDictionary would actually require another 8 bytes per instance (a slot for __weakref__), and the overhead of the dictionary would likely be higher than the memory saved (unless you're regularly creating duplicate IP addresses and storing them for the long haul, but I suspect this is a less common use case than processing many different IP addresses once or twice). I do think it would be a good idea to use __slots__ properly to get the memory down below 100 bytes per instance, but adding automatic IP address interning is probably not worth the effort. In the longer term, a C implementation of ipaddress might be worth it, not for improvements in computational performance, but to get the memory usage down for applications that need to make millions of instances. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22952] multiprocessing doc introduction not in affirmative tone
Raymond Hettinger added the comment: Procedural question ... is a fix for this considered an enhancement? Improvements to documentation are almost always backported. The contrasts with code changes where enhancements (new features, optimizations, or refactorings) are made only on the latest development branch to reduce the risk of destabilizing code that has already shipped. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22952 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Marc-Andre Lemburg added the comment: Josh, you might want to have a look at the lightning talk on this page and the associated slides... http://www.egenix.com/library/presentations/PyCon-UK-2014-When-performance-matters/ After having done the tests, using __slots__ is what I consider the flyweight pattern in Python. -- nosy: +lemburg ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23061] Update pep8 to specify explicitly 'module level' imports at top of file
Raymond Hettinger added the comment: I would argue this is a good time to clean up that code in the standard library. Please leave any clean-ups to the module maintainers. PEP 8 is something you should do to your own code, rather than something you inflict on other people's code. We generally leave clean-ups to the module maintainers rather than creating code churn which can make it more difficult to apply actual bug fixes across versions. Also, GvR has long expressed a preference for holistic refactoring where we avoid making superficial changes across the library and reserve those changes for occasions where we're thinking more deeply about a specific piece of code. The basis for this viewpoint has been confirmed a number of times when we've received Pep 8 patches that introduced bugs into code that was previously correct. -- nosy: +rhettinger ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23061 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Seth Bromberger added the comment: As a test, I tried the following (taken mostly from http://codesnipers.com/?q=python-flyweights): class Foo(object): _Foo = weakref.WeakValueDictionary() def __new__(cls, addr): obj = Foo._Foo.get(addr, None) if obj is None: obj = object.__new__(cls) Foo._Foo[addr] = obj obj.addr = addr return obj I created 10 million instances of Foo(34) in an array. Total space taken: ~80 MB. Times: CPU times: user 6.93 s, sys: 48.7 ms, total: 6.98 s Wall time: 6.98 s I then created 10 million instances of a non-flyweight object, assigning an int to an instance variable: class Bar(object): pass Total space taken: ~1.4 GB. Times: CPU times: user 7.64 s, sys: 794 ms, total: 8.44 s Wall time: 8.44 s This corresponds (roughly) to the space taken by 10 million IPAddr objects. So it appears, informally, that caching / flyweight results in modest time and significant memory savings. I understand that the ship has sailed for a stdlib implementation, but these results are compelling enough for me to create a separate package. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Seth Bromberger added the comment: Sorry for the additional followup, but I re-ran the test with approximately real-world parameters. In this second test, I created 10 million Foo() instances with random values from 1-2000 (using random.randint). This corresponds to 2000 hosts generating 10 million logs. Memory use is ~80 MB. Times: CPU times: user 27.5 s, sys: 101 ms, total: 27.6 s Wall time: 27.6 s For the nonoptimized run (10m instances of 2000 random values): Memory use is ~1.8GB. Times: CPU times: user 28.2 s, sys: 1.29 s, total: 29.5 s Wall time: 29.5 s -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23106] Remove smalltable from set objects
New submission from Raymond Hettinger:
This tracker item is here to record my efforts to re-evaluate whether we were
getting much if any benefit from the smalltable in set objects.
Removing the smalltable special case and instead using a memory allocation had
the following effects:
* Nice simplification of the code, greatly improving the clarity of the
functions for resizing, clearing, and swapping.
* Reduced the memory consumption for sets that were already using memory
allocated tables (saved the memory cost of the unused smalltable).
* Nearly doubled the time to allocate and free set objects (see timings below
for CLang and GCC).
As a percentage change, the time penalty seems like a killer, but then we're
talking about only 1/10th of a μsec per set.
# CLANG #
~/base_cp/python.exe -m timeit -s 'from itertools import repeat, starmap'
'list(starmap(set, repeat((), 25)))'
10 loops, best of 3: 39.1 msec per loop
~/cpython/python.exe -m timeit -s 'from itertools import repeat, starmap'
'list(starmap(set, repeat((), 25)))'
10 loops, best of 3: 76.7 msec per loop
~/base_cp/python.exe -m timeit -s 'from itertools import repeat, starmap'
'list(starmap(set, repeat((), 25)))'
10 loops, best of 3: 38.8 msec per loop
~/cpython/python.exe -m timeit -s 'from itertools import repeat, starmap'
'list(starmap(set, repeat((), 25)))'
10 loops, best of 3: 76.6 msec per loop
~/base_cp/python.exe -m timeit '{1,2,3}'
1000 loops, best of 3: 0.0964 usec per loop
~/cpython/python.exe -m timeit '{1,2,3}'
1000 loops, best of 3: 0.148 usec per loop
~/base_cp/python.exe -m timeit '{1,2,3}'
1000 loops, best of 3: 0.0964 usec per loop
~/cpython/python.exe -m timeit '{1,2,3}'
1000 loops, best of 3: 0.149 usec per loop
# GCC-4.9
~/base_cp $ ~/base_cp/python.exe -m timeit '{1,2,3}'
1000 loops, best of 3: 0.0701 usec per loop
~/base_cp $ ~/cpython/python.exe -m timeit '{1,2,3}'
1000 loops, best of 3: 0.155 usec per loop
~/base_cp $ ~/base_cp/python.exe -m timeit '{1,2,3}'
1000 loops, best of 3: 0.0691 usec per loop
~/base_cp $ ~/cpython/python.exe -m timeit '{1,2,3}'
1000 loops, best of 3: 0.157 usec per loop
~/base_cp $ ~/base_cp/python.exe -m timeit -s 'from itertools import repeat,
starmap' 'list(starmap(set, repeat((), 25)))'
10 loops, best of 3: 34.6 msec per loop
~/base_cp $ ~/cpython/python.exe -m timeit -s 'from itertools import repeat,
starmap' 'list(starmap(set, repeat((), 25)))'
10 loops, best of 3: 77 msec per loop
~/base_cp $ ~/base_cp/python.exe -m timeit -s 'from itertools import repeat,
starmap' 'list(starmap(set, repeat((), 25)))'
10 loops, best of 3: 34.1 msec per loop
~/base_cp $ ~/cpython/python.exe -m timeit -s 'from itertools import repeat,
starmap' 'list(starmap(set, repeat((), 25)))'
10 loops, best of 3: 77.3 msec per loop
--
assignee: rhettinger
components: Interpreter Core
files: no_smalltable2.diff
keywords: patch
messages: 233068
nosy: rhettinger
priority: low
severity: normal
status: open
title: Remove smalltable from set objects
type: performance
versions: Python 3.5
Added file: http://bugs.python.org/file37535/no_smalltable2.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23106
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Josh Rosenberg added the comment: Marc-Andre: Oh, I know. I proselytize to coworkers on the virtues of using __slots__ for classes that will have many instances created (particularly since entirely too much of our stuff is still Py2.7, so we don't have the free savings from shared key dictionaries). Heck, I particularly enjoy inheriting from an inlined collections.namedtuple and an empty __slots__(which does add 8 bytes to the size over declaring the __slots__ directly, but gets you as close to truly immutable instances as you can get when you need them, not just we're all adults here immutable instances). I'm just pointing out that if he thinks 56 bytes per object is too large, he's going to be disappointed with what Python has to offer. General purpose user-defined Python objects don't optimize for low memory usage, and even __slots__ only gets you so far. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23103] ipaddress should be Flyweight
Seth Bromberger added the comment: I'm just pointing out that if he thinks 56 bytes per object is too large, he's going to be disappointed with what Python has to offer. General purpose user-defined Python objects don't optimize for low memory usage, and even __slots__ only gets you so far. He thinks that 1300% overhead is a bit too much for a simple object that can be represented by a 32-bit number, and he has been using python for several years and understands, generally, what the language has to offer (though not nearly so well as some of the respondents here). It may be in the roundoff at n 1e5, but when you start using these objects for real-world analysis, the overhead becomes problematic. I note with some amusement that the overhead is several orders of magnitude greater than those of the protocols these objects represent :) In any case - I have no issues with the decision not to make these objects memory-efficient in stdlib and consequently with closing this out. Rolling my own package appears to be the best solution in any case. Thanks for the discussion. -- resolution: - wont fix status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23103 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21279] str.translate documentation incomplete
John Posner added the comment: issue21279.v5.patch tries to apply the comments in msg233013, msg233014, and msg233025 to the Doc/library/stdtypes.rst writeup. Then it applies some of the same language to the docstring in Objects/unicodeobject.c. -- Added file: http://bugs.python.org/file37536/issue21279.v5.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21279 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23089] Update libffi config files
Changes by Gustavo Temple gustavo.pedr...@eldorado.org.br: -- nosy: +steve.dower ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23089 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23089] Update libffi config files
Changes by Gustavo Temple gustavo.pedr...@eldorado.org.br: -- nosy: +lemburg ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23089 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23089] Update libffi config files
Changes by Gustavo Temple gustavo.pedr...@eldorado.org.br: -- nosy: +meador.inge ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23089 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23089] Update libffi config files
Changes by Gustavo Temple gustavo.pedr...@eldorado.org.br: -- nosy: +tim.golden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23089 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23089] Update libffi config files
Changes by Gustavo Temple gustavo.pedr...@eldorado.org.br: -- nosy: +zach.ware ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23089 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23089] Update libffi config files
Zachary Ware added the comment: Not sure why I was added here, but... I agree with David, we need to see the python-dev discussion to its finish before we make any more changes to any of our private copies of libffi. I hope to get some experimental results from Windows written up and posted soon. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23089 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23107] Tighten-up search loops in sets
New submission from Raymond Hettinger: The lookkey functions currently check for an exact key match in the inner search-loop. Move that test to occur after a matching hash is found rather than testing every entry. This gives a modest speed improvement. --- n = 10,000 --- $ ~/baseline/python.exe -m timeit -s 'from time_tight import s,t,u' 'st' 1000 loops, best of 3: 396 usec per loop $ ~/tight/python.exe -m timeit -s 'from time_tight import s,t,u' 'st' 1000 loops, best of 3: 367 usec per loop $ ~/tight/python.exe -m timeit -s 'from time_tight import s,t,u' 'st' 1000 loops, best of 3: 375 usec per loop $ ~/baseline/python.exe -m timeit -s 'from time_tight import s,t,u' 'st' 1000 loops, best of 3: 389 usec per loop $ $ ~/baseline/python.exe -m timeit -s 'from time_tight import s,t,u' 'su' 1000 loops, best of 3: 656 usec per loop $ ~/tight/python.exe -m timeit -s 'from time_tight import s,t,u' 'su' 1000 loops, best of 3: 657 usec per loop $ ~/baseline/python.exe -m timeit -s 'from time_tight import s,t,u' 'su' 1000 loops, best of 3: 662 usec per loop $ ~/tight/python.exe -m timeit -s 'from time_tight import s,t,u' 'su' 1000 loops, best of 3: 642 usec per loop -- n = 1,000,000 -- $ ~/baseline/python.exe -m timeit -s 'from time_tight import s,t,u' 'st' 10 loops, best of 3: 67 msec per loop $ ~/tight/python.exe -m timeit -s 'from time_tight import s,t,u' 'st' 10 loops, best of 3: 48.2 msec per loop $ ~/baseline/python.exe -m timeit -s 'from time_tight import s,t,u' 'st' 10 loops, best of 3: 59.9 msec per loop $ ~/tight/python.exe -m timeit -s 'from time_tight import s,t,u' 'st' 10 loops, best of 3: 49.1 msec per loop $ ~/baseline/python.exe -m timeit -s 'from time_tight import s,t,u' 'su' 10 loops, best of 3: 173 msec per loop $ ~/tight/python.exe -m timeit -s 'from time_tight import s,t,u' 'su' 10 loops, best of 3: 152 msec per loop $ ~/baseline/python.exe -m timeit -s 'from time_tight import s,t,u' 'su' 10 loops, best of 3: 170 msec per loop $ ~/tight/python.exe -m timeit -s 'from time_tight import s,t,u' 'su' 10 loops, best of 3: 167 msec per loop -- assignee: rhettinger components: Interpreter Core files: tight0.diff keywords: patch messages: 233073 nosy: rhettinger priority: normal severity: normal status: open title: Tighten-up search loops in sets type: performance versions: Python 3.5 Added file: http://bugs.python.org/file37537/tight0.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23107 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23107] Tighten-up search loops in sets
Changes by Raymond Hettinger raymond.hettin...@gmail.com: Added file: http://bugs.python.org/file37538/time_tight.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23107 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
