[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Ehsan Amiri : -- nosy: +esam ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/m

[issue4688] GC optimization: don't track simple tuples and dicts

Antoine Pitrou added the comment: Thanks for the review, by the way! -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsub

[issue4688] GC optimization: don't track simple tuples and dicts

Antoine Pitrou added the comment: I've added tracking statistics for tuples, documentation for gc.is_tracked(), and changed _Py*_Optimize to _Py*_MaybeUntrack on a suggestion by Benjamin. Committed to trunk in r70546. -- resolution: accepted -> fixed status: open -> closed

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Antoine Pitrou : -- assignee: -> pitrou resolution: -> accepted ___ Python tracker ___ ___ Python-bugs-list mailing list U

[issue4688] GC optimization: don't track simple tuples and dicts

Martin v. Löwis added the comment: The patch looks fine to me. Some documentation is missing, still: is_tracked needs to be documented in the regular documentation, and SHOW_TRACK_COUNT should be documented in SpecialBuilds.txt. I'm not sure whether you actually want to integrate SHOW_TRACK_COU

[issue4688] GC optimization: don't track simple tuples and dicts

Antoine Pitrou added the comment: Here is a benchmark ripped from the Computer Language Shootout (http://shootout.alioth.debian.org). Running "binary_trees.py 18" takes the following time: -> without patch: 330s. -> with patch:201s. (40% speedup) -> with GC disabled : 165s. Running

[issue4688] GC optimization: don't track simple tuples and dicts

Antoine Pitrou added the comment: Here is a new patch against trunk. -- Added file: http://bugs.python.org/file13331/containeropts.patch ___ Python tracker ___ __

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Antoine Pitrou : Removed file: http://bugs.python.org/file12397/tuple+dictopts-alt.patch ___ Python tracker ___ ___ Python-bugs-list

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Antoine Pitrou : Removed file: http://bugs.python.org/file12396/tupleopts-alt.patch ___ Python tracker ___ ___ Python-bugs-list mail

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Antoine Pitrou : Removed file: http://bugs.python.org/file12389/dictopts2.patch ___ Python tracker ___ ___ Python-bugs-list mailing

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Antoine Pitrou : Removed file: http://bugs.python.org/file12387/tupleopts.patch ___ Python tracker ___ ___ Python-bugs-list mailing

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Collin Winter : -- nosy: +collinwinter ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.py

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue4688] GC optimization: don't track simple tuples and dicts

Antoine Pitrou added the comment: Hello again, > Still, I think this patch does too much code duplication. There should > be only a single function that does the optional untracking; this then > gets called from multiple places. The point was to avoid slowing down the critical path of tuple cr

[issue4688] GC optimization: don't track simple tuples and dicts

Martin v. Löwis added the comment: > But what counts is where tuples can be created in massive numbers or > sizes: the eval loop, the tuple type's tp_new, and a couple of other > places. We don't need to optimize every single tuple created by the > interpreter or extension modules (and even the,

[issue4688] GC optimization: don't track simple tuples and dicts

Antoine Pitrou added the comment: Here is a patch which combines the alternate approach (untrack simple objects when doing a GC collection) for tuples and dicts. I'm surprised by the results: after a full regression test suite, there are roughly 55000 tracked objects (measured with len(gc.get_ob

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Antoine Pitrou : Added file: http://bugs.python.org/file12396/tupleopts-alt.patch ___ Python tracker ___ ___ Python-bugs-list mailin

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Antoine Pitrou : Removed file: http://bugs.python.org/file12395/tupleopts-alt.patch ___ Python tracker ___ ___ Python-bugs-list mail

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Antoine Pitrou : Added file: http://bugs.python.org/file12395/tupleopts-alt.patch ___ Python tracker ___ ___ Python-bugs-list mailin

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Antoine Pitrou : Removed file: http://bugs.python.org/file12394/dictopts2.patch ___ Python tracker ___ ___ Python-bugs-list mailing

[issue4688] GC optimization: don't track simple tuples and dicts

Antoine Pitrou added the comment: Here is an alternate patch which does the tuple optimization in the GC. I don't like it a lot, first because it puts some type-specific code in gcmodule, second because it invalidates the dict optimization (since tuples are untracked only when walked by the GC,

[issue4688] GC optimization: don't track simple tuples and dicts

Antoine Pitrou added the comment: > > I'd rather see that integrated into collection: e.g. when iterating over > > survivors of a collection, check whether they can be untracked. > > That's what I had tried at first, but it crashes. To be clearer, I tried adding it to the tp_traverse method of

[issue4688] GC optimization: don't track simple tuples and dicts

Antoine Pitrou added the comment: Le jeudi 18 décembre 2008 à 06:12 +, Martin v. Löwis a écrit : > I think this approach is difficult to maintain, and also may miss some > objects. But what counts is where tuples can be created in massive numbers or sizes: the eval loop, the tuple type's tp

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Gregory P. Smith : -- nosy: +gregory.p.smith ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://m

[issue4688] GC optimization: don't track simple tuples and dicts

Martin v. Löwis added the comment: > Unfortunately, the check is O(n), so it can get a little expensive. Not in the typical case, I guess. *If* you have to go through all objects, then likely you end up untracking the object. If you can't untrack, you typically find a tracked object in the cont

[issue4688] GC optimization: don't track simple tuples and dicts

Daniel Stutzbach added the comment: Unfortunately, the check is O(n), so it can get a little expensive. I suppose that's no worse than traversing the tuple to look for a cycle, though. Perhaps it could be done at the same time? Can _PyObject_GC_UNTRACK() be safely called from tp_traverse? __

[issue4688] GC optimization: don't track simple tuples and dicts

Martin v. Löwis added the comment: IIUC, you try to find various places where tuples are created, and check whether they create tuples that don't need tracking. I think this approach is difficult to maintain, and also may miss some objects. I'd rather see that integrated into collection: e.g.

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Antoine Pitrou : Removed file: http://bugs.python.org/file12388/dictopts.patch ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue4688] GC optimization: don't track simple tuples and dicts

Antoine Pitrou added the comment: So, I've tried without the dedicated flag in the dict object and it's as fast as previously! Here is a new patch. Added file: http://bugs.python.org/file12389/dictopts2.patch ___ Python tracker

[issue4688] GC optimization: don't track simple tuples and dicts

Changes by Raymond Hettinger : -- nosy: +rhettinger ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.

[issue4688] GC optimization: don't track simple tuples and dicts

Antoine Pitrou added the comment: Le mercredi 17 décembre 2008 à 23:56 +, Daniel Stutzbach a écrit : > For tuples, I think the rule is: > If an object is immutable, and it points only to untracked objects, > the object can be untracked. Roughly, yes. Exactly, it is "if it points only to

[issue4688] GC optimization: don't track simple tuples and dicts

Daniel Stutzbach added the comment: I've looked very quickly over your patches to try to figure what rule qualifies a tuple or dict as simple enough to be untracked, out of curiosity. For tuples, I think the rule is: If an object is immutable, and it points only to untracked objects, the ob

[issue4688] GC optimization: don't track simple tuples and dicts

Antoine Pitrou added the comment: This additional patch also optimizes simple dicts, it must be applied over the tuples patch. Together these two patches make it so that e.g. a dict of tuples or strings won't be tracked at all, saving CPU cycles when collecting. -- nosy: +loewis Added

[issue4688] GC optimization: don't track simple tuples and dicts

New submission from Antoine Pitrou : Split out from #4074, here is a standalone patch which disables GC tracking for simple tuples (tuples made of atomic objects or immutable containers, possibly nested). The performance improvement can be evaluated using tuple_gc_hell.py from #4074. The patch a