[issue39143] Implementing sub-generation steps in the gc

Pablo Galindo Salgado added the comment: > I'm afraid nailing anything here is hard. For example, I don't know what you > did to "measure memory", but if you're using stats reported by the OS, that's > fraught with dangers too. I am interposing a custom allocator to track all the used

[issue39143] Implementing sub-generation steps in the gc

Tim Peters added the comment: All right! So, at a first look, "buffering" isn't an obvious disaster ;-) I'm afraid nailing anything here is hard. For example, I don't know what you did to "measure memory", but if you're using stats reported by the OS, that's fraught with dangers too.

[issue39143] Implementing sub-generation steps in the gc

Pablo Galindo Salgado added the comment: I have attached two files to the issue. One of them (vanilla-vs-buffered) shows the mean memory profile of running test_asyncio 1000 times with the buffered GC and without the buffered GC. The other file shows the difference between both curves.

[issue39143] Implementing sub-generation steps in the gc

Change by Pablo Galindo Salgado : Added file: https://bugs.python.org/file48807/vanilla-vs-buffered.png ___ Python tracker ___ ___

[issue39143] Implementing sub-generation steps in the gc

Change by Pablo Galindo Salgado : Added file: https://bugs.python.org/file48808/diff-vanilla-buffered.png ___ Python tracker ___ ___

[issue39143] Implementing sub-generation steps in the gc

Pablo Galindo Salgado added the comment: > Oh, I don't expect it to help appreciably - which is why I never did it ;-) > It's aimed at something different, I think, than what you're after: reducing > the burden of cyclic gc on objects that would otherwise soon be reclaimed by >

[issue39143] Implementing sub-generation steps in the gc

Tim Peters added the comment: Oh, I don't expect it to help appreciably - which is why I never did it ;-) It's aimed at something different, I think, than what you're after: reducing the burden of cyclic gc on objects that would otherwise soon be reclaimed by refcounting anyway. But such

[issue39143] Implementing sub-generation steps in the gc

Pablo Galindo Salgado added the comment: > For example, break the youngest generation into parts A & B. Newly tracked > items are added to part A. When a collection occurs, only part B > participates from the youngest generation. When the collection ends, part A > is renamed to part B,

[issue39143] Implementing sub-generation steps in the gc

Joannah Nanjekye added the comment: > About "5 bits", no, we don't have 'em. Even the relatively modest > ugliness > we have now makes it impossible to port Python to a word->addressed machine > (I'm not sure any still exist!). Nothing in C >guarantees the last 2 or 3 > bits of a pointer

[issue39143] Implementing sub-generation steps in the gc

Joannah Nanjekye added the comment: > I suppose we would need to experiment, but for what I have seen I think two or three :) What do you think? IMO, I think experimenting with two steps is good enough. -- ___ Python tracker

[issue39143] Implementing sub-generation steps in the gc

Tim Peters added the comment: Long ago I thought about adding "a buffer in front of" CPython's cyclic gc: newly tracked items would be added there, and not participate in cyclic gc at all until a collection passed. For example, break the youngest generation into parts A & B. Newly tracked

[issue39143] Implementing sub-generation steps in the gc

Pablo Galindo Salgado added the comment: > How many semi-spaces do you think of having in this generation? Most systems > have and recommend two. I suppose we would need to experiment, but for what I have seen I think two or three :) What do you think? > If it is a worse tradeoff to

[issue39143] Implementing sub-generation steps in the gc

Joannah Nanjekye added the comment: >5 bit *per object* is a lot because it scales with the number of objects. It >will quickly obliterate >any gain that we get from some objects being >deallocated sooner (which is what we are trying >to achieve). Using >inter-generations has a constant

[issue39143] Implementing sub-generation steps in the gc

Pablo Galindo Salgado added the comment: > I think we only need about 5 bits to store the age of each object , do we > really need to increase the object back to two word per object ? 5 bit *per object* is a lot because it scales with the number of objects. It will quickly obliterate any

[issue39143] Implementing sub-generation steps in the gc

Joannah Nanjekye added the comment: > What threshold is this? > This is the different thresholds for the generations that you can get using > gc.get_threshold(). >They are in relationship to the number of objects in > every generation (there are slightly different >rules for the latest >

[issue39143] Implementing sub-generation steps in the gc

Pablo Galindo Salgado added the comment: > What threshold is this? This is the different thresholds for the generations that you can get using gc.get_threshold(). They are in relationship to the number of objects in every generation (there are slightly different rules for the latest

[issue39143] Implementing sub-generation steps in the gc

Joannah Nanjekye added the comment: > The most common reason is that when promotions of the youngest generations > happen, >some very young objects that just arrived in the generation are >promoted >because we have >reached a threshold, and its death will be >delayed. What threshold is

[issue39143] Implementing sub-generation steps in the gc

New submission from Pablo Galindo Salgado : While I was re-reading The Garbage Collection Handbook [Moss, Hosking, Jones], I have been doing some benchmarks of different aspects of our garbage collection and taking statistics using the pyperformance suite as long as several "production"