Re: [Tutor] python memory management
Thank you all for your explanations. I really enjoy learning about things like that. Monika -- Original Message -- From: Alan Gauld via Tutor To: zakaria , tutor@python.org Subject: Re: [Tutor] python memory management Date: Sun, 4 Sep 2016 01:22:27 +0100 On 03/09/16 23:20, zakaria wrote: > Is there any practical usage of using reference cycling? There are a (very) few cases where data structures require the creation of cyclic references. One example I've used is in managing comms networks where nodes are multiply and/or cyclically linked and you need to build those linkages into your data model. In those cases you need to manage the cleanup yourself. But most cyclic references occur by accident, the programmer probably didn't intend them to exist but it just so happened that one object points to another which happens to point at another which in turn points at the first. The cycle can span many objects. Think of a directory tree where a deeply nested node has a link pointing back at a root level node. But the root also indirectly points at that low level node by means of the tree structure... As a specific example, It often happens in GUIs where widgets hold references to other widgets higher in the widget parent/child tree. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor Unlock Visions (Sponsored by Content.Ad) 1 Odd Method 'Restores' Your 20/20 Vision. Try This http://thirdpartyoffers.netzero.net/TGL3241/57cb93756ab9c13757ae2st02duc ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python memory management
On 03/09/16 23:20, zakaria wrote: > Is there any practical usage of using reference cycling? There are a (very) few cases where data structures require the creation of cyclic references. One example I've used is in managing comms networks where nodes are multiply and/or cyclically linked and you need to build those linkages into your data model. In those cases you need to manage the cleanup yourself. But most cyclic references occur by accident, the programmer probably didn't intend them to exist but it just so happened that one object points to another which happens to point at another which in turn points at the first. The cycle can span many objects. Think of a directory tree where a deeply nested node has a link pointing back at a root level node. But the root also indirectly points at that low level node by means of the tree structure... As a specific example, It often happens in GUIs where widgets hold references to other widgets higher in the widget parent/child tree. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python memory management
On 03/09/16 22:16, monik...@netzero.net wrote: > So what does [...] mean? Its Python's way of telling you that you have a self referential data structure. Its just a representational thing but without it Python would end up printing an infinite sequence of values. HTH -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python memory management
Is there any practical usage of using reference cycling? On Sat, 2016-09-03 at 14:56 +0100, Alan Gauld via Tutor wrote: > On 03/09/16 04:25, monik...@netzero.net wrote: > > > > > Is this what you mean? > > a = 5 > > b = a > > a = b > > No, you are confusing variable names with objects. > Here you only have one object - the number 5. > For a cycle you need at least 2 objects and those > objects must be able to reference another object. > In practice that means a collection or an instance > of a class. > > a = [] > b = [a] > a.append(b) > > The two lists are now cyclically referring to each > other. We can now delete the names a and b and > the two list objects will continue to exist > in memory even though no variables refer to them. > This is where the second garbage collector comes > into play, it can recognise the link between the > lists and the fact that no variable refers to them. > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python memory management
So what does [...] mean?
-- Original Message --
From: Peter Otten <__pete...@web.de>
To: tutor@python.org
Subject: Re: [Tutor] python memory management
Date: Sat, 03 Sep 2016 14:26:12 +0200
monik...@netzero.net wrote:
>
> By:
> "reference cycles: if one object has a reference to another, and
> that second object also has a reference to the first, that's a cycle."
>
> Is this what you mean?
> a = 5
> b = a
> a = b
No. int instances are immutable. The assignments above bind both /names/ a
and b to the same instance of int -- 5. The last statement a = b is
redundant as a is already bound to 5.
For a reference cycle you need something in an object x to refer to another
y and vice versa. x and y here are not Python names, but the actual objects.
An example using lists:
>>> a = ["a"]
>>> b = ["b"]
Let's append b to a:
>>> a.append(b)
>>> a[1]
['b']
>>> a[1][1]
Traceback (most recent call last):
File "", line 1, in
IndexError: list index out of range
We don't have a cycle yet, so Python eventually complains.
Now let's complete the cycle by appending b to a.
>>> b.append(a)
>>> a[1][1]
['a', ['b', [...]]]
>>> a[1][1][1][1][1]
['b', ['a', [...]]]
We can keep accessing x[1] forever, and if Python weren't smart enough to
detect the cycle instead of producing the [...] it would continue building
the string representation for the nested lists until all available memory
was consumed or the stack overflowed.
Another example with a custom class:
>>> class C:
...def __init__(self, name): self.name = name
...def __repr__(self): return self.name
...
>>> a = C("a")
>>> b = C("b")
>>> a
a
>>> b
b
>>> a.b = b
>>> b.a = a
>>> a.b.a.b.a.b.a.b.a.b.a
a
A cycle may of course consist of more than two objects:
>>> a = C("a")
>>> b = C("b")
>>> c = C("c")
>>> a.b = b
>>> b.c = c
>>> c.a = a
>>> c.a.b.c
c
>>> c.a.b.c.a.b.c.a.b.c
c
The minimal cycle consists of an object referencing itself:
>>> a = C("a")
>>> a.a = a
>>> a.a.a.a.a.a.a
a
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
howlifeworks.com (Sponsored by Content.Ad)
Why Women Are Flocking to This Incredible New Shopping Site
http://thirdpartyoffers.netzero.net/TGL3241/57cb3e00e29ee3e000c94st01duc
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python memory management
On Thu, Sep 01, 2016 at 08:21:36PM +, monik...@netzero.net wrote: > Thank you for your explanation. It is very clear and confirms what I > thought I knew. However, I had a job interview and the interview said > it was a mistake that I did not say that in cases when there are > multiple programmers, there might be some objects/references left and > not deallocated from the memory by python. He asked me how this should > be handles. Without being there, I cannot be sure precisely what he meant. But frankly it sounds like the interviewer doesn't know what he is talking about. The number of programmers who work on a project has nothing to do with how it manages memory. Whether you have one person who writes the code, or a hundred people writing the code, memory is managed exactly the same way. Perhaps you misunderstood the question, or perhaps the interviewer simply wasn't as knowledgable or smart as he thought he was. > So I told him that python has its automatic garbage collection which > keeps track of all objects and references and deletes them as > appropriate (similar what you state). I added that programmers can > keep track of occurrences of object and make sure that it goes down to > 0. And if it does not then use del to delete. However, he did not like > my answer. What you say is technically correct, but if you find yourself using del more than very occasionally, I suggest that you need to think hard about what you are doing. It is not normally necessary to use del. > So Im trying to learn from my mistakes and learn if an > object or reference are still at the end of the program how they > should be deleted. A thought comes to mind... When Python shuts down, it has to deallocate all your remaining objects, giving them a chance to run any finaliser methods. But sometimes the finaliser methods don't get to run until Python has already deallocated the global variables, including modules you might want to use. So for example: import math class Widget: def __del__(self): # Finaliser print(math.sin(10)) but if your widget doesn't get deallocated until the end of your application, it may be that math has been set to None and math.sin is no longer available. Perhaps he was asking about that? What sort of job were you asking for? I expect that would be counted as an extremely advanced corner of Python, not something that most people know or need to know. > I had problem understanding him too well since he > was calling from overseas (there was interference) and I could not > understand his accent well. But I need to learn this for future > interviews. > > Your posting states that this is not a problem but according to the > interviewer it is. (I do not mean to be un-nice to you, sorry) So how > this situation should be handled? There's no way to know the right way. Some interviewers think they know more than they actually do. When you tell them the right answer, they think it is wrong because they are ignorant. Some interviewers are deliberately testing you with really obscure, complex corners of the language. Maybe he was right, and there is some tiny corner where what I told you was incorrect. Maybe you misunderstood the question. Maybe the interviewer was deliberately testing you with false information, to see what you would do. Would you argue with him? Get frustrated? Agree with him even if you knew the answer was actually wrong? There's no way of knowing what they are looking for. Do they want people who will push back and stand their ground when they know they are right, or people who will be good little employees who will agree when the boss says that milk is purple and England is the capital of China? I guess the only thing you can do is answer as best you can, ask for clarifications, and hope. -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python memory management
On 03/09/16 04:25, monik...@netzero.net wrote: > Is this what you mean? > a = 5 > b = a > a = b No, you are confusing variable names with objects. Here you only have one object - the number 5. For a cycle you need at least 2 objects and those objects must be able to reference another object. In practice that means a collection or an instance of a class. a = [] b = [a] a.append(b) The two lists are now cyclically referring to each other. We can now delete the names a and b and the two list objects will continue to exist in memory even though no variables refer to them. This is where the second garbage collector comes into play, it can recognise the link between the lists and the fact that no variable refers to them. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python memory management
On Fri, Sep 2, 2016, at 23:25, monik...@netzero.net wrote: > > By: > "reference cycles: if one object has a reference to another, and > that second object also has a reference to the first, that's a cycle." > > Is this what you mean? > a = 5 > b = a > a = b > > I just want to make sure I understand. > Thank you very much for your explanation. > Monika No, an object that holds reference to other objects is something like a list, tuple, dict. e.g. if you had: a = [] b = [a] a.append(b) you've got two lists, and each one has a reference to the other one. These exist independent of the variables - you could del b and it would still exist in a[0]. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python memory management
monik...@netzero.net wrote:
>
> By:
> "reference cycles: if one object has a reference to another, and
> that second object also has a reference to the first, that's a cycle."
>
> Is this what you mean?
> a = 5
> b = a
> a = b
No. int instances are immutable. The assignments above bind both /names/ a
and b to the same instance of int -- 5. The last statement a = b is
redundant as a is already bound to 5.
For a reference cycle you need something in an object x to refer to another
y and vice versa. x and y here are not Python names, but the actual objects.
An example using lists:
>>> a = ["a"]
>>> b = ["b"]
Let's append b to a:
>>> a.append(b)
>>> a[1]
['b']
>>> a[1][1]
Traceback (most recent call last):
File "", line 1, in
IndexError: list index out of range
We don't have a cycle yet, so Python eventually complains.
Now let's complete the cycle by appending b to a.
>>> b.append(a)
>>> a[1][1]
['a', ['b', [...]]]
>>> a[1][1][1][1][1]
['b', ['a', [...]]]
We can keep accessing x[1] forever, and if Python weren't smart enough to
detect the cycle instead of producing the [...] it would continue building
the string representation for the nested lists until all available memory
was consumed or the stack overflowed.
Another example with a custom class:
>>> class C:
...def __init__(self, name): self.name = name
...def __repr__(self): return self.name
...
>>> a = C("a")
>>> b = C("b")
>>> a
a
>>> b
b
>>> a.b = b
>>> b.a = a
>>> a.b.a.b.a.b.a.b.a.b.a
a
A cycle may of course consist of more than two objects:
>>> a = C("a")
>>> b = C("b")
>>> c = C("c")
>>> a.b = b
>>> b.c = c
>>> c.a = a
>>> c.a.b.c
c
>>> c.a.b.c.a.b.c.a.b.c
c
The minimal cycle consists of an object referencing itself:
>>> a = C("a")
>>> a.a = a
>>> a.a.a.a.a.a.a
a
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python memory management
By: "reference cycles: if one object has a reference to another, and that second object also has a reference to the first, that's a cycle." Is this what you mean? a = 5 b = a a = b I just want to make sure I understand. Thank you very much for your explanation. Monika -- Original Message -- From: Steven D'Aprano To: tutor@python.org Subject: Re: [Tutor] python memory management Date: Fri, 2 Sep 2016 04:10:02 +1000 On Thu, Sep 01, 2016 at 02:12:11PM +, monik...@netzero.net wrote: > Hi: > Can somebody please explain how memory is managed by python? What kind > of memory it uses? What structures use what kind of memory? > If many people work on the same project and have many instances of the > same object how do they ensure that all instances are killed before > the programs exit? Apparently if one of the programmer leaves a > reference to object it might not be automatically deleted by python on > exit. What is the command to do this? > > Could somebody please explain how this works, especially on projects > involving multiple programmers? In general, you (almost) never need to care about memory management, Python will do it for you. The number of programmers writing the code doesn't matter. What matters is how many times the program is running *at the same time*. Each time it runs, your computer's operating system (Windows, Linux, Mac OS X) will start what is called "a process", running the Python interpreter. When the process exits at the end, the OS will reclaim all the memory used and make it available for the next process. While the program is running, the OS has to allocate memory between many different processes. On my computer, right now, I have over 200 processes running. Most of them are handled by the OS, but the others include my email program, my web browser, a few text editors, my desktop manager, and many others. The OS manages the memory allocation. As far as Python is concerned, it manages its own memory from what the OS gives it. When you assign a value: name = "Inigo Montoya" the Python interpreter allocates a chunk of memory in the memory heap to hold the string. It then tracks whether or not the string is being used. So long as the string is being used by your program, or *could possibly* be used, Python will hold onto that string, forever. But as soon as it sees that it can no longer be used, it will free the memory and reuse it. This process is called "garbage collection". You can google for more information, or ask here. Different Python interpreters use different garbage collectors: IronPython uses the .Net garbage collector; Jython uses the Java garbage collector; PyPy has a few different ones that you can choose from; and the CPython (that's the standard Python you are probably running) interpreter has two, a simple "reference counter" GC that works very fast but not very thoroughly, and a more thorough GC that picks up anything the reference counter can't handle. (Mostly reference cycles: if one object has a reference to another, and that second object also has a reference to the first, that's a cycle. The reference counter can't deal with that, but the second GC can.) Let's track the life-span of a chunk of memory. Suppose you write the following code in a module: name = "Inigo Montoya" print(name) name = "The Dread Pirate Roberts" The second assignment frees up the string "Inigo Montoya", as no part of your program can possibly access the old value any more, since it has been replaced by the new value. So the garbage collector frees that chunk of memory and makes it available for something else. This happens automatically, and virtually instantly. You never need to care about allocating or deallocating memory. The interpreter has its own memory manager to do that, with a garbage collector to deallocate memory. So, when do you need to care about memory? - If you are writing a C extension, you have to manage your own memory. - If you're using the ctypes module, you have access to the C code of the interpreter, so you have to care about managing your own memory. - If you're creating massively huge strings or lists, you might have to worry about running out of memory. For example, I once locked up my computer by foolishly creating a HUGE list: # Don't try this at home! L = list(range(100**100)) That would need to find enough memory for a list with one hundred trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion entries. Each entry will take at least four bytes, so the total is ... well, it's a lot. Much more than my poor computer has. On Windo
Re: [Tutor] python memory management
On Thu, Sep 1, 2016 at 4:21 PM, monik...@netzero.net wrote: > Thank you for your explanation. It is very clear and confirms what I thought > I knew. However, I had a job interview and the interview said it was a > mistake that I did not say that in cases when there are multiple > programmers, there might be some objects/references left and not deallocated > from the memory by python. He asked me how this should be handles. > So I told him that python has its automatic garbage collection which keeps > track of all objects and references and deletes them as appropriate (similar > what you state). I added that programmers can keep track of occurrences of > object and make sure that it goes down to 0. And if it does not then use del > to delete. However, he did not like my answer. So Im trying to learn from my > mistakes and learn if an object or reference are still at the end of the > program how they should be deleted. I had problem understanding him too well > since he was calling from overseas (there was interference) and I could not > understand his accent well. But I need to learn this for future interviews. > Your posting states that this is not a problem but according to the > interviewer it is. (I do not mean to be un-nice to you, sorry) So how this > situation should be handled? > Thank you > Monika I would chalk this up to the interviewer wanting to take the upper hand. If he (she?) thought your understanding was wrong, the appropriate thing to do would be to explain how you were wrong, and take it as a teaching moment. If it happens again, ask how your answer was incorrect. I think this shows interest in learning more, and that can't be a bad thing > > -- Original Message ------ > From: Steven D'Aprano > To: tutor@python.org > Subject: Re: [Tutor] python memory management > Date: Fri, 2 Sep 2016 04:10:02 +1000 > > On Thu, Sep 01, 2016 at 02:12:11PM +, monik...@netzero.net wrote: >> Hi: >> Can somebody please explain how memory is managed by python? What kind >> of memory it uses? What structures use what kind of memory? >> If many people work on the same project and have many instances of the >> same object how do they ensure that all instances are killed before >> the programs exit? Apparently if one of the programmer leaves a >> reference to object it might not be automatically deleted by python on >> exit. What is the command to do this? >> >> Could somebody please explain how this works, especially on projects >> involving multiple programmers? > > > In general, you (almost) never need to care about memory management, > Python will do it for you. > > The number of programmers writing the code doesn't matter. What matters > is how many times the program is running *at the same time*. Each time > it runs, your computer's operating system (Windows, Linux, Mac OS X) > will start what is called "a process", running the Python interpreter. > When the process exits at the end, the OS will reclaim all the memory > used and make it available for the next process. > > While the program is running, the OS has to allocate memory between many > different processes. On my computer, right now, I have over 200 > processes running. Most of them are handled by the OS, but the others > include my email program, my web browser, a few text editors, my desktop > manager, and many others. The OS manages the memory allocation. > > As far as Python is concerned, it manages its own memory from what the > OS gives it. When you assign a value: > > name = "Inigo Montoya" > > the Python interpreter allocates a chunk of memory in the memory heap to > hold the string. It then tracks whether or not the string is being used. > So long as the string is being used by your program, or *could possibly* > be used, Python will hold onto that string, forever. > > But as soon as it sees that it can no longer be used, it will free the > memory and reuse it. > > This process is called "garbage collection". You can google for more > information, or ask here. Different Python interpreters use different > garbage collectors: > > IronPython uses the .Net garbage collector; > > Jython uses the Java garbage collector; > > PyPy has a few different ones that you can choose from; > > and the CPython (that's the standard Python you are probably running) > interpreter has two, a simple "reference counter" GC that works very > fast but not very thoroughly, and a more thorough GC that picks up > anything the reference counter can't handle. > > (Mostly reference cycles: if one object has a reference to another, and > that second object also has a refe
Re: [Tutor] python memory management
Thank you for your explanation. It is very clear and confirms what I thought I knew. However, I had a job interview and the interview said it was a mistake that I did not say that in cases when there are multiple programmers, there might be some objects/references left and not deallocated from the memory by python. He asked me how this should be handles. So I told him that python has its automatic garbage collection which keeps track of all objects and references and deletes them as appropriate (similar what you state). I added that programmers can keep track of occurrences of object and make sure that it goes down to 0. And if it does not then use del to delete. However, he did not like my answer. So Im trying to learn from my mistakes and learn if an object or reference are still at the end of the program how they should be deleted. I had problem understanding him too well since he was calling from overseas (there was interference) and I could not understand his accent well. But I need to learn this for future interviews. Your posting states that this is not a problem but according to the interviewer it is. (I do not mean to be un-nice to you, sorry) So how this situation should be handled? Thank you Monika -- Original Message -- From: Steven D'Aprano To: tutor@python.org Subject: Re: [Tutor] python memory management Date: Fri, 2 Sep 2016 04:10:02 +1000 On Thu, Sep 01, 2016 at 02:12:11PM +, monik...@netzero.net wrote: > Hi: > Can somebody please explain how memory is managed by python? What kind > of memory it uses? What structures use what kind of memory? > If many people work on the same project and have many instances of the > same object how do they ensure that all instances are killed before > the programs exit? Apparently if one of the programmer leaves a > reference to object it might not be automatically deleted by python on > exit. What is the command to do this? > > Could somebody please explain how this works, especially on projects > involving multiple programmers? In general, you (almost) never need to care about memory management, Python will do it for you. The number of programmers writing the code doesn't matter. What matters is how many times the program is running *at the same time*. Each time it runs, your computer's operating system (Windows, Linux, Mac OS X) will start what is called "a process", running the Python interpreter. When the process exits at the end, the OS will reclaim all the memory used and make it available for the next process. While the program is running, the OS has to allocate memory between many different processes. On my computer, right now, I have over 200 processes running. Most of them are handled by the OS, but the others include my email program, my web browser, a few text editors, my desktop manager, and many others. The OS manages the memory allocation. As far as Python is concerned, it manages its own memory from what the OS gives it. When you assign a value: name = "Inigo Montoya" the Python interpreter allocates a chunk of memory in the memory heap to hold the string. It then tracks whether or not the string is being used. So long as the string is being used by your program, or *could possibly* be used, Python will hold onto that string, forever. But as soon as it sees that it can no longer be used, it will free the memory and reuse it. This process is called "garbage collection". You can google for more information, or ask here. Different Python interpreters use different garbage collectors: IronPython uses the .Net garbage collector; Jython uses the Java garbage collector; PyPy has a few different ones that you can choose from; and the CPython (that's the standard Python you are probably running) interpreter has two, a simple "reference counter" GC that works very fast but not very thoroughly, and a more thorough GC that picks up anything the reference counter can't handle. (Mostly reference cycles: if one object has a reference to another, and that second object also has a reference to the first, that's a cycle. The reference counter can't deal with that, but the second GC can.) Let's track the life-span of a chunk of memory. Suppose you write the following code in a module: name = "Inigo Montoya" print(name) name = "The Dread Pirate Roberts" The second assignment frees up the string "Inigo Montoya", as no part of your program can possibly access the old value any more, since it has been replaced by the new value. So the garbage collector frees that chunk of memory and makes it available for something else. This happens automatically, and virtually instantly. You never need to care about allocating or deallocating memory. The interpreter has its own memory manager to do that, with a garbage collector to deall
Re: [Tutor] python memory management
On Thu, Sep 01, 2016 at 02:12:11PM +, monik...@netzero.net wrote: > Hi: > Can somebody please explain how memory is managed by python? What kind > of memory it uses? What structures use what kind of memory? > If many people work on the same project and have many instances of the > same object how do they ensure that all instances are killed before > the programs exit? Apparently if one of the programmer leaves a > reference to object it might not be automatically deleted by python on > exit. What is the command to do this? > > Could somebody please explain how this works, especially on projects > involving multiple programmers? In general, you (almost) never need to care about memory management, Python will do it for you. The number of programmers writing the code doesn't matter. What matters is how many times the program is running *at the same time*. Each time it runs, your computer's operating system (Windows, Linux, Mac OS X) will start what is called "a process", running the Python interpreter. When the process exits at the end, the OS will reclaim all the memory used and make it available for the next process. While the program is running, the OS has to allocate memory between many different processes. On my computer, right now, I have over 200 processes running. Most of them are handled by the OS, but the others include my email program, my web browser, a few text editors, my desktop manager, and many others. The OS manages the memory allocation. As far as Python is concerned, it manages its own memory from what the OS gives it. When you assign a value: name = "Inigo Montoya" the Python interpreter allocates a chunk of memory in the memory heap to hold the string. It then tracks whether or not the string is being used. So long as the string is being used by your program, or *could possibly* be used, Python will hold onto that string, forever. But as soon as it sees that it can no longer be used, it will free the memory and reuse it. This process is called "garbage collection". You can google for more information, or ask here. Different Python interpreters use different garbage collectors: IronPython uses the .Net garbage collector; Jython uses the Java garbage collector; PyPy has a few different ones that you can choose from; and the CPython (that's the standard Python you are probably running) interpreter has two, a simple "reference counter" GC that works very fast but not very thoroughly, and a more thorough GC that picks up anything the reference counter can't handle. (Mostly reference cycles: if one object has a reference to another, and that second object also has a reference to the first, that's a cycle. The reference counter can't deal with that, but the second GC can.) Let's track the life-span of a chunk of memory. Suppose you write the following code in a module: name = "Inigo Montoya" print(name) name = "The Dread Pirate Roberts" The second assignment frees up the string "Inigo Montoya", as no part of your program can possibly access the old value any more, since it has been replaced by the new value. So the garbage collector frees that chunk of memory and makes it available for something else. This happens automatically, and virtually instantly. You never need to care about allocating or deallocating memory. The interpreter has its own memory manager to do that, with a garbage collector to deallocate memory. So, when do you need to care about memory? - If you are writing a C extension, you have to manage your own memory. - If you're using the ctypes module, you have access to the C code of the interpreter, so you have to care about managing your own memory. - If you're creating massively huge strings or lists, you might have to worry about running out of memory. For example, I once locked up my computer by foolishly creating a HUGE list: # Don't try this at home! L = list(range(100**100)) That would need to find enough memory for a list with one hundred trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion trillion entries. Each entry will take at least four bytes, so the total is ... well, it's a lot. Much more than my poor computer has. On Windows, this will fail quite quickly with a MemoryError, no real harm done, but on Linux (which I use) the OS will gamely, or perhaps stupidly, try very hard to allocate a trillion trillion trillion ... trillion terabytes of memory, locking up my computer. (I let it run overnight, and eventually needed to just pull the plug to reset it.) Fortunately, even on Linux there are ways to tell the OS not to be so stupid, which means that Python will raise a MemoryError and no real harm is done, but at the time I didn't know about them. So that's about it really... if you're writing long-lasting server class programs, you
Re: [Tutor] python memory management
On 01/09/16 15:12, monik...@netzero.net wrote: > Can somebody please explain how memory is managed by python? > What kind of memory it uses? What structures use what kind of memory? I'm not sure what you have in mind? Do you want to know the internal structure of the various data types? Do you want to know the size of the memory footprint? > If many people work on the same project and have many > instances of the same object how do they ensure that > all instances are killed before the programs exit? Generally each developer runs their own instance of the program and the OS/Python manage the memory for each process entirely separate from all other processes. > Apparently if one of the programmer leaves a reference > to object it might not be automatically deleted > by python on exit. What is the command to do this? When python exits, all objects are deleted regardless of how many references still exist. The issue is more relevant to a server process which never exits. In that case it is possible for a programmer to leave a reference to an obsolete object and prevent it from being disposed of. This is known as a memory leak. But the issue applies just as much to a single programmer as to multiple programmers. It is a design issue that requires that the redundant objects are cleared away correctly with no dangling references. It is one of the many design considerations when building long running code. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] python memory management
Hi: Can somebody please explain how memory is managed by python? What kind of memory it uses? What structures use what kind of memory? If many people work on the same project and have many instances of the same object how do they ensure that all instances are killed before the programs exit? Apparently if one of the programmer leaves a reference to object it might not be automatically deleted by python on exit. What is the command to do this? Could somebody please explain how this works, especially on projects involving multiple programmers? Thank you very much Monika DrAnkurWalia (Sponsored by Content.Ad) Breaking: Delhi Girls Are Getting 5 Shades Fairer (Do This) http://thirdpartyoffers.netzero.net/TGL3241/57c8379d62edd379d57b6st04duc ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
