Re: Garbage collection problem with generators

2016-12-28 Thread Haochuan Guo
Sorry about breaking the rule. I'm just curios about this problem. And I'm using this workaround to prevent redundant resource creation. https://gist.githubusercontent.com/wooparadog/16948ca6c8ffb22214bf491a280406da/raw/- On Wed, Dec 28, 2016 at 9:12 PM Chris Angelico wrote: > On Wed, Dec 28,

Re: Garbage collection problem with generators

2016-12-28 Thread Chris Angelico
On Wed, Dec 28, 2016 at 9:03 PM, Haochuan Guo wrote: > Anyone? The script to reproduce this problem is in: > > https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef > > On Fri, Dec 23, 2016 at 8:39 PM Haochuan Guo wrote: > >> This is reproducible with python2.7, but not in python3.5

Re: Garbage collection problem with generators

2016-12-28 Thread Haochuan Guo
Anyone? The script to reproduce this problem is in: https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef On Fri, Dec 23, 2016 at 8:39 PM Haochuan Guo wrote: > Hi, everyone > > I'm building a http long polling client for our company's discovery > service and something weird happen

Re: garbage collection / reference cycles (cont.)

2009-03-29 Thread Aaron Brady
On Mar 25, 12:12 am, Aaron Brady wrote: > On Mar 25, 12:11 am, Aaron Brady wrote: > > Hello, > > > I am posting the code I mentioned on Saturday that collects garbage > > and cyclic garbage in a flattened two-step process.  The code takes > > 122 lines incl. comments, with 100 in tests.  It shoul

Re: garbage collection / reference cycles (cont.)

2009-03-29 Thread Aaron Brady
On Mar 25, 12:11 am, Aaron Brady wrote: > Hello, > > I am posting the code I mentioned on Saturday that collects garbage > and cyclic garbage in a flattened two-step process.  The code takes > 122 lines incl. comments, with 100 in tests.  It should be in a reply > to this. > > My aim is a buffer-l

Re: garbage collection / reference cycles (cont.)

2009-03-24 Thread Aaron Brady
On Mar 25, 12:11 am, Aaron Brady wrote: > Hello, > > I am posting the code I mentioned on Saturday that collects garbage > and cyclic garbage in a flattened two-step process.  The code takes > 122 lines incl. comments, with 100 in tests.  It should be in a reply > to this. > > My aim is a buffer-l

Re: garbage collection / cyclic references

2009-03-22 Thread Aaron Brady
On Mar 21, 11:59 am, "andrew cooke" wrote: > Aaron Brady wrote: > > My point is, that garbage collection is able to detect when there are > > no program-reachable references to an object.  Why not notify the > > programmer (the programmer's objects) when that happens?  If the > > object does still

Re: garbage collection / cyclic references

2009-03-21 Thread Aaron Brady
On Mar 21, 1:04 pm, John Nagle wrote: > Aaron Brady wrote: > > Hello, > > > I was reading and Googling about garbage collection, reference > > counting, and the problem of cyclic references. > > > Python's garbage collection module claims to be able to detect and > > break cyclic garbage.  Some ot

Re: garbage collection / cyclic references

2009-03-21 Thread John Nagle
Aaron Brady wrote: Hello, I was reading and Googling about garbage collection, reference counting, and the problem of cyclic references. Python's garbage collection module claims to be able to detect and break cyclic garbage. Some other languages merely prohibit it. Is this the place to ask a

Re: garbage collection / cyclic references

2009-03-21 Thread andrew cooke
Aaron Brady wrote: > My point is, that garbage collection is able to detect when there are > no program-reachable references to an object. Why not notify the > programmer (the programmer's objects) when that happens? If the > object does still have other unreachable references, s/he should be > i

Re: garbage collection / cyclic references

2009-03-21 Thread Aaron Brady
On Mar 21, 10:28 am, Aaron Brady wrote: > On Mar 21, 9:50 am, "andrew cooke" wrote: > > > > > Aaron Brady wrote: > > > On Mar 21, 7:54 am, "andrew cooke" wrote: > > >> they should not be used to do things like flushing and closing > > >> files, for example. > > > What is your basis for this clai

Re: garbage collection / cyclic references

2009-03-21 Thread Aaron Brady
On Mar 21, 9:50 am, "andrew cooke" wrote: > Aaron Brady wrote: > > On Mar 21, 7:54 am, "andrew cooke" wrote: > >> they should not be used to do things like flushing and closing > >> files, for example. > > What is your basis for this claim, if it's not the mere unreliability > > of finalization?

Re: garbage collection / cyclic references

2009-03-21 Thread andrew cooke
andrew cooke wrote: > Aaron Brady wrote: >> On Mar 21, 7:54 am, "andrew cooke" wrote: >>> they should not be used to do things like flushing and closing >>> files, for example. >> What is your basis for this claim, if it's not the mere unreliability >> of finalization? IOW, are you not merely beg

Re: garbage collection / cyclic references

2009-03-21 Thread andrew cooke
Aaron Brady wrote: > On Mar 21, 7:54 am, "andrew cooke" wrote: >> they should not be used to do things like flushing and closing >> files, for example. > What is your basis for this claim, if it's not the mere unreliability > of finalization? IOW, are you not merely begging the question? I'm not

Re: garbage collection / cyclic references

2009-03-21 Thread Aaron Brady
On Mar 21, 7:54 am, "andrew cooke" wrote: > Paul Rubin wrote: > > "andrew cooke" writes: > >> the two dominant virtual machines - .net and the jvm both handle > >> circular > >> references with no problem whatever. > > > AFAIK, they also don't guarantee that finalizers ever run, much less > > run

Re: garbage collection / cyclic references

2009-03-21 Thread andrew cooke
Paul Rubin wrote: > "andrew cooke" writes: >> the two dominant virtual machines - .net and the jvm both handle >> circular >> references with no problem whatever. > > AFAIK, they also don't guarantee that finalizers ever run, much less > run in deterministic order. i think you're right, but i'm m

Re: garbage collection / cyclic references

2009-03-21 Thread Martin v. Löwis
> The actual backend of CPython requires garbage-collected container > types to implement tp_inquiry and tp_clear methods, but user-defined > types apparently aren't required to conform. tp_inquiry doesn't exist, you probably mean tp_traverse. tp_traverse is completely irrelevant for python-define

Re: garbage collection / cyclic references

2009-03-21 Thread Aaron Brady
On Mar 20, 8:12 pm, "andrew cooke" wrote: > Aaron Brady wrote: > > [...] > > > caveats and fragilities?  If free software can do it, why isn't it all > > over the industry?  What disqualifies it from solved-problem status? > > the two dominant virtual machines - .net and the jvm both handle circul

Re: garbage collection / cyclic references

2009-03-20 Thread Paul Rubin
"andrew cooke" writes: > the two dominant virtual machines - .net and the jvm both handle circular > references with no problem whatever. AFAIK, they also don't guarantee that finalizers ever run, much less run in deterministic order. -- http://mail.python.org/mailman/listinfo/python-list

Re: garbage collection / cyclic references

2009-03-20 Thread andrew cooke
Aaron Brady wrote: [...] > caveats and fragilities? If free software can do it, why isn't it all > over the industry? What disqualifies it from solved-problem status? the two dominant virtual machines - .net and the jvm both handle circular references with no problem whatever. this is standard

Re: Garbage collection of recursive inner function

2008-08-05 Thread from . future . import
On Aug 5, 5:23 am, Terry Reedy <[EMAIL PROTECTED]> wrote: > To understand this, it helps to realize that Python functions are not, > in themselves, recursive. Recursiveness at any time is a property of a > function in an environment, which latter can change. More specifically, > a function call

Re: Garbage collection of recursive inner function

2008-08-04 Thread Terry Reedy
[EMAIL PROTECTED] wrote: I encountered garbage collection behaviour that I didn't expect when using a recursive function inside another function: To understand this, it helps to realize that Python functions are not, in themselves, recursive. Recursiveness at any time is a property of a fu

Re: Garbage collection of recursive inner function

2008-08-04 Thread Diez B. Roggisch
[EMAIL PROTECTED] schrieb: Hi, I encountered garbage collection behaviour that I didn't expect when using a recursive function inside another function: the definition of the inner function seems to contain a circular reference, which means it is only collected by the mark-and-sweep collector, no

Re: Garbage collection

2008-02-19 Thread Duncan Booth
Nick Craig-Wood <[EMAIL PROTECTED]> wrote: > [<__main__.Y object at 0xb7d9fc8c>, <__main__.Y object at 0xb7d9fcac>, > <__main__.Y object at 0xb7d9fc2c>] [<__main__.Y object at 0xb7d9fc8c>] > > (It behaves slightly differently in the interactive interpreter for > reasons I don't understand - so sa

Re: Garbage collection

2008-02-19 Thread Nick Craig-Wood
Hrvoje Niksic <[EMAIL PROTECTED]> wrote: > Simon Pickles <[EMAIL PROTECTED]> writes: > > > Ken wrote: > >> What is your __del__ method doing? > >> > > Actually, nothing but printing a message when the object is deleted, > > just morbid curiosity. > > > > I've yet to see one of the destructor m

Re: Garbage collection

2008-02-19 Thread Duncan Booth
Jarek Zgoda <[EMAIL PROTECTED]> wrote: > Is that true assumption that __del__ has the same purpose (and same > limitations, i.e. the are not guaranteed to be fired) as Java finalizer > methods? One other point I should have mentioned about __del__: if you are running under Windows and the user h

Re: Garbage collection

2008-02-19 Thread Duncan Booth
Jarek Zgoda <[EMAIL PROTECTED]> wrote: > Duncan Booth napisa³(a): > >> Pretty much. If you have a __del__ method on an object then in the >> worst case the only thing that can be guaranteed is that it will be >> called zero, one or more than one times. (Admittedly the last of >> these only happen

Re: Garbage collection

2008-02-19 Thread Hrvoje Niksic
Simon Pickles <[EMAIL PROTECTED]> writes: > Ken wrote: >> What is your __del__ method doing? >> > Actually, nothing but printing a message when the object is deleted, > just morbid curiosity. > > I've yet to see one of the destructor messages, tho Do your objects participate in reference c

Re: Garbage collection

2008-02-19 Thread Jarek Zgoda
Duncan Booth napisał(a): > Pretty much. If you have a __del__ method on an object then in the worst > case the only thing that can be guaranteed is that it will be called zero, > one or more than one times. (Admittedly the last of these only happens if > you work at it). > > If it is called th

Re: Garbage collection

2008-02-19 Thread Duncan Booth
Jarek Zgoda <[EMAIL PROTECTED]> wrote: > Ken napisa³(a): > >> The good news is that you almost never have to do anything to clean up. >> My guess is that you might not even need to overload __del__ at all. >> People from a C++ background often mistakenly think that they have to >> write destruc

Re: Garbage collection

2008-02-19 Thread Jarek Zgoda
Ken napisał(a): > The good news is that you almost never have to do anything to clean up. > My guess is that you might not even need to overload __del__ at all. > People from a C++ background often mistakenly think that they have to > write destructors when in fact they do not. Is that true ass

Re: Garbage collection

2008-02-19 Thread Simon Pickles
Ken wrote: > What is your __del__ method doing? > Actually, nothing but printing a message when the object is deleted, just morbid curiosity. I've yet to see one of the destructor messages, tho > > from sys import getrefcount > print getrefcount(x) > > Perfect, thanks Simon -- ht

Re: Garbage collection

2008-02-18 Thread Aahz
In article <[EMAIL PROTECTED]>, Ken <[EMAIL PROTECTED]> wrote: >Simon Pickles wrote: >> >> For instance, I have a manager looking after many objects in a dict. >> When those objects are no longer needed, I use del manager[objectid], >> hoping to force the garbage collector to perform the delete.

Re: Garbage collection

2008-02-18 Thread Ken
Simon Pickles wrote: > Hi, > > I'm building a server with python, but coming from a c++ background, > garbage collection seems strange. > > For instance, I have a manager looking after many objects in a dict. > When those objects are no longer needed, I use del manager[objectid], > hoping to for

Re: Garbage collection

2007-03-22 Thread Alex Martelli
Steve Holden <[EMAIL PROTECTED]> wrote: ... > > a. fork > > b. do the memory-hogging work in the child process > > c. meanwhile the parent just waits > > d. the child sends back to the parent the small results > > e. the child terminates > > f. the parent proceeds merrily > > > > I learned this

Re: Garbage collection

2007-03-22 Thread Alex Martelli
Tom Wright <[EMAIL PROTECTED]> wrote: > real programs. I can't help thinking that there are some situations where > you need a lot of memory for a short time though, and it would be nice to > be able to use it briefly and then hand most of it back. Still, I see the > practical difficulties with

Re: Garbage collection

2007-03-22 Thread Steve Holden
Alex Martelli wrote: > Tom Wright <[EMAIL PROTECTED]> wrote: > >> real programs. I can't help thinking that there are some situations where >> you need a lot of memory for a short time though, and it would be nice to >> be able to use it briefly and then hand most of it back. Still, I see the >>

Re: Garbage collection

2007-03-21 Thread Cameron Laird
In article <[EMAIL PROTECTED]>, Dennis Lee Bieber <[EMAIL PROTECTED]> wrote: >On Wed, 21 Mar 2007 15:32:17 +, Tom Wright <[EMAIL PROTECTED]> >declaimed the following in comp.lang.python: > >> >> True, but why does Python hang on to the memory at all? As I understand it, >> it's keeping a big

Re: Garbage collection

2007-03-21 Thread Aahz
In article <[EMAIL PROTECTED]>, Nick Craig-Wood <[EMAIL PROTECTED]> wrote: >Steven D'Aprano <[EMAIL PROTECTED]> wrote: >> >> Or you could just have an "object leak" somewhere. Do you have any >> complicated circular references that the garbage collector can't resolve? >> Lists-of-lists? Trees?

Re: Garbage collection

2007-03-21 Thread Nick Craig-Wood
Steven D'Aprano <[EMAIL PROTECTED]> wrote: > Or you could just have an "object leak" somewhere. Do you have any > complicated circular references that the garbage collector can't resolve? > Lists-of-lists? Trees? Anything where objects aren't being freed when you > think they are? Are you holdi

Re: Garbage collection

2007-03-21 Thread Steven D'Aprano
On Wed, 21 Mar 2007 17:19:23 +, Tom Wright wrote: >> So what's your actual problem that you are trying to solve? > > I have a program which reads a few thousand text files, converts each to a > list (with readlines()), creates a short summary of the contents of each (a > few floating point nu

Re: Garbage collection

2007-03-21 Thread Steve Holden
Tom Wright wrote: > Steven D'Aprano wrote: >> You've described an extremely artificial set of circumstances: you create >> 40,000,000 distinct integers, then immediately destroy them. The obvious >> solution to that "problem" of Python caching millions of integers you >> don't need is not to create

Re: Garbage collection

2007-03-21 Thread Tom Wright
[EMAIL PROTECTED] wrote: > If your program's behavior is: > > * allocate a list of 1e7 ints > * delete that list > > how does the Python interpreter know your next bit of execution won't be > to repeat the allocation? It doesn't know, but if the program runs for a while without repeating

Re: Garbage collection

2007-03-21 Thread Tom Wright
Steve Holden wrote: > Easy to say. How do you know the memory that's not in use is in a > contiguous block suitable for return to the operating system? I can > pretty much guarantee it won't be. CPython doesn't use a relocating > garbage collection scheme Fair point. That is difficult and I don't

Re: Garbage collection

2007-03-21 Thread Tom Wright
Steven D'Aprano wrote: > You've described an extremely artificial set of circumstances: you create > 40,000,000 distinct integers, then immediately destroy them. The obvious > solution to that "problem" of Python caching millions of integers you > don't need is not to create them in the first place

Re: Garbage collection

2007-03-21 Thread Steven D'Aprano
On Wed, 21 Mar 2007 15:32:17 +, Tom Wright wrote: >> Memory contention would be a problem if your Python process wanted to keep >> that memory active at the same time as you were running GIMP. > > True, but why does Python hang on to the memory at all? As I understand it, > it's keeping a bi

Re: Garbage collection

2007-03-21 Thread Steven D'Aprano
On Wed, 21 Mar 2007 15:03:17 +, Tom Wright wrote: [snip] > Ah, thanks for explaining that. I'm a little wiser about memory allocation > now, but am still having problems reclaiming memory from unused objects > within Python. If I do the following: > > (memory use: 15 MB) a = rang

Re: Garbage collection

2007-03-21 Thread skip
Tom> True, but why does Python hang on to the memory at all? As I Tom> understand it, it's keeping a big lump of memory on the int free Tom> list in order to make future allocations of large numbers of Tom> integers faster. If that memory is about to be paged out, then Tom> s

Re: Garbage collection

2007-03-21 Thread Steve Holden
Tom Wright wrote: > [EMAIL PROTECTED] wrote: >> Tom> ...and then I allocate a lot of memory in another process (eg. >> open Tom> a load of files in the GIMP), then the computer swaps the >> Python >> Tom> process out to disk to free up the necessary space. Python's >> Tom> memo

Re: Garbage collection

2007-03-21 Thread Tom Wright
[EMAIL PROTECTED] wrote: > Tom> ...and then I allocate a lot of memory in another process (eg. > open Tom> a load of files in the GIMP), then the computer swaps the > Python > Tom> process out to disk to free up the necessary space. Python's > Tom> memory use is still reported

Re: Garbage collection

2007-03-21 Thread skip
Tom> ...and then I allocate a lot of memory in another process (eg. open Tom> a load of files in the GIMP), then the computer swaps the Python Tom> process out to disk to free up the necessary space. Python's Tom> memory use is still reported as 953 MB, even though nothing like

Re: Garbage collection

2007-03-21 Thread Tom Wright
[EMAIL PROTECTED] wrote: > You haven't forgotten to do anything. Your attempts at freeing memory are > being thwarted (in part, at least) by Python's int free list. I believe > the int free list remains after the 10M individual ints' refcounts drop to > zero. The large storage for the list is gra

Re: Garbage collection

2007-03-21 Thread Thinker
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tom Wright wrote: > Thinker wrote: >> How do you know amount of memory used by Python? ps ? top or >> something? > > $ ps up `pidof python2.5` USER PID %CPU %MEM VSZ RSS TTY > STAT START TIME COMMAND tew24 26275 0.0 11.9 257592 243988 > pts/6 S+ 13:10

Re: Garbage collection

2007-03-21 Thread Tom Wright
Thinker wrote: > How do you know amount of memory used by Python? > ps ? top or something? $ ps up `pidof python2.5` USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND tew2426275 0.0 11.9 257592 243988 pts/6 S+ 13:10 0:00 python2.5 "VSZ" is "Virtual Memory Size" (

Re: Garbage collection

2007-03-21 Thread skip
Tom> I suspect I may be missing something vital here, but Python's Tom> garbage collection doesn't seem to work as I expect it to. Here's Tom> a small test program which shows the problem on python 2.4 and 2.5: Tom> (at this point, Python is using 15MB) >>> a = range(int(1e7)

Re: Garbage collection

2007-03-21 Thread Thinker
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tom Wright wrote: > Hi all > > I suspect I may be missing something vital here, but Python's garbage > collection doesn't seem to work as I expect it to. Here's a small test > program which shows the problem on python 2.4 and 2.5: ... skip

Re: Garbage collection with QT

2005-06-08 Thread Ken Godee
> Is there a way, to find out all references to the QMainWindow or its > hosted QTable, for having a mechanism to destroy them? > Yes, of coarse, the docs are your friend :) QObject::children() QObject::removeChild() QObject::parent() To find all the children for an instance you can create a lo

Re: Garbage collection with QT

2005-06-08 Thread Mike Tammerman
Not all leakage problems caused by qt or python. There is a wrapping layer between Qt and Python provided by SIP. Therefore, SIP may cause leakages. Also PyQt had a paintCell memory leakage problem several months ago. If you're using an old snapshot of PyQt or SIP, that would be a problem. Try usin