Re: Question about garbage collection
So, here's some info about how to see what's going on with Python's memory allocation: https://docs.python.org/3/library/tracemalloc.html . I haven't looked into this in a long time, but it used to be the case that you needed to compile native modules (and probably Python itself?) so that instrumentation is possible (I think incref / decref macros should give you a hint, because they would have to naturally report some of that info). Anyways. The problem of tracing memory allocation / deallocation in Python can be roughly split into these categories: 1. Memory legitimately claimed by objects created through Python runtime, but not reclaimed due to programmer error. I.e. the programmer wrote a program that keeps references to objects which it will never use again. 2. Memory claimed through native objects obtained by means of interacting with Python's allocator. When working with Python C API it's best to interface with Python allocator to deal with dynamic memory allocation and release. However, it's somewhat cumbersome, and some module authors simply might not know about it, or wouldn't want to use it because they prefer a different allocator. Sometimes library authors don't implement memory deallocation well. Which brings us to: 3. Memory claimed by any user-space code that is associated with the Python process. This can be for example shared libraries loaded by means of Python bindings, that is on top of the situation described above. 4. System memory associated with the process. Some system calls need to allocate memory on the system side. Typical examples are opening files, creating sockets etc. Typically, the system will limit the number of such objects, and the user program will hit the numerical limit before it hits the memory limit, but it can also happen that this will manifest as a memory problem (one example I ran into was trying to run conda-build and it would fail due to enormous amounts of memory it requested, but the specifics of the failure were due to it trying to create new sub-processes -- another system resource that requires memory allocation). There isn't a universal strategy to cover all these cases. But, if you have reasons to suspect (4), for example, you'd probably start by using strace utility (on Linux) to see what system calls are executed. For something like the (3), you could try to utilize Valgrind (but it's a lot of work to set it up). It's also possible to use jemalloc to profile a program, but you would have to build Python with its allocator modified to use jemalloc (I've seen an issue in the Python bug tracker where someone wrote a script to do that, so it should be possible). Both of these are quite labor intensive and not trivial to set up. (2) could be often diagnosed with tracemalloc Python module and (1) is something that can be helped with Python's gc module. It's always better though to have an actual error and work from there. Or, at least, have some monitoring data that suggests that your application memory use increases over time. Otherwise you could be spending a lot of time chasing problems you don't have. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 2024-01-17 3:01 AM, Greg Ewing via Python-list wrote: On 17/01/24 1:01 am, Frank Millman wrote: I sometimes need to keep a reference from a transient object to a more permanent structure in my app. To save myself the extra step of removing all these references when the transient object is deleted, I make them weak references. I don't see how weak references help here at all. If the transient object goes away, all references from it to the permanent objects also go away. A weak reference would only be of use if the reference went the other way, i.e. from the permanent object to the transient object. You are right. I got my description above back-to-front. It is a pub/sub scenario. A transient object makes a request to the permanent object to be notified of any changes. The permanent object stores a reference to the transient object and executes a callback on each change. When the transient object goes away, the reference must be removed. Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 17/01/24 1:01 am, Frank Millman wrote: I sometimes need to keep a reference from a transient object to a more permanent structure in my app. To save myself the extra step of removing all these references when the transient object is deleted, I make them weak references. I don't see how weak references help here at all. If the transient object goes away, all references from it to the permanent objects also go away. A weak reference would only be of use if the reference went the other way, i.e. from the permanent object to the transient object. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 17/01/24 4:00 am, Chris Angelico wrote: class Form: def __init__(self): self.elements = [] class Element: def __init__(self, form): self.form = form form.elements.append(self) If you make the reference from Element to Form a weak reference, it won't keep the Form alive after it's been closed. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
> On 16 Jan 2024, at 12:10, Frank Millman via Python-list > wrote: > > My problem is that my app is quite complex, and it is easy to leave a > reference dangling somewhere which prevents an object from being gc'd. What I do to track these problems down is use gc.get_objects() then summerize the number of each type. Part 2 is to print the delta after an interval of a 2nd summary. Leaks of objects show up as the count of a type increasing every time you sample. Barry -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
> On 16 Jan 2024, at 13:17, Thomas Passin via Python-list > wrote: > > The usual advice is to call deleteLater() on objects derived from PyQt > classes. I don't know enough about PyQt to know if this takes care of all > dangling reference problems, though. It works well and robustly. Barry -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On Wed, 17 Jan 2024 at 01:45, Frank Millman via Python-list wrote: > > On 2024-01-16 2:15 PM, Chris Angelico via Python-list wrote: > > > > Where do you tend to "leave a reference dangling somewhere"? How is > > this occurring? Is it a result of an incomplete transaction (like an > > HTTP request that never finishes), or a regular part of the operation > > of the server? > > > > I have a class that represents a database table, and another class that > represents a database column. There is a one-to-many relationship and > they maintain references to each other. > > In another part of the app, there is a class that represents a form, and > another class that represents the gui elements on the form. Again there > is a one-to-many relationship. I don't know when you'd be "done" with the table, so I won't try to give an example, but I'll try this one and maybe it'll give some ideas that could apply to both. When you open the form, you initialize it, display it, etc, etc. This presumably includes something broadly like this: class Form: def __init__(self): self.elements = [] class Element: def __init__(self, form): self.form = form form.elements.append(self) frm = Form(...) Element(frm, ...) # as many as needed frm.show() # present it to the user This is a pretty classic refloop. I don't know exactly what your setup is, but most likely it's going to look something like this. Feel free to correct me if it doesn't. The solution here would be to trap the "form is no longer being displayed" moment. That'll be some sort of GUI event like a "close" or "delete" signal. When that comes through (and maybe after doing other processing), you no longer need the form, and can dispose of it. The simplest solution here is: Empty out frm.elements. That immediately leaves the form itself as a leaf (no references to anything relevant), and the elements still refer back to it, but once nothing ELSE refers to the form, everything can be disposed of. > A gui element that represents a piece of data has to maintain a link to > its database column object. There can be a many-to-one relationship, as > there could be more than one gui element referring to the same column. Okay, so the Element also refers to the corresponding Column. If the Form and Element aren't in a refloop, this shouldn't be a problem. However, if this is the same Table and Column that you referred to above, that might be the answer to my question. Are you "done" with the Table at the same time that the form is no longer visible? If so, you would probably have something similar where the Form refers to the Table, and the Table and Columns refer to each other... so the same solution hopefully should work: wipe out the Table's list of columns. > There are added complications which I won't go into here. The bottom > line is that on some occasions a form which has been closed does not get > gc'd. > > I have been trying to reproduce the problem in my toy app, but I cannot > get it to fail. There is a clue there! I think I have just > over-complicated things. Definitely possible. > I will start with a fresh approach tomorrow. If you don't hear from me > again, you will know that I have solved it! > > Thanks for the input, it definitely helped. Cool cool, happy to help. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 2024-01-16 2:15 PM, Chris Angelico via Python-list wrote: Where do you tend to "leave a reference dangling somewhere"? How is this occurring? Is it a result of an incomplete transaction (like an HTTP request that never finishes), or a regular part of the operation of the server? I have a class that represents a database table, and another class that represents a database column. There is a one-to-many relationship and they maintain references to each other. In another part of the app, there is a class that represents a form, and another class that represents the gui elements on the form. Again there is a one-to-many relationship. A gui element that represents a piece of data has to maintain a link to its database column object. There can be a many-to-one relationship, as there could be more than one gui element referring to the same column. There are added complications which I won't go into here. The bottom line is that on some occasions a form which has been closed does not get gc'd. I have been trying to reproduce the problem in my toy app, but I cannot get it to fail. There is a clue there! I think I have just over-complicated things. I will start with a fresh approach tomorrow. If you don't hear from me again, you will know that I have solved it! Thanks for the input, it definitely helped. Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 1/16/2024 4:17 AM, Barry wrote: On 16 Jan 2024, at 03:49, Thomas Passin via Python-list wrote: This kind of thing can happen with PyQt, also. There are ways to minimize it but I don't know if you can ever be sure all Qt C++ objects will get deleted. It depends on the type of object and the circumstances. When this has been seen in the past it has been promptly fixed by the maintainer. The usual advice is to call deleteLater() on objects derived from PyQt classes. I don't know enough about PyQt to know if this takes care of all dangling reference problems, though. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On Tue, 16 Jan 2024 at 23:08, Frank Millman via Python-list wrote: > > On 2024-01-15 3:51 PM, Frank Millman via Python-list wrote: > > Hi all > > > > I have read that one should not have to worry about garbage collection > > in modern versions of Python - it 'just works'. > > > > I don't want to rely on that. My app is a long-running server, with > > multiple clients logging on, doing stuff, and logging off. They can > > create many objects, some of them long-lasting. I want to be sure that > > all objects created are gc'd when the session ends. > > > > I did not explain myself very well. Sorry about that. > > My problem is that my app is quite complex, and it is easy to leave a > reference dangling somewhere which prevents an object from being gc'd. > > This can create (at least) two problems. The obvious one is a memory > leak. The second is that I sometimes need to keep a reference from a > transient object to a more permanent structure in my app. To save myself > the extra step of removing all these references when the transient > object is deleted, I make them weak references. This works, unless the > transient object is kept alive by mistake and the weak ref is never removed. > > I feel it is important to find these dangling references and fix them, > rather than wait for problems to appear in production. The only method I > can come up with is to use the 'delwatcher' class that I used in my toy > program in my original post. > > I am surprised that this issue does not crop up more often. Does nobody > else have these problems? > It really depends on how big those dangling objects are. My personal habit is to not worry about a few loose objects, by virtue of ensuring that everything either has its reference loops deliberately broken at some point in time, or by keeping things small. An example of deliberately breaking a refloop would be when I track websockets. Usually I'll tag the socket object itself with some kind of back-reference to my own state, but I also need to be able to iterate over all of my own state objects (let's say they're dictionaries for simplicity) and send a message to each socket. So there'll be a reference loop between the socket and the state. But at some point, I will be notified that the socket has been disconnected, and that's when I go to its state object and wipe out its back-reference. It can then be disposed of promptly, since there's no loop. It takes a bit of care, but in general, large state objects won't have these kinds of loops, and dangling references haven't caused me any sort of major issues in production. Where do you tend to "leave a reference dangling somewhere"? How is this occurring? Is it a result of an incomplete transaction (like an HTTP request that never finishes), or a regular part of the operation of the server? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 2024-01-15 3:51 PM, Frank Millman via Python-list wrote: Hi all I have read that one should not have to worry about garbage collection in modern versions of Python - it 'just works'. I don't want to rely on that. My app is a long-running server, with multiple clients logging on, doing stuff, and logging off. They can create many objects, some of them long-lasting. I want to be sure that all objects created are gc'd when the session ends. I did not explain myself very well. Sorry about that. My problem is that my app is quite complex, and it is easy to leave a reference dangling somewhere which prevents an object from being gc'd. This can create (at least) two problems. The obvious one is a memory leak. The second is that I sometimes need to keep a reference from a transient object to a more permanent structure in my app. To save myself the extra step of removing all these references when the transient object is deleted, I make them weak references. This works, unless the transient object is kept alive by mistake and the weak ref is never removed. I feel it is important to find these dangling references and fix them, rather than wait for problems to appear in production. The only method I can come up with is to use the 'delwatcher' class that I used in my toy program in my original post. I am surprised that this issue does not crop up more often. Does nobody else have these problems? Frank -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
> On 16 Jan 2024, at 03:49, Thomas Passin via Python-list > wrote: > > This kind of thing can happen with PyQt, also. There are ways to minimize it > but I don't know if you can ever be sure all Qt C++ objects will get deleted. > It depends on the type of object and the circumstances. When this has been seen in the past it has been promptly fixed by the maintainer. Barry -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 1/15/2024 9:47 PM, Akkana Peck via Python-list wrote: I wrote: Also be warned that some modules (particularly if they're based on libraries not written in Python) might not garbage collect, so you may need to use other methods of cleaning up after those objects. Chris Angelico writes: Got any examples of that? The big one for me was gdk-pixbuf, part of GTK. When you do something like gtk.gdk.pixbuf_new_from_file(), there's a Python object that gets created, but there's also the underlying C code that allocates memory for the pixbuf. When the object went out of scope, the Python object was automatically garbage collected, but the pixbuf data leaked. This kind of thing can happen with PyQt, also. There are ways to minimize it but I don't know if you can ever be sure all Qt C++ objects will get deleted. It depends on the type of object and the circumstances. Calling gc.collect() caused the pixbuf data to be garbage collected too. There used to be a post explaining this on the pygtk mailing list: the link was http://www.daa.com.au/pipermail/pygtk/2003-December/006499.html but that page is gone now and I can't seem to find any other archives of that list (it's not on archive.org either). And this was from GTK2; I never checked whether the extra gc.collect() is still necessary in GTK3, but I figure leaving it in doesn't hurt anything. I use pixbufs in a tiled map application, so there are a lot of small pixbufs being repeatedly read and then deallocated. ...Akkana -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On Tue, 16 Jan 2024 at 13:49, Akkana Peck via Python-list wrote: > > I wrote: > > > Also be warned that some modules (particularly if they're based on > > > libraries not written in Python) might not garbage collect, so you may > > > need to use other methods of cleaning up after those objects. > > Chris Angelico writes: > > Got any examples of that? > > The big one for me was gdk-pixbuf, part of GTK. When you do something like > gtk.gdk.pixbuf_new_from_file(), there's a Python object that gets created, > but there's also the underlying C code that allocates memory for the pixbuf. > When the object went out of scope, the Python object was automatically > garbage collected, but the pixbuf data leaked. Calling gc.collect() caused > the pixbuf data to be garbage collected too. > > There used to be a post explaining this on the pygtk mailing list: the link > was > http://www.daa.com.au/pipermail/pygtk/2003-December/006499.html > but that page is gone now and I can't seem to find any other archives of that > list (it's not on archive.org either). And this was from GTK2; I never > checked whether the extra gc.collect() is still necessary in GTK3, but I > figure leaving it in doesn't hurt anything. I use pixbufs in a tiled map > application, so there are a lot of small pixbufs being repeatedly read and > then deallocated. > Okay, so to clarify: the Python object will always be garbage collected correctly, but a buggy third-party module might have *external* resources (in that case, the pixbuf) that aren't properly released. Either that, or there is a reference loop, which doesn't necessarily mean you NEED to call gc.collect(), but it can help if you want to get rid of them more promptly. (Python will detect such loops at some point, but not always immediately.) But these are bugs in the module, particularly the first case, and should be considered as such. 2003 is fully two decades ago now, and I would not expect that a serious bug like that has been copied into PyGObject (the newer way of using GTK from Python). So, Python's garbage collection CAN be assumed to "just work", unless you find evidence to the contrary. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
I wrote: > > Also be warned that some modules (particularly if they're based on > > libraries not written in Python) might not garbage collect, so you may need > > to use other methods of cleaning up after those objects. Chris Angelico writes: > Got any examples of that? The big one for me was gdk-pixbuf, part of GTK. When you do something like gtk.gdk.pixbuf_new_from_file(), there's a Python object that gets created, but there's also the underlying C code that allocates memory for the pixbuf. When the object went out of scope, the Python object was automatically garbage collected, but the pixbuf data leaked. Calling gc.collect() caused the pixbuf data to be garbage collected too. There used to be a post explaining this on the pygtk mailing list: the link was http://www.daa.com.au/pipermail/pygtk/2003-December/006499.html but that page is gone now and I can't seem to find any other archives of that list (it's not on archive.org either). And this was from GTK2; I never checked whether the extra gc.collect() is still necessary in GTK3, but I figure leaving it in doesn't hurt anything. I use pixbufs in a tiled map application, so there are a lot of small pixbufs being repeatedly read and then deallocated. ...Akkana -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On Tue, 16 Jan 2024 at 06:32, Akkana Peck via Python-list wrote: > > > Frank Millman wrote at 2024-1-15 15:51 +0200: > > >I have read that one should not have to worry about garbage collection > > >in modern versions of Python - it 'just works'. > > Dieter Maurer via Python-list writes: > > There are still some isolated cases when not all objects > > in an unreachable cycle are destroyed > > (see e.g. step 2 of > > "https://devguide.python.org/internals/garbage-collector/index.html#destroying-unreachable-objects";). > > Also be warned that some modules (particularly if they're based on libraries > not written in Python) might not garbage collect, so you may need to use > other methods of cleaning up after those objects. > Got any examples of that? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
> Frank Millman wrote at 2024-1-15 15:51 +0200: > >I have read that one should not have to worry about garbage collection > >in modern versions of Python - it 'just works'. Dieter Maurer via Python-list writes: > There are still some isolated cases when not all objects > in an unreachable cycle are destroyed > (see e.g. step 2 of > "https://devguide.python.org/internals/garbage-collector/index.html#destroying-unreachable-objects";). Also be warned that some modules (particularly if they're based on libraries not written in Python) might not garbage collect, so you may need to use other methods of cleaning up after those objects. ...Akkana -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
Frank Millman wrote at 2024-1-15 15:51 +0200: >I have read that one should not have to worry about garbage collection >in modern versions of Python - it 'just works'. There are still some isolated cases when not all objects in an unreachable cycle are destroyed (see e.g. step 2 of "https://devguide.python.org/internals/garbage-collector/index.html#destroying-unreachable-objects";). But Python's own objects (e.g. traceback cycles) or instances of classes implemented in Python should no longer be affected. Thus, unless you use extensions implemented in C (with "legacy finalizer"s), garbage collection should not make problems. On the other hand, your application, too, must avoid memory leaks. Caches of various forms (with data for several sessions) might introduce them. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
> I do have several circular references. My experience is that if I do not > take some action to break the references when closing the session, the > objects remain alive. Below is a very simple program to illustrate this. > > Am I missing something? All comments appreciated. Python has normal reference counting, but also has a cyclic garbage collector. Here's plenty of detail about how it works: https://devguide.python.org/internals/garbage-collector/index.html Skip -- https://mail.python.org/mailman/listinfo/python-list
Question about garbage collection
Hi all
I have read that one should not have to worry about garbage collection
in modern versions of Python - it 'just works'.
I don't want to rely on that. My app is a long-running server, with
multiple clients logging on, doing stuff, and logging off. They can
create many objects, some of them long-lasting. I want to be sure that
all objects created are gc'd when the session ends.
I do have several circular references. My experience is that if I do not
take some action to break the references when closing the session, the
objects remain alive. Below is a very simple program to illustrate this.
Am I missing something? All comments appreciated.
Frank Millman
==
import gc
class delwatcher:
# This stores enough information to identify the object being watched.
# It does not store a reference to the object itself.
def __init__(self, obj):
self.id = (obj.type, obj.name, id(obj))
print('***', *self.id, 'created ***')
def __del__(self):
print('***', *self.id, 'deleted ***')
class Parent:
def __init__(self, name):
self.type = 'parent'
self.name = name
self.children = []
self._del = delwatcher(self)
class Child:
def __init__(self, parent, name):
self.type = 'child'
self.parent = parent
self.name = name
parent.children.append(self)
self._del = delwatcher(self)
p1 = Parent('P1')
p2 = Parent('P2')
c1_1 = Child(p1, 'C1_1')
c1_2 = Child(p1, 'C1_2')
c2_1 = Child(p2, 'C2_1')
c2_2 = Child(p2, 'C2_2')
input('waiting ...')
# if next 2 lines are included, parent and child can be gc'd
# for ch in p1.children:
# ch.parent = None
# if next line is included, child can be gc'd, but not parent
# p1.children = None
del c1_1
del p1
gc.collect()
input('wait some more ...')
--
https://mail.python.org/mailman/listinfo/python-list
Re: Copy-on-write friendly Python garbage collection (Posting On Python-List Prohibited)
> Niles, if you want to claim wxjmfauth is right, you'll have to present > some actual evidence. He's claimed for years that Python's Unicode > support is buggy (as he does here), without ever demonstrating a bug. > We've long ago tired of trying to reason with him. > > The tradeoffs of memory use for algorithmic complexity are well > understood, and have been endlessly discussed. There is not an > "obviously right" answer to how to make those tradeoffs. > > --Ned. Aah, I didn't know, I'm new here. I thought he was referring to UTF-16 being inefficient, which is mostly what I was agreeing with. -- https://mail.python.org/mailman/listinfo/python-list
Re: Copy-on-write friendly Python garbage collection (Posting On Python-List Prohibited)
On 1/1/18 1:49 PM, Niles Rogoff wrote:
On Mon, 01 Jan 2018 10:42:58 -0800, breamoreboy wrote:
On Monday, January 1, 2018 at 10:14:59 AM UTC, wxjm...@gmail.com wrote:
Le lundi 1 janvier 2018 08:35:53 UTC+1, Lawrence D’Oliveiro a écrit :
On Monday, January 1, 2018 at 7:52:48 AM UTC+13, Paul Rubin wrote:
I wonder if things would suffer if they tried a more aggressive
approach and ditched refcounting completely.
One thing that would suffer is Python’s well-behaved memory usage.
You would need to start imposing heap usage limits, like you do in
Java.
Memory:
sys.getsizeof('abcdefghij' + '$')
36
sys.getsizeof('abcdefghij' + '€')
60
sys.getsizeof(('abcdefghij' + '€').encode('utf-8'))
30
sys.getsizeof('abcdefghij' + '\U0001')
84
sys.getsizeof(('abcdefghij' + '\U0001').encode('utf-8'))
31
Performance:
"anti - utf-32"
Buggyness:
Better to not comment.
Python is the single language, which is presenting the opposite of what
Unicode.org offers on the side of memory *and* on the side of
performance, utf-8 *and* utf-32 !
Happy new year.
He's right though. I would encourage anyone interested to check out
http://utf8everywhere.org/
Niles, if you want to claim wxjmfauth is right, you'll have to present
some actual evidence. He's claimed for years that Python's Unicode
support is buggy (as he does here), without ever demonstrating a bug.
We've long ago tired of trying to reason with him.
The tradeoffs of memory use for algorithmic complexity are well
understood, and have been endlessly discussed. There is not an
"obviously right" answer to how to make those tradeoffs.
--Ned.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Copy-on-write friendly Python garbage collection
On Monday, January 1, 2018 at 12:53:03 PM UTC, Wu Xi wrote: > breamoreboy: > > On Sunday, December 31, 2017 at 6:19:13 PM UTC, Wu Xi wrote: > >> breamoreboy: > >>> An interesting write up on something that is incorporated into Python 3.7 > >>> https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf > >> > >> Appearantly, Erlang is the way to go, when it comes to web frameworks. > > > > What has that got to do with the subject of this thread? > > Well, a little implicitly. Pointing out Erlang superiority in a pythonic > newsgroup is, of course, heresy. Python is fourth in the latest TIOBE index, Erlang doesn't even make the top 20, so in what way is it superior? Python doesn't need Pinky and the Brain in its quest to take over the world. -- Kindest regards. Mark Lawrence. -- https://mail.python.org/mailman/listinfo/python-list
Re: Copy-on-write friendly Python garbage collection (Posting On Python-List Prohibited)
On Mon, 01 Jan 2018 10:42:58 -0800, breamoreboy wrote:
> On Monday, January 1, 2018 at 10:14:59 AM UTC, wxjm...@gmail.com wrote:
>> Le lundi 1 janvier 2018 08:35:53 UTC+1, Lawrence D’Oliveiro a écrit :
>> > On Monday, January 1, 2018 at 7:52:48 AM UTC+13, Paul Rubin wrote:
>> > > I wonder if things would suffer if they tried a more aggressive
>> > > approach and ditched refcounting completely.
>> >
>> > One thing that would suffer is Python’s well-behaved memory usage.
>> > You would need to start imposing heap usage limits, like you do in
>> > Java.
>>
>> Memory:
>>
>> >>> sys.getsizeof('abcdefghij' + '$')
>> 36
>> >>> sys.getsizeof('abcdefghij' + '€')
>> 60
>> >>> sys.getsizeof(('abcdefghij' + '€').encode('utf-8'))
>> 30
>> >>> sys.getsizeof('abcdefghij' + '\U0001')
>> 84
>> >>> sys.getsizeof(('abcdefghij' + '\U0001').encode('utf-8'))
>> 31
>> >>>
>> >>>
>> Performance:
>> "anti - utf-32"
>>
>> Buggyness:
>> Better to not comment.
>>
>> Python is the single language, which is presenting the opposite of what
>> Unicode.org offers on the side of memory *and* on the side of
>> performance, utf-8 *and* utf-32 !
>>
>> Happy new year.
>
He's right though. I would encourage anyone interested to check out
http://utf8everywhere.org/
> Your usual drivel. When are you going to stop banging this drum, you've
> done nothing else for the past five years?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Copy-on-write friendly Python garbage collection
breamore...@gmail.com: > On Sunday, December 31, 2017 at 6:19:13 PM UTC, Wu Xi wrote: >> breamoreboy: >>> An interesting write up on something that is incorporated into Python 3.7 >>> https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf >> >> Appearantly, Erlang is the way to go, when it comes to web frameworks. > > What has that got to do with the subject of this thread? Well, a little implicitly. Pointing out Erlang superiority in a pythonic newsgroup is, of course, heresy. -- https://mail.python.org/mailman/listinfo/python-list
Re: Copy-on-write friendly Python garbage collection
On Sunday, December 31, 2017 at 6:19:13 PM UTC, Wu Xi wrote: > breamoreboy: > > An interesting write up on something that is incorporated into Python 3.7 > > https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf > > Appearantly, Erlang is the way to go, when it comes to web frameworks. What has that got to do with the subject of this thread? -- Kindest regards. Mark Lawrence. -- https://mail.python.org/mailman/listinfo/python-list
Re: Copy-on-write friendly Python garbage collection
FYI: https://bugs.python.org/issue31558 INADA Naoki On Mon, Jan 1, 2018 at 12:39 AM, wrote: > An interesting write up on something that is incorporated into Python 3.7 > https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf > > -- > Kindest regards. > > Mark Lawrence. > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Copy-on-write friendly Python garbage collection
breamore...@gmail.com: > An interesting write up on something that is incorporated into Python 3.7 > https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf Appearantly, Erlang is the way to go, when it comes to web frameworks. -- https://mail.python.org/mailman/listinfo/python-list
Re: Copy-on-write friendly Python garbage collection
they said they run the largest deployment of Django world-wide. be it as it may... many still consider the web guys to be the "funny people". Why did they not switch over to Erlang ? -- https://mail.python.org/mailman/listinfo/python-list
Re: Copy-on-write friendly Python garbage collection
breamore...@gmail.com: > An interesting write up on something that is incorporated into Python 3.7 > https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf kewl py 3.7 does not fully make install here, but it built and works, as far as I can tell -- https://mail.python.org/mailman/listinfo/python-list
Copy-on-write friendly Python garbage collection
An interesting write up on something that is incorporated into Python 3.7 https://engineering.instagram.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf -- Kindest regards. Mark Lawrence. -- https://mail.python.org/mailman/listinfo/python-list
Re: Garbage collection problem with generators
Sorry about breaking the rule. I'm just curios about this problem. And I'm using this workaround to prevent redundant resource creation. https://gist.githubusercontent.com/wooparadog/16948ca6c8ffb22214bf491a280406da/raw/- On Wed, Dec 28, 2016 at 9:12 PM Chris Angelico wrote: > On Wed, Dec 28, 2016 at 9:03 PM, Haochuan Guo > wrote: > > Anyone? The script to reproduce this problem is in: > > > > https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef > > > > On Fri, Dec 23, 2016 at 8:39 PM Haochuan Guo > wrote: > > > >> This is reproducible with python2.7, but not in python3.5. I've also > tried > >> with `thread` instead of `gevent`, it still happens. I'm guessing it's > >> related to garbage collection of generators. > >> > >> Did I bump into a python2 bug? Or am I simply wrong about the way to > close > >> generators...? > > (Please don't top-post.) > > Maybe the fix is to just use Python 3.5+? :) It probably is to do with > the garbage collection of generators; so you may want to consider > using something very explicit (eg a context manager) to ensure that > you call gen.close(). > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Garbage collection problem with generators
On Wed, Dec 28, 2016 at 9:03 PM, Haochuan Guo wrote: > Anyone? The script to reproduce this problem is in: > > https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef > > On Fri, Dec 23, 2016 at 8:39 PM Haochuan Guo wrote: > >> This is reproducible with python2.7, but not in python3.5. I've also tried >> with `thread` instead of `gevent`, it still happens. I'm guessing it's >> related to garbage collection of generators. >> >> Did I bump into a python2 bug? Or am I simply wrong about the way to close >> generators...? (Please don't top-post.) Maybe the fix is to just use Python 3.5+? :) It probably is to do with the garbage collection of generators; so you may want to consider using something very explicit (eg a context manager) to ensure that you call gen.close(). ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Garbage collection problem with generators
Anyone? The script to reproduce this problem is in:
https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef
On Fri, Dec 23, 2016 at 8:39 PM Haochuan Guo wrote:
> Hi, everyone
>
> I'm building a http long polling client for our company's discovery
> service and something weird happened in the following code:
>
> ```python
> while True:
> try:
> r = requests.get("url", stream=True, timeout=3)
> for data in r.iter_lines():
> processing_data...
> except TimeoutException:
> time.sleep(10)
> ```
>
> When I deliberately times out the request and then check the connections
> with `lsof -p process`, I discover that there are *two active
> connections*(ESTABLISH)
> instead of one. After digging around, it turns out it might not be the
> problem with `requests` at all, but gc related to generators.
>
> So I write this script to demonstrate the problem:
>
> https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef
>
> Function `A.a` will return a generator which will raise an exception. And
> in function `b`, I'm building new a new instance of `A` and iterate over
> the exception-raising generator. In the exception handler, I'll close the
> generator, delete it, delete the `A` instance, call `gc.collect()` and do
> the whole process all over again.
>
> There's another greenlet checking the `A` instances by using
> `gc.get_objects()`. It turns out there are always two `A` instances.
>
> This is reproducible with python2.7, but not in python3.5. I've also tried
> with `thread` instead of `gevent`, it still happens. I'm guessing it's
> related to garbage collection of generators.
>
> Did I bump into a python2 bug? Or am I simply wrong about the way to close
> generators...?
>
> Thanks
>
--
https://mail.python.org/mailman/listinfo/python-list
Garbage collection problem with generators
Hi, everyone
I'm building a http long polling client for our company's discovery service
and something weird happened in the following code:
```python
while True:
try:
r = requests.get("url", stream=True, timeout=3)
for data in r.iter_lines():
processing_data...
except TimeoutException:
time.sleep(10)
```
When I deliberately times out the request and then check the connections
with `lsof -p process`, I discover that there are *two active
connections*(ESTABLISH)
instead of one. After digging around, it turns out it might not be the
problem with `requests` at all, but gc related to generators.
So I write this script to demonstrate the problem:
https://gist.github.com/wooparadog/766f8007d4ef1227f283f1b040f102ef
Function `A.a` will return a generator which will raise an exception. And
in function `b`, I'm building new a new instance of `A` and iterate over
the exception-raising generator. In the exception handler, I'll close the
generator, delete it, delete the `A` instance, call `gc.collect()` and do
the whole process all over again.
There's another greenlet checking the `A` instances by using
`gc.get_objects()`. It turns out there are always two `A` instances.
This is reproducible with python2.7, but not in python3.5. I've also tried
with `thread` instead of `gevent`, it still happens. I'm guessing it's
related to garbage collection of generators.
Did I bump into a python2 bug? Or am I simply wrong about the way to close
generators...?
Thanks
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python garbage collection: not releasing memory to OS!
On 04/15/2016 05:25 AM, cshin...@gmail.com wrote: I have written an application with flask and uses celery for a long running task. While load testing I noticed that the celery tasks are not releasing memory even after completing the task. So I googled and found this group discussion.. https://groups.google.com/forum/#!topic/celery-users/jVc3I3kPtlw In that discussion it says, thats how python works. Also the article at https://hbfs.wordpress.com/2013/01/08/python-memory-management-part-ii/ says "But from the OS's perspective, your program's size is the total (maximum) memory allocated to Python. Since Python returns memory to the OS on the heap (that allocates other objects than small objects) only on Windows, if you run on Linux, you can only see the total memory used by your program increase." And I use Linux. So I wrote the below script to verify it. import gc def memory_usage_psutil(): # return the memory usage in MB import resource print 'Memory usage: %s (MB)' % (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1000.0) def fileopen(fname): memory_usage_psutil()# 10 MB f = open(fname) memory_usage_psutil()# 10 MB content = f.read() memory_usage_psutil()# 14 MB def fun(fname): memory_usage_psutil() # 10 MB fileopen(fname) gc.collect() memory_usage_psutil() # 14 MB import sys from time import sleep if __name__ == '__main__': fun(sys.argv[1]) for _ in range(60): gc.collect() memory_usage_psutil()#14 MB ... sleep(1) The input was a 4MB file. Even after returning from the 'fileopen' function the 4MB memory was not released. I checked htop output while the loop was running, the resident memory stays at 14MB. So unless the process is stopped the memory stays with it. So if the celery worker is not killed after its task is finished it is going to keep the memory for itself. I know I can use **max_tasks_per_child** config value to kill the process and spawn a new one. **Is there any other way to return the memory to OS from a python process?.** With situations like this, I normally just fork and do the mem intensive work in the child and then kill it off when done. Might be able to use a thread instead of a fork. But not sure how well all that would work with celery. --Sam -- https://mail.python.org/mailman/listinfo/python-list
Re: Python garbage collection: not releasing memory to OS!
On 04/15/2016 04:25 AM, cshin...@gmail.com wrote: > The input was a 4MB file. Even after returning from the 'fileopen' > function the 4MB memory was not released. I checked htop output while > the loop was running, the resident memory stays at 14MB. So unless > the process is stopped the memory stays with it. I guess the question is, why is this a problem? If there are no leaks, then I confess I don't understand what your concern is. And indeed you say it's not leaking as it never rises above 14 MB. Also there are ways of reading a file without allocating huge amounts of memory. Why not read it in in chunks, or in lines. Take advantage of Python's generator facilities to process your data. > So if the celery worker is not killed after its task is finished it > is going to keep the memory for itself. I know I can use > **max_tasks_per_child** config value to kill the process and spawn a > new one. **Is there any other way to return the memory to OS from a > python process?.** Have you tried using the subprocess module of python? If I understand it correctly, this would allow you to run python code as a subprocess (completely separate process), which would be completely reaped by the OS when it's finished. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python garbage collection: not releasing memory to OS!
On 15 April 2016 at 11:25, wrote: > The input was a 4MB file. Even after returning from the 'fileopen' function > the 4MB memory was not released. I checked htop output while the loop was > running, the resident memory stays at 14MB. So unless the process is stopped > the memory stays with it. When exactly memory gets freed to the OS is unclear but it's possible that your process can reuse the same bits of memory. The real question is whether continuously allocating and deallocating leads to steadily growing memory usage. If you change it so that your code calls fun inside the loop you will see that repeatedly calling fun does not lead to growing memory usage. > So if the celery worker is not killed after its task is finished it is going > to keep the memory for itself. I know I can use **max_tasks_per_child** > config value to kill the process and spawn a new one. **Is there any other > way to return the memory to OS from a python process?.** I don't really understand what you're asking here. You're running celery in a subprocess right? Is the problem about the memory used by subprocesses that aren't killed or is it the memory usage of the Python process? -- Oscar -- https://mail.python.org/mailman/listinfo/python-list
Python garbage collection: not releasing memory to OS!
I have written an application with flask and uses celery for a long running task. While load testing I noticed that the celery tasks are not releasing memory even after completing the task. So I googled and found this group discussion.. https://groups.google.com/forum/#!topic/celery-users/jVc3I3kPtlw In that discussion it says, thats how python works. Also the article at https://hbfs.wordpress.com/2013/01/08/python-memory-management-part-ii/ says "But from the OS's perspective, your program's size is the total (maximum) memory allocated to Python. Since Python returns memory to the OS on the heap (that allocates other objects than small objects) only on Windows, if you run on Linux, you can only see the total memory used by your program increase." And I use Linux. So I wrote the below script to verify it. import gc def memory_usage_psutil(): # return the memory usage in MB import resource print 'Memory usage: %s (MB)' % (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1000.0) def fileopen(fname): memory_usage_psutil()# 10 MB f = open(fname) memory_usage_psutil()# 10 MB content = f.read() memory_usage_psutil()# 14 MB def fun(fname): memory_usage_psutil() # 10 MB fileopen(fname) gc.collect() memory_usage_psutil() # 14 MB import sys from time import sleep if __name__ == '__main__': fun(sys.argv[1]) for _ in range(60): gc.collect() memory_usage_psutil()#14 MB ... sleep(1) The input was a 4MB file. Even after returning from the 'fileopen' function the 4MB memory was not released. I checked htop output while the loop was running, the resident memory stays at 14MB. So unless the process is stopped the memory stays with it. So if the celery worker is not killed after its task is finished it is going to keep the memory for itself. I know I can use **max_tasks_per_child** config value to kill the process and spawn a new one. **Is there any other way to return the memory to OS from a python process?.** -- https://mail.python.org/mailman/listinfo/python-list
Re: Python leaks in cyclic garbage collection
> Wild guess: > maybe when python exits they are called but sys.stdout has already been > closed and nothing gets written on it anymore. Certainly NOT. class Foo(): def __init__(self): self.b = Bar(self) def __del__(self): print "Free Foo" class Bar(): def __init__(self, f): self.f = f def __del__(self): print "Free Bar" f = Foo() print f.b.f.b.f.b.f.b.f.b.f.b.f.b.f.b.f print f.b.f.b.f.b.f.b.f.b.f.b.f.b.f.b.f.b *Console output:*** <__main__.Foo instance at 0x00AF7328> <__main__.Bar instance at 0x00AF7350> D:\v3> -- http://mail.python.org/mailman/listinfo/python-list
Re: Python leaks in cyclic garbage collection
Thanks for your answers! They really helped me out!! :) -- http://mail.python.org/mailman/listinfo/python-list
Re: Python leaks in cyclic garbage collection
Il giorno 19/feb/2011, alle ore 05.10, moerchendiser2k3 ha scritto: > Hi, I have some problems with Python and the garbage collection. In > the following piece of code I create a simple gargabe collection but I > am still wondering why the finalizers are never called - at least on > exit of Py they should be called somehow. What do I miss here? I know, > there is no deterministic way how to resolve this though. > > class Foo(): > >def __init__(self): >self.b=Bar(self) > >def __del__(self): >print "Free Foo" > > class Bar(): >def __init__(self, f): >self.f=f > >def __del__(self): >print "Free Bar" > > f=Foo() > print f > -- > http://mail.python.org/mailman/listinfo/python-list Wild guess: maybe when python exits they are called but sys.stdout has already been closed and nothing gets written on it anymore. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python leaks in cyclic garbage collection
On Fri, 18 Feb 2011 20:49:09 -0800, Chris Rebert wrote: > And following the pointer to gc.garbage's docs: > http://docs.python.org/library/gc.html#gc.garbage : "Objects that have > __del__() methods and are part of a reference cycle ***cause the entire > reference cycle to be uncollectable*** [...] Python doesn’t collect such > cycles automatically because, in general, it isn’t possible for Python > to guess a safe order in which to run the __del__() methods. [...] It’s > generally better to avoid the issue by not creating cycles containing > objects with __del__() methods" Another solution is to manually break the cycle, perhaps by providing a "close" method, and requiring the caller to use it. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Python leaks in cyclic garbage collection
On Fri, Feb 18, 2011 at 8:10 PM, moerchendiser2k3 wrote: > Hi, I have some problems with Python and the garbage collection. In > the following piece of code I create a simple gargabe collection but I > am still wondering why the finalizers are never called - at least on > exit of Py they should be called somehow. What do I miss here? Read The Fine Manual (all emphases added): http://docs.python.org/reference/datamodel.html#object.__del__ : "***It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.***" "Note: [...] x.__del__() — [...] is only called when x‘s reference count reaches zero. Some common situations that may prevent the reference count of an object from going to zero include: circular references between objects [...] Circular references which are garbage are detected when the option cycle detector is enabled (it’s on by default), ***but can only be cleaned up if there are no Python-level __del__() methods involved***. Refer to the documentation for the `gc` module for more information about how __del__() methods are handled by the cycle detector, particularly the description of the `garbage` value." And following the pointer to gc.garbage's docs: http://docs.python.org/library/gc.html#gc.garbage : "Objects that have __del__() methods and are part of a reference cycle ***cause the entire reference cycle to be uncollectable*** [...] Python doesn’t collect such cycles automatically because, in general, it isn’t possible for Python to guess a safe order in which to run the __del__() methods. [...] It’s generally better to avoid the issue by not creating cycles containing objects with __del__() methods" Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Python leaks in cyclic garbage collection
Thanks a lot for any help!!! Bye, moerchendiser2k3 -- http://mail.python.org/mailman/listinfo/python-list
Python leaks in cyclic garbage collection
Hi, I have some problems with Python and the garbage collection. In the following piece of code I create a simple gargabe collection but I am still wondering why the finalizers are never called - at least on exit of Py they should be called somehow. What do I miss here? I know, there is no deterministic way how to resolve this though. class Foo(): def __init__(self): self.b=Bar(self) def __del__(self): print "Free Foo" class Bar(): def __init__(self, f): self.f=f def __del__(self): print "Free Bar" f=Foo() print f -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about python garbage collection when keeping only a reference to an object's member
Some specialists tell that http://bestfinance-blog.com";>loan help people to live their own way, because they are able to feel free to buy necessary things. Moreover, various banks offer small business loan for young and old people. -- http://mail.python.org/mailman/listinfo/python-list
Where's the bug? (cPickle/thread edition) (was Re: How to optimize and monitor garbage collection?)
In article , Steve Holden wrote: > >[aside: in general, if you think your program is not working because of >a bug in Python, look harder at your program]. Good advice, I certainly agree with you. But sometimes it's not so simple. Right now, my company is running up against a problem with CherryPy because cPickle.dumps() doesn't release the GIL. You could argue that it's our fault for using threads, and you could also argue that we're having problems because our server has heavy memory contention (a dumps() that takes a couple of seconds on a laptop takes more than thirty seconds on the server). Nevertheless, I think it's also arguably a bug that dumps() doesn't release the GIL. (cPickle.dump() *does* release the GIL.) (Fortunately, we're savvy enough that it's easy for us to just make a local copy of cPickle that releases the GIL. Much easier than finding the problem in the first place...) -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ "Look, it's your affair if you want to play with five people, but don't go calling it doubles." --John Cleese anticipates Usenet -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about python garbage collection when keeping only a reference to an object's member
On 11/12/2010 2:03 PM George Burdell said... My understanding is that any object which is not pointed to by any variable will be automatically deleted. What if I create a class object, but only keep a reference to one of its members, and not a reference to the object itself? What goes on internally in Python? Does Python retain the whole object, or does it just keep a copy of the referenced member? For example, if I have def myclass: class myclass: def __init__(self): self.x = [1,2,3] self.y = [4,5,6] x = myclass().x This works, and I correctly get x = [1,2,3]. But what happened to the myclass() object initially created, and the member "y"? Garbage collect(ed|able) It only existed (was referenceable) while instantiated and can be gc'd one no longer referenced. Emile -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about python garbage collection when keeping only a reference to an object's member
On 11/12/2010 2:03 PM, George Burdell wrote: > My understanding is that any object which is not pointed to by any > variable will be automatically deleted. What if I create a class > object, but only keep a reference to one of its members, and not a > reference to the object itself? What goes on internally in Python? > Does Python retain the whole object, or does it just keep a copy of > the referenced member? > > For example, if I have > > def myclass: > def __init__(self): >self.x = [1,2,3] >self.y = [4,5,6] > x = myclass().x > > This works, and I correctly get x = [1,2,3]. But what happened to the > myclass() object initially created, and the member "y"? The myclass() call created a myclass instance (which contains a reference to a newly created list, which therefore has a refcount of 1). Attribute access to that instance then returns a second reference to the list, which is bound to the name x in the current local namespace (or the module global namespace if this is top-level code). The binding results in a reference count of two for the list. The instance, having no outstanding references, is then garbage-collected, which means that the objects referenced by its namespace have their reference counts decremented for each reference. So the destruction of the myclass instance reduces the reference count of the list [1, 2, 3] to one, and that of the list [4, 5, 6] to zero (which causes *it* to be garbage collected). And you are left with a local variable x that references a list whose current contents are [1, 2, 3]. Remember, the new values aren't created in local namespace. They are created in a shared area of memory (often referred to as the "heap"), so it's quite easy to "keep objects alive" that were created inside functions - just make sure you hold on to references outside the functions, and you are fine. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 PyCon 2011 Atlanta March 9-17 http://us.pycon.org/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie question about python garbage collection when keeping only a reference to an object's member
On 11/12/10 4:03 PM, George Burdell wrote: My understanding is that any object which is not pointed to by any variable will be automatically deleted. What if I create a class object, but only keep a reference to one of its members, and not a reference to the object itself? What goes on internally in Python? Does Python retain the whole object, or does it just keep a copy of the referenced member? For example, if I have def myclass: def __init__(self): self.x = [1,2,3] self.y = [4,5,6] x = myclass().x This works, and I correctly get x = [1,2,3]. But what happened to the myclass() object initially created, and the member "y"? They get deleted. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Newbie question about python garbage collection when keeping only a reference to an object's member
My understanding is that any object which is not pointed to by any variable will be automatically deleted. What if I create a class object, but only keep a reference to one of its members, and not a reference to the object itself? What goes on internally in Python? Does Python retain the whole object, or does it just keep a copy of the referenced member? For example, if I have def myclass: def __init__(self): self.x = [1,2,3] self.y = [4,5,6] x = myclass().x This works, and I correctly get x = [1,2,3]. But what happened to the myclass() object initially created, and the member "y"? -- http://mail.python.org/mailman/listinfo/python-list
Re: How to optimize and monitor garbage collection?
On 10/24/2010 8:39 PM, kj wrote: I'm designing a system that will be very memory hungry unless it is "garbage-collected" very aggressively. In the past I have had disappointing results with the gc module: I noticed practically no difference in memory usage with and without it. It is possible, however, that I was not measuring memory consumption adequately. What's the most accurate way to monitor memory consumption in a Python program, and thereby ensure that gc is working properly? Also, are there programming techniques that will result in better garbage collection? For example, would it help to explicitly call del on objects that should be gc'd? Python the language is not much concerned with memory. For an interpreter running on a computer, there are four memory sizes to be considered: the virtual memory assigned to the process; the physical memory assigned to the process; the physical memory used by Python objects; and the the memory used by 'active' objects accessible from program code. As far as I know, the OS can only see and report on the first and/or second. If the gap between the second and third (assigned and used physical memory) is large and includes 'blocks' that are totally unused, the interpreter *may* be able to return such blocks. But do not count on it. When this gap expands because the program deletes objects without returning blocks, people get fooled by OS reports of assigned memory not shrinking (even though used memory is). CPython tries to minimize the gap between all objects and active object both with reference counting and cyclic garbage collection (gc). Yes, you can help this along by judicious use of del and gc.collect. The goal should be to minimize the maximum active memory size. Reusing large arrays (rather than deleting and creating a new one) can sometimes help by avoiding fragmentation of allocated memory. Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: How to optimize and monitor garbage collection?
On 10/24/2010 8:39 PM, kj wrote: > What's the most accurate way to monitor memory consumption in a > Python program, and thereby ensure that gc is working properly? Trust me, it is. But don't forget that CPython doesn't actually *use* the garbage collector until you start to create cyclic data structures. In their absence reference counting alone is sufficient to ensure that unused object memory is reclaimed. regards Steve [aside: in general, if you think your program is not working because of a bug in Python, look harder at your program]. -- Steve Holden +1 571 484 6266 +1 800 494 3119 PyCon 2011 Atlanta March 9-17 http://us.pycon.org/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
How to optimize and monitor garbage collection?
I'm designing a system that will be very memory hungry unless it is "garbage-collected" very aggressively. In the past I have had disappointing results with the gc module: I noticed practically no difference in memory usage with and without it. It is possible, however, that I was not measuring memory consumption adequately. What's the most accurate way to monitor memory consumption in a Python program, and thereby ensure that gc is working properly? Also, are there programming techniques that will result in better garbage collection? For example, would it help to explicitly call del on objects that should be gc'd? TIA! ~kj -- http://mail.python.org/mailman/listinfo/python-list
Re: ctypes and garbage collection
> I'd add an "__owner" field to the node, initialised with the owning > container instance. I will - thank you! Joakim -- http://mail.python.org/mailman/listinfo/python-list
Re: ctypes and garbage collection
Joakim Hove wrote: > I have used ctypes to wrap a C-library > [...] > Observe that the c_container_get_node() function does _not_ allocate > memory, it just returns a opaque handle to a node structure, still > fully owned by the container structure. [...] > > class Container: > def __init__(self , filename): > self.c_ptr = c_container_alloc( filename ) > > def __del__( self ): > c_container_free( self.c_ptr ) > > def get_node( self , node_id): > Node( c_container_get_node( self , node_id )) > > > class Node: > def __init__( self , c_ptr ): > self.c_ptr = c_ptr > > def __del__( self ): > pass > > > Now, a use scenario might be like this: > > 1. Instantiate a Container() instance. > 2. Instantiate a Node() instance with the Container.get_node() > function. > 3. Forget about the Container instance and work happily with the Node > instance. > 4. Out of the blue comes the gc - and then? Will the Node instance be > enough to protect the Container instance from beeing garbage > collected? No. You should be able to even see that by logging calls to alloc/free of your library. > I thought maybe the get_node() function should have something like a > incref() call, and the Node.__del__() function a corresponding > decref()? Or ?? I'd add an "__owner" field to the node, initialised with the owning container instance. Uli -- Sator Laser GmbH Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 -- http://mail.python.org/mailman/listinfo/python-list
ctypes and garbage collection
Hello, I have used ctypes to wrap a C-library - it has been a really painless experience! The C-library instantiates a quite large "container-like" structure. There are then several functions to inspect the content of the container, get at items and free the whole thing: /* C - code */ c_container_type * c_container_alloc( const char * filename ); c_node_type * c_container_get_node( c_container_type * container , const char * node_id ); void c_container_free( c_container_type * container ); Observe that the c_container_get_node() function does _not_ allocate memory, it just returns a opaque handle to a node structure, still fully owned by the container structure. I have wrapped this with Python/ctypes roughly like this (severly trimmed pseudo code): class Container: def __init__(self , filename): self.c_ptr = c_container_alloc( filename ) def __del__( self ): c_container_free( self.c_ptr ) def get_node( self , node_id): Node( c_container_get_node( self , node_id )) class Node: def __init__( self , c_ptr ): self.c_ptr = c_ptr def __del__( self ): pass Now, a use scenario might be like this: 1. Instantiate a Container() instance. 2. Instantiate a Node() instance with the Container.get_node() function. 3. Forget about the Container instance and work happily with the Node instance. 4. Out of the blue comes the gc - and then? Will the Node instance be enough to protect the Container instance from beeing garbage collected? I thought maybe the get_node() function should have something like a incref() call, and the Node.__del__() function a corresponding decref()? Or ?? Regards Joakim Hove -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On 27-Aug-2010, at 2:14 AM, Brad wrote: > On Aug 25, 4:05 am, Alex McDonald wrote: >> Your example of writing code with >> memory leaks *and not caring because it's a waste of your time* makes >> me think that you've never been a programmer of any sort. > > "Windows applications are immune from memory leaks since programmers > can count on regular crashes to automatically release previously > allocated RAM." > -- > http://mail.python.org/mailman/listinfo/python-list Sorry if I may sound rude, but I have to do this on the windows applications comment - hahahahaha -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 25, 4:05 am, Alex McDonald wrote: > Your example of writing code with > memory leaks *and not caring because it's a waste of your time* makes > me think that you've never been a programmer of any sort. "Windows applications are immune from memory leaks since programmers can count on regular crashes to automatically release previously allocated RAM." -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 25, 4:01 pm, John Passaniti wrote: > On Aug 25, 5:01 pm, Joshua Maurice wrote: > > > I agree. Sadly, with managers, especially non-technical > > managers, it's hard to make this case when the weasel > > guy says "See! It's working.". > > Actually, it's not that hard. The key to communicating the true cost > of software development to non-technical managers (and even some > technical ones!) is to express the cost in terms of a metaphor they > can understand. Non-technical managers may not understand the > technology or details of software development, but they can probably > understand money. So finding a metaphor along those lines can help > them to understand. > > http://c2.com/cgi/wiki?WardExplainsDebtMetaphor > > I've found that explaining the need to improve design and code quality > in terms of a debt metaphor usually helps non-technical managers have > a very real, very concrete understanding of the problem. For example, > telling a non-technical manager that a piece of code is poorly written > and needs to be refactored may not resonate with them. To them, the > code "works" and isn't that the only thing that matters? But put in > terms of a debt metaphor, it becomes easier for them to see the > problem. But then it becomes a game of "How bad is this code exactly?" and "How much technical debt have we accrued?". At least in my company's culture, it is quite hard. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 25, 5:01 pm, Joshua Maurice wrote: > I agree. Sadly, with managers, especially non-technical > managers, it's hard to make this case when the weasel > guy says "See! It's working.". Actually, it's not that hard. The key to communicating the true cost of software development to non-technical managers (and even some technical ones!) is to express the cost in terms of a metaphor they can understand. Non-technical managers may not understand the technology or details of software development, but they can probably understand money. So finding a metaphor along those lines can help them to understand. http://c2.com/cgi/wiki?WardExplainsDebtMetaphor I've found that explaining the need to improve design and code quality in terms of a debt metaphor usually helps non-technical managers have a very real, very concrete understanding of the problem. For example, telling a non-technical manager that a piece of code is poorly written and needs to be refactored may not resonate with them. To them, the code "works" and isn't that the only thing that matters? But put in terms of a debt metaphor, it becomes easier for them to see the problem. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
John Passaniti writes: > On Aug 24, 8:00 pm, Hugh Aguilar wrote: >> The C programmers reading this are likely wondering why I'm being >> attacked. The reason is that Elizabeth Rather has made it clear to >> everybody that this is what she wants: [http://tinyurl.com/2bjwp7q] > > Hello to those outside of comp.lang.forth, where Hugh usually leaves > his slime trail. I seriously doubt many people will bother to read > the message thread Hugh references, but if you do, you'll get to > delight in the same nonsense Hugh has brought to comp.lang.forth. > Here's the compressed version: I did :-). I have somewhat followed Forth from a far, far distance since the 80's (including hardware), and did read several messages in the thread, also since it was not clear what Hugh was referring to. -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl & Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 25, 1:44 pm, John Passaniti wrote: > On Aug 24, 9:05 pm, Hugh Aguilar wrote: > > > What about using what I learned to write programs that work? > > Does that count for anything? > > It obviously counts, but it's not the only thing that matters. Where > I'm employed, I am currently managing a set of code that "works" but > the quality of that code is poor. The previous programmer suffered > from a bad case of cut-and-paste programming mixed with a > unsophisticated use of the language. The result is that this code > that "works" is a maintenance nightmare, has poor performance, wastes > memory, and is very brittle. The high level of coupling between code > means that when you change virtually anything, it invariably breaks > something else. > > And then you have the issue of the programmer thinking the code > "works" but it doesn't actually meet the needs of the customer. The > same code I'm talking about has a feature where you can pass message > over the network and have the value you pass configure a parameter. > It "works" fine, but it's not what the customer wants. The customer > wants to be able to bump the value up and down, not set it to an > absolute value. So does the code "work"? Depends on the definition > of "work." > > In my experience, there are a class of software developers who care > only that their code "works" (or more likely, *appears* to work) and > think that is the gold standard. It's an attitude that easy for > hobbyists to take, but not one that serious professionals can afford > to have. A hobbyist can freely spend hours hacking away and having a > grand time writing code. Professionals are paid for their efforts, > and that means that *someone* is spending both time and money on the > effort. A professional who cares only about slamming out code that > "works" is invariably merely moving the cost of maintaining and > extending the code to someone else. It becomes a hidden cost, but why > do they care... it isn't here and now, and probably won't be their > problem. I agree. Sadly, with managers, especially non-technical managers, it's hard to make this case when the weasel guy says "See! It's working.". -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 24, 9:05 pm, Hugh Aguilar wrote: > What about using what I learned to write programs that work? > Does that count for anything? It obviously counts, but it's not the only thing that matters. Where I'm employed, I am currently managing a set of code that "works" but the quality of that code is poor. The previous programmer suffered from a bad case of cut-and-paste programming mixed with a unsophisticated use of the language. The result is that this code that "works" is a maintenance nightmare, has poor performance, wastes memory, and is very brittle. The high level of coupling between code means that when you change virtually anything, it invariably breaks something else. And then you have the issue of the programmer thinking the code "works" but it doesn't actually meet the needs of the customer. The same code I'm talking about has a feature where you can pass message over the network and have the value you pass configure a parameter. It "works" fine, but it's not what the customer wants. The customer wants to be able to bump the value up and down, not set it to an absolute value. So does the code "work"? Depends on the definition of "work." In my experience, there are a class of software developers who care only that their code "works" (or more likely, *appears* to work) and think that is the gold standard. It's an attitude that easy for hobbyists to take, but not one that serious professionals can afford to have. A hobbyist can freely spend hours hacking away and having a grand time writing code. Professionals are paid for their efforts, and that means that *someone* is spending both time and money on the effort. A professional who cares only about slamming out code that "works" is invariably merely moving the cost of maintaining and extending the code to someone else. It becomes a hidden cost, but why do they care... it isn't here and now, and probably won't be their problem. > If I don't have a professor to pat me on the back, will my > programs stop working? What a low bar you set for yourself. Does efficiency, clarity, maintainability, extensibility, and elegance not matter to you? -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 24, 8:00 pm, Hugh Aguilar wrote:
> The C programmers reading this are likely wondering why I'm being
> attacked. The reason is that Elizabeth Rather has made it clear to
> everybody that this is what she wants: [http://tinyurl.com/2bjwp7q]
Hello to those outside of comp.lang.forth, where Hugh usually leaves
his slime trail. I seriously doubt many people will bother to read
the message thread Hugh references, but if you do, you'll get to
delight in the same nonsense Hugh has brought to comp.lang.forth.
Here's the compressed version:
1. Hugh references code ("symtab") that he wrote (in Factor) to
manage symbol tables.
2. I (and others) did some basic analysis and found it to be a poor
algorithm-- both in terms of memory use and performance-- especially
compared to the usual solutions (hash tables, splay trees, etc.).
3. I stated that symtab sucked for the intended application.
4. Hugh didn't like that I called his baby ugly and decided to expose
his bigotry.
5. Elizabeth Rather said she didn't appreciate Hugh's bigotry in the
newsgroup.
Yep, that's it. What Hugh is banking on is that you won't read the
message thread, and that you'll blindly accept that Elizabeth is some
terrible ogre with a vendetta against Hugh. The humor here is that
Hugh himself provides a URL that disproves that! So yes, if you care,
do read the message thread. It won't take long for you to get a clear
impression of Hugh's character.
--
http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On 19 Aug, 16:25, c...@tiac.net (Richard Harter) wrote: > On Wed, 18 Aug 2010 01:39:09 -0700 (PDT), Nick Keighley > wrote: > >On 17 Aug, 18:34, Standish P wrote: > >> How are these heaps being implemented ? Is there some illustrative > >> code or a book showing how to implement these heaps in C for example ? > > >any book of algorithms I'd have thought my library is currently inaccessible. Normally I'd have picked up Sedgewick and seen what he had to say on the subject. And possibly Knuth (though that requires taking more of a deep breath). Presumably Plauger's library book includes an implementation of malloc()/free() so that might be a place to start. > >http://en.wikipedia.org/wiki/Dynamic_memory_allocation > >http://www.flounder.com/inside_storage_allocation.htm > > >I've no idea how good either of these is serves me right for not checking :-( > The wikipedia page is worthless. odd really, you'd think basic computer science wasn't that hard... I found even wikipedia's description of a stack confusing and heavily biased towards implementation > The flounder page has > substantial meat, but the layout and organization is a mess. A > quick google search didn't turn up much that was general - most > articles are about implementations in specific environments. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
Alex McDonald writes: > Your example of writing code with >memory leaks *and not caring because it's a waste of your time* makes >me think that you've never been a programmer of any sort. Ever. Well, I find his approach towards memory leaks as described in <779b992b-7199-4126-bf3a-7ec40ea80...@j18g2000yqd.googlegroups.com> quite sensible, use something like that myself, and recommend it to others. Followups set to c.l.f (adjust as appropriate). - anton -- M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html New standard: http://www.forth200x.org/forth200x.html EuroForth 2010: http://www.euroforth.org/ef10/ -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On 25 Aug, 01:00, Hugh Aguilar wrote: > On Aug 24, 4:17 pm, Richard Owlett wrote: > > > Hugh Aguilar wrote: > > > [SNIP ;] > > > > The real problem here is that C, Forth and C++ lack automatic garbage > > > collection. If I have a program in which I have to worry about memory > > > leaks (as described above), I would be better off to ignore C, Forth > > > and C++ and just use a language that supports garbage collection. Why > > > should I waste my time carefully freeing up heap space? I will very > > > likely not find everything but yet have a few memory leaks anyway. > > > IOW Hugh has surpassed GIGO to achieve AGG - > > *A*utomatic*G*arbage*G*eneration ;) > > The C programmers reading this are likely wondering why I'm being > attacked. The reason is that Elizabeth Rather has made it clear to > everybody that this is what she > wants:http://groups.google.com/group/comp.lang.forth/browse_thread/thread/c... > > Every Forth programmer who aspires to get a job at Forth Inc. is > obliged to attack me. Attacking my software that I posted on the FIG > site is preferred, but personal attacks work too. It is a loyalty > test. Complete bollox. A pox on your persecution fantasies. This isn't about Elizabeth Rather or Forth Inc. It's about your massive ego and blind ignorance. Your example of writing code with memory leaks *and not caring because it's a waste of your time* makes me think that you've never been a programmer of any sort. Ever. In a commercial environment, your slide rule code would be rejected during unit testing, and you'd be fired and your code sent to the bit bucket. This isn't about CS BS; this is about making sure that banks accounts square, that planes fly, that nuclear reactors stay sub-critical; that applications can run 24 by 7, 365 days a year without requiring any human attention. So who designs and writes compilers for fail-safe systems? Who designs and writes operating systems that will run for years, non-stop? Where do they get the assurance that what they're writing is correct -- and provably so? From people that do research, hard math, have degrees, and design algorithms and develop all those other abstract ideas you seem so keen to reject as high-falutin' nonsense. I'd rather poke myself in the eye than run any of the crap you've written. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
Hugh Aguilar writes: > On Aug 24, 5:16 pm, Paul Rubin wrote: >> Anyway, as someone else once said, studying a subject like CS isn't done >> by reading. It's done by writing out answers to problem after problem. >> Unless you've been doing that, you haven't been studying. > > What about using what I learned to write programs that work? Does that > count for anything? No. Having put together a cupboard that holds some books without falling apart does not make you a carpenter, much less an architect. -- David Kastrup -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
John Bokma writes: > At an university which languages you see depend a lot on what your > teachers use themselves. A language is just a verhicle to get you from a > to b. Addendum: or to illustrate a concept (e.g. functional programming, oop) [..] > Like you, you mean? You consider yourself quite the expert on how people > educate and what they learn when educated in a formal > environment. Without (if I recall correctly) only second hand ^^^ Should've written "With", of course. > information and guessing. -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl & Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
Hugh Aguilar writes: > This is also the attitude that I find among college graduates. They > just believe what their professors told them in college, and there is > no why. Which college is that? It doesn't agree with my experiences. In CS quite a lot has to be proven with a formal proof, exactly the opposite from what you claim. And after some time students want to see the proof and certainly don't accept "there is no why!" unless it's a trivial thing. Maybe it's because your anecdote is an interpretation from a distance, not based on the actual experience? > This is essentially the argument being made above --- that C > is taught in college and Forth is not, therefore C is good and Forth > is bad --- THERE IS NO WHY! At an university which languages you see depend a lot on what your teachers use themselves. A language is just a verhicle to get you from a to b. What a good study should teach you is how to drive the verhicle without accidents and not that a red one is the best. From top of my head I've seen 20+ languages during my study at the University of Utrecht. Forth wasn't one of them, but I already knew about Forth before I went to the UU. On top of that I had written an extremely minimalistic Forth in Z80 assembly years before I went to the UU (based on the work of someone else). > People who promote "idiomatic" programming are essentially trying to > be Yoda. They want to criticize people even when those people's > programs work. "Works" doesn't mean that a program is good or what. There is a lot to say about a program that works, even one that works flawless. I do it all the time about my own programs. It's good to be critical about your own work. And if you're a teacher, it's good to provide positive feedback. > They are just faking up their own expertise --- Like you, you mean? You consider yourself quite the expert on how people educate and what they learn when educated in a formal environment. Without (if I recall correctly) only second hand information and guessing. > many of them have never actually written a program that works > themselves. Quite some part of CS can be done without writing a single line of code. > The reason why I like programming is because there is an inherent anti- > bullshit mechanism in programming. Your program either works or it > doesn't. Now can you provide a formal proof that it works, or do you just consider running the program a few times sufficient proof that "it works"? > If your program doesn't work, then it doesn't matter if it is > idiomatic, if you have a college degree, etc., etc.. That is the way I > see it, anyway. Well, you see it wrong. A program that doesn't work and is idiomatic is easier to make work and to verify by others that it works. A program that's the result of trial-and-error (that's what quite some people end up doing who are self-taught) is a pain in the ass (pardon my French) to maintain or to extend. > This perspective doesn't hold for much on > comp.lang.forth where we have people endlessly spouting blather > *about* programming, and you are different how? Also note that your post is crossposted to several other groups. > without actually doing any programming themselves. This is why I don't > take c.l.f. very seriously; people attack me all of the time and I > don't really care heh, hence all the replies you write, and mentioning it in this post. -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl & Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
Hugh Aguilar writes: > On Aug 24, 5:16 pm, Paul Rubin wrote: >> Anyway, as someone else once said, studying a subject like CS isn't done >> by reading. It's done by writing out answers to problem after problem. >> Unless you've been doing that, you haven't been studying. > > What about using what I learned to write programs that work? Does that > count for anything? Of course it does; but who's going to verify your program? > If I don't have a professor to pat me on the back, will my programs > stop working? That sounds more like magic than technology. I am sure you know what Paul means. As for patting on the back: you must make a hell of an effort to get that. -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl & Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 21, 10:57 pm, Steven D'Aprano wrote:
> Anyway, I'm looking forward to hear why overuse of the return stack is a
> big reason why people use GCC rather than Forth. (Why GCC? What about
> other C compilers?) Me, in my ignorance, I thought it was because C was
> invented and popularised by the same universities which went on to teach
> it to millions of programmers, and is firmly in the poplar and familiar
> Algol family of languages, while Forth barely made any impression on
> those universities, and looks like line-noise and reads like Yoda. (And
> I'm saying that as somebody who *likes* Forth and wishes he had more use
> for it.) In my experience, the average C programmer wouldn't recognise a
> return stack if it poked him in the eye.
"The Empire Strikes Back" was a popular movie. I read an article ("The
puppet like, I do not") criticizing the movie though. At one point,
Luke asked why something was true that Yoda had told him, and Yoda
replied: "There is no why!" The general idea is that the sudent (Luke)
was supposed to blindly accept what the professor (Yoda) tells him. If
he asks "why?," he gets yelled at.
This is also the attitude that I find among college graduates. They
just believe what their professors told them in college, and there is
no why. This is essentially the argument being made above --- that C
is taught in college and Forth is not, therefore C is good and Forth
is bad --- THERE IS NO WHY!
People who promote "idiomatic" programming are essentially trying to
be Yoda. They want to criticize people even when those people's
programs work. They are just faking up their own expertise --- many of
them have never actually written a program that works themselves.
The reason why I like programming is because there is an inherent anti-
bullshit mechanism in programming. Your program either works or it
doesn't. If your program doesn't work, then it doesn't matter if it is
idiomatic, if you have a college degree, etc., etc.. That is the way I
see it, anyway. This perspective doesn't hold for much on
comp.lang.forth where we have people endlessly spouting blather
*about* programming, without actually doing any programming
themselves. This is why I don't take c.l.f. very seriously; people
attack me all of the time and I don't really care --- I know that my
programs work, which is what matters in the real world.
(Pardon my use of the word "bullshit" above; there is no better term
available.)
--
http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 24, 5:16 pm, Paul Rubin wrote: > Anyway, as someone else once said, studying a subject like CS isn't done > by reading. It's done by writing out answers to problem after problem. > Unless you've been doing that, you haven't been studying. What about using what I learned to write programs that work? Does that count for anything? If I don't have a professor to pat me on the back, will my programs stop working? That sounds more like magic than technology. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
Paul Rubin writes: > Hugh Aguilar writes: >> I've read a lot of graduate-level CS books. > > Reading CS books doesn't make you a computer scientist any more than > listening to violin records makes you a violinist. Write out answers to > all the exercises in those books, and get your answers to the more > difficult ones checked by a professor, and you'll be getting somewhere. > That's the point someone else was making about self-study: without > someone checking your answers at first, it's easy to not learn to > recogize your own mistakes. > > Anyway, as someone else once said, studying a subject like CS isn't done > by reading. It's done by writing out answers to problem after problem. > Unless you've been doing that, you haven't been studying. Yup. I would like to add the following three: 1) being able to teach to peers what you've read. As explained in a post I made: during several courses I took you got a paper from your teacher and had to teach in front of the class the next week. Those papers are quite hard to grasp on the first reading even if you know quite a bit of the topic. Understanding it enough to teach in front of a class and being able to handle the question round, in which the teacher participates, is quite a killer. 2) being able to program on paper / understand programs on paper. On several exams I had to write small programs on paper. The solutions had to compile (i.e. missing a ; for languages that required so was counted against you, or using optional ;). One exam was about OOP and several OO languages were taught, and hence on paper one had to provide solutions in C++, Objective-C, Object Pascal, Smalltalk, Eiffel, etc. No compiler(s) handy. And of course questions like: what's wrong with this piece of code and how should it be written. 3) being able to write papers and a thesis (or two) No explanation needed, quite some people have no problem reading the required books, passing the exams, but need quite some time to do this (and some give up on it). -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl & Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
Hugh Aguilar writes: > I've read a lot of graduate-level CS books. Reading CS books doesn't make you a computer scientist any more than listening to violin records makes you a violinist. Write out answers to all the exercises in those books, and get your answers to the more difficult ones checked by a professor, and you'll be getting somewhere. That's the point someone else was making about self-study: without someone checking your answers at first, it's easy to not learn to recogize your own mistakes. Anyway, as someone else once said, studying a subject like CS isn't done by reading. It's done by writing out answers to problem after problem. Unless you've been doing that, you haven't been studying. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
Hugh Aguilar wrote: On Aug 24, 4:17 pm, Richard Owlett wrote: Hugh Aguilar wrote: [SNIP ;] The real problem here is that C, Forth and C++ lack automatic garbage collection. If I have a program in which I have to worry about memory leaks (as described above), I would be better off to ignore C, Forth and C++ and just use a language that supports garbage collection. Why should I waste my time carefully freeing up heap space? I will very likely not find everything but yet have a few memory leaks anyway. IOW Hugh has surpassed GIGO to achieve AGG - *A*utomatic*G*arbage*G*eneration ;) The C programmers reading this are likely wondering why I'm being attacked. The reason is that Elizabeth Rather has made it clear to everybody that this is what she wants: http://groups.google.com/group/comp.lang.forth/browse_thread/thread/c37b473ec4da66f1 Every Forth programmer who aspires to get a job at Forth Inc. is obliged to attack me. Attacking my software that I posted on the FIG site is preferred, but personal attacks work too. It is a loyalty test. *SNICKER SNICKER LOL* I am not now, nor have been a professional programmer. I still recognize you. P.S. - ever read "The Emperor's New Clothes" -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 24, 4:17 pm, Richard Owlett wrote: > Hugh Aguilar wrote: > > [SNIP ;] > > > The real problem here is that C, Forth and C++ lack automatic garbage > > collection. If I have a program in which I have to worry about memory > > leaks (as described above), I would be better off to ignore C, Forth > > and C++ and just use a language that supports garbage collection. Why > > should I waste my time carefully freeing up heap space? I will very > > likely not find everything but yet have a few memory leaks anyway. > > IOW Hugh has surpassed GIGO to achieve AGG - > *A*utomatic*G*arbage*G*eneration ;) The C programmers reading this are likely wondering why I'm being attacked. The reason is that Elizabeth Rather has made it clear to everybody that this is what she wants: http://groups.google.com/group/comp.lang.forth/browse_thread/thread/c37b473ec4da66f1 Every Forth programmer who aspires to get a job at Forth Inc. is obliged to attack me. Attacking my software that I posted on the FIG site is preferred, but personal attacks work too. It is a loyalty test. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
Hugh Aguilar writes: > On Aug 22, 11:12 am, John Bokma wrote: > >> And my >> experience is that a formal study in CS can't compare to home study >> unless you're really good and have the time and drive to read formal >> books written on CS. And my experience is that most self-educaters don't >> have that time. > > I've read a lot of graduate-level CS books. I think most self-educated > programmers have read more of these books than have 4-year degree > students who were not required to in order to get their Bachelors > degree and who were too busy during college to read anything that > wasn't required. I doubt it. But this all comes back to what I earlier wrote: those with a CS degree think they are better than people without, and people without think they can achieve the same or better by just buying a few books and reading them. On top of that, most of the people I knew in my final year were very fanatic regarding CS: it was a hobby to them. During coffeebreaks we talked about approximation algorithms for TSPs for example. Not always, but it happened. I read plenty of books during my studies that were not on the list, as did other students I knew. If I recall correctly, you don't have a CS degree. I do, and I can tell you that your /guess/ (since that is all it is) is wrong. For most exams I've done one had not only to have read the entire book (often in a very short time), but also the hand-outs. And for quite some courses additional material was given during the course itself, so not attending all classes could result in a lower score. Reading additional books and papers helped. Sometimes reading a book by a different author could be a real eye opener (and the students I had contact with did exactly this). On top of that, often in class excercises were done, and with some courses I had to hand in home work (yikes). Also, most books are easy to read compared to CS papers. In my final two years I did several courses which solely consisted of reading a CS paper and giving a presentation on the subject in front of your classmates (and sometimes other interested people). Reading and understanding such a paper is one (and quite an effort). Teaching it in front of a (small) class within a few days is not easy, to say the least. We also had to attend several talks by guest speakers. I went to more than the required number, including a guest talk by Linus. When there was a break-through in proving Fermat's last theorem there was a talk, which I attended, like several other class mates. I am sure there are students who are there just to get a degree and to make money. But my class mates didn't fall into that category, or I have missed something. So yes, I am convinced that there are plenty of self-educated people who can code circles around me or plenty of other people with a CS degree. But IMO those people are very hard to find. Most people overestimate their skills, with or without a degree; I am sure I do. And it wouldn't surprise me if self-educated people do this more so. >> On the other hand: some people I knew during my studies had no problem >> at all with introducing countless memory leaks in small programs (and >> turning off compiler warnings, because it gave so much noise...) > > I do this all the time. My slide-rule program, for example, has beau- > coup memory leaks. When I have time to mess with the program I clean > up these memory leaks, but it is not a big deal. The program just > runs, generates the gcode and PostScript, and then it is done. I don't > really worry about memory leaks except with programs that are run > continuously and have a user-interface, because they can eventually > run out of memory. Oh boy, I think you just made my point for me... > The real problem here is that C, Forth and C++ lack automatic garbage > collection. If I have a program in which I have to worry about memory > leaks (as described above), I would be better off to ignore C, Forth > and C++ and just use a language that supports garbage collection. Several languages that support garbage collection still are able to leak memory when circular datastructures are used (for example). Also, allocating memory and never giving it back (by keeping a reference to it) can also be memory leaking. And the wrong form of optimization can result in a program using more memory than necessary. On top of that, you have to understand when the gc releases memory, and things like memory fragmentation. In short: you still have to use your head (on some occasions even more). > Why should I waste my time carefully freeing up heap space? I will > very likely not find everything but yet have a few memory leaks > anyway. Why should you waste time with carefully checking for other issues? In my experience, once you become sloppy with one aspect it's
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 24, 9:24 am, David Kastrup wrote: > Anybody worth his salt in his profession has a trail of broken things in > his history. When I was employed as a Forth programmer, I worked for two brothers. The younger one told me a funny story about when he was 13 or 14 years old. He bought a radio at a garage sale. The radio worked perfectly, except that it had no case. He was mighty proud of his radio and was admiring it, but he noticed that the tubes were dusty. That wouldn't do! Such a wonderful radio ought to look as good as it sounds! So he removed the tubes and cleaned them all off with a soft cloth. At this time it occurred to him that maybe he should have kept track of which sockets the tubes had come out of. He put the tubes back in so that they looked correct, but he couldn't be sure. Fortunately, his older brother who was in high school knew *everything* about electronics, or at least, that is what he claimed. So the boy gets his big brother and asks him. The brother says: "There is one way to know for sure if the tubes are in correctly or not --- plug the radio in." He plugs in the radio; it makes a crackling noise and begins to smoke. The boy desperately yanks the cord, but it is too late; his wonderful radio is toast. The older brother says: "Now you know!" -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
Hugh Aguilar wrote: [SNIP ;] The real problem here is that C, Forth and C++ lack automatic garbage collection. If I have a program in which I have to worry about memory leaks (as described above), I would be better off to ignore C, Forth and C++ and just use a language that supports garbage collection. Why should I waste my time carefully freeing up heap space? I will very likely not find everything but yet have a few memory leaks anyway. IOW Hugh has surpassed GIGO to achieve AGG - *A*utomatic*G*arbage*G*eneration ;) -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 22, 11:12 am, John Bokma wrote: > And my > experience is that a formal study in CS can't compare to home study > unless you're really good and have the time and drive to read formal > books written on CS. And my experience is that most self-educaters don't > have that time. I've read a lot of graduate-level CS books. I think most self-educated programmers have read more of these books than have 4-year degree students who were not required to in order to get their Bachelors degree and who were too busy during college to read anything that wasn't required. > On the other hand: some people I knew during my studies had no problem > at all with introducing countless memory leaks in small programs (and > turning off compiler warnings, because it gave so much noise...) I do this all the time. My slide-rule program, for example, has beau- coup memory leaks. When I have time to mess with the program I clean up these memory leaks, but it is not a big deal. The program just runs, generates the gcode and PostScript, and then it is done. I don't really worry about memory leaks except with programs that are run continuously and have a user-interface, because they can eventually run out of memory. The real problem here is that C, Forth and C++ lack automatic garbage collection. If I have a program in which I have to worry about memory leaks (as described above), I would be better off to ignore C, Forth and C++ and just use a language that supports garbage collection. Why should I waste my time carefully freeing up heap space? I will very likely not find everything but yet have a few memory leaks anyway. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
David Kastrup writes: > John Bokma writes: > >> On the other hand: some people I knew during my studies had no problem >> at all with introducing countless memory leaks in small programs (and >> turning off compiler warnings, because it gave so much noise...) > > [...] > >> As for electrical engineering: done that (BSc) and one of my class >> mates managed to connect a transformer the wrong way >> around twice. Yet he had the highest mark in our class. > > Anybody worth his salt in his profession has a trail of broken things in > his history. Sure. The long version is: he blew up his work when he connected the transformer wrong. He borrowed someone else's board and blew that one up as well. > The faster it thinned out, the better he learned. He he he, his internships went along similar lines. Maybe he loved to blow up things. > The only reliable way never to break a thing is not to touch it in the > first place. But that will not help you if it decides to break on its > own. I don't think transfomers connect themselfs in the wrong way ;-). I agree with that accidents do happen, but some people just manage to make accidents happen way above average. And in that case they might start to think if it's a good idea them touching things. -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl & Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 21, 12:18 pm, ehr...@dk3uz.ampr.org (Edmund H. Ramm) wrote: > In <2d59bfaa-2aa5-4396-bd03-22200df8c...@x21g2000yqa.googlegroups.com> Hugh > Aguilar writes: > > > [...] > > I really recommend that people spend a lot more time writing code, > > and a lot less time with all of this pseudo-intellectual nonsense. > > [...] > > I energetically second that! > -- > e-mail: dk3uz AT arrl DOT net | AMPRNET: dk...@db0hht.ampr.org > If replying to a Usenet article, please use above e-mail address. > Linux/m68k, the best U**x ever to hit an Atari! What open-source code have you posted publicly? BTW, why did you request that your post not be archived, and be removed in a few days? That doesn't seem very energetic. Also, now that I've responded to it, it will be archived forever. It is so rare that anybody agrees with me, I wanted to make a permanent record. :-) -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
David Kastrup wrote: John Bokma writes: On the other hand: some people I knew during my studies had no problem at all with introducing countless memory leaks in small programs (and turning off compiler warnings, because it gave so much noise...) [...] As for electrical engineering: done that (BSc) and one of my class mates managed to connect a transformer the wrong way around twice. Yet he had the highest mark in our class. Anybody worth his salt in his profession has a trail of broken things in his history. The faster it thinned out, the better he learned. The only reliable way never to break a thing is not to touch it in the first place. But that will not help you if it decides to break on its own. *LOL* !!! I remember the day a very senior field service engineer for a multi-national minicomputer mfg plugged 16k (or was it 32k) of core (back when a core was visible to naked eye ;) the wrong way into a backplane. After the smoke cleared ... snicker snicker. I also remember writing a failure report because someone installed a grounding strap 100 degrees out of orientation on a piece of multi kV switchgear.(don't recall nominal capacity, buck backup generator was rated for 1.5 MW continuous ;) P.S. failure was demonstrated as manufacturer's senior sales rep was demonstrating how easy it was to do maintenance on the system. There were times I had fun writing up inspection reports. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
John Bokma writes: > On the other hand: some people I knew during my studies had no problem > at all with introducing countless memory leaks in small programs (and > turning off compiler warnings, because it gave so much noise...) [...] > As for electrical engineering: done that (BSc) and one of my class > mates managed to connect a transformer the wrong way > around twice. Yet he had the highest mark in our class. Anybody worth his salt in his profession has a trail of broken things in his history. The faster it thinned out, the better he learned. The only reliable way never to break a thing is not to touch it in the first place. But that will not help you if it decides to break on its own. -- David Kastrup -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On 24 Aug, 01:00, Hugh Aguilar wrote: > On Aug 21, 12:32 pm, Alex McDonald wrote: > > > "Scintilla" gets about 2,080,000 results on google; "blather" gets > > about 876,000 results. O Hugh, you pseudo-intellectual you! > > > > with gutter language such as > > > "turd" > > > About 5,910,000 results. It has a long history, even getting a mention > > in the Wyclif's 13th century bible. > > You looked up "blather" and "turd" on google *AND* you are not a > pseudo-intellectual??? That is funny! > > I don't consider myself to be a pseudo-intellectual. I don't have any > education however, so a pseudo-intellectual is the only kind of > intellectual that I could be. I don't have any formal CS education, nor a degree in anything else. But that doesn't make me an anti-intellectual by instinct (the instinct would be jealousy, I guess), nor does it stop me from learning. Or using Google, something I'm sure you do too. We have a great degree of admiration and fondness for intellectuals in Europe; the French in particular hold them in very high regard. Perhaps disdain of learning and further education is peculiar to a certain section of American society, as the label "intellectual" (often, "liberal intellectual") appears to be used as a derogatory term. I have no idea what a pseudo-intellectual might be, but it's evident you mean it in much the same way. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 22, 3:40 pm, 1001nuits <1001nu...@gmail.com> wrote: > Another thing you learn in studying in University is the fact that you can > be wrong, which is quite difficult to accept for self taught people. Yet another thing you learn in studying in University, is the art of apple polishing! LOL If a person has graduated from college, it is not clear what if anything he has learned of a technical nature --- but it can be assumed that he has learned to be a head-bobber (someone who habitually bobs his head up and down in agreement when the boss is speaking) and has learned to readily admit to being wrong when pressured (when the boss looks at him without smiling for more than two seconds). These are the traits that bosses want in an employee --- that prove the employee to be "trainable." BTW, has anybody actually looked at my software? http://www.forth.org/novice.html All this pseudo-intellectual nonsense (including this post) is getting boring. Why don't we try discussing software for a while? I wrote that slide-rule program as a showcase of Forth. I've been thinking of porting it over to another language, possibly C. Maybe one of you C experts could write the C program though, as a comparison --- to show how much better C is than Forth. You can demonstrate that my code was badly written and strangely designed --- with a concrete example, rather than just a lot hand-waving and chest-thumping. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 21, 12:32 pm, Alex McDonald wrote: > "Scintilla" gets about 2,080,000 results on google; "blather" gets > about 876,000 results. O Hugh, you pseudo-intellectual you! > > > with gutter language such as > > "turd" > > About 5,910,000 results. It has a long history, even getting a mention > in the Wyclif's 13th century bible. You looked up "blather" and "turd" on google *AND* you are not a pseudo-intellectual??? That is funny! I don't consider myself to be a pseudo-intellectual. I don't have any education however, so a pseudo-intellectual is the only kind of intellectual that I could be. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
Le Sun, 22 Aug 2010 20:12:36 +0200, John Bokma a écrit: David Kastrup writes: John Bokma writes: David Kastrup writes: John Passaniti writes: Amen! All this academic talk is useless. Who cares about things like the big-O notation for program complexity. Can't people just *look* at code and see how complex it is?! And take things like the years of wasted effort computer scientists have put into taking data structures (like hashes and various kinds of trees) and extending them along various problem domains and requirements. Real programmers don't waste their time with learning that junk. What good did any of that ever do anyone?! It is my experience that in particular graduated (and in particular Phd) computer scientists don't waste their time _applying_ that junk. Question: do you have a degree in computer science? Since in my experience: people who talk about their experience with graduated people often missed the boat themselves and think that reading a book or two equals years of study. I have a degree in electrical engineering. But that's similarly irrelevant. Nah, it's not: your attitude towards people with a degree in computer science agrees with what I wrote. That has not particularly helped my respect towards CS majors and PhDs in the function of programmers (and to be honest: their education is not intended to make them good programmers, but to enable them to _lead_ good programmers). I disagree. That does not mean that I am incapable of analyzing, say quicksort and mergesort, Oh, that's what I was not implying. I am convinced that quite some people who do self-study can end up with better understanding of things than people who do it for a degree. I have done both: I already was programming in several languages before I was studying CS. And my experience is that a formal study in CS can't compare to home study unless you're really good and have the time and drive to read formal books written on CS. And my experience is that most self-educaters don't have that time. On the other hand: some people I knew during my studies had no problem at all with introducing countless memory leaks in small programs (and turning off compiler warnings, because it gave so much noise...) Donald Knuth never studied computer science. Yes, yes, and Albert Einstein worked at an office. Those people are very rare. But my experience (see for plenty of examples: Slashdot) is that quite some people who don't have a degree think that all that formal education is just some paper pushing and doesn't count. While some of those who do have the paper think they know it all. Those people who are right in either group are a minority in my experience. As for electrical engineering: done that (BSc) and one of my class mates managed to connect a transformer the wrong way around twice. Yet he had the highest mark in our class. So in short: yes, self-study can make you good at something. But self-study IMO is not in general a replacement for a degree. Someone who can become great after self-study would excel at a formal study and learn more. Study works best if there is competition and if there are challenges. I still study a lot at home, but I do miss the challenges and competition. Hi all, I quite agree with the fact that self learning is not enough. Another thing you learn in studying in University is the fact that you can be wrong, which is quite difficult to accept for self taught people. When you work in groups, you are bound to admit that you don't have the best solution all the time. To my experience, self-taught people I worked with had tremendous difficulties to accept that they were wrong, that their design was badly done, that their code was badly written or strangely designed. Because self teaching was done with a lot of efforts, in particular to figure out complex problems on their own. Most of the time, the self learned people are attached to the things they learned by themselves and have difficulties to envisage that being right of wrong is often not an issue provided the group comes to the best option. They often live contradiction as a personal offense while it is just work, you know. That's another interest of the degree, confrontation with other people that have the same background. And letting the things learned at the place they should be and not in the affective area. 1001 -- Utilisant le logiciel de courrier révolutionnaire d'Opera : http://www.opera.com/mail/ -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
David Kastrup writes: > John Bokma writes: > >> David Kastrup writes: >> >>> John Passaniti writes: >>> Amen! All this academic talk is useless. Who cares about things like the big-O notation for program complexity. Can't people just *look* at code and see how complex it is?! And take things like the years of wasted effort computer scientists have put into taking data structures (like hashes and various kinds of trees) and extending them along various problem domains and requirements. Real programmers don't waste their time with learning that junk. What good did any of that ever do anyone?! >>> >>> It is my experience that in particular graduated (and in particular Phd) >>> computer scientists don't waste their time _applying_ that junk. >> >> Question: do you have a degree in computer science? >> >> Since in my experience: people who talk about their experience with >> graduated people often missed the boat themselves and think that reading >> a book or two equals years of study. > > I have a degree in electrical engineering. But that's similarly > irrelevant. Nah, it's not: your attitude towards people with a degree in computer science agrees with what I wrote. > That has not particularly helped my respect towards CS majors and PhDs > in the function of programmers (and to be honest: their education is not > intended to make them good programmers, but to enable them to _lead_ > good programmers). I disagree. > That does not mean that I am incapable of analyzing, say quicksort and > mergesort, Oh, that's what I was not implying. I am convinced that quite some people who do self-study can end up with better understanding of things than people who do it for a degree. I have done both: I already was programming in several languages before I was studying CS. And my experience is that a formal study in CS can't compare to home study unless you're really good and have the time and drive to read formal books written on CS. And my experience is that most self-educaters don't have that time. On the other hand: some people I knew during my studies had no problem at all with introducing countless memory leaks in small programs (and turning off compiler warnings, because it gave so much noise...) > Donald Knuth never studied computer science. Yes, yes, and Albert Einstein worked at an office. Those people are very rare. But my experience (see for plenty of examples: Slashdot) is that quite some people who don't have a degree think that all that formal education is just some paper pushing and doesn't count. While some of those who do have the paper think they know it all. Those people who are right in either group are a minority in my experience. As for electrical engineering: done that (BSc) and one of my class mates managed to connect a transformer the wrong way around twice. Yet he had the highest mark in our class. So in short: yes, self-study can make you good at something. But self-study IMO is not in general a replacement for a degree. Someone who can become great after self-study would excel at a formal study and learn more. Study works best if there is competition and if there are challenges. I still study a lot at home, but I do miss the challenges and competition. -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl & Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
John Bokma writes: > David Kastrup writes: > >> John Passaniti writes: >> >>> Amen! All this academic talk is useless. Who cares about things like >>> the big-O notation for program complexity. Can't people just *look* >>> at code and see how complex it is?! And take things like the years of >>> wasted effort computer scientists have put into taking data structures >>> (like hashes and various kinds of trees) and extending them along >>> various problem domains and requirements. Real programmers don't >>> waste their time with learning that junk. What good did any of that >>> ever do anyone?! >> >> It is my experience that in particular graduated (and in particular Phd) >> computer scientists don't waste their time _applying_ that junk. > > Question: do you have a degree in computer science? > > Since in my experience: people who talk about their experience with > graduated people often missed the boat themselves and think that reading > a book or two equals years of study. I have a degree in electrical engineering. But that's similarly irrelevant. I have a rather thorough background with computers (started with punched cards), get along with about a dozen assembly languages and quite a few other higher level languages. I've had to write the BIOS for my first computer and a number of other stuff and did digital picture enhancement on DOS computers with EMM (programming 80387 assembly language and using a variant of Hartley transforms). I have rewritten digital map processing code from scratch that has been designed and optimized by graduated computer scientists (including one PhD) to a degree where it ran twice as fast as originally, at the cost of occasional crashes and utter unmaintainability. Twice as fast meaning somewhat less than a day of calculation time for medium size data sets (a few 10 of data points, on something like a 25MHz 68020 or something). So I knew the problem was not likely to be easy. Took me more than a week. After getting the thing to compile and fixing the first few crashing conditions, I got stuck in debugging. The thing just terminated after about 2 minutes of runtime without an apparent reason. I spent almost two more days trying to find the problem before bothering to even check the output. The program just finished regularly. That has not particularly helped my respect towards CS majors and PhDs in the function of programmers (and to be honest: their education is not intended to make them good programmers, but to enable them to _lead_ good programmers). That does not mean that I am incapable of analyzing, say quicksort and mergesort, and come up with something reasonably close to a closed form for average, min, and max comparisons (well, unless a close approximation is good enough, you have to sum about lg n terms which is near instantaneous, with a real closed form mostly available when n is special, like a power of 2). And I know how to work with more modern computer plagues, like the need for cache coherency. So in short, I have a somewhat related scientific education, but I can work the required math. And I can work the computers. > Oh, and rest assured, it works both ways: people who did graduate are > now and then thinking it's the holy grail and no body can beat it with > home study. > > Both are wrong, by the way. Depends. In my personal opinion, living close to the iron and being sharp enough can make a lot of a difference. Donald Knuth never studied computer science. He more or less founded it. As a programmer, he is too much artist and too little engineer for my taste: you can't take his proverbial masterpiece "TeX" apart without the pieces crumbling. He won't write inefficient programs: he has the respective gene and the knowledge to apply it. But the stuff he wrote is not well maintainable and reusable. Of course, he has no need for reuse if he can rewrite as fast as applying an interface. -- David Kastrup -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
Oh, I am so going to regret getting sucked into this tarpit... oh well. On Sat, 21 Aug 2010 09:58:18 -0700, Hugh Aguilar wrote: > The > following is a pretty good example, in which Alex mixes big pseudo- > intellectual words such as "scintilla" with gutter language such as > "turd" in an ungrammatical mish-mash You say that like it's a bad thing. Besides, scintilla isn't a "big pseudo-intellectual" word. It might seem so to those whose vocabulary (that's another big word, like "patronizing" and "fatuousness") is lacking, but it's really quite a simple word. It means "a spark", hence "scintillating", as in "he thinks he's quite the scintillating wit, and he's half right". It also means "an iota, a smidgen, a scarcely detectable amount", and if anyone can't see the connection between a spark and a smidgen, there's probably no hope for them. Nothing intellectual about it, let alone pseudo-intellectual, except that it comes from Latin. But then so do well more half the words in the English language. Anyway, I'm looking forward to hear why overuse of the return stack is a big reason why people use GCC rather than Forth. (Why GCC? What about other C compilers?) Me, in my ignorance, I thought it was because C was invented and popularised by the same universities which went on to teach it to millions of programmers, and is firmly in the poplar and familiar Algol family of languages, while Forth barely made any impression on those universities, and looks like line-noise and reads like Yoda. (And I'm saying that as somebody who *likes* Forth and wishes he had more use for it.) In my experience, the average C programmer wouldn't recognise a return stack if it poked him in the eye. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 21, 3:36 am, David Kastrup wrote: > > I think there must be some programmer gene. It is not enough to be able > to recognize O(n^k) or worse (though it helps having a more exact rather > than a fuzzy notion of them _if_ you have that gene). Some of the best minds in comp.lang.forth have a penchant for sarcasm - one of the reasons I always read their posts. Maybe it gets lost on the international crowd, but I love it. -Brad -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
David Kastrup writes: > John Passaniti writes: > >> Amen! All this academic talk is useless. Who cares about things like >> the big-O notation for program complexity. Can't people just *look* >> at code and see how complex it is?! And take things like the years of >> wasted effort computer scientists have put into taking data structures >> (like hashes and various kinds of trees) and extending them along >> various problem domains and requirements. Real programmers don't >> waste their time with learning that junk. What good did any of that >> ever do anyone?! > > It is my experience that in particular graduated (and in particular Phd) > computer scientists don't waste their time _applying_ that junk. Question: do you have a degree in computer science? Since in my experience: people who talk about their experience with graduated people often missed the boat themselves and think that reading a book or two equals years of study. Oh, and rest assured, it works both ways: people who did graduate are now and then thinking it's the holy grail and no body can beat it with home study. Both are wrong, by the way. -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl & Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On 21 Aug, 17:58, Hugh Aguilar wrote: > On Aug 21, 5:29 am, Alex McDonald wrote: > > > On 21 Aug, 06:42, Standish P wrote: > > > Admittedly, I am asking a question that would be thought > > > provoking to those who claim to be "experts" but these experts are > > > actually very stingy and mean business people, most certainly worse > > > than Bill Gates, only it did not occur to them his ideas and at the > > > right time. > > > What surprises may is that anyone bothered to answer, as your question > > was neither "thought provoking" nor in need of attention from an > > expert. Their generosity in the face of so much stupidity stands out > > as remarkable. > > I wouldn't call the OP "stupid," which is just mean-spirited. Perhaps I'm just getting less forgiving the older I get, or the more I read here. The internet is a fine resource for research, and tools like google, archivx and so on are easy to access and take but a little effort to use. > That is > not much of a welcome wagon for somebody who might learn Forth > eventually and join our rather diminished ranks. I care neither to be included in your "diminished ranks", nor do I take much regard of popularity as you define it. Standish P doesn't want to join anything; he (like you) has an agenda for yet another club with a membership of one. > Lets go with "over- > educated" instead! I thought that his question was vague. It seemed > like the kind of question that students pose to their professor in > class to impress him with their thoughtfulness, so that he'll forget > that they never did get any of their homework-assignment programs to > actually work. It didn't work. He hasn't done any homework, neither do you, and it shows. > I yet maintain that writing programs is what > programming is all about. You remind me of those that would build a house without an architect, or fly without bothering to study the weather. > > I see a lot of pseudo-intellectual blather on comp.lang.forth. The > following is a pretty good example, in which Alex mixes big pseudo- > intellectual words such as "scintilla" "Scintilla" gets about 2,080,000 results on google; "blather" gets about 876,000 results. O Hugh, you pseudo-intellectual you! > with gutter language such as > "turd" About 5,910,000 results. It has a long history, even getting a mention in the Wyclif's 13th century bible. > in an ungrammatical mish-mash --- and defends the overuse of > the return stack for holding temporary data as being readable(?!): I did? Where? You're making stuff up. Again. > http://groups.google.com/group/comp.lang.forth/browse_thread/thread/4... > > On Jul 23, 4:43 pm, Alex McDonald wrote: > > > Whereas yours contained several tens, and nearly every one of them is > > wrong. Hugh, do you actually have any evidence -- even a scintilla -- > > that supports this log winded opinions-as-fact post? Take any of the > > statements you make, and demonstrate that you can justify it. > > Reminding us that you said it before doesn't count. > > > Start with this turd of an assertion and see if you can polish it; > > "Most of the time, when Forth code gets really ugly, it is because of > > an overuse of >R...R> --- that is a big reason why people use GCC > > rather than Forth." > Something you never did address, probably because the statement you made is just another symptom of Aguilar's Disease; presenting as fact an opinion based on personal experience, limited observation and no research. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On Aug 21, 5:29 am, Alex McDonald wrote: > On 21 Aug, 06:42, Standish P wrote: > > Admittedly, I am asking a question that would be thought > > provoking to those who claim to be "experts" but these experts are > > actually very stingy and mean business people, most certainly worse > > than Bill Gates, only it did not occur to them his ideas and at the > > right time. > > What surprises may is that anyone bothered to answer, as your question > was neither "thought provoking" nor in need of attention from an > expert. Their generosity in the face of so much stupidity stands out > as remarkable. I wouldn't call the OP "stupid," which is just mean-spirited. That is not much of a welcome wagon for somebody who might learn Forth eventually and join our rather diminished ranks. Lets go with "over- educated" instead! I thought that his question was vague. It seemed like the kind of question that students pose to their professor in class to impress him with their thoughtfulness, so that he'll forget that they never did get any of their homework-assignment programs to actually work. I yet maintain that writing programs is what programming is all about. I see a lot of pseudo-intellectual blather on comp.lang.forth. The following is a pretty good example, in which Alex mixes big pseudo- intellectual words such as "scintilla" with gutter language such as "turd" in an ungrammatical mish-mash --- and defends the overuse of the return stack for holding temporary data as being readable(?!): http://groups.google.com/group/comp.lang.forth/browse_thread/thread/4b9f67406c6852dd/0218831f02564410 On Jul 23, 4:43 pm, Alex McDonald wrote: > Whereas yours contained several tens, and nearly every one of them is > wrong. Hugh, do you actually have any evidence -- even a scintilla -- > that supports this log winded opinions-as-fact post? Take any of the > statements you make, and demonstrate that you can justify it. > Reminding us that you said it before doesn't count. > > Start with this turd of an assertion and see if you can polish it; > "Most of the time, when Forth code gets really ugly, it is because of > an overuse of >R...R> --- that is a big reason why people use GCC > rather than Forth." -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On 21 Aug, 06:42, Standish P wrote: > On Aug 20, 3:51 pm, Hugh Aguilar wrote: > > > > > On Aug 18, 6:23 pm, Standish P wrote: > > > > On Aug 17, 6:38 pm, John Passaniti wrote: > > > > > You asked if Forth "borrowed" lists from Lisp. It did not. In Lisp, > > > > lists are constructed with pair of pointers called a "cons cell". > > > > That is the most primitive component that makes up a list. Forth has > > > > no such thing; in Forth, the dictionary (which is traditionally, but > > > > not necessarily a list) is a data structure that links to the previous > > > > word with a pointer. > > > > Would you show me a picture, ascii art or whatever for Forth ? I know > > > what lisp lists look like so I dont need that for comparison. Forth > > > must have a convention and a standard or preferred practice for its > > > dicts. However, let me tell you that in postscript the dictionaries > > > can be nested inside other dictionaries and any such hiearchical > > > structure is a nested associative list, which is what linked list, > > > nested dictionaries, nested tables are. > > > You can see an example of lists in my novice package (in the list.4th > > file):http://www.forth.org/novice.html > > Also in there is symtab, which is a data structure intended to be used > > for symbol tables (dictionaries). Almost nobody uses linked lists for > > the dictionary anymore (the FIG compilers of the 1970s did, but they > > are obsolete). > > > I must say, I've read through this entire thread and I didn't > > understand *anything* that *anybody* was saying (especially the OP). > > You didnt understand anything because no one explained anything > coherently. It indicates that you're asking a question that *you don't understand*. I'm continually amazed that people come to Usenet, wikis, websites and other fora and ask questions that even the most basic of research (and a bit of care with terminology aka "using the right words") would show to be confused. A quick scan of the available literature on garbage collection and stacks, starting with the fundamentals, would surely show you what you need to know. > Admittedly, I am asking a question that would be thought > provoking to those who claim to be "experts" but these experts are > actually very stingy and mean business people, most certainly worse > than Bill Gates, only it did not occur to them his ideas and at the > right time. > What surprises may is that anyone bothered to answer, as your question was neither "thought provoking" nor in need of attention from an expert. Their generosity in the face of so much stupidity stands out as remarkable. -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
John Passaniti writes: > Amen! All this academic talk is useless. Who cares about things like > the big-O notation for program complexity. Can't people just *look* > at code and see how complex it is?! And take things like the years of > wasted effort computer scientists have put into taking data structures > (like hashes and various kinds of trees) and extending them along > various problem domains and requirements. Real programmers don't > waste their time with learning that junk. What good did any of that > ever do anyone?! It is my experience that in particular graduated (and in particular Phd) computer scientists don't waste their time _applying_ that junk. They have learnt to analyze it, they could tell you how bad their own algorithms are (if they actually bothered applying their knowledge), but it does not occur to them to replace them by better ones. Or even factor their solutions in a way that the algorithms and data structures are actually isolated. I think there must be some programmer gene. It is not enough to be able to recognize O(n^k) or worse (though it helps having a more exact rather than a fuzzy notion of them _if_ you have that gene). You have to fear it. It has to hurt. You need to feel compassion with the CPU. It's not enough to sit there in your easychair, occasionally sucking on your pipeline and listen to its story about a hard realtime youth and its strained connection to its motherboard. When it stops, you have to see its benchmarks and feel their pain in your own backplane. -- David Kastrup -- http://mail.python.org/mailman/listinfo/python-list
