On 11/15/05, Jeremy Hylton <[EMAIL PROTECTED]> wrote: > On 11/15/05, Niko Matsakis <[EMAIL PROTECTED]> wrote: > > > As Neal pointed out, it's tricky to write code for the AST parser > > > and compiler > > > without accidentally letting memory leak when the parser or > > > compiler runs into > > > a problem and has to bail out on whatever it was doing. Thomas's > > > patch got to > > > v5 (based on Neal's review comments) with memory leaks still in it, > > > my review > > > got rid of some of them, and we think Neal's last review of v6 of > > > the patch > > > got rid of the last of them. > > > > Another lurker's 2 cents: > > > > My experience with compilers in particular is that an arena is the > > way to go for memory management. I haven't looked at the AST code, > > but this can take a variety of forms: anything from linked lists of > > pointers to free from something which allocates memory in large > > blocks and parcels them out. The goal is just to be able to free the > > memory en-masse whatever happens and not have to track individual > > pointers. > > Thanks for the message. I was going to suggest the same thing. I > think it's primarily a question of how to add an arena layer. The AST > phase has a mixture of malloc/free and Python object allocation. It > should be straightforward to change the malloc/free code to use an > arena API. We'd probably need a separate mechanism to associate a set > of PyObject* with the arena and have those DECREFed. >
Might just need two lists; malloc'ed pointers and PyObject pointers. Could redefine Py_INCREF and Py_DECREF locally for ast.c and compile.c to use the arena API and thus hide the detail. Otherwise just a big, flashing "USE THIS API" sign will be needed. I have gone ahead and added this as a possible topic to sprint on at PyCon. -Brett > Jeremy > > > > > Generally, compilers have memory allocations which operate in phases > > and so are very amenable to arenas. You might have one memory pool > > for long lived representation, one that is freed and recreated > > between passes, etc. > > > > If you need to keep the AST around long term, then a mark-sweep > > garbage collector combined with a linked list might even be a good idea. > > > > Obviously, the whole thing is a tradeoff of peak memory size (which > > goes up) against correctness (which is basically ensured, and at > > least easily auditable). > > > > > > Niko > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev@python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com