As the writer of the crappy code that sparked this conversation, I feel I should say something :)
Brett Cannon wrote: >On 11/15/05, Neal Norwitz <[EMAIL PROTECTED]> wrote: > > >>On 11/15/05, Jeremy Hylton <[EMAIL PROTECTED]> wrote: >> >> >>>Thanks for the message. I was going to suggest the same thing. I >>>think it's primarily a question of how to add an arena layer. The AST >>>phase has a mixture of malloc/free and Python object allocation. It >>>should be straightforward to change the malloc/free code to use an >>>arena API. We'd probably need a separate mechanism to associate a set >>>of PyObject* with the arena and have those DECREFed. >>> >>> >>Well good. It seems we all agree there is a problem and on the >>general solution. I haven't thought about Brett's idea to see if it >>could work or not. It would be great if we had someone start working >>to improve the situation. It could well be that we live with the >>current code for 2.5, but it would be great to use arenas for 2.6 at >>least. >> >> >> > > I have been thinking about this some more to put off doing homework >and I have some random ideas I just wanted to toss out there to make >sure I am not thinking about arena memory management incorrectly >(never actually encountered it directly before). > >I think an arena API is going to be the best solution. Pulling >trickery with redefining Py_INCREF and such like I suggested seems >like a pain and possibly error-prone. With the compiler being a >specific corner of the core having a special API for handling the >memory for PyObject* stuff seems reasonable. > > > I agree. And it raises the learning curve for poor saps like myself. :) >We might need PyArena_Malloc() and PyArena_New() to handle malloc() >and PyObject* creation. We could then have a struct that just stored >pointers to the allocated memory (linked list for each pointer which >gives high memory overhead or linked list of arrays that should lower >memory but make having possible holes in the array for stuff already >freed a pain to handle). We would then have PyArena_FreeAll() that >would be strategically placed in the code for when bad things happen >that would just traverse the lists and free everything. I assume >having a way to free individual items might be useful. Could have the >PyArena_New() and _Malloc() return structs with the needed info for a >PyArena_Free(location_struct) to be able to fee the specific item >without triggering a complete freeing of all memory. But this usage >should be discouraged and only used when proper memory management is >guaranteed. > > > An arena/pool (as I understood it from my quick skim) for the AST would probably best be implemented (IMHO) as an ADT based on a linked-list: typedef struct _ast_pool_node { struct _ast_pool_node *next; PyObject *object; /* == NULL when data != NULL */ void *data; /* == NULL when object != NULL */ }ast_pool_node; deallocating a node could then be as simple as: /* ast_pool_node *n */ PyObject_Free(n->object); if (n->data != NULL) free(n->data); /* save n->next */ free(n); /* then go on to free n->next */ I haven't really thought all that deeply about this, so somebody shoot me down if I'm completely off-base (Neal? :D). Every allocation of a seq/stmt within ast.c would have its memory saved to the pool within the function it's allocated in. Then before we return, we can just deallocate the pool/arena/whatever you want to call it. The problem with this is that should we get to the end of the function and everything actually went okay (i.e. we return non-NULL), we then have to run through and deallocate all the nodes anyway (without deallocating n->object or n->data). Bah. Maybe we *would* be better off with a monolithic cleanup. I don't know. >Boy am I wanting RAII from C++ for automatic freeing when scope is >left. Maybe we need to come up with a similar thing, like all memory >that should be freed once a scope is left must use some special struct >that stores references to all created memory locally and then a free >call must be made at all exit points in the function using the special >struct. Otherwise the pointer is stored in the arena and handled >en-mass later. > > > Which is basically what I just rambled on about up above, I think :) >Hopefully this is all made some sense. =) Is this the basic strategy >that an arena setup would need? if not can someone enlighten me? > > >-Brett >_______________________________________________ >Python-Dev mailing list >Python-Dev@python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >http://mail.python.org/mailman/options/python-dev/krumms%40gmail.com > > > _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com