[Python-Dev] Memory use of PEP 563 vs PEP 649

2021-04-17 Thread Mark Shannon
Hi, There has been some discussion on this mailing list about the memory use of PEP 563 vs PEP 649. It doesn't matter. The memory use of either is small, and any measurements are merely measuring artifacts of the current implementations, and the current on-disk representation of code object

[Python-Dev] Memory address vs serial number in reprs

2020-07-19 Thread Serhiy Storchaka
I have problem with the location of hexadecimal memory address in custom reprs. vs The long hexadecimal number makes the repr longer and distracts attention from other useful information. We could get rid of it, but it is useful if we want to distinguish objects of the same type.

Re: [Python-Dev] Memory bitmaps for the Python cyclic garbage collector

2017-09-08 Thread Tim Peters
[Tim] >> In that case, it's because Python >> _does_ mutate the objects' refcount members under the covers, and so >> the OS ends up making fresh copies of the memory anyway. [Greg Ewing ] > Has anyone ever considered addressing that by moving the > refcounts out of the objects and keeping them so

Re: [Python-Dev] Memory bitmaps for the Python cyclic garbage collector

2017-09-08 Thread Greg Ewing
Tim Peters wrote: In that case, it's because Python _does_ mutate the objects' refcount members under the covers, and so the OS ends up making fresh copies of the memory anyway. Has anyone ever considered addressing that by moving the refcounts out of the objects and keeping them somewhere else

Re: [Python-Dev] Memory bitmaps for the Python cyclic garbage collector

2017-09-08 Thread Tim Peters
[Neil Schemenauer ] > Python objects that participate in cyclic GC (things like lists, dicts, > sets but not strings, ints and floats) have extra memory overhead. I > think it is possible to mostly eliminate this overhead. Also, while > the GC is running, this GC state is mutated, which destroys

Re: [Python-Dev] Memory bitmaps for the Python cyclic garbage collector

2017-09-07 Thread INADA Naoki
Big +1. I love the idea. str (especially, docstring), dict, and tuples are major memory eater in Python. This may reduce tuple memory usage massively. INADA Naoki On Fri, Sep 8, 2017 at 2:30 AM, Neil Schemenauer wrote: > Python objects that participate in cyclic GC (things like lists, dicts

Re: [Python-Dev] Memory bitmaps for the Python cyclic garbage collector

2017-09-07 Thread Antoine Pitrou
On Thu, 7 Sep 2017 11:30:12 -0600 Neil Schemenauer wrote: > > * The GC process would work nearly the same as it does now. Rather than > only traversing the linked list, we would also have to crawl over the > GC object arenas, check blocks of memory that have the tracked bit > set. Small n

[Python-Dev] Memory bitmaps for the Python cyclic garbage collector

2017-09-07 Thread Neil Schemenauer
Python objects that participate in cyclic GC (things like lists, dicts, sets but not strings, ints and floats) have extra memory overhead. I think it is possible to mostly eliminate this overhead. Also, while the GC is running, this GC state is mutated, which destroys copy-on-write optimizations.

Re: [Python-Dev] Memory

2016-05-01 Thread Burkhard Meier
Sorry. Somebody take the keyboard away from me after midnight. I think Google Chrome does have that feature. Definitely Not related to Python dev... Google translate doesn't correctly understand everything. My apologies (again ) for this unrelated post. And, yes, I am the author of the "Pytho

Re: [Python-Dev] Memory

2016-05-01 Thread Guido van Rossum
That would certainly be the most favorable interpretation. On Sun, May 1, 2016 at 10:41 AM, Carl Kleffner wrote: > AFAIK Burkhard A. Meier is the author of the Python GUI Programming > Cookbook (Packtpub). I wonder if someone has hacked is gmail account? > > 2016-05-01 18:51 GMT+02:00 Steven D'A

Re: [Python-Dev] Memory

2016-05-01 Thread Carl Kleffner
AFAIK Burkhard A. Meier is the author of the Python GUI Programming Cookbook (Packtpub). I wonder if someone has hacked is gmail account? 2016-05-01 18:51 GMT+02:00 Steven D'Aprano : > On Sun, May 01, 2016 at 08:08:22AM -0500, Ryan Gonzalez wrote: > > Well, I put this in Google Translate...and go

Re: [Python-Dev] Memory

2016-05-01 Thread Steven D'Aprano
On Sun, May 01, 2016 at 08:08:22AM -0500, Ryan Gonzalez wrote: > Well, I put this in Google Translate...and got this: > > The disk clatters > the Spontie giggles > ~ > hopefully > alliance insures ... > > Not sure if this a useless post or Translate just being weird. Leaning > towards the latter.

Re: [Python-Dev] Memory

2016-05-01 Thread Ryan Gonzalez
Well, I put this in Google Translate...and got this: The disk clatters the Spontie giggles ~ hopefully alliance insures ... Not sure if this a useless post or Translate just being weird. Leaning towards the latter... -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your pr

[Python-Dev] Memory

2016-05-01 Thread Burkhard Meier
~ Die Scheibe klirrt der Spontie kichert ~ hoffentlich Alliiance versichert... ~ https://youtu.be/m3SjCzA71eM ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailm

Re: [Python-Dev] Memory BIO for _ssl

2014-07-12 Thread Geert Jansen
On Mon, Jul 7, 2014 at 1:49 AM, Antoine Pitrou wrote: > Le 05/07/2014 14:04, Geert Jansen a écrit : > >> Since I need this for my Gruvi async framework, I want to volunteer to >> write a patch. It should be useful as well to Py3K's asyncio and other >> async frameworks. It would be good to get so

Re: [Python-Dev] Memory BIO for _ssl

2014-07-06 Thread Antoine Pitrou
Hi, Le 05/07/2014 14:04, Geert Jansen a écrit : Since I need this for my Gruvi async framework, I want to volunteer to write a patch. It should be useful as well to Py3K's asyncio and other async frameworks. It would be good to get some feedback before I start on this. Thanks for volunteerin

[Python-Dev] Memory BIO for _ssl

2014-07-05 Thread Geert Jansen
Hi, the topic of a memory BIO for the _ssl module in the stdlib was discussed before here: http://mail.python.org/pipermail/python-ideas/2012-November/017686.html Since I need this for my Gruvi async framework, I want to volunteer to write a patch. It should be useful as well to Py3K's asyncio a

Re: [Python-Dev] Memory tests in Unicode

2008-08-16 Thread Terry Reedy
Antoine Pitrou wrote: Facundo Batista gmail.com> writes: The test_raiseMemError() in test_unicode.py is complicating me the regression tests: tries to use all the memory in my system, which starts to swap as crazy, and almost freezes everything. When the test finishes (always pass ok), I have

Re: [Python-Dev] Memory tests in Unicode

2008-08-16 Thread Facundo Batista
2008/8/16 "Martin v. Löwis" <[EMAIL PROTECTED]>: > See bug http://bugs.python.org/issue3556 > > If no solution is forthcoming quickly, I recommend to remove/disable the > test. The Antoine patch works ok for me, and solves the whole issue. I'm on linux, if somebody can give it a try in another p

Re: [Python-Dev] Memory tests in Unicode

2008-08-16 Thread Martin v. Löwis
> Are you having also this issue? Do you think that it should run not *always*? See bug http://bugs.python.org/issue3556 If no solution is forthcoming quickly, I recommend to remove/disable the test. Regards, Martin ___ Python-Dev mailing list Python-

Re: [Python-Dev] Memory tests in Unicode

2008-08-16 Thread Antoine Pitrou
Facundo Batista gmail.com> writes: > > I do *not* want to remove the test. You misunderstood my remark. If the test takes a lot of time and eats a lot of memory, it's precisely because the MemoryError isn't raised; and the test needs MemoryError to be raised in order to be meaningful. I was pro

Re: [Python-Dev] Memory tests in Unicode

2008-08-16 Thread Jean-Paul Calderone
On Sat, 16 Aug 2008 13:01:33 -0300, Facundo Batista <[EMAIL PROTECTED]> wrote: 2008/8/16 Antoine Pitrou <[EMAIL PROTECTED]>: If the test does allocate the very large string, it means MemoryError isn't raised, which defeats the purpose of the test. I do *not* want to remove the test. Antoine

Re: [Python-Dev] Memory tests in Unicode

2008-08-16 Thread Facundo Batista
2008/8/16 Antoine Pitrou <[EMAIL PROTECTED]>: > If the test does allocate the very large string, it means MemoryError isn't > raised, which defeats the purpose of the test. I do *not* want to remove the test. I just want to execute it *only* when I run "-u all" or "-u memory", not everytime I ru

Re: [Python-Dev] Memory tests in Unicode

2008-08-16 Thread Antoine Pitrou
Facundo Batista gmail.com> writes: > > The test_raiseMemError() in test_unicode.py is complicating me the > regression tests: tries to use all the memory in my system, which > starts to swap as crazy, and almost freezes everything. When the test > finishes (always pass ok), I have all the memory

Re: [Python-Dev] Memory tests in Unicode

2008-08-16 Thread Victor Stinner
Hi, Le Saturday 16 August 2008 15:43:28 Facundo Batista, vous avez écrit : > The test_raiseMemError() in test_unicode.py is complicating me the > regression tests: tries to use all the memory in my system, which > starts to swap as crazy, and almost freezes everything. On UNIX, it's possible to l

Re: [Python-Dev] Memory tests in Unicode

2008-08-16 Thread Nick Coghlan
Facundo Batista wrote: > The test_raiseMemError() in test_unicode.py is complicating me the > regression tests: tries to use all the memory in my system, which > starts to swap as crazy, and almost freezes everything. When the test > finishes (always pass ok), I have all the memory flushed so it ta

[Python-Dev] Memory tests in Unicode

2008-08-16 Thread Facundo Batista
The test_raiseMemError() in test_unicode.py is complicating me the regression tests: tries to use all the memory in my system, which starts to swap as crazy, and almost freezes everything. When the test finishes (always pass ok), I have all the memory flushed so it take a few seconds to go back to

Re: [Python-Dev] Memory Error while reading large file

2008-07-31 Thread Greg Ewing
Martin v. Löwis wrote: (maybe the use of the question mark is more typical in German than in English; my stomach turns around when I read a question that ends with a full stop) No, it's required in English, too. -- Greg ___ Python-Dev mailing lis

Re: [Python-Dev] Memory Error while reading large file

2008-07-31 Thread Guido van Rossum
On Thu, Jul 31, 2008 at 2:38 PM, Scott Dial <[EMAIL PROTECTED]> wrote: > Martin v. Löwis wrote: >> >> (maybe the use of the question mark is more typical in German >> than in English; my stomach turns around when I read a question >> that ends with a full stop) > > There is no loss in translation h

Re: [Python-Dev] Memory Error while reading large file

2008-07-31 Thread Scott Dial
Martin v. Löwis wrote: (maybe the use of the question mark is more typical in German than in English; my stomach turns around when I read a question that ends with a full stop) There is no loss in translation here. Proper English requires the use of a question mark just the same as German,

Re: [Python-Dev] Memory Error while reading large file

2008-07-31 Thread Martin v. Löwis
> If you have an actual question I'd like to stress this point as well. Any good posting one wants an answer to must include a question, and that question must be explicitly phrased, and terminated with a question mark. (maybe the use of the question mark is more typical in German than in Engl

Re: [Python-Dev] Memory Error while reading large file

2008-07-31 Thread Steven D'Aprano
On Thu, 31 Jul 2008 03:01:42 pm Sumant Gupta wrote: > Hi > > I have a problem reading very large text file. > When I call the len function to get the total lines in python file.i > get memory error . I am reading the list of files in a loop ,2 files > are read properly but when the third file is re

[Python-Dev] Memory Error while reading large file

2008-07-30 Thread Sumant Gupta
Hi I have a problem reading very large text file. When I call the len function to get the total lines in python file.i get memory error . I am reading the list of files in a loop ,2 files are read properly but when the third file is read , It gives an memory error . Sumant Gupta Software Engine

Re: [Python-Dev] Memory benchmarking?

2007-11-29 Thread M.-A. Lemburg
On 2007-11-29 11:52, Titus Brown wrote: > Hi all, > > is there a good, or standard memory benchmarking system for Python? > pybench doesn't return significantly different results when Python 2.6 > is compiled with pymalloc and without pymalloc. Thinking on it, I'm not > too surprised -- pybench p

[Python-Dev] Memory benchmarking?

2007-11-29 Thread Titus Brown
Hi all, is there a good, or standard memory benchmarking system for Python? pybench doesn't return significantly different results when Python 2.6 is compiled with pymalloc and without pymalloc. Thinking on it, I'm not too surprised -- pybench probably benchmarks a lot of stuff -- but some guidan

Re: [Python-Dev] Memory Error the right error for coding cookie promise violation?

2006-02-21 Thread Neal Norwitz
On 2/20/06, Bengt Richter <[EMAIL PROTECTED]> wrote: > Perhaps a more informative message would be nice. > Here's an easy way to trigger it: > > >>> compile("#-*- coding: ascii -*-\nprint 'ab%c'\n"%0x80, '','exec') > Traceback (most recent call last): >File "", line 1, in ? > MemoryError Th

Re: [Python-Dev] Memory Error the right error for coding cookie promise violation?

2006-02-20 Thread Alex Martelli
On Feb 21, 2006, at 6:53 AM, Bengt Richter wrote: > Perhaps a more informative message would be nice. > Here's an easy way to trigger it: > compile("#-*- coding: ascii -*-\nprint 'ab%c'\n"%0x80, '','exec') > Traceback (most recent call last): >File "", line 1, in ? > MemoryError Defin

[Python-Dev] Memory Error the right error for coding cookie promise violation?

2006-02-20 Thread Bengt Richter
Perhaps a more informative message would be nice. Here's an easy way to trigger it: >>> compile("#-*- coding: ascii -*-\nprint 'ab%c'\n"%0x80, '','exec') Traceback (most recent call last): File "", line 1, in ? MemoryError Regards, Bengt Richter

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-12-01 Thread Nick Coghlan
Martin v. Löwis wrote: > Nick Coghlan wrote: > > The ast C structs are already auto-generated by a Python script > (asdl_c.py, to > > be precise). The trick is to make that script generate full PyObjects > rather > > than the simple C structures that it generates now. > > See the ast-object b

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-30 Thread Jeremy Hylton
Sure. If they're immutable sharing is fine, but you end up making a copy anyway if you want to make changes, right? Jeremy On 11/30/05, Greg Ewing <[EMAIL PROTECTED]> wrote: > Jeremy Hylton wrote: > > > I still think passing copies is better than sharing live > > objects between Python and C, >

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-30 Thread Greg Ewing
Jeremy Hylton wrote: > I still think passing copies is better than sharing live > objects between Python and C, Even if the objects are immutable? -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | A citizen of NewZealandCorp, a

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-30 Thread Jeremy Hylton
On 11/30/05, Neal Norwitz <[EMAIL PROTECTED]> wrote: > On 11/30/05, Thomas Lee <[EMAIL PROTECTED]> wrote: > > > > Quick semi-related question: where are the marshal_* functions called? > > They're all static in Python-ast.c and don't seem to be actually called > > anywhere. Can we ditch them? > > I

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-30 Thread Martin v. Löwis
Nick Coghlan wrote: > The ast C structs are already auto-generated by a Python script (asdl_c.py, to > be precise). The trick is to make that script generate full PyObjects rather > than the simple C structures that it generates now. See the ast-object branch. > The second step is to then m

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-30 Thread Neil Schemenauer
Neal Norwitz <[EMAIL PROTECTED]> wrote: > If everything is a PyObject, wouldn't [the marshal functions] be > redundant? You could be right. Spending time to kept them working is probably wasted effort. Neil ___ Python-Dev mailing list Python-Dev@pyt

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-30 Thread Neal Norwitz
On 11/30/05, Neil Schemenauer <[EMAIL PROTECTED]> wrote: > Thomas Lee <[EMAIL PROTECTED]> wrote: > > Quick semi-related question: where are the marshal_* functions called? > > They're all static in Python-ast.c and don't seem to be actually called > > anywhere. Can we ditch them? > > They are inten

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-30 Thread Neil Schemenauer
Thomas Lee <[EMAIL PROTECTED]> wrote: > Quick semi-related question: where are the marshal_* functions called? > They're all static in Python-ast.c and don't seem to be actually called > anywhere. Can we ditch them? They are intended to be used to make the AST available to Python code. It would

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-30 Thread Neal Norwitz
On 11/30/05, Thomas Lee <[EMAIL PROTECTED]> wrote: > > Quick semi-related question: where are the marshal_* functions called? > They're all static in Python-ast.c and don't seem to be actually called > anywhere. Can we ditch them? I *think* they are not necessary. My guess is that they were there

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-30 Thread A.M. Kuchling
On Wed, Nov 30, 2005 at 07:42:20PM +1000, Nick Coghlan wrote: > The second step is to then modify ast.c to use the new structures. A branch > probably wouldn't help much with initial development (this is a "break the > world, check in when stuff compiles again" kind of change, which is hard to >

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-30 Thread Thomas Lee
Nick Coghlan wrote: >Greg Ewing wrote: > > >>Neal Norwitz wrote: >> >> >> >>>I'm mostly convinced that using PyObjects would be a good thing. >>>However, making the change isn't free as all the types need to be >>>created and this is likely quite a bit of code. >>> >>> >>Since they're

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-30 Thread Michael Hudson
Greg Ewing <[EMAIL PROTECTED]> writes: > Guido van Rossum wrote: > >>>To me, that's an argument in favour of always generating >>>a .pyc, even for scripts. >> >> I'm not sure I follow the connection. > > You were saying that if the parser and compiler were > slow, it would slow down single-file s

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-30 Thread Nick Coghlan
Greg Ewing wrote: > Neal Norwitz wrote: > >> I'm mostly convinced that using PyObjects would be a good thing. >> However, making the change isn't free as all the types need to be >> created and this is likely quite a bit of code. > > Since they're all so similar, perhaps they could be > auto-gen

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-29 Thread Greg Ewing
Neal Norwitz wrote: > I'm mostly convinced that using PyObjects would be a good thing. > However, making the change isn't free as all the types need to be > created and this is likely quite a bit of code. Since they're all so similar, perhaps they could be auto-generated by a fairly simple scrip

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-29 Thread Greg Ewing
Neal Norwitz wrote: > On 11/28/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > > he assumed that FunctionDef would *consume* the references > > being passed (whether it is successful or not). > > I keep resisting this solution, though I'm not sure why. One reason for not liking it is that i

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-29 Thread Greg Ewing
Guido van Rossum wrote: >>To me, that's an argument in favour of always generating >>a .pyc, even for scripts. > > I'm not sure I follow the connection. You were saying that if the parser and compiler were slow, it would slow down single-file scripts that didn't have a .pyc (or at least that's w

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-29 Thread Greg Ewing
Nick Coghlan wrote: > Declaring that none of the AST node creation methods steal references would > be > consistent with most of the existing C API (e.g. PySequence_SetItem, > PySequence_Tuple, PySequence_List), Agreed, although the rest of your proposal (while admirably cunning) requires that

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-29 Thread Greg Ewing
Neal Norwitz wrote: > For those watching, Greg's and Martin's version were almost the same. > However, Greg's version left in the memory leak, while Martin fixed it > by letting the result fall through. I addressed the memory leak by stipulating that FunctionDef should steal references to its ar

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-29 Thread Brett Cannon
On 11/29/05, Neal Norwitz <[EMAIL PROTECTED]> wrote: > On 11/29/05, Nick Coghlan <[EMAIL PROTECTED]> wrote: > > > > When working on the CST->AST parser, there were only a few things I found to > > be seriously painful about the memory management: > > > >1. Remembering which free_* variant to ca

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-29 Thread Brett Cannon
On 11/29/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > On 11/28/05, Greg Ewing <[EMAIL PROTECTED]> wrote: > > [...] My mental model > > of parsing & compiling in the presence of a parse tree > > is like this: > > > >[source] -> scanner -> [tokens] > > -> parser -> [AST] -> code_generat

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-29 Thread Neal Norwitz
On 11/28/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > Neal Norwitz wrote: > > For those watching, Greg's and Martin's version were almost the same. > > However, Greg's version left in the memory leak, while Martin fixed it > > by letting the result fall through. > > Actually, Greg said (corre

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-29 Thread Neal Norwitz
On 11/29/05, Nick Coghlan <[EMAIL PROTECTED]> wrote: > > When working on the CST->AST parser, there were only a few things I found to > be seriously painful about the memory management: > >1. Remembering which free_* variant to call for AST nodes >2. Remembering which asdl_seq_*_free varian

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-29 Thread Guido van Rossum
On 11/28/05, Greg Ewing <[EMAIL PROTECTED]> wrote: > [...] My mental model > of parsing & compiling in the presence of a parse tree > is like this: > >[source] -> scanner -> [tokens] > -> parser -> [AST] -> code_generator -> [code] > > The fact that there still seems to be another kind of

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-29 Thread Nick Coghlan
Neal Norwitz wrote: > On 11/28/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: >> Neal Norwitz wrote: >>> Hope this helps explain a bit. Please speak up with how this can be >>> improved. Gotta run. >> I would rewrite it as > > [code snipped] > > For those watching, Greg's and Martin's version

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-29 Thread Fredrik Lundh
Jeremy Hylton wrote: > > Me neither. Adding yet another memory allocation scheme to Python's > > already staggering number of memory allocation strategies sounds like > > a bad idea. > > The reason this thread started was the complaint that reference > counting in the compiler is really difficult.

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Martin v. Löwis
Neal Norwitz wrote: > For those watching, Greg's and Martin's version were almost the same. > However, Greg's version left in the memory leak, while Martin fixed it > by letting the result fall through. Actually, Greg said (correctly) that his version also fixes the leak: he assumed that Function

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Neal Norwitz
On 11/28/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > Neal Norwitz wrote: > > Hope this helps explain a bit. Please speak up with how this can be > > improved. Gotta run. > > I would rewrite it as [code snipped] For those watching, Greg's and Martin's version were almost the same. Howeve

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Greg Ewing
Brett Cannon wrote: > Is there a specific reason you are leaving out the AST, Greg, or do > you count that as part of the bytecode compiler No, I consider it part of the parser. My mental model of parsing & compiling in the presence of a parse tree is like this: [source] -> scanner -> [tokens

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Neal Norwitz
On 11/28/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > I guess I don't understand the AST compiler code enough to participate > in this discussion. I hope everyone while chime in here. This is important to improve and learn from others. Let me try to describe the current situation with a s

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Brett Cannon
On 11/28/05, Greg Ewing <[EMAIL PROTECTED]> wrote: > Here's a somewhat radical idea: > > Why not write the parser and bytecode compiler in Python? > > A .pyc could be bootstrapped from it and frozen into > the executable. > Is there a specific reason you are leaving out the AST, Greg, or do you co

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Martin v. Löwis
Neal Norwitz wrote: > Hope this helps explain a bit. Please speak up with how this can be > improved. Gotta run. I would rewrite it as static PyObject* ast_for_funcdef(struct compiling *c, const node *n) { /* funcdef: [decorators] 'def' NAME parameters ':' suite */ PyObject *name = NU

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Greg Ewing
Here's a somewhat radical idea: Why not write the parser and bytecode compiler in Python? A .pyc could be bootstrapped from it and frozen into the executable. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | A citizen of NewZeal

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Greg Ewing
Neal Norwitz wrote: > This is an entire function from Python/ast.c. > Sequences do not know what type they hold, so there needs to be > different dealloc functions to free them properly (asdl_*_seq_free()). Well, that's one complication that would go away if the nodes were PyObjects. > The memo

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Greg Ewing
Jeremy Hylton wrote: > Almost every line of > code can lead to an error exit. The code becomes quite cluttered when > it uses reference counting. I don't see why very many more error exits should become possible just by introducing refcounting. Errors are possible whenever you allocate something

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Brett Cannon
On 11/28/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > [Guido] > > > Then I don't understand why there was discussion of alloca() earlier > > > on -- surely the lifetime of a node should not be limited by the stack > > > frame that allocated it? > > [Jeremy] > > Actually this is a pretty good l

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Guido van Rossum
[Guido] > > Then I don't understand why there was discussion of alloca() earlier > > on -- surely the lifetime of a node should not be limited by the stack > > frame that allocated it? [Jeremy] > Actually this is a pretty good limit, because all these data > structures are temporaries used by the

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Neil Schemenauer
On Mon, Nov 28, 2005 at 03:47:07PM -0500, Jeremy Hylton wrote: > The reason this thread started was the complaint that reference > counting in the compiler is really difficult. I don't think that's exactly right. The problem is that the AST compiler mixes its own memory management strategy with r

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Martin v. Löwis
Jeremy Hylton wrote: > The reason this thread started was the complaint that reference > counting in the compiler is really difficult. Almost every line of > code can lead to an error exit. The code becomes quite cluttered when > it uses reference counting. Right now, the AST is created with

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Jeremy Hylton
On 11/28/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > The code becomes quite cluttered when > > it uses reference counting. Right now, the AST is created with > > malloc/free, but that makes it hard to free the ast at the right time. > > Would fixing the code to add free() calls in all the

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Guido van Rossum
On 11/28/05, Jeremy Hylton <[EMAIL PROTECTED]> wrote: > On 11/28/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > > On 11/18/05, Neil Schemenauer <[EMAIL PROTECTED]> wrote: > > > Perhaps we should use the memory management technique that the rest > > > of Python uses: reference counting. I don't

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Jeremy Hylton
On 11/28/05, Guido van Rossum <[EMAIL PROTECTED]> wrote: > On 11/18/05, Neil Schemenauer <[EMAIL PROTECTED]> wrote: > > Perhaps we should use the memory management technique that the rest > > of Python uses: reference counting. I don't see why the AST > > structures couldn't be PyObjects. > > Me n

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-28 Thread Guido van Rossum
On 11/18/05, Neil Schemenauer <[EMAIL PROTECTED]> wrote: > Perhaps we should use the memory management technique that the rest > of Python uses: reference counting. I don't see why the AST > structures couldn't be PyObjects. Me neither. Adding yet another memory allocation scheme to Python's alre

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-23 Thread Thomas Lee
Neil Schemenauer wrote: >Fredrik Lundh <[EMAIL PROTECTED]> wrote: > > >>Thomas Lee wrote: >> >> >> >>>Even if it meant we had just one function call - one, safe function call >>>that deallocated all the memory allocated within a function - that we >>>had to put before each and every return,

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-20 Thread Neil Schemenauer
Fredrik Lundh <[EMAIL PROTECTED]> wrote: > Thomas Lee wrote: > >> Even if it meant we had just one function call - one, safe function call >> that deallocated all the memory allocated within a function - that we >> had to put before each and every return, that's better than what we >> have. > > all

[Python-Dev] Memory management in the AST parser & compiler

2005-11-18 Thread Jim Jewett
There is a public-domain implementation of alloca at http://www.cs.purdue.edu/homes/apm/courses/BITSC461-fall03/listen-code/listen-1.0-dave/lsl_cpp/alloca.c It would still fail on architectures that don't use a stack frame; other than that, it seems like a reasonable fallback, if alloca is otherw

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-18 Thread Michael Walter
The behavior of libiberty's alloca() replacement might be interesting as well: http://gcc.gnu.org/onlinedocs/libiberty/Functions.html#index-alloca-59 Regards, Michael On 11/18/05, Alex Martelli <[EMAIL PROTECTED]> wrote: > > On Nov 17, 2005, at 5:00 PM, Thomas Lee wrote: > > > Portability may al

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-17 Thread Alex Martelli
On Nov 17, 2005, at 5:00 PM, Thomas Lee wrote: > Portability may also be an issue to take into consideration: Of course -- but so is anno domini... the eskimo.com FAQ is (C) 1995, and the neohapsis.com page just points to the eskimo.com one: > http://www.eskimo.com/~scs/C-faq/q7.32.html > htt

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-17 Thread Thomas Lee
Portability may also be an issue to take into consideration: http://www.eskimo.com/~scs/C-faq/q7.32.html http://archives.neohapsis.com/archives/postfix/2001-05/1305.html Cheers, Tom Alex Martelli wrote: >On Nov 17, 2005, at 12:46 PM, Brett Cannon wrote: >... > > >>>alloca? >>> >>>(duck) >

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-17 Thread Alex Martelli
On Nov 17, 2005, at 12:46 PM, Brett Cannon wrote: ... >> alloca? >> >> (duck) >> > > But how widespread is its support (e.g., does Windows have it)? Yep, spelled with a leading underscore: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/ vclib/html/_crt__alloca.asp Alex _

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-17 Thread Brett Cannon
On 11/16/05, Thomas Lee <[EMAIL PROTECTED]> wrote: > Just messing around with some ideas. I was trying to avoid the ugly > macros (note my earlier whinge about a learning curve) but they're the > cleanest way I could think of to get around the problem without > resorting to a mass deallocation righ

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-17 Thread Brett Cannon
On 11/16/05, Fredrik Lundh <[EMAIL PROTECTED]> wrote: > Thomas Lee wrote: > > > Even if it meant we had just one function call - one, safe function call > > that deallocated all the memory allocated within a function - that we > > had to put before each and every return, that's better than what we

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-16 Thread Thomas Lee
Just messing around with some ideas. I was trying to avoid the ugly macros (note my earlier whinge about a learning curve) but they're the cleanest way I could think of to get around the problem without resorting to a mass deallocation right at the end of the AST run. Which may not be all that

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-16 Thread Nick Coghlan
Thomas Lee wrote: > As the writer of the crappy code that sparked this conversation, I feel > I should say something :) Don't feel bad about it. It turned out the 'helpful' review comments from Neal and I didn't originally work out very well either ;) With the AST compiler being so new, this is

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-16 Thread Collin Winter
On 11/16/05, Niko Matsakis <[EMAIL PROTECTED]> wrote: > - Another idea is to have trees of arenas: the idea is that when an > arena is created, it is assigned a parent. When an arena is freed, > an arenas in its subtree are also freed. This way you can have one > master arena for exception handli

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-16 Thread Fredrik Lundh
Thomas Lee wrote: > Even if it meant we had just one function call - one, safe function call > that deallocated all the memory allocated within a function - that we > had to put before each and every return, that's better than what we > have. alloca? (duck)

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-16 Thread Thomas Lee
By the way, I liked the sound of the arena/pool tree - really good idea. Thomas Lee wrote: >Niko Matsakis wrote: > > > >>>Boy am I wanting RAII from C++ for automatic freeing when scope is >>>left. Maybe we need to come up with a similar thing, like all memory >>>that should be freed once a sc

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-16 Thread Thomas Lee
Niko Matsakis wrote: >>Boy am I wanting RAII from C++ for automatic freeing when scope is >>left. Maybe we need to come up with a similar thing, like all memory >>that should be freed once a scope is left must use some special struct >>that stores references to all created memory locally and then

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-16 Thread Niko Matsakis
> Boy am I wanting RAII from C++ for automatic freeing when scope is > left. Maybe we need to come up with a similar thing, like all memory > that should be freed once a scope is left must use some special struct > that stores references to all created memory locally and then a free > call must be

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-16 Thread Nick Coghlan
Nick Coghlan wrote: > Marek Baczek Baczyński wrote: >> 2005/11/15, Nick Coghlan <[EMAIL PROTECTED]>: >>> It avoids the potential for labelling problems that arises when goto's are >>> used for resource cleanup. It's a far cry from real exception handling, but >>> it's the best solution I've seen wi

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-16 Thread Thomas Lee
As the writer of the crappy code that sparked this conversation, I feel I should say something :) Brett Cannon wrote: >On 11/15/05, Neal Norwitz <[EMAIL PROTECTED]> wrote: > > >>On 11/15/05, Jeremy Hylton <[EMAIL PROTECTED]> wrote: >> >> >>>Thanks for the message. I was going to suggest t

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-16 Thread Brett Cannon
On 11/15/05, Neal Norwitz <[EMAIL PROTECTED]> wrote: > On 11/15/05, Jeremy Hylton <[EMAIL PROTECTED]> wrote: > > > > Thanks for the message. I was going to suggest the same thing. I > > think it's primarily a question of how to add an arena layer. The AST > > phase has a mixture of malloc/free a

Re: [Python-Dev] Memory management in the AST parser & compiler

2005-11-15 Thread Neal Norwitz
On 11/15/05, Jeremy Hylton <[EMAIL PROTECTED]> wrote: > > Thanks for the message. I was going to suggest the same thing. I > think it's primarily a question of how to add an arena layer. The AST > phase has a mixture of malloc/free and Python object allocation. It > should be straightforward to

  1   2   >