[issue42763] Exposing a race in the "_warnings" resulting Python parser crash

Change by Irit Katriel : -- versions: +Python 3.11 -Python 3.7, Python 3.8 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue42889] Incorrect behavior of Python parser after ast node of test program being modified

Guido van Rossum added the comment: Yeah, there's supposed to be a checker. -- nosy: +lys.nikolaou, serhiy.storchaka ___ Python tracker ___

[issue42889] Incorrect behavior of Python parser after ast node of test program being modified

Xinmeng Xia added the comment: Nice suggestion! I change the argument and I can' find segfault program in transforming ast.Name. But I do find a segfault program in transforming ast.BinOp! Seeing the following example, this program will cause a segmentation fault on Python 3.10. No error

[issue42889] Incorrect behavior of Python parser after ast node of test program being modified

Guido van Rossum added the comment: Hm, if your argument is just that you can make invalid identifiers this way, I still don't see why that's a big deal. You can do `x.__dict__["1"] = 1` and now you've given x an attribute that's not a valid identifier. You could also call the code object

[issue42889] Incorrect behavior of Python parser after ast node of test program being modified

Xinmeng Xia added the comment: Sorry, my description is a little confusing. My points lie on function 'compile' and 'exec'. Yes, I agree. AST can be modified and don't correspond to valid programs. But I don't think this invaild program can be compiled and exec without any check. It's

[issue42889] Incorrect behavior of Python parser after ast node of test program being modified

Guido van Rossum added the comment: I don't think this is a bug, unless you can show an example where the bytecode compiler or the interpreter actually crashes. Basically you can change AST nodes in lots of ways that don't correspond to valid programs, and as long as you can't make CPython

[issue42889] Incorrect behavior of Python parser after ast node of test program being modified

New submission from Xinmeng Xia : The following program will lead to a incorrect behavior of Python parser. We change all variable to integer( forcely transformed to string) via ast.NodeTransformer. Then we compile the new code and execute it. It's surprising that code like "1=2; pr

[issue42763] Exposing a race in the "_warnings" resulting Python parser crash

Xinmeng Xia added the comment: Now I see. By the way, I think this case should be moved back to "cpython/Lib/test/crashers/" since the bug still exists. It is not fixed completely, the old case is outdated. I suggest we can put the new case into directory "crashers". --

[issue42763] Exposing a race in the "_warnings" resulting Python parser crash

Steve Stagg added the comment: If we take your minimal example (that /sometimes/ crashes), and look at what python is doing: import threading t = threading.Thread(target=1, daemon=True).start() --- The main thread does the following: M1. Startup interpreter, parse code M2. Import &

[issue42763] Exposing a race in the "_warnings" resulting Python parser crash

Xinmeng Xia added the comment: Thank you, but I don't think this is a duplicate of issue42717. This crash is probably caused by the parameter "target" of Thread. "Target" accept the "callable object". Defaults to None. In this program, it's assigned to a "Int object". I think a pre-checking

[issue42763] Exposing a race in the "_warnings" resulting Python parser crash

Steve Stagg added the comment: Looks like a duplicate of issue42717. Causing a daemonic thread to terminate while doing IO will trigger the above abort -- nosy: +stestagg ___ Python tracker

[issue42763] Exposing a race in the "_warnings" resulting Python parser crash

threads Python runtime state: finalizing (tstate=0x2679180) Current thread 0x7f3481d3a700 (most recent call first): Aborted (core dumped) -- components: Interpreter Core messages: 383883 nosy: xxm priority: normal severity: normal status: open title: Exposing a race in the "_warnings" r

[issue42284] The grammar specification is inconsistent with the implementation of Python parser.

Raymond Hettinger added the comment: I concur with Georg. In writing compilers, it is often convenient to allow the grammar to be permissive and save the enforcement of additional syntax restrictions for the downstream semantic analysis phase. For example, the C language grammar

[issue42284] The grammar specification is inconsistent with the implementation of Python parser.

Georg Brandl added the comment: This grammar specification doesn't contain a full specification of code that won't raise SyntaxError. There are several conditions that aren't checked by the generated parser, but at a later stage in the compilation process. While probably possible to express

[issue42284] The grammar specification is inconsistent with the implementation of Python parser.

' %%% However, the implementation of the Python parser requires the 'break' can only be embedded into a loop statement. See the following example: Example A(without loop): >>> compile("if True:break",'','exec') Traceback (most recent call l

[issue26415] Excessive peak memory consumption by the Python parser

Pablo Galindo Salgado added the comment: Closing as we have a new parser in 3.9 and the current one will be removed in Python 3.10. -- resolution: -> rejected stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue26415] Excessive peak memory consumption by the Python parser

Change by Inada Naoki : -- nosy: +inada.naoki ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue26415] Excessive peak memory consumption by the Python parser

Change by Pablo Galindo Salgado : -- nosy: +pablogsal ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue26415] Excessive peak memory consumption by the Python parser

Change by Serhiy Storchaka : -- assignee: -> serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue26415] Excessive peak memory consumption by the Python parser

A. Skrobov added the comment: I've run pyperformance (0.7.0) with my updated patch, and posted results at the PR page. They look encouraging enough. -- ___ Python tracker

[issue26415] Excessive peak memory consumption by the Python parser

A. Skrobov added the comment: @Serhiy: incredibly, this patch from 2.5 years ago required very minor changes to apply to the latest master. Shows how ossified the parser is :-) Now posted as https://github.com/python/cpython/pull/10995 -- ___

[issue26415] Excessive peak memory consumption by the Python parser

Change by Roundup Robot : -- keywords: +patch pull_requests: +10236 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue26415] Excessive peak memory consumption by the Python parser

Change by Brett Cannon : -- nosy: -brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue26415] Excessive peak memory consumption by the Python parser

Change by Giampaolo Rodola' : -- nosy: -giampaolo.rodola ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue26415] Excessive peak memory consumption by the Python parser

Serhiy Storchaka added the comment: Do you mind to create a pull request for reviewing? I can not promise that it will be merged, but it will have a chance. For reference, the article about this patch (on Russian): https://habr.com/post/314062/ . -- versions: +Python 3.8 -Python

[issue26415] Excessive peak memory consumption by the Python parser

A. Skrobov added the comment: To bump this year-old issue, I have delivered a talk at EuroPython 2017, explaining what my patch does, how it does what it does, and why it's a good thing to do. You can watch my talk at https://www.youtube.com/watch?v=dyRDmcsTwhE=1h52m38s --

[issue26415] Excessive peak memory consumption by the Python parser

Changes by Charalampos Stratakis : -- nosy: +cstratak ___ Python tracker ___ ___

[issue26415] Excessive peak memory consumption by the Python parser

Changes by Petr Viktorin : -- nosy: +encukou ___ Python tracker ___ ___ Python-bugs-list

[issue26415] Excessive peak memory consumption by the Python parser

A. Skrobov added the comment: Updated the comment for Init_ValidationGrammar() -- Added file: http://bugs.python.org/file44370/patch ___ Python tracker

[issue26415] Excessive peak memory consumption by the Python parser

A. Skrobov added the comment: Xavier, the big picture description of my patch is in http://bugs.python.org/file43665/devguide_patch The heap fragmentation was observed by Victor, not by myself. Victor, could you please create a new ticket for your python_memleak.py reproducer? --

[issue26415] Excessive peak memory consumption by the Python parser

Xavier Combelle added the comment: Looks that there is two bug as partial solution of the main bug which is reduce memory consumption of the parser: - The compression thing - the reducing of heap fragmentation Could each sub bug have it's own bug tracker and mark them blocking for the main bug

[issue26415] Excessive peak memory consumption by the Python parser

A. Skrobov added the comment: Fixing whitespace in the patch, and including an update for the docs -- Added file: http://bugs.python.org/file43664/patch ___ Python tracker

[issue26415] Excessive peak memory consumption by the Python parser

Changes by A. Skrobov : Added file: http://bugs.python.org/file43665/devguide_patch ___ Python tracker ___

[issue26415] Excessive peak memory consumption by the Python parser

A. Skrobov added the comment: (Updating the issue title, to avoid confusion between two orthogonal concerns) -- title: Fragmentation of the heap memory in the Python parser -> Excessive peak memory consumption by the Python parser ___ Python trac

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: My current patch avoids the memory peak *and* doesn't add any memory fragmentation on top of whatever is already there. In other words, it makes the parser better in this one aspect, and it doesn't make it worse in any aspect. --

[issue26415] Fragmentation of the heap memory in the Python parser

STINNER Victor added the comment: Benjamin Peterson: "It seems to me a simpler solution would be allocate all nodes for a parse tree in an arena." Exactly, that's the real fix. Just make sure that deallocating this arena does punch holes in the "heap memory". For example, on Linux, it means

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: An arena might help reclaim the memory once the parsing is complete, but it wouldn't reduce the peak memory consumption by the parser, and so it wouldn't prevent a MemoryError when parsing a 35MB source on a PC with 2GB of RAM. --

[issue26415] Fragmentation of the heap memory in the Python parser

Benjamin Peterson added the comment: It seems to me a simpler solution would be allocate all nodes for a parse tree in an arena. -- ___ Python tracker

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: Now that #26526 landed (thanks to everybody involved!), I'm requesting a review on an updated version of my patch, which addresses the excessive memory consumption by the parser. The description of my original patch still applies: > The attached patch for the

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: Ping? This patch is two months old now. -- ___ Python tracker ___ ___

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: I've now tried it with "perf.py -r -m", and the memory savings are as follows: ### 2to3 ### Mem max: 45976.000 -> 47440.000: 1.0318x larger ### chameleon_v2 ### Mem max: 436968.000 -> 401088.000: 1.0895x smaller ### django_v3 ### Mem max: 23808.000 -> 22584.000:

[issue26415] Fragmentation of the heap memory in the Python parser

Changes by Serhiy Storchaka : -- stage: -> patch review ___ Python tracker ___ ___

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: The attached patch for the parser reduces "Maximum resident set size (kbytes)" threefold, for the degenerate example of 'import ast; ast.parse("0,"*100, mode="eval")', by eliminating many CST nodes that have a single child. According to the comment in node.c

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: @Serhiy: if your build is 32-bit, then every node is half the size, as it mostly consists of pointers. The amount of heap fragmentation can also depend on gcc/glibc version. -- ___ Python tracker

[issue26415] Fragmentation of the heap memory in the Python parser

Serhiy Storchaka added the comment: On my computer peak memory usage in non-debug build is about 450 MB. Peak memory usage in debug build is about 730 MB. I'm not sure this is due to memory fragmentation. -- ___ Python tracker

[issue26415] Fragmentation of the heap memory in the Python parser

STINNER Victor added the comment: libapr library of Apache is designed to group all memory allocations required to handle an HTTP request. It is possible to release *all* memory allocations of the request at once. It looks like the the "pool" object: *

[issue26415] Fragmentation of the heap memory in the Python parser

STINNER Victor added the comment: My misc notes about memory fragmentation: https://haypo-notes.readthedocs.org/heap_fragmentation.html -- ___ Python tracker

[issue26415] Fragmentation of the heap memory in the Python parser

Changes by STINNER Victor <victor.stin...@gmail.com>: -- title: Out of memory, trying to parse a 35MB dict -> Fragmentation of the heap memory in the Python parser ___ Python tracker <rep...@bugs.python.org> <http://bugs.pyt

Re: Python parser problem

Dave, Thanks for the reply. The script was originally edited on Windows with proper \r\n endings, but the PythonTidy script somehow does the doubling (I guess it assumes UNIX format only), i.e., \r\r\n . So indeed, that's kind of messy (and the Python Lang Reference specifies clearly

Re: Python parser problem

On Fri, Dec 14, 2012 at 6:12 AM, RCU alex.e.s...@gmail.com wrote: Dave, Thanks for the reply. The script was originally edited on Windows with proper \r\n endings, It's worth noting that many Windows-based editors and interpreters are quite happy with \n line endings. You may be able

Re: Python parser problem

Chris Angelico ros...@gmail.com writes: On Fri, Dec 14, 2012 at 6:12 AM, RCU alex.e.s...@gmail.com wrote: Dave, Thanks for the reply. The script was originally edited on Windows with proper \r\n endings, It's worth noting that many Windows-based editors and interpreters are quite

Re: Python parser problem

On Fri, Dec 14, 2012 at 6:30 PM, Paul Rudin paul.nos...@rudin.co.uk wrote: Chris Angelico ros...@gmail.com writes: On Fri, Dec 14, 2012 at 6:12 AM, RCU alex.e.s...@gmail.com wrote: Dave, Thanks for the reply. The script was originally edited on Windows with proper \r\n endings,

Python parser problem

Hello. I would like to report a parser bug manifesting on Python 2.5, 2.7 (but not on 2.2) and 3.3. Please see the attached script. Basically this bug appeared after applying PythonTidy on a valid script. More exactly, when running: python -c import iCam_GIT5_5

Re: Python parser problem

On 12/12/2012 02:10 PM, RCU wrote: Hello. I would like to report a parser bug manifesting on Python 2.5, 2.7 (but not on 2.2) and 3.3. Please see the attached script. Basically this bug appeared after applying PythonTidy on a valid script. More exactly, when running:

Re: Python parser problem

On Wed, Dec 12, 2012 at 2:10 PM, RCU alex.e.s...@gmail.com wrote: With this occasion I would like to ask also what are the limits of the Python 2.x and 3.x parser. Where can I find what are the limits on the size/lines of the parsed script? The Python Language Reference is probably what

Re: Python parser problem

On 12/12/2012 2:10 PM, RCU wrote: I would like to report a parser bug manifesting on Python 2.5, 2.7 (but not on 2.2) and 3.3. You are not the first to erroneously attribute a problem to Python itself. But seriously, the interpreter itself is so thoroughly tested on a daily basis that

python-parser running Beautiful Soup only spits out one line of 10. What i have gotten wrong here?

Hello dear Community,. I am trying to get a scraper up and running: And keep running into problems. when I try what you have i have learnedd so far I only get: strongSchuldaten/strong Here is the code that I used: import urllib2 from BeautifulSoup import BeautifulSoup page =

Re: python-parser running Beautiful Soup only spits out one line of 10. What i have gotten wrong here?

Your program is doing what you asked it to do. It finds the first table with class 'bp_ergebnis_tab_info'. Then it ignores that results. Then it finds the first td item in the document, and prints the contents of that. Then it exits. What did you want it to do? Try this. It prints

Re: python-parser running Beautiful Soup needs to be reviewed

I've no opinion. I'm just struggling with BeautifulSoup myself, finding it one of the toughest libs I've seen ;-) Really? While I'm by no means an expert, I find it very easy to work with. It's very well structured IMHO. I think the cause lies in the documentation. The PySide documentation

python-parser running Beautiful Soup needs to be reviewed

Hello commnity i am new to Python and to Beatiful Soup also! It is told to be a great tool to parse and extract content. So here i am...: I want to take the content of a td-tag of a table in a html document. For example, i have this table table class=bp_ergebnis_tab_info tr td

Re: python-parser running Beautiful Soup needs to be reviewed

try using lxml ... its very useful On Sat, Dec 11, 2010 at 11:24 AM, Martin Kaspar martin.kas...@campus-24.com wrote: Hello commnity i am new to Python and to Beatiful Soup also! It is told to be a great tool to parse and extract content. So here i am...: I want to take the content of a

Re: python-parser running Beautiful Soup needs to be reviewed

On 11-12-2010 17:24, Martin Kaspar wrote: Hello commnity i am new to Python and to Beatiful Soup also! It is told to be a great tool to parse and extract content. So here i am...: I want to take the content of a td-tag of a table in a html document. For example, i have this table table

Re: python-parser running Beautiful Soup needs to be reviewed

On Sat, 11 Dec 2010 22:38:43 +0100, Stef Mientki wrote: [snip] So the simplest solution I came up with: Text = table class=bp_ergebnis_tab_info tr td This is a sample text /td td This is the second sample

Re: python-parser running Beautiful Soup needs to be reviewed

On 11.12.2010 22:38, Stef Mientki wrote: On 11-12-2010 17:24, Martin Kaspar wrote: Hello commnity i am new to Python and to Beatiful Soup also! It is told to be a great tool to parse and extract content. So here i am...: I want to take the content of atd-tag of a table in a html document. For

Re: python parser overridden by pymol

On Wed, Nov 11, 2009 at 7:48 PM, Dave Angel da...@ieee.org wrote: Jeremiah wrote: Hello, I'm fairly new to python (version 2.5.4), and am writing a program which uses both pymol (version 1.2r1) and numpy (version 1.3.0) from debian. It appears that when I add pymol to $PYTHONPATH, that

Re: python parser overridden by pymol

Jeremiah H. Savage wrote: To use pymol and numpy together, I now do the following: To ~/.bashrc add: PYMOL_PATH=/usr/lib/pymodules/python2.5/pymol export PYMOL_PATH Then I can do the following in python: import numpy numpy.save('123',numpy.array([1,2,3]))

Re: python parser overridden by pymol

Robert Kern wrote: div class=moz-text-flowed style=font-family: -moz-fixedJeremiah H. Savage wrote: To use pymol and numpy together, I now do the following: To ~/.bashrc add: PYMOL_PATH=/usr/lib/pymodules/python2.5/pymol export PYMOL_PATH Then I can do the following in

Re: python parser overridden by pymol

Jeremiah wrote: However, a problem arises once $PYTHONPATH includes the pymol directory. To use the pymol api, I add the following to ~/.bashrc: PYMOL_PATH=/usr/lib/pymodules/python2.5/pymol export PYMOL_PATH PYTHONPATH=$PYMOL_PATH export PYTHONPATH Don't change your PYTHONPATH

Re: python parser overridden by pymol

Jeremiah wrote: Hello, I'm fairly new to python (version 2.5.4), and am writing a program which uses both pymol (version 1.2r1) and numpy (version 1.3.0) from debian. It appears that when I add pymol to $PYTHONPATH, that parser.expr() is no longer available, and so I am unable to use

Re: Python parser

Kay Schluehr wrote: You'll most likely need a GLR parser. i'm not sure why you think this. as far as i can tell, the OP needs a parser that is suitable for whatever grammar they find (and the grammar will probably be written for a particular parser, which may not be GLR). however, if you are

Re: Python parser

Gabriel Genellina wrote: Do you mean the simpleparser project in Sourceforge? http://simpleparse.sourceforge.net/ I thought this to be one of the most famous and useful Python parsers, because of its combination of simplicity and speed. Anyway, it is *very* good, and not having a version for

Re: Python parser

This reminds me: the SimpleParse developers ran into some troubles porting to Python 2.6. It would be great if someone could give them a hand. Alan Isaac -- http://mail.python.org/mailman/listinfo/python-list

Re: Python parser

En Tue, 03 Mar 2009 22:39:19 -0200, Alan G Isaac ais...@american.edu escribió: This reminds me: the SimpleParse developers ran into some troubles porting to Python 2.6. It would be great if someone could give them a hand. Do you mean the simpleparser project in Sourceforge? Latest alpha

Re: Python parser

On 2 Mrz., 23:14, Clarendon jine...@hotmail.com wrote: Thank you, Lie and Andrew for your help. I have studied NLTK quite closely but its parsers seem to be only for demo. It has a very limited grammar set, and even a parser that is supposed to be large does not have enough grammar to cover

Python parser

Can somebody recommend a good parser that can be used in Python programs? I need a parser with large grammar that can cover a large amount of random texts. Thank you very much. -- http://mail.python.org/mailman/listinfo/python-list

Re: Python parser

Clarendon wrote: Can somebody recommend a good parser that can be used in Python programs? Do you want parser that can parse python source code or parser that works in python? If the latter, pyparsing is a popular choice. Ply is another. There are many choice:

Re: Python parser

if this is for natural language texts you may want to look at http://www.nltk.org/ andrew Clarendon wrote: Can somebody recommend a good parser that can be used in Python programs? I need a parser with large grammar that can cover a large amount of random texts. Thank you very much. --

Re: Python parser

Thank you, Lie and Andrew for your help. I have studied NLTK quite closely but its parsers seem to be only for demo. It has a very limited grammar set, and even a parser that is supposed to be large does not have enough grammar to cover common words like I. I need to parse a large amount of

Re: Python parser

On 2009-03-02 16:14, Clarendon wrote: Thank you, Lie and Andrew for your help. I have studied NLTK quite closely but its parsers seem to be only for demo. It has a very limited grammar set, and even a parser that is supposed to be large does not have enough grammar to cover common words like I.

Re: Python parser

Clarendon wrote: [...] I need to parse a large amount of texts collected from the web (around a couple hundred sentences at a time) very quickly, so I need a parser with a broad scope of grammar, enough to cover all these texts. This is what I mean by 'random'. so the most important things

Help needed with controlling the python parser

the python parser to do this? Thanks, VJ -- http://mail.python.org/mailman/listinfo/python-list

Re: Help needed with controlling the python parser

False, mesg .return True, '' I would augment it with a regular expression to catch __*. [This is left as an exercise to the reader ;-)] Is scripting a tool like pylint the way to go? Or is it fairly easy to control the python parser to do this? I don't know what pylint can do for you

Re: seek python parser

I tend to use the compiler module: http://docs.python.org/lib/compiler.html With the output of the parsing functions I selectively inspect the AST nodes using the named attributes listed in the documentation: http://docs.python.org/lib/module-compiler.ast.html For cases where one just needs to

seek python parser

Hi, Does anyone know of a python source parser program which can return the parsed source file as a coherent dom-like data structure? I'm playing around with ast, and the output of ast.tolist(), but it's got a lot of chaff, and would need a lot of hacking to break it down to simple data

python parser

Hello everyone I have to write a verilog parser in python for a class project. I was wondering if all you folks could advise me on choosing the right python parser module. I am not comfortable with lex/yacc and as a result find myself strugging with any module which use lex/yacc syntax/philosophy

Re: python parser

tuxlover wrote: I have to write a verilog parser in python for a class project. I was wondering if all you folks could advise me on choosing the right python parser module. I am not comfortable with lex/yacc and as a result find myself strugging with any module which use lex/yacc syntax

Re: python parser

tuxlover wrote: Hello everyone I have to write a verilog parser in python for a class project. I was wondering if all you folks could advise me on choosing the right python parser module. I am not comfortable with lex/yacc and as a result find myself strugging with any module which use lex

Re: python parser

On Tue, 12 Jul 2005 13:30:14 -0700, Robert Kern [EMAIL PROTECTED] wrote: tuxlover wrote: Hello everyone I have to write a verilog parser in python for a class project. I was wondering if all you folks could advise me on choosing the right python parser module. I am not comfortable with lex