[issue40661] The new parser segfaults when parsing invalid input

Pablo Galindo Salgado added the comment: New changeset 7b7a21bc4fd063b26a2d1882fddc458861497812 by Lysandros Nikolaou in branch 'master': bpo-40661: Fix segfault when parsing invalid input (GH-20165) https://github.com/python/cpython/commit/7b7a21bc4fd063b26a2d1882fddc458861497812

[issue40661] The new parser segfaults when parsing invalid input

Change by Pablo Galindo Salgado : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue40661] The new parser segfaults when parsing invalid input

Guido van Rossum added the comment: Okay, deferring the blocker. But I'll still land Lysandros' fix ASAP. -- priority: release blocker -> deferred blocker ___ Python tracker

[issue40661] The new parser segfaults when parsing invalid input

STINNER Victor added the comment: I don't think that such bug should block Python 3.9 beta1 (I'm talking about the "release blocker" priority). There is still time before 3.9 final to fix it. -- nosy: +vstinner ___ Python tracker

[issue40661] The new parser segfaults when parsing invalid input

Guido van Rossum added the comment: Unfortunately, I do not understand enough about how Hypothes* works to be helpful here, other than by offering encouragement. (And please don't try to educate me. I have too many other things. Sorry.) -- ___

[issue40661] The new parser segfaults when parsing invalid input

Zac Hatfield-Dodds added the comment: I know what else it might find either, but I still think it's worth running property-based tests in CI to find out! The demo I wrote for my language summit talk doesn't have any parser tests, but still would have caught this bug in the pull request

[issue40661] The new parser segfaults when parsing invalid input

Guido van Rossum added the comment: Zac: The reproducer here apparently uses a long string of weird accented characters. I'm not sure how to generalize from that to other things that Hyothes* could find. But maybe this helps:

[issue40661] The new parser segfaults when parsing invalid input

Zac Hatfield-Dodds added the comment: I understand from Paul Ganssle that this bug was found using Hypothesmith in my stdlib property tests (reported at https://github.com/Zac-HD/stdlib-property-tests/issues/14). As discussed in https://github.com/we-like-parsers/cpython/issues/91 and

[issue40661] The new parser segfaults when parsing invalid input

Change by Lysandros Nikolaou : -- pull_requests: +19464 pull_request: https://github.com/python/cpython/pull/20165 ___ Python tracker ___

[issue40661] The new parser segfaults when parsing invalid input

Change by Guido van Rossum : -- keywords: +patch pull_requests: +19462 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/20159 ___ Python tracker

[issue40661] The new parser segfaults when parsing invalid input

Guido van Rossum added the comment: I see almost no time difference for 'make time_stdlib': before 3.471, after 3.451. But I see a serious difference for 'make time_compile': before 3.474, after 4.996. That's over 40% slower (on the extreme case, xxl.py). I'll prepare a PR just in case.

[issue40661] The new parser segfaults when parsing invalid input

Pablo Galindo Salgado added the comment: A quick benchmark using xxl.py: Base time (master): Time: 9.386 seconds on an average of 20 runs With the patch in this issue: Time: 9.498 seconds on an average of 20 runs Sadly I could not test with PGO/LTO and without CPU isolation, so it would be

[issue40661] The new parser segfaults when parsing invalid input

Guido van Rossum added the comment: How costly is PyErr_Occurred()? That worries me most, otherwise I’d accept this right away. -- ___ Python tracker ___

[issue40661] The new parser segfaults when parsing invalid input

Pablo Galindo Salgado added the comment: Indeed, that diff solves the problem -- ___ Python tracker ___ ___ Python-bugs-list

[issue40661] The new parser segfaults when parsing invalid input

Pablo Galindo Salgado added the comment: I think we may need to test for the error indicator (and maybe PyErr_Ocurred for safety) before every alternative. Something like: diff --git a/Tools/peg_generator/pegen/c_generator.py b/Tools/peg_generator/pegen/c_generator.py index

[issue40661] The new parser segfaults when parsing invalid input

New submission from Pablo Galindo Salgado : The new peg parser segfaults when parsing the attached reproducer. this is due to the fact that the exception set by `tokenizer_error` is not properly propagated. -- components: Interpreter Core files: reproducer.py messages: 369132 nosy: