Our experience with automatic testing is that unfortunately is very
difficult to extract real problems with it. We tried some of the new
experimental source generators on top of hypothesis (
https://pypi.org/project/hypothesmith/) and sadly we could not catch many
important things that parsing exis
In this case, you can use the old parser as an oracle, at least for python
3.8 syntax. The new parser should produce a syntax error if and only if the
old one does. And if it doesn't the AST should be the same I guess (I'm not
sue if the AST structure changed)
On Thu, 8 Oct 2020, 03:12 Terry Reedy
On 10/6/2020 2:02 PM, Guido van Rossum wrote:
That's appreciated, but I think what's needed more is someone who
actually wants to undertake this project. It's not just a matter of
running a small script for hours -- someone will have to come up with a
way to fuzz that is actually useful for thi
That's appreciated, but I think what's needed more is someone who actually
wants to undertake this project. It's not just a matter of running a small
script for hours -- someone will have to come up with a way to fuzz that is
actually useful for this particular situation (inserting random character
Hi Guido, Pablo & Lysandros,
I'm excited about this improvement to Python, and was interested to hear
about it at the language summit as well. I happen to be friends with
Alessandro Warth, whom you cited in the PEP as developing the packrat
parsing technique you use (at least in part). I wrote t
On Tue, Apr 21, 2020 at 9:35 PM Gregory P. Smith wrote:
> Could we go ahead and mark lib2to3 as Pending Deprecation in 3.9 so we can
> get it out of the stdlib by 3.11 or 3.12?
>
I'm going ahead and tracking the idea in https://bugs.python.org/issue40360.
>
> lib2to3 is the basis of all sorts
Could we go ahead and mark lib2to3 as Pending Deprecation in 3.9 so we can
get it out of the stdlib by 3.11 or 3.12?
lib2to3 is the basis of all sorts of general source code manipulation
tooling. Its name and original reason d'etre have moved on. It is
actively used to parse and rewrite Python 3
Great! Please submit a PR to update the [lib]2to3 docs and CC me
(@gvanrossum).
While perhaps it wouldn't hurt if the PEP mentioned lib2to3, it was just
accepted by the Steering Council without such language, and I wouldn't want
to imply that the SC agrees with everything I said. So I still think
On Sat, Apr 18, 2020 at 10:38 PM Guido van Rossum wrote:
>
> Note that, while there is indeed a docs page about 2to3, the only docs for
> lib2to3 in the standard library reference are a link to the source code and a
> single "Note: The lib2to3 API should be considered unstable and may change
>
On Sat, Apr 18, 2020 at 9:45 PM Guido van Rossum wrote:
> But I never found the time to make it an appealing parser generator tool
> for other languages, even though that was on my mind as a future
> possibility.
>
I simply want to +1 on this. A general purpose parser library in the stdlib
woul
On Sat, Apr 18, 2020 at 4:53 PM Carl Meyer wrote:
> The PEP is exciting and is very clearly presented, thank you all for
> the hard work!
>
> Considering the comments in the PEP about the new parser not
> preserving a parse tree or CST, I have some questions about the future
> options for Python
The PEP is exciting and is very clearly presented, thank you all for
the hard work!
Considering the comments in the PEP about the new parser not
preserving a parse tree or CST, I have some questions about the future
options for Python language-services tooling which requires a CST in
order to roun
After the feedback received in the language summit, we have made a modification
to the
proposed migration plan in PEP 617 so the new parser will be the default in
3.9alpha6:
https://github.com/python/peps/pull/1369
___
Python-Dev mailing list -- python
On Mon, Apr 06, 2020 at 07:03:30PM -0700, Guido van Rossum wrote:
> After 30 years am I not allowed to take new information into account and
> consider a change of heart? :-)
Of course :-)
--
Steven
___
Python-Dev mailing list -- python-dev@python.org
Another point in favour of always-reserved keywords is that they
make life a lot easier for syntax highlighters.
--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python
On 7/04/20 6:54 am, Guido van Rossum wrote:
I'm not sure that that was the conclusion. At the time the point was
that we *wanted* all keywords to be reserved everywhere, an `as` was an
ugly exception to that rule, which we got rid of as soon as we could --
not because it was a bad idea but bec
On 7/04/20 5:43 am, Guido van Rossum wrote:
The biggest difference is that the `|` operator is no longer
symmetrical (since if you have alternatives `A | B`, and both match at
some point in the input, PEG reports A, while the old generator would
reject the grammar as being ambiguous.
I'm stil
After 30 years am I not allowed to take new information into account and
consider a change of heart? :-)
On Mon, Apr 6, 2020 at 6:21 PM Steven D'Aprano wrote:
> On Mon, Apr 06, 2020 at 11:54:54AM -0700, Guido van Rossum wrote:
>
> > (In an early version of the PEG parser, all keywords were
> > c
On Mon, Apr 06, 2020 at 11:54:54AM -0700, Guido van Rossum wrote:
> (In an early version of the PEG parser, all keywords were
> context-sensitive, and there were only very few places in the grammar where
> this required us to insert negative lookaheads to make edge cases parse
> correctly. The res
On Mon, Apr 6, 2020 at 8:04 PM Guido van Rossum wrote:
> On Mon, Apr 6, 2020 at 11:36 AM Steven D'Aprano
> wrote:
>
>>
>> Personally, I would not like to have to explain to newcomers why `match`
>> is a keyword but you can still use it as a function or variable, but not
>> other keywords like `r
On Tue, Apr 7, 2020 at 5:03 AM Guido van Rossum wrote:
>
> On Mon, Apr 6, 2020 at 11:36 AM Steven D'Aprano wrote:
>> Personally, I would not like to have to explain to newcomers why `match`
>> is a keyword but you can still use it as a function or variable, but not
>> other keywords like `raise`,
On Mon, Apr 6, 2020 at 11:36 AM Steven D'Aprano wrote:
> On Mon, Apr 06, 2020 at 10:43:11AM -0700, Guido van Rossum wrote:
>
> > I've been toying with the idea of introducing a "match" statement
> > similar to Scala's match expression by making "match" a keyword only when
> > followed by an expre
On Mon, Apr 06, 2020 at 10:43:11AM -0700, Guido van Rossum wrote:
> I've been toying with the idea of introducing a "match" statement
> similar to Scala's match expression by making "match" a keyword only when
> followed by an expression and a colon.)
Didn't we conclude from `as` that having cont
On Mon, Apr 6, 2020 at 5:18 AM Jeff Allen wrote:
> The PEP gives a good exposition of the problem and proposed solution,
> thanks.
>
> If I understand correctly, the proposal is that the PEG grammar should
> become the definitive grammar for Python at some point, probably for Python
> 3.10, so it
On Mon, Apr 6, 2020 at 4:03 AM Fabio Zadrozny wrote:
> I think using a PEG parser is interesting, but I do have some questions
> related to what's to expect in the future for other people which have to
> follow the Python grammar, so, can you shed some light on this?
>
> Does that mean that the g
The PEP gives a good exposition of the problem and proposed solution,
thanks.
If I understand correctly, the proposal is that the PEG grammar should
become the definitive grammar for Python at some point, probably for
Python 3.10, so it may evolve without the LL(1) restrictions. I'd like
to r
On Thu, Apr 2, 2020 at 3:16 PM Guido van Rossum wrote:
> Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros
> Nikolaou and myself have been working on a new parser for CPython. We are
> now far enough along that we present a PEP we've written:
>
> https://www.python.org/dev
On Sun, Apr 5, 2020 at 5:16 PM Greg Ewing
wrote:
> On 6/04/20 4:48 am, Guido van Rossum wrote:
> > There's no need to worry about this: in almost all cases the error
> > indicator points to the same spot in the source code as with the old
> > parser.
>
> I'm curious about how that works. From the
On 6/04/20 4:48 am, Guido van Rossum wrote:
There's no need to worry about this: in almost all cases the error
indicator points to the same spot in the source code as with the old
parser.
I'm curious about how that works. From the description in the PEP,
it seems that none of the individual pa
The tl;dr is that actions specified in the grammar are specific to the
target language. So if you want to use the pegen tool to generate both
Python and C code for the same grammar, you would need two grammar files
with the same grammar but different actions. Since our goal here is just to
generate
> The only thing I'm missing from the PEP is more detail about how the
cross-language nature of the parser actions are handled.
Expanded the "actions" section in the PEP here:
https://github.com/python/peps/pull/1357
___
Python-Dev mailing list -- pytho
> On 6/04/20 2:08 am, Jelle Zijlstra wrote:
> > The current CPython parser usually just produces "SyntaxError: invalid
> > syntax" for any error, while other languages that I work with usually
> > say something more precise like 'expected x, got y'. What will the error
> > messages in the PEG parse
On 6/04/20 2:08 am, Jelle Zijlstra wrote:
The current CPython parser usually just produces "SyntaxError: invalid
syntax" for any error, while other languages that I work with usually
say something more precise like 'expected x, got y'. What will the error
messages in the PEG parser look like? M
El jue., 2 abr. 2020 a las 11:19, Guido van Rossum ()
escribió:
> Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros
> Nikolaou and myself have been working on a new parser for CPython. We are
> now far enough along that we present a PEP we've written:
>
> https://www.python
On 4/04/20 9:29 am, Brett Cannon wrote:
I think "needs" is a bit strong. It would be nice, though. Regardless, as long
as this is a net improvement over the status quo I don't see this being rejected on the
grounds that an LR or LALR parser would be better since we have a working PEG parser
to
Greg Ewing wrote:
> On 3/04/20 7:10 am, Guido van Rossum wrote:
> > Since last fall's core sprint in London, Pablo
> > Galindo Salgado,
> > Lysandros Nikolaou and myself have been working on a new parser for
> > CPython. We are now far enough along that we present a PEP we've written:
> > https:/
>The only thing I'm missing from the PEP is more detail about how the
> cross-language nature of the parser actions are handled. The example covers
> just C, and the description of the actions says they're C expressions. The
> only mention of Python code generation is for alternatives without actio
> That paragraph seems rather confused. I think what it might be
> trying to say is that a PEG parser allows you to write productions
> with overlapping first sets (which would be "ambiguous" for an
> LL parser), but still somehow guarantees that a unique parse tree
> is produced. The latter sugges
Thanks, Guido, Pablo, Lysandros, that's a great PEP. Also thanks to
everyone else working on the PEG parser over the last year, like Emily. I
know it's a lot of work but as someone who's intimately aware of the
headaches caused by the LL(1) parser, I greatly appreciate it :).
The only thing I'm mi
On Thu, Apr 02, 2020 at 08:57:30PM -0700, Guido van Rossum wrote:
> On Thu, Apr 2, 2020 at 7:55 PM Matt Billenstein wrote:
>
> Even just running it in a dev build against the corpus of the top few
> thousand packages on pypi might give enough confidence -- I had a script
> to download
On Thu, Apr 2, 2020 at 7:55 PM Matt Billenstein wrote:
> On Thu, Apr 02, 2020 at 05:17:31PM -0700, Guido van Rossum wrote:
> > On Thu, Apr 2, 2020 at 4:20 PM Nathaniel Smith wrote:
> >
> > If the AST is supposed to be the same, then would it make sense to
> > temporarily – maybe just dur
On Thu, Apr 02, 2020 at 05:17:31PM -0700, Guido van Rossum wrote:
> On Thu, Apr 2, 2020 at 4:20 PM Nathaniel Smith wrote:
>
> If the AST is supposed to be the same, then would it make sense to
> temporarily – maybe just during the alpha/beta period – always run
> *both* parsers and co
On 3/04/20 3:22 pm, Guido van Rossum wrote:
This allows more freedom in designing a grammar.
For example, it would let a language designer solve the "dangling else"
problem from the Wikipedia page, by writing the form including the
"else" clause first .
I'm inclined to think that such problem
On 3/04/20 2:13 pm, Victor Stinner wrote:
"Unlike LL(1) parsers PEG-based parsers cannot be ambiguous: if a
string parses, it has exactly one valid parse tree. This means that a
PEG-based parser cannot suffer from the ambiguity problems described
in the previous section."
That paragraph seems r
Le ven. 3 avr. 2020 à 02:58, Greg Ewing a
écrit :
> On 3/04/20 10:33 am, Victor Stinner wrote:
> > I also like the fact that PEG is deterministic, whereas
> > LL(1) parsers are not.
>
> Where do you get that LL(1) parsers are not deterministic?
> That's news to me!
On Thu, Apr 2, 2020 at 6:15 PM
Wonderful news! I'm really excited to see what is coming alongside
this flexible parser.
On Thu, Apr 2, 2020 at 9:16 PM Guido van Rossum wrote:
>
> Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros
> Nikolaou and myself have been working on a new parser for CPython. We ar
Sorry, I was referring to *ambiguous* grammar rules. Extract of the PEP:
"Unlike LL(1) parsers PEG-based parsers cannot be ambiguous: if a
string parses, it has exactly one valid parse tree. This means that a
PEG-based parser cannot suffer from the ambiguity problems described
in the previous sect
On 3/04/20 10:33 am, Victor Stinner wrote:
I also like the fact that PEG is deterministic, whereas
LL(1) parsers are not.
Where do you get that LL(1) parsers are not deterministic?
That's news to me!
--
Greg
___
Python-Dev mailing list -- python-dev@
On 3/04/20 7:10 am, Guido van Rossum wrote:
Since last fall's core sprint in London, Pablo Galindo Salgado,
Lysandros Nikolaou and myself have been working on a new parser for
CPython. We are now far enough along that we present a PEP we've written:
https://www.python.org/dev/peps/pep-0617/
On Thu, Apr 2, 2020 at 4:20 PM Nathaniel Smith wrote:
> On Thu, Apr 2, 2020 at 2:48 PM Pablo Galindo Salgado
> wrote:
> >
> > > About the migration, can I ask who is going to (help to) fix projects
> > which rely on the AST?
> >
> > I think you misunderstood: The AST is exactly the same as the o
On Thu, Apr 2, 2020 at 2:55 PM Pablo Galindo Salgado
wrote:
> > About the migration, can I ask who is going to (help
> to) fix projects which rely on the AST?
>
> Whoops, I send the latest email before finishing it by mistake. Here is
> the extended version of the answer:
>
> I think there is a m
On Thu, Apr 2, 2020 at 2:48 PM Pablo Galindo Salgado
wrote:
>
> > About the migration, can I ask who is going to (help to) fix projects
> which rely on the AST?
>
> I think you misunderstood: The AST is exactly the same as the old and the new
> parser. The only
> the thing that the new parser doe
On Thu, Apr 2, 2020 at 12:43 PM Paul Moore wrote:
> On Thu, 2 Apr 2020 at 19:20, Guido van Rossum wrote:
> >
> > Since last fall's core sprint in London, Pablo Galindo Salgado,
> Lysandros Nikolaou and myself have been working on a new parser for
> CPython. We are now far enough along that we pr
On Thu, Apr 2, 2020 at 1:21 PM Barry Warsaw wrote:
> On Apr 2, 2020, at 13:07, Pablo Galindo Salgado
> wrote:
> >
> >> Just to clarify, this means that 3.9 will ship with the PEG parser as
> default,
> >> right? If so, this would be a new feature, post beta. Since that is
> counter to our
> >>
> About the migration, can I ask who is going to (help
to) fix projects which rely on the AST?
Whoops, I send the latest email before finishing it by mistake. Here is the
extended version of the answer:
I think there is a misunderstanding here: The new parser generates the same AST
as the old p
> About the migration, can I ask who is going to (help to) fix projects
which rely on the AST?
I think you misunderstood: The AST is exactly the same as the old and the new
parser. The only
the thing that the new parser does is not generate an immediate CST (Concrete
Syntax Tree) and that
is onl
Hi,
It's great to see that you finally managed to come up with a PEP, this
work becomes concrete: congrats!
I started to read the PEP, and it's really well written! I heard that
LL(1) parsers have limits, but this PEP explains very well that the
current Python grammar was already "hacked" to work
On Apr 2, 2020, at 13:07, Pablo Galindo Salgado wrote:
>
>> Just to clarify, this means that 3.9 will ship with the PEG parser as
>> default,
>> right? If so, this would be a new feature, post beta. Since that is
>> counter to our
>> general policy, we would need to get explicit RM approval f
> Just to clarify, this means that 3.9 will ship with the PEG parser as default,
> right? If so, this would be a new feature, post beta. Since that is counter
> to our
> general policy, we would need to get explicit RM approval for such a change.
The idea is merging it *before beta* and make it
Great to see this new work pay off!
On Apr 2, 2020, at 11:10, Guido van Rossum wrote:
> 2. After Python 3.9 Beta 1 the default parser will be the new parser.
Just to clarify, this means that 3.9 will ship with the PEG parser as default,
right? If so, this would be a new feature, post beta.
On Thu, 2 Apr 2020 at 19:20, Guido van Rossum wrote:
>
> Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros
> Nikolaou and myself have been working on a new parser for CPython. We are now
> far enough along that we present a PEP we've written:
>
> https://www.python.org/de
These are good news. I think the new parser is indeed both simpler and more
flexible - great!
--
Ivan
On Thu, 2 Apr 2020 at 19:19, Guido van Rossum wrote:
> Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros
> Nikolaou and myself have been working on a new parser for CP
62 matches
Mail list logo
