[issue32911] Doc strings no longer stored in body of AST

Serhiy Storchaka added the comment: New changeset d6bbb57855da3f92521edd901f49705be46fb203 by Serhiy Storchaka in branch '3.7': bpo-32911: Update the historical note about the magic number. (GH-7272) https://github.com/python/cpython/commit/d6bbb57855da3f92521edd901f49705be46fb203 -

[issue32911] Doc strings no longer stored in body of AST

Serhiy Storchaka added the comment: New changeset 941ec210aaef517cf818b043ec28345962f78465 by Serhiy Storchaka in branch 'master': bpo-32911: Add the historical note about the magic number. (GH-7273) https://github.com/python/cpython/commit/941ec210aaef517cf818b043ec28345962f78465 -

[issue32911] Doc strings no longer stored in body of AST

Change by Serhiy Storchaka : -- pull_requests: +6900 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://

[issue32911] Doc strings no longer stored in body of AST

Change by Serhiy Storchaka : -- pull_requests: +6899 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://

[issue32911] Doc strings no longer stored in body of AST

Nick Coghlan added the comment: In that case, closing this as resolved. Folks interested in the idea of a "suite" compilation mode to compile a series of statements without special-casing the first one as a potential docstring, as well as the idea of a dedicated DocString AST node may want t

[issue32911] Doc strings no longer stored in body of AST

Serhiy Storchaka added the comment: I concur. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue32911] Doc strings no longer stored in body of AST

INADA Naoki added the comment: I think it's OK. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://m

[issue32911] Doc strings no longer stored in body of AST

Ned Deily added the comment: Thank you all for your input and thank you, Inada-san and Serhiy, for the PRs. Is there anything more that needs to go in for 3.7.0b5 tomorrow so that this can be tested before the release candidate? If not, can we now close this again? If new issues arise wit

[issue32911] Doc strings no longer stored in body of AST

Serhiy Storchaka added the comment: New changeset 73cbe7a01a22d02dbe1ec841e8779c775cad3d08 by Serhiy Storchaka in branch 'master': bpo-32911: Revert bpo-29463. (GH-7121) (GH-7197) https://github.com/python/cpython/commit/73cbe7a01a22d02dbe1ec841e8779c775cad3d08 -- _

[issue32911] Doc strings no longer stored in body of AST

Change by Serhiy Storchaka : -- pull_requests: +6831 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://

[issue32911] Doc strings no longer stored in body of AST

Serhiy Storchaka added the comment: New changeset 2641ee5040abb090e66e4ff80c33b76729b36e75 by Serhiy Storchaka in branch '3.7': bpo-32911: Revert bpo-29463. (GH-7121) https://github.com/python/cpython/commit/2641ee5040abb090e66e4ff80c33b76729b36e75 -- __

[issue32911] Doc strings no longer stored in body of AST

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.py

[issue32911] Doc strings no longer stored in body of AST

Benjamin Ragan-Kelley added the comment: That should work well for us. Our patches for this are all conditional on the module body being empty, so reverting causes us no issues at all. Thank you! -- nosy: +Benjamin Ragan-Kelley ___ Python tracker

[issue32911] Doc strings no longer stored in body of AST

Ned Deily added the comment: OK, since I believe everyone who has spoken up so far has chosen B or a variation on it, I think we can eliminate option A. And there also seems to be a consensus so far among the core developers who have spoken up for the approach in PR 7121. Before we commit t

[issue32911] Doc strings no longer stored in body of AST

INADA Naoki added the comment: I'm +1 on PR-7121 too. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue32911] Doc strings no longer stored in body of AST

Nick Coghlan added the comment: After looking at potential implementation approaches, I believe the following would be the ASDL code generator changes needed to present the docstring as part of the body in the Python API, while keeping it as a separate attribute in the C API: * in ast2obj, a

[issue32911] Doc strings no longer stored in body of AST

Nick Coghlan added the comment: Just noting that I'm going to attempt an alternative, hopefully lower impact, approach at implementing the reversion, which will be to modify Parser/asdl_c.py to inject the following constructor postamble for ASDL nodes that define both a "body" and a "docstrin

[issue32911] Doc strings no longer stored in body of AST

Matthias Bussonnier added the comment: > Just to quickly touch on Matthias' question about opt-in or deprecations, a > key thing to note is the ast module is automatically generated from the ASDL > file, so either of those approaches would require developing a new mechanism > in the code gene

[issue32911] Doc strings no longer stored in body of AST

Brett Cannon added the comment: Just to quickly touch on Matthias' question about opt-in or deprecations, a key thing to note is the ast module is automatically generated from the ASDL file, so either of those approaches would require developing a new mechanism in the code generator to suppor

[issue32911] Doc strings no longer stored in body of AST

Serhiy Storchaka added the comment: PR 7121 is based on PR 5927 and goes one step further. It doesn't introduce a new AST type DocString. Third-party code should not need any adaptation for 3.7. -- ___ Python tracker

[issue32911] Doc strings no longer stored in body of AST

Change by Serhiy Storchaka : -- keywords: +patch pull_requests: +6756 stage: commit review -> patch review ___ Python tracker ___ ___

[issue32911] Doc strings no longer stored in body of AST

INADA Naoki added the comment: > B. Revert the feature now for 3.7.0, retargeting for 3.8, and produce a > 3.7.0b5 on a somewhat shorter cycle to allow downstream users to adapt to the > removal. Please note that it can't be reverted simply. The change was introduced to ease AST-layer optimiz

[issue32911] Doc strings no longer stored in body of AST

Matthias Bussonnier added the comment: Łukasz Langa wrote: > Inadasan, I think what we should do is to amend `ast.parse()` and `compile()` > docs that mode="exec" treats given code as a module, we should even give the > docstring handling as an example. That is what I proposed in https://bug

[issue32911] Doc strings no longer stored in body of AST

Nick Coghlan added the comment: Ouch, I'd completely missed the fact that this would affect the parsing of bare strings to a simple AST (I was focused on functions and classes, as in Mark's original example). So even though I'm the author of https://bugs.python.org/issue11549#msg193656 (wher

[issue32911] Doc strings no longer stored in body of AST

Change by Eitan Adler : -- nosy: +eitan.adler ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue32911] Doc strings no longer stored in body of AST

Min RK added the comment: In the A/B vote, I cast mine for B, for what it is worth, but it is not strongly held. >From the IPython side, I don't view our particular issue as a major regression >for users. The only affected case for us is interactively typed string >literals in single stateme

[issue32911] Doc strings no longer stored in body of AST

Ned Deily added the comment: Based on more recent discussions and indirect feedback from downstream users (primarily the recent IPython experience), I am reluctantly re-opening this issue for 3.7.0. I take responsibility for encouraging us earlier in the beta phase to continue with the featu

[issue32911] Doc strings no longer stored in body of AST

Ned Deily added the comment: For the record, Serhiy commented on this issue today in a thread on the python-committers list: "I have doubts about two issues. I feel the responsibility for them because I had the opportunity to solve them before, but I lost it. 1. Changes in the AST. Few third

[issue32911] Doc strings no longer stored in body of AST

Benjamin Peterson added the comment: Regardless of the value of .docstring change, it seems it had some unexpected consequences. I think providing a new mode for compile mulitline statements is a reasonable way to address those consequences going forward. That said, it's extremely late in the

[issue32911] Doc strings no longer stored in body of AST

STINNER Victor added the comment: IMHO Python 3.7 will be simpler to process, since it avoids to check if the first node is a docstring or not. I agree that it's a backward incompatible, but it has been done on purpose. In fact, there is no warranty about backward compatibility on the genreat

[issue32911] Doc strings no longer stored in body of AST

Ned Deily added the comment: > * Merge my patch in 3.7rc1, or Sorry but we are not going to revisit this decision for 3.7.0. It's too late! -- ___ Python tracker ___

[issue32911] Doc strings no longer stored in body of AST

INADA Naoki added the comment: If Python 3.7 is released with current AST form, I don't want to change it again. I prefer * Merge my patch in 3.7rc1, or * Add compile(..., "multi") mode for "sequence of statements" -- ___ Python tracker

[issue32911] Doc strings no longer stored in body of AST

Serhiy Storchaka added the comment: Maybe we will need to return to this issue in future and merge Inada's patch. IPython has been needed to add a workaround for this change. [1] And maybe Python will be needed to add that workaround if once it will add the support of pasting multiple complex

[issue32911] Doc strings no longer stored in body of AST

Łukasz Langa added the comment: Oh, and Francis, you might actually want to use lib2to3.pytree instead, which will provide you with a Concrete Syntax Tree ("lossless AST"). That tree will keep information like whitespace, comments, and other semantically insignificant tokens like commas and

[issue32911] Doc strings no longer stored in body of AST

Łukasz Langa added the comment: Inadasan, what Francis is referring to is that people often use `ast.parse()` to parse code fragments, not entire modules. For example: >>> some_code = '"just a string"' >>> some_code_ast = ast.parse(some_code) The way `ast.parse()` works, it always consid

[issue32911] Doc strings no longer stored in body of AST

INADA Naoki added the comment: > in practice `ast.parse()` is often used with smaller fragments of source code > that don't directly correspond to a complete module, and this behaviour makes > no sense in such cases. > > The resulting bug in kdev-python took a while to track down (because I ha

[issue32911] Doc strings no longer stored in body of AST

Francis Herne added the comment: Note: Because this also applies to module-level docstrings, a rather strange effect that, for example, `ast.parse("'foo'")` is now an empty module. While the root node is always an instance of `ast.Module`, in practice `ast.parse()` is often used with smaller

[issue32911] Doc strings no longer stored in body of AST

Ned Deily added the comment: Thanks everyone for your inputs. Let's go with the status quo -> closing this issue. -- priority: deferred blocker -> resolution: -> wont fix stage: -> resolved status: open -> closed ___ Python tracker

[issue32911] Doc strings no longer stored in body of AST

Mark Shannon added the comment: I still think that option 2 would be best, but given that it is late in the release cycle, I will accept the status quo. A couple of comments for the record: Constant folding occurs after AST creation, so doesn't make the AST worse. The parse tree created by l

[issue32911] Doc strings no longer stored in body of AST

Łukasz Langa added the comment: Python's AST already doesn't include a lot of syntax that has no runtime effect: - comments; - whitespace and punctuation. The situation is now made "worse" with constant folding. You won't have full line information on the full expression, you don't even fully

[issue32911] Doc strings no longer stored in body of AST

INADA Naoki added the comment: There are no enough +1 for merging GH-5927 yet on the ML discussion. https://mail.python.org/pipermail/python-dev/2018-February/152311.html @Mark.Shannon, would you comment your opinion to the ML thread? For now, I don't want to break tools supporting Python 3.7

[issue32911] Doc strings no longer stored in body of AST

Ned Deily added the comment: Since we are already past the 3.7.0b2 cutoff time and there does not seen to be a consensus that the current 3.7 behavior needs to change and the proposed change is quite large, I think we should not change anything now for b2. You can have a discussion on python

[issue32911] Doc strings no longer stored in body of AST

Serhiy Storchaka added the comment: Sorry, Python-Dev, of course. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubs

[issue32911] Doc strings no longer stored in body of AST

INADA Naoki added the comment: Python-list? -dev? -ideas? -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue32911] Doc strings no longer stored in body of AST

Serhiy Storchaka added the comment: Would be nice to discuss this design question on Python-list. -- ___ Python tracker ___ ___ Pyth

[issue32911] Doc strings no longer stored in body of AST

INADA Naoki added the comment: I'm implementing (2). Please check GH-5927. -- keywords: -patch stage: patch review -> ___ Python tracker ___

[issue32911] Doc strings no longer stored in body of AST

Change by INADA Naoki : -- keywords: +patch pull_requests: +5698 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-list

[issue32911] Doc strings no longer stored in body of AST

INADA Naoki added the comment: I tried this patch: diff --git a/Python/ast.c b/Python/ast.c index e2092f0f85..93be2bc839 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -3537,9 +3537,9 @@ docstring_from_stmts(asdl_seq *stmts) if (s->kind == Expr_kind && s->v.Expr.value->kind == Str_kin

[issue32911] Doc strings no longer stored in body of AST

Ned Deily added the comment: Is this going to get resolved in time for 3.7.0b2? If we need to change this behavior, I really do not want to delay this for b3 which is when the 3.7.0 ABI freezes; otherwise such a major behavior change would likely need to wait until 3.8. -- priority:

[issue32911] Doc strings no longer stored in body of AST

Change by ppperry : -- title: Doc strings omitted from AST -> Doc strings no longer stored in body of AST ___ Python tracker ___ ___