Re: [Python-Dev] Improving the bytecode
> On Jun 4, 2016, at 1:08 AM, Serhiy Storchaka wrote: > > Following the converting 8-bit bytecode to 16-bit bytecode (wordcode), there > are other issues for improving the bytecode. > > 1. http://bugs.python.org/issue27129 > Make the bytecode more 16-bit oriented. I don' think this should be done. Adding the /2 and *2 just complicates the code and messes with my ability to reason about jumps. With VM opcodes, there is always a tension between being close to implementation (what byte address are we jumping to) and being high level (what is the word offset). In this case, I think we should stay with the former because they are primarily used in ceval.c and peephole.c which are close to the implementation. At the higher level, there isn't any real benefit either (because dis.py already does a nice job of translating the jump targets). Here is one example of the parts of the diff that cause concern that future maintenance will be made more difficult by the change: -j = blocks[j + i + 2] - blocks[i] - 2; +j = (blocks[j * 2 + i + 2] - blocks[i] - 2) / 2; Reviewing the original line only gives me a mild headache while the second one really makes me want to avert my eyes ;-) > 2. http://bugs.python.org/issue27140 > Add new opcode BUILD_CONST_KEY_MAP for building a dict with constant keys. > This optimize the common case and especially helpful for two following issues > (creating and calling functions). This shows promise. The proposed name BUILD_CONST_KEY_MAP is much more clear than BUILD_MAP_EX. > 3. http://bugs.python.org/issue27095 > Simplify MAKE_FUNCTION/MAKE_CLOSURE. Instead packing three numbers in oparg > the new MAKE_FUNCTION takes built tuples and dicts from the stack. > MAKE_FUNCTION and MAKE_CLOSURE are merged in the single opcode. > > 4. http://bugs.python.org/issue27213 > Rework CALL_FUNCTION* opcodes. Replace four existing opcodes with three > simpler and more efficient opcodes. +1 > 5. http://bugs.python.org/issue27127 > Rework the for loop implementation. I'm unclear what problem is being solved by requiring that GET_ITER always followed immediately by FOR_ITER. > 6. http://bugs.python.org/issue17611 > Move unwinding of stack for "pseudo exceptions" from interpreter to compiler. I have mixed feelings on this one, at once applauding efforts to simplify an eternally messy part of the eval loop and at the same time worried that it throws aways years of tweaks and improvements that came beforehand. This is more of a major surgery than the other patches. Raymond Hettinger ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improving the bytecode
On 05.06.16 21:24, Raymond Hettinger wrote: On Jun 4, 2016, at 1:08 AM, Serhiy Storchaka wrote: 1. http://bugs.python.org/issue27129 Make the bytecode more 16-bit oriented. I don' think this should be done. Adding the /2 and *2 just complicates the code and messes with my ability to reason about jumps. With VM opcodes, there is always a tension between being close to implementation (what byte address are we jumping to) and being high level (what is the word offset). In this case, I think we should stay with the former because they are primarily used in ceval.c and peephole.c which are close to the implementation. At the higher level, there isn't any real benefit either (because dis.py already does a nice job of translating the jump targets). Here is one example of the parts of the diff that cause concern that future maintenance will be made more difficult by the change: -j = blocks[j + i + 2] - blocks[i] - 2; +j = (blocks[j * 2 + i + 2] - blocks[i] - 2) / 2; Reviewing the original line only gives me a mild headache while the second one really makes me want to avert my eyes ;-) The /2 and *2 are added just because Victor wants to keep f_lineno counting bytes. Please look at my first patch. It doesn't contain /2 and *2. It even contains much less +2 and -2. For example the above change looks as: -j = blocks[j + i + 2] - blocks[i] - 2; +j = blocks[j + i + 1] - blocks[i] - 1; Doesn't this give you less headache? 2. http://bugs.python.org/issue27140 Add new opcode BUILD_CONST_KEY_MAP for building a dict with constant keys. This optimize the common case and especially helpful for two following issues (creating and calling functions). This shows promise. The proposed name BUILD_CONST_KEY_MAP is much more clear than BUILD_MAP_EX. If you accept this patch, I'll commit it. At least two other issues wait this. 5. http://bugs.python.org/issue27127 Rework the for loop implementation. I'm unclear what problem is being solved by requiring that GET_ITER always followed immediately by FOR_ITER. As I understand, the purpose was to decrease the number of executed opcodes. It looks to me that existing patch is not acceptable, because there is a reason for using two opcodes in the for loop start. But I think that we can use other optimization here. I'll try to write a patch. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C99
Guido van Rossum wrote: > I'm talking about 3rd party extensions. Those may require source > compatibility with older Python versions. All I'm asking for is to not > require source-level use of C99 features. This of course removes a lot of its usefulness. E.g. macros cannot be replaced by inline functions, as header files must still be plain C89. Sturla Molden ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C99
> -Original Message- > From: Python-Dev [mailto:python-dev-bounces+tritium- > [email protected]] On Behalf Of Sturla Molden > Sent: Sunday, June 5, 2016 10:29 PM > To: [email protected] > Subject: Re: [Python-Dev] C99 > > Guido van Rossum wrote: > > > I'm talking about 3rd party extensions. Those may require source > > compatibility with older Python versions. All I'm asking for is to not > > require source-level use of C99 features. > > This of course removes a lot of its usefulness. E.g. macros cannot be > replaced by inline functions, as header files must still be plain C89. > > > Sturla Molden > I share Guido's priority there - source compatibility is more important than smoothing a few of C's rough edges. Maybe the next breaking change release this should be considered (python 4000... python 5000?) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C99
> From: Python-Dev [mailto:python-dev- > [email protected]] On Behalf Of tritium- > [email protected] > Sent: Sunday, June 05, 2016 10:35 PM > To: 'Sturla Molden'; [email protected] > Subject: Re: [Python-Dev] C99 > > > -Original Message- > > From: Python-Dev [mailto:python-dev-bounces+tritium- > > [email protected]] On Behalf Of Sturla Molden > > Sent: Sunday, June 5, 2016 10:29 PM > > To: [email protected] > > Subject: Re: [Python-Dev] C99 > > > > Guido van Rossum wrote: > > > > > I'm talking about 3rd party extensions. Those may require source > > > compatibility with older Python versions. All I'm asking for is to not > > > require source-level use of C99 features. > > > > This of course removes a lot of its usefulness. E.g. macros cannot be > > replaced by inline functions, as header files must still be plain C89. > > > > > > Sturla Molden > > > > I share Guido's priority there - source compatibility is more important than > smoothing a few of C's rough edges. Correct me if I'm wrong, but I think that Guido meant that the third-party extensions might require their own code (not CPython's) to be compatible with versions of CPython < 3.6, and so PEP 7 shouldn't force them to break their own backwards compatibility. Either way I'm +1 for allowing (but not enforcing) C99 syntax. > Maybe the next breaking change release > this should be considered (python 4000... python 5000?) Let's not! -Emanuel ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C99
wrote: > I share Guido's priority there - source compatibility is more important than > smoothing a few of C's rough edges. Maybe the next breaking change release > this should be considered (python 4000... python 5000?) I was simply pointing out that Guido's priority removes a lot of the usefulness of C99 at source level. I was not saying I disagreed. If we have to keep header files clean of C99 I think this proposal just adds clutter. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C99
I'm not sure I meant that. But if I have a 3rd party extension that compiles with 3.5 headers using C89, then it should still compile with 3.6 headers using C99. Also if I compile it for 3.5 and it only uses the ABI it should still be linkable with 3.6. On Sun, Jun 5, 2016 at 7:42 PM, Sturla Molden wrote: > wrote: > >> I share Guido's priority there - source compatibility is more important than >> smoothing a few of C's rough edges. Maybe the next breaking change release >> this should be considered (python 4000... python 5000?) > > I was simply pointing out that Guido's priority removes a lot of the > usefulness of C99 at source level. I was not saying I disagreed. If we have > to keep header files clean of C99 I think this proposal just adds clutter. > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: replace custom validation logic in the parse module with a simple DFA validator
On Sat, Jun 4, 2016, at 17:26, Christian Heimes wrote:
> On 2016-06-02 11:32, benjamin.peterson wrote:
> > https://hg.python.org/cpython/rev/4a9159ea2536
> > changeset: 101601:4a9159ea2536
> > user:Benjamin Peterson
> > date:Thu Jun 02 11:30:18 2016 -0700
> > summary:
> > replace custom validation logic in the parse module with a simple DFA
> > validator (closes #26526)
> >
> > Patch from A. Skrobov.
> >
> > files:
> > Misc/NEWS | 3 +
> > Modules/parsermodule.c | 2545 +--
> > 2 files changed, 96 insertions(+), 2452 deletions(-)
> >
> >
> > diff --git a/Misc/NEWS b/Misc/NEWS
> > --- a/Misc/NEWS
> > +++ b/Misc/NEWS
> > @@ -22,6 +22,9 @@
> > Library
> > ---
> >
> > +- Issue #26526: Replace custom parse tree validation in the parser
> > + module with a simple DFA validator.
> > +
> > - Issue #27114: Fix SSLContext._load_windows_store_certs fails with
> >PermissionError
> >
> > diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c
> > --- a/Modules/parsermodule.c
> > +++ b/Modules/parsermodule.c
> > @@ -670,9 +670,75 @@
> >
> >
> > static node* build_node_tree(PyObject *tuple);
> > -static int validate_expr_tree(node *tree);
> > -static int validate_file_input(node *tree);
> > -static int validate_encoding_decl(node *tree);
> > +
> > +static int
> > +validate_node(node *tree)
> > +{
> > +int type = TYPE(tree);
> > +int nch = NCH(tree);
> > +dfa *nt_dfa;
> > +state *dfa_state;
> > +int pos, arc;
> > +
> > +assert(ISNONTERMINAL(type));
> > +type -= NT_OFFSET;
> > +if (type >= _PyParser_Grammar.g_ndfas) {
> > +PyErr_Format(parser_error, "Unrecognized node type %d.",
> > TYPE(tree));
> > +return 0;
> > +}
> > +nt_dfa = &_PyParser_Grammar.g_dfa[type];
> > +REQ(tree, nt_dfa->d_type);
> > +
> > +/* Run the DFA for this nonterminal. */
> > +dfa_state = &nt_dfa->d_state[nt_dfa->d_initial];
> > +for (pos = 0; pos < nch; ++pos) {
> > +node *ch = CHILD(tree, pos);
> > +int ch_type = TYPE(ch);
> > +for (arc = 0; arc < dfa_state->s_narcs; ++arc) {
> > +short a_label = dfa_state->s_arc[arc].a_lbl;
> > +assert(a_label < _PyParser_Grammar.g_ll.ll_nlabels);
> > +if (_PyParser_Grammar.g_ll.ll_label[a_label].lb_type ==
> > ch_type) {
> > + /* The child is acceptable; if non-terminal, validate
> > it recursively. */
> > +if (ISNONTERMINAL(ch_type) && !validate_node(ch))
> > +return 0;
> > +
> > +/* Update the state, and move on to the next child. */
> > +dfa_state =
> > &nt_dfa->d_state[dfa_state->s_arc[arc].a_arrow];
> > +goto arc_found;
> > +}
> > +}
> > +/* What would this state have accepted? */
> > +{
> > +short a_label = dfa_state->s_arc->a_lbl;
> > +int next_type;
> > +if (!a_label) /* Wouldn't accept any more children */
> > +goto illegal_num_children;
> > +
> > +next_type = _PyParser_Grammar.g_ll.ll_label[a_label].lb_type;
> > +if (ISNONTERMINAL(next_type))
> > +PyErr_Format(parser_error, "Expected node type %d, got
> > %d.",
> > + next_type, ch_type);
> > +else
> > +PyErr_Format(parser_error, "Illegal terminal: expected
> > %s.",
> > + _PyParser_TokenNames[next_type]);
>
> Coverity doesn't that line:
>
> CID 1362505 (#1 of 1): Out-of-bounds read (OVERRUN)
> 20. overrun-local: Overrunning array _PyParser_TokenNames of 58 8-byte
> elements at element index 255 (byte offset 2040) using index next_type
> (which evaluates to 255).
I don't think this can cause a problem because it doesn't ever come from
user-provided input.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
