Re: [Python-Dev] [Python-checkins] peps: New DSL syntax and slightly changed semantics for the Argument Clinic DSL.

[Larry Hastings]

On 03/17/2013 03:26 PM, Stefan Krah wrote:

While I like the syntax better and appreciate the option to condense the
function declaration I still fear that the amount of implicitness will
distract from what is important: programming in C.

This applies especially if people start declaring converters using the
[python] feature.

So I hope that at least converters can be declared statically in a header
file, like I suggested in PEP 437.


The Argument Clinic prototype is written in Python; I don't know how 
"declared static in a header file" applies to a Python implementation.  
Currently the converters are declared directly in clinic.py, somewhere 
in the middle.


For what it's worth, I think the new syntax (dictated more-or-less by 
Guido and Nick) makes things slightly less explicit.  In my original 
syntax, you declared the *type* of each parameter in C, and the flags 
dictated how Clinic should perform the conversion.  In the new syntax, 
you declare the *converter function* for each parameter, and the type of 
the resulting C variable is inferred.




A couple of comments:

As of CPython 3.3, builtin functions nearly always parse their arguments
with one of two functions: the original ``PyArg_ParseTuple()``, [1]_ and
the more modern ``PyArg_ParseTupleAndKeywords()``. [2]_ The former
only handles positional parameters; the latter also accommodates keyword
and keyword-only parameters, and is preferred for new code.

What is the source for this? I seem to remember a discussion on python-ideas
(but cannot find it now) where some developers preferred non-keyword functions
for some use cases.

For example it's strange to write div(x=10, y=3), or worse, div(y=3, x=10).
Using positional-only arguments prevents this "feature".


(I don't know to what "div" function you refer, but its arguments are 
poorly named.)


I thought this was obviously true.  But realized I didn't have any 
justification for it.  So I checked into it.  Thomas Wouters found me a 
thread from python-ideas from last March about changing range() to 
support keyword-only parameters.  Guido participated in the thread, and 
responded with this pronouncement:


   [...] let's forget about "fixing" range. And that's final.

   http://mail.python.org/pipermail/python-ideas/2012-March/014380.html

I double-checked with him about this and he confirmed: positional 
parameters are here to stay.


This has some consequences.  For example, inspect.getfullargspec, 
inspect.Signature, and indeed types.FunctionObject and types.CodeObject 
have no currently defined mechanism for communicating that a parameter 
is positional-only.  I strongly assert we need such a mechanism, though 
it could be as simple as having the parameter name be an empty string or 
None.


Anyway, it turns out this "keyword-only is preferred" was a 
misconception on my part.  Python supports, and will continue to 
support, positional-only parameters as part of the language. Currently 
it isn't possible to define functions in Python that have them.  But 
builtins have them, and will continue to have them, and therefore 
Argument Clinic needs to support them.


I'll amend my PEP soonish to reflect this.  Specifically the semantics 
of the /, [, and ] lines in the parameter section.




path: path_t(allow_fd=1)

I do not see where the C initialization or the cleanup are specified. Are
they part of the converter specification?


The extension interface isn't yet well-enough defined to be in the PEP.  
But yes, the generation of cleanup code will be part of the job of the 
conversion functions.  I'm not sure I have any provision for 
initialization apart from assignment, but I agree that it should have one.




curses.window.addch

The parameters appear to be in the wrong order.


Fixed.



The return annotation is also optional.  If skipped, the arrow ("``->``")
must also be omitted.

Why is it optional? Aren't type annotations important?


I'm not aware of any builtins that have annotations.  In fact, I'm not 
sure there's any place in the API where you can attach annotations to a 
builtin.  I expect this will change when we add reflection support for 
signatures, though I doubt many builtins will use the facility.


This raises an interesting point, though: Argument Clinic provides a 
place to provide a function return annotation, but has never supported 
per-parameter annotations.  I'll meditate on this and see if I can come 
up with a reasonable amendment to the current syntax.




How are the converters specified? Inside the preprocessor source? Are 
initialization
and cleanup part of the specification, e.g. is a converter represented by a 
class?


Respectively: in Python, yes, and yes.  The current prototype has an 
experimental extension interface; this is used to add support for 
"path_t" in Modules/posixmodule.c.  It supports initialization and cleanup.




I think there should be a table that lists which values are converted and what
the result of the conversion is.

Re: [Python-Dev] [Python-checkins] peps: New DSL syntax and slightly changed semantics for the Argument Clinic DSL.

[Larry Hastings]

On 03/18/2013 12:16 AM, Larry Hastings wrote:
I'll amend my PEP soonish to reflect this.  Specifically the semantics 
of the /, [, and ] lines in the parameter section.


I've just posted this revision.

I'd like to draw everyone's attention to the top entry in the Notes 
section, reproduced below:


* The DSL currently makes no provision for specifying per-parameter
  type annotations.  This is something explicitly supported in Python;
  it should be supported for builtins too, once we have reflection support.

  It seems to me that the syntax for parameter lines--dictated by
  Guido--suggests conversion functions are themselves type annotations.
  This makes intuitive sense.  But my thought experiments in how to
  convert the conversion function specification into a per-parameter
  type annotation ranged from obnoxious to toxic; I don't think that
  line of thinking will bear fruit.

  Instead, I think we need to add a new syntax allowing functions
  to explicitly specify a per-parameter type annotation.  The problem:
  what should that syntax be?  I've only had one idea so far, and I
  don't find it all that appealing: allow a optional second colon
  on the parameter line, and the type annotation would be specified...
  somewhere, either between the first and second colons, or between
  the second colon and the (optional) default.

  Also, I don't think this could specify any arbitrary Python value.
  I suspect it would suffer heavy restrictions on what types and
  literals it could use.  Perhaps the best solution would be to
  store the exact string in static data, and have Python evaluate
  it on demand?  If so, it would be safest to restrict it to Python
  literal syntax, permitting no function calls (even to builtins).


Syntax suggestions welcome,


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] peps: New DSL syntax and slightly changed semantics for the Argument Clinic DSL.

[Ronald Oussoren]

On 18 Mar, 2013, at 8:16, Larry Hastings  wrote:

> 
> This has some consequences.  For example, inspect.getfullargspec, 
> inspect.Signature, and indeed types.FunctionObject and types.CodeObject have 
> no currently defined mechanism for communicating that a parameter is 
> positional-only.  I strongly assert we need such a mechanism, though it could 
> be as simple as having the parameter name be an empty string or None.

inspect.Signature does have support for positional-only arguments, they have 
inspect.Parameter.POSITIONAL_ONLY as their kind.  The others probably don't 
have support for this kind of parameters because there is no Python syntax for 
creating them.

Ronald
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] peps: New DSL syntax and slightly changed semantics for the Argument Clinic DSL.

[Stefan Krah]

Larry Hastings  wrote:
> So I hope that at least converters can be declared statically in a header
> file, like I suggested in PEP 437.
> 
> 
> The Argument Clinic prototype is written in Python; I don't know how "declared
> static in a header file" applies to a Python implementation.  Currently the
> converters are declared directly in clinic.py, somewhere in the middle.

It applies in the same way to a Python implementation as declaring the
DSL comment blocks in a C file applies to a Python implementation. This
is exactly the same.

1) I think that third party tools should be able to extract *all* required
   information from the DSL only.

2) After writing a new custom converter, I'd rather edit a header file and
   not the preprocessor source.

3) Likewise, I'd rather edit a header file than inserting a magic [python]
   block into the C source that registers the required information with
   the preprocessor in a completely implementation defined way.


> 
> The example in posixmodule.c takes up a lot of space and from the 
> perspective
> of auditing the effects it's a little like following a longjmp.
> 
> 
> I got strong feedback that I needed more examples.  That was the logical place
> for them.  Can you suggest a better spot, or spots?

I'm concerned about the whole concept (see above).


Stefan Krah


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] peps: New DSL syntax and slightly changed semantics for the Argument Clinic DSL.

[Stefan Krah]

Larry Hastings  wrote:
> * The DSL currently makes no provision for specifying per-parameter
>   type annotations.  This is something explicitly supported in Python;
>   it should be supported for builtins too, once we have reflection support.
> 
>   It seems to me that the syntax for parameter lines--dictated by
>   Guido--suggests conversion functions are themselves type annotations.
>   This makes intuitive sense.

Really, did you read PEP 437? It's all in there.


>  But my thought experiments in how to convert the conversion function
>  specification into a per-parameter type annotation ranged from obnoxious
>  to toxic; I don't think that
>  line of thinking will bear fruit.

Did you look at the patch that I posted in issue #16612? It's already
implemented:

$ ./printsemant Tools/preprocess/testcases/posix_stat.c
PROGRAM[
  SOURCE[...],
  DEFINE[
CNAME posix_stat,
SPEC[
  DECLARATION
 { fun_fqname = os.stat,
   fun_name = stat,
   fun_cname = posix_stat,
   fun_kind = Keywords,
   fun_params = [
 { param_name = path,
   param_type = [bytes, int, str], <== here it is
   param_default = NONE,
   param_kind = (PosOrKwd, Required),
   param_conv = path_converter,
   param_parseargs = [
 ConvArg { arg_name = path_converter,
   arg_type = int (*converter)(PyObject *, void *)
   arg_use_ptr = false },
 MainArg { arg_name = path,
   arg_type = path_t,
   arg_use_ptr = true }]},
[...]



Stefan Krah


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] peps: Update for 436, explicitly supporting positional parameters forever, amen.

[Stefan Krah]

larry.hastings  wrote:
> +Establishes that all the *proceeding* arguments are
> +positional-only.  For now, Argument Clinic does not
> +support functions with both positional-only and
> +non-positional-only arguments; therefore, if ``/``
> +is specified for a function, currently it must always
> +be after the last parameter.  Also, Argument Clinic
> +does not currently support default values for
> +positional-only parameters.
> +
> +(The semantics of ``/`` follow a syntax for positional-only
> +parameters in Python once proposed by Guido. [5]_ )

I think the entire PEP would be easier to understand if the main sections
only contained the envisaged end result and all current preprocessor
deficiencies were listed in a single isolated section.


Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] can't assign to function call

[Neal Becker]

def F(x):
return x

x = 2
F(x) = 3

F(x) = 3
SyntaxError: can't assign to function call

Do we really need this restriction?  There do exist other languages without it.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] can't assign to function call

[Joao S. O. Bueno]

On 18 March 2013 10:50, Neal Becker  wrote:
> def F(x):
> return x
>
> x = 2
> F(x) = 3
>
> F(x) = 3
> SyntaxError: can't assign to function call
>
> Do we really need this restriction?  There do exist other languages without 
> it.

What?
I mean...what are you even talking about?
Assignments are to "names" - names are not Python objects and it is
not something that can be returned from a function call.

If you are meaning mathematical equation like functionality, I
recommend you to try "SymPy" - the Library for symbolic mathematics.

I can't make sense of what you want to perform by "assigning to a
function call", and given the time without a reply to this e-mail, I
think I am not the only one there.


  js
 -><-





>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] can't assign to function call

[Steven D'Aprano]

On 19/03/13 00:50, Neal Becker wrote:

def F(x):
 return x

x = 2
F(x) = 3

 F(x) = 3
SyntaxError: can't assign to function call

Do we really need this restriction?  There do exist other languages without it.



What meaning would you give to "F(x) = 3", and why?



--
Steven
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] can't assign to function call

[Chris Angelico]

On Tue, Mar 19, 2013 at 12:50 AM, Neal Becker  wrote:
> def F(x):
> return x
>
> x = 2
> F(x) = 3
>
> F(x) = 3
> SyntaxError: can't assign to function call
>
> Do we really need this restriction?  There do exist other languages without 
> it.

The languages that permit you to assign to a function call all have
some notion of a reference type. In C++, for instance, you can return
a reference to an object, and assigning to the function call achieves
the same thing as assigning to the referent. C has similar semantics
with pointers; you can dereference a returned pointer:

int somevalue;
int *F() {return &somevalue;}

*F() = 5; /* will assign to somevalue */

With Python, there are no pointers, there are no variables. But you
can do something somewhat similar:

>>> x = [0]
>>> def F():
return x

>>> F()[0]=3;
>>> x[0]
3

If you think of x as a pointer to the value x[0], then Python lets you
"dereference" the function return value. But this is fiddling with
terminology; the concept of assigning to a function return value
doesn't really make sense in Python.

Further discussion on exactly _why_ this is the case can be found on
python-list's or python-tutor's archives, such as this excellent post
by Steven D'Aprano:

http://mail.python.org/pipermail/tutor/2010-December/080505.html

TLDR: Python != C. :)

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] can't assign to function call

[Xavier Morel]

On 2013-03-18, at 15:23 , Chris Angelico wrote:

> On Tue, Mar 19, 2013 at 12:50 AM, Neal Becker  wrote:
>> def F(x):
>>return x
>> 
>> x = 2
>> F(x) = 3
>> 
>>F(x) = 3
>> SyntaxError: can't assign to function call
>> 
>> Do we really need this restriction?  There do exist other languages without 
>> it.
> 
> The languages that permit you to assign to a function call all have
> some notion of a reference type.

Alternatively they're functional language defining "match cases" e.g. in
Haskell a function is defined as

foo a b c = someOperation a b c

which is functionally equivalent to Python's

def foo(a, b, c):
return someOperation(a, b, c)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of XML fixes

[Christian Heimes]

Am 17.03.2013 19:25, schrieb Eli Bendersky:
> I'll gladly review the _elementtree changes and can help with the expat
> & pyexpat changes as well. Until now I had the impression that the
> patches aren't ready for review yet. If they are, that's great.

The modifications to expat, pyexpat and _elementtree are available for
weeks. I just hadn't have time to create proper patches yet.

> Do you have a patch in the issue tracker (so it can be reviewed with
> Rietveld)? ISTM the current form is just a file (say _elementtree.c) in
> your Bitbucket repo. Should that be just diffed with the trunk file to
> see the changes?

I have pushed all changes from defusedexpat to a clone of Python's hg
repository. You can find the clone at https://bitbucket.org/tiran/xmlbomb/ .

The repository also contains a quick draft for a XML security API.
https://bitbucket.org/tiran/xmlbomb/commits/c033abd0f7747c5b215e1b32f90372dd96e397ba

I have to port tests from my other branch and add tests for the new API,
too.

Christian

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of XML fixes

[Christian Heimes]

Am 17.03.2013 19:59, schrieb Antoine Pitrou:
>> Why keep the libraries vulnerable for another year (3.4 final is expected
>> for early 2014), if there is something we can do about them now?
> 
> Well, Christian said that his stdlib patch wasn't ready yet.

The patch is > 90% finished. All the hard work is already done.

Christian

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] can't assign to function call

[Hrvoje Niksic]

On 03/18/2013 03:23 PM, Chris Angelico wrote:

The languages that permit you to assign to a function call all have
some notion of a reference type.


Assigning to function calls is orthogonal to reference types.  For 
example, Python manages assignment to subscripts without having 
references just fine:


val = obj[index]  # val = obj.__getitem__(index)
obj[index] = val  # obj.__setitem__(index, val)

In analogy with that, Python could implement what looks like assignment 
to function call like this:


val = f(arg)  # val = f.__call__(arg)
f(arg) = val  # f.__setcall__(arg, val)

I am not arguing that this should be added, I'm only pointing out that 
Python's object customization is not fundamentally at odds with 
assignment to function calls.  Having said that, I am in fact arguing 
that Python doesn't need them.  All C++ uses of operator() overloads can 
be implemented with the subscript operator.


Even if one needs more different assignments than there are operators, 
Python can provide it as easily as C++.  For example, on 
std::vector::operator[] provides access to the container without error 
checking, and std::vector::at() checks bounds:


vec[i] = val  // no error checking
vec.at(i) = val   // error checking

This is trivially translated to Python as:

vec[i] = val  # primary functionality, use __setitem__
vec.at[i] = val   # secondary functionality, __setitem__ on a proxy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] can't assign to function call

[Steven D'Aprano]

On 19/03/13 02:01, Hrvoje Niksic wrote:

On 03/18/2013 03:23 PM, Chris Angelico wrote:

The languages that permit you to assign to a function call all have
some notion of a reference type.


Assigning to function calls is orthogonal to reference types.  For example, 
Python manages assignment to subscripts without having references just fine:

val = obj[index]  # val = obj.__getitem__(index)
obj[index] = val  # obj.__setitem__(index, val)

In analogy with that, Python could implement what looks like assignment to 
function call like this:

val = f(arg)  # val = f.__call__(arg)
f(arg) = val  # f.__setcall__(arg, val)


That's all very well, but what would it do? It's not enough to say that the 
syntax could exist, we also need to have semantics. What's the use-case here? 
(That question is mostly aimed at the original poster.)

Aside: I'd reverse the order of the arg, val in any such hypothetical 
__setcall__, so as to support functions with zero or more arguments:

f(*args, **kwargs) = val  <=>  f.__setcall__(val, *args, **kwargs)



--
Steven
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] peps: New DSL syntax and slightly changed semantics for the Argument Clinic DSL.

[Nick Coghlan]

On Mon, Mar 18, 2013 at 3:13 AM, Stefan Krah  wrote:
> Larry Hastings  wrote:
>> So I hope that at least converters can be declared statically in a header
>> file, like I suggested in PEP 437.
>>
>>
>> The Argument Clinic prototype is written in Python; I don't know how 
>> "declared
>> static in a header file" applies to a Python implementation.  Currently the
>> converters are declared directly in clinic.py, somewhere in the middle.
>
> It applies in the same way to a Python implementation as declaring the
> DSL comment blocks in a C file applies to a Python implementation. This
> is exactly the same.
>
> 1) I think that third party tools should be able to extract *all* required
>information from the DSL only.
>
> 2) After writing a new custom converter, I'd rather edit a header file and
>not the preprocessor source.
>
> 3) Likewise, I'd rather edit a header file than inserting a magic [python]
>block into the C source that registers the required information with
>the preprocessor in a completely implementation defined way.

We didn't spend much time on this (we were focused on the per-function
DSL), but I agree a DSL for converters would be highly desirable, and
quite like the one in PEP 437.

It would require some tweaks to correctly handle the converter
parameterisation (for example, in the "es#" case, "encoding" is an
input to the converter rather than an output. This is why PEP 436 now
allows a callable notation for type converter references)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] can't assign to function call

[Skip Montanaro]

On Mon, Mar 18, 2013 at 8:50 AM, Neal Becker  wrote:
> def F(x):
> return x
>
> x = 2
> F(x) = 3
>
> F(x) = 3
> SyntaxError: can't assign to function call
>
> Do we really need this restriction?  There do exist other languages without 
> it.

I think this belongs on python-ideas before launching into discussions
of syntax and semantics on python-dev.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] peps: Update for 436, explicitly supporting positional parameters forever, amen.

[Nick Coghlan]

On Mon, Mar 18, 2013 at 1:47 AM, larry.hastings
 wrote:
>  Notes / TBD
>  ===
>
> +* The DSL currently makes no provision for specifying per-parameter
> +  type annotations.  This is something explicitly supported in Python;
> +  it should be supported for builtins too, once we have reflection support.
> +
> +  It seems to me that the syntax for parameter lines--dictated by
> +  Guido--suggests conversion functions are themselves type annotations.
> +  This makes intuitive sense.  But my thought experiments in how to
> +  convert the conversion function specification into a per-parameter
> +  type annotation ranged from obnoxious to toxic; I don't think that
> +  line of thinking will bear fruit.
> +
> +  Instead, I think wee need to add a new syntax allowing functions
> +  to explicitly specify a per-parameter type annotation.  The problem:
> +  what should that syntax be?  I've only had one idea so far, and I
> +  don't find it all that appealing: allow a optional second colon
> +  on the parameter line, and the type annotation would be specified...
> +  somewhere, either between the first and second colons, or between
> +  the second colon and the (optional) default.
> +
> +  Also, I don't think this could specify any arbitrary Python value.
> +  I suspect it would suffer heavy restrictions on what types and
> +  literals it could use.  Perhaps the best solution would be to
> +  store the exact string in static data, and have Python evaluate
> +  it on demand?  If so, it would be safest to restrict it to Python
> +  literal syntax, permitting no function calls (even to builtins).
> +

I think the hack we're using for the default-as-shown-in-Python will
work here as well: use the converter parameterisation notation.

Then "pydefault" (I think that is a better name than the current
"default") and "pynote" would control what is shown for the conversion
in the first line of the docstring and in any future introspection
support. If either is not given, then the C default would be passed
through as the Python default and the annotation would be left blank.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 437] A DSL for specifying signatures, annotations and argument converters

[Nick Coghlan]

On Sun, Mar 17, 2013 at 12:14 AM, Larry Hastings  wrote:
> On 03/16/2013 02:17 AM, Stefan Krah wrote:
>
> Both PEPs were discussed at PyCon.  The state of affairs is that a
> compromise is being worked upon and will be published by Larry in
> a revised PEP.
>
>
> I've pushed an update to PEP 436, the Argument Clinic PEP.  It's now live on
> python.org.

Thanks for that. A few comments.

* I'm confused by the leading table - is it possible/expected to have
a module declaration, class declaration *and* function declaration in
the same block? If not, it seems more appropriate to have 3 tables, or
else simplify the initial table to omit the declaration details, and
explain the available options later.

* Is it possible to use "module_name.class_name.method_name" when
declaring a function that will be used as a method in a class?

* To match the behaviour of Python functions, function docstrings
should also be optional in Argument Clinic. We'll always include at
least the function prototype, even if no other content is specified.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] can't assign to function call

[Hrvoje Niksic]

On 03/18/2013 04:40 PM, Steven D'Aprano wrote:

In analogy with that, Python could implement what looks like assignment to 
function call like this:

val = f(arg)  # val = f.__call__(arg)
f(arg) = val  # f.__setcall__(arg, val)


That's all very well, but what would it do? It's not enough to say

> that the syntax could exist, we also need to have semantics.

I am not the best person to answer because I go on to argue that this 
syntax is not needed in Python at all (anything it can do can be 
implemented with __setitem__ at no loss of clarity).  Still, if such a 
feature existed in Python, I imagine people would use it to set the same 
resource that the function obtains, where such a thing is applicable.



Aside: I'd reverse the order of the arg, val in any such hypothetical

> __setcall__, so as to support functions with zero or more arguments:


f(*args, **kwargs) = val  <=>  f.__setcall__(val, *args, **kwargs)


That would be a better design, I agree.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] can't assign to function call

[Serhiy Storchaka]

On 18.03.13 17:40, Steven D'Aprano wrote:

On 19/03/13 02:01, Hrvoje Niksic wrote:

Assigning to function calls is orthogonal to reference types.  For
example, Python manages assignment to subscripts without having
references just fine:

val = obj[index]  # val = obj.__getitem__(index)
obj[index] = val  # obj.__setitem__(index, val)

In analogy with that, Python could implement what looks like
assignment to function call like this:

val = f(arg)  # val = f.__call__(arg)
f(arg) = val  # f.__setcall__(arg, val)


That's all very well, but what would it do? It's not enough to say that
the syntax could exist, we also need to have semantics. What's the
use-case here? (That question is mostly aimed at the original poster.)


Python could use parenthesis instead of brackets for indexing and a 
dictionary lookup. However it is too late to discuss this idea.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] peps: New DSL syntax and slightly changed semantics for the Argument Clinic DSL.

[Guido van Rossum]

On Mon, Mar 18, 2013 at 3:36 AM, Stefan Krah  wrote:
> Larry Hastings  wrote:
>> * The DSL currently makes no provision for specifying per-parameter
>>   type annotations.  This is something explicitly supported in Python;
>>   it should be supported for builtins too, once we have reflection support.
>>
>>   It seems to me that the syntax for parameter lines--dictated by
>>   Guido--suggests conversion functions are themselves type annotations.
>>   This makes intuitive sense.
>
> Really, did you read PEP 437? It's all in there.

This attitude is unhelpful. Please stop being outright hostile. If you
want to have any influence on the outcome at all, consider looking
into compromises.

>>  But my thought experiments in how to convert the conversion function
>>  specification into a per-parameter type annotation ranged from obnoxious
>>  to toxic; I don't think that
>>  line of thinking will bear fruit.
>
> Did you look at the patch that I posted in issue #16612? It's already
> implemented:
>
> $ ./printsemant Tools/preprocess/testcases/posix_stat.c
> PROGRAM[
>   SOURCE[...],
>   DEFINE[
> CNAME posix_stat,
> SPEC[
>   DECLARATION
>  { fun_fqname = os.stat,
>fun_name = stat,
>fun_cname = posix_stat,
>fun_kind = Keywords,
>fun_params = [
>  { param_name = path,
>param_type = [bytes, int, str], <== here it is
>param_default = NONE,
>param_kind = (PosOrKwd, Required),
>param_conv = path_converter,
>param_parseargs = [
>  ConvArg { arg_name = path_converter,
>arg_type = int (*converter)(PyObject *, void *)
>arg_use_ptr = false },
>  MainArg { arg_name = path,
>arg_type = path_t,
>arg_use_ptr = true }]},
> [...]

I can assure you nobody downloaded your binaries. The security
implications are just too scary.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] can't assign to function call

[Guido van Rossum]

Move. This. Thread. Out. Of. Python-Dev. Now. (python-ideas is the right place.)

On Mon, Mar 18, 2013 at 10:25 AM, Serhiy Storchaka  wrote:
> On 18.03.13 17:40, Steven D'Aprano wrote:
>>
>> On 19/03/13 02:01, Hrvoje Niksic wrote:
>>>
>>> Assigning to function calls is orthogonal to reference types.  For
>>> example, Python manages assignment to subscripts without having
>>> references just fine:
>>>
>>> val = obj[index]  # val = obj.__getitem__(index)
>>> obj[index] = val  # obj.__setitem__(index, val)
>>>
>>> In analogy with that, Python could implement what looks like
>>> assignment to function call like this:
>>>
>>> val = f(arg)  # val = f.__call__(arg)
>>> f(arg) = val  # f.__setcall__(arg, val)
>>
>>
>> That's all very well, but what would it do? It's not enough to say that
>> the syntax could exist, we also need to have semantics. What's the
>> use-case here? (That question is mostly aimed at the original poster.)
>
>
> Python could use parenthesis instead of brackets for indexing and a
> dictionary lookup. However it is too late to discuss this idea.
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org



--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] peps: Update for 436, explicitly supporting positional parameters forever, amen.

[Guido van Rossum]

On Mon, Mar 18, 2013 at 8:57 AM, Nick Coghlan  wrote:
> On Mon, Mar 18, 2013 at 1:47 AM, larry.hastings
>  wrote:
>>  Notes / TBD
>>  ===
>>
>> +* The DSL currently makes no provision for specifying per-parameter
>> +  type annotations.  This is something explicitly supported in Python;
>> +  it should be supported for builtins too, once we have reflection support.
>> +
>> +  It seems to me that the syntax for parameter lines--dictated by
>> +  Guido--suggests conversion functions are themselves type annotations.
>> +  This makes intuitive sense.  But my thought experiments in how to
>> +  convert the conversion function specification into a per-parameter
>> +  type annotation ranged from obnoxious to toxic; I don't think that
>> +  line of thinking will bear fruit.
>> +
>> +  Instead, I think wee need to add a new syntax allowing functions
>> +  to explicitly specify a per-parameter type annotation.  The problem:
>> +  what should that syntax be?  I've only had one idea so far, and I
>> +  don't find it all that appealing: allow a optional second colon
>> +  on the parameter line, and the type annotation would be specified...
>> +  somewhere, either between the first and second colons, or between
>> +  the second colon and the (optional) default.
>> +
>> +  Also, I don't think this could specify any arbitrary Python value.
>> +  I suspect it would suffer heavy restrictions on what types and
>> +  literals it could use.  Perhaps the best solution would be to
>> +  store the exact string in static data, and have Python evaluate
>> +  it on demand?  If so, it would be safest to restrict it to Python
>> +  literal syntax, permitting no function calls (even to builtins).
>> +
>
> I think the hack we're using for the default-as-shown-in-Python will
> work here as well: use the converter parameterisation notation.
>
> Then "pydefault" (I think that is a better name than the current
> "default") and "pynote" would control what is shown for the conversion
> in the first line of the docstring and in any future introspection
> support. If either is not given, then the C default would be passed
> through as the Python default and the annotation would be left blank.

Right. In fact, I think the decision of what (if anything) should be
put in the annotation should be up to the converter class. It can be a
specific method on the converter object.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Recent changes to TextIOWrapper and its tests

[Jeff Allen]

I'm pulling recent changes in the io module across to Jython. I am 
looking for help understanding the changes in

http://hg.python.org/cpython/rev/19a33ef3821d

That change set is about what should happen if the underlying buffer 
does not return bytes when read, but instead, for example, unicode 
characters. The test test_read_nonbytes() constructs a pathological text 
stream reader t where the usual BytesIO or BufferedReader is replaced 
with a StringIO. It then checks that r.read(1) and t.readlines() raise a 
TypeError, that is, it tests that TextIOWrapper checks the type of what 
it reads from the buffer.


The puzzle is that it requires t.read() to succeed.

When I insert a check for bytes type in all the places it seems 
necessary in my code, I pass the first two conditions, but since 
t.read() also raises TypeError, the overall test fails. Is reading the 
stream with read() intended to succeed? Why is this desired?


Jeff Allen



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] can't assign to function call

[Greg Ewing]

Hrvoje Niksic wrote:
I am not the best person to answer because I go on to argue that this 
syntax is not needed in Python at all (anything it can do can be 
implemented with __setitem__ at no loss of clarity).


I would even argue that the proxy solution is even *better*
for that particular use case, because it makes both operations
look like forms of indexing, which they are.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] peps: New DSL syntax and slightly changed semantics for the Argument Clinic DSL.

[Larry Hastings]

On 03/18/2013 02:29 AM, Ronald Oussoren wrote:

On 18 Mar, 2013, at 8:16, Larry Hastings  wrote:

This has some consequences.  For example, inspect.getfullargspec, 
inspect.Signature, and indeed types.FunctionObject and types.CodeObject have no 
currently defined mechanism for communicating that a parameter is 
positional-only.

inspect.Signature does have support for positional-only arguments, they have 
inspect.Parameter.POSITIONAL_ONLY as their kind.


You're right!  And I should have remembered that--I was one of the 
authors of the inspect.Signature PEP.  It's funny, it can represent 
something that it has no way of inferring ;-)



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] peps: New DSL syntax and slightly changed semantics for the Argument Clinic DSL.

[Ronald Oussoren]

On 18 Mar, 2013, at 23:43, Larry Hastings  wrote:

> On 03/18/2013 02:29 AM, Ronald Oussoren wrote:
>> On 18 Mar, 2013, at 8:16, Larry Hastings  wrote:
>>> This has some consequences.  For example, inspect.getfullargspec, 
>>> inspect.Signature, and indeed types.FunctionObject and types.CodeObject 
>>> have no currently defined mechanism for communicating that a parameter is 
>>> positional-only.
>> inspect.Signature does have support for positional-only arguments, they have 
>> inspect.Parameter.POSITIONAL_ONLY as their kind.
> 
> You're right!  And I should have remembered that--I was one of the authors of 
> the inspect.Signature PEP.  It's funny, it can represent something that it 
> has no way of inferring ;-)

It doesn't necessarily have to, builtin functions could grow a __signature__ 
attribute that calculates the signature (possibly from the DSL data). I've done 
something like that in a pre-release version of PyObjC, and with some patching 
of pydoc and inspect (see #17053) I now have useful help information for what 
are basicly builtin functions with positional-only arguments.

Ronald

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] peps: New DSL syntax and slightly changed semantics for the Argument Clinic DSL.

[Nick Coghlan]

On Mon, Mar 18, 2013 at 11:02 AM, Guido van Rossum  wrote:
> On Mon, Mar 18, 2013 at 3:36 AM, Stefan Krah  wrote:
>> Larry Hastings  wrote:
>>> * The DSL currently makes no provision for specifying per-parameter
>>>   type annotations.  This is something explicitly supported in Python;
>>>   it should be supported for builtins too, once we have reflection support.
>>>
>>>   It seems to me that the syntax for parameter lines--dictated by
>>>   Guido--suggests conversion functions are themselves type annotations.
>>>   This makes intuitive sense.
>>
>> Really, did you read PEP 437? It's all in there.
>
> This attitude is unhelpful. Please stop being outright hostile. If you
> want to have any influence on the outcome at all, consider looking
> into compromises.

While I actually agree with Stefan that it's important to eventually
have a converter DSL, I don't think it's necessary to have it in the
initial implementation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Recent changes to TextIOWrapper and its tests

[Serhiy Storchaka]

On 18.03.13 22:26, Jeff Allen wrote:

The puzzle is that it requires t.read() to succeed.

When I insert a check for bytes type in all the places it seems
necessary in my code, I pass the first two conditions, but since
t.read() also raises TypeError, the overall test fails. Is reading the
stream with read() intended to succeed? Why is this desired?


This is not desired. I just registered the current behavior. Python 3 is 
more strict and always raises an exception.


Perhaps this test should be relaxed. I.e. use

with self.maybeRaises(TypeError):
t.read()

and define maybeRaises() as:

@contextlib.contextmanager
def maybeRaises(self, *args, **kwds):
try:
yield
except args:
pass


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] newbuffer support in python 2.7

[Kristján Valur Jónsson]

Hi python dev.
I have two languishing defects regarding 2.7 and how buffer support isn't 
complete there.
http://bugs.python.org/issue10211
http://bugs.python.org/issue10212

In both cases, the new style buffer support is incomplete, and the patches 
close usability holes having to do with memoryview objects.
It was suggested to me that I put it to python-dev to decide if we should 
consider this a bug to be fixed or not, and hopefully get a consensus before 
2.7.4 freeze.
I have been running a local patch of 2.7 with those fixes for two years now.

Cheers,
Kristján


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Slides from today's parallel/async Python talk

[Christian Tismer]

Hi Trent,

I just started to try to understand the idea and the implications.
Removing almost all of your message since that is already too long
to work with:
The reference is

http://mail.python.org/pipermail/python-dev/2013-March/124690.html

On 3/14/13 11:45 AM, Trent Nelson wrote:

On Wed, Mar 13, 2013 at 07:05:41PM -0700, Trent Nelson wrote:

 Just posted the slides for those that didn't have the benefit of
 attending the language summit today:

 
https://speakerdeck.com/trent/parallelizing-the-python-interpreter-an-alternate-approach-to-async

 Someone on /r/python asked if I could elaborate on the "do Y" part
 of "if we're in a parallel thread, do Y, if not, do X", which I
 (inadvertently) ended up replying to in detail.  I've included the
 response below.  (I'll work on converting this into a TL;DR set of
 slides soon.)


Can you go into a bit of depth about "X" here?

That's a huge topic that I'm hoping to tackle ASAP.  The basic premise
is that parallel 'Context' objects (well, structs) are allocated for
each parallel thread callback.  The context persists for the lifetime of
the "parallel work".





So, the remaining challenge is preventing the use case alluded to
earlier where someone tries to modify an object that hasn't been "async
protected".  That's a bit harder.  The idea I've got in mind is to
instrument the main CPython ceval loop, such that we do these checks as
part of opcode processing.  That allows us to keep all the logic in the
one spot and not have to go hacking the internals of every single
object's C backend to ensure correctness.

Now, that'll probably work to an extent.  I mean, after all, there are
opcodes for all the things we'd be interested in instrumenting,
LOAD_GLOBAL, STORE_GLOBAL, SETITEM etc.  What becomes challenging is
detecting arbitrary mutations via object calls, i.e. how do we know,
during the ceval loop, that foo.append(x) needs to be treated specially
if foo is a main-thread object and x is a parallel thread object?

There may be no way to handle that *other* than hacking the internals of
each object, unfortunately.  So, the viability of this whole approach
may rest on whether or that's deemed as an acceptable tradeoff (a
necessary evil, even) to the Python developer community.


This is pretty much my concern:
In order to make this waterproof, as required for CPython, you will quite
likely have to do something on very many objects, and this is hard
to chime into CPython.



If it's not, then it's unlikely this approach will ever see the light of
day in CPython.  If that turns out to be the case, then I see this
project taking the path that Stackless took (forking off and becoming a
separate interpreter).


We had that discussion quite often for Stackless, and I would love to find
a solution that allows to add special versions and use cases to CPython
in a way that avoids the forking as we did it.

It would be a nice thing if we could come up with a way to keep CPython
in place, but to swap the interpreter out and replace it with a specialized
version, if the application needs it. I wonder to what extent that would be
possible.
What I would like to achieve, after having given up on Stackless integration
is a way to let it piggyback onto CPython that works like an extension
module, although it hat effectively replace larger parts of the interpreter.
I wonder if that might be the superior way to have more flexibility, 
without forcing

everything and all go into CPython.
If we can make the interpreter somehow pluggable at runtime, a lot of issues
would become much simpler.



There's nothing wrong with that; I am really excited about the
possibilities afforded by this approach, and I'm sure it will pique the
interest of commercial entities out there that have problems perfectly
suited to where this pattern excels (shared-nothing, highly concurrent),
much like the relationship that developed between Stackless and Eve
Online.



What do you think: does it make sense to think of a framework that
allows to replace the interpreter at runtime, without making normal
CPython really slower?

cheers - chris

--
Christian Tismer :^)   
Software Consulting  : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Slides from today's parallel/async Python talk

[Trent Nelson]

On Mon, Mar 18, 2013 at 05:27:33PM -0700, Christian Tismer wrote:
> Hi Trent,

Hi Christian!  Thanks for taking the time to read my walls of text
;-)

> > So, the remaining challenge is preventing the use case alluded to
> > earlier where someone tries to modify an object that hasn't been "async
> > protected".  That's a bit harder.  The idea I've got in mind is to
> > instrument the main CPython ceval loop, such that we do these checks as
> > part of opcode processing.  That allows us to keep all the logic in the
> > one spot and not have to go hacking the internals of every single
> > object's C backend to ensure correctness.
> >
> > Now, that'll probably work to an extent.  I mean, after all, there are
> > opcodes for all the things we'd be interested in instrumenting,
> > LOAD_GLOBAL, STORE_GLOBAL, SETITEM etc.  What becomes challenging is
> > detecting arbitrary mutations via object calls, i.e. how do we know,
> > during the ceval loop, that foo.append(x) needs to be treated specially
> > if foo is a main-thread object and x is a parallel thread object?
> >
> > There may be no way to handle that *other* than hacking the internals of
> > each object, unfortunately.  So, the viability of this whole approach
> > may rest on whether or that's deemed as an acceptable tradeoff (a
> > necessary evil, even) to the Python developer community.
> 
> This is pretty much my concern:
> In order to make this waterproof, as required for CPython, you will quite
> likely have to do something on very many objects, and this is hard
> to chime into CPython.

Actually, I think I was unnecessarily pessimistic here.  When I sent
that follow-up mail with cross-references, I realized I'd forgotten
the nitty gritty details of how I implemented the async protection
support.

It turns out I'd already started on protecting lists (or rather,
PySequenceMethods), but decided to stop as the work I'd done on the
PyMappingMethods was sufficient for my needs at the time.

All I *really* want to do is raise an exception if a parallel object
gets assigned to a main-thread container object (list/dict etc) that
hasn't been "async protected".  (As opposed to now, where it'll
either segfault or silently corrupt stuff, then segfault later.)

I've already got all the infrastructure in place to test that (I use
it extensively within pyparallel.c):

Py_ISPY(obj) - detect a main-thread object
Py_ISPX(obj) - detect a parallel-thread object
Py_IS_PROTECTED(obj) - detect if a main-thread object has
   been protected*

[*]: actually, this isn't in a macro form right now, it's a
cheeky inline:

__inline
char
_protected(PyObject *obj)
{
return (obj->px_flags & Py_PXFLAGS_RWLOCK);
}

As those macros are exposed in the public ,
they can be used in other parts of the code base.  So, it's just a
matter of finding the points where an `lvalue = rvalue` takes place;
where: ``Py_ISPY(lvalue) && Py_ISPX(rvalue)``.  Then a test to see
if lvalue is protected; if not, raise an exception.  If so, then
nothing else needs to be done.

And there aren't that many places where this happens.  (It didn't
take long to get the PyMappingMethods intercepts nailed down.)

That's the idea anyway.  I need to get back to coding to see how it
all plays out in practice.  "And there aren't many places where this
happens" might be my famous last words.

> > If it's not, then it's unlikely this approach will ever see the light of
> > day in CPython.  If that turns out to be the case, then I see this
> > project taking the path that Stackless took (forking off and becoming a
> > separate interpreter).
> 
> We had that discussion quite often for Stackless, and I would love to find
> a solution that allows to add special versions and use cases to CPython
> in a way that avoids the forking as we did it.
> 
> It would be a nice thing if we could come up with a way to keep CPython
> in place, but to swap the interpreter out and replace it with a specialized
> version, if the application needs it. I wonder to what extent that would be
> possible.
> What I would like to achieve, after having given up on Stackless integration
> is a way to let it piggyback onto CPython that works like an extension
> module, although it hat effectively replace larger parts of the interpreter.
> I wonder if that might be the superior way to have more flexibility, 
> without forcing
> everything and all go into CPython.
> If we can make the interpreter somehow pluggable at runtime, a lot of issues
> would become much simpler.
> 
> >
> > There's nothing wrong with that; I am really excited about the
> > possibilities afforded by this approach, and I'm sure it will pique the
> > interest of commercial entities out there that h

[Python-Dev] Rough idea for adding introspection information for builtins

[Larry Hastings]

The original impetus for Argument Clinic was adding introspection 
information for builtins--it seemed like any manual approach I came up 
with would push the builtins maintenance burden beyond the pale.


Assuming that we have Argument Clinic or something like it, we don't 
need to optimize for ease of use from the API end--we can optimize for 
data size.  So the approach writ large: store a blob of data associated 
with each entry point, as small as possible. Reconstitute the 
appropriate inspect.Signature on demand by reading that blob.


Where to store the data?  PyMethodDef is the obvious spot, but I think 
that structure is part of the stable ABI.  So we'd need a new 
PyMethodDefEx and that'd be a little tiresome.  Less violent to the ABI 
would be defining a new array of pointers-to-introspection-blobs, 
parallel to the PyMethodDef array, passed in via a new entry point.



On to the representation.  Consider the function

   def foo(arg, b=3, *, kwonly='a'):
pass

I considered four approaches, each listed below along with its total 
size if it was stored as C static data.


1. A specialized bytecode format, something like pickle, like this:

   bytes([ PARAMETER_START_LENGTH_3, 'a', 'r', 'g',
  PARAMETER_START_LENGTH_1, 'b', PARAMETER_DEFAULT_LENGTH_1, '3',
  KEYWORD_ONLY,
  PARAMETER_START_LENGTH_6, 'k', 'w', 'o', 'n', 'l', 'y',
   PARAMETER_DEFAULT_LENGTH_3, '\'', 'a', '\'',
  END
  ])

Length: 20 bytes.

2. Just use pickle--pickle the result of inspect.signature() run on a 
mocked-up signature, just store that.   Length: 130 bytes. (Assume a 
two-byte size stored next to it.)


3. Store a string that, if eval'd, would produce the inspect.Signature.  
Length: 231 bytes.  (This could be made smaller if we could assume "from 
inspect import *" or "p = inspect.Parameter" or something, but it'd 
still be easily the heaviest.)


4. Store a string that looks like the Python declaration of the 
signature, and parse it (Nick's suggestion).  For foo above, this would 
be "(arg,b=3,*,kwonly='a')".  Length: 23 bytes.


Of those, Nick's suggestion seems best.  It's slightly bigger than the 
specialized bytecode format, but it's human-readable (and 
human-writable!), and it'd be the easiest to implement.



My first idea for implementation: add a "def x" to the front and ": 
pass" to the end, then run it through ast.parse.  Iterate over the tree, 
converting parameters into inspect.Parameters and handling the return 
annotation if present.  Default values and annotations would be turned 
into values by ast.eval_literal.  (It wouldn't surprise me if there's a 
cleaner way to do it than the fake function definition; I'm not familiar 
with the ast module.)


We'd want one more mild hack: the DSL will support positional 
parameters, and inspect.Signature supports positional parameters, so 
it'd be nice to render that information.  But we can't represent that in 
Python syntax (or at least not yet!), so we can't let ast.parse see it.  
My suggestion: run it through ast.parse, and if it throws a SyntaxError 
see if the problem was a slash.  If it was, remove the slash, reprocess 
through ast.parse, and remember that all parameters are positional-only 
(and barf if there are kwonly, args, or kwargs).



Thoughts?


//arry/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rough idea for adding introspection information for builtins

[Stefan Behnel]

Larry Hastings, 19.03.2013 05:45:
> The original impetus for Argument Clinic was adding introspection
> information for builtins [...]
> On to the representation.  Consider the function
> 
>def foo(arg, b=3, *, kwonly='a'):
> pass
> [...]
> 4. Store a string that looks like the Python declaration of the signature,
> and parse it (Nick's suggestion).  For foo above, this would be
> "(arg,b=3,*,kwonly='a')".

I had already noted that this would be generally useful, specifically for
Cython, so I'm all for going this route. No need to invent something new here.


>  Length: 23 bytes.

I can't see why the size would matter in any way.


> Of those, Nick's suggestion seems best.  It's slightly bigger than the
> specialized bytecode format, but it's human-readable (and human-writable!),
> and it'd be the easiest to implement.

Plus, if it becomes the format how C level signatures are expressed anyway,
it wouldn't require any additional build time preprocessing.


> My first idea for implementation: add a "def x" to the front and ": pass"
> to the end

Why not require it to be there already? Maybe more like

def foo(arg, b=3, *, kwonly='a'):
 ...

(i.e. using Ellipsis instead of pass, so that it's clear that it's not an
empty function but one the implementation of which is hidden)


> then run it through ast.parse.  Iterate over the tree,
> converting parameters into inspect.Parameters and handling the return
> annotation if present.  Default values and annotations would be turned into
> values by ast.eval_literal.  (It wouldn't surprise me if there's a cleaner
> way to do it than the fake function definition; I'm not familiar with the
> ast module.)

IMHO, if there is no straight forward way currently to convert a function
header from a code blob into a Signature object in Python code, preferably
using the ast module (either explicitly or implicitly through inspect.py),
then that's a bug.


> We'd want one more mild hack: the DSL will support positional parameters,
> and inspect.Signature supports positional parameters, so it'd be nice to
> render that information.  But we can't represent that in Python syntax (or
> at least not yet!), so we can't let ast.parse see it.  My suggestion: run
> it through ast.parse, and if it throws a SyntaxError see if the problem was
> a slash.  If it was, remove the slash, reprocess through ast.parse, and
> remember that all parameters are positional-only (and barf if there are
> kwonly, args, or kwargs).

Is sounds simpler to me to just make it a Python syntax feature. Or at
least an optional one, supported by the ast module with a dedicated
compiler flag.

Stefan


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com