Re: [Python-Dev] The docstring hack for signature information has to go
On 02/03/2014 08:19 PM, Guido van Rossum wrote: But why do you even need a flag? Reading issue 20075 where the complaint started, it really feels that the change was an overreaction to a very minimal problem. I'll cop to that. I'm pretty anxious about trying to "get it right". My worry was (and is) that this hiding-the-signature-in-the-docstring approach is a cheap hack, and it will have unexpected and undesirable side-effects that will in retrospect seem obvious. This is FUD I admit. But it seems to me if we did it "the right way", with a "PyMethodDefEx", we'd be able to do a lot better job predicting the side-effects. A few docstrings appear truncated. Big deal. We can rewrite the ones that are reported as broken (either by adjusting the docstring to not match the patter or by adjusting it to match the pattern better, depending on the case). Tons of docstrings contain incorrect info, we just fix them when we notice the issue, we don't declare the language broken. I don't think #20075 touches on it, but my biggest concern was third-party modules. If you maintain a Python module, you very well might compile for 3.4 only to find that the first line of your docstrings have mysteriously vanished. You'd have to be very current on changes in Python 3.4 to know what was going on. It seemed like an overly efficient way of pissing off external module maintainers. (Why would they vanish? The mechanical separator for __doc__ vs __text_signature__ would accept them, but unless they're 100% compatible Python ast.parse will reject them. So they'd get stripped from your docstring, but you wouldn't get a valid signature in return.) I'd feel much better with an explicit flag--explicit is better than implicit, after all. But here's a reminder, to make it easier for you to say "no". That would mean adding an explicit flag to all the objects which support a signature hidden in the docstring: * PyTypeObject (has tp_flags, we've used 18 of 32 bits by my count) * PyMethodDef (has ml_flags, 7 of 32 bits in use) * PyMethodDescrObject (reuses PyMethodDef) * PyWrapperDescrObject (has d_base->flags, 1 of 32 bits in use) * wrapperobject (reuses PyWrapperDescrObject) Argument Clinic would write the PyMethodDefs, so we'd get those for free. The last three all originate from a PyMethodDef, so when we copied out the docstring pointer we could also propagate the flag. But we'd have to add the flag by hand to the PyTypeObjects. If you won't let me have a flag, can I at least have a more-clever marker? How about this: (...)\n \n Yes, the last four characters are right-parenthesis, newline, space, and newline. Benefits: * The odds of finding *that* in the wild seem remote. * If this got displayed as help in 3.3 the user would never notice the space. For the record, here are things that may be in the signature that aren't legal Python syntax and therefore might be surprising: * "self" parameters (and "module" and "type") are prefixed with '$'. * Positional-only parameters will soon be delimited by '/', just as keyword-only parameters are currently delimited by '*'. (Hasn't happened yet. Needs to happen for 3.4, in order for inspect.Signature to be accurate.) //arry/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The docstring hack for signature information has to go
Am 04.02.2014 10:12, schrieb Larry Hastings: > If you won't let me have a flag, can I at least have a more-clever marker? > How > about this: > > (...)\n \n > > Yes, the last four characters are right-parenthesis, newline, space, and > newline. Benefits: > > * The odds of finding *that* in the wild seem remote. > * If this got displayed as help in 3.3 the user would never notice the > space. Clever, but due to the "hidden" space it also increases the frustration factor for people trying to find out "why is this accepted as a signature and not this". I don't think a well-chosen visible separator is worse off, such as "--\n". Georg ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The docstring hack for signature information has to go
On 02/03/2014 12:55 PM, Terry Reedy wrote: I think the solution adopted should be future-oriented (ie, clean in the future) even if the cost is slight awkwardness in 3.3. Just a minor point: I keep saying 3.3, but I kind of mean "3.2 through 3.3". I believe the binary ABI shipped with 3.2. However, in practice I suspect there are few installations that * are still on 3.2, and * will ever use binary-ABI-clean third-party modules compiled against 3.4+ that contain signatures. //arry/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The docstring hack for signature information has to go
On 02/03/2014 01:10 PM, Zachary Ware wrote:
What about just choosing a marker value that is somewhat less
unsightly? "signature = (", or "parameters: (", or something (better)
to that effect? It may not be beautiful in 3.3, but we can at least
make it make sense.
It's a reasonable enough idea, and we could consider it if we stick with
something like "sig=". However, see later in the thread where Guido
says to return to the old somewhat-ambiguous form with the function
name. ;-)
//arry/
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The docstring hack for signature information has to go
On 02/03/2014 02:06 PM, Gregory P. Smith wrote: Wouldn't your proposal to extend the PyMethodDef structure would require ifdef's and make it impossible to include the type information in something compiled against the 3.3 headers that you want to use in 3.4 without recompiling? It might use #ifdefs. However, my proposal was forwards-compatible. When iterating over the methoddef array passed in with a type, if the PyMethodDef flags parameter had METH_SIGNATURE set, I'd advance by sizeof(PyMethodDefEx) bytes, otherwise I'd advance by sizeof(PyMethodDef) bytes. Modules compiled against 3.3 would not have the flag set, therefore I'd advance by the right amount, therefore they should be fine. //arry/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The docstring hack for signature information has to go
On 02/03/2014 02:26 PM, Antoine Pitrou wrote:
How do you create an array that mixes PyMethodDefs and PyMethodDefExs
together?
You're right, you wouldn't be able to. For my PyMethodDefEx proposal,
the entire array would have to be one way or the other.
It sounds like METH_SIGNATURE is the wrong mechanism.
Instead, you may want a tp_methods_ex as well as as a
Py_TPFLAGS_HAVE_METHODS_EX.
You may well be right. We'd already need a flag on the type object
anyway, to indicate "tp_doc start with a signature". So if we had such
a flag, it could do double-duty to also indicate "tp_methods points to
PyMethodDefEx structures".
My only concern here: __text_signature__ is supported on five internal
objects: PyCFunctionObject, PyTypeObject, PyMethodDescr_Type,
_PyMethodWrapper_Type, and PyWrapperDescr_Type. I'm not certain that all
of those carry around their own pointer back to their original type object.
If you went off the "self" parameter, you wouldn't have one if you were
an unbound method. And you might get the wrong answer if the user bound
you to a different class, or if you were accessed through a subclass.
(I say "might" not to mean "it could happen sometimes", but rather "I
don't know what the correct answer is".)
Note that this constrains future growth to only add pointer fields,
unless you also add a couple of long fields. But at least it sounds
workable.
Ah, in the back of my mind I meant to say "add some unused union {void
*p; long i;} fields". Though in practice I don't think we support any
platforms where sizeof(long) > sizeof(void *).
Uh... If you write a "conversion function", you may as well make it so
it converts the "sig=" line to a plain signature line in 3.3, which
avoids the issue entirely.
Yeah, that's an improvement, though it makes the conversion function a
lot more complicated, and presumably uses more memory.
(and how would that conversion function be shipped to the user anyway?
Python 3.3 and the stable ABI don't have it)
As a C function in a text file, that they'd have to copy into their
program. I know it's ugly.
//arry/
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The docstring hack for signature information has to go
On 02/04/2014 01:41 AM, Georg Brandl wrote:
Clever, but due to the "hidden" space it also increases the frustration factor
for people trying to find out "why is this accepted as a signature and not
this".
I don't think a well-chosen visible separator is worse off, such as "--\n".
I could live with that. To be explicit: the signature would then be of
the form
The scanning function would look for "(" at the
front. If it found it it'd scan forwards in the docstring for
")\n--\n". If it found *that*, then it would declare success.
//arry/
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The docstring hack for signature information has to go
On 02/03/2014 08:05 AM, Stefan Krah wrote: I think we may slowly get into PEP territory here. Just imagine that we settle on X, then decide at a later point to have a standard way of adding type annotations, then find that X does not work because of (unknown). I'm mentioning this because signatures get really interesting for me if they contain type information. I simultaneously share your interest, and also suspect that maybe Python is the wrong language for that. After all, Python has always been about duck-typing. Even if it did happen, it won't be for quite a while yet. The logical mechanism for type information in pure Python is annotations, and afaik they're not getting any large-scale real-world use for type annotating. (If I'm misinformed I'd love to hear counterexamples.) //arry/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The docstring hack for signature information has to go
On Tue, 04 Feb 2014 02:21:51 -0800
Larry Hastings wrote:
> On 02/04/2014 01:41 AM, Georg Brandl wrote:
> > Clever, but due to the "hidden" space it also increases the frustration
> > factor
> > for people trying to find out "why is this accepted as a signature and not
> > this".
> >
> > I don't think a well-chosen visible separator is worse off, such as "--\n".
>
> I could live with that. To be explicit: the signature would then be of
> the form
>
>
> The scanning function would look for "(" at the
> front. If it found it it'd scan forwards in the docstring for
> ")\n--\n". If it found *that*, then it would declare success.
This would have to be checked for layout regressions. If the docstring
is formatted using a ReST-to-HTML converter, what will be the effect?
Regards
Antoine.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] __doc__ regression
Hello, Following the previous clinic thread, I realize that the latest signature improvements actually entail a regression in __doc__. Compare: $ ./python -c "print(dict.fromkeys.__doc__)" Returns a new dict with keys from iterable and values equal to value. $ python3.3 -c "print(dict.fromkeys.__doc__)" dict.fromkeys(S[,v]) -> New dict with keys from S and values equal to v. v defaults to None. As you see the signature string has vanished from the __doc__ contents. This means that any tool directly processing __doc__ attributes to generate (e.g.) automatic API docs will produce less useful docs. I think the signature should be restored, and a smarter way of reconciling __doc__ and the signature for help() should be found. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __doc__ regression
On 02/04/2014 04:17 AM, Antoine Pitrou wrote: As you see the signature string has vanished from the __doc__ contents. This means that any tool directly processing __doc__ attributes to generate (e.g.) automatic API docs will produce less useful docs. Why couldn't these tools use inspect.Signature? //arry/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __doc__ regression
2014-02-04 Larry Hastings : > Why couldn't these tools use inspect.Signature? inspect.Signature was added in Python 3.3. Python 2 is still widely used, and some Linux distro only provide Python 3.2. By the way, help(dict.fromkeys) looks to use __doc__, not the signature, because the prototype is also missing. It's a regression. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __doc__ regression
On Tue, 04 Feb 2014 05:08:28 -0800 Larry Hastings wrote: > On 02/04/2014 04:17 AM, Antoine Pitrou wrote: > > As you see the signature string has vanished from the __doc__ contents. > > This means that any tool directly processing __doc__ attributes to > > generate (e.g.) automatic API docs will produce less useful docs. > > Why couldn't these tools use inspect.Signature? Because it's new in 3.3? Most Python code has been written before that. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] expat symbols should be namespaced in pyexpat (Issue19186 - no progress in four months)
Hi everyone, Just wondering if anyone has looked into http://bugs.python.org/issue19186- priority has been changed to critical four months ago but nothing has happened since. I think it would be nice to get this sorted before python3.4 release Cheers, Lucas ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The docstring hack for signature information has to go
Georg Brandl writes: > I don't think a well-chosen visible separator is worse off, such as "--\n". Don't you mean "-- \n"? L'Ancien Mail-guique-ly y'rs, ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The docstring hack for signature information has to go
On 4 February 2014 02:04, Larry Hastings wrote: > On 02/03/2014 07:08 AM, Barry Warsaw wrote: > > On Feb 03, 2014, at 06:43 AM, Larry Hastings wrote: > > But that only fixes part of the problem. Our theoretical extension that > wants to be binary-compatible with 3.3 and 3.4 still has a problem: how can > they support signatures? They can't give PyMethodDefEx structures to 3.3, > it > will blow up. But if they don't use PyMethodDefEx, they can't have > signatures. > > Can't an extension writer #ifdef around this? Yeah, it's ugly, but it's a > pretty standard approach for making C extensions multi-version compatible. > > > For source compatibility, yes. But I thought the point of the binary ABI > was to allow compiling a single extension that worked unmodified with > multiple versions of Python. If we simply don't support that, then an ifdef > would be fine. Then the solution appears straightforward to me: Python 3.4 will not support providing introspection information through the stable ABI. If you want to provide signature info for your C extension without an odd first line in your 3.3 docstring, you must produce version specific binaries (which allows #ifdef hackery). Then PEP 457 can address this properly for 3.5 along with the other issues it needs to cover. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] test whatever failed
Dear support, we tried to install Phyton-3.3.3 but we got some problems with "make test". Test whatever failed. We trid also "make testall" but we got: Ran 45 tests in 7.897s FAILED (failures=25, skipped=15) test test_gdb failed make: *** [testall] Errore 1 Please could you help us? Thanks Adelaide Adelaide Milani, PhD student Division of Comparative Biomedical Sciences. OIE/FAO and National Reference Laboratory for Avian Influenza and Newcastle Disease FAO Reference Centre for Rabies OIE Collaborating Centre for Infectious Diseases at the Human- Animal Interface Istituto Zooprofilattico Sperimentale delle Venezie Viale dell'Università, 10, 35020, Legnaro (Padova), Italy Tel: 0039 049 8084381 Fax 0039 049 8084360 ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The docstring hack for signature information has to go
Am 04.02.2014 13:14, schrieb Antoine Pitrou:
> On Tue, 04 Feb 2014 02:21:51 -0800
> Larry Hastings wrote:
>> On 02/04/2014 01:41 AM, Georg Brandl wrote:
>> > Clever, but due to the "hidden" space it also increases the frustration
>> > factor
>> > for people trying to find out "why is this accepted as a signature and not
>> > this".
>> >
>> > I don't think a well-chosen visible separator is worse off, such as "--\n".
>>
>> I could live with that. To be explicit: the signature would then be of
>> the form
>>
>> >
>> The scanning function would look for "(" at the
>> front. If it found it it'd scan forwards in the docstring for
>> ")\n--\n". If it found *that*, then it would declare success.
>
> This would have to be checked for layout regressions. If the docstring
> is formatted using a ReST-to-HTML converter, what will be the effect?
The "--" will be added after the signature in the same paragraph.
However, I don't think this is a valid concern: if you process signatures
as ReST you will already have to deal with lots of markup errors (e.g. due
to unpaired "*" and "**").
Tools that extract the docstrings and treat them specially (such as Sphinx)
will adapt anyway.
Georg
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __doc__ regression
Am 04.02.2014 14:19, schrieb Victor Stinner: > 2014-02-04 Larry Hastings : >> Why couldn't these tools use inspect.Signature? > > inspect.Signature was added in Python 3.3. Python 2 is still widely > used, and some Linux distro only provide Python 3.2. Well, Python 2 won't be able to introspect C modules written and compiled for Python 3.x anyway. And __text_signature__ can be used by any Python 3.x with no loss of expressivity (handling strings in both cases). Georg ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test whatever failed
This is a mailing list for the development *of* Python, not *with* Python. The best way to get help would be to email [email protected] or [email protected]. On Tue, Feb 4, 2014 at 10:15 AM, Milani Adelaide wrote: > Dear support, > we tried to install Phyton-3.3.3 but we got some problems with "make > test". Test whatever failed. > > We trid also "make testall" but we got: > > Ran 45 tests in 7.897s > > FAILED (failures=25, skipped=15) > test test_gdb failed > make: *** [testall] Errore 1 > > Please could you help us? > Thanks > Adelaide > > > Adelaide Milani, PhD student > Division of Comparative Biomedical Sciences. > OIE/FAO and National Reference Laboratory for Avian Influenza and > Newcastle Disease > FAO Reference Centre for Rabies > OIE Collaborating Centre for Infectious Diseases at the Human- Animal > Interface > Istituto Zooprofilattico Sperimentale delle Venezie > Viale dell'Università, 10, 35020, Legnaro (Padova), Italy > Tel: 0039 049 8084381 > Fax 0039 049 8084360 > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] expat symbols should be namespaced in pyexpat (Issue19186 - no progress in four months)
On 2/4/2014 9:03 AM, Lukas Vacek wrote: Hi everyone, Just wondering if anyone has looked into http://bugs.python.org/issue19186 - priority has been changed to critical four months ago but nothing has happened since. I think it would be nice to get this sorted before python3.4 release Benjamin applied patch and closed issue. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __doc__ regression
On 02/04/2014 05:19 AM, Victor Stinner wrote: 2014-02-04 Larry Hastings : Why couldn't these tools use inspect.Signature? inspect.Signature was added in Python 3.3. Python 2 is still widely used, and some Linux distro only provide Python 3.2. By the way, help(dict.fromkeys) looks to use __doc__, not the signature, because the prototype is also missing. It's a regression. In 3.4, inspect.getfullargspec and inspect.getargspec are being reimplemented using inspect.Signature. I don't understand your bringing up Python 2 and Python 3.2. Are there programs that run under Python 2 and Python 3.2 that examine docstrings from Python 3.4? //arry/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __doc__ regression
On 02/04/2014 05:24 AM, Antoine Pitrou wrote: On Tue, 04 Feb 2014 05:08:28 -0800 Larry Hastings wrote: On 02/04/2014 04:17 AM, Antoine Pitrou wrote: As you see the signature string has vanished from the __doc__ contents. This means that any tool directly processing __doc__ attributes to generate (e.g.) automatic API docs will produce less useful docs. Why couldn't these tools use inspect.Signature? Because it's new in 3.3? Most Python code has been written before that. inspect.getargspec and inspect.getfullargspec in 3.4 use inspect.Signature to get the signatures of builtins. Does this address your concerns? //arry/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Decision on the docstring hack (was Re: The docstring hack for signature information has to go)
In the absence of a ruling from above I'm making a decision. I say we
keep the docstring hack, but go back to the human-readable version, only
this time with a *much* stricter signature. That should reduce the
number of false positives to zero.
Furthermore, I say we don't add any flags--adding a flag to the callable
would mean doing it three different ways, and adding a flag to the type
might not work in all cases.
And finally, I agree with Nick, that publishing signatures shall not be
a supported API for Python 3.4. If external modules publish signatures,
and we break their support in 3.5, it's on them.
The rules enforced on the signature will be as per Georg's suggestion
but made slightly more stringent:
The signature must start with "(".
The signature must end with ")\n--\n\n".
The signature may contain newlines, but must not contain empty lines.
The signature may have non-Python syntax in it, such as "$"
indicating a self parameter, "/" to indicate positional-only
parameters, and "[" and "]" to indicate optional groups (though I
don't expect 3.4 will ship with support for this last one).
Here it is in non-working pseudo-code:
s = function.docstring
if not s.startswith(function.name + "(")
return failure
end = s.find(")\n--\n\n")
if end < 0:
return failure
newline = s.find("n\n")
if newline > 0 and newline < end:
return failure
return success
(The actual implementation will be in C, in find_signature in
Objects/typeobject.c.)
Expect a patch in about 24 hours,
//arry/
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Decision on the docstring hack (was Re: The docstring hack for signature information has to go)
Sounds like a good compromise to me - and we certainly have a lot of interesting experience to feed into the design of PEP 457 for 3.5 ;) Cheers, Nick. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
