Re: [Python-ideas] a new namedtuple

[Guido van Rossum]

On Mon, Jul 17, 2017 at 6:25 PM, Eric Snow 
wrote:

> On Mon, Jul 17, 2017 at 6:01 PM, Ethan Furman  wrote:
> > Guido has decreed that namedtuple shall be reimplemented with speed in
> mind.
>
> FWIW, I'm sure that any changes to namedtuple will be kept as minimal
> as possible.  Changes would be limited to the underlying
> implementation, and would not include the namedtuple() signature, or
> using metaclasses, etc.  However, I don't presume to speak for Guido
> or Raymond. :)
>

Indeed. I referred people here for discussion of ideas like this:

>>> a = (x=1, y=0)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[David Mertz]

Can you try across a range of tuple sizes? E.g. what about with 100 items?
1000?

On Jul 17, 2017 7:56 PM, "Ethan Furman"  wrote:

On 07/17/2017 06:34 PM, Steven D'Aprano wrote:

> On Mon, Jul 17, 2017 at 05:01:58PM -0700, Ethan Furman wrote:
>

Guido has decreed that namedtuple shall be reimplemented with speed in mind.
>>
>> I haven't timed it (I'm hoping somebody will volunteer to be the bench
>> mark
>> guru), I'll offer my NamedTuple implementation from my aenum [1] library.
>>
>
> With respect Ethan, if you're going to offer up NamedTuple as a faster
> version of namedtuple, you should at least do a quick proof of
> concept to demonstrate that it actually *is* faster.
>

I suck at benchmarking, so thank you for providing those quick-and-dirty
hints.


Full bench marking
> can wait, but you should be able to do at least something like:
>
>
> python3 -m timeit --setup "from collections import namedtuple" \
>  "K = namedtuple('K', 'a b c')"
>

546 usec


versus
>
> python3 -m timeit --setup "from aenum import NamedTuple" \
>  "K = NamedTuple('K', 'a b c')"
>

250 usec


So it seems to be faster!  :)

It is also namedtuple compatible, except for the _source attribute.

--
~Ethan~

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[Ethan Furman]

On 07/17/2017 06:34 PM, Steven D'Aprano wrote:

On Mon, Jul 17, 2017 at 05:01:58PM -0700, Ethan Furman wrote:



Guido has decreed that namedtuple shall be reimplemented with speed in mind.

I haven't timed it (I'm hoping somebody will volunteer to be the bench mark
guru), I'll offer my NamedTuple implementation from my aenum [1] library.


With respect Ethan, if you're going to offer up NamedTuple as a faster
version of namedtuple, you should at least do a quick proof of
concept to demonstrate that it actually *is* faster.


I suck at benchmarking, so thank you for providing those quick-and-dirty hints.


Full bench marking
can wait, but you should be able to do at least something like:


python3 -m timeit --setup "from collections import namedtuple" \
 "K = namedtuple('K', 'a b c')"


546 usec


versus

python3 -m timeit --setup "from aenum import NamedTuple" \
 "K = NamedTuple('K', 'a b c')"


250 usec


So it seems to be faster!  :)

It is also namedtuple compatible, except for the _source attribute.

--
~Ethan~
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[Steven D'Aprano]

On Mon, Jul 17, 2017 at 05:01:58PM -0700, Ethan Furman wrote:
> Guido has decreed that namedtuple shall be reimplemented with speed in mind.
> 
> I haven't timed it (I'm hoping somebody will volunteer to be the bench mark 
> guru), I'll offer my NamedTuple implementation from my aenum [1] library.  

With respect Ethan, if you're going to offer up NamedTuple as a faster 
version of namedtuple, you should at least do a quick proof of 
concept to demonstrate that it actually *is* faster. Full bench marking 
can wait, but you should be able to do at least something like:


python3 -m timeit --setup "from collections import namedtuple" \
"K = namedtuple('K', 'a b c')"

versus 

python3 -m timeit --setup "from aenum import NamedTuple" \
"K = NamedTuple('K', 'a b c')"

(or whatever the interface is). If there's only a trivial speed up, or 
if its slower, then there's no point even considing it unless you speed 
it up first.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[Eric Snow]

On Mon, Jul 17, 2017 at 6:01 PM, Ethan Furman  wrote:
> Guido has decreed that namedtuple shall be reimplemented with speed in mind.

FWIW, I'm sure that any changes to namedtuple will be kept as minimal
as possible.  Changes would be limited to the underlying
implementation, and would not include the namedtuple() signature, or
using metaclasses, etc.  However, I don't presume to speak for Guido
or Raymond. :)

-eric
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[Joseph Jevnik]

If we are worried about speed but want to keep the same API I have a
near drop in replacement for collections.namedtuple that dramatically
improves class and instance creation speed [1]. The only things
missing from this implementation are `_source` and `verbose` which
could be dynamically computed to provide equivalent Python source.
This project was originally proposed as a replacement for the standard
namedtuple, but after talking to Raymond we decided the performance
did not outweigh the simplicity of the existing implementation. Now
that people seem more concerned with performance, I wanted to bring
this up again.

[1] https://github.com/ll/cnamedtuple

On Mon, Jul 17, 2017 at 8:04 PM, Ivan Levkivskyi  wrote:
> Just FYI, typing.NamedTuple is there for almost a year and already supports
> default values, methods, docstrings etc.
> Also there is ongoing work towards dataclasses PEP, see
> https://github.com/ericvsmith/dataclasses
>
> So that would keep namedtuple API as it is, and focus only on performance
> improvements.
>
> --
> Ivan
>
>
>
> On 18 July 2017 at 02:01, Ethan Furman  wrote:
>>
>> Guido has decreed that namedtuple shall be reimplemented with speed in
>> mind.
>>
>> I haven't timed it (I'm hoping somebody will volunteer to be the bench
>> mark guru), I'll offer my NamedTuple implementation from my aenum [1]
>> library.  It uses the same metaclass techniques as Enum, and offers doc
>> string and default value support in the class-based form.
>>
>> --
>> ~Ethan~
>>
>>
>> [1] https://pypi.python.org/pypi/aenum/1.4.5
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[Ivan Levkivskyi]

Just FYI, typing.NamedTuple is there for almost a year and already supports
default values, methods, docstrings etc.
Also there is ongoing work towards dataclasses PEP, see
https://github.com/ericvsmith/dataclasses

So that would keep namedtuple API as it is, and focus only on performance
improvements.

--
Ivan



On 18 July 2017 at 02:01, Ethan Furman  wrote:

> Guido has decreed that namedtuple shall be reimplemented with speed in
> mind.
>
> I haven't timed it (I'm hoping somebody will volunteer to be the bench
> mark guru), I'll offer my NamedTuple implementation from my aenum [1]
> library.  It uses the same metaclass techniques as Enum, and offers doc
> string and default value support in the class-based form.
>
> --
> ~Ethan~
>
>
> [1] https://pypi.python.org/pypi/aenum/1.4.5
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] a new namedtuple

[Ethan Furman]

On 07/17/2017 05:01 PM, Ethan Furman wrote:


I haven't timed it (I'm hoping somebody will volunteer to be the bench mark 
guru), I'll offer my NamedTuple
implementation from my aenum [1] library.  It uses the same metaclass 
techniques as Enum, and offers doc string and
default value support in the class-based form.


Oh, and to be clear, there are some other nice-to-have and/or neat features (such as variable-sized NamedTuples), that I 
would expect to be trimmed from a stdlib version.


--
~Ethan~

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] a new namedtuple

[Ethan Furman]

Guido has decreed that namedtuple shall be reimplemented with speed in mind.

I haven't timed it (I'm hoping somebody will volunteer to be the bench mark guru), I'll offer my NamedTuple 
implementation from my aenum [1] library.  It uses the same metaclass techniques as Enum, and offers doc string and 
default value support in the class-based form.


--
~Ethan~


[1] https://pypi.python.org/pypi/aenum/1.4.5
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Is this PEP viable?

[Brett Cannon]

This doesn't require a PEP as it falls under "adding" to a pre-existing
module: https://devguide.python.org/stdlibchanges/.

On Sun, 16 Jul 2017 at 22:06 Evan Adler  wrote:

> I would like to submit the following proposal. In the logging module, I
> would like handlers (like file handlers and stream handlers) to have a
> field for exc_info printing. This way, a call to logger.exception() will
> write the stack trace to the handlers with this flag set, and only print
> the message and other info to handlers without the flag set. This allows a
> single logger to write to a less detailed console output, a less detailed
> run log, and a more detailed error log.
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Arguments to exceptions

[Stephen J. Turnbull]

First, I would like to make it clear that I am not claiming that
the current design of standard Exceptions cannot be improved, nor do I
necessarily have a problem with doing it at the BaseException level.
My argument has always been directed to the issue of "does changing
BaseException actually save work in design and implementation of the
interpreter, of standard derived Exceptions, and of new user
Exceptions?"  I don't think it does.

But Ken and you have taken opposition to Ken's proposal to indicate
that we don't see a problem with the current implementation of the
standard Exception hierarchy.  That's not true.  We do understand the
problems.  We differ on importance: Steven d'A seems to deprecate the
NameError in "data-sets-formatted-as-Python-containers" problem as
rare and only encountered by a few users.  I think that's a big enough
class of use cases (I do it myself, though I've never encountered
Ken's use case) to worry about, though I wouldn't be willing to do the
work myself on the evidence so far presented.  But we do understand.

What you and Ken haven't done yet is to show

1.  the implementation of an improvement is simple and generic, and
2.  enough so to justify what seems to be an improvement restricted to
an uncommon set of use cases.

Nick has helped to convince me that (1) can be addressed, though he
warned that it's not all that easy (especially if you're a DRY
fanatic).  The question then is the balance between (1) and (2).

Jeff Walker writes:

 > The first problem is that there is no direct access to the
 > components that make up the error in some of the standard Python
 > exceptions.

That's not a problem in BaseException, though.  BaseException allows
access to arbitrary Python objects, if, and only if, the developer
puts them in the exception.

 > Why wasn't that done?

I don't know, but it's not prevented by BaseException.

 > That leads us to the second problem: the base exception does not
 > handle arguments gracefully unless you only pass an error message
 > all by itself.

I think it would have been a mistake to try to do something graceful.
The point is that looking at the design, I would suppose that
BaseException had only one job: conveying whatever information the
programmer chose to put in it faithfully to an exception handler.  It
does that, even if you hand it something really wierd, and in several
pieces.

This is called "humility in design when you have no idea what you're
designing for".  The Zen expresses it "But sometimes never is better
than *right* now."  It's easy to design a better BaseException for a
handful of cases.  It's not at all clear that such a BaseException
would work well for more than a double handful.  If it doesn't, it
would tend to cause the same problem for exceptions that don't fit it
very well but the programmer doesn't think are important enough to
deserve a well-defined derived Exception.  Since it would necessarily
be more complex than the current BaseException, it might have even
uglier failure modes.

 > >>> try:
 > >>> name = 'foo'
 > >>> raise NameError(name, '%s: name is not defined.' % name)
l > >>> except NameError as e:
 > >>> print(str(e))
 > ('foo', 'foo: name is not defined.')
 > 
 > In this case, printing the exception cast to a string does not
 > result in a reasonable error message.

Sure, but that's because the error message was poorly chosen.  I would

raise NameError(name, 'error: undefined name')

which would stringify as

('foo', 'error: undefined name')

(and there are probably even better ways to phrase that).  But in the
case of NameError, I don't understand why a string is used at all.  I
really don't see a problem with the interpreter printing

NameError: foo

and a traceback indicating where "foo" was encountered, and having
programmatic handlers do something appropriate with e.args[0].  It's
not obvious to me that the same is not true of most of the "stupid
string formatting" Exceptions that have been given as examples.

 > So the basic point is that the lack of reasonable behavior for
 > str(e)

I hate to tell you this, but str doesn't guarantee anything like
reasonable behavior for error handling.  Even repr doesn't guarantee
you can identify an object passed to it, nor reproduce it.

I have to assume that the developers who designed these Exceptions
believed that in the vast majority of cases the mere fact that an
instance of particular Exception was raised is enough to identify the
offending object.  Perhaps they were mistaken.  But if they were, the
places in the interpreter where they are raised and handled all must
be be checked to see if they need to be changed, whether we change
BaseException or the individual derived Exceptions.  And if we change
BaseException, I expect it will be very desirable to change individual
derived Exceptions as well (to give them intelligible error message
templates).

Also, it's not obvious to me that default templates are 

Re: [Python-ideas] Custom Code Folding: Standardized Rules and Syntax?

[Rhodri James]

On 16/07/17 16:15, Steven D'Aprano wrote:

Hi Connor, and welcome!

On Sun, Jul 16, 2017 at 10:37:26AM -0400, Connor Farrell wrote:

Background: I work in scientific computing and use Community Pycharm IDE.
In the end, I end up with very well folded code (except for large parameter
lists) and a bunch of co-workers asking about all the "editor-fold"
comments that don't work in their API.


I'm afraid I'm not understanding you here either. What's an
"editor-fold" comment? What do they mean by API? API for which
application? How does the programming interface to an application relate
to code folding in a text editor?


The usual mechanism for folding (in those editors that don't think they 
know best) is to delimit them with comments that start with a particular 
sequence of characters.  You might have:


class Shrubbery:
  def __init__(self, size, planting, rockery=False):
# {{{ Boring bits you don't need to see
self.size = size
self.rockery = rockery
# }}}
self.cost = complicated_function_of(planting)

...and so on.


Python was a trail-blazer in terms of emphasizing the importance of code
readability and effective syntax. I think that we should consider some sort
of standard for folding comments, if not only to promote stronger code
organizations. I know standards don't usually interact with IDEs, but hey,
the 'typing' module is pretty dang nice.

TL;DR: Code folding is great, custom code folding is great, let's upgrade
it to a language feature.


What does that even mean? Are you suggesting that the Python interpreter
should raise a SyntaxError or something if your code was written in an
editor that didn't support code folding? How would it know?

Python is a programming language. The source code is text. I should be
able to write Python code in NotePad if I want. Why should the Python
core devs try to force every text editor and IDE fold code exactly the
same way? That sounds awful to me. People choose different editors
because they like different features, and that may include the
particular way the editor folds code. Or to not fold it at all.

I'm sorry to be so negative, but I don't understand your proposal, and
the bits that I *think* I understand sound pretty awful to me. Perhaps
you can explain a bit better what you mean and why it should be a
language feature, apart from "I want everyone to lay out their source
code the way I do". Because that's what it sounds like to me.


I think I do understand the proposal, but I still agree with Steven. 
Folding is a feature of IDEs, not languages, and trying to legislate 
about it in a language definition is a bad plan.  I say this as someone 
who wrote a folding editor to make programming in Occam bearable :-)


--
Rhodri James *-* Kynesim Ltd
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Arguments to exceptions

[Nick Coghlan]

On 17 July 2017 at 04:59, Ken Kundert  wrote:
> Nick,
> I see. The Python C interface provides a very simple way of raising an
> exception in the case where the exception is only passed one argument, the 
> error
> message. It even makes it easy to interpolate arguments into that error 
> message.
> If it is important to include other arguments, as with OSError, one would
> presumably use another mechanism that requires more effort.
>
> Your suggestion of providing an alternative to PyErr_Format() that squirreled
> away its arguments into the exception and deferred their interpolation into 
> the
> error message seems like a very nice approach. But as you say, it does not 
> seem
> trivial to implement in C. If it could be extended to support named arguments,
> it does have the benefit that it could unify the way exceptions are raise from
> C.

I'll note that while it wouldn't be trivial, it *should* at least
theoretically be possible to share the string segmentation and
processing steps with the f-string implementation.

However, I also realised that a much simpler approach to the same idea
would instead look like:

PyErr_SetFromAttributes(PyExc_NameError, "name '%.200s' is not
defined", "name", name);

Where the field names alternate with the field values in the argument
list, rather than being embedded in the format string.

Ignoring error handling, the implementation would then be something like:

- split va_list into a list of field names and a list of field values
- set the exception via `PyErr_SetObject(exc_type,
PyUnicode_Format(exc_msg, field_values))`
- retrieve the just set exception and do `PyObject_SetAttr(exc,
field_name, field_value)` for each field with a non-NULL name

> The alternative would be to simply enhance the individual exceptions on an as
> needed basis as you suggested earlier. That could be easy if the exceptions 
> are
> only raised in one or two places. Do you have a sense for how many places 
> raise
> some of these common exceptions such as NameError, KeyError, etc.?

A quick check shows around a dozen hits for PyExc_NameError in c files
(all in ceval.c), and most of the hits for NameError in Python files
being related to catching it rather than raising it.

PyExc_KeyError is mentioned in a dozen or so C files (most of them
raising it, a few of them check for it being raised elsewhere), and
while the hits in Python files still mostly favour catching it, it's
raised explicitly in the standard library more often than NameError
is.

The nice thing about the PyErr_SetFromAttributes approach at the C API
level is that it pairs nicely with the following approach at the
constructor level:

class NameError(Exception):
def __init__(self, message, *, name=None):
super().__init__(message)
self.name = name

Something like KeyError may want to distinguish between "key value was
None" and "key value wasn't reported when raising the exception" (by
not defining the attribute at all in the latter case), but
PyErr_SetFromAttributes wouldn't need to care about those kinds of
details.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/