Re: [Python-ideas] Better error messages [was: (no subject)]
On 30 November 2016 at 02:14, Stephen J. Turnbull wrote: > How about: > > class Blog: > pass > > blog = get_blog_for_date(someday) > > logn = log(blog.size) > > NameError: Python doesn't recognize the function "log". Did you > mean "Blog"? > > Wouldn't > > NameError: Python doesn't recognize the name "log". Perhaps > you need to import the "math" module? ... and of course up until this example, I'd assumed you were talking about the log function from the logging module :-) I'm a strong +1 on better error messages, but there's always a risk with heuristics that the resulting messages end up worse, not better. Maybe keep it simpler: NameError: Python doesn't recognize the name "log". Maybe you misspelled the name, or did you mean to import the function from a module? and don't try to guess the user's intent. Paul ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Add optional defaults to namedtuple
It would be nice to have a supported way to add defaults to namedtuple, so the slightly hacky solution here does not have to be used: http://stackoverflow.com/a/18348004/2570866 Jelte ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] (no subject)
On 29/11/2016 20:09, Terry Reedy wrote: On 11/29/2016 11:32 AM, Rob Cliffe wrote: On 29/11/2016 04:58, victor rajewski wrote: Traceback (most recent call last): File "foo.py", line 2, in l[10]=14 IndexError: list assignment index out of range A better message might be: You tried to use l[10] when l is only 4 elements long. You can add items to l using l.append(value), or check your index value to make sure that's really the position you wanted to access. It would make sense to me to upgrade this particular error message to IndexError: list assignment index 10 out of range 0 to 3 if it can be done without too much difficulty or overhead. (An empty list, and perhaps one with only 1 element, would be special cases.) Come to think of it, is the word "assignment" needed? It would help if the line were "l1[10] = 2 * l2[13] + 3". You're right of course; "assignment" IS meaningful; I missed it. >>> [][0] Traceback (most recent call last): File "", line 1, in IndexError: list index out of range ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Better error messages [was: (no subject)]
On 30 November 2016 at 19:05, Paul Moore wrote: > On 30 November 2016 at 02:14, Stephen J. Turnbull > wrote: >> Wouldn't >> >> NameError: Python doesn't recognize the name "log". Perhaps >> you need to import the "math" module? > > ... and of course up until this example, I'd assumed you were talking > about the log function from the logging module :-) > > I'm a strong +1 on better error messages, but there's always a risk > with heuristics that the resulting messages end up worse, not better. > > Maybe keep it simpler: > > NameError: Python doesn't recognize the name "log". Maybe you > misspelled the name, or did you mean to import the function from a > module? > > and don't try to guess the user's intent. This brings up a point that I was going to mention earlier: when it comes to specialised learning environments, the folks developing the curriculum also know *what problem the student is working on*, and can tailor their error messages accordingly. The reference interpreter is never going to be able to guess intent like that due to the sheer scope of Python's use cases - by covering everything from students writing "Guess a number" games to mechanical engineers modeling and tuning race car performance to sysadmins automating service deployments to web developers responding to user requests to data analysts trying to make sense of noisy data, we end up being *really* limited in the assumptions we can make about what a user was really trying to do when they accidentally ask for something nonsensical. Tweaking some of the default representations to mention common problems should be OK, though. While some veteran programmers may find such prompts a bit condescending, they'd be better equipped than beginners to opt in to alternative exception display options that omit the hints. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Decorator to avoid a mistake
Hello, While I think this should not be "on by default", I don't see the harm in being able to opt-in to this behavior. I also figured spending a few minutes attempting to write this would be fun: https://gist.github.com/veloutin/2ec3e5246651f5de78442516d8e24fc1 François: sorry about the double reply, I forgot to reply to the list. Vince ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add optional defaults to namedtuple
On 11/30/2016 02:32 AM, Jelte Fennema wrote:
It would be nice to have a supported way to add defaults to namedtuple,
so the slightly hacky solution here does not have to be used:
http://stackoverflow.com/a/18348004/2570866
Actually, the solution right below it is better [1]:
--> from collections import namedtuple
--> class Node(namedtuple('Node', ['value', 'left', 'right'])):
--> __slots__ = ()
--> def __new__(cls, value, left=None, right=None):
--> return super(Node, cls).__new__(cls, value, left, right)
But even more readable than that is using the NamedTuple class from my aenum
[3] library (and on SO as [3]):
--> from aenum import NamedTuple
--> class Node(NamedTuple):
--> val = 0
--> left = 1, 'previous Node', None
--> right = 2, 'next Node', None
shamelessly-plugging-my-own-solutions'ly yrs,
--
~Ethan~
[1] http://stackoverflow.com/a/16721002/208880
[2] https://pypi.python.org/pypi/aenum
[3] http://stackoverflow.com/a/40891597/208880
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add optional defaults to namedtuple
On Wed, Nov 30, 2016 at 7:09 AM, Ethan Furman wrote:
> On 11/30/2016 02:32 AM, Jelte Fennema wrote:
>
> It would be nice to have a supported way to add defaults to namedtuple,
>> so the slightly hacky solution here does not have to be used:
>> http://stackoverflow.com/a/18348004/2570866
>>
>
> Actually, the solution right below it is better [1]:
>
> --> from collections import namedtuple
> --> class Node(namedtuple('Node', ['value', 'left', 'right'])):
> --> __slots__ = ()
> --> def __new__(cls, value, left=None, right=None):
> --> return super(Node, cls).__new__(cls, value, left, right)
>
> But even more readable than that is using the NamedTuple class from my
> aenum [3] library (and on SO as [3]):
>
> --> from aenum import NamedTuple
> --> class Node(NamedTuple):
> --> val = 0
> --> left = 1, 'previous Node', None
> --> right = 2, 'next Node', None
>
> shamelessly-plugging-my-own-solutions'ly yrs,
>
Ditto: with PEP 526 and the latest typing.py (in 3.6) you will be able to
do this:
class Employee(NamedTuple):
name: str
id: int
We should make it so that the initial value in the class is used as the
default value, too. (Sorry, this syntax still has no room for a docstring
per attribute.)
--
--Guido van Rossum (python.org/~guido )
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] (no subject)
On Tue, 29 Nov 2016 at 12:32 MRAB wrote:
> On 2016-11-29 19:45, Brendan Barnwell wrote:
> > On 2016-11-29 09:43, Brett Cannon wrote:
> >> One way to make this cheap is to have a reasonable default message and
> >> use attributes on the exceptions trigger the use of the default message.
> >> Nearly a year ago I filed a bunch of issues for ideas on providing
> >> attributes on exceptions where it made sense, e.g. an index attribute on
> >> IndexError (http://bugs.python.org/issue18162). If we did this then for
> >> classes like IndexError there constructor could be `IndexError(index=10,
> >> start=0, end=3)` and then __str__() can lazily construct the string
> >> representation using a default message, e.g. `"index {} is out of range
> >> of{} to {}".format(index, start, end)`. Make the arguments keyword-only
> >> and they become backwards-compatible and so the only overhead you pay
> >> for these richer messages are keyword-based construction if you simply
> >> never access the repr for the exception.
> >
> > I absolutely think this is the way to go. Having the relevant
> > information (the list that was too short, the index that was too big,
> > the key that wasn't there, etc.) is useful in many situations, and it's
> > much better to have that information in a programmatic form than just
> > squashed into an error message. This then makes it relatively easy to
> > write wrappers that take bubbling-up exceptions and try to construct
> > more detailed messages for a less experienced audience. Right now this
> > is difficult or impossible because the exception objects don't record
> > the information that would be needed for these expanded messages.
> >
> Couldn't that result in objects being held for longer, taking up memory,
> not being collected as promptly, and not releasing resources as quickly?
>
Sure, just like passing any other object into BaseException's initializer
so it gets stored in the args attribute. Notice how my example only used
ints, and that was on purpose. If you want to only pass in the repr of the
type for the message, then simply enforce that or at least encourage it to
prevent/discourage people from using whole objects instead of ints and
strings which are cheaper than the whole string message that is currently
constructed eagerly.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Allow random.choice, random.sample to work on iterators
Currently these functions fail if the supplied object has no len(). There are algorithms for this task that can work on any finite iterator (for example, files as a stream of lines), and the functions could fall back to these if there is no len(). ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Add optional defaults to namedtuple
On 30 November 2016 at 13:09, Ethan Furman wrote: > But even more readable than that is using the NamedTuple class from my aenum > [3] library (and on SO as [3]): > > --> from aenum import NamedTuple > --> class Node(NamedTuple): > --> val = 0 > --> left = 1, 'previous Node', None > --> right = 2, 'next Node', None > > shamelessly-plugging-my-own-solutions'ly yrs, Sorry - taking the boat to even-more-shamelessly anounce extradict.extratuple.defaultnamedtuple - in the newly released extradict v. 0.2.5 https://pypi.python.org/pypi/extradict/0.2.5 It allows one to build an default-paremetrized namedtuple by passing a sequence of 2-tuples with key, values, or, on Python 3.6, pass in the default values as keywords to the defaultnamedtuple factory. (The "extradict" package, with a faster reimplementation of namedtuple already existed, of course - maybe someone can pick some other weird idea I have into there to put it into more day-to-day use) > -- > ~Ethan~ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow random.choice, random.sample to work on iterators
On 2016-11-30 17:25, Random832 wrote: Currently these functions fail if the supplied object has no len(). There are algorithms for this task that can work on any finite iterator (for example, files as a stream of lines), and the functions could fall back to these if there is no len(). I like the idea, as long as it does not add too much overhead to currently existing code. It could be a special code path for reservoir sampling (I assume) for both functions (the first taking only one sample from the stream). -- Bernardo Sulzbach http://www.mafagafogigante.org/ mafagafogiga...@gmail.com ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow random.choice, random.sample to work on iterators
This was also brought up back in April: https://mail.python.org/pipermail//python-ideas/2016-April/039707.html It got a few replies from Guido (https://mail.python.org/pipermail//python-ideas/2016-April/039713.html for one of them). It seems the idea got dropped due to problems with making it properly random (practically, the sampling has issues in cases of very large sequences, let along infinite) and performance (either large numbers of calls to random, or copying the sequence, each of which has its own problems). There are also issues with how it should behave on iterables that cannot be re-iterated (eg, random.choice will consume the iterator, and could only be called once safely). Chris On Wed, Nov 30, 2016 at 11:42 AM, Bernardo Sulzbach wrote: > On 2016-11-30 17:25, Random832 wrote: >> >> Currently these functions fail if the supplied object has no len(). >> There are algorithms for this task that can work on any finite iterator >> (for example, files as a stream of lines), and the functions could fall >> back to these if there is no len(). > > > I like the idea, as long as it does not add too much overhead to currently > existing code. > > It could be a special code path for reservoir sampling (I assume) for both > functions (the first taking only one sample from the stream). > > > -- > Bernardo Sulzbach > http://www.mafagafogigante.org/ > mafagafogiga...@gmail.com > > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow random.choice, random.sample to work on iterators
On Wed, Nov 30, 2016 at 11:52 AM, Chris Kaynor wrote: > There are also issues with how it should behave on iterables that > cannot be re-iterated (eg, random.choice will consume the iterator, > and could only be called once safely). I meant to include a sample in my previous e-mail: Consider that this code will not produce the "correct" results (for a reasonable definition of correct): a = (i for i in range(100)) # Pretend this does something more interesting, and isn't a trivial generator - maybe a file object reading by line. randomEntries = [random.choice(a) for i in range(10)] # May not be quite as obvious, such as the choices could be chosen in a more complex loop. randomEntries may not contain 10 items (or the generation may error due to having insufficent items, depending on implementation), it will not contain duplicates, and will be sorted. This occurs because the first call to random.choice will consume elements from a until it picks one. The next call then consumes from there until it picks one, and so forth. Chris ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow random.choice, random.sample to work on iterators
On 2016-11-30 17:57, Chris Kaynor wrote: On Wed, Nov 30, 2016 at 11:52 AM, Chris Kaynor wrote: There are also issues with how it should behave on iterables that cannot be re-iterated (eg, random.choice will consume the iterator, and could only be called once safely). I meant to include a sample in my previous e-mail: Consider that this code will not produce the "correct" results (for a reasonable definition of correct): a = (i for i in range(100)) # Pretend this does something more interesting, and isn't a trivial generator - maybe a file object reading by line. randomEntries = [random.choice(a) for i in range(10)] In such a case you should explicitly use a sample. I see your example as the caller's fault, which ignored the fact that the iterator would change after calls to choice. Hold the first 10 (in this case). For every subsequent element, randomly choose to replace one of the "held" ones by it (with a diminishing probability). Assume this does not offer the same performance as loading everything into memory. But it isn't meant to do so, as if you need / can / want, you could just shove it all into a list and use what we currently have. -- Bernardo Sulzbach http://www.mafagafogigante.org/ mafagafogiga...@gmail.com ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow random.choice, random.sample to work on iterators
Is the goal to allow them to consume a finite generator of *unknown* length (requires reservoir sampling https://en.wikipedia.org/wiki/Reservoir_sampling with N random calls, which seemed to be the rub before?) or just consume a generator with known length that's not indexable (a rare beast?). Consuming iterables if they have a length like below wouldn't be so bad, but might be too niche. class X: def __init__(self, ele): self.ele = ele def __len__(self): return len(self.ele) def __iter__(self): return iter(self.ele) x = X([1, 2, 3, 4, 5]) random.choice(x) # TypeError: 'X' object does not support indexing Would allowing an optional 'len' argument alongside the iterator to sample/choice be too narrow to be useful? On Wed, Nov 30, 2016 at 2:21 PM, Bernardo Sulzbach < mafagafogiga...@gmail.com> wrote: > On 2016-11-30 17:57, Chris Kaynor wrote: > >> On Wed, Nov 30, 2016 at 11:52 AM, Chris Kaynor >> wrote: >> >>> There are also issues with how it should behave on iterables that >>> cannot be re-iterated (eg, random.choice will consume the iterator, >>> and could only be called once safely). >>> >> >> I meant to include a sample in my previous e-mail: >> >> Consider that this code will not produce the "correct" results (for a >> reasonable definition of correct): >> >> a = (i for i in range(100)) # Pretend this does something more >> interesting, and isn't a trivial generator - maybe a file object >> reading by line. >> randomEntries = [random.choice(a) for i in range(10)] >> > > In such a case you should explicitly use a sample. > > I see your example as the caller's fault, which ignored the fact that the > iterator would change after calls to choice. > > Hold the first 10 (in this case). For every subsequent element, randomly > choose to replace one of the "held" ones by it (with a diminishing > probability). > > Assume this does not offer the same performance as loading everything into > memory. But it isn't meant to do so, as if you need / can / want, you could > just shove it all into a list and use what we currently have. > > -- > Bernardo Sulzbach > http://www.mafagafogigante.org/ > mafagafogiga...@gmail.com > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow random.choice, random.sample to work on iterators
On Wed, Nov 30, 2016 at 12:21 PM, Bernardo Sulzbach wrote: > On 2016-11-30 17:57, Chris Kaynor wrote: >> >> On Wed, Nov 30, 2016 at 11:52 AM, Chris Kaynor >> wrote: >>> >>> There are also issues with how it should behave on iterables that >>> cannot be re-iterated (eg, random.choice will consume the iterator, >>> and could only be called once safely). >> >> >> I meant to include a sample in my previous e-mail: >> >> Consider that this code will not produce the "correct" results (for a >> reasonable definition of correct): >> >> a = (i for i in range(100)) # Pretend this does something more >> interesting, and isn't a trivial generator - maybe a file object >> reading by line. >> randomEntries = [random.choice(a) for i in range(10)] > > > In such a case you should explicitly use a sample. > > I see your example as the caller's fault, which ignored the fact that the > iterator would change after calls to choice. > > Hold the first 10 (in this case). For every subsequent element, randomly > choose to replace one of the "held" ones by it (with a diminishing > probability). > > Assume this does not offer the same performance as loading everything into > memory. But it isn't meant to do so, as if you need / can / want, you could > just shove it all into a list and use what we currently have. It would be the caller's fault: they failed to follow the documentation. That said, it would be a fairly subtle difference, and in many cases may go unnoticed in simple testing. There is a good probability that if you needed 10 samples out of 1000 entries, my code would appear to work correctly (I don't care to do the math to figure out the actual chance). Only on deeper testing or analysis of the results, would you notice that the probabilities are messed up. Most likely, my exact example would be fairly obvious, but imagine if the code were more complex, with the random.choice call inside a loop, or even another function. If random.choice were updated to use a reservoir sampling as a fallback, it also means that it is more likely for performance to degrade with some types. Giving a list (and maybe other sequences) would be O(1), but if a generator gets passed in, it falls back to O(n), and with lots of random.random calls that are generally naturally slow anyways. All that said, I would not be opposed to Python including a random.reservoir_choice (probably not the best name) function *in addition* to random.choice. The algorithm has its uses, but enough drawbacks and gotchas that it likely is not a good candidate for a fallback. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow random.choice, random.sample to work on iterators
On 2016-11-30 19:11, Chris Kaynor wrote: All that said, I would not be opposed to Python including a random.reservoir_choice (probably not the best name) function *in addition* to random.choice. The algorithm has its uses, but enough drawbacks and gotchas that it likely is not a good candidate for a fallback. I think this may be the path of least resistance for this. Even if this does imply one or two new functions in random, it may be better than changing random.choice. If these functions would be used enough by enough end users to justify this change is debatable, however. -- Bernardo Sulzbach http://www.mafagafogigante.org/ mafagafogiga...@gmail.com ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Allow random.choice, random.sample to work on iterators
On Wed, Nov 30, 2016 at 11:57:46AM -0800, Chris Kaynor wrote:
> Consider that this code will not produce the "correct" results (for a
> reasonable definition of correct):
>
> a = (i for i in range(100)) # Pretend this does something more
> interesting, and isn't a trivial generator - maybe a file object
> reading by line.
> randomEntries = [random.choice(a) for i in range(10)] # May not be
> quite as obvious, such as the choices could be chosen in a more
> complex loop.
>
> randomEntries may not contain 10 items (or the generation may error
> due to having insufficent items, depending on implementation),
Indeed. Given a iterator with 100 items, there's a 9% chance that the
first choice will be in the last nine items, which implies that failures
here will be common.
> it will
> not contain duplicates, and will be sorted.
Right. In other words, its a highly biased, non-random "random sample".
It may be worth adding a "reservoir sample" function, but I think it is
a mistake to try modifying the existing functions to deal with
iterators. Its too hard to get the same characteristics when sampling
from a sequence and an iterator. Better to keep them as separate
functions so that the caller knows what they're getting.
Here's my first attempt at implementing Algorithm R from Wikipedia:
https://en.wikipedia.org/wiki/Reservoir_sampling#Algorithm_R
from random import randrange, shuffle
import itertools
def reservoir_sample(iterable, count=1):
"""Return a list of count items sampled without replacement
from iterable, using reservoir sampling "Algorithm R".
This will exhaust the iterable, which must be finite.
"""
it = iter(iterable)
reservoir = list(itertools.islice(it, count))
if len(reservoir) < count:
raise ValueError('iterator is too short to sample %d items' % count)
shuffle(reservoir)
for i, value in enumerate(it, count+1):
j = randrange(0, i)
if j < count:
reservoir[j] = value
return reservoir
--
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
