[Python-ideas] Re: !$? operators for gratuitous overloading
On Feb 3, 2020, at 21:14, Random832 wrote: > > If we are going to have a general binary operator mechanism, maybe it should > be something more haskell-like, That’s exactly what I suggested—but more to argue that we don’t want a general binary operator mechanism at all. (The right way to do it would be like Haskell, we don’t want anything like whet Haskell does, therefore we shouldn’t do it.) > with an open-ended set of binary operator names [`identifier` as haskell, > perhaps? and/or maybe the unicode math ops section, with stuff like circled > plus] and a way to define precedence before the point of use for the parser. Well, in Haskell, it’s any _string_ of symbol characters, not just any one symbol character. And I think you’d want that even if you open it up to all of Unicode Sm. Not just because most of the non-ASCII characters are hard to type in most environments, but because for an awful lot of operations there’s no obvious single character. For example, what Unicode character would be as good as >> for rshift even to read, much less to type? How could you distinguish |<, <|, |>, >|, and the other map insert/update operators by using single characters for each? And so on. And operator strings don’t work in Python, where things like 2+-3 are perfectly valid sequences of two operators (and 2-=3 is valid and not even an operator). Also, without sectioning, it would be hard to refer to operators—Python has the operator module for all the builtin ones, and operator.add looks just as nice as (+), but that doesn’t help if you need a Python equivalent to (|<>) or a non-ASCII character. Even the backticked names thing, while I don’t think it has any syntactic ambiguity, I don’t think it would be readable in Python, because you don’t read Python in terms of curried functions. For example, in Haskell, if I write odd `dropwhile` xs, that makes sense, because you think of it as the partially applied function (dropwhile odd) being applied to xs, but in Python nobody would think of it as partial(dropwhile, odd)(xs). So it would at best look backward and at worst incomprehensible. But anyway, even if there weren’t any problems with either one, I think the odds of convincing the Python community to add it would be pretty close to nil. That being said, somewhere or other I have a hacky import hook that I could dig up if you want it that lets you do something like this. (Hacky not just in the way that it’s implemented, but also in what it does and doesn’t allow—basically an operator is any string of Sm characters that happens to generate a single ERRORTOKEN when tokenized as normal Python, IIRC.) I can’t imagine using it in real life code, but it was fun to implement. > Perhaps it would also be useful to provide a utility function for performing > the __op__/__rop__/NotImplemented/most-derived-class binary operator > execution rules on an arbitrary pair of method names, without *any* > syntactical support [in order to, say, allow evaluation of custom ASTs for > non-python languages on python objects] That part is not too hard to write, and doesn’t change from version to version, and if we don’t add custom operators (or encourage people to do so with import hooks) I think it will come up very rarely. So I think it’s probably not needed in the stdlib. (If you’ve got an idea for the opposite problem, though, a tool to make it easier to implement __op__ and __rop__ pairs on your types without having to go through the whole mess seen in fractions.Fraction, that would be pretty handy in the stdlib…) ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/BVXLH2Y3F77C2USGULGHK6DKZA2NLJSS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: addition of "nameof" operator
On Mon, Feb 3, 2020 at 3:58 AM Richard Damon
wrote:
>
> IF Python had a proper refactoring tool (see my other message where I
> question the ability to do that) then the nameof operator would clearly
> help keep logging/debug strings in line with the program definitions.
>
> A statement like print("bar", foo.bar) would be very hard to notice that
> the "bar" match the foo.bar, while
>
> print(nameof(foo.bar), foo.bar) makes the connection explicit.
>
What you're actually asking for is a way for a refactoring tool to
recognize that the string "bar" is a reference to the bar in foo.bar.
You're NOT asking for the Python compiler to recognize that. There are lots
of ways to do this. And Ricky Teachey's examples show that this is pretty
ugly.
On Mon, Feb 3, 2020 at 9:38 AM Ricky Teachey wrote:
> To restate the motivation: the hope is that there is a potentially large
> benefit of being able to more easily refactor code with something like a
> nameof().
>
>
>
> class FunctionLogger:
> # NOTE: this docstring will result in a runtime error, so can't even use
> nameof() here to help ourselves
> f"""Records use of a function.
> The optional `{nameof(FunctionLogger.log)}` parameter is to be a callable
> that accepts a string.
> It is logging.info by default.
> """
> *...*logging.exception(f"failed to log {func_name!r} call; is
> {cls_name}.{log_param_name} set correctly?")
>
> *Bottom line: refactoring remains a chore. The win is minimal. Do nothing
> option is preferred.*
>
> Here is a much more readable alternative which has the advantage that it
is already supported by Python. I'm going to show a harder example where I
want to reference foo.bar and allow both foo and bar to be refactored.
Here's the original proposal:
f"The field {nameof(foo)}.{nameof(foo.bar)} can be refactored."
Here's my alternative:
f"""The field {"foo.bar"} can be refactored."""
Since there's no real reason that you'd use {} around a literal string, we
have our refactoring tool recognize that syntax as a variable reference. It
has no net effect on the running code. If we don't want foo to be
refactored, we write instead:
f"""The field foo.{"bar"} can be refactored."""
And if we just want the string to reference "bar" but want to make sure
it's foo.bar not something_else.bar, we could write this:
f"""The field {"foo" and "bar"} can be refactored."""
--- Bruce
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/J2DZZLAOV72SS4XXJISX7KPZNJPYUZVE/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: !$? operators for gratuitous overloading
On Sun, Feb 2, 2020, at 11:41, MRAB wrote: > On 2020-02-02 15:00, Karl Ramm wrote: > > We have all shared in the observation that all but the most carefully > > considered operator overloading tends to reduce code readability. There > > also many programmers that feel that they need it to make there code terse > > (or perhaps shiny) enough. > > > > I propose adding ! and $ as (normally unimplemented) binary operators and ? > > as a unary operator so as to explicitly to give people the rope they want > > but clearly tagged "here there be shenanigans". > > > In the past there have been suggestions to add None-coalescing > operators > such as "?." and "??", comparable to the null-coalescing operators of > C#, so I'm -1 on just adding "!", "$" and "?" without a solid use-case, > in case we find a better use for them later in the future. Before the nameof discussion got shut down, I was going to propose $nameof as a way to avoid collision with a function called "nameof", and I still think it's viable as a general way to make an open-ended set of keywords that don't collide with identifiers [frozenset literals?]... so I'm also -1 on cutting down the number of reserved ASCII characters without a very good concrete use case. If we are going to have a general binary operator mechanism, maybe it should be something more haskell-like, with an open-ended set of binary operator names [`identifier` as haskell, perhaps? and/or maybe the unicode math ops section, with stuff like circled plus] and a way to define precedence before the point of use for the parser. Perhaps it would also be useful to provide a utility function for performing the __op__/__rop__/NotImplemented/most-derived-class binary operator execution rules on an arbitrary pair of method names, without *any* syntactical support [in order to, say, allow evaluation of custom ASTs for non-python languages on python objects] ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GISGMZLBZ3FRT77QWJ3EXUGJTGJC2MEG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: allow full expressions in decorators
I’ve always resisted changing this, but it keeps coming up, and in other
cases we don’t restrict the grammar (except when there are real
ambiguities). So maybe the SC can accept a PRP for this?
On Mon, Feb 3, 2020 at 15:47 Ben Avrahami wrote:
> Hi all, decorators are a very powerful feature in python, but it's syntax
> is strangely restrictive.
>
> decorator ::= "@" dotted_name ["(" [argument_list [","]] ")"] NEWLINE
>
> for 99% of cases this is not a hindrance, but sometimes we'd rather be
> able to use full expression syntax. consider the following example (using
> PyQt):
>
> # make 10 buttons
> buttons = [QPushButton(f'button #{i}') for i in range(10)]
> # for now we only want to bind an event to the first button
> @buttons[0].clicked.connect
> def foo():
> ... # do whatever
>
> in cases such as this, it would be useful to have any expression
> acceptable as a decorator.
>
> In addition to other use cases such as chained function calling
> (@buttons_dict.get("foo"). clicked.connect), the restrictiveness of the
> decorator syntax seems like a strange oversight. Arbitrary expressions are
> already allowed in decorator lines as arguments, so allowing them as the
> called decorator shouldn't be too difficult.
>
> Any thoughts on this?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/WOWD4P323DYDIGUQVWMESDWUG6QOW4MP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
--Guido (mobile)
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/FKE7ZFGUDCU5WVOE2QTD5XGMCNCOMETV/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] allow full expressions in decorators
Hi all, decorators are a very powerful feature in python, but it's syntax
is strangely restrictive.
decorator ::= "@" dotted_name ["(" [argument_list [","]] ")"] NEWLINE
for 99% of cases this is not a hindrance, but sometimes we'd rather be able
to use full expression syntax. consider the following example (using PyQt):
# make 10 buttons
buttons = [QPushButton(f'button #{i}') for i in range(10)]
# for now we only want to bind an event to the first button
@buttons[0].clicked.connect
def foo():
... # do whatever
in cases such as this, it would be useful to have any expression acceptable
as a decorator.
In addition to other use cases such as chained function calling
(@buttons_dict.get("foo"). clicked.connect), the restrictiveness of the
decorator syntax seems like a strange oversight. Arbitrary expressions are
already allowed in decorator lines as arguments, so allowing them as the
called decorator shouldn't be too difficult.
Any thoughts on this?
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/WOWD4P323DYDIGUQVWMESDWUG6QOW4MP/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
Sincerely I would have to agree that it's seems a bit excessive the `cancel_on_error`, unless it enabled by default and implemented in the abstract class it should probably not be included, that was just an idea to keep backwards compatibility. I will personally simply add subclass of my prefered executor with the following for my particular use case: def __exit__(self, exc_type, exc_val, exc_tb): self.shutdown(wait=True, cancel_futures=exc_val is not None) return False Again, I think aborting futures on uncaught exceptions make sense, but it would break backwards compatibility. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/3XDUZPZS3VCGX2PJMZHILI2L65V7BEGS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: addition of "nameof" operator
On Feb 3, 2020, at 10:25, Andrew Barnert wrote: > > This is the same as Smalltalk, ObjC, Ruby, f-script, and all of the other > languages that use the same dynamic type/data model as Python. And Smalltalk > was the first language to have practical refactoring tools (although I don’t > think they were called that until Ralph Johnson’s refactoring object browser > in the early 90s). > > One difference between Python and most of the other Smalltalk-model languages > is that they use symbols or selectors or some other string-like type to name > methods, while Python uses plain old strings. If you have print(:bar, > foo.bar) obviously bar is the name of an attribute rather than normal text. > Dropping selectors from the language turns out to cause a lot fewer problems > than any Smalltalk devotee would have believed, just a few very small > problems—like the one nameof is trying to fix. > > Meanwhile, unlike Smalltalk and most other languages with the same model, > Python has optional static typing. The other major Smalltalk-model language > that added optional static types is ObjectiveC—and one of the reasons was to > improve the navigation and refactoring tools. For example, if you have a > function that takes a `foo: Any` param (spelled `id foo` in ObjC) and there > are three unrelated types that all have a `bar` method, the tool will tell > you that it’s ambiguous which one you meant by `nameof bar` (spelled > `@sel(bar)`), and it can suggest that you either specify a type for `foo`, or > create an implicit ABC (a `@protocol` in ObjC terms) to own the method. If > you do the first, renaming Foo.bar to baz will change the selector; if you do > the latter, it will warn you that Foo.bar makes Foo conform to the Barable > protocol that you defined and there are selectors in your program that depend > on it being Barable, so do you want to break that or do you want to rename > Barable.bar and all of the classes that implement it instead? If you do > neither, it doesn’t guess, it tells you that there are two ambiguous uses of > the selectors and lets you see the two uses of the selector or the three > classes that define it, so you can proceed manually. > > Of course this means that an IDE, object browser, or standalone refactoring > tool has to build a mapping from selector names to a list of implicit uses, > explicit uses, and definitions (including knowing which definitions are > inherited, and which are relied on by protocols) for the whole program. But > that’s exactly what they do. That’s why RJBrowser takes a while to start up, > and why Xcode is constantly running clang on bits of your code in the > background as you edit. > > Couldn’t a Python tool just also keep track of every string and substring > that has at least one match as a use or definition of an attribute? Yes. But > I think it’s obvious why this is a lot more work and at least a little less > useful. And that may be part of the reason Python has always had fewer > refactoring tools available than Smalltalk, ObjC, etc., which is why nameof > could be a useful addition. All that being said, I’m not sure _how_ useful it would be in practice. Python has a chicken-and-egg problem that C#, ObjC, etc. don’t. If the C#/ObjC devs add a new feature to their language, the Visual Studio/Xcode team next door adds a nameof-driven renaming feature to the refactoring tools in the IDE that 90% of C#/ObjC programmers use, and then then everyone who wants that feature starts using nameof. If the Python devs add nameof, PyCharm, PyDev, etc. may not take advantage of it, and they may already have an often-good-enough heuristic-guessing feature for people who don’t use it. If not many people start using it, there will be even less motivation for PyCharm to take advantage of it, etc. If a new feature has an obvious large benefit (and/or is fun enough to implement), like optional static typing, you can jump past that hurdle. If a new feature has other uses even without IDE support, like @dataclass, you can spiral over it. But nameof is neither; it might just be a dead-on-arrival feature that’s only ever used by a handful of large teams who build their own internal tools. (And that large internal-tool-building team could just define a builtin `def nameof(x: str): return x` to use while debugging and write an install-time or import-time transformer that just replaces every `nameof(x)` with `x` to eliminate the deployed runtime cost.) Add in the fact that most people use the newest version of C# or ObjC for any new project, while Python developers often stay a version or 3 behind, and a breaking change like this would probably need a __future__ and/or deprecation schedule on top of that, and the problem looks even worse. So, someone who wants this feature has to not only resolve all the confusion over what exactly is being proposed and how it helps, but also make the case that Python tools and Python
[Python-ideas] Re: addition of "nameof" operator
On Feb 3, 2020, at 03:59, Richard Damon wrote:
>
> On 2/3/20 12:02 AM, Bruce Leban wrote:
>> People keep mentioning refactoring. It's a red herring.
>>
>> No refactoring tool needs a nameof operator added to the language in order
>> to do its job. And certainly an operator that requires running the program
>> in order to use it is not going to be helpful. Refactoring tools analyze the
>> program; they don't run it.
>>
>> And f-strings are another red herring. There is nothing magical about
>> expressions inside f-strings compared to those outside.
>>
>> Ignoring the red herrings, I see no utility in a nameof operator and no way
>> for it to actually do anything useful.
>>
>> --- Bruce
>
> IF Python had a proper refactoring tool (see my other message where I
> question the ability to do that) then the nameof operator would clearly help
> keep logging/debug strings in line with the program definitions.
>
> A statement like print("bar", foo.bar) would be very hard to notice that the
> "bar" match the foo.bar, while
>
> print(nameof(foo.bar), foo.bar) makes the connection explicit.
>
> This is where such an operation could help with refactoring, having an actual
> Python attribute reference that the refactoring operation could see, so it
> knows that the string is based on that attribute and not something else.
>
> As I mentioned elsewhere, the issue is that in Python, if we change the
> attribute bar in class Foo, then we have the problem that we often can't tell
> if in foo.bar if foo is of type Foo, so we should apply the change. This
> isn't a problem is statically typed languages, as there we can determine the
> type of foo, and see if the change applies (at least in most cases).
This is the same as Smalltalk, ObjC, Ruby, f-script, and all of the other
languages that use the same dynamic type/data model as Python. And Smalltalk
was the first language to have practical refactoring tools (although I don’t
think they were called that until Ralph Johnson’s refactoring object browser in
the early 90s).
One difference between Python and most of the other Smalltalk-model languages
is that they use symbols or selectors or some other string-like type to name
methods, while Python uses plain old strings. If you have print(:bar, foo.bar)
obviously bar is the name of an attribute rather than normal text. Dropping
selectors from the language turns out to cause a lot fewer problems than any
Smalltalk devotee would have believed, just a few very small problems—like the
one nameof is trying to fix.
Meanwhile, unlike Smalltalk and most other languages with the same model,
Python has optional static typing. The other major Smalltalk-model language
that added optional static types is ObjectiveC—and one of the reasons was to
improve the navigation and refactoring tools. For example, if you have a
function that takes a `foo: Any` param (spelled `id foo` in ObjC) and there are
three unrelated types that all have a `bar` method, the tool will tell you that
it’s ambiguous which one you meant by `nameof bar` (spelled `@sel(bar)`), and
it can suggest that you either specify a type for `foo`, or create an implicit
ABC (a `@protocol` in ObjC terms) to own the method. If you do the first,
renaming Foo.bar to baz will change the selector; if you do the latter, it will
warn you that Foo.bar makes Foo conform to the Barable protocol that you
defined and there are selectors in your program that depend on it being
Barable, so do you want to break that or do you want to rename Barable.bar and
all of the classes that implement it instead? If you do neither, it doesn’t
guess, it tells you that there are two ambiguous uses of the selectors and lets
you see the two uses of the selector or the three classes that define it, so
you can proceed manually.
Of course this means that an IDE, object browser, or standalone refactoring
tool has to build a mapping from selector names to a list of implicit uses,
explicit uses, and definitions (including knowing which definitions are
inherited, and which are relied on by protocols) for the whole program. But
that’s exactly what they do. That’s why RJBrowser takes a while to start up,
and why Xcode is constantly running clang on bits of your code in the
background as you edit.
Couldn’t a Python tool just also keep track of every string and substring that
has at least one match as a use or definition of an attribute? Yes. But I think
it’s obvious why this is a lot more work and at least a little less useful. And
that may be part of the reason Python has always had fewer refactoring tools
available than Smalltalk, ObjC, etc., which is why nameof could be a useful
addition.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/a
[Python-ideas] Re: addition of "nameof" operator
To restate the motivation: the hope is that there is a potentially large
benefit of being able to more easily refactor code with something like a
nameof().
I am going to make the claim that:
1. this benefit is actually minimal and does not address a lot of other
existing refactoring problems/chores, and
2. comes at the cost of a possible loss of readability, and therefore
isn't worth the trade
Consider the following typical code (I hope it is nice; I feel like I can
certainly read it):
import logging
import functools
class FunctionLogger:
"""Records use of a function.
The optional `log` parameter is to be a callable that accepts a string.
It is logging.info by default.
"""
log = logging.info
def __init__(self, log=None):
if log is not None:
self.log = log
def __call__(self, func):func_name = func.__name__
@functools.wraps(func)def wrapper(*args, **kwargs):
try:self.log(f"called {func_name!r}")
except:cls_name = type(self).__name__
logging.exception(f"failed to log {func_name!r} call; is
{cls_name}.log set correctly?")finally:
return func(*args, **kwargs)return wrapper
Let's think through: what refactoring must occur to change the "log"
parameter to something else? And, how readable does the code remain
afterward?
At least some of the code could certainly benefit from being more
easily refactored (by an IDE) with nameof(), but how much?
import logging
import functools
class FunctionLogger:
# NOTE: this docstring will result in a runtime error, so can't
even use nameof() here to help ourselves
f"""Records use of a function.
The optional `{nameof(FunctionLogger.log)}` parameter is to be a
callable that accepts a string.
It is logging.info by default.
"""
log = logging.info # no need for nameof() here, but see below *
def __init__(self, log=None): # IDE will refactor the init
signature SEPARATELY (see below *)
if log is not None: # no help from nameof() in __init__ body
# IDE will get this one just fine anyway; shouldn't use
nameof() in this situation
# using it actually hurts readability
setattr(self, nameof(self.log), log)
def __call__(self, func):
func_name = nameof(func) # it's different, but NOT an
improvement over func.__name__
@functools.wraps(func)
def wrapper(*args, **kwargs):
try:
self.log(f"called {func_name!r}")
except:
cls_name = nameof(type(self)) # different, but NOT an
improvement over type(self).__name__
log_param_name = nameof(self.log) # ok, HERE is a
place we are definitely going to benefit from nameof()
# readability of next line might actually be slightly
improved...? But perhaps it's also worse...?
# but the IDE will refactor next line automatically,
which is handy.logging.exception(f"failed to log
{func_name!r} call; is {cls_name}.{log_param_name} set correctly?")
finally:return func(*args, **kwargs)
return wrapper
* For the class level member and init signature/body: the IDE won't know to
include the class member and the init signature/body in a refactor of a
member-level variable, and this problem isn't helped by nameof(). Multiple
refactoring chores have to be completed to change the parameter name: 1.
the init signature, 2. the class level variable, 3. object level variable,
4. the class docstring, 5. all of the f strings. Using a nameof() only
totally prevents one of them (the f strings; it can't help with the
docstring).
So I guess there is SOME benefit here. But I think it comes at the cost of
some potential loss of readability, and there are some major places in
which nameof() doesn't bring any help at all (looking at you, class
docstring).
And there is a ready alternative: use a global if you think you might
rename your parameters later (see code at bottom). This rolls TWO of the
refactoring chores into one (the f strings and the docstring).
*Bottom line: refactoring remains a chore. The win is minimal. Do nothing
option is preferred.*
import logging
import functools
_FOO = "foo" # haven't decided on .foo member name yet...
class FunctionLogger:
# with a global, this docstring will work!
f"""Records use of a function.
The optional `{_FOO}` parameter is to be a callable that accepts a string.
It is logging.info by default.
"""
log = logging.info
def __init__(self, log=None):
if log is not None:
self.log = log
def __call__(self, func):
func_name = func.__name__
@functools.wraps(func)
def wrapper(*args, **kwargs):
try:
self.log(f"called {func_name!r}")
except:
cls_name = type(self).__name__
# employ the global below for the e
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
Thanks for all your hard work on the `cancel_futures` feature! As you said, there is a complexity cost (both in terms of the API and the implementation) whenever a new feature is added. The current ProcessPoolExecutor implementation, in particular, is complex enough that I can't easily easily reason about it's behavior and there are some long-standing bugs that have been difficult to resolve. I was actually hoping that we wouldn't touch the API for a while so I could do some stability work. I'm -0 on the `cancel_on_error` feature but maybe it would help to hear about a concrete use case that can't be easily accomplished using the current API? Cheers, Brian On Mon, Feb 3, 2020 at 12:36 AM Kyle Stanley wrote: > Miguel Ángel Prosper wrote: > > Thank you so much for the work, I was very confused on how to even start > implementing it in the ProcessPoolExecutor, but you finished everything > super quick! > > No problem! The ProcessPoolExecutor implementation wasn't immediately > clear to me either, but after some experimentation (and some great > suggestions from Brian Quinlain to improve it) I was able to get it > functioning as intended. Thank you for the proposal, I think it will be a > good addition for the Executor API. > > Miguel Ángel Prosper wrote: > > I'm suppose that this might be better in another thread but... Could it > be possible to changing the context manager of the executor to cancel > futures on uncaught exceptions? > > Hmm, it should be possible. Do you specifically mean cancelling the > pending futures once a single one of the submitted functions raises an > exception, or cancelling the pending futures when the Executor itself > raises an exception (I.E. BrokenProcessPool)? I would assume the prior, > since that seems more useful to me. > > Miguel Ángel Prosper wrote: > > If not I could also be added as an optional parameter to the executor > constructor, "cancel_on_error" set to False to not changing anything. > > Assuming we did implement something like this, I'd say it should be done > as an optional parameter for the constructor (as stated above) rather than > as the default option when an exception occurs, because it would > significantly different from the current behavior. At the moment, the > executor doesn't mind when exceptions are raised from submitted functions; > it simply sets the exception to the future (via ``future.set_exception()``) > and moves on to the next. > > So, cancelling the pending futures after one raises an exception would be > a significant behavioral change; one that could easily result in breaking > existing code. As a result, I think it should be an optional parameter if > anything. > > As far as whether or not it should be implemented, I'm not 100% certain. > While I'm not against the idea of it, I think it would be reasonable to > wait until the new parameter is released in Python 3.9 and see what users > think of it prior to adding a similar parameter to the executor constructor > as well. The cost of adding a new feature demanded by users is relatively > minimal, but once it's added we're stuck with maintaining it. > > If the idea is approved, I'd be happy to help with implementing it. I just > want to make sure that we actually want it in the first place. > > What do you think, Guido? I've also CC'd Brian Quinlain and Antoine > Pitrou, in case they have any feedback but didn't see the original thread. > > On Mon, Feb 3, 2020 at 2:27 AM Miguel Ángel Prosper < > miguelangel.pros...@gmail.com> wrote: > >> Thank you so much for the work, I was very confused on how to even start >> implementing it in the ProcessPoolExecutor, but you finished everything >> super quick! >> >> I'm suppose that this might be better in another thread but... Could it >> be possible to changing the context manager of the executor to cancel >> futures on uncaught exceptions? If not I could also be added as an optional >> parameter to the executor constructor, "cancel_on_error" set to False to >> not changing anything. Personally I was thinking of using the executor like >> that; currently I have to either place everything in a try except block, >> handle it and then reraise it or subclass each executor. >> ___ >> Python-ideas mailing list -- python-ideas@python.org >> To unsubscribe send an email to python-ideas-le...@python.org >> https://mail.python.org/mailman3/lists/python-ideas.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-ideas@python.org/message/R5VMWP75K35KF3DJN2F5MBZXD5UOQIV3/ >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5OK7BUEUACAHB7ZYHOPTXODVWCZWS5YB/ Code
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
> But, it would potentially risk adding an underutilized parameter to the > executor constructor (which contributes to feature bloat). That's true, personally I would always enable cancel_on_error (making it redundant and implementing it in the abstract class), but that's just my use case. You can always add that functionality later with your own subclass. Personally I think it make sense to do it by default but that's just me and it also changes current behaviour. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/MMS3OGNA6XNKT63DYWVB4DZFHESLKOXM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Annotated string literals
Why don't you define a LOCALIZE function (x → x) like in your C example?
Le lun. 3 févr. 2020 à 13:01, Soni L. a écrit :
>
>
>
> On 2020-02-02 11:29 p.m., Eric V. Smith wrote:
> > On 2/2/2020 8:28 PM, Soni L. wrote:
> >> It'd be cool to attach metadata to string literals that doesn't end
> >> up in the resulting string object. This metadata could be used by all
> >> sorts of tools, everything from localization to refactoring.
> >>
> >> In C, some localization libraries use macros for it, such as:
> >>
> >> #define LOCALIZE(s) s
> >>
> >> and it acts like an annotation, and then you do:
> >>
> >> printf("%s", get_localized(locale, LOCALIZE("This string gets
> >> extracted and used in language files")));
> >>
> >> And I think Python could do better and have special string literals
> >> explicitly for this purpose.
> >>
> >> It could even be something with fstrings, like: f"{#localize}This
> >> string gets extracted and used in language files"
> >>
> >> Ofc, there's nothing preventing one from using something like
> >> f"{(lambda _:'')('localize')}This string gets extracted and used in
> >> language files" today, but I feel like having some sort of dedicated
> >> syntax for it would be an improvement.
> >
> > You might want to look at PEP 501.
> >
> > Eric
>
> Oh. No, I want it to return plain strings.
>
> e.g. you should be able to replace an existing
>
> MY_STRING = "This string gets extracted and used in language files"
>
> with
>
> MY_STRING = [uhh idk what would go here tbh]
>
> and maintain full backwards-compatibility.
>
> It's for external tools to consume, not for python code to consume.
>
> >
> > ___
> > Python-ideas mailing list -- python-ideas@python.org
> > To unsubscribe send an email to python-ideas-le...@python.org
> > https://mail.python.org/mailman3/lists/python-ideas.python.org/
> > Message archived at
> > https://mail.python.org/archives/list/python-ideas@python.org/message/GEBY52WVQBFB7J27MRJPZZPJMWUZZJWO/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/HY6N3CA2FEJL5EGY5QWJHBLP5KVETTD3/
> Code of Conduct: http://python.org/psf/codeofconduct/
--
Antoine Rozo
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/DGSVOSOXX4RS4M2JA4NBE5LGKUMJDCE2/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Annotated string literals
On 2020-02-02 11:29 p.m., Eric V. Smith wrote:
On 2/2/2020 8:28 PM, Soni L. wrote:
It'd be cool to attach metadata to string literals that doesn't end
up in the resulting string object. This metadata could be used by all
sorts of tools, everything from localization to refactoring.
In C, some localization libraries use macros for it, such as:
#define LOCALIZE(s) s
and it acts like an annotation, and then you do:
printf("%s", get_localized(locale, LOCALIZE("This string gets
extracted and used in language files")));
And I think Python could do better and have special string literals
explicitly for this purpose.
It could even be something with fstrings, like: f"{#localize}This
string gets extracted and used in language files"
Ofc, there's nothing preventing one from using something like
f"{(lambda _:'')('localize')}This string gets extracted and used in
language files" today, but I feel like having some sort of dedicated
syntax for it would be an improvement.
You might want to look at PEP 501.
Eric
Oh. No, I want it to return plain strings.
e.g. you should be able to replace an existing
MY_STRING = "This string gets extracted and used in language files"
with
MY_STRING = [uhh idk what would go here tbh]
and maintain full backwards-compatibility.
It's for external tools to consume, not for python code to consume.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/GEBY52WVQBFB7J27MRJPZZPJMWUZZJWO/
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/HY6N3CA2FEJL5EGY5QWJHBLP5KVETTD3/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: addition of "nameof" operator
On 2/3/20 12:02 AM, Bruce Leban wrote:
People keep mentioning refactoring. It's a red herring.
No refactoring tool needs a nameof operator added to the language in
order to do its job. And certainly an operator that requires running
the program in order to use it is not going to be helpful. Refactoring
tools analyze the program; they don't run it.
And f-strings are another red herring. There is nothing magical about
expressions inside f-strings compared to those outside.
Ignoring the red herrings, I see no utility in a nameof operator and
no way for it to actually do anything useful.
--- Bruce
IF Python had a proper refactoring tool (see my other message where I
question the ability to do that) then the nameof operator would clearly
help keep logging/debug strings in line with the program definitions.
A statement like print("bar", foo.bar) would be very hard to notice that
the "bar" match the foo.bar, while
print(nameof(foo.bar), foo.bar) makes the connection explicit.
This is where such an operation could help with refactoring, having an
actual Python attribute reference that the refactoring operation could
see, so it knows that the string is based on that attribute and not
something else.
As I mentioned elsewhere, the issue is that in Python, if we change the
attribute bar in class Foo, then we have the problem that we often can't
tell if in foo.bar if foo is of type Foo, so we should apply the change.
This isn't a problem is statically typed languages, as there we can
determine the type of foo, and see if the change applies (at least in
most cases).
--
Richard Damon
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/COJJ4EG4B5J2DKXZIZSQV7D2USZ3QGCS/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
> Then if Executor.__exit__ detects an exception it would call shutdown with cancel_futures set to True. Oh, I see. That should be rather simple: ``` def __exit__(self, exc_type, exc_val, exc_tb): if exc_val is not None and self._cancel_on_error: self.shutdown(wait=True, cancel_futures=True) else: self.shutdown(wait=True) return False ``` (I believe the above would have to be specified as an override for each of ThreadPoolExecutor and ProcessPoolExecutor if *cancel_on_error* were supplied as a parameter to the executor constructor. They both currently use the ``__exit__`` specified in the abstract Executor class, which wouldn't have access to *cancel_on_error*.) My previous consideration about waiting for user input of *cancel_futures *after the release of 3.9*, *before adding *cancel_on_error* to the constructor still applies to some degree though. If it directly uses executor.shutdown() with cancel_futures set to True, the maintenance would be rather minimal. But, it would potentially risk adding an underutilized parameter to the executor constructor (which contributes to feature bloat). On Mon, Feb 3, 2020 at 4:03 AM Miguel Ángel Prosper < miguelangel.pros...@gmail.com> wrote: > > Hmm, it should be possible. Do you specifically mean cancelling the > pending > > futures once a single one of the submitted functions raises an exception, > > or cancelling the pending futures when the Executor itself raises an > > exception (I.E. BrokenProcessPool)? I would assume the prior, since that > > seems more useful to me. > > I was referring to any exception within the context manager of the > executor, not the futures or just the executor's exceptions, something such > as the following: > > with ThreadPoolExecutor(cancel_on_error=True) as executor: > # Code here > raise Exception > > Then if Executor.__exit__ detects an exception it would call shutdown > with cancel_futures set to True. > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/OAMG4JUFK54FTAASHQG5WNAA7BU7KRS7/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GCKG347LGTROINXHYUVTN3IN4QTLGYRK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
> Hmm, it should be possible. Do you specifically mean cancelling the pending > futures once a single one of the submitted functions raises an exception, > or cancelling the pending futures when the Executor itself raises an > exception (I.E. BrokenProcessPool)? I would assume the prior, since that > seems more useful to me. I was referring to any exception within the context manager of the executor, not the futures or just the executor's exceptions, something such as the following: with ThreadPoolExecutor(cancel_on_error=True) as executor: # Code here raise Exception Then if Executor.__exit__ detects an exception it would call shutdown with cancel_futures set to True. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/OAMG4JUFK54FTAASHQG5WNAA7BU7KRS7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: concurrent.futures cancel pending at Executor.shutdown
Miguel Ángel Prosper wrote: > Thank you so much for the work, I was very confused on how to even start implementing it in the ProcessPoolExecutor, but you finished everything super quick! No problem! The ProcessPoolExecutor implementation wasn't immediately clear to me either, but after some experimentation (and some great suggestions from Brian Quinlain to improve it) I was able to get it functioning as intended. Thank you for the proposal, I think it will be a good addition for the Executor API. Miguel Ángel Prosper wrote: > I'm suppose that this might be better in another thread but... Could it be possible to changing the context manager of the executor to cancel futures on uncaught exceptions? Hmm, it should be possible. Do you specifically mean cancelling the pending futures once a single one of the submitted functions raises an exception, or cancelling the pending futures when the Executor itself raises an exception (I.E. BrokenProcessPool)? I would assume the prior, since that seems more useful to me. Miguel Ángel Prosper wrote: > If not I could also be added as an optional parameter to the executor constructor, "cancel_on_error" set to False to not changing anything. Assuming we did implement something like this, I'd say it should be done as an optional parameter for the constructor (as stated above) rather than as the default option when an exception occurs, because it would significantly different from the current behavior. At the moment, the executor doesn't mind when exceptions are raised from submitted functions; it simply sets the exception to the future (via ``future.set_exception()``) and moves on to the next. So, cancelling the pending futures after one raises an exception would be a significant behavioral change; one that could easily result in breaking existing code. As a result, I think it should be an optional parameter if anything. As far as whether or not it should be implemented, I'm not 100% certain. While I'm not against the idea of it, I think it would be reasonable to wait until the new parameter is released in Python 3.9 and see what users think of it prior to adding a similar parameter to the executor constructor as well. The cost of adding a new feature demanded by users is relatively minimal, but once it's added we're stuck with maintaining it. If the idea is approved, I'd be happy to help with implementing it. I just want to make sure that we actually want it in the first place. What do you think, Guido? I've also CC'd Brian Quinlain and Antoine Pitrou, in case they have any feedback but didn't see the original thread. On Mon, Feb 3, 2020 at 2:27 AM Miguel Ángel Prosper < miguelangel.pros...@gmail.com> wrote: > Thank you so much for the work, I was very confused on how to even start > implementing it in the ProcessPoolExecutor, but you finished everything > super quick! > > I'm suppose that this might be better in another thread but... Could it be > possible to changing the context manager of the executor to cancel futures > on uncaught exceptions? If not I could also be added as an optional > parameter to the executor constructor, "cancel_on_error" set to False to > not changing anything. Personally I was thinking of using the executor like > that; currently I have to either place everything in a try except block, > handle it and then reraise it or subclass each executor. > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/R5VMWP75K35KF3DJN2F5MBZXD5UOQIV3/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/PMIV2XIYGHULEBZVDG2Q5PBQJJXYB2LD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] concurrent.futures deinitializer for Executor
Hi, I thought maybe a callback before the worker of a Executor finishes would be helpful. It would allow the setup and teardown resources (such as file, databases, connections, ...) across workers. My particular use case is for closing down requests.Session objects, that currently its very complicated to cleanup properly. There are many other use cases, a generic format would be something like the following: import threading from concurrent.futures import ThreadPoolExecutor worker_local = threading.local() def session_initializer(): # Initialize costly resources for a particular worker worker_local.resources = ... def session_deinitializer(): # Deinitialize worker resources carefully worker_local.resources ... with ThreadPoolExecutor(initializer=session_initializer, deinitializer=session_deinitializer) as executor: executor.submit(task) ... ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/BOKCVLL4B7COBVXFS5BPTTFTRFU3JKJD/ Code of Conduct: http://python.org/psf/codeofconduct/
