[Python-ideas] Re: Error handling toggle / levels

[Ilya Kulakov]

Wow, I have never realized that except’s expression is a full featured 
expression!

Best Regards
Ilya Kulakov

> On Sep 26, 2019, at 7:00 AM, Serhiy Storchaka  wrote:
> 
> 26.09.19 12:07, salemalbr...@gmail.com пише:
>> So when coding, at deferent stages we need different levels of error 
>> handling. For example at stage of finding the concept, initial 
>> implementation, alpha, releasing testing, etc.
>> By some bits of code we can simulate enable/disable error handling.
>> But it would be very helpful to have such thing built in the language with 
>> some more pythonic salt 😁.
>> For example,
>> Assume this behavior..
>> 
>> SetExceptionLevel(50)
>> try:
>> x = 1 / 0
>> except(0) Exception as e:
>> print(e)
>> except (40) ZeroDivisionError as e1:
>> x = math.inf
>> except (40) ValueError as e2:
>> x = None
>> except(60) Exception as e3:
>> raise e3
>> 
>> i.e. one exception can have more than one handler and each handler has a 
>> level to enable it,
>> so now for this example the first 2 exception will run only,
>> If we want the last one to run also we need to change the first line to  
>> "SetExceptionLevel(70)" for example.
>> The third one " ValueError " will never be triggered.
>> And if no exception met the requirements "level and exception type" then the 
>> error will be raised out.
> 
> It is not clear what your propose syntax means, but you can use e.g. such 
> code:
> 
>   except (ZeroDivisionError if level > 40 or ()) as e1:
>   x = math.inf
> 
> If level > 0, you will catch ZeroDivisionError and set x = math.inf.
> If level <= 40 the ZeroDivisionError exception will pass to other handlers.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/K57OE77UM4ZSHP4E2L5FEKRRBNALCCGR/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GQ6PEX5V6QFAO5ZTG3Q4RIRPYLUXK3B4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a time-based synchronization primitive

[Ilya Kulakov]

Complement to the proposed idea.

Some time ago I thought how cool it would be if one could implement a custom 
scheduler for Python's asyncio that used runtime metrics (defined by the user) 
to dynamically adjust priorities (with respect to the defined execution order).
E.g. if the developer prioritizes database related operations over maintenance 
and current application load demands a lot of the former, the scheduler will 
proactively delay maintenance.
In code it may look like decorator annotations assigning tasks to groups.

> On Sep 25, 2019, at 2:02 PM, Viktor Roytman  wrote:
> 
> The asyncio and threading modules include a number of synchronization 
> primitives. In particular, a Semaphore allows you to limit the number of 
> concurrent tasks if you need to stay under some capacity constraint.
> 
> However, none of the existing primitives provide for rate limiting, as in 
> making sure there are no more than n tasks started per second.
> 
> I believe this Stack Overflow question shows that adding such a primitive 
> would be useful: 
> https://stackoverflow.com/questions/35196974/aiohttp-set-maximum-number-of-requests-per-second
>  The asker clearly wants rate limiting, but the answers provided limit 
> concurrency instead.
> 
> I found an excellent answer by Martijn Pieters, which includes an 
> implementation of the leaky bucket algorithm, here 
> https://stackoverflow.com/a/45502319/1475412 The AsyncLeakyBucket is used in 
> exactly the same way as a Semaphore.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/45ZKM62GS5EINE75KNOLFTJ3AHOYVHRZ/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/IMQXORSBERIDRTIX7CTB5KQ5AHDLH74W/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Deprecation utilities for the warnings module

[Ilya Kulakov]

After spending more time thinking about the implementation I came to a 
conclusion that
it's not easy to generalize replacement of classes.
Yes, with some work it's possible to ensure that old name references a new one.
But that's not sufficient. If new class has different interface then user's 
code will fail.

Library's author could provide a mapping via a dict or by manually implementing 
methods
but that'd be only half of the problem. User's code would still pass "old" 
objects back to the library.
So effectively the library would have to support both old and new making all 
work meaningless.

I think it's possible to design a smart wrapper that behaves as an old class
in user's code, but as a new class inside library. E.g. by analyzing 
traceback.extract_stack.
But that is too clever and almost certainly comes with its own bag of flaws.

I'm confident that this problem leaves the "obsolete" decorators out of scope 
of this enhancement.

Would anyone be interested in using plain deprecation decorators? They are 
still quite useful:
- Can warn when class is subclassed or class-level attributes are accessed (vs 
warn inside __init__/__new__)
- Much easier to be seen by static analyzers


Best Regards,
Ilya Kulakov

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Deprecation utilities for the warnings module

[Ilya Kulakov]

>> Therefore it's not redundant to subclass *Warning for namespacing alone.
> 
> Not redundant? You mean you must subclass? In that case my concern stands. 


An unfortunate typo, meant "it's redundant". 

> And what does that match against? The module name of the exception type right?

It matches agains a location where warn is called after taking stacklevel into 
account.

Consider the following example:

test.py:
import warnings
warnings.warn("test")
warnings.warn("__main__ from test", stacklevel=2)

$ python -c "import warnings, test; warnings.warn('__main__')"

test.py:2: UserWarning: test
  warnings.warn("test")
-c:1: UserWarning: __main__ from test
-c:1: UserWarning: __main__

$ python -W "ignore:::test:" -c "import warnings, test; 
warnings.warn('__main__')"

-c:1: UserWarning: __main__ from test
-c:1: UserWarning: __main__

$ python -W "ignore:::__main__:" -c "import warnings, test; 
warnings.warn('__main__')"

test.py:2: UserWarning: test
  warnings.warn("test")

End-user can distinguish warnings of the same category by specifying their 
origin (where warning is issued in runtime).


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Deprecation utilities for the warnings module

[Ilya Kulakov]

Hi Anders,

If correctly understood your concern, it's about usage of stdlib's *Warning 
classes directly
that makes all warnings coming from different libraries indistinguishable.

I think that's not the case, since warnings.filterwarnings allows
to specify custom filter using a regular expression to match module names.
Therefore it's not redundant to subclass *Warning for namespacing alone.

> On Sep 13, 2018, at 11:07 PM, Anders Hovmöller  wrote:
> 
> 
>> I'd like to propose an extension for the warnings module
>> to address this problem.
> 
> I like all of that. The only issue I have with it is that the warnings module 
> is designed to namespace depredations so you can turn them on per library and 
> this code doesn’t seem to handle that. We really want to avoid libraries 
> using these convenience functions instead of creating their own warning that 
> can be properly filtered. 
> 
> I’m not sure what the solution to this would be. Maybe just accessing 
> .DeprecationWarning dynamically? Seems a bit magical though. 
> 
> / Anders

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Deprecation utilities for the warnings module

[Ilya Kulakov]

(Apologies if it's a duplicate. I originally posted to 
python-id...@googlegroups.com)

I've been recently working on an internal library written
entirely in Python. The library while being under development
was as actively used by other teams. The product was under
pressure and hasty design decisions were made.
Certain  interfaces were exposed before scope of the problem
was entirely clear.

Didn't take long before the code started to beg for refactoring.
But compatibility needed to be preserved. As a result a set of utilities,
which I'd like to share, came to life.

---

The warnings module is a robust foundation. Authors can warn
what should be avoided and users can choose how to act.
However in its current form it still requires utilities
to be useful. E.g. it's non-trivial to properly replace
a deprecated class.

I'd like to propose an extension for the warnings module
to address this problem.

The extensions consists of 4 decorators:
- @deprecated
- @obsolete
- @deprecated_arg
- @obsolete_arg

The @deprecated decorator marks an object to issue a warning
if it's used:
- Callable objects issue a warning upon a call
- Property attributes issue a warning upon an access
- Classes issue a warning upon instantiation and subclassing
  (directly or via a metaclass)

The @obsolete decorator marks an object in the same way
as @deprecated does but forwards usage to the replacement:
- Callable object redirect the call
- Property attribute redirect the access (get / set / del)
- Class is replaced in a way that during both instantiation
  and subclassing replacement is used

In case of classes extra care is taken to preserve validity
of existing isinstance and issubclass checks.

The @deprecated_arg and @obsolete_arg work with signatures
of callable objects. Upon a call either a warning is issued
or an argument mapping is performed.

Please take a look at the implementation and a few examples:
https://gist.github.com/Kentzo/53df97c7a54609d3febf5f8eb6b67118

Why I think it should be a part of stdlib:
- Library authors are reluctant to add dependencies
  especially when it's for internal usage
- Ease of use will reduce compatibility problems
  and ease migration burden since the soltion will be
  readily available
- IDEs and static analyzers are more likely
  to support it

---

Joshua Harlow shared a link to OpenStack's debtcollector:
https://docs.openstack.org/debtcollector/latest/reference/index.html
Certain aspects of my implementation are inspired by it.

Please let me know what you think about the idea in general
and implementation in particular. If that's something the community
is interested in, I'm willing to work on it.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Extend the warnings module with Deprecation utility methods and classes

[Ilya Kulakov]

Very nice, thank you.

It would be a good start to collect usage patterns like this as seen in the 
wild for design consideration and test cases.

> On Aug 30, 2018, at 10:43 PM, Joshua Harlow  wrote:
> 
> And mirrored at https://github.com/openstack/debtcollector 
> 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Extend the warnings module with Deprecation utility methods and classes

[Ilya Kulakov]

Sooner or later authors and maintainers of libraries change public 
interfaces of their creations. 
Usually one of the two approaches is taken:

1. Outright breaking change
2. Soft deprecation later followed by [1]

While [1] is perfectly suitable for libraries with limited audience, [2] is 
what popular libraries
have to do (to remain popular).

Python's stdlib contributed to the process via PEP 230 and, recently, PEP 
565.
The `warn` function and subclasses of Warning are omnipresent. However, 
when it comes
to practical application it's still up to a developer to write utility 
methods and classes.


I propose that Python should extend the warnings module with a set of 
utilities to cover
basic needs as seen across the industry. The extension should include:

1. A decorator for functions
2. A decorator for classes
3. A context manager


Function Decorator

A function can be deprecated:

- Via a rename when an author decides there is a better name
- Via a change of its interface
- By becoming obsolete

Therefore the job of the decorator is to modify the behavior such as upon a 
call:

- A warning is issued
- Optionally an aliased (new) function is called
- Optionally both positional and keyword arguments are mapped to satisfy 
new interface;
might be followed by an extra warning for each remap


Class Decorator

A class can be deprecated:

- Via a rename when an author decides there is a better name
- By becoming obsolete

Therefore a job of the decorator is to ensure that:

- A warning is issued when class is either instantiated or used as a base 
class
- If aliased, subclasses of a deprecated class (from the end user 
perspective) pass `issubclass` and `isinstance` checks against an aliased 
new class
- If aliased, subclasses of a new class pass `issubclass` and `isinstance` 
checks against a deprecated class


Context Manager

Context manager is useful when behavior of a particular function changes 
without visible changes to its name or interface.
E.g. when author realizes (too late) that arguments are mutually exclusive. 
Instead of introducing a breaking change outright, better solution is to 
wrap that particular case with a warning and consistent behavior.


Each of these utilities should allow to specify a condition via a callable. 
E.g. when deprecation only makes sense for certain platforms or python 
interpreter versions.


I think most if not all can be implemented using existing runtime and 
stdlib. But before diving into code, what do you think of this idea overall?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Stub class for Generic to further improve PEP 560

[Ilya Kulakov]

> Anyway, my expectation is that going along this way (i.e. removing all 
> runtime API apart from a necessary minimum)
> will give a minor speed-up as compared to PEP 560 at the cost of a breaking 
> change (even for small number of developers).

I don't think the change will be breaking: usage of this class will be entirely 
voluntarily and does not replace typing.Generic

> PEP 560 already gives overhead of 80% as compared to normal classes in worst 
> case scenario
> (empty body with a single generic base). This is actually less than for ABCs 
> (they can give up to 120% in worst case scenario).

GenericMeta inherits from ABCMeta. Do you mean that it will be removed after 
560 lands?

> Moreover, performance is not a single motivation for PEP 560, there are other 
> arguments such as metaclass conflicts which will
> not be solved without the machinery proposed by the PEP.

Perhaps you can consider designing Generic / GenericMeta in a way that will 
allow end user to create GenericStub-alike class without much trouble?___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Stub class for Generic to further improve PEP 560

[Ilya Kulakov]

A very rough implementation:

typing.py:

class _GenericMetaNoop(type):
def __getitem__(self, params):
return self

class _GenericNoop(metaclass=_GenericMetaNoop)
pass

GenericStub = Generic if TYPE_CHECKING else _GenericNoop

elsewhere.py:

import typing

T = typing.TypeVar('T')

class MyClass(typing.GenericStub[T]):
pass

Now when run under type checkers MyClass will inherit typing.Generic with all 
runtime complexity.
However when run outside of them code will just work.


> On Nov 30, 2017, at 1:35 PM, Ivan Levkivskyi  wrote:
> 
> Could you please give some examples for these statements? They still look to 
> abstract for me.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Stub class for Generic to further improve PEP 560

[Ilya Kulakov]

My point is to have a class with public interface identical to typing.Generic, 
but without all runtime overhead.
It's based on the assumption that few developers benefit with typing.Generic 
addition during production application execution.

Those who do, can subclass typing.Generic directly.

> On Nov 30, 2017, at 1:22 PM, Ivan Levkivskyi  wrote:
> 
> FWIW, with PEP 560 generic classes will stay generic at runtime. I am not 
> sure what you mean by this, but
> practically all existing runtime APIs (with few exceptions, see PEP 560) for 
> generics will stay.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Stub class for Generic to further improve PEP 560

[Ilya Kulakov]

As explained in PEP 560, creation of Generic subclasses is much slower (~7x on 
my machine).
My guess is that a limited number of applications actually need Generic classes 
at "production" runtime (i.e. when not under analysis of mypy or IDE).

I propose a special class that will alias typing.Generic whenever 
typing.TYPE_CHECKING == True and a no-op stub otherwise.


Best Regards,
Ilya Kulakov

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Complete typing.Union with the rest of set-like functions

[Ilya Kulakov]

I needed to declare a type that would mean "Any but None" and didn't find how.

Current implementation allows to expand set of allowed types, not narrow it.
Perhaps typing needs the rest of set operators in addition to Union?


Best Regards,
Ilya Kulakov

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Thread.__init__ should call super()

[Ilya Kulakov]

Neil, thank you for doing much better job explaining the problem.

Generally, I'm cool with Python's standard library classes not calling super(), 
as many of them
are not designed for subclassing. But those which are should do that. E.g. take 
a look at more
recent asyncio's Protocol and Transport classes: they all properly call super().

One potential problem is that it will break existing code:

class X(Thread, SomethingElse):
def __init__(self):
Thread.__init__(self)
SomethingElse.__init__(self)

SomethingElse.__init__ will be called twice.
Is it a good reason for "old" classes to lag behind? I don't know.

Perhaps some mechanism (invisible to a user) can be designed to avoid that.
E.g. super() may leave a flag which should signal interpreter to "skip" all 
direct calls
of a function and warn about it (DeprecationWarning?).

Best Regards,
Ilya Kulakov

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Thread.__init__ should call super()

[Ilya Kulakov]

Since one of the legit use-cases of using the Thread class is subclassing,
I think it's __init__ should call super() to support cooperative inheritance.

Or perhaps there is a good reason for not doing so?

Best Regards,
Ilya Kulakov

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/