date:20200218

It is hard to distinguish a sequence and a mapping in Python 3. Both
have __getitem__, __len__ and __iter__ methods, but with different semantic.

In Python code you can use isinstance() and issubclass() checks against
collections.abc.Mapping, but it works only with types which was
explicitly registered in Mapping. Autodectection does not work as for
Iterable or Hashable. Also importing collections.abc.Mapping and calling
PyObject_IsInstance() or PyObject_IsSubclass() is cumbersome and
inefficient in the C code. It is rarely used.

It is more common to check for the existence of the keys() method. The
dict constructor and update() method do this, as well as other code
which emulates the behavior of dict. This is the only non-dunder method
used by the core operations. There are problems with this:

* Since it is not a dunder method, it is not reserved, and may conflicts
with the user attribute.

* Since it is not a dunder method, it is checked for the instance, not
for the type. Instances of the same type can look as a mapping or as not
a mapping for the same code. Checking the attribute of the class does
not give you information about the attribute of the instance.

* There is no a corresponding slot in the type object. So checking the
existence of the keys attribute is slow, non-atomic, can execute an
arbitrary code and raise an exception. It can't be used in
PyMapping_Check() and slows down the dict constructor.

* The keys() method is not called in the dict constructor. Just the
existence of the keys attribute is checked, its value is not used.

* It is a special case. All other implicitly called methods which
determined the behavior of builtin types are dunder names.

I propose to add support for special methods `__keys__` or `__items__`
and corresponding type slot (`tp_mapping->mp_keys` or
`tp_mapping->mp_items`). At first stage, the code which checked for
"keys" should check also for the special method and use them if defined.
At second stage it should emit a warning if "keys" is defined, but the
special method is not defined. At third stage, it should use the result
of the special method and ignore "keys". PyMapping_Check() can be made
checking the corresponding slot.

I am not sure about some details:

1. What special method should be added, `__keys__` or `__items__`? The
former returns keys, it needs to call `__getitem__` repeatedly to get
values. The latter returns key-value pairs, it does not need to call
`__getitem__`, but you should always pay the cost of creating a tuple
even if you do not need values.

2. What type should they return?

* An iterator.
* An iterable which can be iterated only once. Easy to implement in
Python as generator methods.

* An iterable which can be iterated multiple times.
* More complex view object which may support other protocols (for
example support `__or__` and `__and__`).

What do you think about this?
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/A3CK7Y2ICIIVYKMAXSCUVYLK7IC7CH6G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add keys or items protocol

2020-02-18 Thread Chris Angelico

On Tue, Feb 18, 2020 at 7:25 PM Serhiy Storchaka  wrote:
> * The keys() method is not called in the dict constructor. Just the
> existence of the keys attribute is checked, its value is not used.

Given that that's already been the case, my preferred colour for the
bike shed is...

> 1. What special method should be added, `__keys__` or `__items__`? The
> former returns keys, it needs to call `__getitem__` repeatedly to get
> values. The latter returns key-value pairs, it does not need to call
> `__getitem__`, but you should always pay the cost of creating a tuple
> even if you do not need values.

... __items__, because it'd be more efficient for the dict
constructor; and if anything needs the opposite behaviour, it can
check for the presence of __items__ and then simply iterate over the
object, to get the keys. (Document that calling __items__ should
return tuples in the same order as iteration returns keys, just like a
dict does.)

> 2. What type should they return?
>
> * An iterator.
> * An iterable which can be iterated only once. Easy to implement in
> Python as generator methods.
> * An iterable which can be iterated multiple times.
> * More complex view object which may support other protocols (for
> example support `__or__` and `__and__`).
>

I'd be inclined to mandate as little as possible; to be precise: it
must return an iterable, but it's okay if that iterable be single-use,
and it's okay either way whether it's a snapshot or a view. So any of
the above would be compliant.

+1 for (eventually) removing the special-case of using keys() as a signal.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/YZFEQQPSTPWXRD3Z6CZX6WTPKBMTV4MD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a way to execute code in the context of different frame

2020-02-18 Thread M.-A. Lemburg

On 18.02.2020 08:33, Serhiy Storchaka wrote:
> The idea is inspired by the following StackOverflow question:
> https://stackoverflow.com/questions/40945752/inspect-who-imported-me and
> the corresponding BPO issue: https://bugs.python.org/issue39643.
> 
> In the older Python versions the f_back attribute of the frame in which
> the module code is executed pointed to the frame which executes the
> import statement or calls __import__() or import_module(). But after
> rewriting the import machinery in Python, there are other frames between
> the caller frame and the calling frame. This caused problems with
> tracebacks which was worked around by removing intermediate frames
> related to the import machinery when the exception floats to the caller.
> But there is still unresolved problem with warnings, and all frames of
> the import machinery are visible from the imported code.
> 
> I propose to add possibility to execute the code in the context of
> different frame. Either add a frame argument in exec() and eval(), which
> will allow to pass an arbitrary frame. Or add a stacklevel argument in
> exec() and eval() (similar to warnings.warn()), which will limit
> possible frames to the parent frames of the current frame. I do not know
> what is more feasible.
> 
> This will automatically fix problems with warnings. This will allow to
> get rid the workaround for exceptions, and may make the code simpler.
> This will allow the imported code to do miscellaneous magic things with
> the code which performs the import.
> 
> What do you think about this?

This sounds like a nice idea, but it may make sense to limit the
frame to one already on the stack. Otherwise, the code you're
executing may never return to your current stack...

OTOH, perhaps that could be regarded and used as a feature, similar
to what PyPy exposes:

https://doc.pypy.org/en/latest/stackless.html

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 18 2020)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XNFL4FBRSXJL2MGNIWILP4HHGECRGOSH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a way to execute code in the context of different frame


18.02.20 11:28, M.-A. Lemburg пише:

This sounds like a nice idea, but it may make sense to limit the
frame to one already on the stack. Otherwise, the code you're
executing may never return to your current stack...


I do not think there will be such problem. exec() will temporary replace 
the stack of frames before executing the code and restore the original 
stack after it be finished. Since the stack is implemented as a 
single-linked list, and multiple lists can share the same tail, it all 
should be zero-overhead.



OTOH, perhaps that could be regarded and used as a feature, similar
to what PyPy exposes:

https://doc.pypy.org/en/latest/stackless.html


Thank you for the link. It is interesting. The difference with PyPy is 
that I do not propose to create cycles. And since CPython uses the 
single C stack it has nothing with stackless and do not allow to get rid 
of the recursion depth limit. But there is something common in these 
ideas -- both work with f_back. Maybe more.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/MQQ547VS24A6TQ5MGX4CP2HKYIT74PXM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add keys or items protocol


18.02.20 10:34, Chris Angelico пише:

Given that that's already been the case, my preferred colour for the
bike shed is...


Thank you for your feedback. This is my preferable color too, for both 
parts.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/5YF4BSNER7MYOJ52JOUDPWWGW7UOI2CJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] a new itertools.product variant that supports kwargs and returns dicts

2020-02-18 Thread Daniel Grießhaber

When doing a grid-search over some hyper-parameters of an algorithm I often
need to specify the possible value space for each parameter and then
evaluate on every combination of these values.

In this case, the product function of the itertools module is handy:

from itertools import product

def train_and_evaluate(lr, num_layers, dataset):
print(f'training on {dataset} with {num_layers=} and {lr=}')

params = (
(0.01, 0.1, 0.5), #learning rate
range(10, 40, 10), #number of hidden neurons
('MNLI', 'SNLI', 'QNLI') # training dataset
)

for config in product(*params):
train_and_evaluate(*config)

However, this code relies on the order of parameters in the function
train_and_evaluate, which could be considered not very pythonic, as
explicit is better than implicit.

In the same way, the 'intention' for the values is only clear with the
context of the function signature (or the comments, if someone bothered to
write them).

Therefore I propose a new variant of the product() function that supports
Mappings instead of Iterables to enable to explicitly specify the function
of an iterables values in the product. A trivial implementation could be:

def named_product(repeat=1, **kwargs):
for combination in itertools.product(*kwargs.values(), repeat=repeat):
yield dict(zip(kwargs.keys(), combination))

which could then be used like this:

params = {
'lr': (0.01, 0.1, 0.5), #learning rate
'num_layers': range(10, 40, 10), #number of hidden neurons
'dataset': ('MNLI', 'SNLI', 'QNLI') # training dataset
}

for config in named_product(**params):
train_and_evaluate(**config)

This has the advantage that the order of the parameters in the kwargs does
not depend on the method signature of train_and_evaluate.
Open Questions

   -

   Support scalar values?

   I the example use-case it may be nice to not have each kwarg to be an
   iterable, because you may want to specify all parameters to the function,
   even if they do not vary in the product items (factors?)

 params = named_product(a=(1, 2), b=3)

   instead of

 params = named_product(a=(1, 2), b=(3, ))

   However, this may be unexpected to users when they supply a string which
   would then result in an iteration over its characters.
   -

   Would this may be suited for the more-itertools package?

   However, I would love to have this build-in and not depend on a
   non-stdlib module
   -

   More functionality:

   In my toolbox-package I implemented a variant with much more
   functionality:
   
   https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html. This
   version offers additional functionality compared to the proposed function:
   - accept both, a dict as *arg or **kwargs and merges them
  - accepts scalar values, not only iterables but treats strings as
  scalars
  - if the value of any item is a dict itself (not scalar or Sequence),
  iterate over the nested dict as a seperate call to named_product
would and
  update the 'outer' dict with those values
  - copies values before yielding which is useful if the returned
  values are mutable (can be excluded)

   Maybe some of these features would also be useful for a general version?
   However, this version is mostly developed for my special use-case and some
   behaviour is solely based on my peculiarities

Example usage:

Here is an example from my actual code which may clarify some decisions on
functionality of the features of

https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html

configs = named_product({
'nb_epoch': 3,
'acquisition_iterations': 0,
'training_function': {
partial(train_bert, decoder_function=create_decoder):{
'decoder_name': 'FFNN'
},
partial(train_bert, decoder_function=create_cnn_decoder):{
'decoder_name': 'CNN'
}
},
'training_data': partial(load_glue, initial_training_data_size=10),
'number_queries': 100,
'batch_size': 64,
'dataset_class': [QNLIDataset, MNLIDataset],
'num_frozen_layers': 0,
'acquire_function': {
acquire_uncertain: {
'certainty_function': calculate_random,
'predict_function': 'predict',
'dropout_iterations': 1
}
}
})
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/IAVDAMJUQXI4GBNH6JXBYJE5MJYQRVXP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] A new itertools.product variant that supports kwargs and returns dicts

2020-02-18 Thread Daniel Grießhaber

When doing a grid-search over some hyper-parameters of an algorithm I often 
need to specify the possible value space for each parameter and then evaluate 
on every combination of these values.
In this case, the product function of the itertools module is handy:

from itertools import product

def train_and_evaluate(lr, num_layers, dataset):
print(f'training on {dataset} with {num_layers=} and {lr=}')

params = (
(0.01, 0.1, 0.5), #learning rate
range(10, 40, 10), #number of hidden neurons
('MNLI', 'SNLI', 'QNLI') # training dataset
)

for config in product(*params):
train_and_evaluate(*config)

However, this code relies on the order of parameters in the function 
train_and_evaluate, which could be considered not very pythonic, as explicit is 
better than implicit. 
In the same way, the 'intention' for the values is only clear with the context 
of the function signature (or the comments, if someone bothered to write them).

Therefore I propose a new variant of the product() function that supports 
Mappings instead of Iterables to enable to explicitly specify the function of 
an iterables values in the product. A trivial implementation could be:

def named_product(repeat=1, **kwargs):
for combination in itertools.product(*kwargs.values(), repeat=repeat):
yield dict(zip(kwargs.keys(), combination))

which could then be used like this:

params = {
'lr': (0.01, 0.1, 0.5), #learning rate
'num_layers': range(10, 40, 10), #number of hidden neurons
'dataset': ('MNLI', 'SNLI', 'QNLI') # training dataset
}

for config in named_product(**params):
train_and_evaluate(**config)

This has the advantage that the order of the parameters in the kwargs does not 
depend on the method signature of train_and_evaluate.

I also would appreciate your input on the following questions:

- Support scalar values?
I the example use-case it may be nice to not have each kwarg to be an 
iterable, because you may want to specify all parameters to the function, even 
if they do not vary in the product items (factors?)

params = named_product(a=(1, 2), b=3)

instead of

params = named_product(a=(1, 2), b=(3, ))

However, this may be unexpected to users when they supply a string which 
would then result in an iteration over its characters.

- Would this may be suited for the more-itertools package?
However, I would love to have this build-in and not depend on a non-stdlib 
module

- More functionality:
In my toolbox-package I implemented a variant with much more functionality: 
[https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html](https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html).
 This version offers additional functionality compared to the proposed function:
- accept both, a dict as *arg or **kwargs and merges them
- accepts scalar values, not only iterables but treats strings as scalars
- if the value of any item is a dict itself (not scalar or Sequence), 
iterate over the nested dict as a seperate call to named_product would and 
update the 'outer' dict with those values
- copies values before yielding which is useful if the returned values are 
mutable (can be excluded)

Maybe some of these features would also be useful for a general version? 
However, this version is mostly developed for my special use-case and some 
behaviour is solely based on my peculiarities

## Example usage:
Here is an example from my actual code which may clarify some decisions on 
functionality of the features of 
[https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html](https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html)

configs = named_product({
'nb_epoch': 3,
'acquisition_iterations': 0, 
'training_function': {
partial(train_bert, decoder_function=create_decoder):{
'decoder_name': 'FFNN'
},
partial(train_bert, decoder_function=create_cnn_decoder):{
'decoder_name': 'CNN'
}
},
'training_data': partial(load_glue, initial_training_data_size=10),
'number_queries': 100,
'batch_size': 64,
'dataset_class': [QNLIDataset, MNLIDataset],
'num_frozen_layers': 0,
'acquire_function': {
acquire_uncertain: {
'certainty_function': calculate_random, 
'predict_function': 'predict',
'dropout_iterations': 1
}
}
})
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/EVW2F3PD4O25S5QWQFRCBPFB7YO6BEEA/
Code o

[Python-ideas] Re: Add keys or items protocol

2020-02-18 Thread Andrew Svetlov

Sounds reasonable to me

On Tue, Feb 18, 2020 at 12:32 PM Serhiy Storchaka  wrote:
>
> 18.02.20 10:34, Chris Angelico пише:
> > Given that that's already been the case, my preferred colour for the
> > bike shed is...
>
> Thank you for your feedback. This is my preferable color too, for both
> parts.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/5YF4BSNER7MYOJ52JOUDPWWGW7UOI2CJ/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Thanks,
Andrew Svetlov
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/A6QP5JMRFSFP2YP7CBQ7I3V4DBYX77L2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add keys or items protocol

+1

On Tue, Feb 18, 2020 at 5:01 AM Andrew Svetlov 
wrote:

> Sounds reasonable to me
>
> On Tue, Feb 18, 2020 at 12:32 PM Serhiy Storchaka 
> wrote:
> >
> > 18.02.20 10:34, Chris Angelico пише:
> > > Given that that's already been the case, my preferred colour for the
> > > bike shed is...
> >
> > Thank you for your feedback. This is my preferable color too, for both
> > parts.
> > ___
> > Python-ideas mailing list -- python-ideas@python.org
> > To unsubscribe send an email to python-ideas-le...@python.org
> > https://mail.python.org/mailman3/lists/python-ideas.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/5YF4BSNER7MYOJ52JOUDPWWGW7UOI2CJ/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Thanks,
> Andrew Svetlov
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/A6QP5JMRFSFP2YP7CBQ7I3V4DBYX77L2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/A4VJ7TDZ2YXLIXPG6HOQMCYTALN4DXWG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a way to execute code in the context of different frame

I am a little confused how you get from "there are extra frames in the
traceback" to "modify exec() to run code in another frame".

Also, with PEP 558 (Defined semantics for locals(), by Nick Coghlan) we
might be able to just pass the frame's globals and locals to exec() or
eval() without further changes (or perhaps just with changes to allow
passing a proxy instead of a true dict).

On Mon, Feb 17, 2020 at 11:36 PM Serhiy Storchaka 
wrote:

> The idea is inspired by the following StackOverflow question:
> https://stackoverflow.com/questions/40945752/inspect-who-imported-me and
> the corresponding BPO issue: https://bugs.python.org/issue39643.
>
> In the older Python versions the f_back attribute of the frame in which
> the module code is executed pointed to the frame which executes the
> import statement or calls __import__() or import_module(). But after
> rewriting the import machinery in Python, there are other frames between
> the caller frame and the calling frame. This caused problems with
> tracebacks which was worked around by removing intermediate frames
> related to the import machinery when the exception floats to the caller.
> But there is still unresolved problem with warnings, and all frames of
> the import machinery are visible from the imported code.
>
> I propose to add possibility to execute the code in the context of
> different frame. Either add a frame argument in exec() and eval(), which
> will allow to pass an arbitrary frame. Or add a stacklevel argument in
> exec() and eval() (similar to warnings.warn()), which will limit
> possible frames to the parent frames of the current frame. I do not know
> what is more feasible.
>
> This will automatically fix problems with warnings. This will allow to
> get rid the workaround for exceptions, and may make the code simpler.
> This will allow the imported code to do miscellaneous magic things with
> the code which performs the import.
>
> What do you think about this?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/ETBOLQKHTZFH7ELNOJG3TWRAXCOGYX2D/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/FUUDB734UEXB6B5G3FXPBZOPSBHG3ZVG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add keys or items protocol

On 2020-02-18 5:34 a.m., Chris Angelico wrote:

On Tue, Feb 18, 2020 at 7:25 PM Serhiy Storchaka  wrote:
> * The keys() method is not called in the dict constructor. Just the
> existence of the keys attribute is checked, its value is not used.

Given that that's already been the case, my preferred colour for the
bike shed is...

> 1. What special method should be added, `__keys__` or `__items__`? The
> former returns keys, it needs to call `__getitem__` repeatedly to get
> values. The latter returns key-value pairs, it does not need to call
> `__getitem__`, but you should always pay the cost of creating a tuple
> even if you do not need values.

... __items__, because it'd be more efficient for the dict
constructor; and if anything needs the opposite behaviour, it can
check for the presence of __items__ and then simply iterate over the
object, to get the keys. (Document that calling __items__ should
return tuples in the same order as iteration returns keys, just like a
dict does.)

I like __items__. additionally, lists should implement it to return 
enumerate(self), and sets should implement it to return (v, v for v in 
self), and as such there should be no requirement with regards to 
__items__ and __iter__, and whether __iter__ returns keys or values.

we can then call it __pairs__ instead of __items__. __items__ feels a 
lot like "outputs from __getitem__" rather than "pairs representing 
valid inputs and outputs of __getitem__".

> 2. What type should they return?
>
> * An iterator.
> * An iterable which can be iterated only once. Easy to implement in
> Python as generator methods.
> * An iterable which can be iterated multiple times.
> * More complex view object which may support other protocols (for
> example support `__or__` and `__and__`).
>

I'd be inclined to mandate as little as possible; to be precise: it
must return an iterable, but it's okay if that iterable be single-use,
and it's okay either way whether it's a snapshot or a view. So any of
the above would be compliant.

+1 for (eventually) removing the special-case of using keys() as a signal.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/YZFEQQPSTPWXRD3Z6CZX6WTPKBMTV4MD/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/EJQH42G5EOWGXDW3OCQKCQCOQ6KYEX4R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a way to execute code in the context of different frame


18.02.20 20:13, Guido van Rossum пише:
I am a little confused how you get from "there are extra frames in the 
traceback" to "modify exec() to run code in another frame".


Sorry for skipping several steps. I'll try to expand them here. PEP 558 
does not have relation to this, as frame is not the part of locals().


exec() is used by the import machinery to execute the module code. 
Currently frames look like this:


import foo  # user code
# many frames in the import machinery
# ...
# ...
exec(code, module.__dict__)  # still in the import machinery
# user code in foo.py

I propose to add a parameter to exec() which will allow to execute the 
code in the imported mode as exec() was called immediately from the 
importer code:


import foo  # user code
# user code in foo.py

As it was when the import machinery was implemented in C.

I did not propose to modify exec() to run code in another frame, I 
proposde to modify exec() to run code in new frame whose parent is a 
specified frame (either arbitrary frame or on of ancestors of the caller 
frame).


I think it may be not difficult to do. I ask whether there are other 
applications of this or similar feature and what form of the feature 
will be more convenient.


It just occurred to me that it may be not a feature of exec(), but a 
separate function which will allow to execute an arbitrary function, not 
just a Python code, as it was called from the different frame.


apply_in_frame(frame, func, /, *args, **kwargs)

or

apply_with_frames_skipped(stacklevel, func, /, *args, **kwargs)

The question is how to call this function and where to put it. In the 
first variant it could be a method of frame.


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LEYVARMREKR74BKUOPSSIKN7ZF674WDQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add keys or items protocol


18.02.20 19:00, Soni L. пише:
I like __items__. additionally, lists should implement it to return 
enumerate(self), and sets should implement it to return (v, v for v in 
self), and as such there should be no requirement with regards to 
__items__ and __iter__, and whether __iter__ returns keys or values.


The point is that it **should not** be implemented by list and set.

dict([(1, 2)]) should return {1: 2}, not {0: (1, 2)}.
dict({(1, 2)}) should return {1: 2}, not {(1, 2): (1, 2)}.

If you want to propose a new special method common for dict, list and 
set, please open a separate thread for this.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/K6AD73WSPAB3HAYBCLZNAYXBRJE54KZB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Add a __valid_getitem_requests__ protocol

It'd be nice to have a __valid_getitem_requests__ protocol that, if 
present, yields pairs such that:


for key, value in items(obj):
  assert obj[key] == value

for any obj. Note that this definition explicitly allows subsets, which 
makes it useful for sequences and defaultdicts, as they can have all 
sorts of keys (or even infinitely many in the case of defaultdict):


>>> [1, 2, 3, 4, 5][1]
2
>>> [1, 2, 3, 4, 5][1:3]
[2, 3]
>>> [1, 2, 3, 4, 5][-1]
5

and as such we should consider limiting the results for at least these 2 
cases.


I currently use a similar idea in my code but I feel like having this 
would make my code more extensible without ppl having to monkeypatch it.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/N4L5JV3ZUD5CTFTTT4DDUASE6HODW5SK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add keys or items protocol

2020-02-18 Thread Ethan Furman


On 02/18/2020 12:24 AM, Serhiy Storchaka wrote:


1. What special method should be added, `__keys__` or `__items__`? The former 
returns keys, it needs to call `__getitem__` repeatedly to get values. The 
latter returns key-value pairs, it does not need to call `__getitem__`, but you 
should always pay the cost of creating a tuple even if you do not need values.


`__items__`



2. What type should they return?

* An iterator.
* An iterable which can be iterated only once. Easy to implement in Python as 
generator methods.
* An iterable which can be iterated multiple times.
* More complex view object which may support other protocols (for example 
support `__or__` and `__and__`).


Whatever `dict.items()` returns, both for consistency and because 
`dict.items()` can become a simple alias for `dict.__items__`.



What do you think about this?


+1

--
~Ethan~
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LCWMA6WMFLLSWXPKYC6FEUDHZBR3MYPE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add keys or items protocol

2020-02-18 Thread waszka23

+1 on the proposed addition. I think that `__items__` would be a better
choice for the method name.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/QTS4VLUNHIN6DTD4A5JQZ5MNBE3QMDUX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol

2020-02-18 Thread Rhodri James


On 18/02/2020 19:43, Soni L. wrote:
It'd be nice to have a __valid_getitem_requests__ protocol that, if 
present, yields pairs such that:


for key, value in items(obj):
   assert obj[key] == value

for any obj.


OK, I'll bite.  What is this "items()" function you apply to the 
arbitrary object?


--
Rhodri James *-* Kynesim Ltd
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XFDXBFSDQC6BELVFHKPLQEZ5ERBRO545/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol




On 2020-02-18 5:08 p.m., Rhodri James wrote:

On 18/02/2020 19:43, Soni L. wrote:
It'd be nice to have a __valid_getitem_requests__ protocol that, if 
present, yields pairs such that:


for key, value in items(obj):
   assert obj[key] == value

for any obj.


OK, I'll bite.  What is this "items()" function you apply to the 
arbitrary object?



Similar to len(). Just a shitty wrapper for __valid_getitem_requests__.

We kinda have a requirement that __iter__ and __contains__ have to be 
related, see e.g. dict.__iter__ and dict.__contains__. In an ideal world 
dict would yield pairs by default. and dict() would just iterate 
whatever. But anyway I digress. As far as my proposal goes, 
__valid_getitem_requests__ would provide slightly different semantics 
for converting-to-dict:


dict(items(['a', 'b', 'c'])) == {0: 'a', 1: 'b', 2: 'c'}
dict(items({1: 'a', 2: 'b'})) == {1: 'a', 2: 'b'}
# TODO find other things with __getitem__ and put them here

In other words: it'd allow creating a dict whose __getitem__ behaviour 
is similar to that of the original collection, in the most 
straightforward case. (note how the first dict doesn't contain negative 
indices or the infinitely many slice objects that can be used to index a 
three-element list.)


(perhaps a better name for this property is needed? how about "canonical 
keys"? as in, __valid_getitem_requests__ should contain only the 
canonical "keys" (the python datamodel specifically says __getitem__ 
takes a self and a key[1]) of a collection, and their respective values.)


[1] https://docs.python.org/3/reference/datamodel.html#object.__getitem__
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/CEHZRUGM63P7WF3RTOUPLM2GIR3FDLZS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol

2020-02-18 Thread Rhodri James


On 18/02/2020 20:33, Soni L. wrote:



On 2020-02-18 5:08 p.m., Rhodri James wrote:

On 18/02/2020 19:43, Soni L. wrote:
It'd be nice to have a __valid_getitem_requests__ protocol that, if 
present, yields pairs such that:


for key, value in items(obj):
   assert obj[key] == value

for any obj.


OK, I'll bite.  What is this "items()" function you apply to the 
arbitrary object?



Similar to len(). Just a shitty wrapper for __valid_getitem_requests__.


Language, sunshine.

Do you have a use case for this, or is it just theoretically nice to 
have?  I have to say it isn't nice enough for me to actually want it, 
and I say that as someone who regularly forgets that iterating over a 
dict gets you its keys.


--
Rhodri James *-* Kynesim Ltd
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3W4UBUL6QTONUZN2XFSVAPVKZTHO62PU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol

2020-02-18 Thread Chris Angelico

On Wed, Feb 19, 2020 at 7:34 AM Soni L.  wrote:
>
> On 2020-02-18 5:08 p.m., Rhodri James wrote:
> > On 18/02/2020 19:43, Soni L. wrote:
> >> It'd be nice to have a __valid_getitem_requests__ protocol that, if
> >> present, yields pairs such that:
> >>
> >> for key, value in items(obj):
> >>assert obj[key] == value
> >>
> >> for any obj.
> >
> > OK, I'll bite.  What is this "items()" function you apply to the
> > arbitrary object?
> >
> Similar to len(). Just a shitty wrapper for __valid_getitem_requests__.
>
> We kinda have a requirement that __iter__ and __contains__ have to be
> related, see e.g. dict.__iter__ and dict.__contains__. In an ideal world
> dict would yield pairs by default. and dict() would just iterate
> whatever. But anyway I digress. As far as my proposal goes,
> __valid_getitem_requests__ would provide slightly different semantics
> for converting-to-dict:
>
> dict(items(['a', 'b', 'c'])) == {0: 'a', 1: 'b', 2: 'c'}
> dict(items({1: 'a', 2: 'b'})) == {1: 'a', 2: 'b'}
> # TODO find other things with __getitem__ and put them here
>
> In other words: it'd allow creating a dict whose __getitem__ behaviour
> is similar to that of the original collection, in the most
> straightforward case. (note how the first dict doesn't contain negative
> indices or the infinitely many slice objects that can be used to index a
> three-element list.)
>

Is this something you do a lot of? Can you show some real-world (or
real-world-cut-down) code that would benefit from this?

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/FTDZGPQ6WLMBLWEK2MMFTZBEGMIK76NF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol





On 2020-02-18 5:43 p.m., Chris Angelico wrote:

On Wed, Feb 19, 2020 at 7:34 AM Soni L.  wrote:
>
> On 2020-02-18 5:08 p.m., Rhodri James wrote:
> > On 18/02/2020 19:43, Soni L. wrote:
> >> It'd be nice to have a __valid_getitem_requests__ protocol that, if
> >> present, yields pairs such that:
> >>
> >> for key, value in items(obj):
> >>assert obj[key] == value
> >>
> >> for any obj.
> >
> > OK, I'll bite.  What is this "items()" function you apply to the
> > arbitrary object?
> >
> Similar to len(). Just a shitty wrapper for __valid_getitem_requests__.
>
> We kinda have a requirement that __iter__ and __contains__ have to be
> related, see e.g. dict.__iter__ and dict.__contains__. In an ideal world
> dict would yield pairs by default. and dict() would just iterate
> whatever. But anyway I digress. As far as my proposal goes,
> __valid_getitem_requests__ would provide slightly different semantics
> for converting-to-dict:
>
> dict(items(['a', 'b', 'c'])) == {0: 'a', 1: 'b', 2: 'c'}
> dict(items({1: 'a', 2: 'b'})) == {1: 'a', 2: 'b'}
> # TODO find other things with __getitem__ and put them here
>
> In other words: it'd allow creating a dict whose __getitem__ behaviour
> is similar to that of the original collection, in the most
> straightforward case. (note how the first dict doesn't contain negative
> indices or the infinitely many slice objects that can be used to index a
> three-element list.)
>

Is this something you do a lot of? Can you show some real-world (or
real-world-cut-down) code that would benefit from this?


Oh yes absolutely. I have a whole library focused on this sort of 
"canonical keys"-based iteration: 
https://ganarchy.autistic.space/project/0f74bd87a23b515b45da7e6f5d9cc82380443dab/


(Friendly warning, it does handle sets in a rather unusual way. I've 
never used that feature outside unit tests, either. Pay no attention to 
that however, as it's not relevant to this proposal.)

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/2FRJDMBOYU3P3VJWPTKWCFEQ22VUSX4R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol

Presumably this was inspired by the other thread, "Add __keys__ or
__items__ protocol", where an __items__() method was proposed instead of
recognizing mappings by the presence of a keys() method.

On Tue, Feb 18, 2020 at 12:46 PM Rhodri James  wrote:

> On 18/02/2020 20:33, Soni L. wrote:
> >
> >
> > On 2020-02-18 5:08 p.m., Rhodri James wrote:
> >> On 18/02/2020 19:43, Soni L. wrote:
> >>> It'd be nice to have a __valid_getitem_requests__ protocol that, if
> >>> present, yields pairs such that:
> >>>
> >>> for key, value in items(obj):
> >>>assert obj[key] == value
> >>>
> >>> for any obj.
> >>
> >> OK, I'll bite.  What is this "items()" function you apply to the
> >> arbitrary object?
> >>
> > Similar to len(). Just a shitty wrapper for __valid_getitem_requests__.
>
> Language, sunshine.
>
> Do you have a use case for this, or is it just theoretically nice to
> have?  I have to say it isn't nice enough for me to actually want it,
> and I say that as someone who regularly forgets that iterating over a
> dict gets you its keys.
>
> --
> Rhodri James *-* Kynesim Ltd
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/3W4UBUL6QTONUZN2XFSVAPVKZTHO62PU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/Q26OAZRX44VGBEPWIJAVTEXSJIKQBUDO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol

2020-02-18 Thread Chris Angelico

On Wed, Feb 19, 2020 at 7:58 AM Soni L.  wrote:
>
>
>
> On 2020-02-18 5:43 p.m., Chris Angelico wrote:
> > On Wed, Feb 19, 2020 at 7:34 AM Soni L.  wrote:
> > >
> > > On 2020-02-18 5:08 p.m., Rhodri James wrote:
> > > > On 18/02/2020 19:43, Soni L. wrote:
> > > >> It'd be nice to have a __valid_getitem_requests__ protocol that, if
> > > >> present, yields pairs such that:
> > > >>
> > > >> for key, value in items(obj):
> > > >>assert obj[key] == value
> > > >>
> > > >> for any obj.
> > > >
> > > > OK, I'll bite.  What is this "items()" function you apply to the
> > > > arbitrary object?
> > > >
> > > Similar to len(). Just a shitty wrapper for __valid_getitem_requests__.
> > >
> > > We kinda have a requirement that __iter__ and __contains__ have to be
> > > related, see e.g. dict.__iter__ and dict.__contains__. In an ideal world
> > > dict would yield pairs by default. and dict() would just iterate
> > > whatever. But anyway I digress. As far as my proposal goes,
> > > __valid_getitem_requests__ would provide slightly different semantics
> > > for converting-to-dict:
> > >
> > > dict(items(['a', 'b', 'c'])) == {0: 'a', 1: 'b', 2: 'c'}
> > > dict(items({1: 'a', 2: 'b'})) == {1: 'a', 2: 'b'}
> > > # TODO find other things with __getitem__ and put them here
> > >
> > > In other words: it'd allow creating a dict whose __getitem__ behaviour
> > > is similar to that of the original collection, in the most
> > > straightforward case. (note how the first dict doesn't contain negative
> > > indices or the infinitely many slice objects that can be used to index a
> > > three-element list.)
> > >
> >
> > Is this something you do a lot of? Can you show some real-world (or
> > real-world-cut-down) code that would benefit from this?
>
> Oh yes absolutely. I have a whole library focused on this sort of
> "canonical keys"-based iteration:
> https://ganarchy.autistic.space/project/0f74bd87a23b515b45da7e6f5d9cc82380443dab/
>
> (Friendly warning, it does handle sets in a rather unusual way. I've
> never used that feature outside unit tests, either. Pay no attention to
> that however, as it's not relevant to this proposal.)

Too short, didn't read. Specifically, I'm not going to go digging
through your links to find code to justify why you want this feature.
Post some code here, or at very least, link directly to the code that
would be improved by this feature.

Soni, you're very close to my mental blocklist. Please put a bit more
effort into your posts to make it clear what you're actually
proposing, rather than making each of us do the work.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HVXVW7TPZNWBLMWU5D7BEGPPOVM773ZA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol

2020-02-18 Thread Paul Moore

On Tue, 18 Feb 2020 at 20:47, Rhodri James  wrote:
>
> On 18/02/2020 20:33, Soni L. wrote:
> >
> >
> > On 2020-02-18 5:08 p.m., Rhodri James wrote:
> >> On 18/02/2020 19:43, Soni L. wrote:
> >>> It'd be nice to have a __valid_getitem_requests__ protocol that, if
> >>> present, yields pairs such that:
> >>>
> >>> for key, value in items(obj):
> >>>assert obj[key] == value
> >>>
> >>> for any obj.
> >>
> >> OK, I'll bite.  What is this "items()" function you apply to the
> >> arbitrary object?
> >>
> > Similar to len(). Just a shitty wrapper for __valid_getitem_requests__.
>
> Language, sunshine.
>
> Do you have a use case for this, or is it just theoretically nice to
> have?  I have to say it isn't nice enough for me to actually want it,
> and I say that as someone who regularly forgets that iterating over a
> dict gets you its keys.

This looks to me like Lua's `items()` function. It's useful in Lua,
but I'm not sure there's any obvious reason to assume it'll be a
natural fit for idiomatic Python code. As people have said, do you
have a good Python example of real code where this would be useful?
(And where dct.items() or enumerate(lst) wouldn't be sufficient - code
that expects to work equally with lists or dictionaries seems more
Lua-like than Pythonic, IMO).

Paul
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RWWNHLGO2ONXHPS56DKGTXNVFJOHPZO2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol

On Tue, Feb 18, 2020 at 1:06 PM Chris Angelico  wrote:

> On Wed, Feb 19, 2020 at 7:58 AM Soni L.  wrote:
> > [...] I have a whole library focused on this sort of
> > "canonical keys"-based iteration:
> >
> https://ganarchy.autistic.space/project/0f74bd87a23b515b45da7e6f5d9cc82380443dab/
> >
> > (Friendly warning, it does handle sets in a rather unusual way. I've
> > never used that feature outside unit tests, either. Pay no attention to
> > that however, as it's not relevant to this proposal.)
>

That link leads to a page where there's a link to
https://soniex2.autistic.space/git-repos/abdl.git/ which is not accessible
without further authentication.

> Too short, didn't read. Specifically, I'm not going to go digging
> through your links to find code to justify why you want this feature.
> Post some code here, or at very least, link directly to the code that
> would be improved by this feature.
>
> Soni, you're very close to my mental blocklist. Please put a bit more
> effort into your posts to make it clear what you're actually
> proposing, rather than making each of us do the work.
>

I have to agree with Chris here -- Soni, please put a bit more effort in
your posts, if you want to remain welcome here. From your URL I gather you
may not have the best social skills. That's fine. We all try to accommodate
a diverse audience here. But at some point, if you don't at least show that
you're trying your best, you will wear out your welcome here. Maybe that
has happened elsewhere. Don't blame it entirely on others or whatever
handicap you may have (we have no idea here, except that we observe your
posts to be somewhat cryptic). You have started a large number of threads
in a short amount of time, and most of them have generated little of use to
the Python community. Please think about how you can contribute something
more constructive.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZPQHDMUZCGYCMHKF6REC3L6GYPJY4VUM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol




On 2020-02-18 6:04 p.m., Chris Angelico wrote:

Too short, didn't read. Specifically, I'm not going to go digging
through your links to find code to justify why you want this feature.
Post some code here, or at very least, link directly to the code that
would be improved by this feature.

Soni, you're very close to my mental blocklist. Please put a bit more
effort into your posts to make it clear what you're actually
proposing, rather than making each of us do the work.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HVXVW7TPZNWBLMWU5D7BEGPPOVM773ZA/
Code of Conduct: http://python.org/psf/codeofconduct/

Alright, sorry.

The documentation states:

"ABDL expressions are regex-like constructs for matching and validating 
object structures. They can be used with JSON and similar formats, and 
even self-referential data structures."
"ABDL expressions have the ability to iterate, index, validate and 
filter data structures, through the use of the syntax elements below."
"An arrow is ``->`` and indicates indexing/iteration (Mappings, 
Sequences, Sets). Whether indexing or iteration is used is defined by 
the elements that follow, with iteration being used by default."
"A variable is a sequence of alphanumeric characters, not starting with 
a digit. A ``(key, value)`` tuple containing the respective matched 
element will be identified by this name in the results dict."


And this sums up the core aspect of the library: the library is built on 
the idea of iterating "canonical keys" and their corresponding values, 
and validating and filtering them. ("[...] iteration being used by 
default", meaning indexing takes additional syntax - this is unlike some 
JSON and XML query languages where indexing takes /less/ syntax and 
filtering is generally highly limited and verbose.)


Iteration is done thusly:

def _pairs(obj):
    if isinstance(obj, collections.abc.Mapping):
    return iter(obj.items())
    elif isinstance(obj, collections.abc.Sequence):
    return iter(enumerate(obj, 0))
    elif isinstance(obj, collections.abc.Set):
    return iter(((e, e) for e in obj))
    else:
    # maybe there's more stuff I can implement later
    raise TypeError

I think having a proper protocol for this behaviour would be beneficial, 
as no (monkey)patching would be needed to add support for additional 
collections and data types. The comment saying there may be additional 
data types I've missed would then become irrelevant, as they'd be 
covered by __valid_getitem_requests__.


Sometimes you just want to treat everything as a (read-only) dict. 
Mostly when dealing with user input (e.g. configs), but not only.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/BO75Q4QMZ7UDMSMK3F2MM5BBLKVV7ZME/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol

On Tue, Feb 18, 2020 at 1:45 PM Soni L.  wrote:

> [...]
> Iteration is done thusly:
>
> def _pairs(obj):
>  if isinstance(obj, collections.abc.Mapping):
>  return iter(obj.items())
>  elif isinstance(obj, collections.abc.Sequence):
>  return iter(enumerate(obj, 0))
>  elif isinstance(obj, collections.abc.Set):
>  return iter(((e, e) for e in obj))
>  else:
>  # maybe there's more stuff I can implement later
>  raise TypeError
>
> I think having a proper protocol for this behaviour would be beneficial,
> as no (monkey)patching would be needed to add support for additional
> collections and data types. The comment saying there may be additional
> data types I've missed would then become irrelevant, as they'd be
> covered by __valid_getitem_requests__.
>
> Sometimes you just want to treat everything as a (read-only) dict.
> Mostly when dealing with user input (e.g. configs), but not only.
>

But what will you do with the pairs?

For mappings and sequences, the pairs represent (key, value) or (index,
value) pairs, which makes sense, since (as you write in your first post)
one can write obj[key] to obtain values from a mapping or sequence (key ~~
index).

But then you go on and propose returning pairs (value, value) for sets, and
there's nothing corresponding to obj[key]. So I'm not sure what your intent
for the proposed is. Why do you want pairs for everything? Suppose obj is a
number x, perhaps you would want to get a single pair (x, x) from the
iteration? (I know that's probably bending the metaphor too far. :-)

Even if you were to answer this question, consider that perhaps your
application is rather unique in this insistence on pairs. Or perhaps you
should've stopped at proposing an API that unifies mappings and sequences
(which more sensible since both support __getitem__) -- isn't that what
your original post was about?

However even for mappings and sequences the analogy only goes so far, since
*iterating* over them behaves very differently: for a mapping, iteration
yields keys, while for a sequence, it yields values. A legitimate question
that comes up from time to time is actually how to *distinguish* between
mappings and sequences -- se the other python-ideas thread that I mentioned
previously.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/MLWXVXG5JIT34E27XM3UNA3OTYF2XFSG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a way to execute code in the context of different frame

On Tue, Feb 18, 2020 at 11:09 AM Serhiy Storchaka 
wrote:

> 18.02.20 20:13, Guido van Rossum пише:
> > I am a little confused how you get from "there are extra frames in the
> > traceback" to "modify exec() to run code in another frame".
>
> Sorry for skipping several steps. I'll try to expand them here. PEP 558
> does not have relation to this, as frame is not the part of locals().
>

Ah, fair.

exec() is used by the import machinery to execute the module code.
> Currently frames look like this:
>
>  import foo  # user code
>  # many frames in the import machinery
>  # ...
>  # ...
>  exec(code, module.__dict__)  # still in the import machinery
>  # user code in foo.py
>
> I propose to add a parameter to exec() which will allow to execute the
> code in the imported mode as exec() was called immediately from the
> importer code:
>
>  import foo  # user code
>  # user code in foo.py
>
> As it was when the import machinery was implemented in C.
>

This makes sense (except when debugging the import machinery, which is also
a valid use case. :-).

The caller would still have to provide a dict for the globals (the module
dict), and I guess it would have to point to the *previous* frame (the one
executing `import foo`) so that exec() can create a new frame pointing back
there. And somehow we would need to set things up so that after exec()
completes or raises we automatically return or raise to the parent frame,
rather than passing control to exec()'s caller. Or alternatively, we'd need
exec()'s caller (deep down inside importlib) to be able to say "pretend to
return or raise from this (new) frame".

> I did not propose to modify exec() to run code in another frame, I
> proposde to modify exec() to run code in new frame whose parent is a
> specified frame (either arbitrary frame or on of ancestors of the caller
> frame).
>

I understand that now.

I think it may be not difficult to do. I ask whether there are other
> applications of this or similar feature and what form of the feature
> will be more convenient.
>

My gut feeling says that the only other use cases will be alternatives to
importlib, perhaps other kinds of import hooks.

> It just occurred to me that it may be not a feature of exec(), but a
> separate function which will allow to execute an arbitrary function, not
> just a Python code, as it was called from the different frame.
>
>  apply_in_frame(frame, func, /, *args, **kwargs)
>
> or
>
>  apply_with_frames_skipped(stacklevel, func, /, *args, **kwargs)
>
> The question is how to call this function and where to put it. In the
> first variant it could be a method of frame.
>

So either way it would create a new frame, right? And it would not pass
control to the caller, but to the indicated frame.

I'm not that keen on passing a stacklevel -- that could easily cause
confusion, and to calculate the right level the caller would have to crawl
up the stack anyway.

A problem I see with either solution is that it looks like except and
finally blocks in the intervening frames would be skipped. Or perhaps not,
but at that point I suspect that the implementation won't be all that easy
after all.

So *if* we end up doing this optimization I think it may be best to make it
something that's clearly meant for the importlib use case. Sine we already
have sys._getframe(), maybe it could be another magic thing in sys?

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/JY2RCOZKES5RESIZBFTKWDPXAXTHUUPV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a __valid_getitem_requests__ protocol