[issue47006] PEP 646: Decide on substitution behavior

[Guido van Rossum]

Guido van Rossum  added the comment:

We need to move on this, because the outcome of this discussion is a release 
blocker for 3.11b1 -- the next release!

--
priority: normal -> release blocker
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Matthew Rahtz]

Matthew Rahtz  added the comment:

Ok, https://github.com/python/cpython/pull/32341/files is a reference of how 
the current implementation behaves. Fwiw, it *is* mostly correct - with a few 
minor tweaks it might be alright for at least the 3.11 release.

In particular, instead of dealing with the thorny issue of what to do about 
splitting unpacked arbitrary-length tuples over multiple type variables - e.g. 
C[T, *Ts][*tuple[int, ...]] - instead either deciding to try and evaluate it 
properly and living with the complexity, or leaving it unsimplified and living 
with the __args__, __parameters__ and __origin__ problem - for now, we could 
just raise an exception for any substitutions which involve an unpacked 
arbitrary-length tuple, since I'd guess it's going to be an extremely rare 
use-case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Matthew Rahtz]

Change by Matthew Rahtz :


--
keywords: +patch
pull_requests: +30396
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/32341

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Matthew Rahtz]

Matthew Rahtz  added the comment:

[Guido]

> 1. Some edge case seems to be that if *tuple[...] is involved on either side 
> we will never simplify.

Alright, let me think this through with some examples to get my head round it.

It would prohibit the following difficult case:

class C(Generic[*Ts]): ...
Alias = C[T, *Ts]
Alias[*tuple[int, ...]]  # Does not simplify; stays C[T, *Ts][*tuple[int, ...]]

That seems pretty reasonable. It would also prohibit these other relatively 
simple cases, but I guess that's fine:

Alias = C[*Ts]
Alias[*tuple[int, ...]]  # Does not simplify; stays C[*Ts][*tuple[int, ...]]

Alias = C[T, *tuple[int, ...]]
Alias[str]  # Does not simplify; stays C[T, *tuple[int, ...]][str]


> Or perhaps a better rule is that *tuple[...] is never simplified away (but 
> fixed items before and after it may be).

Is this to say that we effectively prohibit binding *tuple[...] to anything? If 
we can simplify without binding *tuple[...] to anything, then we do simplify, 
but otherwise, we don't simplify? So under this rule, the following WOULD work?

Alias = C[T, *tuple[int, ...]]
Alias[str]  # Simplifies to C[str, *tuple[int, ...]], because we didn't have to 
bind *tuple[int, ...] to do it


> 2. Another edge case is that if neither side has any starred items we will 
> always simplify (since this is the existing behavior in 3.10). This may raise 
> an error if the number of subscripts on the right does not match the number 
> of parameters on the left.

Alright, so this is business as usual.


> 3. If there's a single *Ts on the left but not on the right, we should be 
> able to simplify, which again may raise an error if there are not enough 
> values on the right, but if there are more than enough, the excess will be 
> consumed by *Ts (in fact that's the only way *Ts is fed).

So then:

class C(Generic[*Ts]): ...
Alias = C[T, *Ts]
Alias[()]  # Raises error
Alias[int] # Simplifies to C[int, *Ts]
Alias[int, str]# Simplifies to C[int, str]
Alias[int, str, bool]  # Simplifies to C[int, str, bool]

Yup, seems straightforward.


> 4. If there's a *Ts on the right but not on the left, we should _not_ 
> simplify, since whatever we have on the left serves as a constraint for *Ts.

Ok, so this is about the following situations:

class C(Generic[*Ts]): ...
Alias = C[T1, T2]
Alias[*Ts]  # Does not simplify; stays C[T1, T2][*Ts]

Yikes - in fact, this is actually super hairy; I hadn't thought about this edge 
case at all in the PEP.

Agreed that it seems reasonable not to simplify here.


> E.g. tuple[int, int][*Ts] constrains *Ts to being (int, int).

Was that a typo? Surely tuple[int, int][*Ts] isn't valid - since tuple[int, 
int] doesn't have any free parameters?


> 5. If there's exactly one *Ts on the left and one on the right, we _might__ 
> be able to simplify if the prefix and suffix of the __parameters__ match the 
> prefix and suffix of the subscript on the right. E.g. C[int, T, *Ts, 
> float][str, *Ts] can be simplified to C[int, str, *Ts, float]. OTOH C[int, T, 
> *Ts, float][*Ts] cannot be simplified -- but we cannot flag it as an error 
> either. Note that __parameters__ in this example is (T, Ts); we have to 
> assume that typevartuples in __parameters__ are always used as *Ts (since the 
> PEP recognizes no valid unstarred uses of Ts).

Ok, this also makes sense.


---

Still, though, doesn't the point that Serhiy brought up about __origin__, 
__parameters__ and __args__ still apply? In cases where we *don't* simplify, 
there'd still be the issue of what we'd set these things to be.

This evening I'll also revisit the PRs adding tests for substitution to try and 
make them a comprehensive reference as to what's currently possible.

--

___
Python tracker 
<https://bugs.python.org/issue47006>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Matthew Rahtz]

Matthew Rahtz  added the comment:

Apologies for the slow reply - coming back to this now that the docs and 
pickling issues are mostly sorted.

[Serhiy]

> > Alias = C[T, *Ts]
> > Alias2 = Alias[*tuple[int, ...]]
> > # Alias2 should be C[int, *tuple[int, ...]]
>
> tuple[int, ...] includes also an empty tuple, and in this case there is no 
> value for T.

This was my initial intuition too, but Pradeep pointed out to me in 
https://github.com/python/cpython/pull/31021#discussion_r815853784 that for 
tuple[int, ...], Python has chosen the opposite mindset: instead of assuming 
the worst-case scenario, we assume the best-case scenario. Thus, the following 
type-checks correctly with mypy 
(https://mypy-play.net/?mypy=latest=3.10=b9ca66fb7d172f939951a741388836a6):

def return_first(tup: tuple[int, ...]) -> int:
return tup[0]
tup: tuple[()] = ()
return_first(tup)

> > We actually deliberately chose not to unpack concrete tuple types - see the 
> > description of https://github.com/python/cpython/pull/30398, under the 
> > heading 'Starred tuple types'. (If you see another way around it, though, 
> > let me know.)
> 
> You assumed that *tuple[str, bool] in def foo(*args: *tuple[str, bool]) 
> should give foo.__annotations__['args'] = tuple[str, bool], but it should 
> rather give (str, bool). No confusion with tuple[str, bool].

Fair point, we could *technically* distinguish between tuple[str, bool] and 
(str, bool). But if I was a naive user and I saw `foo.__annotations__['args'] 
== (str, bool)`, I don't think it'd be immediately obvious to me that the type 
of `args` was `*tuple[str, bool]`.

Also though, there's a second reason mentioned in 
https://github.com/python/cpython/pull/30398 why `(str, bool)` wouldn't be the 
best choice. We decided that the runtime behaviour of `*args: *something` 
should be that we essentially do `(*something,)[0]`. If we made `tuple[int, 
str]` unpack to `(int, str)`, then we'd end up with `__annotations__['args'] == 
(int,)`.

> And one of PEP 646 options is to implement star-syntax only in subscription, 
> not in var-parameter type annotations.

As in, we would allow `Generic[*Ts]`, but not `*args: *Ts`? That'd be a *major* 
change to the PEP - not an option I'm willing to consider at this stage in the 
process.

> > I'm also not sure about this one; disallowing unpacked TypeVarTuples in 
> > argument lists to generic aliases completely (if I've understood right?)
>
> No, it will only be disallowed in substitution of a VarType. Tuple[T][*Ts] -- 
> error. Tuple[*Ts][*Ts2] -- ok.

Ah, gotcha. My mistake.

[Guido]

I ran out of time this evening :) Will reply properly soon.

--

___
Python tracker 
<https://bugs.python.org/issue47006>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

> 1. Some edge case seems to be that if *tuple[...] is involved on either side 
> we will never simplify. Or perhaps a better rule is that *tuple[...] is never 
> simplified away (but fixed items before and after it may be).

I do not understand this. Do you forbid simplifying of tuple[*Ts, float][str, 
*tuple[int, ...]] to tuple[str, *tuple[int, ...], float]?

I think that the rule should be that *tuple[X, ...] cannot split between 
different variables. Or that it cannot substitute a TypeVar. A more strong 
variant of rule 4.

> 5. ... but we cannot flag it as an error either.

I think that it will better to flag it as an error now. Later, after all code 
be merged and all edge cases be handled we can return here and reconsider this.

There are workarounds for this.

* You should not use Generic[*Ts] if you require at least one item, but 
Generic[*Ts, T].
* Instead of `def foo(*args: *Ts)` use `def foo(*args: *tuple[*Ts, T])`.

These tricks are common in functional programming.

The rest of the rules match my implementations more or less.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

> Alias = C[T, *Ts]
> Alias2 = Alias[*tuple[int, ...]]
> # Alias2 should be C[int, *tuple[int, ...]]

tuple[int, ...] includes also an empty tuple, and in this case there is no 
value for T.

> Oh, also interesting - I didn't know about this one either. Could you give an 
> example?

If __origin__, __parameters__, __args__ are a mess, it will definitely break a 
code which use them.


> We actually deliberately chose not to unpack concrete tuple types - see the 
> description of https://github.com/python/cpython/pull/30398, under the 
> heading 'Starred tuple types'. (If you see another way around it, though, let 
> me know.)

You assumed that *tuple[str, bool] in def foo(*args: *tuple[str, bool]) should 
give foo.__annotations__['args'] = tuple[str, bool], but it should rather give 
(str, bool). No confusion with tuple[str, bool].

And one of PEP 646 options is to implement star-syntax only in subscription, 
not in var-parameter type annotations.

> I'm also not sure about this one; disallowing unpacked TypeVarTuples in 
> argument lists to generic aliases completely (if I've understood right?)

No, it will only be disallowed in substitution of a VarType. Tuple[T][*Ts] -- 
error. Tuple[*Ts][*Ts2] -- ok.

I propose to implement simple and strict rules, and later add support of new 
cases where it makes sense.

--

___
Python tracker 
<https://bugs.python.org/issue47006>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Guido van Rossum]

Guido van Rossum  added the comment:

I'd like to look at this as a case of simplifying something to its simplest 
canonical form, but no simpler. This is what the existing fixed-typevar 
expansion does: e.g. tuple[str, T, T][int] becomes tuple[str, int, int].

I propose that we try to agree on a set of rules for what can be simplified 
further and what cannot, when we have B = C[...]; A = B[...], (IOW A = 
C[...][...]), for various shapes of the subscripts to C and B. Note that what's 
relevant for the second subscript is C[...].__parameters__, so I'll call that 
"left" below.

1. Some edge case seems to be that if *tuple[...] is involved on either side we 
will never simplify. Or perhaps a better rule is that *tuple[...] is never 
simplified away (but fixed items before and after it may be).

2. Another edge case is that if neither side has any starred items we will 
always simplify (since this is the existing behavior in 3.10). This may raise 
an error if the number of subscripts on the right does not match the number of 
parameters on the left.

3. If there's a single *Ts on the left but not on the right, we should be able 
to simplify, which again may raise an error if there are not enough values on 
the right, but if there are more than enough, the excess will be consumed by 
*Ts (in fact that's the only way *Ts is fed).

4. If there's a *Ts on the right but not on the left, we should _not_ simplify, 
since whatever we have on the left serves as a constraint for *Ts. (E.g. 
tuple[int, int][*Ts] constrains *Ts to being (int, int).)

5. If there's exactly one *Ts on the left and one on the right, we _might__ be 
able to simplify if the prefix and suffix of the __parameters__ match the 
prefix and suffix of the subscript on the right. E.g. C[int, T, *Ts, 
float][str, *Ts] can be simplified to C[int, str, *Ts, float]. OTOH C[int, T, 
*Ts, float][*Ts] cannot be simplified -- but we cannot flag it as an error 
either. Note that __parameters__ in this example is (T, Ts); we have to assume 
that typevartuples in __parameters__ are always used as *Ts (since the PEP 
recognizes no valid unstarred uses of Ts).

TBH case 5 is the most complex and I may have overlooked something. I'm more 
sure of cases 1-4.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Matthew Rahtz]

Matthew Rahtz  added the comment:

P.s. To be clear, (I think?) these are all substitutions that are computable. 
We *could* implement the logic to make all these evaluate correctly if we 
wanted to. It's just a matter of how much complexity we want to allow in 
typing.py (or in the runtime in general, if we say farmed some of this logic 
out to a separate module).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Matthew Rahtz]

Matthew Rahtz  added the comment:

[Guido]

> What would be an example of a substitution that's too complex to do?

We also need to remember the dreaded arbitrary-length tuple. For example, I 
think it should be the case that:

```python
T = TypeVar('T')
Ts = TypeVarTuple('Ts')
class C(Generic[*Ts]): pass
Alias = C[T, *Ts]
Alias2 = Alias[*tuple[int, ...]]
# Alias2 should be C[int, *tuple[int, ...]]
```

Ok, this is a bit of a silly example, but if we're committing to evaluating 
substitutions correctly, we should probably make even this kind of example 
behave correctly so that users who accidentally do something silly can debug 
what's gone wrong.

[Serhiy]

> A repr can be less readable.

Definitely true.

> It will break equality comparison and hashing. Good bye caching.

Huh, I didn't know about this one. Fair enough, this is totally a downside.

> What about __origin__, __parameters__, __args__? How will they be calculated?

This could admittedly be thorny. We'd have to think it through carefully. 
Admittedly also a downside.

> It can break code which uses annotations for something. For example it can 
> break dataclasses.

Oh, also interesting - I didn't know about this one either. Could you give an 
example?

> The first case will be practically fixed by GH 32030 after chenging the 
> grammar to allow unpacking in index tuple: A[*B].

We actually deliberately chose not to unpack concrete tuple types - see the 
description of https://github.com/python/cpython/pull/30398, under the heading 
'Starred tuple types'. (If you see another way around it, though, let me know.)

> Two other cases will be fixed by GH 32031. It does not require any C code.

I'm also not sure about this one; disallowing unpacked TypeVarTuples in 
argument lists to generic aliases completely (if I've understood right?) seems 
like too restrictive a solution. I can imagine there might be completely 
legitimate cases where the ability to do this would be important. For example:

```python
DType = TypeVar('DType')
Shape = TypeVarTuple('Shape')
class Tensor(Generic[DType, *Shape]): ...
Uint8Tensor = Tensor[uint8, *Shape]
Unit8BatchTensor = Uint8Tensor[Batch, *Shape]
```

> Note that the alternative proposition is even more lenient to errors.

True, but at least it's predictably lenient to errors - I think the repr makes 
it very clear that "Woah, you're doing something advanced here. You're on your 
own!" I think it better fits the principle of least astonishment to have 
something that consistently lets through all errors of a certain class than 
something that sometimes catches errors and sometimes doesn't.

--

___
Python tracker 
<https://bugs.python.org/issue47006>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

The first case will be practically fixed by GH 32030 after chenging the grammar 
to allow unpacking in index tuple: A[*B].

Two other cases will be fixed by GH 32031. It does not require any C code.

In the last case no error is raised because some error checks are skipped if 
any of Generic arguments is a TypeVarTuple. We just need to add such checks. 
This is Python-only code too.

Note that the alternative proposition is even more lenient to errors.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Jelle Zijlstra]

Jelle Zijlstra  added the comment:

It's simple if you only look at simple examples.

Here are some examples current main (with Serhiy's patch for the Python version 
of typing) gets wrong:

>>> from typing import *
>>> Ts = TypeVarTuple("Ts")
>>> T1 = TypeVar("T1")
>>> T2 = TypeVar("T2")
>>> Tuple[T1, Unpack[Ts], T2][int, Unpack[tuple[int]]]  # expect error
typing.Tuple[int, *tuple[int]]
>>> Tuple[T1, Unpack[Ts], str, T2][int, Unpack[Ts]]  # expect error (T2 missing)
typing.Tuple[int, str, *Ts]  # it put *Ts in the wrong place
>>> Tuple[T1, Unpack[Ts], str, T2][int, Unpack[Ts], Unpack[Ts]]  # expect error 
>>> (*Ts can't substitute T2)
typing.Tuple[int, *Ts, str, *Ts]
>>> class G(Generic[T1, Unpack[Ts], T2]): pass
... 
>>> G[int]  # expect error
__main__.G[int]

We can probably fix that, but I'm not looking forward to implementing the fixed 
logic in both Python and C. Also, I'm worried that it won't work with future 
extensions to the type system (e.g., the rumored Map operator) that may go into 
3.12 or later versions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Guido van Rossum]

Guido van Rossum  added the comment:

I think I'm with Serhiy, I don't understand the hesitance to transform 
tuple[*Ts][int, str] into tuple[int, str].

What would be an example of a substitution that's too complex to do?

--

___
Python tracker 
<https://bugs.python.org/issue47006>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I am for consistent behavior. If return GenericAlias(GenericAlias(tuple, 
Unpack[Ts]), (int, str)) for tuple[*Ts][int, str], we should also return 
GenericAlias(GenericAlias(list, T), int) for list[T][int], etc. And it will 
cause multiple problems:

* A repr can be less readable.
* It will break equality comparison and hashing. Good bye caching.
* What about __origin__, __parameters__, __args__? How will they be calculated?
* It can break code which uses annotations for something. For example it can 
break dataclasses.

It may be that will need to use it as a fallback for cases like tuple[T, 
*Ts][*Ts2] (currently it is error). But I am not sure that such cases should be 
supported.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Alex Waygood]

Change by Alex Waygood :


--
nosy: +AlexWaygood

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Jelle Zijlstra]

Jelle Zijlstra  added the comment:

Thanks Matthew! Merged PRs can still be reverted, and we have some time before 
the feature freeze. I'd like to hear what Guido and Ken think too.

If we go with the GenericAlias substitution, we need to make sure that such 
aliases still work as base class. That would need some C work to make 
types.GenericAlias.__mro_entries__ recurse if the alias's origin is itself a 
GenericAlias. There's a few other subtleties to think about; I can work on that 
but don't have a ton of time today.

--

___
Python tracker 
<https://bugs.python.org/issue47006>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Matthew Rahtz]

Matthew Rahtz  added the comment:

(Having said that, to be clear: my preferred solution currently would still be 
the solution where we just return a new GenericAlias for anything involving a 
TypeVarTuple. The crux is what Serhiy is happy with.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Matthew Rahtz]

Matthew Rahtz  added the comment:

Thanks for starting this, Jelle - I was a bit unsure about how to proceed here.

Given that https://github.com/python/cpython/pull/31800 is already merged, I'd 
also propose something halfway between the two extremes: return a sensible 
substitution when the logic to compute that isn't too onerous, and a new 
GenericAlias object when it is. The upsides are that we'd probably be able to 
return reasonable substitutions for the vast majority of cases, and that we 
wouldn't have to remove what's already been merged. The downsides would be lack 
of consistency, and the potential for changing rules about what does and 
doesn't return a full substitution as time goes on and new features are added.

--
nosy: +matthew.rahtz

___
Python tracker 
<https://bugs.python.org/issue47006>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Jelle Zijlstra]

Change by Jelle Zijlstra :


--
nosy: +mrahtz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47006] PEP 646: Decide on substitution behavior

[Jelle Zijlstra]

New submission from Jelle Zijlstra :

We've had some disagreement about the behavior of TypeVarTuple substitution 
related to PEP 646, and the discussion has now spilled around multiple PRs. I'd 
like to use this issue to come to an agreement so we don't have to chase 
through so many different places.

Links:
- GH-31021 (merged) implements PEP 646 in typing.py. Matthew initially 
implemented full TypeVar substitution support, but took it out after I 
suggested an alternative solution in 
https://github.com/python/cpython/pull/31021#pullrequestreview-890627941.
- GH-31800 (merged without review) implements TypeVarTuple substitution in 
Python.
- GH-31828 implements TypeVarTuple substitution in C.
- GH-31844 and GH-31846 add additional test cases.
- GH-31804 (closed) implements my proposed solution.

I'd like to ask that until we come to an agreement we hold off on making any 
more changes, so we don't have to go back and forth and we ensure that the 
eventual solution covers all edge cases.

The disagreement is about what to do with TypeVarTuple substitution: the 
behavior when a generic type is subscripted, like `tuple[*Ts][int, str]`.

There are two possible extreme approaches:
- Implement full substitution support, just as we have it for existing 
TypeVars. This is complicated because TypeVarTuple makes it much harder to 
match up the types correctly. However, it is consistent with the behavior for 
other TypeVar-like objects. My example would turn into GenericAlias(tuple, 
(int, str)).
- Give up on substitution and just return a new GenericAlias object: 
GenericAlias(GenericAlias(tuple, Unpack[Ts]), (int, str). This avoids 
implementing any complex runtime behavior, but it inconsistent with existing 
behavior and less pretty when you print out the type. I prefer this approach 
because there's less risk that future enhancements to typing will break it. I 
also want to explore extending this approach to ParamSpec substitution.

--
assignee: JelleZijlstra
messages: 415100
nosy: JelleZijlstra, gvanrossum, kj, serhiy.storchaka
priority: normal
severity: normal
status: open
title: PEP 646: Decide on substitution behavior
versions: Python 3.11

___
Python tracker 
<https://bugs.python.org/issue47006>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Mehdi2277]

Mehdi2277  added the comment:

Concatenate[int, ...] I would interpret as a function signature with first 
argument int, followed by arbitrary arguments afterwards. I haven't run into 
this case, but ... is allowed in other spots Paramspec is allowed currently.

P = Paramspec("P")

class Foo(Generic[P]):
  ...

Foo[...] # Allowed 
Callable[..., None] # Allowed

Are there any other places a paramspec is allowed? Generic type argument, first 
argument to callable, last to concatenate, anything else?

I'm unaware of any type checking use case for Concatenate[int, ...]. You can 
practically get same thing by using a paramspec variable without using it 
elsewhere so that it captures, but then effectively discards that information.

def use_func1(f: Callable[Concatenate[int, P], None]) -> None:
  ...

def use_func2(f: Callable[Concatenate[int, ...], None]) -> None:
  ...

feels like those two signatures should encode same information.

--
nosy: +med2277

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Jelle Zijlstra]

Jelle Zijlstra  added the comment:

I'm looking at https://github.com/python/cpython/pull/30969 and I'm not sure 
what the motivation for the change is. PEP 612 is quite precise here 
(https://www.python.org/dev/peps/pep-0612/#id1) and allows only a ParamSpec as 
the last argument to Concatenate.

What is the use case for using ... as the last argument? What should it mean to 
a static type checker?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Marc Mueller]

Change by Marc Mueller :


--
nosy: +cdce8p

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Alex Waygood]

Change by Alex Waygood :


--
nosy: +sobolevn

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Alex Waygood]

Change by Alex Waygood :


--
nosy: +AlexWaygood

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Guido van Rossum]

Guido van Rossum  added the comment:

Before you start supporting things that contradict PEP 612 this should be 
discussed on typing-sig (or in the python/typing tracker).

Honestly I'd feel more comfortable if there was agreement on typing-sig with 
your previous PRs in this issue as well; "it does not contradict it" is not a 
ringing endorsement. :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

PR 27518 fixes a substitution of a ParamSpec variable with a Concatenate nad a 
list of types. It is not specified explicitly in PEP 612, but it does not 
contradict it.

PR 30969 makes an ellipsis be valid as the last argument of Concatenate to fix 
a substitution of a ParamSpec variable with an ellipsis. It contradicts with 
PEP 612 which allows only a ParamSpec variable there.

--

___
Python tracker 
<https://bugs.python.org/issue44791>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +29147
pull_request: https://github.com/python/cpython/pull/30969

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Jelle Zijlstra]

Change by Jelle Zijlstra :


--
nosy: +Jelle Zijlstra

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[miss-islington]

miss-islington  added the comment:


New changeset 89db09029566cf3af04b540e33fe1ff9b32f8c8b by Miss Islington (bot) 
in branch '3.10':
bpo-44791: Fix substitution of ParamSpec in Concatenate with different 
parameter expressions (GH-27518)
https://github.com/python/cpython/commit/89db09029566cf3af04b540e33fe1ff9b32f8c8b


--

___
Python tracker 
<https://bugs.python.org/issue44791>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset ecfacc362dd7fef7715dcd94f2e2ca6c622ef115 by Serhiy Storchaka in 
branch 'main':
bpo-44791: Fix substitution of ParamSpec in Concatenate with different 
parameter expressions (GH-27518)
https://github.com/python/cpython/commit/ecfacc362dd7fef7715dcd94f2e2ca6c622ef115


--

___
Python tracker 
<https://bugs.python.org/issue44791>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 3.0 -> 4.0
pull_requests: +29138
pull_request: https://github.com/python/cpython/pull/30959

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44801] Type expression is coerced to a list of parameter arguments in substitution of ParamSpec

[Łukasz Langa]

Change by Łukasz Langa :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44801] Type expression is coerced to a list of parameter arguments in substitution of ParamSpec

[miss-islington]

miss-islington  added the comment:


New changeset 536e35ae6a2555a01f4b51a68ad71dbf7923536d by Miss Islington (bot) 
in branch '3.10':
bpo-44801: Check arguments in substitution of ParamSpec in Callable (GH-27585)
https://github.com/python/cpython/commit/536e35ae6a2555a01f4b51a68ad71dbf7923536d


--

___
Python tracker 
<https://bugs.python.org/issue44801>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44801] Type expression is coerced to a list of parameter arguments in substitution of ParamSpec

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 4.0 -> 5.0
pull_requests: +26099
pull_request: https://github.com/python/cpython/pull/27598

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44801] Type expression is coerced to a list of parameter arguments in substitution of ParamSpec

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 3875a6954741065b136650db67ac533bc70a3eac by Serhiy Storchaka in 
branch 'main':
bpo-44801: Check arguments in substitution of ParamSpec in Callable (GH-27585)
https://github.com/python/cpython/commit/3875a6954741065b136650db67ac533bc70a3eac


--
nosy: +lukasz.langa

___
Python tracker 
<https://bugs.python.org/issue44801>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44801] Type expression is coerced to a list of parameter arguments in substitution of ParamSpec

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +26088
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/27585

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44801] Type expression is coerced to a list of parameter arguments in substitution of ParamSpec

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Thank you Ken Jin. So the problem is the __args__ (x, str) is interpreted 
differently depending on x, and after substituting x the interpretation can be 
changed. If x was ParamSpec, it was interpreted in one way, but if it becomes 
int, it is now interpreted in other way.

The solution is to forbid substitution of P with wrong values (not parameters 
expression). Some normalization is also needed, before and after substitution.

Other related example is:

>>> from typing import *
>>> P = ParamSpec("P")
>>> class Z(Generic[P]): pass
... 
>>> A = Z[[int]]
>>> B = Z[int]
>>> A
__main__.Z[(,)]
>>> B
__main__.Z[int]
>>> A.__args__
((,),)
>>> B.__args__
(,)

It is expected that A and B should the same.

--

___
Python tracker 
<https://bugs.python.org/issue44801>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44793] Arguments ignored in substitution in typing.Callable

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44793] Arguments ignored in substitution in typing.Callable

[miss-islington]

miss-islington  added the comment:


New changeset c8db292012dd84aab81eb3ed9146709696a3d290 by Miss Islington (bot) 
in branch '3.10':
bpo-44793: Fix checking the number of arguments when subscribe a generic type 
with ParamSpec parameter. (GH-27515)
https://github.com/python/cpython/commit/c8db292012dd84aab81eb3ed9146709696a3d290


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44793] Arguments ignored in substitution in typing.Callable

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 3.0 -> 4.0
pull_requests: +26047
pull_request: https://github.com/python/cpython/pull/27537

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44793] Arguments ignored in substitution in typing.Callable

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset f92b9133ef67e77605cbd315b6b6c81036ce110e by Serhiy Storchaka in 
branch 'main':
bpo-44793: Fix checking the number of arguments when subscribe a generic type 
with ParamSpec parameter. (GH-27515)
https://github.com/python/cpython/commit/f92b9133ef67e77605cbd315b6b6c81036ce110e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44802] Substitution does not work after ParamSpec substitution of the user generic with a list of TypeVars

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

If the user generic with ParamSpec parameter substituted with a parametrised 
list containing TypeVar, that TypeVar cannot be substituted.

>>> from typing import *
>>> T = TypeVar("T")
>>> P = ParamSpec("P")
>>> class X(Generic[P]):
... f: Callable[P, int]
... 
>>> Y = X[[int, T]]
>>> Y
__main__.X[(, ~T)]
>>> Y[str]
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
return func(*args, **kwds)
   ^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 1028, in __getitem__
_check_generic(self, params, len(self.__parameters__))
^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 228, in _check_generic
raise TypeError(f"{cls} is not a generic class")

TypeError: __main__.X[(, ~T)] is not a generic class

Expected result equal to X[[int, str]].

--
components: Library (Lib)
messages: 398694
nosy: gvanrossum, kj, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Substitution does not work after ParamSpec substitution of the user 
generic with a list of TypeVars
type: behavior
versions: Python 3.10, Python 3.11

___
Python tracker 
<https://bugs.python.org/issue44802>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44801] Type expression is coerced to a list of parameter arguments in substitution of ParamSpec

[Ken Jin]

Ken Jin  added the comment:

> Type expression is coerced to a list of parameter arguments in substitution 
> of ParamSpec.

It's not, only the repr is like that. Internally it's not coerced.

>>> C[int, str]
typing.Callable[[int], str]
>>> C[int, str].__args__
(, )

Because Callable's correct form is Callable[[type], type] (where type is not 
ParamSpec or Concatenate) so the repr reflects that.

Internally, Callable also flattens the list of args:

>>> Callable[[int, str], int].__args__
(, , )

Because __args__ *must* be hashable and immutable. Otherwise, it will not work 
with Union or Literal. Some time ago we tried converting to nested tuple. But 
that was too difficult (and backwards incompatible) because every other typing 
operation expected __args__ to contain types, not a tuple.

--

___
Python tracker 
<https://bugs.python.org/issue44801>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44801] Type expression is coerced to a list of parameter arguments in substitution of ParamSpec

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
components: +Library (Lib)
dependencies: +Arguments ignored in substitution in typing.Callable
type:  -> behavior
versions: +Python 3.10, Python 3.11

___
Python tracker 
<https://bugs.python.org/issue44801>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44801] Type expression is coerced to a list of parameter arguments in substitution of ParamSpec

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

Type expression is coerced to a list of parameter arguments in substitution of 
ParamSpec. For example:

>>> from typing import *
>>> T = TypeVar('T')
>>> P = ParamSpec('P')
>>> C = Callable[P, T]
>>> C[int, str]
typing.Callable[[int], str]

int becomes [int]. There is even a dedicated test for this.

But it is not followed from PEP 612. Furthermore, it contradicts one of 
examples in the PEP:

>>> class X(Generic[T, P]):
... f: Callable[P, int]
... x: T
... 
>>> X[int, int]  # Should be rejected
__main__.X[int, int]

It makes the implementation (at least the code in issue44796) more complex and 
makes the user code more errorprone.

--
messages: 398687
nosy: gvanrossum, kj, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Type expression is coerced to a list of parameter arguments in 
substitution of ParamSpec

___
Python tracker 
<https://bugs.python.org/issue44801>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +26033
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/27518

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

My understanding is that type expression is valid when substitute TypeVar and 
parameters expression is valid when substitute ParamSpec.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44793] Arguments ignored in substitution in typing.Callable

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

There is also similar bug in Generic:

>>> from typing import *
>>> T = TypeVar("T")
>>> P = ParamSpec("P")
>>> class X(Generic[T, P]):
... f: Callable[P, int]
... x: T
... 
>>> P_2 = ParamSpec("P_2")
>>> X[int, P_2, str]
__main__.X[int, ~P_2]

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44793] Arguments ignored in substitution in typing.Callable

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +26030
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/27515

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44793] Arguments ignored in substitution in typing.Callable

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

If Callable is nested, it works as expected:

>>> typing.List[C][int, str, float]
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
return func(*args, **kwds)
   ^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 1028, in __getitem__
_check_generic(self, params, len(self.__parameters__))
^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 231, in _check_generic
raise TypeError(f"Too {'many' if alen > elen else 'few'} parameters for 
{cls};"

^^^
TypeError: Too many parameters for typing.List[typing.Callable[~P, ~T]]; actual 
3, expected 2

collections.abc.Callable raises an error too.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Ken Jin]

Ken Jin  added the comment:

Should Concatenate support substitution to begin with? PEP 612 doesn't say 
anything, and I am fairly certain it's a special typing form, not a generic. So 
I don't really understand what it means to substitute Concatenate.

Then again, Callable with a nested Concatenate can be substituted, and we 
currently implement that by using Concatenate's substitution behavior as proxy. 
But again, the meaning isn't clear to me.

I also noticed this strange part in PEP 612 about user-defined generic classes:

"`Generic[P]` makes a class generic on `parameters_expressions` (when P is a 
ParamSpec)":

...
class X(Generic[T, P]):
  f: Callable[P, int]
  x: T

def f(x: X[int, Concatenate[int, P_2]]) -> str: ...  # Accepted (KJ: What?)
...

The grammar for `parameters_expression` is:

parameters_expression ::=
  | "..."
  | "[" [ type_expression ("," type_expression)* ] "]"
  | parameter_specification_variable
  | concatenate "["
   type_expression ("," type_expression)* ","
   parameter_specification_variable
"]"

I'm very confused. Does this mean Concatenate is valid when substituting user 
generics? Maybe I should ask the PEP authors?

My general sense when I implemented the PEP was that it was intended primarily 
for static type checking only. IMO, runtime correctness wasn't its concern.

--

___
Python tracker 
<https://bugs.python.org/issue44791>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44793] Arguments ignored in substitution in typing.Callable

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

>>> import typing
>>> T = typing.TypeVar('T')
>>> P = typing.ParamSpec('P')
>>> C = typing.Callable[P, T]
>>> C[int, str, float]
typing.Callable[[int], str]

int substitutes P as [int], str substitutes T, and the third argument float is 
ignored.

--
components: Library (Lib)
messages: 398636
nosy: gvanrossum, kj, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Arguments ignored in substitution in typing.Callable
type: behavior
versions: Python 3.10, Python 3.11

___
Python tracker 
<https://bugs.python.org/issue44793>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44791] Substitution of ParamSpec in Concatenate

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

Substitution of ParamSpec in Concatenate produces weird results:

>>> import typing
>>> P = typing.ParamSpec('P')
>>> typing.Concatenate[str, P][int]
typing.Concatenate[str, int]
>>> typing.Concatenate[str, P][[int]]
typing.Concatenate[str, (,)]
>>> typing.Concatenate[str, P][typing.Concatenate[int, P]]
typing.Concatenate[str, typing.Concatenate[int, ~P]]

But all these results are invalid:

>>> typing.Concatenate[str, int]
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
return func(*args, **kwds)
   ^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 400, in __getitem__
return self._getitem(self, parameters)
   ^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 595, in Concatenate
raise TypeError("The last parameter to Concatenate should be a "

TypeError: The last parameter to Concatenate should be a ParamSpec variable.
>>> typing.Concatenate[str, (int,)]
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
return func(*args, **kwds)
   ^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 400, in __getitem__
return self._getitem(self, parameters)
   ^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 595, in Concatenate
raise TypeError("The last parameter to Concatenate should be a "

TypeError: The last parameter to Concatenate should be a ParamSpec variable.
>>> typing.Concatenate[str, [int]]
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
return func(*args, **kwds)
   ^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 400, in __getitem__
return self._getitem(self, parameters)
   ^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 595, in Concatenate
raise TypeError("The last parameter to Concatenate should be a "

TypeError: The last parameter to Concatenate should be a ParamSpec variable.
>>> typing.Concatenate[str, typing.Concatenate[int, P]]
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
return func(*args, **kwds)
   ^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 400, in __getitem__
return self._getitem(self, parameters)
   ^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 595, in Concatenate
raise TypeError("The last parameter to Concatenate should be a "

TypeError: The last parameter to Concatenate should be a ParamSpec variable.

I expect that

1. The last parameter to Concatenate can be a Concatenate. Inner Concatenate 
should merge with the external one:

Concatenate[str, Concatenate[int, P]] -> Concatenate[str, int, P]
Concatenate[str, P][Concatenate[int, P]] -> Concatenate[str, int, P]

2. The last parameter to Concatenate can be a list of types. The result should 
be a list of types.

Concatenate[str, [int, dict]] -> [str, int, dict]
Concatenate[str, P][[int, dict]] -> [str, int, dict]

3. The last parameter to Concatenate can be an ellipsis.

Concatenate[str, ...]
Concatenate[str, P][...] -> Concatenate[str, ...]

--
components: Library (Lib)
messages: 398632
nosy: gvanrossum, kj, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Substitution of ParamSpec in Concatenate
versions: Python 3.10, Python 3.11

___
Python tracker 
<https://bugs.python.org/issue44791>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Guido van Rossum]

Guido van Rossum  added the comment:

Thanks for taking over here, Ł!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Łukasz Langa]

Łukasz Langa  added the comment:

Thanks Serhiy for the report and the fix, and Ken for reviews.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 21db59fc75b6ebb01bf120a8e5930fe032174f73 by Miss Islington (bot) 
in branch '3.10':
bpo-44653: Support typing types in parameter substitution in the union type. 
(GH-27247) (#27296)
https://github.com/python/cpython/commit/21db59fc75b6ebb01bf120a8e5930fe032174f73


--

___
Python tracker 
<https://bugs.python.org/issue44653>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 5.0 -> 6.0
pull_requests: +25839
pull_request: https://github.com/python/cpython/pull/27296

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 2e3744d50b6e30ea24351e55b4352dcc58fd469e by Serhiy Storchaka in 
branch 'main':
bpo-44653: Support typing types in parameter substitution in the union type. 
(GH-27247)
https://github.com/python/cpython/commit/2e3744d50b6e30ea24351e55b4352dcc58fd469e


--
nosy: +lukasz.langa

___
Python tracker 
<https://bugs.python.org/issue44653>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Guido van Rossum]

Guido van Rossum  added the comment:

I need someone else to own this, sorry.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

There is also problem with other typing types:

>>> (int | T)[typing.List]
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Each union argument must be a type, got typing.List
>>> (int | T)[typing.List[int]]
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Each union argument must be a type, got typing.List[int]
>>> (int | T)[typing.Hashable]
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Each union argument must be a type, got typing.Hashable
>>> (int | T)[typing.Callable[[int], str]]
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Each union argument must be a type, got typing.Callable[[int], str]
>>> (int | T)[typing.ParamSpec('P')]
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Each union argument must be a type, got ~P

Despite the fact that they support the | operator.

We can add one by one special support of different types supporting the | 
operator (ParamSpec, _GenericAlias, _CallableGenericAlias, _UnionGenericAlias, 
_LiteralGenericAlias, _ConcatenateGenericAlias, _AnnotatedAlias, 
_SpecialGenericAlias, _CallableType, _TupleType), but it is cumbersome and 
errorprone. We will need to synchronize code of unionobject.c with typing every 
time we add new kind of types.

PR 27247 uses more general approach. It calls the | operator for arguments 
after substitution. So all types which support the | operator are now 
automatically supported. But the result of parameter substitution can now be 
typing.Union instead of types.Union.

>>> import typing
>>> import collections.abc
>>> T = typing.TypeVar('T')
>>> (int | T)[list]
int | list
>>> (int | T)[typing.List]
typing.Union[int, typing.List]
>>> (int | T)[list[int]]
int | list[int]
>>> (int | T)[typing.List[int]]
typing.Union[int, typing.List[int]]
>>> (int | T)[collections.abc.Hashable]
int | collections.abc.Hashable
>>> (int | T)[typing.Hashable]
typing.Union[int, typing.Hashable]
>>> (int | T)[collections.abc.Sequence[int]]
int | collections.abc.Sequence[int]
>>> (int | T)[typing.Sequence[int]]
typing.Union[int, typing.Sequence[int]]
>>> (int | T)[collections.abc.Callable[[int], str]]
int | collections.abc.Callable[[int], str]
>>> (int | T)[typing.Callable[[int], str]]
typing.Union[int, typing.Callable[[int], str]]
>>> (int | T)[typing.TypeVar('S')]
int | ~S
>>> (int | T)[typing.ParamSpec('P')]
typing.Union[int, ~P]
>>> (int | T)[str | list]
int | str | list
>>> (int | T)[typing.Union[str, list]]
typing.Union[int, str, list]

--

___
Python tracker 
<https://bugs.python.org/issue44653>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +25795
pull_request: https://github.com/python/cpython/pull/27247

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Ken Jin]

Ken Jin  added the comment:

@Serhiy, this doesn't just affect typing.Union, it seems that the rest of the 
typing types don't substitute:

>>> (int | T)[typing.List[str]]
Traceback (most recent call last):
  File "", line 1, in 
TypeError: Each union arg must be a type, got typing.List[str]

We should probably loosen the check during the union_getitem substitution -- no 
need to raise the TypeError or check for is_union, just blindly replace the 
TypeVar. We already do this for types.GenericAlias:

>>> list[T][1]
list[1]

Or if you want to continue checking, maybe checking for PyCallable_Check(obj) 
in substitution is enough - typing internally accepts callable(o) too:

https://github.com/python/cpython/blob/3.10/Lib/typing.py#L146

--

___
Python tracker 
<https://bugs.python.org/issue44653>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +25780
pull_request: https://github.com/python/cpython/pull/27232

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests:  -25757

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Yurii Karabas]

Change by Yurii Karabas <1998uri...@gmail.com>:


--
pull_requests: +25757
pull_request: https://github.com/python/cpython/pull/26980

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests:  -25745

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Yurii Karabas]

Change by Yurii Karabas <1998uri...@gmail.com>:


--
keywords: +patch
nosy: +uriyyo
nosy_count: 3.0 -> 4.0
pull_requests: +25745
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/26980

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44653] Parameter substitution in the union type does not work with typing.Union

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

>>> import typing
>>> T = typing.TypeVar('T')
>>> (int | T)[typing.Union[str, list]]
NotImplemented

See also issue44633. But in this case the expected result is int | str | list 
or typing.Union[init, str, list].

--
components: Interpreter Core
messages: 397624
nosy: gvanrossum, kj, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Parameter substitution in the union type does not work with typing.Union
type: behavior
versions: Python 3.10, Python 3.11

___
Python tracker 
<https://bugs.python.org/issue44653>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25087] Type variable substitution in type instances

[Guido van Rossum]

Guido van Rossum  added the comment:

Yup, see comment in https://github.com/python/typing/pull/308 regarding #156.

--
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25087] Type variable substitution in type instances

[Batuhan]

Batuhan  added the comment:

It looks like the issue in typing tracker closed, can we also close this?

--
nosy: +BTaskaya

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14619] Enhanced variable substitution for databases

[Raymond Hettinger]

Change by Raymond Hettinger :


--
resolution:  -> third party
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: zenity substitution

[Bob Gailer]

On Oct 28, 2018 10:17 PM, "listo factor via Python-list" <
python-list@python.org> wrote:
>
> Hi all,
> I'm new to Python, but not to programming.
>
> As a teaching exercise, I am converting a bunch of bash shell
> scripts to Python, so that they can be run on all three OS-es
> (Linux, Windows, MacOS).
>
> The scripts in questions make extensive use of Linux "zenity"
> dialogs.
>
> Is there an equivalent facility in Python 3? If so, what is it?

Look up zenity in Wikipeda. Scroll down to cross-platform script.

Bob Gailer
-- 
https://mail.python.org/mailman/listinfo/python-list

zenity substitution

[listo factor via Python-list]

Hi all,
I'm new to Python, but not to programming.

As a teaching exercise, I am converting a bunch of bash shell
scripts to Python, so that they can be run on all three OS-es
(Linux, Windows, MacOS).

The scripts in questions make extensive use of Linux "zenity"
dialogs.

Is there an equivalent facility in Python 3? If so, what is it?

TIA
--
https://mail.python.org/mailman/listinfo/python-list

[issue30720] re.sub substitution match group contains wrong value after unmatched pattern was processed

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Atomic groups can help you: '((?>.*?))'.

But this feature is not supported in the re module yet (see issue433030). You 
can use the third-party regex module which is compatible with the re module and 
supports atomic grouping.

>>> import regex as re
>>> pattern = re.compile('((?>.*?))', flags=re.DOTALL)
>>> print(re.sub(pattern, '\\1',
...  'foo123456789\n'
...  'bar\n'))
foo123456789
bar

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30720] re.sub substitution match group contains wrong value after unmatched pattern was processed

[William Budd]

William Budd added the comment:

Doh! This has a really easy solution, doesn't it; just replace "." with "[^<]": 
re.compile('([^<]*?)', flags=re.DOTALL).

Sorry about the noise.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30720] re.sub substitution match group contains wrong value after unmatched pattern was processed

[William Budd]

William Budd added the comment:

I now see you're right of course. Not a bug after all. Thank you.

I mistakenly assumed that the group boundary ")" would delimit the end of the 
non-greedy match group. I.e., ".*?" versus ".*?".

I don't see a way to accomplish the "even less greedy" variant I'm looking for 
though...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30720] re.sub substitution match group contains wrong value after unmatched pattern was processed

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Yes, it is non-greedy. But it needs matching not just '', but ''. 
After finding the first '' it doesn't see the '' after it and 
continue searching until found ''.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30720] re.sub substitution match group contains wrong value after unmatched pattern was processed

[William Budd]

William Budd added the comment:

I don't understand... Isn't the "?" in ".*?" supposed to make the ".*" matching 
non-greedy, hence matching the first "" rather than the last ""?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30720] re.sub substitution match group contains wrong value after unmatched pattern was processed

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

It works correctly. It finds a substring that starts with '' and ends 
with '' from left to right. The leftmost found substring starts from 
index 0 and ends before the final '\n'. Overlapped substrings are not found.

--
nosy: +serhiy.storchaka
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30720] re.sub substitution match group contains wrong value after unmatched pattern was processed

[William Budd]

New submission from William Budd:

pattern = re.compile('(.*?)', flags=re.DOTALL)



# This works as expected in the following case:

print(re.sub(pattern, '\\1',
 'foo\n'
 'bar123456789\n'))

# which outputs:

foo
bar123456789



# However, it does NOT work as I expect in this case:

print(re.sub(pattern, '\\1',
 'foo123456789\n'
 'bar\n'))

# actual output:

foo123456789
bar

# expected output:

foo123456789
bar



It seems that pattern matching/substitution iterations only go haywire once the 
matching iteration immediately prior to it turned out not to be a match. Maybe 
some internal variable is not cleaned up properly in an edge(?) case triggered 
by the example above?

--
components: Regular Expressions
messages: 296506
nosy: William Budd, ezio.melotti, mrabarnett
priority: normal
severity: normal
status: open
title: re.sub substitution match group contains wrong value after unmatched 
pattern was processed
versions: Python 3.6

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue30720>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Manual parameter substitution in sqlite3

[Tim Chase]

On 2017-03-02 09:55, Christian Gollwitzer wrote:
> you could do it "the (theoretical) right way", which is using
> another table. Insert your keys into a table, maybe temporary one,
> and then do
> 
> select * from mumble where key in (select key from keytable)
> 
> In theory that should also be faster, but unless you have a
> thousand keys, there will be no noticeable difference.

It would depend on the frequency with which the same query is
issued.  Bulk inserts, followed by the query, followed by cleaning up
the data would almost certainly be slower.  Bulk inserting the data
and then issuing thousands of queries against it could prove much
faster (though as with all things performance-related, test, don't
guess). But if the set of criteria changes frequently, the overhead
of inserting/deleting the data will almost certainly swamp any
potential speed-up.

-tkc





-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Manual parameter substitution in sqlite3

[Christian Gollwitzer]

Am 28.02.17 um 18:28 schrieb Skip Montanaro:

Most of the time (well, all the time if you're smart), you let the
database adapter do parameter substitution for you to avoid SQL
injection attacks (or stupid users). So:

curs.execute("select * from mumble where key = ?", (key,))

If you want to select from several possible keys, it would be nice to
be able to do this:

curs.execute("select * from mumble where key in (?)", (keys,))

but that doesn't work.


Instead of building the query on the fly (with quoting hell etc.), you 
could do it "the (theoretical) right way", which is using another table. 
Insert your keys into a table, maybe temporary one, and then do


select * from mumble where key in (select key from keytable)

In theory that should also be faster, but unless you have a thousand 
keys, there will be no noticeable difference.


Christian

--
https://mail.python.org/mailman/listinfo/python-list

Re: Manual parameter substitution in sqlite3

[Serhiy Storchaka]

On 28.02.17 19:28, Skip Montanaro wrote:

Most of the time (well, all the time if you're smart), you let the
database adapter do parameter substitution for you to avoid SQL
injection attacks (or stupid users). So:

curs.execute("select * from mumble where key = ?", (key,))

If you want to select from several possible keys, it would be nice to
be able to do this:

curs.execute("select * from mumble where key in (?)", (keys,))

but that doesn't work. Instead, you need to do your own parameter
substitution. The quick-and-insecure way to do this is:

curs.execute("select * from mumble where key in (%s)" %
",".join([repr(k) for k in keys]))

I'm pretty sure that's breakable.


curs.execute("select * from mumble where %s" %
 " or ".join(["key = ?"] * len(keys)),
 *keys)


--
https://mail.python.org/mailman/listinfo/python-list

Re: Manual parameter substitution in sqlite3

[Tim Chase]

On 2017-03-01 04:40, Chris Angelico wrote:
> curs.execute("select * from mumble where key in (" +
> ",".join(["?"]*len(keys)) + ")", keys)
> 
> If this is combined with another parameter, it'd be messier, but you
> could do something like:
> 
> curs.execute("select * from mumble where key in (" +
> ",".join(["?"]*len(keys)) + ") and category = ?",
> tuple(keys) + (cat,))
> 
> Either way, you're still letting the sqlite connector do the
> processing of the elements, but handling the collection yourself.

These are the methods I use both with sqlite and with the SQL Server
adapter (at $DAYJOB).  Works great and also pretty easy to debug if
you preformulate the strings into a variable:

  PLACEHOLDER = "?" # SQL Server's place-holder

  params = [3, 1, 4, 5, 9]
  field2 = "Some value"

  sql = """
SELECT *
FROM tbl
WHERE
  field1 IN (%(all_placeholders)s)
  AND field2 = %(placeholder)s
""" % dict(
 placeholders=",".join(
   PLACEHOLDER
   for _
   in params
   ),
 placeholder=PLACEHOLDER,
 )
  cur.execute(sql, params + [field2,])

-tkc






-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Manual parameter substitution in sqlite3

[Neil Cerutti]

On 2017-02-28, Skip Montanaro <skip.montan...@gmail.com> wrote:
> Some database adapters provide a function to do explicit
> substitution (e.g., mySQLdb.escape, psycopg2._param_escape),
> but the sqlite3 adapter doesn't.

It's clunky but you can use sqlite's core "quote" function.

quote(X)

   The quote(X) function returns the text of an SQL literal which
   is the value of its argument suitable for inclusion into an
   SQL statement. Strings are surrounded by single-quotes with
   escapes on interior quotes as needed. BLOBs are encoded as
   hexadecimal literals. Strings with embedded NUL characters
   cannot be represented as string literals in SQL and hence the
   returned string literal is truncated prior to the first NUL. 

The time I found it useful was when I thought I needed to build a
query with parameters where the DB-API didn't allow them, e.g.,
in the column-names portion of an INSERT statement.

quoted_val, = c.execute("SELECT quote(?);", (val,)).fetchone()

-- 
Neil Cerutti

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Manual parameter substitution in sqlite3

[Skip Montanaro]

On Tue, Feb 28, 2017 at 12:42 PM, Chris Angelico  wrote:
> That isn't what you were doing in your post, so it seemed worth
> mentioning.

Sorry, my original post was a bit abbreviated. I can't copy text from
inside to outside, so have to retype everything. I guess I missed
that.

S
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Manual parameter substitution in sqlite3

[Chris Angelico]

On Wed, Mar 1, 2017 at 5:40 AM, Skip Montanaro  wrote:
> On Tue, Feb 28, 2017 at 11:40 AM, Chris Angelico  wrote:
>> Testing with PostgreSQL (which *does* transform lists) suggests that
>> "in" doesn't work; I used "key = any(%s)". I'd try that with sqlite3
>> first, just in case it makes a difference. Probably it won't, but
>> worth a try.
>
> Yeah, doesn't work in Sqlite. It doesn't appear to have an any() function.

Not surprised, but worth a try.

>> Third recommendation: Instead of making yourself completely
>> vulnerable, just go one level in:
>
> That's what I was doing.
>

That isn't what you were doing in your post, so it seemed worth
mentioning. Sounds like you have the best available options already at
your fingertips. Have at it!

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Manual parameter substitution in sqlite3

[Skip Montanaro]

On Tue, Feb 28, 2017 at 11:40 AM, Chris Angelico  wrote:
> Testing with PostgreSQL (which *does* transform lists) suggests that
> "in" doesn't work; I used "key = any(%s)". I'd try that with sqlite3
> first, just in case it makes a difference. Probably it won't, but
> worth a try.

Yeah, doesn't work in Sqlite. It doesn't appear to have an any() function.

> Second recommendation: Switch to PostgreSQL, because then this happens
> automatically :)

It's on my likely list of upgrades. Sqlite3 was just easier to start with.

> Third recommendation: Instead of making yourself completely
> vulnerable, just go one level in:

That's what I was doing.

On Tue, Feb 28, 2017 at 12:05 PM, Peter Otten <__pete...@web.de> wrote:
> If all else fails use quote()
>
> http://www.sqlite.org/lang_corefunc.html#quote

This looks like it will do the trick nicely. I was only looking at the
API for the Python package, not considering the Sqlite3 core. To allow
simpler migration to another database in the future, I'll likely hide
your example in a function.

Thanks for the responses.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Manual parameter substitution in sqlite3

[Peter Otten]

Skip Montanaro wrote:

> Most of the time (well, all the time if you're smart), you let the
> database adapter do parameter substitution for you to avoid SQL
> injection attacks (or stupid users). So:
> 
> curs.execute("select * from mumble where key = ?", (key,))
> 
> If you want to select from several possible keys, it would be nice to
> be able to do this:
> 
> curs.execute("select * from mumble where key in (?)", (keys,))
> 
> but that doesn't work. Instead, you need to do your own parameter
> substitution. The quick-and-insecure way to do this is:
> 
> curs.execute("select * from mumble where key in (%s)" %
> ",".join([repr(k) for k in keys]))
> 
> I'm pretty sure that's breakable.
> 
> Some database adapters provide a function to do explicit substitution
> (e.g., mySQLdb.escape, psycopg2._param_escape), but the sqlite3
> adapter doesn't. Is there a function floating around out there which
> does the right thing, allowing you to safely construct these sorts of
> set inclusion clauses?
> 
> Thx,
> 
> Skip

If all else fails use quote()

http://www.sqlite.org/lang_corefunc.html#quote

>>> db = sqlite3.connect(":memory:")
>>> cs = db.cursor()
>>> next(cs.execute("select quote(?)", ("foo 'bar' \"baz\"",)))[0]
'\'foo \'\'bar\'\' "baz"\''
>>> print(_)
'foo ''bar'' "baz"'

With the documented limitation:

>>> next(cs.execute("select quote(?)", ("foo 'bar'\0 \"baz\"",)))[0]
"'foo ''bar'''"


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Manual parameter substitution in sqlite3

[Chris Angelico]

On Wed, Mar 1, 2017 at 4:28 AM, Skip Montanaro <skip.montan...@gmail.com> wrote:
> Some database adapters provide a function to do explicit substitution
> (e.g., mySQLdb.escape, psycopg2._param_escape), but the sqlite3
> adapter doesn't. Is there a function floating around out there which
> does the right thing, allowing you to safely construct these sorts of
> set inclusion clauses?

Testing with PostgreSQL (which *does* transform lists) suggests that
"in" doesn't work; I used "key = any(%s)". I'd try that with sqlite3
first, just in case it makes a difference. Probably it won't, but
worth a try.

Second recommendation: Switch to PostgreSQL, because then this happens
automatically :)

Third recommendation: Instead of making yourself completely
vulnerable, just go one level in:

curs.execute("select * from mumble where key in (" +
",".join(["?"]*len(keys)) + ")", keys)

If this is combined with another parameter, it'd be messier, but you
could do something like:

curs.execute("select * from mumble where key in (" +
",".join(["?"]*len(keys)) + ") and category = ?",
tuple(keys) + (cat,))

Either way, you're still letting the sqlite connector do the
processing of the elements, but handling the collection yourself.

I may or may not have needed to do this once before with MySQL
but I'm not going to admit to it. Because I also had to use PHP to
talk to the same database. And I don't admit to knowing PHP.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Manual parameter substitution in sqlite3

[Skip Montanaro]

Most of the time (well, all the time if you're smart), you let the
database adapter do parameter substitution for you to avoid SQL
injection attacks (or stupid users). So:

curs.execute("select * from mumble where key = ?", (key,))

If you want to select from several possible keys, it would be nice to
be able to do this:

curs.execute("select * from mumble where key in (?)", (keys,))

but that doesn't work. Instead, you need to do your own parameter
substitution. The quick-and-insecure way to do this is:

curs.execute("select * from mumble where key in (%s)" %
",".join([repr(k) for k in keys]))

I'm pretty sure that's breakable.

Some database adapters provide a function to do explicit substitution
(e.g., mySQLdb.escape, psycopg2._param_escape), but the sqlite3
adapter doesn't. Is there a function floating around out there which
does the right thing, allowing you to safely construct these sorts of
set inclusion clauses?

Thx,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

tox-2.3.1: fix cross-section substitution for setenv

[holger krekel]

just did a quick bugfix release, tox-2.3.1, which re-allows cross-section
substitution for setenv.  Thanks all for the patience and the reporting.

For docs, as always see http://testrun.org/tox

have fun,
holger
-- 
https://mail.python.org/mailman/listinfo/python-announce-list

Support the Python Software Foundation:
http://www.python.org/psf/donations/

[issue25087] Type variable substitution in type instances

[Stefan Krah]

New submission from Stefan Krah:

If a type scheme is instantiated, should the type variables in the
class body be substituted?  This is an example (typed by hand on
a locked down Windows machine, may contain errors):


alpha = TypeVar('alpha')
beta = TypeVar('beta')

class ABTuple(Generic[alpha, beta]):
def __init__(self, a : alpha, b : beta):
self.value = (a, b)

get_type_hints(ABTuple.__init__)
 ==> {'b': ~beta, 'a': ~alpha}


IntIntTuple = ABTuple[int, int]

IntIntTuple
 ==> __main__.ABTuple[int, int]

get_type_hints(IntIntTuple.__init__)
{'b': ~beta, 'a': ~alpha}
  ^^  ^^


Since the type has been specialized, these should ideally be 'int'.

--
components: Interpreter Core
messages: 250565
nosy: gvanrossum, skrah
priority: normal
severity: normal
status: open
title: Type variable substitution in type instances
type: behavior
versions: Python 3.5

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue25087>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25087] Type variable substitution in type instances

[Guido van Rossum]

Guido van Rossum added the comment:

Good question. Let's discuss this in the type hinting / pep 484 tracker:

https://github.com/ambv/typehinting/issues/156

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Regular expression and substitution, unexpected duplication

[Laurent Pointal]

MRAB wrote:

 On 2015-08-18 22:42, Laurent Pointal wrote:
 Hello,
zip
 ellipfind_re = re.compile(r((?=\.\.\.)|…), re.IGNORECASE|re.VERBOSE)
 ellipfind_re.sub(' ... ',
 C'est un essai... avec différents caractères… pour voir.)

 (?=...) is a lookahead; a non-capture group is (?:...).
 
 The regex should be r((?:\.\.\.)|…), which can be simplified to just
 r\.\.\.|… for your use-case. (You don't need the
 re.IGNORECASE|re.VERBOSE either!)

Thanks,

I made same mistake in another place, but dont see it here.

A+
Laurent.


-- 
https://mail.python.org/mailman/listinfo/python-list

Regular expression and substitution, unexpected duplication

[Laurent Pointal]

Hello,

I want to make a replacement in a string, to ensure that ellipsis are 
surrounded by spaces (this is not a typographycal problem, but a preparation 
for late text chunking).

I tried with regular expressions and the SRE_Pattern.sub() method, but I 
have an unexpected duplication of the replacement pattern:


The code:

ellipfind_re = re.compile(r((?=\.\.\.)|…), re.IGNORECASE|re.VERBOSE)
ellipfind_re.sub(' ... ', 
   C'est un essai... avec différents caractères… pour voir.)

And I retrieve:

C'est un essai ... ... avec différents caractères ...  pour voir.
^^^

I tested with/without group capture, same result.

My Python version:
Python 3.4.3 (default, Mar 26 2015, 22:03:40) 
[GCC 4.9.2] on linux

Any idea ?

Thanks.
Laurent.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Regular expression and substitution, unexpected duplication

[MRAB]

On 2015-08-18 22:42, Laurent Pointal wrote:

Hello,

I want to make a replacement in a string, to ensure that ellipsis are
surrounded by spaces (this is not a typographycal problem, but a preparation
for late text chunking).

I tried with regular expressions and the SRE_Pattern.sub() method, but I
have an unexpected duplication of the replacement pattern:


The code:

ellipfind_re = re.compile(r((?=\.\.\.)|…), re.IGNORECASE|re.VERBOSE)
ellipfind_re.sub(' ... ',
C'est un essai... avec différents caractères… pour voir.)

And I retrieve:

C'est un essai ... ... avec différents caractères ...  pour voir.
 ^^^

I tested with/without group capture, same result.

My Python version:
Python 3.4.3 (default, Mar 26 2015, 22:03:40)
[GCC 4.9.2] on linux

Any idea ?


(?=...) is a lookahead; a non-capture group is (?:...).

The regex should be r((?:\.\.\.)|…), which can be simplified to just
r\.\.\.|… for your use-case. (You don't need the
re.IGNORECASE|re.VERBOSE either!)

--
https://mail.python.org/mailman/listinfo/python-list

Re: Regex substitution trouble

[Eugene]

massi_...@msn.com wrote:

 Hi everyone,
 
 I'm not really sure if this is the right place to ask about regular
 expressions, but since I'm usin python I thought I could give a try 
:-)
 Here is the problem, I'm trying to write a regex in order to 
substitute
 all the occurences in the form $somechars with another string. This 
is
 what I wrote:
 
 newstring = re.sub(ur(?u)(\$\[\s\w]+\), subst, oldstring)
 
 This works pretty well, but it has a problem, I would need it also to
 handle the case in which the internal string contains the double 
quotes,
 but only if preceeded by a backslash, that is something like
 $somechars_with\\doublequotes. Can anyone help me to correct it?
 
 Thanks in advance!
Hi!

Next snippet works for me:

re.sub(r'\$([\s\w]+(\\)*[\s\w]+)+', 'noop', r'$te\sts\tri\ng')

-- 
https://mail.python.org/mailman/listinfo/python-list

Regex substitution trouble

[massi_srb]

Hi everyone,

I'm not really sure if this is the right place to ask about regular 
expressions, but since I'm usin python I thought I could give a try :-)
Here is the problem, I'm trying to write a regex in order to substitute all the 
occurences in the form $somechars with another string. This is what I wrote:

newstring = re.sub(ur(?u)(\$\[\s\w]+\), subst, oldstring)

This works pretty well, but it has a problem, I would need it also to handle 
the case in which the internal string contains the double quotes, but only if 
preceeded by a backslash, that is something like 
$somechars_with\\doublequotes.
Can anyone help me to correct it?

Thanks in advance!
-- 
https://mail.python.org/mailman/listinfo/python-list