Hi Marten,
All valid questions; there are indeed a lot of moving parts in Python
typing, and NumPy's stubs don't make it any easier, to say the least :P
You mention:
>
> > # missing scalar type
>
> > I think it's important that we also introduce a companion scalar type
> > for this, instead of using the `builtins.bytes` for this.
>
> To me this seems somewhat orthogonal to the NEP. If it is important for
> `ByteStringDType`, it is even more important for `StringDType`, which
> will be substantially more used, and presumably the solution would be
> the same for both (some franken-subclass of `bytes` or `str` and
> `np.generic`, just like `np.float64` (which subclasses both `float` and
> `np.generic`; its mro is `np.floating`, `np.inexact`, `np.number`,
> `np.generic`, `float`, `object`).
The missing scalar type for StringDType is indeed a painful problem right
now for static typing.
I tried my best to make the most out of it, but this required adding a
bunch of special-casing
(in the form of overloads) to function stubs that support StringDType, as
well as several
# type: ignore comments to silence type-checker (rightly) errors where str
is now used in places
where only numpy.generic subtypes are allowed. This was quite a lot of
work, and made the stubs
even more complicated than they already were. And today, there are still
functions (e.g. np.min, iirc)
that don't support StringDType because they don't have special-cased
workaround overloads yet.
So BytesDType will come without a scalar type, we'll also have to apply
similar workarounds for it,
and the stubs will become an even bigger mess.
Last meeting Nathan told me that adding these scalar types is a very tricky
thing though, so I realize
that it's a big ask. But if BytesDType gets a scalar type, then I believe
it's indeed not that much more
difficult to also add the missing StringDType scalar type, which would make
this a 2-for-1 deal :)
> Currently `StringDType` is simply type-unsafe, and it's impossible to
> > express a `StringDType` array using the widely used
> > `numpy.typing.NDArray`. For example, this leads to `f(x:
> > npt.NDArray[np.generic])` rejecting every `StringDType` array, even
> > though `npt.NDArray[np.generic]` is supposed to represent the "top
> > type" of `ndarray`.
>
> As a non-typer, this feels weird. It suggests `NDArray` is treated a
> bit like a list, but then what is the problem with `NDArray[str]`? Or
> why would one not write `NDArray[Any]`?
>
> Maybe more broadly, why treat it like a list? Not all dtypes will have
> associated scalars; e.g., the SFloatDescr that is one of numpy's test
> cases, does not have a scalar type at all (I'm building on that example
> for astropy, so it has real-world uses).
>
> Indeed, the concept of scalars associated with dtypes is inconsistent
> with the Array API. At some level, it would be rather lovely if we
> could get rid of them altogether...
>
> I guess to ask it differently, why can one not write it with the dtype,
> i.e., `NDArray[StringDType]` so that the most general form would be
> `NDArray[np.dtype]`?
>
> Now probably this has all been discussed to death already, but I guess
> the general question is whether rather than try to adjust fairly logical
> choices to typing, it is possible to expand what typing can do so that
> it can express those choices...
Today, numpy.typing.NDArray is defined as:
type NDArray[ScalarT: np.generic] = np.ndarray[_AnyShape, np.dtype[ScalarT]]
The important bit here is that the ScalarT type parameter is restricted to
(subtypes of) np.generic.
This means that NDArray[str] or NDArray[bytes] are not allowed (by static
type-checkers).
Forcing it by ignoring the resulting typing errors is possible, but results
in undefined behavior. So we,
and downstream users, are now not able to use NDArray to express StringDType
or BytesDType arrays,
even though that's precisely what NDArray is intended for. So the only
option is to spell it out in full as
np.ndarray[_AnyShape, np.dtypes.BytesDType] (where _AnyShape = tuple[Any,
...]), which is
clearly verbose, and I don't expect many downstream users to even consider
it an option. Because, after all,
for any other dtype, NDArray is all you need.
Removing the bound on ScalarT might seem like an obvious solution here, but
doing so would be a breaking
change for many downstream libraries, and a backwards-incompatible one at
that. Because, for example, where
it currently is always possible to assign an element of an NDArray to some x:
np.generic, that will then result in
a type error, as that would indeed be type-unsafe.
You suggested an alternative to this, where we'd allow writing
NDArray[StringDType]. But that would require
some form of pattern matching in the declaration of type NDArray[ScalarT:
np.generic | np.dtype] to
deal with the two possible types that ScalarT could then resolve to.
Perhaps this will be possible to express in
the future, but for now it's not an option I'm afraid.
> # na_object
>
> > As you probably already know, this feature of `StringDType` is
> > problematic for static typing, because there is no good way to express
> > this functionality in the stubs. And although I understand that it
> > would be strange if the direct dual to `StringDType` wouldn't have the
> > same `na_object` functionality, I'd rather we not repeat the mistakes
> > of the past, taking the resulting inconsistency for granted.
>
> I'm a bit confused about this one. How is this different from, e.g.,
> the concept of byte order, which is not captured by the typing either?
> Also, pandas has particular integer values to indicate missing. Isn't
> that similar? Can that be captured by typing?
NumPy's stubs and the numpy.typing types have been built around the idea
that the elements of arrays can be
fully statically described in terms of their scalar type. But the na_object,
which can be any object, is defined on
the *dtype itself.* So by only considering the scalar type, as we always
have, we lose that na_object information,
i.e. which type it can be. A dedicated scalar type could solve this,
because we could make it a generic type with
the type of na_object as type parameter.
But what a dedicated scalar type wouldn't help with, is the problem that
na_object can be *any* object, even
scalar-like objects such as float that's already associated with float64 scalar
types everywhere in the stubs.
So this introduces ambiguity in the inference of functions accepting
array-likes like np.array. This is why the
na_object of StringDType isn't well supported in the stubs at the moment.
In the last community call we talked about a potential solution for this
ambiguity: restricting the na_object to a
limited set of allowed types, e.g. None, and/or a dedicated sentinel or
some enum. Nathan suggested including
float, specifically float("nan"). But the issue with that is that, unlike
e.g. int and str, float("nan") cannot
be expressed as a typing.Literal. This might be something that could be
changed in the typing spec, but I'll
have to investigate a bit more to see how feasible that would be.
I hope that answers your questions :)
Cheers,
Joren
On Wed, 26 Aug 2026 at 00:07, Marten van Kerkwijk via NumPy-Discussion <
[email protected]> wrote:
> Hi Joren,
>
> As a person who doesn't use typing, some perhaps naive questions about
> your comments, but ones that express my worry that the current state of
> typing, which would seem a fast-moving and still relatively in-flux
> feature, starts to influence the convenience of code.
>
> You mention:
>
> > # missing scalar type
>
> > I think it's important that we also introduce a companion scalar type
> > for this, instead of using the `builtins.bytes` for this.
>
> To me this seems somewhat orthogonal to the NEP. If it is important for
> `ByteStringDType`, it is even more important for `StringDType`, which
> will be substantially more used, and presumably the solution would be
> the same for both (some franken-subclass of `bytes` or `str` and
> `np.generic`, just like `np.float64` (which subclasses both `float` and
> `np.generic`; its mro is `np.floating`, `np.inexact`, `np.number`,
> `np.generic`, `float`, `object`).
>
> > Currently `StringDType` is simply type-unsafe, and it's impossible to
> > express a `StringDType` array using the widely used
> > `numpy.typing.NDArray`. For example, this leads to `f(x:
> > npt.NDArray[np.generic])` rejecting every `StringDType` array, even
> > though `npt.NDArray[np.generic]` is supposed to represent the "top
> > type" of `ndarray`.
>
> As a non-typer, this feels weird. It suggests `NDArray` is treated a
> bit like a list, but then what is the problem with `NDArray[str]`? Or
> why would one not write `NDArray[Any]`?
>
> Maybe more broadly, why treat it like a list? Not all dtypes will have
> associated scalars; e.g., the SFloatDescr that is one of numpy's test
> cases, does not have a scalar type at all (I'm building on that example
> for astropy, so it has real-world uses).
>
> Indeed, the concept of scalars associated with dtypes is inconsistent
> with the Array API. At some level, it would be rather lovely if we
> could get rid of them altogether...
>
> I guess to ask it differently, why can one not write it with the dtype,
> i.e., `NDArray[StringDType]` so that the most general form would be
> `NDArray[np.dtype]`?
>
> Now probably this has all been discussed to death already, but I guess
> the general question is whether rather than try to adjust fairly logical
> choices to typing, it is possible to expand what typing can do so that
> it can express those choices...
>
> > # na_object
>
> > As you probably already know, this feature of `StringDType` is
> > problematic for static typing, because there is no good way to express
> > this functionality in the stubs. And although I understand that it
> > would be strange if the direct dual to `StringDType` wouldn't have the
> > same `na_object` functionality, I'd rather we not repeat the mistakes
> > of the past, taking the resulting inconsistency for granted.
>
> I'm a bit confused about this one. How is this different from, e.g.,
> the concept of byte order, which is not captured by the typing either?
> Also, pandas has particular integer values to indicate missing. Isn't
> that similar? Can that be captured by typing?
>
> Again, apologies for what are probably naive questions...
>
> All the best,
>
> Marten
>
> _______________________________________________
> NumPy-Discussion mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3//lists/numpy-discussion.python.org
> Member address: [email protected]
>
_______________________________________________
NumPy-Discussion mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3//lists/numpy-discussion.python.org
Member address: [email protected]
