Julian-J-S opened a new issue, #55900:
URL: https://github.com/apache/spark/issues/55900
Using pyspark we always try to use concrete types everywhere instead of
strings to improve quality and maintainability.
We recently came across this:
```python
from pyspark.sql import functions as F
from pyspark.sql import types as T
_ = F.from_json(
F.col("value"),
schema=T.MapType(T.StringType(), T.StringType()) # >>> TYPE ERROR
)
```
This code works! ✅
However, type checkers are not happy: 🛑
Type Error: Argument to function `from_json` is incorrect: Expected
`ArrayType | StructType | Column | str`, found `MapType`.
Is there a reason that `MapType` is not included as a type hint?
There is even a concrete example using `MAP` in the function documentation
of `from_json`:
> df.select(sf.from_json(df.value, "MAP<STRING,INT>").alias("json")).show()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
