iamkhav opened a new issue, #56141: URL: https://github.com/apache/spark/issues/56141
## Description A static typechecker can't evaluate `is_remote_only()` which makes the type annotation of relevant `DataFrame` methods be a `Union` of the property method or a `Column` (because of `__getattr__`). https://github.com/apache/spark/blob/4a61083f22e605fa0a4c73915499514200902b79/python/pyspark/sql/dataframe.py#L173-L191 I read the reasoning on the PR #45053. In addition, I think it makes sense to add logic for static typecheckers. ## Example ```python _ = df.rdd.flatMap(some_fn) ``` Typechecker doesn't know if `rdd` is the callable property `rdd` or a `Column` returned by `__getattr__`. Calling `flatMap()` will make the typechecker throw an error because it doesn't know if the `Callable` or `Column` was returned. Tested with https://github.com/astral-sh/ty and `pyspark==4.1.2`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
