Hi there,


I believe I have found an error with the type hints for `DataFrame.drop` in
pyspark. The first overload at
https://github.com/apache/spark/blob/0bc38acc615ad411a97779c6a1ff43d4391c0c3d/python/pyspark/sql/dataframe.py#L5559-L5568
isn’t
as a `*args` argument, and therefore doesn’t allow specifying multiple
`Columns` in `drop`. Additionally, according to the python docs
<https://docs.python.org/3/library/typing.html#typing.overload>, type hints
on the non-overloaded definition should be ignored by a type checker and so
mixing specifying the `str` name of a field with `Column` expressions in
the final `drop` definition should not be used by a type checker. However,
the code in both the connect
<https://github.com/apache/spark/blob/0bc38acc615ad411a97779c6a1ff43d4391c0c3d/python/pyspark/sql/connect/dataframe.py#L489-L496>
 and classic
<https://github.com/apache/spark/blob/0bc38acc615ad411a97779c6a1ff43d4391c0c3d/python/pyspark/sql/classic/dataframe.py#L1712-L1720>
implementation doesn't
appear to have any issues with mixing `Columns` with `str`, nor specifying
multiple `Column`s, so I think the overloads here are unnecessary
altogether and the final declaration is sufficient asis.


Is my understanding correct? Or is there something more that I'm missing as
to why that is typed like that?


Thanks,

Olly

Reply via email to