Philip Kahn created SPARK-47997:
-----------------------------------
Summary: Pandas-on-Spark incompletely implements DataFrame.drop
Key: SPARK-47997
URL: https://issues.apache.org/jira/browse/SPARK-47997
Project: Spark
Issue Type: Bug
Components: Pandas API on Spark
Affects Versions: 3.4.3
Reporter: Philip Kahn
For Pandas v1.0+, `drop` supports the `errors` kwarg:
[https://pandas.pydata.org/pandas-docs/version/1.0/reference/api/pandas.DataFrame.drop.html]
Pandas-on-Spark does not implement it. This is especially glaring since the
pyspark drop is a no-op on absent columns, behaving like `errors='ignore'`, so
_extra_ work needed to be done to implement the raise behaviour.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
