[
https://issues.apache.org/jira/browse/SPARK-35806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haejoon Lee updated SPARK-35806:
Description:
pandas and pandas-on-Spark both have an argument named `mode` in the
[DataFrame.to_csv,
|https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.to_csv.html]but
the acceptable strings are different.
pandas can accept the "w", "w+", "a", "a+" where as pandas-on-Spark can accept
"append", "overwrite", "ignore", "error" or "errorifexists".
We should map these acceptable strings to pandas.
e.g. "w" will work as Spark's "overwrite". In addition, mode can take Spark's
"overwrite" too.
was:
pandas and pandas-on-Spark both have a argument named `mode` in the
[DataFrame.to_csv.|https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.to_csv.html]
And pandas has same argument, but the acceptable strings are different.
So, we should map the acceptable string to pandas.
e.g. mode=w will work as Spark's overwrite. In addition, mode can take Spark's
overwrite too.
> Mapping the `mode` argument to pandas in DataFrame.to_csv
> -
>
> Key: SPARK-35806
> URL: https://issues.apache.org/jira/browse/SPARK-35806
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 3.2.0
>Reporter: Haejoon Lee
>Priority: Major
>
> pandas and pandas-on-Spark both have an argument named `mode` in the
> [DataFrame.to_csv,
> |https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.to_csv.html]but
> the acceptable strings are different.
> pandas can accept the "w", "w+", "a", "a+" where as pandas-on-Spark can
> accept "append", "overwrite", "ignore", "error" or "errorifexists".
> We should map these acceptable strings to pandas.
> e.g. "w" will work as Spark's "overwrite". In addition, mode can take Spark's
> "overwrite" too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
