This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 1bb9e332faa [SPARK-40117][PYTHON][SQL] Convert condition to java in
DataFrameWriterV2.overwrite
1bb9e332faa is described below
commit 1bb9e332faadd0fe8650285874b7f320d60405e6
Author: Wenli Looi <wl...@ucalgary.ca>
AuthorDate: Wed Aug 17 15:28:55 2022 +0900
[SPARK-40117][PYTHON][SQL] Convert condition to java in
DataFrameWriterV2.overwrite
### What changes were proposed in this pull request?
Fix DataFrameWriterV2.overwrite() fails to convert the condition parameter
to java. This prevents the function from being called.
It is caused by the following commit that deleted the `_to_java_column`
call instead of fixing it:
https://github.com/apache/spark/commit/a1e459ed9f6777fb8d5a2d09fda666402f9230b9
### Why are the changes needed?
DataFrameWriterV2.overwrite() cannot be called.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually checked whether the arguments are sent to JVM or not.
Closes #37547 from looi/fix-overwrite.
Authored-by: Wenli Looi <wl...@ucalgary.ca>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
(cherry picked from commit 46379863ab0dd2ee8fcf1e31e76476ff18397f60)
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
python/pyspark/sql/readwriter.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index 9529cf0fdae..cfa2f3fffb0 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -1570,6 +1570,7 @@ class DataFrameWriterV2(object):
Overwrite rows matching the given filter condition with the contents
of the data frame in
the output table.
"""
+ condition = _to_java_column(condition)
self._jwriter.overwrite(condition)
@since(3.1)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
