(spark-connect-swift) branch main updated: [SPARK-52151] `DataFrameWriterV2.overwrite` should handle `condition`

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git


The following commit(s) were added to refs/heads/main by this push:
     new 89b4b7d  [SPARK-52151] `DataFrameWriterV2.overwrite` should handle 
`condition`
89b4b7d is described below

commit 89b4b7db9c221a67b589519092a53708e5c3476f
Author: Dongjoon Hyun <dongj...@apache.org>
AuthorDate: Wed May 14 19:59:46 2025 -0700

    [SPARK-52151] `DataFrameWriterV2.overwrite` should handle `condition`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix `DataFrameWriterV2.overwrite` to handle the `condition` 
parameter.
    
    ### Why are the changes needed?
    
    This was missed at the initial implementation.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, but this is a bug fix.
    
    ### How was this patch tested?
    
    Pass the CIs. Apache Iceberg CI will be added soon.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #149 from dongjoon-hyun/SPARK-52151.
    
    Authored-by: Dongjoon Hyun <dongj...@apache.org>
    Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
---
 Sources/SparkConnect/DataFrameWriterV2.swift | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/Sources/SparkConnect/DataFrameWriterV2.swift 
b/Sources/SparkConnect/DataFrameWriterV2.swift
index 55b8503..6deac16 100644
--- a/Sources/SparkConnect/DataFrameWriterV2.swift
+++ b/Sources/SparkConnect/DataFrameWriterV2.swift
@@ -109,7 +109,7 @@ public actor DataFrameWriterV2: Sendable {
   /// output table.
   /// - Parameter condition: A filter condition.
   public func overwrite(condition: String) async throws {
-    try await executeWriteOperation(.overwrite)
+    try await executeWriteOperation(.overwrite, condition)
   }
 
   /// Overwrite all partition for which the ``DataFrame`` contains at least 
one row with the contents
@@ -120,7 +120,10 @@ public actor DataFrameWriterV2: Sendable {
     try await executeWriteOperation(.overwritePartitions)
   }
 
-  private func executeWriteOperation(_ mode: WriteOperationV2.Mode) async 
throws {
+  private func executeWriteOperation(
+    _ mode: WriteOperationV2.Mode,
+    _ overwriteCondition: String? = nil
+  ) async throws {
     var write = WriteOperationV2()
 
     let plan = await self.df.getPlan() as! Plan
@@ -139,6 +142,9 @@ public actor DataFrameWriterV2: Sendable {
     for property in self.tableProperties.toStringDictionary() {
       write.tableProperties[property.key] = property.value
     }
+    if let overwriteCondition {
+      write.overwriteCondition = overwriteCondition.toExpression
+    }
     write.mode = mode
 
     var command = Spark_Connect_Command()


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org