Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/21477#discussion_r193892217
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/ForeachWriter.scala
---
@@ -71,23 +110,17 @@ abstract class ForeachWriter[T] extends Serializable {
// TODO: Move this to org.apache.spark.sql.util or consolidate this with
batch API.
/**
- * Called when starting to process one partition of new data in the
executor. The `version` is
- * for data deduplication when there are failures. When recovering from
a failure, some data may
- * be generated multiple times but they will always have the same
version.
- *
- * If this method finds using the `partitionId` and `version` that this
partition has already been
- * processed, it can return `false` to skip the further data processing.
However, `close` still
- * will be called for cleaning up resources.
+ * Called when starting to process one partition of new data in the
executor.
*
* @param partitionId the partition id.
- * @param version a unique id for data deduplication.
+ * @param epochId a unique id for data deduplication.
* @return `true` if the corresponding partition and version id should
be processed. `false`
* indicates the partition should be skipped.
*/
- def open(partitionId: Long, version: Long): Boolean
+ def open(partitionId: Long, epochId: Long): Boolean
--- End diff --
okay. I checked but there is no compatibility issue in this case. The
source compatibility issue arises when a code calls a method with param name
(e.g. `func(paramName=value)`) and the param name changes. In this case, users
are overriding the method (overriding does not care about the exact name, only
the types) and Spark is internally calling the method (not by param name). So
this is fine.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
