sunchao commented on code in PR #5854:
URL: https://github.com/apache/datafusion-comet/pull/5854#discussion_r4049790875
##########
spark/src/main/scala/org/apache/comet/CometExecIterator.scala:
##########
@@ -358,6 +358,13 @@ object CometExecIterator extends Logging {
CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED.key,
CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED.get(SQLConf.get).toString)
+ // The native map constructors (map_from_arrays, map_from_entries,
str_to_map) resolve
+ // duplicate keys with this policy, which the native side reads as
+ // `datafusion.spark.map_key_dedup_policy`.
+ builder.putEntries(
+ SQLConf.MAP_KEY_DEDUP_POLICY.key,
+ SQLConf.get.getConf(SQLConf.MAP_KEY_DEDUP_POLICY).toString)
Review Comment:
[P2] Preserve the map policy across repeated actions
This reads the current task SQLConf on every new native iterator, while
Spark's ArrayBasedMapBuilder captures the policy once and the executed
expression retains that builder. Reusing the same Dataset after changing the
policy therefore gives different behavior:
```scala
val path =
java.nio.file.Files.createTempDirectory("map-policy").resolve("data").toString
spark.range(0, 1, 1, 1).write.parquet(path)
spark.conf.set("spark.sql.mapKeyDedupPolicy", "LAST_WIN")
val df = spark.read.parquet(path)
.selectExpr("map_from_arrays(array(id, id), array(1, 2)) AS m")
df.collect() // {0 -> 2}
spark.conf.set("spark.sql.mapKeyDedupPolicy", "EXCEPTION")
df.collect() // Spark still returns {0 -> 2}
```
In the second action, the exact current Comet serializer sends EXCEPTION;
CometExecRDD creates a fresh iterator/native session, and the native wrapper
raises DUPLICATED_MAP_KEY for these inputs. The old LAST_WIN fallback preserved
Spark's behavior. The same policy-capture mismatch affects map_from_entries and
str_to_map.
Please preserve the policy with the expression/plan and add a regression
that executes the same Dataset twice across a policy change. Constructing a new
Dataset for each policy does not cover this case. I executed the Spark
reference, exact serializer, and native wrapper effects separately; the full
JNI validation limitation is recorded in the review summary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
