andygrove commented on code in PR #4863:
URL: https://github.com/apache/datafusion-comet/pull/4863#discussion_r3545701402
##########
spark/src/main/scala/org/apache/comet/serde/maps.scala:
##########
@@ -75,8 +75,44 @@ object CometMapExtract extends
CometExpressionSerde[GetMapValue] {
}
}
+/**
+ * Shared support for the `spark.sql.mapKeyDedupPolicy` divergence between
Spark and Comet.
+ *
+ * Spark's `ArrayBasedMapBuilder` (used by `MapFromArrays` and
`MapFromEntries`) rejects null map
+ * keys and applies the policy on duplicate keys (default `EXCEPTION`, or
`LAST_WIN`). Comet's
+ * native map construction implements neither behaviour, so the two only agree
when the config is
+ * at its default and the user input contains no null / duplicate keys. This
helper narrows the
+ * divergence to the config-driven half: when the policy has been switched
away from the default
+ * we mark the expression as incompatible so it falls back to Spark. The
null-key / duplicate-key
+ * data cases under the default policy remain user-observable divergences
tracked separately.
+ *
+ * Other map-building expressions (`CreateMap`, `MapConcat`, `TransformKeys`)
already delegate to
+ * Spark's own `doGenCode` via `CometCodegenDispatch`, so they inherit Spark's
dedup semantics for
+ * free and do not need this gate.
+ */
+private object MapKeyDedupPolicySupport {
+ val incompatibleReason: String =
+ s"`${SQLConf.MAP_KEY_DEDUP_POLICY.key}` is set to a non-default value;
Comet does not " +
+ "enforce Spark's map-key dedup semantics (LAST_WIN vs EXCEPTION) or
reject null map keys."
+
+ def isDefault: Boolean =
+ SQLConf.get.getConf(SQLConf.MAP_KEY_DEDUP_POLICY) ==
+ SQLConf.MAP_KEY_DEDUP_POLICY.defaultValueString
Review Comment:
Checking for default vs non-default value seems brittle. What if the default
changes in a later Spark version? Shouldn't we check explicitly for `EXCEPTION`
vs `LAST_WIN`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
