gengliangwang commented on code in PR #59077:
URL: https://github.com/apache/spark/pull/59077#discussion_r4426016226
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala:
##########
@@ -66,7 +67,15 @@ object SimplifyExtractValueOps extends Rule[LogicalPlan] {
// out of bounds, mimic the runtime behavior and return null
Literal(null, ga.dataType)
}
- case GetMapValue(CreateMap(elems, _), key) => CaseKeyWhen(key, elems)
+ case GetMapValue(CreateMap(elems, _), key) =>
+ // CASE returns the first matching value, whereas LAST_WIN keeps the
last map entry.
+ val entries = if (conf.getConf(SQLConf.MAP_KEY_DEDUP_POLICY) ==
+ SQLConf.MapKeyDedupPolicy.LAST_WIN) {
+ elems.grouped(2).toSeq.reverse.flatten
Review Comment:
**[P1] Preserve signed-zero lookups when normalization is disabled**
With legacy map-key normalization disabled, ArrayBasedMapBuilder uses an
identity key normalizer and a Java HashMap for double keys, preserving +0.0 and
-0.0 as distinct entries. GetMapValue's LinearExecutor nevertheless compares
them using SQLOrderingUtil.compareDoubles, which equates signed zeros, and
returns the first entry's value. CaseKeyWhen uses the same SQL equality, so
reversing its branches now selects the other value. The dynamic id prevents
constant folding, and floating-number normalization does not apply to this
projection. Skip this simplification for floating keys in this legacy mode, or
preserve the builder's deduplication and lookup semantics.
**Failure scenario:** Set spark.sql.mapKeyDedupPolicy=LAST_WIN and
spark.sql.legacy.disableMapKeyNormalization=true, then run SELECT map(CAST(id
AS DOUBLE), 10, CAST('-0.0' AS DOUBLE), 20)[CAST(id AS DOUBLE)] FROM range(1).
The new rewrite returns 20; constructing and looking up the map, and the
previous rewrite, return 10.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
