uros-b commented on code in PR #57629:
URL: https://github.com/apache/spark/pull/57629#discussion_r3713915659
##########
sql/core/src/test/resources/sql-tests/inputs/distinct-map-aggregates.sql:
##########
@@ -0,0 +1,95 @@
+-- Test DISTINCT aggregates with MapType arguments.
+
+CREATE OR REPLACE TEMPORARY VIEW distinct_map_data AS SELECT * FROM VALUES
+ (2, map('a', 1, 'b', 2), 1, true),
+ (2, map('b', 2, 'a', 1), 1, true),
+ (1, map('a', 1, 'b', 2), 1, true),
+ (1, map('a', 3), 2, false)
+AS distinct_map_data(g, m, id, should_keep);
+
+SELECT COUNT(DISTINCT m) FROM distinct_map_data;
+
+SELECT SIZE(COLLECT_LIST(DISTINCT m)) FROM distinct_map_data;
+
+SELECT map_entries(m)
+FROM (
+ SELECT EXPLODE(COLLECT_LIST(DISTINCT m)) AS m
+ FROM distinct_map_data
+) AS collected_maps
+ORDER BY element_at(m, 'a');
+
+SELECT map_entries(FIRST(DISTINCT m)), map_entries(LAST(DISTINCT m)),
COUNT(DISTINCT m)
+FROM VALUES (map('b', 2, 'a', 1)) AS single_map_data(m);
+
+SELECT COUNT(DISTINCT m, id) FROM distinct_map_data;
+
+SELECT COUNT(DISTINCT m), COUNT(DISTINCT id) FROM distinct_map_data;
+
+SELECT g, COUNT(DISTINCT m)
+FROM distinct_map_data
+GROUP BY g
+ORDER BY g;
+
+SELECT m, COUNT(DISTINCT m), COLLECT_LIST(DISTINCT m)
+FROM distinct_map_data
+GROUP BY m
+ORDER BY element_at(m, 'a');
+
+SELECT COUNT(DISTINCT m) FILTER (WHERE should_keep) FROM distinct_map_data;
+
+SELECT MAX(map_values(m)[0])
+FROM distinct_map_data
+WHERE id = 1;
+
+SELECT MAX(map_values(m)[0]), COUNT(DISTINCT m)
+FROM distinct_map_data
+WHERE id = 1;
+
+SELECT g
+FROM distinct_map_data
+GROUP BY g
+ORDER BY COUNT(DISTINCT m), g;
+
+SELECT g
+FROM distinct_map_data
+GROUP BY g
+HAVING COUNT(DISTINCT m) = 1
+ORDER BY g;
+
+SELECT COUNT(DISTINCT named_struct('m', m)) FROM distinct_map_data;
+
+SELECT COUNT(DISTINCT array(m)) FROM distinct_map_data;
+
+SELECT COUNT(DISTINCT map('m', m)) FROM distinct_map_data;
+
+SELECT COUNT(DISTINCT m), COLLECT_LIST(DISTINCT m)
+FROM VALUES
+ (CAST(map() AS MAP<STRING, INT>)),
+ (CAST(map() AS MAP<STRING, INT>)),
+ (CAST(NULL AS MAP<STRING, INT>))
+AS null_and_empty_map_data(m);
+
+SELECT g, GROUPING(g), COUNT(DISTINCT m)
+FROM distinct_map_data
+GROUP BY GROUPING SETS ((g), ())
+ORDER BY GROUPING(g), g;
+
+SELECT COUNT(DISTINCT named_struct('m', m, 'n', n))
+FROM VALUES
+ (map('a', 1, 'b', 2), map('x', 1, 'y', 2)),
+ (map('b', 2, 'a', 1), map('y', 2, 'x', 1))
+AS grouped_distinct_map_data(m, n)
+GROUP BY m;
+
+SET spark.sql.optimizer.insertMapSortInDistinctAggregates.enabled=false;
+
+SELECT COUNT(DISTINCT m), COLLECT_LIST(DISTINCT m) FROM distinct_map_data;
Review Comment:
This case can't show what it's meant to show. HiveResult.toHiveString sorts
map entries when formatting output:
```
case (m: Map[_, _], MapType(kType, vType, _)) =>
m.map { case (key, value) => ... }.toSeq.sorted.mkString("{", ",", "}")
```
So the recorded result reads as 3 [{"a":1,"b":2},{"a":1,"b":2},{"a":3}] even
though the second element is really map('b', 2, 'a', 1). A reader can't
distinguish this flag-off case from the enabled one — the arrays look
identically sorted in both. You already work around this elsewhere in the file
with map_entries and explode-plus-ORDER BY; applying the same treatment here
would make the legacy behavior actually visible, which is the point of having
the flag-off cases at all.
That also fixes a second problem: COLLECT_LIST ordering isn't deterministic,
so asserting on the full three-element array is a flakiness risk across
shuffle-partition counts and across the suites that replay these inputs. The
single-element cases elsewhere in the file are fine; this is the only one that
pins down a multi-element ordering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
