github-actions[bot] commented on code in PR #68639:
URL: https://github.com/apache/doris/pull/68639#discussion_r4132147936
##########
be/src/exprs/function/cast/cast_base.cpp:
##########
@@ -19,9 +19,35 @@
#include <cstdint>
+#include "core/column/column_array.h"
+#include "core/column/column_map.h"
+#include "core/column/column_struct.h"
#include "util/jsonb_writer.h"
namespace doris::CastWrapper {
+// Remove the duplicated keys of every map in the column, also the maps inside
arrays, structs,
+// map keys and map values. The last value of a key wins, as in map().
+static Status deduplicate_map_keys(IColumn& column) {
+ if (auto* nullable = check_and_get_column<ColumnNullable>(column)) {
+ return deduplicate_map_keys(nullable->get_nested_column());
+ }
+ if (auto* array = check_and_get_column<ColumnArray>(column)) {
+ return deduplicate_map_keys(array->get_data());
+ }
+ if (auto* st = check_and_get_column<ColumnStruct>(column)) {
+ for (size_t i = 0; i < st->tuple_size(); ++i) {
+ RETURN_IF_ERROR(deduplicate_map_keys(st->get_column(i)));
+ }
+ return Status::OK();
+ }
+ if (auto* map = check_and_get_column<ColumnMap>(column)) {
+ RETURN_IF_ERROR(deduplicate_map_keys(map->get_keys()));
+ RETURN_IF_ERROR(deduplicate_map_keys(map->get_values()));
Review Comment:
[P2] Filter duplicate outer keys before normalizing discarded values. This
walk deduplicates all maps in `map->get_values()` before
`map->deduplicate_keys()` removes earlier entries with the same outer key. For
a string cast to `MAP<STRING,ARRAY<MAP<STRING,INT>>>` with many repeated `"k"`
entries, every nested map in the superseded values is serialized and hashed
even though only the last value survives; those nested maps can have multiple
keys, so the singleton-map shortcut does not help. Normalize map keys first,
deduplicate the outer map, then visit its surviving values.
##########
be/src/exprs/function/cast/cast_base.cpp:
##########
@@ -19,9 +19,35 @@
#include <cstdint>
+#include "core/column/column_array.h"
+#include "core/column/column_map.h"
+#include "core/column/column_struct.h"
#include "util/jsonb_writer.h"
namespace doris::CastWrapper {
+// Remove the duplicated keys of every map in the column, also the maps inside
arrays, structs,
+// map keys and map values. The last value of a key wins, as in map().
+static Status deduplicate_map_keys(IColumn& column) {
+ if (auto* nullable = check_and_get_column<ColumnNullable>(column)) {
+ return deduplicate_map_keys(nullable->get_nested_column());
+ }
+ if (auto* array = check_and_get_column<ColumnArray>(column)) {
+ return deduplicate_map_keys(array->get_data());
+ }
+ if (auto* st = check_and_get_column<ColumnStruct>(column)) {
+ for (size_t i = 0; i < st->tuple_size(); ++i) {
+ RETURN_IF_ERROR(deduplicate_map_keys(st->get_column(i)));
+ }
+ return Status::OK();
+ }
+ if (auto* map = check_and_get_column<ColumnMap>(column)) {
+ RETURN_IF_ERROR(deduplicate_map_keys(map->get_keys()));
+ RETURN_IF_ERROR(deduplicate_map_keys(map->get_values()));
+ return map->deduplicate_keys();
Review Comment:
[P2] Skip key serialization for singleton nested maps. This new recursion
calls `deduplicate_keys()` for maps inside arrays and structs, but that method
allocates a reference for every key and serializes the entire key column before
it checks each map's entry count. For example, an `ARRAY<MAP<STRING,INT>>` cast
containing many one-entry maps copies every key into an Arena even though no
map can have duplicate keys; a large block can retain substantial extra memory
and spend time serializing data it cannot deduplicate. Check the map offsets
first and skip deduplication when all rows have at most one entry (or serialize
only rows with multiple entries).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
