Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24852 )
Change subject: IMPALA-15358: Cap Kudu DML writers to partition count ...................................................................... Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/24852/12//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/24852/12//COMMIT_MSG@9 PS12, Line 9: For INSERT/UPSERT into a partitioned Kudu table, the planner inserts a : KUDU-partitioned exchange ahead of the KuduTableSink. At runtime, : KrpcDataStreamSender routes each row to channel (partition_index % : num_channels), where partition_index is the actual Kudu tablet index : (bounded by the table's tablet count). The writer fragment's instance : count, however, was sized like any other fragment (mt_dop / cost-based : scaling) with no awareness of the target table's partitioning. > I'm curious where KuduPartitioner is found in this bigger picture. Perhaps In the case we're covering here, Impala add's a partitioned exchange based on KuduPartitionExpr, which uses a KuduPartitioner - https://github.com/apache/impala/blob/master/be/src/exprs/kudu-partition-expr.cc#L67C17-L67C32 - to direct results to recipients. So we pre-partition the data along Kudu's partitioning scheme; if there are more recipients than partitions, some recipients receive no data. This patch is primarily a scheduling fix to avoid scheduling nodes that can't receive data, which is particularly bad when mt_dop>1 and all the active receivers are on a small set of nodes. -- To view, visit http://gerrit.cloudera.org:8080/24852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia3644fb245ceac72e91ed70c8c0392f65df75fde Gerrit-Change-Number: 24852 Gerrit-PatchSet: 12 Gerrit-Owner: Michael Smith <[email protected]> Gerrit-Reviewer: Aleksandr Efimov <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: David Rorke <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Fri, 02 Oct 2026 17:24:13 +0000 Gerrit-HasComments: Yes
