alamb commented on code in PR #219:
URL: https://github.com/apache/datafusion-site/pull/219#discussion_r4567837069
##########
content/blog/2026-11-13-topk-per-partition.md:
##########
@@ -0,0 +1,356 @@
+---
+layout: post
+title: Top N Per Group with PartitionedTopKExec in Apache DataFusion
+date: 2026-11-13
+author: Vadim Lazuk
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*[Vadim Lazuk](https://github.com/Inf1mum)*
+
+"Find the top N rows within each group" is one of the most common shapes in
+analytics: the five best-selling products in each category, the three most
+recent events per device, the two highest-revenue rows per store. In SQL this
+is usually written with a ranking window function and an outer filter:
+
+```sql
+SELECT * FROM (
+ SELECT ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS
rn, *
+ FROM sales
+) WHERE rn <= 5;
+```
+
+Until recently, Apache DataFusion evaluated this query by sorting the *entire*
+input to compute the window function, and then throwing almost all of the
+sorted output away. DataFusion 55 recognizes this pattern and, when
+[datafusion.optimizer.enable_window_topn] is enabled, replaces the full sort
+with a new `PartitionedTopKExec` operator that keeps only the top N rows per
+partition in a bounded heap.
+
+This post explains the use case, shows how to enable and use the feature,
+walks through the `EXPLAIN` output before and after, describes how the
+operator works, and reports the measured performance — including the cases
+where it is *not* a good fit.
+
+This post grew out of a draft section [@alamb] wrote for the [DataFusion
Review Comment:
this is irrelevant detail
##########
content/blog/2026-11-13-topk-per-partition.md:
##########
@@ -0,0 +1,356 @@
+---
+layout: post
+title: Top N Per Group with PartitionedTopKExec in Apache DataFusion
+date: 2026-11-13
+author: Vadim Lazuk
+categories: [performance]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+*[Vadim Lazuk](https://github.com/Inf1mum)*
+
+"Find the top N rows within each group" is one of the most common shapes in
+analytics: the five best-selling products in each category, the three most
+recent events per device, the two highest-revenue rows per store. In SQL this
+is usually written with a ranking window function and an outer filter:
+
+```sql
+SELECT * FROM (
+ SELECT ROW_NUMBER() OVER (PARTITION BY category ORDER BY revenue DESC) AS
rn, *
+ FROM sales
+) WHERE rn <= 5;
+```
+
+Until recently, Apache DataFusion evaluated this query by sorting the *entire*
+input to compute the window function, and then throwing almost all of the
+sorted output away. DataFusion 55 recognizes this pattern and, when
+[datafusion.optimizer.enable_window_topn] is enabled, replaces the full sort
+with a new `PartitionedTopKExec` operator that keeps only the top N rows per
+partition in a bounded heap.
+
+This post explains the use case, shows how to enable and use the feature,
+walks through the `EXPLAIN` output before and after, describes how the
+operator works, and reports the measured performance — including the cases
+where it is *not* a good fit.
+
+This post grew out of a draft section [@alamb] wrote for the [DataFusion
+55.0.0 release post][draft-55-post], which was dropped because a release
+summary has limited room to explain how an operator actually works. The
+feature itself was implemented by [@SubhamSinghal], resolving a [feature
+request from 2023][#6899].
+
+[@alamb]: https://github.com/alamb
+[@SubhamSinghal]: https://github.com/SubhamSinghal
+[datafusion.optimizer.enable_window_topn]:
https://datafusion.apache.org/user-guide/configs.html
+[draft-55-post]: https://github.com/apache/datafusion/issues/27233
+[#6899]: https://github.com/apache/datafusion/issues/6899
+
+## The "Top N Per Group" Pattern
+
+The query above computes the top five rows per `category`. Other instances of
+the same pattern:
+
+- The latest N events per device in a telemetry table (`ORDER BY time DESC`).
+- The top N products by revenue per store.
+- The N highest-paid employees per department.
+
+It is also the shape of query 8 of the [h2o group-by benchmark] ("top 2 rows
+by value per group"), which DataFusion tracks as part of its effort to stay
+competitive on [db-benchmark][#13586] — a well-known public benchmark for data
+processing engines.
+
+[h2o group-by benchmark]: https://github.com/h2oai/db-benchmark
+[#13586]: https://github.com/apache/datafusion/issues/13586
+
+The important characteristic is that the filter keeps a *small, fixed number
+of rows per group* (`rn <= K`), while the input per group is large. When that
+holds, almost all the work of a sort-based plan is discarded.
+
+## The Default Plan Sorts Everything
+
+A ranking window function must see each partition of the data as one
+contiguous, ordered run. The straightforward way to guarantee that is to sort
+the whole input by `(partition keys, order keys)` and evaluate the window over
+the sorted stream. With the optimization disabled, the physical plan for the
+query above is (abridged):
+
+```text
+ProjectionExec
+ FilterExec: rn <= 5
+ BoundedWindowAggExec: ROW_NUMBER() PARTITION BY [category] ORDER BY
[revenue DESC]
+ SortExec: expr=[category ASC, revenue DESC]
+ DataSourceExec
+```
+
+This plan:
Review Comment:
can you please expand here with an explanation of how this operator works?
This is not enough detail to follow I don't think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
