Akshat-Jain opened a new pull request, #17141:
URL: https://github.com/apache/druid/pull/17141
### Description
In `WindowOperatorQueryFrameProcessor`, currently we were not adding the
partition boost column to the `ColumnSelectorFactory` used to create the
`FrameWriter`. Because of this, the boost column was not being written to the
frame.
This was causing the quantile sketch to misjudge the number of output
partitions to create, as it was creating a single partition boundary because we
had the same value for boost column (null byte) across all rows.
For example, for the following query:
```sql
REPLACE INTO "test_1" OVERWRITE ALL
SELECT
"__time",
"countryName",
"cityName",
count(cityName) over (partition by countryName) as countCity,
ROW_NUMBER() over (partition by countryName) as rowNumberCity
FROM "wikipedia"
group by __time, countryName, cityName
PARTITIONED BY HOUR
CLUSTERED BY countryName
```
Currently, we were getting 1 worker in the Scan stage after the Window stage:
<img width="1213" alt="image"
src="https://github.com/user-attachments/assets/9c2b079b-d9f5-4bb0-bfaf-41231d933094";>
With this PR's change, we get multiple workers in the Scan stage after the
Window stage:
<img width="1274" alt="image"
src="https://github.com/user-attachments/assets/b3f610e1-c8db-4cf8-b2a5-91af214befa2";>
### Test changes of the PR
- I had to update some MSQ drill tests to use single worker as they were
giving a different correct result. I could add some `order by` to try and make
the output deterministically correct for queries of these tests, but since
these were tests from the drill suite, I thought it's best to leave them
untouched and just use a single worker for these.
- The changes in `testWindowOnMixOfEmptyAndNonEmptyOverWithMultipleWorkers`
reflects the above situation where previously we were getting 1 worker in the
stage after the Window stage, but now we're getting 4 workers (as we use
`maxNumTasks=5` in this particular test).
<hr>
This PR has:
- [x] been self-reviewed.
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in
[licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
