This is an automated email from the ASF dual-hosted git repository. lakshsingla pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push: new 3c72ec8413f docs: Migration guide for subquery limit (#16519) 3c72ec8413f is described below commit 3c72ec8413fc066dbb56e9f02d6fbc6e3b28bbe1 Author: Jill Osborne <jill.osbo...@imply.io> AuthorDate: Fri May 31 04:56:07 2024 +0100 docs: Migration guide for subquery limit (#16519) Adds a migration guide for Druid 30 to help users understand the new byte-based subquery limit property maxSubqueryBytes --- docs/release-info/migr-subquery-limit.md | 60 ++++++++++++++++++++++++++++++++ website/sidebars.json | 3 +- 2 files changed, 62 insertions(+), 1 deletion(-) diff --git a/docs/release-info/migr-subquery-limit.md b/docs/release-info/migr-subquery-limit.md new file mode 100644 index 00000000000..853be50294d --- /dev/null +++ b/docs/release-info/migr-subquery-limit.md @@ -0,0 +1,60 @@ +--- +id: migr-subquery-limit +title: "Migration guide: Subquery limit" +sidebar_label: Subquery limit +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +Druid now allows you to set a byte-based limit on subquery size, to prevent brokers from running out of memory when handling large subqueries. +Druid uses subqueries as joins as well as in common table expressions, such as WITH. + +The byte-based subquery limit overrides Druid's row-based subquery limit. + +:::info +We recommend that you move towards using byte-based limits starting in Druid 30.0. +::: + +For queries that generate a large number of rows (5 million or more), we recommend that you don't use `maxSubqueryBytes` from the outset. +You can increase `maxSubqueryRows` and then configure the byte-based limit if you find that Druid needs it to process the query. + +## Row-based subquery limit + +Druid uses the `maxSubqueryRows` property to limit the number of rows Druid returns in a subquery. +Because this is a row-based limit, it doesn't restrict the overall size of the returned data. + +The `maxSubqueryRows` property is set to 100,000 by default. + +## Enable a byte-based subquery limit + +Set the optional property `maxSubqueryBytes` to set a maximum number of returned bytes. +This property takes precedence over `maxSubqueryRows`. + +## Usage considerations + +You can set both `maxSubqueryRows` and `maxSubqueryBytes` at cluster level and override them in individual queries. +See [Overriding default query context values](../configuration#overriding-default-query-context-values) for more information. + +## Learn more + +See the following topics for more information: + +- [Query context](../querying/query-context.md) for information on setting query context parameters. +- [Broker configuration reference](../configuration#guardrails-for-materialization-of-subqueries) for more information on `maxSubqueryRows` and `maxSubqueryBytes`. diff --git a/website/sidebars.json b/website/sidebars.json index 286269953d9..5e07945ca7d 100644 --- a/website/sidebars.json +++ b/website/sidebars.json @@ -363,9 +363,10 @@ "id": "release-info/migration-guide" }, "items": [ + "release-info/migr-subquery-limit" ] } ]}, "misc/papers-and-talks" ] -} \ No newline at end of file +} --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org