FrankChen021 commented on code in PR #20646: URL: https://github.com/apache/druid/pull/20646#discussion_r4325292617
########## docs/querying/query-context-reference.md: ########## @@ -25,124 +25,1584 @@ sidebar_label: "Query context reference" The query context provides runtime configuration for individual queries in Apache® Druid. Each parameter in the query context controls a specific aspect of query behavior—from execution timeouts and resource limits to caching policies and processing strategies. -This reference contains context parameters organized by their scope: +This reference documents every query context parameter, grouped by where it applies: -- **General parameters**: Applies to all query types. -- **Parameters by query type**: Applies to a specific type of query, such as TopN. -- **Vectorization parameters**: Controls vectorized query execution for supported queries. +- **General parameters**: Apply to native queries and, unless noted, to SQL queries. Some also apply to other engines. +- **Druid SQL parameters**: Apply to SQL queries only. +- **Query type parameters**: Apply to a specific native query type, such as TopN or GroupBy. +- **SQL-based ingestion parameters**: Apply to queries that run on the multi-stage query (MSQ) engine. +- **Dart parameters**: Apply to [Dart](dart.md) queries only. -To learn how to set the query context, see [Set query context](./query-context.md). +Parameters are listed in alphabetical order within each group. Each parameter shows its type, default value, +the query interfaces (`JSON` for native queries, `SQL` for Druid SQL) and engines that honor it, and the Druid version +that introduced it. -For reference on query context parameters specific to Druid SQL, visit [SQL query context](sql-query-context.md). -For context parameters related to SQL-based ingestion, see the [SQL-based ingestion reference](../multi-stage-query/reference.md#context-parameters). +To learn how to set the query context, see [Set query context](./query-context.md). +<a id="vectorization-parameters"></a>To learn which queries can run vectorized, see [Vectorization](vectorization.md). +<a id="join-filter"></a> Review Comment: [P2] Restore the Join filter section **Finding:** The former Join filter section was removed, while its compatibility anchor remains immediately before General parameters. Links to #join-filter therefore land at the wrong section, and the five join-only parameters are no longer grouped by scope. **Suggestion:** Restore a dedicated Join filter heading and generated block with the anchor on that section. ########## docs/querying/vectorization.md: ########## @@ -0,0 +1,57 @@ +--- +id: vectorization +title: "Vectorization" +sidebar_label: "Vectorization" +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +The GroupBy, Timeseries, TimeBoundary, and Scan (in MSQ only) query types can run in _vectorized_ mode, which speeds up +query execution by processing batches of rows at a time. Not all queries can be vectorized. In particular, vectorization +currently has the following requirements: + +- All query-level filters must either be able to run on bitmap indexes or must offer vectorized row-matchers. These +include `selector`, `bound`, `in`, `like`, `regex`, `search`, `and`, `or`, and `not`. +- All filters in filtered aggregators must offer vectorized row-matchers. +- All aggregators must offer vectorized implementations. These include `count`, `doubleSum`, `floatSum`, `longSum`. `longMin`, + `longMax`, `doubleMin`, `doubleMax`, `floatMin`, `floatMax`, `longAny`, `doubleAny`, `floatAny`, `stringAny`, + `hyperUnique`, `filtered`, `approxHistogram`, `approxHistogramFold`, and `fixedBucketsHistogram` (with numerical input). Review Comment: [P3] Remove trailing whitespace **Finding:** The new page contains trailing whitespace, which is reported by git diff --check. **Suggestion:** Remove the trailing space at the end of the line. ########## docs/multi-stage-query/reference.md: ########## @@ -380,50 +380,58 @@ For more information about clustering, see [Clustering](concepts.md#clustering). <a name="context"></a> -## Context parameters +## Query context parameters {#context-parameters} The multi-stage query task engine supports the [SQL context parameters](../querying/sql-query-context.md), as well as its own context parameters described in this section. Use these parameters to tailor how Druid executes your query. You can specify the context parameters in SELECT, INSERT, or REPLACE statements. For detailed instructions on configuring query context parameters, refer to [Set query context](../querying/query-context.md). -The following table lists the context parameters for the MSQ task engine: +The following context parameters apply to the MSQ task engine. See the [query context reference](../querying/query-context-reference.md#sql-based-ingestion-parameters) for details. -| Parameter | Description | Default value | -|---|---|---| -|`maxNumTasks`|SELECT, INSERT, REPLACE<br /><br />The maximum total number of tasks to launch, including the controller task. The lowest possible value for this setting is 2: one controller and one worker. All tasks must be able to launch simultaneously. If they cannot, the query returns a `TaskStartTimeout` error code after approximately 10 minutes.|`2`| <!-- GENERATED QUERY CONTEXT PARAMETER: maxNumTasks --> -|`taskAssignment`|SELECT, INSERT, REPLACE<br /><br />Determines how many tasks to use. Possible values include: <ul><li>`max`: Uses as many tasks as possible, up to `maxNumTasks`.</li><li>`auto`: When file sizes can be determined through directory listing (for example: local files, S3, GCS, HDFS) uses as few tasks as possible without exceeding 512 MiB or 10,000 files per task, unless exceeding these limits is necessary to stay within `maxNumTasks`. When calculating the size of files, the weighted size is used, which considers the file format and compression format used if any. When file sizes cannot be determined through directory listing (for example: http), behaves the same as `max`.</li></ul>|`max`| <!-- GENERATED QUERY CONTEXT PARAMETER: taskAssignment --> -|`finalizeAggregations`|SELECT, INSERT, REPLACE<br /><br />Determines the type of aggregation to return. If true, Druid finalizes the results of complex aggregations that directly appear in query results. If false, Druid returns the aggregation's intermediate type rather than finalized type. This parameter is useful during ingestion, where it enables storing sketches directly in Druid tables. For more information about aggregations, see [SQL aggregation functions](../querying/sql-aggregations.md).|`true`| <!-- GENERATED QUERY CONTEXT PARAMETER: finalizeAggregations --> -|`arrayIngestMode`|INSERT, REPLACE<br /><br />Controls how ARRAY type values are stored in Druid segments. When set to `array` (the default, recommended for SQL compliance), Druid will store all ARRAY typed values in [ARRAY typed columns](../querying/arrays.md), and supports storing both VARCHAR and numeric typed arrays. When set to `mvd` (for backwards compatibility), Druid only supports VARCHAR typed arrays, and will store them as [multi-value string columns](../querying/multi-value-dimensions.md). See [`arrayIngestMode`] in the [Arrays](../querying/arrays.md) page for more details.|`array`| <!-- GENERATED QUERY CONTEXT PARAMETER: arrayIngestMode --> -|`sqlJoinAlgorithm`|SELECT, INSERT, REPLACE<br /><br />Algorithm to use for `JOIN`: `broadcast` for broadcast hash join or `sortMerge` for sort-merge join. Affects all joins in the query. This is a hint to the MSQ engine, and the actual joins may proceed differently. The native engine only supports `broadcast`. See [Joins](../multi-stage-query/reference.md#joins) for more details.|`broadcast`| <!-- GENERATED QUERY CONTEXT PARAMETER: sqlJoinAlgorithm --> -|`maxRowsInMemory`|INSERT, REPLACE<br /><br />Maximum number of rows to store in memory at once before flushing to disk during the segment generation process. Ignored for non-INSERT queries. In most cases, use the default value. You may need to override the default if you run into one of the [known issues](../multi-stage-query/known-issues.md) around memory usage.|`100000`| <!-- GENERATED QUERY CONTEXT PARAMETER: maxRowsInMemory --> -|`rowsInMemory`|INSERT, REPLACE<br /><br />Alternate spelling of `maxRowsInMemory`. Ignored if `maxRowsInMemory` is set.|`100000`| <!-- GENERATED QUERY CONTEXT PARAMETER: rowsInMemory --> -|`segmentSortOrder`|INSERT, REPLACE<br /><br />Normally, Druid sorts rows in individual segments using `__time` first, followed by the [CLUSTERED BY](#clustered-by) clause. When you set `segmentSortOrder`, Druid uses the order from this context parameter instead. Provide the column list as comma-separated values or as a JSON array in string form.<br /><br />For example, consider an INSERT query that uses `CLUSTERED BY country` and has `segmentSortOrder` set to `__time,city,country`. Within each time chunk, Druid assigns rows to segments based on `country`, and then within each of those segments, Druid sorts those rows by `__time` first, then `city`, then `country`.|empty list| <!-- GENERATED QUERY CONTEXT PARAMETER: segmentSortOrder --> -|`forceSegmentSortByTime`|INSERT, REPLACE<br /><br />When set to `true` (the default), Druid prepends `__time` to [CLUSTERED BY](#clustered-by) when determining the sort order for individual segments. Druid also requires that `segmentSortOrder`, if provided, starts with `__time`.<br /><br />When set to `false`, Druid uses the [CLUSTERED BY](#clustered-by) alone to determine the sort order for individual segments, and does not require that `segmentSortOrder` begin with `__time`. Setting this parameter to `false` is an experimental feature; see [Sorting](../ingestion/partitioning.md#sorting) for details.|`true`| <!-- GENERATED QUERY CONTEXT PARAMETER: forceSegmentSortByTime --> -|`maxParseExceptions`|SELECT, INSERT, REPLACE<br /><br />Maximum number of parse exceptions that are ignored while executing the query before it stops with `TooManyWarningsFault`. To ignore all the parse exceptions, set the value to -1.|`0`| <!-- GENERATED QUERY CONTEXT PARAMETER: maxParseExceptions --> -|`rowsPerSegment`|INSERT, REPLACE<br /><br />The number of rows per segment to target. The actual number of rows per segment may be somewhat higher or lower than this number. In most cases, use the default. For general information about sizing rows per segment, see [Segment Size Optimization](../operations/segment-optimization.md).|`3000000`| <!-- GENERATED QUERY CONTEXT PARAMETER: rowsPerSegment --> -|`indexSpec`|INSERT, REPLACE<br /><br />An [`indexSpec`](../ingestion/ingestion-spec.md#indexspec) to use when generating segments. May be a JSON string or object. See [Front coding](../ingestion/ingestion-spec.md#front-coding) for details on configuring an `indexSpec` with front coding.|See [`indexSpec`](../ingestion/ingestion-spec.md#indexspec).| <!-- GENERATED QUERY CONTEXT PARAMETER: indexSpec --> -|`durableShuffleStorage`|SELECT, INSERT, REPLACE<br /><br />Whether to use durable storage for shuffle mesh. To use this feature, configure the durable storage at the server level using `druid.msq.intermediate.storage.enable=true`). If these properties are not configured, any query with the context variable `durableShuffleStorage=true` fails with a configuration error.|`false`| <!-- GENERATED QUERY CONTEXT PARAMETER: durableShuffleStorage --> -|`faultTolerance`|SELECT, INSERT, REPLACE<br /><br />Whether to turn on fault tolerance mode or not. Failed workers are retried based on [Limits](#limits). Cannot be used when `durableShuffleStorage` is explicitly set to false.|`false`| <!-- GENERATED QUERY CONTEXT PARAMETER: faultTolerance --> -|`selectDestination`|SELECT<br /><br />Controls where the final result of the select query is written. <br />Use `taskReport`(the default) to write select results to the task report. <b> This is not scalable since task reports size explodes for large results </b> <br/>Use `durableStorage` to write results to durable storage location. <b>For large results sets, its recommended to use `durableStorage` </b>. To configure durable storage see [`this`](#durable-storage) section.|`taskReport`| <!-- GENERATED QUERY CONTEXT PARAMETER: selectDestination --> -|`waitUntilSegmentsLoad`|INSERT, REPLACE<br /><br />If set, the ingest query waits for the generated segments to be loaded before exiting, else the ingest query exits without waiting. The task and live reports contain the information about the status of loading segments if this flag is set. This will ensure that any future queries made after the ingestion exits will include results from the ingestion. The drawback is that the controller task will stall till the segments are loaded.|`false`| <!-- GENERATED QUERY CONTEXT PARAMETER: waitUntilSegmentsLoad --> -|`includeSegmentSource`|SELECT, INSERT, REPLACE<br /><br />Controls the sources Druid queries for results in addition to the segments present on deep storage. Can be `NONE` or `REALTIME`. If set to `NONE`, only non-realtime (published and used) segments are downloaded from deep storage. If set to `REALTIME`, results are also included from realtime tasks. `REALTIME` cannot be used while writing data into the same datasource it is read from.|`NONE` for the MSQ task engine, `REALTIME` for Dart| <!-- GENERATED QUERY CONTEXT PARAMETER: includeSegmentSource --> -|`rowsPerPage`|SELECT<br /><br />The number of rows per page to target. The actual number of rows per page may be somewhat higher or lower than this number. In most cases, use the default.<br /> This property comes into effect only when `selectDestination` is set to `durableStorage`|`100000`| <!-- GENERATED QUERY CONTEXT PARAMETER: rowsPerPage --> -|`skipTypeVerification`|INSERT, REPLACE<br /><br />During query validation, Druid validates that [string arrays](../querying/arrays.md) and [multi-value dimensions](../querying/multi-value-dimensions.md) are not mixed in the same column. If you are intentionally migrating from one to the other, use this context parameter to disable type validation.<br /><br />Provide the column list as comma-separated values or as a JSON array in string form.|empty list| <!-- GENERATED QUERY CONTEXT PARAMETER: skipTypeVerification --> -|`failOnEmptyInsert`|INSERT, REPLACE<br /><br />When set to false (the default), an INSERT query generating no output rows will be no-op, and a REPLACE query generating no output rows will delete all data that matches the OVERWRITE clause. When set to true, an ingest query generating no output rows will throw an `InsertCannotBeEmpty` fault.|`false`| <!-- GENERATED QUERY CONTEXT PARAMETER: failOnEmptyInsert --> -|`storeCompactionState`|REPLACE<br /><br />When set to true, a REPLACE query stores as part of each segment's metadata a `lastCompactionState` field that captures the various specs used to create the segment. Future compaction jobs skip segments whose `lastCompactionState` matches the desired compaction state. Works the same as [`storeCompactionState`](../ingestion/tasks.md#context-parameters) task context flag.|`false`| <!-- GENERATED QUERY CONTEXT PARAMETER: storeCompactionState --> -|`removeNullBytes`|SELECT, INSERT, REPLACE<br /><br />The MSQ engine can't process null bytes in strings and throws `InvalidNullByteFault` if it encounters them in the source data. If the parameter is set to true, the MSQ engine removes the null bytes in string fields when reading the data.|`false`| <!-- GENERATED QUERY CONTEXT PARAMETER: removeNullBytes --> -|`includeAllCounters`|SELECT, INSERT, REPLACE<br /><br />Whether to include counters that were added in Druid 31 or later. This is a backwards compatibility option that must be set to `false` during a rolling update from versions prior to Druid 31.|`true`| <!-- GENERATED QUERY CONTEXT PARAMETER: includeAllCounters --> -|`maxFrameSize`|SELECT, INSERT, REPLACE<br /><br />Size of frames used for data transfer within the MSQ engine. You generally do not need to change this unless you have very large rows.|`1000000` (1 MB)| <!-- GENERATED QUERY CONTEXT PARAMETER: maxFrameSize --> -|`maxInputFilesPerWorker`|SELECT, INSERT, REPLACE<br /><br />Maximum number of input files or segments per worker. If a single worker would need to read more than this number of files, the query fails with a `TooManyInputFiles` error. In this case, you should either increase this limit if your tasks have enough memory to handle more files, add more workers by increasing `maxNumTasks`, or split your query into smaller queries that process fewer files.|`10000`| <!-- GENERATED QUERY CONTEXT PARAMETER: maxInputFilesPerWorker --> -|`backgroundFetchExternalFiles`|SELECT, INSERT, REPLACE<br /><br />Controls how workers read cloud storage files (e.g. `s3`, `gs`, `azure`) referenced by [`EXTERN`](#extern-function). When `true` (the default), each worker fetches its input files asynchronously when possible before reading them. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.<br /><br />This setting has no effect for input sources that use `prefix`, or input sources that request [system fields](../ingestion/input-sources.md).|`true`| <!-- GENERATED QUERY CONTEXT PARAMETER: backgroundFetchExternalFiles --> -|`maxPartitions`|SELECT, INSERT, REPLACE<br /><br />Maximum number of output partitions for any single stage. For INSERT or REPLACE queries, this controls the maximum number of segments that can be generated. If the query would exceed this limit, it fails with a `TooManyPartitions` error. You can increase this limit if needed, break your query into smaller queries, or use a larger target segment size (via `rowsPerSegment`).|`25000`| <!-- GENERATED QUERY CONTEXT PARAMETER: maxPartitions --> -|`maxThreads`|SELECT, INSERT, REPLACE<br /><br />Maximum number of threads to use for processing. This only has an effect if it is greater than zero and less than the default thread count based on system configuration. Otherwise, it is ignored, and workers use the default thread count.|Not set (use default thread count)| <!-- GENERATED QUERY CONTEXT PARAMETER: maxThreads --> -|`taskLockType`|INSERT, REPLACE<br /><br />Type of time chunk lock that an INSERT or REPLACE query acquires: `EXCLUSIVE`, `SHARED`, `APPEND` or `REPLACE`. If not set, REPLACE queries use `EXCLUSIVE` and INSERT queries use `SHARED`. Ignored when `useConcurrentLocks` is true.|`EXCLUSIVE` for REPLACE, `SHARED` for INSERT| <!-- GENERATED QUERY CONTEXT PARAMETER: taskLockType --> -|`useConcurrentLocks`|INSERT, REPLACE<br /><br />Whether an INSERT or REPLACE query uses concurrent locks: `APPEND` locks for INSERT and `REPLACE` locks for REPLACE. See [concurrent append and replace](../ingestion/concurrent-append-replace.md).|`false`| <!-- GENERATED QUERY CONTEXT PARAMETER: useConcurrentLocks --> -|`maxNumSegments`|INSERT, REPLACE<br /><br />Maximum number of segments that an ingestion query can generate in a single time chunk. If the query would exceed this limit, it fails with a `TooManySegmentsInTimeChunk` error.|No limit| <!-- GENERATED QUERY CONTEXT PARAMETER: maxNumSegments --> -|`maxRowsMaterializedInWindow`|SELECT, INSERT, REPLACE<br /><br />Maximum number of rows that window functions can materialize in a single window partition. If a partition exceeds this limit, the query fails with a `TooManyRowsInAWindow` error.|`100000`| <!-- GENERATED QUERY CONTEXT PARAMETER: maxRowsMaterializedInWindow --> +<!-- BEGIN GENERATED QUERY CONTEXT PARAMETER TABLE: engines~MSQ engines!~NATIVE --> + +|Parameter|Statements|Description|Default| +|---|---|---|---| +|[`arrayIngestMode`](../querying/query-context-reference.md#arrayingestmode)|INSERT, REPLACE|Controls how ARRAY type values are stored in Druid segments.|`array`| +|[`backgroundFetchExternalFiles`](../querying/query-context-reference.md#backgroundfetchexternalfiles)|SELECT, INSERT, REPLACE|Controls how workers read cloud storage files (e.g. `s3`, `gs`, `azure`) referenced by [`EXTERN`](../multi-stage-query/reference.md#extern-function).|`true`| +|[`catalogValidationEnabled`](../querying/query-context-reference.md#catalogvalidationenabled)||If false, Druid skips the catalog-specific validation of `INSERT` and `REPLACE` statements that write to tables defined in the Druid catalog.|`true`| Review Comment: [P2] Document catalogValidationEnabled statement scope **Finding:** The generated MSQ table has a Statements column, but this row is blank even though the parameter only controls validation for INSERT and REPLACE. The canonical parameter metadata also lacks this statement scope. **Suggestion:** Add INSERT and REPLACE statement metadata to CATALOG_VALIDATION_ENABLED and regenerate the documentation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
