cryptoe commented on code in PR #14609:
URL: https://github.com/apache/druid/pull/14609#discussion_r1280078477
##########
docs/api-reference/sql-api.md:
##########
@@ -186,4 +186,815 @@ Druid returns an HTTP 404 response in the following cases:
- `sqlQueryId` is incorrect.
- The query completes before your cancellation request is processed.
-Druid returns an HTTP 403 response for authorization failure.
\ No newline at end of file
+Druid returns an HTTP 403 response for authorization failure.
+
+## Query from deep storage
+
+> Query from deep storage is an [experimental
feature](../development/experimental.md).
+
+You can use the `sql/statements` endpoint to query segments that exist only in
deep storage and are not loaded onto your Historical processes as determined by
your load rules.
+
+Note that at least one segment of a datasource must be available on a
Historical process so that the Broker can plan your query. A quick way to check
if this is true is whether or not a datasource is visible in the Druid console.
+
+
+For more information, see [Query from deep
storage](../querying/query-from-deep-storage.md).
+
+### Submit a query
+
+Submit a query for data stored in deep storage. Any data ingested into Druid
is placed into deep storage. The query is contained in the "query" field in the
JSON object within the request payload.
+
+Note that at least part of a datasource must be available on a Historical
process so that Druid can plan your query and only the user who submits a query
can see the results.
+
+#### URL
+
+<code class="postAPI">POST</code> <code>/druid/v2/sql/statements</code>
+
+#### Request body
+
+Generally, the `sql` and `sql/statements` endpoints support the same response
body fields with minor differences. For general information about the available
fields, see [Submit a query to the `sql` endpoint](#submit-a-query).
+
+Keep the following in mind when submitting queries to the `sql/statements`
endpoint:
+
+- There are additional context parameters for `sql/statements`:
+
+ - `executionMode` determines how query results are fetched. Druid
currently only supports `ASYNC`. You must manually retrieve your results after
the query completes.
+ - `selectDestination` determines where final results get written. By
default, results are written to task reports. Set this parameter to
`DURABLESTORAGE` Druid to write the results from SELECT queries to durable
storage, which allows you to fetch larger result sets.. Note that this requires
you to have [durable storage for MSQ enabled](../operations/durable-storage.md).
+
+- The only supported value for `resultFormat` is JSON.
Review Comment:
```suggestion
- The only supported value for `resultFormat` is JSON LINES .
```
##########
docs/api-reference/sql-api.md:
##########
@@ -186,4 +186,815 @@ Druid returns an HTTP 404 response in the following cases:
- `sqlQueryId` is incorrect.
- The query completes before your cancellation request is processed.
-Druid returns an HTTP 403 response for authorization failure.
\ No newline at end of file
+Druid returns an HTTP 403 response for authorization failure.
+
+## Query from deep storage
+
+> Query from deep storage is an [experimental
feature](../development/experimental.md).
+
+You can use the `sql/statements` endpoint to query segments that exist only in
deep storage and are not loaded onto your Historical processes as determined by
your load rules.
+
+Note that at least one segment of a datasource must be available on a
Historical process so that the Broker can plan your query. A quick way to check
if this is true is whether or not a datasource is visible in the Druid console.
+
+
+For more information, see [Query from deep
storage](../querying/query-from-deep-storage.md).
+
+### Submit a query
+
+Submit a query for data stored in deep storage. Any data ingested into Druid
is placed into deep storage. The query is contained in the "query" field in the
JSON object within the request payload.
+
+Note that at least part of a datasource must be available on a Historical
process so that Druid can plan your query and only the user who submits a query
can see the results.
+
+#### URL
+
+<code class="postAPI">POST</code> <code>/druid/v2/sql/statements</code>
+
+#### Request body
+
+Generally, the `sql` and `sql/statements` endpoints support the same response
body fields with minor differences. For general information about the available
fields, see [Submit a query to the `sql` endpoint](#submit-a-query).
+
+Keep the following in mind when submitting queries to the `sql/statements`
endpoint:
+
+- There are additional context parameters for `sql/statements`:
+
+ - `executionMode` determines how query results are fetched. Druid
currently only supports `ASYNC`. You must manually retrieve your results after
the query completes.
+ - `selectDestination` determines where final results get written. By
default, results are written to task reports. Set this parameter to
`DURABLESTORAGE` Druid to write the results from SELECT queries to durable
storage, which allows you to fetch larger result sets.. Note that this requires
you to have [durable storage for MSQ enabled](../operations/durable-storage.md).
Review Comment:
```suggestion
- `selectDestination` determines where final results get written. By
default, results are written to task reports. Set this parameter to
`durableStorage` to instruct druid to write the results from SELECT queries to
durable storage, which allows you to fetch larger result sets. Note that this
requires you to have [durable storage for MSQ
enabled](../operations/durable-storage.md).
```
##########
docs/api-reference/sql-api.md:
##########
@@ -186,4 +186,815 @@ Druid returns an HTTP 404 response in the following cases:
- `sqlQueryId` is incorrect.
- The query completes before your cancellation request is processed.
-Druid returns an HTTP 403 response for authorization failure.
\ No newline at end of file
+Druid returns an HTTP 403 response for authorization failure.
+
+## Query from deep storage
+
+> Query from deep storage is an [experimental
feature](../development/experimental.md).
+
+You can use the `sql/statements` endpoint to query segments that exist only in
deep storage and are not loaded onto your Historical processes as determined by
your load rules.
+
+Note that at least one segment of a datasource must be available on a
Historical process so that the Broker can plan your query. A quick way to check
if this is true is whether or not a datasource is visible in the Druid console.
+
+
+For more information, see [Query from deep
storage](../querying/query-from-deep-storage.md).
+
+### Submit a query
+
+Submit a query for data stored in deep storage. Any data ingested into Druid
is placed into deep storage. The query is contained in the "query" field in the
JSON object within the request payload.
+
+Note that at least part of a datasource must be available on a Historical
process so that Druid can plan your query and only the user who submits a query
can see the results.
+
+#### URL
+
+<code class="postAPI">POST</code> <code>/druid/v2/sql/statements</code>
+
+#### Request body
+
+Generally, the `sql` and `sql/statements` endpoints support the same response
body fields with minor differences. For general information about the available
fields, see [Submit a query to the `sql` endpoint](#submit-a-query).
+
+Keep the following in mind when submitting queries to the `sql/statements`
endpoint:
+
+- There are additional context parameters for `sql/statements`:
+
+ - `executionMode` determines how query results are fetched. Druid
currently only supports `ASYNC`. You must manually retrieve your results after
the query completes.
+ - `selectDestination` determines where final results get written. By
default, results are written to task reports. Set this parameter to
`DURABLESTORAGE` Druid to write the results from SELECT queries to durable
storage, which allows you to fetch larger result sets.. Note that this requires
you to have [durable storage for MSQ enabled](../operations/durable-storage.md).
+
+- The only supported value for `resultFormat` is JSON.
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully queried from deep storage*
+
+<!--400 BAD REQUEST-->
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with
the following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorClass": "Class of exception that caused this error.",
+ "host": "The host on which the error occurred.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/statements"; \
+--header 'Content-Type: application/json' \
+--data '{
+ "query": "SELECT * FROM wikipedia WHERE user='\''BlueMoon2662'\''",
+ "context": {
+ "executionMode":"ASYNC"
+ }
+}'
+```
+
+<!--HTTP-->
+
+```HTTP
+POST /druid/v2/sql/statements HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+Content-Length: 134
+
+{
+ "query": "SELECT * FROM wikipedia WHERE user='BlueMoon2662'",
+ "context": {
+ "executionMode":"ASYNC"
+ }
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+ ```json
+{
+ "queryId": "query-b82a7049-b94f-41f2-a230-7fef94768745",
+ "state": "ACCEPTED",
+ "createdAt": "2023-07-26T21:16:25.324Z",
+ "schema": [
+ {
+ "name": "__time",
+ "type": "TIMESTAMP",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "channel",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "cityName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "comment",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "isAnonymous",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isMinor",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isNew",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isRobot",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isUnpatrolled",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "metroCode",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "namespace",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "page",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "user",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "delta",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "added",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "deleted",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ }
+ ],
+ "durationMs": -1
+}
+ ```
+</details>
+
+### Get query status
+
+Retrieves information about the query associated with the given query ID. The
response matches the response from the POST API if the query is accepted or
running and the execution mode is `ASYNC`. In addition to the fields that this
endpoint shares with `POST /sql/statements`, a completed query's status
includes the following:
+
+- A `result` object that summarizes information about your results, such as
the total number of rows and a sample record
+- A `pages` object that includes the following information for each page of
results:
+ - `numRows`: the number of rows in that page of results
+ - `sizeInBytes`: the size of the page
+ - `id`: the page number that you can use to reference a specific page when
you get query results
+
+#### URL
+
+<code class="getAPI">GET</code> <code>/druid/v2/sql/statements/:queryId</code>
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
Review Comment:
Looks like the status code is not visible in the docs.
##########
docs/api-reference/sql-api.md:
##########
@@ -186,4 +186,815 @@ Druid returns an HTTP 404 response in the following cases:
- `sqlQueryId` is incorrect.
- The query completes before your cancellation request is processed.
-Druid returns an HTTP 403 response for authorization failure.
\ No newline at end of file
+Druid returns an HTTP 403 response for authorization failure.
+
+## Query from deep storage
+
+> Query from deep storage is an [experimental
feature](../development/experimental.md).
+
+You can use the `sql/statements` endpoint to query segments that exist only in
deep storage and are not loaded onto your Historical processes as determined by
your load rules.
+
+Note that at least one segment of a datasource must be available on a
Historical process so that the Broker can plan your query. A quick way to check
if this is true is whether or not a datasource is visible in the Druid console.
+
+
+For more information, see [Query from deep
storage](../querying/query-from-deep-storage.md).
+
+### Submit a query
+
+Submit a query for data stored in deep storage. Any data ingested into Druid
is placed into deep storage. The query is contained in the "query" field in the
JSON object within the request payload.
+
+Note that at least part of a datasource must be available on a Historical
process so that Druid can plan your query and only the user who submits a query
can see the results.
+
+#### URL
+
+<code class="postAPI">POST</code> <code>/druid/v2/sql/statements</code>
+
+#### Request body
+
+Generally, the `sql` and `sql/statements` endpoints support the same response
body fields with minor differences. For general information about the available
fields, see [Submit a query to the `sql` endpoint](#submit-a-query).
+
+Keep the following in mind when submitting queries to the `sql/statements`
endpoint:
+
+- There are additional context parameters for `sql/statements`:
+
+ - `executionMode` determines how query results are fetched. Druid
currently only supports `ASYNC`. You must manually retrieve your results after
the query completes.
+ - `selectDestination` determines where final results get written. By
default, results are written to task reports. Set this parameter to
`DURABLESTORAGE` Druid to write the results from SELECT queries to durable
storage, which allows you to fetch larger result sets.. Note that this requires
you to have [durable storage for MSQ enabled](../operations/durable-storage.md).
+
+- The only supported value for `resultFormat` is JSON.
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully queried from deep storage*
+
+<!--400 BAD REQUEST-->
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with
the following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorClass": "Class of exception that caused this error.",
+ "host": "The host on which the error occurred.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/statements"; \
+--header 'Content-Type: application/json' \
+--data '{
+ "query": "SELECT * FROM wikipedia WHERE user='\''BlueMoon2662'\''",
+ "context": {
+ "executionMode":"ASYNC"
+ }
+}'
+```
+
+<!--HTTP-->
+
+```HTTP
+POST /druid/v2/sql/statements HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+Content-Length: 134
+
+{
+ "query": "SELECT * FROM wikipedia WHERE user='BlueMoon2662'",
+ "context": {
+ "executionMode":"ASYNC"
+ }
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+ ```json
+{
+ "queryId": "query-b82a7049-b94f-41f2-a230-7fef94768745",
+ "state": "ACCEPTED",
+ "createdAt": "2023-07-26T21:16:25.324Z",
+ "schema": [
+ {
+ "name": "__time",
+ "type": "TIMESTAMP",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "channel",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "cityName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "comment",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "isAnonymous",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isMinor",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isNew",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isRobot",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isUnpatrolled",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "metroCode",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "namespace",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "page",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "user",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "delta",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "added",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "deleted",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ }
+ ],
+ "durationMs": -1
+}
+ ```
+</details>
+
+### Get query status
+
+Retrieves information about the query associated with the given query ID. The
response matches the response from the POST API if the query is accepted or
running and the execution mode is `ASYNC`. In addition to the fields that this
endpoint shares with `POST /sql/statements`, a completed query's status
includes the following:
+
+- A `result` object that summarizes information about your results, such as
the total number of rows and a sample record
Review Comment:
```suggestion
- A `result` object that summarizes information about your results, such as
the total number of rows and sample records.
```
##########
docs/operations/durable-storage.md:
##########
@@ -0,0 +1,47 @@
+---
+id: durable-storage
+title: "Durable storage for the multi-stage query engine"
+sidebar_label: "Durable storage"
+---
+
+You can use durable storage to improve querying from deep storage and
SQL-based ingestion.
+
+> Note that only S3 is supported as a durable storage location.
+
+Durable storage for queries from deep storage provides a location where you
can write the results of deep storage queries to. Durable storage for SQL-based
ingestion is used to temporarily house intermediate files, which can improve
reliability.
+
+## Enable durable storage
+
+To enable durable storage, you need to set the following common service
properties:
+
+```
Review Comment:
The tempDir is used more as a staging directory before pushing out bytes to
s3. Its not related to results of the query.
##########
docs/api-reference/sql-api.md:
##########
@@ -186,4 +186,815 @@ Druid returns an HTTP 404 response in the following cases:
- `sqlQueryId` is incorrect.
- The query completes before your cancellation request is processed.
-Druid returns an HTTP 403 response for authorization failure.
\ No newline at end of file
+Druid returns an HTTP 403 response for authorization failure.
+
+## Query from deep storage
+
+> Query from deep storage is an [experimental
feature](../development/experimental.md).
+
+You can use the `sql/statements` endpoint to query segments that exist only in
deep storage and are not loaded onto your Historical processes as determined by
your load rules.
+
+Note that at least one segment of a datasource must be available on a
Historical process so that the Broker can plan your query. A quick way to check
if this is true is whether or not a datasource is visible in the Druid console.
+
+
+For more information, see [Query from deep
storage](../querying/query-from-deep-storage.md).
+
+### Submit a query
+
+Submit a query for data stored in deep storage. Any data ingested into Druid
is placed into deep storage. The query is contained in the "query" field in the
JSON object within the request payload.
+
+Note that at least part of a datasource must be available on a Historical
process so that Druid can plan your query and only the user who submits a query
can see the results.
+
+#### URL
+
+<code class="postAPI">POST</code> <code>/druid/v2/sql/statements</code>
+
+#### Request body
+
+Generally, the `sql` and `sql/statements` endpoints support the same response
body fields with minor differences. For general information about the available
fields, see [Submit a query to the `sql` endpoint](#submit-a-query).
+
+Keep the following in mind when submitting queries to the `sql/statements`
endpoint:
+
+- There are additional context parameters for `sql/statements`:
+
+ - `executionMode` determines how query results are fetched. Druid
currently only supports `ASYNC`. You must manually retrieve your results after
the query completes.
+ - `selectDestination` determines where final results get written. By
default, results are written to task reports. Set this parameter to
`DURABLESTORAGE` Druid to write the results from SELECT queries to durable
storage, which allows you to fetch larger result sets.. Note that this requires
you to have [durable storage for MSQ enabled](../operations/durable-storage.md).
+
+- The only supported value for `resultFormat` is JSON.
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully queried from deep storage*
+
+<!--400 BAD REQUEST-->
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with
the following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorClass": "Class of exception that caused this error.",
+ "host": "The host on which the error occurred.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/statements"; \
+--header 'Content-Type: application/json' \
+--data '{
+ "query": "SELECT * FROM wikipedia WHERE user='\''BlueMoon2662'\''",
+ "context": {
+ "executionMode":"ASYNC"
+ }
+}'
+```
+
+<!--HTTP-->
+
+```HTTP
+POST /druid/v2/sql/statements HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+Content-Length: 134
+
+{
+ "query": "SELECT * FROM wikipedia WHERE user='BlueMoon2662'",
+ "context": {
+ "executionMode":"ASYNC"
+ }
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+ ```json
+{
+ "queryId": "query-b82a7049-b94f-41f2-a230-7fef94768745",
+ "state": "ACCEPTED",
+ "createdAt": "2023-07-26T21:16:25.324Z",
+ "schema": [
+ {
+ "name": "__time",
+ "type": "TIMESTAMP",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "channel",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "cityName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "comment",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "isAnonymous",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isMinor",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isNew",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isRobot",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isUnpatrolled",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "metroCode",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "namespace",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "page",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "user",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "delta",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "added",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "deleted",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ }
+ ],
+ "durationMs": -1
+}
+ ```
+</details>
+
+### Get query status
+
+Retrieves information about the query associated with the given query ID. The
response matches the response from the POST API if the query is accepted or
running and the execution mode is `ASYNC`. In addition to the fields that this
endpoint shares with `POST /sql/statements`, a completed query's status
includes the following:
+
+- A `result` object that summarizes information about your results, such as
the total number of rows and a sample record
+- A `pages` object that includes the following information for each page of
results:
+ - `numRows`: the number of rows in that page of results
+ - `sizeInBytes`: the size of the page
+ - `id`: the page number that you can use to reference a specific page when
you get query results
+
+#### URL
+
+<code class="getAPI">GET</code> <code>/druid/v2/sql/statements/:queryId</code>
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully retrieved query status*
+
+<!--400 BAD REQUEST-->
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with
the following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample request
+
+The following example retrieves the status of a query with specified ID
`query-9b93f6f7-ab0e-48f5-986a-3520f84f0804`.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl
"http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/statements/query-9b93f6f7-ab0e-48f5-986a-3520f84f0804";
+```
+
+<!--HTTP-->
+
+```HTTP
+GET /druid/v2/sql/statements/query-9b93f6f7-ab0e-48f5-986a-3520f84f0804
HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+ ```json
+{
+ "queryId": "query-9b93f6f7-ab0e-48f5-986a-3520f84f0804",
+ "state": "SUCCESS",
+ "createdAt": "2023-07-26T22:57:46.620Z",
+ "schema": [
+ {
+ "name": "__time",
+ "type": "TIMESTAMP",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "channel",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "cityName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "comment",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "isAnonymous",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isMinor",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isNew",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isRobot",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isUnpatrolled",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "metroCode",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "namespace",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "page",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "user",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "delta",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "added",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "deleted",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ }
+ ],
+ "durationMs": 25591,
+ "result": {
+ "numTotalRows": 1,
+ "totalSizeInBytes": 375,
+ "dataSource": "__query_select",
+ "sampleRecords": [
+ [
+ 1442018873259,
+ "#ja.wikipedia",
+ "",
+ "/* 対戦通算成績と得失点 */",
+ "",
+ "",
+ 0,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ "Main",
+ "アルビレックス新潟の年度別成績一覧",
+ "",
+ "",
+ "BlueMoon2662",
+ 14,
+ 14,
+ 0
+ ]
+ ],
+ "pages": [
+ {
+ "id": 0,
+ "numRows": 1,
+ "sizeInBytes": 375
+ }
+ ]
+ }
+}
+ ```
+</details>
+
+
+### Get query results
+
+Retrieves results for completed queries. Results are separated into pages, so
you can use the optional `page` parameter to refine the results you get. Druid
returns information about the composition of each page and its page number
(`id`). For information about pages, see [Get query status](#get-query-status).
+
+If a page number isn't passed, all results are returned sequentially in the
same response. If you have large result sets, you may encounter timeouts based
on the value configured for `druid.router.http.readTimeout`.
+
+When getting query results, keep the following in mind:
+
+- JSON is the only supported result format.
+- Getting the query results for an ingestion query returns an empty response.
+
+#### URL
+
+<code class="getAPI">GET</code>
<code>/druid/v2/sql/statements/:queryId/results</code>
+
+#### Query parameters
+* `page`
+ * Int (optional)
+ * Refine paginated results
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully retrieved query results*
+
+<!--400 BAD REQUEST-->
+
+*Query in progress. Returns a JSON object detailing the error with the
following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--404 NOT FOUND-->
+
+*Query failed or canceled*
+
+<!--500 SERVER ERROR-->
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with
the following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+The following example retrieves the status of a query with specified ID
`query-f3bca219-173d-44d4-bdc7-5002e910352f`.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl
"http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/statements/query-f3bca219-173d-44d4-bdc7-5002e910352f/results";
+```
+
+<!--HTTP-->
+
+```HTTP
+GET
/druid/v2/sql/statements/query-f3bca219-173d-44d4-bdc7-5002e910352f/results
HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+ ```json
+[
+ {
+ "__time": 1442018818771,
+ "channel": "#en.wikipedia",
+ "cityName": "",
+ "comment": "added project",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 0,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Talk",
+ "page": "Talk:Oswald Tilghman",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "GELongstreet",
+ "delta": 36,
+ "added": 36,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018820496,
+ "channel": "#ca.wikipedia",
+ "cityName": "",
+ "comment": "Robot inserta {{Commonscat}} que enllaça amb
[[commons:category:Rallicula]]",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 1,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Rallicula",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "PereBot",
+ "delta": 17,
+ "added": 17,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018825474,
+ "channel": "#en.wikipedia",
+ "cityName": "Auburn",
+ "comment": "/* Status of peremptory norms under international law */
fixed spelling of 'Wimbledon'",
+ "countryIsoCode": "AU",
+ "countryName": "Australia",
+ "isAnonymous": 1,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 0,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Peremptory norm",
+ "regionIsoCode": "NSW",
+ "regionName": "New South Wales",
+ "user": "60.225.66.142",
+ "delta": 0,
+ "added": 0,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018828770,
+ "channel": "#vi.wikipedia",
+ "cityName": "",
+ "comment": "fix Lỗi CS1: ngày tháng",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 1,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Apamea abruzzorum",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "Cheers!-bot",
+ "delta": 18,
+ "added": 18,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018831862,
+ "channel": "#vi.wikipedia",
+ "cityName": "",
+ "comment": "clean up using [[Project:AWB|AWB]]",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Atractus flammigerus",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "ThitxongkhoiAWB",
+ "delta": 18,
+ "added": 18,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018833987,
+ "channel": "#vi.wikipedia",
+ "cityName": "",
+ "comment": "clean up using [[Project:AWB|AWB]]",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Agama mossambica",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "ThitxongkhoiAWB",
+ "delta": 18,
+ "added": 18,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018837009,
+ "channel": "#ca.wikipedia",
+ "cityName": "",
+ "comment": "/* Imperi Austrohongarès */",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 0,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Campanya dels Balcans (1914-1918)",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "Jaumellecha",
+ "delta": -20,
+ "added": 0,
+ "deleted": 20
+ },
+ {
+ "__time": 1442018839591,
+ "channel": "#en.wikipedia",
+ "cityName": "",
+ "comment": "adding comment on notability and possible COI",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 1,
+ "isRobot": 0,
+ "isUnpatrolled": 1,
+ "metroCode": 0,
+ "namespace": "Talk",
+ "page": "Talk:Dani Ploeger",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "New Media Theorist",
+ "delta": 345,
+ "added": 345,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018841578,
+ "channel": "#en.wikipedia",
+ "cityName": "",
+ "comment": "Copying assessment table to wiki",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "User",
+ "page": "User:WP 1.0 bot/Tables/Project/Pubs",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "WP 1.0 bot",
+ "delta": 121,
+ "added": 121,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018845821,
+ "channel": "#vi.wikipedia",
+ "cityName": "",
+ "comment": "clean up using [[Project:AWB|AWB]]",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Agama persimilis",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "ThitxongkhoiAWB",
+ "delta": 18,
+ "added": 18,
+ "deleted": 0
+ }
+]
+ ```
+</details>
+
+### Cancel a query
+
+Cancels a running or accepted query.
+
+#### URL
+
+<code class="deleteAPI">DELETE</code>
<code>/druid/v2/sql/statements/:queryId</code>
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 OK-->
+
+*Unsuccessfully canceled query, query is already complete*
Review Comment:
```suggestion
*A no op operation since the query is not in a state to be cancelled.*
```
##########
docs/multi-stage-query/reference.md:
##########
@@ -343,59 +343,23 @@ CLUSTERED BY user
The context parameter that sets `sqlJoinAlgorithm` to `sortMerge` is not shown
in the above example.
-## Durable Storage
+## Durable storage
-Using durable storage with your SQL-based ingestion can improve their
reliability by writing intermediate files to a storage location temporarily.
+SQL-based ingestion supports using durable storage to store intermediate files
temporarily. Enabling it can improve reliability. For more information, see
[Durable storage](../operations/durable-storage.md).
-To prevent durable storage from getting filled up with temporary files in case
the tasks fail to clean them up, a periodic
-cleaner can be scheduled to clean the directories corresponding to which there
isn't a controller task running. It utilizes
-the storage connector to work upon the durable storage. The durable storage
location should only be utilized to store the output
-for cluster's MSQ tasks. If the location contains other files or directories,
then they will get cleaned up as well.
-
-Enabling durable storage also enables the use of local disk to store temporary
files, such as the intermediate files produced
-by the super sorter. Tasks will use whatever has been configured for their
temporary usage as described in [Configuring task storage
sizes](../ingestion/tasks.md#configuring-task-storage-sizes)
-If the configured limit is too low, `NotEnoughTemporaryStorageFault` may be
thrown.
-
-### Enable durable storage
-
-To enable durable storage, you need to set the following common service
properties:
-
-```
-druid.msq.intermediate.storage.enable=true
-druid.msq.intermediate.storage.type=s3
-druid.msq.intermediate.storage.bucket=YOUR_BUCKET
-druid.msq.intermediate.storage.prefix=YOUR_PREFIX
-druid.msq.intermediate.storage.tempDir=/path/to/your/temp/dir
-```
-
-For detailed information about the settings related to durable storage, see
[Durable storage configurations](#durable-storage-configurations).
-
-
-### Use durable storage for queries
-
-When you run a query, include the context parameter `durableShuffleStorage`
and set it to `true`.
-
-For queries where you want to use fault tolerance for workers, set
`faultTolerance` to `true`, which automatically sets `durableShuffleStorage` to
`true`.
-
-Set `selectDestination`:`durableStorage` for select queries that want to write
the final results to durable storage instead of the task reports. Saving the
results in the durable
-storage allows users to fetch large result sets. The location where the
workers write the intermediate results is different than the location where
final results get stored. Therefore, `durableShuffleStorage`:`false` and
-`selectDestination`:`durableStorage` is a valid configuration to use in the
query context, that instructs the controller to persist only the final result
in the durable storage, and not the
-intermediate results.
-
-
-## Durable storage configurations
+### Durable storage configurations
The following common service properties control how durable storage behaves:
|Parameter |Default | Description
|
|-------------------|----------------------------------------|----------------------|
+|`druid.msq.intermediate.storage.enable` | true | Required. Whether to enable
durable storage for the cluster.|
+|`druid.msq.intermediate.storage.type` | `s3` if your deep storage is S3 |
Required. The type of storage to use. You can either set this to `local` or
`s3`. |
Review Comment:
https://github.com/apache/druid/pull/14720 has some incoming changes.
##########
docs/api-reference/sql-api.md:
##########
@@ -186,4 +186,815 @@ Druid returns an HTTP 404 response in the following cases:
- `sqlQueryId` is incorrect.
- The query completes before your cancellation request is processed.
-Druid returns an HTTP 403 response for authorization failure.
\ No newline at end of file
+Druid returns an HTTP 403 response for authorization failure.
+
+## Query from deep storage
+
+> Query from deep storage is an [experimental
feature](../development/experimental.md).
+
+You can use the `sql/statements` endpoint to query segments that exist only in
deep storage and are not loaded onto your Historical processes as determined by
your load rules.
+
+Note that at least one segment of a datasource must be available on a
Historical process so that the Broker can plan your query. A quick way to check
if this is true is whether or not a datasource is visible in the Druid console.
+
+
+For more information, see [Query from deep
storage](../querying/query-from-deep-storage.md).
+
+### Submit a query
+
+Submit a query for data stored in deep storage. Any data ingested into Druid
is placed into deep storage. The query is contained in the "query" field in the
JSON object within the request payload.
+
+Note that at least part of a datasource must be available on a Historical
process so that Druid can plan your query and only the user who submits a query
can see the results.
+
+#### URL
+
+<code class="postAPI">POST</code> <code>/druid/v2/sql/statements</code>
+
+#### Request body
+
+Generally, the `sql` and `sql/statements` endpoints support the same response
body fields with minor differences. For general information about the available
fields, see [Submit a query to the `sql` endpoint](#submit-a-query).
+
+Keep the following in mind when submitting queries to the `sql/statements`
endpoint:
+
+- There are additional context parameters for `sql/statements`:
+
+ - `executionMode` determines how query results are fetched. Druid
currently only supports `ASYNC`. You must manually retrieve your results after
the query completes.
+ - `selectDestination` determines where final results get written. By
default, results are written to task reports. Set this parameter to
`DURABLESTORAGE` Druid to write the results from SELECT queries to durable
storage, which allows you to fetch larger result sets.. Note that this requires
you to have [durable storage for MSQ enabled](../operations/durable-storage.md).
+
+- The only supported value for `resultFormat` is JSON.
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully queried from deep storage*
+
+<!--400 BAD REQUEST-->
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with
the following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorClass": "Class of exception that caused this error.",
+ "host": "The host on which the error occurred.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/statements"; \
+--header 'Content-Type: application/json' \
+--data '{
+ "query": "SELECT * FROM wikipedia WHERE user='\''BlueMoon2662'\''",
+ "context": {
+ "executionMode":"ASYNC"
+ }
+}'
+```
+
+<!--HTTP-->
+
+```HTTP
+POST /druid/v2/sql/statements HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+Content-Length: 134
+
+{
+ "query": "SELECT * FROM wikipedia WHERE user='BlueMoon2662'",
+ "context": {
+ "executionMode":"ASYNC"
+ }
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+ ```json
+{
+ "queryId": "query-b82a7049-b94f-41f2-a230-7fef94768745",
+ "state": "ACCEPTED",
+ "createdAt": "2023-07-26T21:16:25.324Z",
+ "schema": [
+ {
+ "name": "__time",
+ "type": "TIMESTAMP",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "channel",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "cityName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "comment",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "isAnonymous",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isMinor",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isNew",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isRobot",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isUnpatrolled",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "metroCode",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "namespace",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "page",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "user",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "delta",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "added",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "deleted",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ }
+ ],
+ "durationMs": -1
+}
+ ```
+</details>
+
+### Get query status
+
+Retrieves information about the query associated with the given query ID. The
response matches the response from the POST API if the query is accepted or
running and the execution mode is `ASYNC`. In addition to the fields that this
endpoint shares with `POST /sql/statements`, a completed query's status
includes the following:
+
+- A `result` object that summarizes information about your results, such as
the total number of rows and a sample record
+- A `pages` object that includes the following information for each page of
results:
+ - `numRows`: the number of rows in that page of results
+ - `sizeInBytes`: the size of the page
+ - `id`: the page number that you can use to reference a specific page when
you get query results
+
+#### URL
+
+<code class="getAPI">GET</code> <code>/druid/v2/sql/statements/:queryId</code>
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully retrieved query status*
+
+<!--400 BAD REQUEST-->
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with
the following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample request
+
+The following example retrieves the status of a query with specified ID
`query-9b93f6f7-ab0e-48f5-986a-3520f84f0804`.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl
"http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/statements/query-9b93f6f7-ab0e-48f5-986a-3520f84f0804";
+```
+
+<!--HTTP-->
+
+```HTTP
+GET /druid/v2/sql/statements/query-9b93f6f7-ab0e-48f5-986a-3520f84f0804
HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+ ```json
+{
+ "queryId": "query-9b93f6f7-ab0e-48f5-986a-3520f84f0804",
+ "state": "SUCCESS",
+ "createdAt": "2023-07-26T22:57:46.620Z",
+ "schema": [
+ {
+ "name": "__time",
+ "type": "TIMESTAMP",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "channel",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "cityName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "comment",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "isAnonymous",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isMinor",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isNew",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isRobot",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isUnpatrolled",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "metroCode",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "namespace",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "page",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "user",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "delta",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "added",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "deleted",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ }
+ ],
+ "durationMs": 25591,
+ "result": {
+ "numTotalRows": 1,
+ "totalSizeInBytes": 375,
+ "dataSource": "__query_select",
+ "sampleRecords": [
+ [
+ 1442018873259,
+ "#ja.wikipedia",
+ "",
+ "/* 対戦通算成績と得失点 */",
+ "",
+ "",
+ 0,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ "Main",
+ "アルビレックス新潟の年度別成績一覧",
+ "",
+ "",
+ "BlueMoon2662",
+ 14,
+ 14,
+ 0
+ ]
+ ],
+ "pages": [
+ {
+ "id": 0,
+ "numRows": 1,
+ "sizeInBytes": 375
+ }
+ ]
+ }
+}
+ ```
+</details>
+
+
+### Get query results
+
+Retrieves results for completed queries. Results are separated into pages, so
you can use the optional `page` parameter to refine the results you get. Druid
returns information about the composition of each page and its page number
(`id`). For information about pages, see [Get query status](#get-query-status).
+
+If a page number isn't passed, all results are returned sequentially in the
same response. If you have large result sets, you may encounter timeouts based
on the value configured for `druid.router.http.readTimeout`.
+
+When getting query results, keep the following in mind:
+
+- JSON is the only supported result format.
+- Getting the query results for an ingestion query returns an empty response.
+
+#### URL
+
+<code class="getAPI">GET</code>
<code>/druid/v2/sql/statements/:queryId/results</code>
+
+#### Query parameters
+* `page`
+ * Int (optional)
+ * Refine paginated results
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully retrieved query results*
+
+<!--400 BAD REQUEST-->
+
+*Query in progress. Returns a JSON object detailing the error with the
following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--404 NOT FOUND-->
+
+*Query failed or canceled*
+
+<!--500 SERVER ERROR-->
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with
the following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+The following example retrieves the status of a query with specified ID
`query-f3bca219-173d-44d4-bdc7-5002e910352f`.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl
"http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/statements/query-f3bca219-173d-44d4-bdc7-5002e910352f/results";
+```
+
+<!--HTTP-->
+
+```HTTP
+GET
/druid/v2/sql/statements/query-f3bca219-173d-44d4-bdc7-5002e910352f/results
HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+ ```json
+[
+ {
+ "__time": 1442018818771,
+ "channel": "#en.wikipedia",
+ "cityName": "",
+ "comment": "added project",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 0,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Talk",
+ "page": "Talk:Oswald Tilghman",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "GELongstreet",
+ "delta": 36,
+ "added": 36,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018820496,
+ "channel": "#ca.wikipedia",
+ "cityName": "",
+ "comment": "Robot inserta {{Commonscat}} que enllaça amb
[[commons:category:Rallicula]]",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 1,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Rallicula",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "PereBot",
+ "delta": 17,
+ "added": 17,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018825474,
+ "channel": "#en.wikipedia",
+ "cityName": "Auburn",
+ "comment": "/* Status of peremptory norms under international law */
fixed spelling of 'Wimbledon'",
+ "countryIsoCode": "AU",
+ "countryName": "Australia",
+ "isAnonymous": 1,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 0,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Peremptory norm",
+ "regionIsoCode": "NSW",
+ "regionName": "New South Wales",
+ "user": "60.225.66.142",
+ "delta": 0,
+ "added": 0,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018828770,
+ "channel": "#vi.wikipedia",
+ "cityName": "",
+ "comment": "fix Lỗi CS1: ngày tháng",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 1,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Apamea abruzzorum",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "Cheers!-bot",
+ "delta": 18,
+ "added": 18,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018831862,
+ "channel": "#vi.wikipedia",
+ "cityName": "",
+ "comment": "clean up using [[Project:AWB|AWB]]",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Atractus flammigerus",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "ThitxongkhoiAWB",
+ "delta": 18,
+ "added": 18,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018833987,
+ "channel": "#vi.wikipedia",
+ "cityName": "",
+ "comment": "clean up using [[Project:AWB|AWB]]",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Agama mossambica",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "ThitxongkhoiAWB",
+ "delta": 18,
+ "added": 18,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018837009,
+ "channel": "#ca.wikipedia",
+ "cityName": "",
+ "comment": "/* Imperi Austrohongarès */",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 0,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Campanya dels Balcans (1914-1918)",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "Jaumellecha",
+ "delta": -20,
+ "added": 0,
+ "deleted": 20
+ },
+ {
+ "__time": 1442018839591,
+ "channel": "#en.wikipedia",
+ "cityName": "",
+ "comment": "adding comment on notability and possible COI",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 1,
+ "isRobot": 0,
+ "isUnpatrolled": 1,
+ "metroCode": 0,
+ "namespace": "Talk",
+ "page": "Talk:Dani Ploeger",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "New Media Theorist",
+ "delta": 345,
+ "added": 345,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018841578,
+ "channel": "#en.wikipedia",
+ "cityName": "",
+ "comment": "Copying assessment table to wiki",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "User",
+ "page": "User:WP 1.0 bot/Tables/Project/Pubs",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "WP 1.0 bot",
+ "delta": 121,
+ "added": 121,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018845821,
+ "channel": "#vi.wikipedia",
+ "cityName": "",
+ "comment": "clean up using [[Project:AWB|AWB]]",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Agama persimilis",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "ThitxongkhoiAWB",
+ "delta": 18,
+ "added": 18,
+ "deleted": 0
+ }
+]
+ ```
+</details>
+
+### Cancel a query
+
+Cancels a running or accepted query.
+
+#### URL
+
+<code class="deleteAPI">DELETE</code>
<code>/druid/v2/sql/statements/:queryId</code>
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 OK-->
+
+*Unsuccessfully canceled query, query is already complete*
+
+<!--202 ACCEPTED-->
+
+*Successfully cancelled query*
Review Comment:
```suggestion
*Successfully accepted query for cancellation*
```
##########
docs/api-reference/sql-api.md:
##########
@@ -186,4 +186,815 @@ Druid returns an HTTP 404 response in the following cases:
- `sqlQueryId` is incorrect.
- The query completes before your cancellation request is processed.
-Druid returns an HTTP 403 response for authorization failure.
\ No newline at end of file
+Druid returns an HTTP 403 response for authorization failure.
+
+## Query from deep storage
+
+> Query from deep storage is an [experimental
feature](../development/experimental.md).
+
+You can use the `sql/statements` endpoint to query segments that exist only in
deep storage and are not loaded onto your Historical processes as determined by
your load rules.
+
+Note that at least one segment of a datasource must be available on a
Historical process so that the Broker can plan your query. A quick way to check
if this is true is whether or not a datasource is visible in the Druid console.
+
+
+For more information, see [Query from deep
storage](../querying/query-from-deep-storage.md).
+
+### Submit a query
+
+Submit a query for data stored in deep storage. Any data ingested into Druid
is placed into deep storage. The query is contained in the "query" field in the
JSON object within the request payload.
+
+Note that at least part of a datasource must be available on a Historical
process so that Druid can plan your query and only the user who submits a query
can see the results.
+
+#### URL
+
+<code class="postAPI">POST</code> <code>/druid/v2/sql/statements</code>
+
+#### Request body
+
+Generally, the `sql` and `sql/statements` endpoints support the same response
body fields with minor differences. For general information about the available
fields, see [Submit a query to the `sql` endpoint](#submit-a-query).
+
+Keep the following in mind when submitting queries to the `sql/statements`
endpoint:
+
+- There are additional context parameters for `sql/statements`:
+
+ - `executionMode` determines how query results are fetched. Druid
currently only supports `ASYNC`. You must manually retrieve your results after
the query completes.
+ - `selectDestination` determines where final results get written. By
default, results are written to task reports. Set this parameter to
`DURABLESTORAGE` Druid to write the results from SELECT queries to durable
storage, which allows you to fetch larger result sets.. Note that this requires
you to have [durable storage for MSQ enabled](../operations/durable-storage.md).
+
+- The only supported value for `resultFormat` is JSON.
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully queried from deep storage*
+
+<!--400 BAD REQUEST-->
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with
the following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorClass": "Class of exception that caused this error.",
+ "host": "The host on which the error occurred.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl "http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/statements"; \
+--header 'Content-Type: application/json' \
+--data '{
+ "query": "SELECT * FROM wikipedia WHERE user='\''BlueMoon2662'\''",
+ "context": {
+ "executionMode":"ASYNC"
+ }
+}'
+```
+
+<!--HTTP-->
+
+```HTTP
+POST /druid/v2/sql/statements HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+Content-Type: application/json
+Content-Length: 134
+
+{
+ "query": "SELECT * FROM wikipedia WHERE user='BlueMoon2662'",
+ "context": {
+ "executionMode":"ASYNC"
+ }
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+ ```json
+{
+ "queryId": "query-b82a7049-b94f-41f2-a230-7fef94768745",
+ "state": "ACCEPTED",
+ "createdAt": "2023-07-26T21:16:25.324Z",
+ "schema": [
+ {
+ "name": "__time",
+ "type": "TIMESTAMP",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "channel",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "cityName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "comment",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "isAnonymous",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isMinor",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isNew",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isRobot",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isUnpatrolled",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "metroCode",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "namespace",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "page",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "user",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "delta",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "added",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "deleted",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ }
+ ],
+ "durationMs": -1
+}
+ ```
+</details>
+
+### Get query status
+
+Retrieves information about the query associated with the given query ID. The
response matches the response from the POST API if the query is accepted or
running and the execution mode is `ASYNC`. In addition to the fields that this
endpoint shares with `POST /sql/statements`, a completed query's status
includes the following:
+
+- A `result` object that summarizes information about your results, such as
the total number of rows and a sample record
+- A `pages` object that includes the following information for each page of
results:
+ - `numRows`: the number of rows in that page of results
+ - `sizeInBytes`: the size of the page
+ - `id`: the page number that you can use to reference a specific page when
you get query results
+
+#### URL
+
+<code class="getAPI">GET</code> <code>/druid/v2/sql/statements/:queryId</code>
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully retrieved query status*
+
+<!--400 BAD REQUEST-->
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with
the following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample request
+
+The following example retrieves the status of a query with specified ID
`query-9b93f6f7-ab0e-48f5-986a-3520f84f0804`.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl
"http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/statements/query-9b93f6f7-ab0e-48f5-986a-3520f84f0804";
+```
+
+<!--HTTP-->
+
+```HTTP
+GET /druid/v2/sql/statements/query-9b93f6f7-ab0e-48f5-986a-3520f84f0804
HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+ ```json
+{
+ "queryId": "query-9b93f6f7-ab0e-48f5-986a-3520f84f0804",
+ "state": "SUCCESS",
+ "createdAt": "2023-07-26T22:57:46.620Z",
+ "schema": [
+ {
+ "name": "__time",
+ "type": "TIMESTAMP",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "channel",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "cityName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "comment",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "countryName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "isAnonymous",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isMinor",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isNew",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isRobot",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "isUnpatrolled",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "metroCode",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "namespace",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "page",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionIsoCode",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "regionName",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "user",
+ "type": "VARCHAR",
+ "nativeType": "STRING"
+ },
+ {
+ "name": "delta",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "added",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ },
+ {
+ "name": "deleted",
+ "type": "BIGINT",
+ "nativeType": "LONG"
+ }
+ ],
+ "durationMs": 25591,
+ "result": {
+ "numTotalRows": 1,
+ "totalSizeInBytes": 375,
+ "dataSource": "__query_select",
+ "sampleRecords": [
+ [
+ 1442018873259,
+ "#ja.wikipedia",
+ "",
+ "/* 対戦通算成績と得失点 */",
+ "",
+ "",
+ 0,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ "Main",
+ "アルビレックス新潟の年度別成績一覧",
+ "",
+ "",
+ "BlueMoon2662",
+ 14,
+ 14,
+ 0
+ ]
+ ],
+ "pages": [
+ {
+ "id": 0,
+ "numRows": 1,
+ "sizeInBytes": 375
+ }
+ ]
+ }
+}
+ ```
+</details>
+
+
+### Get query results
+
+Retrieves results for completed queries. Results are separated into pages, so
you can use the optional `page` parameter to refine the results you get. Druid
returns information about the composition of each page and its page number
(`id`). For information about pages, see [Get query status](#get-query-status).
+
+If a page number isn't passed, all results are returned sequentially in the
same response. If you have large result sets, you may encounter timeouts based
on the value configured for `druid.router.http.readTimeout`.
+
+When getting query results, keep the following in mind:
+
+- JSON is the only supported result format.
+- Getting the query results for an ingestion query returns an empty response.
+
+#### URL
+
+<code class="getAPI">GET</code>
<code>/druid/v2/sql/statements/:queryId/results</code>
+
+#### Query parameters
+* `page`
+ * Int (optional)
+ * Refine paginated results
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 SUCCESS-->
+
+*Successfully retrieved query results*
+
+<!--400 BAD REQUEST-->
+
+*Query in progress. Returns a JSON object detailing the error with the
following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--404 NOT FOUND-->
+
+*Query failed or canceled*
+
+<!--500 SERVER ERROR-->
+
+*Error thrown due to bad query. Returns a JSON object detailing the error with
the following format:*
+
+```json
+{
+ "error": "Summary of the encountered error.",
+ "errorCode": "Well-defined error code.",
+ "persona": "Role or persona associated with the error.",
+ "category": "Classification of the error.",
+ "errorMessage": "Summary of the encountered issue with expanded
information.",
+ "context": "Additional context about the error."
+}
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+---
+
+#### Sample request
+
+The following example retrieves the status of a query with specified ID
`query-f3bca219-173d-44d4-bdc7-5002e910352f`.
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--cURL-->
+
+```shell
+curl
"http://ROUTER_IP:ROUTER_PORT/druid/v2/sql/statements/query-f3bca219-173d-44d4-bdc7-5002e910352f/results";
+```
+
+<!--HTTP-->
+
+```HTTP
+GET
/druid/v2/sql/statements/query-f3bca219-173d-44d4-bdc7-5002e910352f/results
HTTP/1.1
+Host: http://ROUTER_IP:ROUTER_PORT
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
+
+#### Sample response
+
+<details>
+ <summary>Click to show sample response</summary>
+
+ ```json
+[
+ {
+ "__time": 1442018818771,
+ "channel": "#en.wikipedia",
+ "cityName": "",
+ "comment": "added project",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 0,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Talk",
+ "page": "Talk:Oswald Tilghman",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "GELongstreet",
+ "delta": 36,
+ "added": 36,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018820496,
+ "channel": "#ca.wikipedia",
+ "cityName": "",
+ "comment": "Robot inserta {{Commonscat}} que enllaça amb
[[commons:category:Rallicula]]",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 1,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Rallicula",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "PereBot",
+ "delta": 17,
+ "added": 17,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018825474,
+ "channel": "#en.wikipedia",
+ "cityName": "Auburn",
+ "comment": "/* Status of peremptory norms under international law */
fixed spelling of 'Wimbledon'",
+ "countryIsoCode": "AU",
+ "countryName": "Australia",
+ "isAnonymous": 1,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 0,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Peremptory norm",
+ "regionIsoCode": "NSW",
+ "regionName": "New South Wales",
+ "user": "60.225.66.142",
+ "delta": 0,
+ "added": 0,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018828770,
+ "channel": "#vi.wikipedia",
+ "cityName": "",
+ "comment": "fix Lỗi CS1: ngày tháng",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 1,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Apamea abruzzorum",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "Cheers!-bot",
+ "delta": 18,
+ "added": 18,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018831862,
+ "channel": "#vi.wikipedia",
+ "cityName": "",
+ "comment": "clean up using [[Project:AWB|AWB]]",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Atractus flammigerus",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "ThitxongkhoiAWB",
+ "delta": 18,
+ "added": 18,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018833987,
+ "channel": "#vi.wikipedia",
+ "cityName": "",
+ "comment": "clean up using [[Project:AWB|AWB]]",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Agama mossambica",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "ThitxongkhoiAWB",
+ "delta": 18,
+ "added": 18,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018837009,
+ "channel": "#ca.wikipedia",
+ "cityName": "",
+ "comment": "/* Imperi Austrohongarès */",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 0,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Campanya dels Balcans (1914-1918)",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "Jaumellecha",
+ "delta": -20,
+ "added": 0,
+ "deleted": 20
+ },
+ {
+ "__time": 1442018839591,
+ "channel": "#en.wikipedia",
+ "cityName": "",
+ "comment": "adding comment on notability and possible COI",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 1,
+ "isRobot": 0,
+ "isUnpatrolled": 1,
+ "metroCode": 0,
+ "namespace": "Talk",
+ "page": "Talk:Dani Ploeger",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "New Media Theorist",
+ "delta": 345,
+ "added": 345,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018841578,
+ "channel": "#en.wikipedia",
+ "cityName": "",
+ "comment": "Copying assessment table to wiki",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "User",
+ "page": "User:WP 1.0 bot/Tables/Project/Pubs",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "WP 1.0 bot",
+ "delta": 121,
+ "added": 121,
+ "deleted": 0
+ },
+ {
+ "__time": 1442018845821,
+ "channel": "#vi.wikipedia",
+ "cityName": "",
+ "comment": "clean up using [[Project:AWB|AWB]]",
+ "countryIsoCode": "",
+ "countryName": "",
+ "isAnonymous": 0,
+ "isMinor": 0,
+ "isNew": 0,
+ "isRobot": 1,
+ "isUnpatrolled": 0,
+ "metroCode": 0,
+ "namespace": "Main",
+ "page": "Agama persimilis",
+ "regionIsoCode": "",
+ "regionName": "",
+ "user": "ThitxongkhoiAWB",
+ "delta": 18,
+ "added": 18,
+ "deleted": 0
+ }
+]
+ ```
+</details>
+
+### Cancel a query
+
+Cancels a running or accepted query.
+
+#### URL
+
+<code class="deleteAPI">DELETE</code>
<code>/druid/v2/sql/statements/:queryId</code>
+
+#### Responses
+
+<!--DOCUSAURUS_CODE_TABS-->
+
+<!--200 OK-->
+
+*Unsuccessfully canceled query, query is already complete*
+
+<!--202 ACCEPTED-->
+
+*Successfully cancelled query*
+
+<!--500 SERVER ERROR-->
Review Comment:
This should be 404 if the query id is not found.
https://github.com/apache/druid/pull/14706
##########
docs/operations/durable-storage.md:
##########
@@ -0,0 +1,86 @@
+---
+id: durable-storage
+title: "Durable storage for the multi-stage query engine"
+sidebar_label: "Durable storage"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+You can use durable storage to improve querying from deep storage and
SQL-based ingestion.
+
+> Note that only S3 is supported as a durable storage location.
+
+Durable storage for queries from deep storage provides a location where you
can write the results of deep storage queries to. Durable storage for SQL-based
ingestion is used to temporarily house intermediate files, which can improve
reliability.
+
+Enabling durable storage also enables the use of local disk to store temporary
files, such as the intermediate files produced
+by the super sorter. Tasks will use whatever has been configured for their
temporary usage as described in [Configuring task storage
sizes](../ingestion/tasks.md#configuring-task-storage-sizes).
Review Comment:
```suggestion
Enabling durable storage also enables the use of local disk to store
temporary files, such as the intermediate files produced
while sorting the data. Tasks will use whatever has been configured for
their temporary usage as described in [Configuring task storage
sizes](../ingestion/tasks.md#configuring-task-storage-sizes).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
