317brian commented on code in PR #14609:
URL: https://github.com/apache/druid/pull/14609#discussion_r1281349938
##########
docs/multi-stage-query/reference.md:
##########
@@ -343,59 +343,23 @@ CLUSTERED BY user
The context parameter that sets `sqlJoinAlgorithm` to `sortMerge` is not shown
in the above example.
-## Durable Storage
+## Durable storage
-Using durable storage with your SQL-based ingestion can improve their
reliability by writing intermediate files to a storage location temporarily.
+SQL-based ingestion supports using durable storage to store intermediate files
temporarily. Enabling it can improve reliability. For more information, see
[Durable storage](../operations/durable-storage.md).
-To prevent durable storage from getting filled up with temporary files in case
the tasks fail to clean them up, a periodic
-cleaner can be scheduled to clean the directories corresponding to which there
isn't a controller task running. It utilizes
-the storage connector to work upon the durable storage. The durable storage
location should only be utilized to store the output
-for cluster's MSQ tasks. If the location contains other files or directories,
then they will get cleaned up as well.
-
-Enabling durable storage also enables the use of local disk to store temporary
files, such as the intermediate files produced
-by the super sorter. Tasks will use whatever has been configured for their
temporary usage as described in [Configuring task storage
sizes](../ingestion/tasks.md#configuring-task-storage-sizes)
-If the configured limit is too low, `NotEnoughTemporaryStorageFault` may be
thrown.
-
-### Enable durable storage
-
-To enable durable storage, you need to set the following common service
properties:
-
-```
-druid.msq.intermediate.storage.enable=true
-druid.msq.intermediate.storage.type=s3
-druid.msq.intermediate.storage.bucket=YOUR_BUCKET
-druid.msq.intermediate.storage.prefix=YOUR_PREFIX
-druid.msq.intermediate.storage.tempDir=/path/to/your/temp/dir
-```
-
-For detailed information about the settings related to durable storage, see
[Durable storage configurations](#durable-storage-configurations).
-
-
-### Use durable storage for queries
-
-When you run a query, include the context parameter `durableShuffleStorage`
and set it to `true`.
-
-For queries where you want to use fault tolerance for workers, set
`faultTolerance` to `true`, which automatically sets `durableShuffleStorage` to
`true`.
-
-Set `selectDestination`:`durableStorage` for select queries that want to write
the final results to durable storage instead of the task reports. Saving the
results in the durable
-storage allows users to fetch large result sets. The location where the
workers write the intermediate results is different than the location where
final results get stored. Therefore, `durableShuffleStorage`:`false` and
-`selectDestination`:`durableStorage` is a valid configuration to use in the
query context, that instructs the controller to persist only the final result
in the durable storage, and not the
-intermediate results.
-
-
-## Durable storage configurations
+### Durable storage configurations
The following common service properties control how durable storage behaves:
|Parameter |Default | Description
|
|-------------------|----------------------------------------|----------------------|
+|`druid.msq.intermediate.storage.enable` | true | Required. Whether to enable
durable storage for the cluster.|
+|`druid.msq.intermediate.storage.type` | `s3` if your deep storage is S3 |
Required. The type of storage to use. You can either set this to `local` or
`s3`. |
Review Comment:
I manually incorporated the changes and suggestions from that PR to avoid
conflicts later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
