abhishekrb19 commented on code in PR #18124: URL: https://github.com/apache/druid/pull/18124#discussion_r2141507184
########## docs/configuration/index.md: ########## @@ -1186,6 +1191,15 @@ The following properties pertain to segment metadata caching on the Overlord tha |`druid.manager.segments.useIncrementalCache`|Denotes the usage mode of the segment metadata incremental cache. Possible modes are: (a) `never`: Cache is disabled. (b) `always`: Reads are always done from the cache. Service start-up will be blocked until cache has synced with the metadata store at least once. Transactions will block until cache has synced with the metadata store at least once after becoming leader. (c) `ifSynced`: Reads are done from the cache only if it has already synced with the metadata store. This mode does not block service start-up or transactions.|`never`| |`druid.manager.segments.pollDuration`|Duration (in ISO 8601 format) between successive syncs of the cache with the metadata store. This property is used only when `druid.manager.segments.useIncrementalCache` is set to `always` or `ifSynced`.|`PT1M` (1 minute)| +##### Auto-kill unused segments (Experimental) + +These configs pertain to the new embedded mode of running [kill tasks on the Overlord](../data-management/delete.md#auto-kill-data-on-the-overlord-experimental). + Review Comment: I think it would be good to explicitly call out that none of the coordinator-based auto-kill configs apply to this overlord-based auto-kill feature (even in the combined coordinator+OL mode). Bringing this up because I had that assumption initially when reviewing https://github.com/apache/druid/pull/18028. The reasons are already cross-linked in `data-management/delete.md`, so it should be clear for anyone curious. ########## docs/data-management/delete.md: ########## @@ -116,3 +116,33 @@ Some of the parameters used in the task payload are further explained below: **WARNING:** The `kill` task permanently removes all information about the affected segments from the metadata store and deep storage. This operation cannot be undone. +### Auto-kill data using Coordinator duties + +Instead of submitting `kill` tasks manually to permanently delete data for a given interval, you can enable auto-kill of unused segments on the Coordinator. +The Coordinator runs a duty periodically to identify intervals containing unused segments that are eligible for kill. It then launches a `kill` task for each of these intervals. + +Refer to [Data management on the Coordinator](../configuration/index.md#data-management) to configure auto-kill of unused segments on the Coordinator. + +### Auto-kill data on the Overlord (Experimental) + +:::info +This feature MUST NOT be enabled if auto-kill of unused segments is already enabled on the Coordinator. +::: + +This is an experimental feature to run kill tasks in an "embedded" mode on the Overlord itself. + +These embedded tasks offer several advantages over auto-kill performed by the Coordinator as they: +- avoid a lot of unnecessary REST API calls to the Overlord from tasks or the Coordinator. +- kill unused segments as soon as they become eligible. +- run on the Overlord and do not take up task slots. +- finish faster as they save on the overhead of launching a task process. +- kill a small number of segments per task, to ensure that locks on an interval are not held for too long. +- skip locked intervals to avoid head-of-line blocking in kill tasks. +- require little to no configuration. +- can keep up with a large number of unused segments in the cluster. +- take advantage of the segment metadata cache on the Overlord. + +This feature can be used only if [segment metadata caching](../configuration/index.md#segment-metadata-cache-experimental) is enabled on the Overlord. Review Comment: It would be better to move this prerequisite above, perhaps into a separate `::info` section for visibility. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
