This is an automated email from the ASF dual-hosted git repository. roryqi pushed a commit to branch branch-0.10 in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git
commit c4da702357f85a23d1acc233d017c37bf39945ee Author: maobaolong <[email protected]> AuthorDate: Mon Dec 2 01:13:24 2024 +0800 [#436][FOLLOWUP] docs: Add documents for localStorageManagerClass and MultiPartLocalStorageManager (#2266) ### What changes were proposed in this pull request? Add config localStorageManagerClass description to introduce the default value and MultiPartLocalStorageManager. ### Why are the changes needed? Add missing docs. Fix: #436 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No need. --- docs/server_guide.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/server_guide.md b/docs/server_guide.md index 3c6b7750a..af2901235 100644 --- a/docs/server_guide.md +++ b/docs/server_guide.md @@ -124,13 +124,14 @@ This document will introduce how to deploy Uniffle shuffle servers. | rss.server.blockIdManagerClass | org.apache.uniffle.server.block.DefaultShuffleBlockIdManager | The block id manager class. It is used to manage block id. [...] ### Advanced Configurations -| Property Name | Default | Description | -|--------------------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| rss.server.storageMediaProvider.from.env.key | - | Sometimes, the local storage type/media info is provided by external system. RSS would read the env key defined by this configuration and get info about the storage media of its basePaths | -| rss.server.decommission.check.interval | 60000 | The interval(ms) to check if all applications have finish when server is decommissioning | -| rss.server.decommission.shutdown | true | Whether shutdown the server after server is decommissioned | -| rss.server.health.checker.script.path | - | The health script path for `HealthScriptChecker`. To enable `HealthScriptChecker`, need to set `rss.server.health.checker.class.names` and set `rss.server.health.check.enable` to true. | -| rss.server.health.checker.script.execute.timeout | 5000 | Timeout for `HealthScriptChecker` execute health script.(ms) | +| Property Name | Default | Description [...] +|--------------------------------------------------|-------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [...] +| rss.server.storageMediaProvider.from.env.key | - | Sometimes, the local storage type/media info is provided by external system. RSS would read the env key defined by this configuration and get info about the storage media of its basePaths [...] +| rss.server.decommission.check.interval | 60000 | The interval(ms) to check if all applications have finish when server is decommissioning [...] +| rss.server.decommission.shutdown | true | Whether shutdown the server after server is decommissioned [...] +| rss.server.health.checker.script.path | - | The health script path for `HealthScriptChecker`. To enable `HealthScriptChecker`, need to set `rss.server.health.checker.class.names` and set `rss.server.health.check.enable` to true. [...] +| rss.server.health.checker.script.execute.timeout | 5000 | Timeout for `HealthScriptChecker` execute health script.(ms) [...] +| rss.server.localStorageManagerClass | org.apache.uniffle.server.storage.LocalStorageManager | The class for implementing local storage management can be set to `org.apache.uniffle.server.storage.MultiPartLocalStorageManager`. This class enables the storage of each partition across multiple disks in multiple parts. Please note that this configuration requires coordination with the client. Therefore, ensure that the client version is equal to or newer than the this version b [...] ### Huge Partition Optimization A huge partition is a common problem for Spark/MR and so on, caused by data skew. And it can cause the shuffle server to become unstable. To solve this, we introduce some mechanisms to limit the writing of huge partitions to avoid affecting regular partitions, and introduce a hard limit config to reject extremely huge partition, more details can be found in [ISSUE-378](https://github.com/apache/incubator-uniffle/issues/378). The basic rules for limiting large partitions are memory usage [...]
