This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 286af3bce97 [doc](file-cache) fix wrong config (#29156)
286af3bce97 is described below
commit 286af3bce976d16bc563717c533b3f1fd28c8a94
Author: Mingyu Chen <[email protected]>
AuthorDate: Wed Dec 27 18:51:01 2023 +0800
[doc](file-cache) fix wrong config (#29156)
* [doc](file-cache) fix wrong config
* 2
---
docs/en/docs/lakehouse/filecache.md | 55 +++++++++++++++++-----------------
docs/zh-CN/docs/lakehouse/filecache.md | 53 ++++++++++++++++----------------
2 files changed, 55 insertions(+), 53 deletions(-)
diff --git a/docs/en/docs/lakehouse/filecache.md
b/docs/en/docs/lakehouse/filecache.md
index fa9272e7088..c2a0304f913 100644
--- a/docs/en/docs/lakehouse/filecache.md
+++ b/docs/en/docs/lakehouse/filecache.md
@@ -53,42 +53,43 @@ SET GLOBAL enable_file_cache = true;
> The File Cache is only applicable to external queries for files (such as
> Hive, Hudi). It has no effect on internal table queries, or non-file
> external queries (such as JDBC, Elasticsearch), etc.
### Configurations for BE
+
Add settings to the BE node's configuration file `conf/be.conf`, and restart
the BE node for the configuration to take effect.
-| Parameter | Description |
-| --- | --- |
-| `enable_file_cache` | Whether to enable File Cache, default false |
-| `file_cache_path` | Parameters about cache path, json format, for exmaple:
`[{"path": "/path/to/file_cache1", "total_size":53687091200,"query_limit":
"10737418240"},{"path": "/path/to/file_cache2",
"total_size":53687091200,"query_limit": "10737418240"},{"path":
"/path/to/file_cache3", "total_size":53687091200,"query_limit":
"10737418240"}]`. `path` is the path to save cached data; `total_size` is the
max size of cached data; `query_limit` is the max size of cached data for a
single query. |
-| `file_cache_min_file_segment_size` | Min size of a single cached block,
default 1MB, should greater than 4096 |
-| `file_cache_max_file_segment_size` | Max size of a single cached block,
default 4MB, should greater than 4096 |
-| `enable_file_cache_query_limit` | Whether to limit the cache size used by a
single query, default false |
-| `clear_file_cache` | Whether to delete the previous cache data when the BE
restarts, default false |
+| Parameter | Required | Description |
+| --- | --- | --- |
+| `enable_file_cache` | Yes | Whether to enable File Cache, default false |
+| `file_cache_path` | Yes | Parameters about cache path, json format, for
exmaple: `[{"path": "/path/to/file_cache1",
"total_size":53687091200,"query_limit": 10737418240},{"path":
"/path/to/file_cache2", "total_size":53687091200,"query_limit":
10737418240},{"path": "/path/to/file_cache3",
"total_size":53687091200,"query_limit": 10737418240}]`. `path` is the path to
save cached data; `total_size` is the max size of cached data; `query_limit` is
the max size of cached data for a single query. |
+| `file_cache_min_file_segment_size` | No | Min size of a single cached block,
default 1MB, should greater than 4096 |
+| `file_cache_max_file_segment_size` | No | Max size of a single cached block,
default 4MB, should greater than 4096 |
+| `enable_file_cache_query_limit` | No | Whether to limit the cache size used
by a single query, default false |
+| `clear_file_cache` | No | Whether to delete the previous cache data when the
BE restarts, default false |
## Check whether a query hits cache
Execute `set enable_profile = true` to enable the session variable, and you
can view the query profile in the Queris tab of FE's web page. The metrics
related to File Cache are as follows:
```
-- FileCache:
- - IOHitCacheNum: 552
- - IOTotalNum: 835
- - ReadFromFileCacheBytes: 19.98 MB
- - ReadFromWriteCacheBytes: 0.00
- - ReadTotalBytes: 29.52 MB
- - WriteInFileCacheBytes: 915.77 MB
- - WriteInFileCacheNum: 283
+- FileCache: 0ns
+ - BytesScannedFromCache: 2.02 GB
+ - BytesScannedFromRemote: 0.00
+ - BytesWriteIntoCache: 0.00
+ - LocalIOUseTimer: 2s723ms
+ - NumLocalIOTotal: 444
+ - NumRemoteIOTotal: 0
+ - NumSkipCacheIOTotal: 0
+ - RemoteIOUseTimer: 0ns
+ - WriteCacheIOUseTimer: 0ns
```
-- `IOTotalNum`: Number of remote access
-- `IOHitCacheNum`: Number of cache hits
-- `ReadFromFileCacheBytes`: Amount of data read from cache file
-- `ReadTotalBytes`: Total amount of data read
-- `SkipCacheBytes`: Failed to create the cache file, or the cache file was
deleted. The amount of data that needs to be read from the remote again
-- `WriteInFileCacheBytes`: Amount of data saved to cache file
-- `WriteInFileCacheNum`: The number of blocks saved, so
'WriteInFileCacheBytes'/' WriteInFileCacheBytes' is the average size of blocks
-
-`IOHitCacheNum` / `IOTotalNum` Equal to 1, indicating that read data only from
file cache
+- `BytesScannedFromCache`: The amount of data read from the local cache.
+- `BytesScannedFromRemote`: The amount of data read from the remote end.
+- `BytesWriteIntoCache`: The amount of data written to the cache.
+- `LocalIOUseTimer`: Local cached IO time.
+- `RemoteIOUseTimer`: IO time for remote reading.
+- `NumLocalIOTotal`: The number of locally cached IOs.
+- `NumRemoteIOTotal`: Number of remote IOs.
+- `WriteCacheIOUseTimer`: IO time to write cache.
-`ReadFromFileCacheBytes` / `ReadTotalBytes` Equal to 1, indicating that read
data only from file cache
+If `BytesScannedFromRemote` is 0, it means all caches are hit.
-`ReadFromFileCacheBytes` The smaller the better, the smaller the amount of
data read from remote
diff --git a/docs/zh-CN/docs/lakehouse/filecache.md
b/docs/zh-CN/docs/lakehouse/filecache.md
index 79035141e2d..d153c811f9d 100644
--- a/docs/zh-CN/docs/lakehouse/filecache.md
+++ b/docs/zh-CN/docs/lakehouse/filecache.md
@@ -57,39 +57,40 @@ SET GLOBAL enable_file_cache = true;
添加参数到 BE 节点的配置文件 conf/be.conf 中，并重启 BE 节点让配置生效。
-| 参数 | 说明 |
-| --- | --- |
-| `enable_file_cache` | 是否启用 File Cache，默认 false |
-| `file_cache_path` | 缓存目录的相关配置，json格式，例子: `[{"path": "/path/to/file_cache1",
"total_size":53687091200,"query_limit": "10737418240"},{"path":
"/path/to/file_cache2", "total_size":53687091200,"query_limit":
"10737418240"},{"path": "/path/to/file_cache3",
"total_size":53687091200,"query_limit": "10737418240"}]`。`path`
是缓存的保存路径，`total_size` 是缓存的大小上限，`query_limit` 是单个查询能够使用的最大缓存大小。 |
-| `file_cache_min_file_segment_size` | 单个 Block 的大小下限，默认 1MB，需要大于 4096 |
-| `file_cache_max_file_segment_size` | 单个 Block 的大小上限，默认 4MB，需要大于 4096 |
-| `enable_file_cache_query_limit` | 是否限制单个 query 使用的缓存大小，默认 false |
-| `clear_file_cache` | BE 重启时是否删除之前的缓存数据，默认 false |
+| 参数 | 必选项 | 说明 |
+| --- | --- | --- |
+| `enable_file_cache` | 是 | 是否启用 File Cache，默认 false |
+| `file_cache_path` | 是 | 缓存目录的相关配置，json格式，例子: `[{"path":
"/path/to/file_cache1", "total_size":53687091200,"query_limit":
10737418240},{"path": "/path/to/file_cache2",
"total_size":53687091200,"query_limit": 10737418240},{"path":
"/path/to/file_cache3", "total_size":53687091200,"query_limit":
10737418240}]`。`path` 是缓存的保存路径，`total_size` 是缓存的大小上限，`query_limit`
是单个查询能够使用的最大缓存大小。 |
+| `file_cache_min_file_segment_size` | 否 | 单个 Block 的大小下限，默认 1MB，需要大于 4096 |
+| `file_cache_max_file_segment_size` | 否 | 单个 Block 的大小上限，默认 4MB，需要大于 4096 |
+| `enable_file_cache_query_limit` | 否 | 是否限制单个 query 使用的缓存大小，默认 false |
+| `clear_file_cache` | 否 | BE 重启时是否删除之前的缓存数据，默认 false |
### 查看 File Cache 命中情况
执行 `set enable_profile=true` 打开会话变量，可以在 FE 的 web 页面的 Queris 标签中查看到作业的
Profile。File Cache 相关的指标如下:
```
-- FileCache:
- - IOHitCacheNum: 552
- - IOTotalNum: 835
- - ReadFromFileCacheBytes: 19.98 MB
- - ReadFromWriteCacheBytes: 0.00
- - ReadTotalBytes: 29.52 MB
- - WriteInFileCacheBytes: 915.77 MB
- - WriteInFileCacheNum: 283
+- FileCache: 0ns
+ - BytesScannedFromCache: 2.02 GB
+ - BytesScannedFromRemote: 0.00
+ - BytesWriteIntoCache: 0.00
+ - LocalIOUseTimer: 2s723ms
+ - NumLocalIOTotal: 444
+ - NumRemoteIOTotal: 0
+ - NumSkipCacheIOTotal: 0
+ - RemoteIOUseTimer: 0ns
+ - WriteCacheIOUseTimer: 0ns
```
-- `IOTotalNum`: 远程访问的次数
-- `IOHitCacheNum`: 命中缓存的次数
-- `ReadFromFileCacheBytes`: 从缓存文件中读取的数据量
-- `ReadTotalBytes`: 总共读取的数据量
-- `SkipCacheBytes`: 创建缓存文件失败，或者缓存文件被删，需要再次从远程读取的数据量
-- `WriteInFileCacheBytes`: 保存到缓存文件中的数据量
-- `WriteInFileCacheNum`: 保存的 Block 数量，所以
`WriteInFileCacheBytes`/`WriteInFileCacheBytes` 为 Block 的平均大小
+- `BytesScannedFromCache`：从本地缓存中读取的数据量。
+- `BytesScannedFromRemote`：从远端读取的数据量。
+- `BytesWriteIntoCache`：写入缓存的数据量。
+- `LocalIOUseTimer`：本地缓存的 IO 时间。
+- `RemoteIOUseTimer`：远端读取的 IO 时间。
+- `NumLocalIOTotal`：本地缓存的 IO 次数。
+- `NumRemoteIOTotal`：远端 IO 次数。
+- `WriteCacheIOUseTimer`：写入缓存的 IO 时间。
-`IOHitCacheNum` / `IOTotalNum` 等于1，表示缓存完全命中
-
-`ReadFromFileCacheBytes` / `ReadTotalBytes` 等于1，表示缓存完全命中
+如果 `BytesScannedFromRemote` 为 0，表示全部命中缓存。
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
