This patch series introduces a caching mechanism for expensive status
update calls made from pvestatd.
As a first step, I introduced the cache to the arguably
most expensive call, namely `storage_info` from pve-storage. Instead
of caching the results of the `storage_info` function as a whole, we
only cache the results from status updates from individual storage
plugins.
Regarding the cache implementation: I really tried it keep it as simple
as possible. Cached values are not queried terribly often and are
rather small, so I ended up with with simply creating a json-file per
cache key in a directory in tmpfs.
Changes from v1 -> v2:
- Incorporated changes from the first review (thx @Wolfgang, thx @Max)
- Made sure to invalidate the cache for storage plugin if necessary:
- When changing storage config in some way
- When allocating volumes
- When uploading ISOs/Images (might change the amount of free space)
- This allows to increase the expiration time - I've set it now to 60s
- Added a flag to storage status API/CLI that allows one to ignore cached
values,
in case somebody needs up-to-date values at all cost (--ignore-cache)
- Added functions that allow the user of proxmox-cache to lock cache entries
(locking is accomplished using lock files and `flock`)
- Added get_or_update helper - this one tries to get the cached value, and if
it does not exist/is expired computes the new value from a closure/function
passed in as a parameter, all while holding the lock on the cache entry.
Locking while updating the value eliminates possible race conditions if
multiple processes want to update the value at the same time.
Versions of this patch series:
- v1: https://lists.proxmox.com/pipermail/pve-devel/2023-August/058806.html
proxmox:
Lukas Wagner (5):
sys: fs: remove unnecessary clippy allow directive
sys: fs: let CreateOptions::apply_to take RawFd instead of File
sys: fs: use inline formatting for bail! macro
sys: add make_tmp_dir
cache: add new crate 'proxmox-shared-cache'
Cargo.toml | 1 +
proxmox-shared-cache/Cargo.toml | 18 +
proxmox-shared-cache/debian/changelog | 5 +
proxmox-shared-cache/debian/control | 53 ++
proxmox-shared-cache/debian/copyright | 18 +
proxmox-shared-cache/debian/debcargo.toml | 7 +
proxmox-shared-cache/examples/performance.rs | 113 +++++
proxmox-shared-cache/src/lib.rs | 485 +++++++++++++++++++
proxmox-shared-memory/src/lib.rs | 4 +-
proxmox-sys/src/fs/dir.rs | 76 ++-
proxmox-sys/src/fs/file.rs | 4 +-
proxmox-sys/src/fs/mod.rs | 15 +-
12 files changed, 780 insertions(+), 19 deletions(-)
create mode 100644 proxmox-shared-cache/Cargo.toml
create mode 100644 proxmox-shared-cache/debian/changelog
create mode 100644 proxmox-shared-cache/debian/control
create mode 100644 proxmox-shared-cache/debian/copyright
create mode 100644 proxmox-shared-cache/debian/debcargo.toml
create mode 100644 proxmox-shared-cache/examples/performance.rs
create mode 100644 proxmox-shared-cache/src/lib.rs
proxmox-perl-rs:
Lukas Wagner (1):
cache: add bindings for `SharedCache`
common/pkg/Makefile | 1 +
common/pkg/Proxmox/RS/SharedCache.pm | 46 ++++++++++++++
common/src/mod.rs | 1 +
common/src/shared_cache.rs | 89 ++++++++++++++++++++++++++++
pve-rs/Cargo.toml | 1 +
5 files changed, 138 insertions(+)
create mode 100644 common/pkg/Proxmox/RS/SharedCache.pm
create mode 100644 common/src/shared_cache.rs
pve-storage:
Lukas Wagner (1):
stats: api: cache storage plugin status
src/PVE/API2/Storage/Config.pm | 18 +++++++
src/PVE/API2/Storage/Content.pm | 15 ++++++
src/PVE/API2/Storage/Status.pm | 38 ++++++++++++++-
src/PVE/Storage.pm | 84 ++++++++++++++++++++++++---------
4 files changed, 131 insertions(+), 24 deletions(-)
Summary over all repositories:
21 files changed, 1049 insertions(+), 43 deletions(-)
--
murpp v0.4.0
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
