This is an automated email from the ASF dual-hosted git repository.
zwoop pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/trafficserver.git
The following commit(s) were added to refs/heads/master by this push:
new 8ea9548fbe Cache volumes: RAM cache settings and remap option (#12717)
8ea9548fbe is described below
commit 8ea9548fbe7619f5aebf7346cb26fffd4c471c61
Author: Leif Hedstrom <[email protected]>
AuthorDate: Fri Mar 13 23:17:03 2026 -0700
Cache volumes: RAM cache settings and remap option (#12717)
* First cut at new volume configs for RAM cache
* Added support for @volume= in remap.config
This gets a bit complicated because we allow for ATS to start
up with remap before the cache is properly started.
* Adds a default volume list to records.yaml
* Adds some autests
* Address Chris' review comment
* Address Copilot review comments
- foreach_mapping: remove const (Trie only has const_iterator, so
const_cast is still needed but function is now semantically correct)
- Cache.cc: document default_volumes_host_rec as intentional singleton
- CacheHosting.cc: restore tmp++ (tmp=line_end broke multi-param lines),
remove unreachable null check after new, use char array for "volume"
literal instead of const_cast
- ReverseProxy.cc: add acquire()/release() around rewrite_table access
in init_remap_volume_host_records, document startup-only constraint
- Test files: remove trailing commas from ATSReplayTest calls
* Address review comments from bryancall
- CacheHosting.cc: restrict suffix skip loops to uppercase KMGT to
match ink_atoi64's uppercase-only handling
- RemapConfig.cc: use swoc::TextView to validate @volume= segments
(rejects empty, zero, out-of-range, trailing comma)
- ReverseProxy.h/cc, HttpSM.cc: use std::atomic<UrlRewrite*> for
rewrite_table to make atomic access explicit
- cache_volume_features.replay.yaml: remove misleading @volume=99
remap entry and its corresponding session test
* Try to prove the code correctly parses configs
---
configs/volume.config.default | 33 ++++-
doc/admin-guide/files/records.yaml.en.rst | 38 ++++++
doc/admin-guide/files/remap.config.en.rst | 48 +++++++
doc/admin-guide/files/volume.config.en.rst | 86 ++++++++++--
include/iocore/cache/Cache.h | 5 +-
include/proxy/ReverseProxy.h | 7 +-
include/proxy/http/HttpTransact.h | 8 +-
include/proxy/http/remap/RemapConfig.h | 1 +
include/proxy/http/remap/UrlMapping.h | 37 +++++-
include/proxy/http/remap/UrlMappingPathIndex.h | 13 ++
src/iocore/cache/Cache.cc | 102 ++++++++++-----
src/iocore/cache/CacheHosting.cc | 120 +++++++++++++++--
src/iocore/cache/CacheProcessor.cc | 121 +++++++++++++----
src/iocore/cache/CacheVC.cc | 12 +-
src/iocore/cache/P_CacheHosting.h | 49 +++++--
src/iocore/cache/P_CacheInternal.h | 11 +-
src/iocore/cache/Stripe.h | 2 +
src/proxy/ReverseProxy.cc | 76 +++++++++--
src/proxy/http/HttpCacheSM.cc | 20 ++-
src/proxy/http/HttpSM.cc | 2 +-
src/proxy/http/remap/RemapConfig.cc | 76 ++++++++++-
src/proxy/http/remap/RemapProcessor.cc | 3 +
src/proxy/http/remap/UrlMapping.cc | 33 +++++
src/records/RecordsConfig.cc | 2 +
src/traffic_server/traffic_server.cc | 4 +
tests/gold_tests/autest-site/ats_replay.test.ext | 12 ++
.../cache/cache_volume_defaults.replay.yaml | 144 +++++++++++++++++++++
.../gold_tests/cache/cache_volume_defaults.test.py | 24 ++++
.../cache/cache_volume_features.replay.yaml | 141 ++++++++++++++++++++
.../gold_tests/cache/cache_volume_features.test.py | 25 ++++
30 files changed, 1133 insertions(+), 122 deletions(-)
diff --git a/configs/volume.config.default b/configs/volume.config.default
index c1aa03644a..6eafd85aed 100644
--- a/configs/volume.config.default
+++ b/configs/volume.config.default
@@ -29,6 +29,37 @@
# disk (assuming each disk has enough free space available).
#
# To create one volume of size 10% of the total cache space and
-# another 1 Gig volume,
+# another 1 Gig volume,
# volume=1 scheme=http size=10%
# volume=2 scheme=http size=1024
+#
+# Additional optional parameters:
+#
+# ramcache=true/false
+# Enable or disable RAM cache for this volume (default: true)
+#
+# ram_cache_size=<size>
+# Allocate a dedicated RAM cache pool for this volume (e.g., 512M, 2G)
+# This amount is automatically subtracted from the global ram_cache.size
+# setting, with the remainder shared among other volumes.
+#
+# ram_cache_cutoff=<size>
+# Override the global ram_cache_cutoff for this volume (e.g., 64K, 1M)
+# Objects larger than this will not be stored in RAM cache.
+#
+# avg_obj_size=<size>
+# Override the global min_average_object_size for this volume
+#
+# fragment_size=<size>
+# Override the global target_fragment_size for this volume (max: 4MB)
+#
+# Advanced RAM cache configuration examples:
+#
+# Example 1: Volume with dedicated 2GB RAM cache
+# volume=1 scheme=http size=40% ram_cache_size=2G
+#
+# Example 2: Small objects with custom cutoff and dedicated RAM
+# volume=2 scheme=http size=20% ram_cache_size=512M ram_cache_cutoff=64K
+#
+# Example 3: Large media with higher cutoff (shares remaining RAM pool)
+# volume=3 scheme=http size=40% ram_cache_cutoff=1M
diff --git a/doc/admin-guide/files/records.yaml.en.rst
b/doc/admin-guide/files/records.yaml.en.rst
index 2232c0b3fd..f6691e00d4 100644
--- a/doc/admin-guide/files/records.yaml.en.rst
+++ b/doc/admin-guide/files/records.yaml.en.rst
@@ -2681,6 +2681,34 @@ Cache Control
used in determining the number of :term:`directory buckets <directory
bucket>`
to allocate for the in-memory cache directory.
+.. ts:cv:: CONFIG proxy.config.cache.default_volumes STRING ""
+
+ Specifies a comma-separated list of cache volume numbers to use as the
default
+ for cache stripe selection when no more specific volume configuration
applies.
+ For example, ``"1,2"`` would use volumes 1 and 2 as the default.
+
+ The volume selection priority order is:
+
+ 1. ``@volume=`` directive in :file:`remap.config` (highest priority)
+ 2. Hostname matching in :file:`hosting.config`
+ 3. ``proxy.config.cache.default_volumes`` (if non-empty)
+ 4. All available cache volumes (lowest priority)
+
+ An empty string (the default) disables this feature, causing |TS| to fall
+ back directly to using all available volumes when no other configuration
+ matches.
+
+ This is useful for scenarios where you want to restrict default caching to
+ specific volumes without configuring hostname patterns in
:file:`hosting.config`.
+ For example, you might want to reserve certain volumes for specific remap
rules
+ while having a different set of default volumes for all other traffic.
+
+.. topic:: Example
+
+ Assign volumes 1 and 2 as defaults for general traffic ::
+
+ CONFIG proxy.config.cache.default_volumes STRING "1,2"
+
.. ts:cv:: CONFIG proxy.config.cache.permit.pinning INT 0
:reloadable:
@@ -2821,6 +2849,11 @@ RAM Cache
Alternatively, it can be set to a fixed value such as
**20GB** (21474836480)
+ This global setting can be overridden on a per-volume basis using the
+ ``ram_cache_size`` parameter in :file:`volume.config`. Per-volume
+ allocations are subtracted from the total RAM cache size before
+ distributing the remainder among volumes without explicit settings.
+
.. ts:cv:: CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304
Objects greater than this size will not be kept in the RAM cache.
@@ -2828,6 +2861,11 @@ RAM Cache
in memory in order to improve performance.
**4MB** (4194304)
+ This global setting can be overridden on a per-volume basis using the
+ ``ram_cache_cutoff`` parameter in :file:`volume.config`. When set,
+ the per-volume cutoff takes precedence over this global setting for
+ that specific volume.
+
.. ts:cv:: CONFIG proxy.config.cache.ram_cache.algorithm INT 1
Two distinct RAM caches are supported, the default (1) being the simpler
diff --git a/doc/admin-guide/files/remap.config.en.rst
b/doc/admin-guide/files/remap.config.en.rst
index 7578750216..7301282373 100644
--- a/doc/admin-guide/files/remap.config.en.rst
+++ b/doc/admin-guide/files/remap.config.en.rst
@@ -455,6 +455,54 @@ will pass "1" and "2" to plugin1.so and "3" to plugin2.so.
This will pass "1" and "2" to plugin1.so and "3" to plugin2.so
+.. _remap-config-cache-volume-selection:
+
+Cache Volume Selection
+======================
+
+The ``@volume`` directive allows you to override the default cache volume
selection
+for specific remap rules, bypassing the hostname-based volume selection
configured in
+:file:`hosting.config`. This provides fine-grained control over which cache
volumes
+are used for different URL patterns.
+
+Format
+------
+
+::
+
+ @volume=<volume_list>
+
+Where ``<volume_list>`` can be either:
+
+- A single volume number: ``@volume=4``
+- Multiple comma-separated volume numbers: ``@volume=3,4,5``
+
+Volume numbers must be between 1 and 255 (volume 0 is reserved and not usable).
+All specified volumes must be defined in :file:`volume.config`.
+
+Examples
+--------
+
+::
+
+ # Single volume for API requests (backward compatibility)
+ map https://api.example.com/ https://api-origin.example.com/ @volume=4
+
+ # Multiple volumes for load distribution across SSD volumes
+ map https://cdn.example.com/ https://cdn-origin.example.com/ @volume=2,3,4
+
+ # Single high-performance volume for critical services
+ map https://checkout.example.com/ https://checkout-origin.example.com/
@volume=1
+
+ # Everything else gets the default volume allocations (hosting.config
rules)
+ map https://www.example.com/ https://origin.example.com/
+
+.. note::
+
+ When using ``@volume``, ensure that the target volumes have appropriate
disk space and
+ performance characteristics for the expected traffic patterns. For multiple
volumes,
+ consider the combined capacity and performance of all specified volumes.
+
.. _remap-config-named-filters:
NextHop Selection Strategies
diff --git a/doc/admin-guide/files/volume.config.en.rst
b/doc/admin-guide/files/volume.config.en.rst
index 70ec2725af..f26a122eef 100644
--- a/doc/admin-guide/files/volume.config.en.rst
+++ b/doc/admin-guide/files/volume.config.en.rst
@@ -73,20 +73,67 @@ Optional directory entry sizing
You can also add an option ``avg_obj_size=<size>`` to the volume configuration
line. This overrides the global
:ts:cv:`proxy.config.cache.min_average_object_size`
-configuration for this volume. This is useful if you have a volume that is
dedicated
-for say very small objects, and you need a lot of directory entries to store
them.
+configuration for this volume. The size supports multipliers (K, M, G, T) for
+convenience (e.g., ``avg_obj_size=64K`` or ``avg_obj_size=1M``). This is useful
+if you have a volume that is dedicated for say very small objects, and you need
+a lot of directory entries to store them.
Optional fragment size setting
------------------------------
You can also add an option ``fragment_size=<size>`` to the volume configuration
line. This overrides the global
:ts:cv:`proxy.config.cache.target_fragment_size`
-configuration for this volume. This allows for a smaller, or larger, fragment
size
-for a particular volume. This may be useful together with ``avg_obj_size`` as
well,
-since a larger fragment size could reduce the number of directory entries
needed
-for a large object.
-
-Note that this setting has a maximmum value of 4MB.
+configuration for this volume. The size supports multipliers (K, M, G, T) for
+convenience (e.g., ``fragment_size=512K`` or ``fragment_size=2M``). This allows
+for a smaller, or larger, fragment size for a particular volume. This may be
+useful together with ``avg_obj_size`` as well, since a larger fragment size
could
+reduce the number of directory entries needed for a large object.
+
+Note that this setting has a maximum value of 4MB.
+
+Optional RAM cache size allocation
+-----------------------------------
+
+You can add an option ``ram_cache_size=<size>`` to the volume configuration
line
+to allocate a dedicated RAM cache pool for this volume. The size supports
+multipliers (K, M, G, T) for convenience (e.g., ``ram_cache_size=512M`` or
+``ram_cache_size=2G``). Setting ``ram_cache_size=0`` disables the RAM cache
+for this volume, which is equivalent to ``ramcache=false``.
+
+When ``ram_cache_size`` is specified for a volume, that amount is
**automatically
+subtracted** from the global :ts:cv:`proxy.config.cache.ram_cache.size`
setting,
+and the remainder is shared among volumes without private allocations. This
ensures
+total RAM cache usage never exceeds the configured global limit.
+
+For example, if the global RAM cache size is 4GB and you allocate 1GB to
volume 1
+and 512MB to volume 2, the remaining 2.5GB will be distributed among other
volumes
+using the normal proportional allocation based on disk space.
+
+**Important notes:**
+
+* If the sum of all ``ram_cache_size`` allocations exceeds the global RAM
cache size,
+ Traffic Server will fail to start with a fatal error. Increase
+ :ts:cv:`proxy.config.cache.ram_cache.size` or reduce the per-volume
allocations.
+* If ``ramcache=false`` is set alongside ``ram_cache_size``, the
``ram_cache_size``
+ is ignored (with a warning) since the RAM cache is disabled for that volume.
+* This setting only takes effect when
:ts:cv:`proxy.config.cache.ram_cache.size`
+ is set to a positive value (not ``-1`` for automatic sizing).
+
+Optional RAM cache cutoff override
+-----------------------------------
+
+You can add an option ``ram_cache_cutoff=<size>`` to the volume configuration
line
+to override the global :ts:cv:`proxy.config.cache.ram_cache_cutoff` setting for
+this specific volume. The size supports multipliers (K, M, G, T) for
convenience
+(e.g., ``ram_cache_cutoff=64K`` or ``ram_cache_cutoff=1M``).
+
+This cutoff determines the maximum object size that will be stored in the RAM
cache.
+Objects larger than this size will only be stored on disk. Setting different
cutoffs
+per volume allows you to:
+
+* Use larger cutoffs for volumes serving frequently accessed large objects
+* Use smaller cutoffs for volumes with many small objects to maximize RAM
cache hits
+* Disable RAM caching entirely for certain objects by setting a very low cutoff
Exclusive spans and volume sizes
================================
@@ -126,5 +173,24 @@ ramcache has been disabled.::
volume=1 scheme=http size=20%
volume=2 scheme=http size=20%
volume=3 scheme=http size=20%
- volume=4 scheme=http size=20% avg_obj_size=4096
- volume=5 scheme=http size=20% ramcache=false fragment_size=524288
+ volume=4 scheme=http size=20% avg_obj_size=4K
+ volume=5 scheme=http size=20% ramcache=false fragment_size=512K
+
+The following example shows advanced RAM cache configuration with dedicated
+allocations and custom cutoffs::
+
+ # Volume 1: General content with 2GB dedicated RAM cache
+ volume=1 scheme=http size=40% ram_cache_size=2G
+
+ # Volume 2: Small API responses with custom cutoff and 512MB RAM cache
+ volume=2 scheme=http size=20% ram_cache_size=512M ram_cache_cutoff=64K
+
+ # Volume 3: Large media with higher cutoff for thumbnails
+ volume=3 scheme=http size=40% ram_cache_cutoff=1M
+
+In this example, assuming a global ``proxy.config.cache.ram_cache.size`` of
4GB:
+
+* Volume 1 gets a dedicated 2GB RAM cache allocation
+* Volume 2 gets a dedicated 512MB RAM cache allocation and only caches objects
up to 64KB
+* Volume 3 shares from the remaining 1.5GB pool (4GB - 2GB - 512MB) and caches
objects up to 1MB
+* The automatic subtraction ensures total RAM usage stays within the 4GB limit
diff --git a/include/iocore/cache/Cache.h b/include/iocore/cache/Cache.h
index 2e8369c909..3a7da523b5 100644
--- a/include/iocore/cache/Cache.h
+++ b/include/iocore/cache/Cache.h
@@ -52,6 +52,7 @@ struct CacheDisk;
class URL;
class HTTPHdr;
class HTTPInfo;
+struct CacheHostRecord;
using CacheHTTPHdr = HTTPHdr;
using CacheURL = URL;
@@ -83,9 +84,9 @@ struct CacheProcessor : public Processor {
Action *scan(Continuation *cont, std::string_view hostname =
std::string_view{}, int KB_per_second = SCAN_KB_PER_SECOND);
Action *lookup(Continuation *cont, const HttpCacheKey *key, CacheFragType
frag_type = CACHE_FRAG_TYPE_HTTP);
Action *open_read(Continuation *cont, const HttpCacheKey *key, CacheHTTPHdr
*request, const HttpConfigAccessor *params,
- CacheFragType frag_type = CACHE_FRAG_TYPE_HTTP);
+ CacheFragType frag_type = CACHE_FRAG_TYPE_HTTP, const
CacheHostRecord *volume_host_rec = nullptr);
Action *open_write(Continuation *cont, const HttpCacheKey *key,
CacheHTTPInfo *old_info, time_t pin_in_cache = 0,
- CacheFragType frag_type = CACHE_FRAG_TYPE_HTTP);
+ CacheFragType frag_type = CACHE_FRAG_TYPE_HTTP, const
CacheHostRecord *volume_host_rec = nullptr);
Action *remove(Continuation *cont, const HttpCacheKey *key, CacheFragType
frag_type = CACHE_FRAG_TYPE_HTTP);
/** Mark physical disk/device/file as offline.
diff --git a/include/proxy/ReverseProxy.h b/include/proxy/ReverseProxy.h
index e7aca26ca0..c24748b796 100644
--- a/include/proxy/ReverseProxy.h
+++ b/include/proxy/ReverseProxy.h
@@ -32,6 +32,8 @@
#pragma once
+#include <atomic>
+
#include "records/RecProcess.h"
#include "tscore/ink_defs.h"
@@ -45,7 +47,7 @@
class url_mapping;
struct host_hdr_info;
-extern UrlRewrite *rewrite_table;
+extern std::atomic<UrlRewrite *> rewrite_table;
// API Functions
int init_reverse_proxy();
@@ -57,4 +59,5 @@ bool response_url_remap(HTTPHdr *response_header,
UrlRewrite *table);
bool reloadUrlRewrite();
bool urlRewriteVerify();
-int url_rewrite_CB(const char *name, RecDataT data_type, RecData data, void
*cookie);
+void init_remap_volume_host_records();
+int url_rewrite_CB(const char *name, RecDataT data_type, RecData data, void
*cookie);
diff --git a/include/proxy/http/HttpTransact.h
b/include/proxy/http/HttpTransact.h
index 0e05c1e14b..52c92a9cb7 100644
--- a/include/proxy/http/HttpTransact.h
+++ b/include/proxy/http/HttpTransact.h
@@ -102,6 +102,7 @@ using ink_time_t = time_t;
struct HttpConfigParams;
class HttpSM;
+struct CacheHostRecord;
#include "iocore/net/ConnectionTracker.h"
#include "tscore/InkErrno.h"
@@ -494,6 +495,8 @@ public:
URL *parent_selection_url = nullptr;
URL parent_selection_url_storage;
+ const CacheHostRecord *volume_host_rec = nullptr;
+
_CacheLookupInfo() {}
};
@@ -715,8 +718,9 @@ public:
MgmtByte cache_open_write_fail_action = 0;
- HttpConfigParams *http_config_param = nullptr;
- CacheLookupInfo cache_info;
+ HttpConfigParams *http_config_param = nullptr;
+ CacheLookupInfo cache_info;
+
ResolveInfo dns_info;
RedirectInfo redirect_info;
ConnectionTracker::TxnState outbound_conn_track_state;
diff --git a/include/proxy/http/remap/RemapConfig.h
b/include/proxy/http/remap/RemapConfig.h
index 8456dd846c..559604e560 100644
--- a/include/proxy/http/remap/RemapConfig.h
+++ b/include/proxy/http/remap/RemapConfig.h
@@ -40,6 +40,7 @@ class UrlRewrite;
#define REMAP_OPTFLG_INTERNAL 0x0080u /* only allow internal
requests to hit this remap */
#define REMAP_OPTFLG_IN_IP 0x0100u /* "in_ip=" option (used for
ACL filtering)*/
#define REMAP_OPTFLG_STRATEGY 0x0200u /* "strategy=" the name of
the nexthop selection strategy */
+#define REMAP_OPTFLG_VOLUME 0x0400u /* "volume=" cache volume
override */
#define REMAP_OPTFLG_MAP_ID 0x0800u /* associate a map ID with
this rule */
#define REMAP_OPTFLG_INVERT 0x80000000u /* "invert" the rule (for
src_ip and src_ip_category at least) */
#define REMAP_OPTFLG_ALL_FILTERS \
diff --git a/include/proxy/http/remap/UrlMapping.h
b/include/proxy/http/remap/UrlMapping.h
index dabab07118..8888e1568f 100644
--- a/include/proxy/http/remap/UrlMapping.h
+++ b/include/proxy/http/remap/UrlMapping.h
@@ -24,6 +24,8 @@
#pragma once
+#include <atomic>
+#include <string>
#include <vector>
#include "tscore/ink_config.h"
@@ -36,6 +38,7 @@
#include "tscore/List.h"
class NextHopSelectionStrategy;
+struct CacheHostRecord;
/**
* Used to store http referrer strings (and/or regexp)
@@ -112,20 +115,41 @@ public:
bool ip_allow_check_enabled_p = false;
acl_filter_rule *filter = nullptr; // acl filtering
(linked list of rules)
LINK(url_mapping, link); // For use with the
main Queue linked list holding all the mapping
- NextHopSelectionStrategy *strategy = nullptr;
- std::string remapKey;
- std::atomic<uint64_t> _hitCount = 0; // counter can overflow
+ NextHopSelectionStrategy *strategy = nullptr;
+ std::string remapKey;
+ std::atomic<uint64_t> _hitCount = 0; // counter can overflow
+ std::atomic<CacheHostRecord *> volume_host_rec = nullptr;
+
+ CacheHostRecord *
+ getVolumeHostRec() const
+ {
+ return volume_host_rec.load(std::memory_order_acquire);
+ }
+
+ void
+ setVolume(const char *str)
+ {
+ if (str && *str) {
+ _volume_str = str;
+ }
+ }
+
+ const std::string &
+ getVolume() const
+ {
+ return _volume_str;
+ }
int
getRank() const
{
return _rank;
- };
+ }
void
setRank(int rank)
{
_rank = rank;
- };
+ }
void
setRemapKey()
@@ -145,9 +169,12 @@ public:
_hitCount++;
}
+ bool initVolumeHostRec(char *errbuf, size_t errbufsize);
+
private:
std::vector<RemapPluginInst *> _plugin_inst_list;
int _rank = 0;
+ std::string _volume_str;
};
/**
diff --git a/include/proxy/http/remap/UrlMappingPathIndex.h
b/include/proxy/http/remap/UrlMappingPathIndex.h
index 63bc90cd6e..a74156c9f2 100644
--- a/include/proxy/http/remap/UrlMappingPathIndex.h
+++ b/include/proxy/http/remap/UrlMappingPathIndex.h
@@ -40,6 +40,19 @@ public:
void Print() const;
std::string PrintUrlMappingPathIndex() const;
+ // Apply a function to each url_mapping in this index.
+ // Note: Trie only exposes const_iterator, so const_cast is required.
+ template <typename Func>
+ void
+ foreach_mapping(Func &&f)
+ {
+ for (auto &trie_pair : m_tries) {
+ for (auto const &mapping : *trie_pair.second) {
+ f(const_cast<url_mapping &>(mapping));
+ }
+ }
+ }
+
private:
using UrlMappingTrie = Trie<url_mapping>;
diff --git a/src/iocore/cache/Cache.cc b/src/iocore/cache/Cache.cc
index 512a5e7bf7..14855e98fa 100644
--- a/src/iocore/cache/Cache.cc
+++ b/src/iocore/cache/Cache.cc
@@ -100,6 +100,8 @@ ClassAllocator<EvacuationBlock, false>
evacuationBlockAllocator("evacuationBl
ClassAllocator<CacheRemoveCont, false>
cacheRemoveContAllocator("cacheRemoveCont");
ClassAllocator<EvacuationKey, false>
evacuationKeyAllocator("evacuationKey");
std::unordered_set<std::string> known_bad_disks;
+// Process-lifetime singleton: allocated during cache init and intentionally
never freed.
+CacheHostRecord *default_volumes_host_rec = nullptr;
namespace
{
@@ -231,7 +233,7 @@ Cache::open_done()
}
ReplaceablePtr<CacheHostTable>::ScopedReader hosttable(&this->hosttable);
- if (hosttable->gen_host_rec.num_cachevols == 0) {
+ if (hosttable->getGenHostRecCacheVols() == 0) {
ready = CacheInitState::FAILED;
} else {
ready = CacheInitState::INITIALIZED;
@@ -242,6 +244,22 @@ Cache::open_done()
Emergency("Failed to initialize cache host table");
}
+ // Initialize default_volumes_host_rec from
proxy.config.cache.default_volumes
+ if (ready == CacheInitState::INITIALIZED && default_volumes_host_rec ==
nullptr) {
+ auto default_volumes_str =
RecGetRecordStringAlloc("proxy.config.cache.default_volumes");
+
+ if (default_volumes_str && !default_volumes_str.value().empty()) {
+ char errbuf[256];
+
+ default_volumes_host_rec =
createCacheHostRecord(default_volumes_str.value().c_str(), errbuf,
sizeof(errbuf));
+ if (default_volumes_host_rec != nullptr) {
+ Dbg(dbg_ctl_cache_init, "Initialized default_volumes from '%s'",
default_volumes_str.value().c_str());
+ } else {
+ Warning("Failed to parse proxy.config.cache.default_volumes '%s': %s",
default_volumes_str.value().c_str(), errbuf);
+ }
+ }
+ }
+
cacheProcessor.cacheInitialized();
return 0;
@@ -529,7 +547,7 @@ Cache::scan(Continuation *cont, std::string_view hostname,
int KB_per_second) co
Action *
Cache::open_read(Continuation *cont, const CacheKey *key, CacheHTTPHdr
*request, const HttpConfigAccessor *params,
- CacheFragType type, std::string_view hostname) const
+ CacheFragType type, std::string_view hostname, const
CacheHostRecord *volume_host_rec) const
{
if (!CacheProcessor::IsCacheReady(type)) {
cont->handleEvent(CACHE_EVENT_OPEN_READ_FAILED, reinterpret_cast<void
*>(-ECACHE_NOT_READY));
@@ -537,7 +555,7 @@ Cache::open_read(Continuation *cont, const CacheKey *key,
CacheHTTPHdr *request,
}
ink_assert(caches[type] == this);
- StripeSM *stripe = key_to_stripe(key, hostname);
+ StripeSM *stripe = key_to_stripe(key, hostname, volume_host_rec);
Dir result, *last_collision = nullptr;
ProxyMutex *mutex = cont->mutex.get();
OpenDirEntry *od = nullptr;
@@ -603,8 +621,8 @@ Lcallreturn:
// main entry point for writing of http documents
Action *
-Cache::open_write(Continuation *cont, const CacheKey *key, CacheHTTPInfo
*info, time_t apin_in_cache, CacheFragType type,
- std::string_view hostname) const
+Cache::open_write(Continuation *cont, const CacheKey *key, CacheHTTPInfo
*old_info, time_t pin_in_cache, CacheFragType type,
+ std::string_view hostname, const CacheHostRecord
*volume_host_rec) const
{
if (!CacheProcessor::IsCacheReady(type)) {
cont->handleEvent(CACHE_EVENT_OPEN_WRITE_FAILED, reinterpret_cast<void
*>(-ECACHE_NOT_READY));
@@ -613,7 +631,7 @@ Cache::open_write(Continuation *cont, const CacheKey *key,
CacheHTTPInfo *info,
ink_assert(caches[type] == this);
intptr_t err = 0;
- int if_writers = reinterpret_cast<uintptr_t>(info) ==
CACHE_ALLOW_MULTIPLE_WRITES;
+ int if_writers = reinterpret_cast<uintptr_t>(old_info) ==
CACHE_ALLOW_MULTIPLE_WRITES;
CacheVC *c = new_CacheVC(cont);
c->vio.op = VIO::WRITE;
c->first_key = *key;
@@ -629,10 +647,10 @@ Cache::open_write(Continuation *cont, const CacheKey
*key, CacheHTTPInfo *info,
} while (DIR_MASK_TAG(c->key.slice32(2)) ==
DIR_MASK_TAG(c->first_key.slice32(2)));
c->earliest_key = c->key;
c->frag_type = CACHE_FRAG_TYPE_HTTP;
- c->stripe = key_to_stripe(key, hostname);
+ c->stripe = key_to_stripe(key, hostname, volume_host_rec);
StripeSM *stripe = c->stripe;
- c->info = info;
- if (c->info && reinterpret_cast<uintptr_t>(info) !=
CACHE_ALLOW_MULTIPLE_WRITES) {
+ c->info = old_info;
+ if (c->info && reinterpret_cast<uintptr_t>(old_info) !=
CACHE_ALLOW_MULTIPLE_WRITES) {
/*
Update has the following code paths :
a) Update alternate header only :
@@ -664,9 +682,9 @@ Cache::open_write(Continuation *cont, const CacheKey *key,
CacheHTTPInfo *info,
c->f.update = 1;
c->op_type = static_cast<int>(CacheOpType::Update);
DDbg(dbg_ctl_cache_update, "Update called");
- info->object_key_get(&c->update_key);
+ old_info->object_key_get(&c->update_key);
ink_assert(!(c->update_key.is_zero()));
- c->update_len = info->object_size_get();
+ c->update_len = old_info->object_size_get();
} else {
c->op_type = static_cast<int>(CacheOpType::Write);
}
@@ -674,7 +692,7 @@ Cache::open_write(Continuation *cont, const CacheKey *key,
CacheHTTPInfo *info,
ts::Metrics::Gauge::increment(cache_rsb.status[c->op_type].active);
ts::Metrics::Gauge::increment(stripe->cache_vol->vol_rsb.status[c->op_type].active);
// coverity[Y2K38_SAFETY:FALSE]
- c->pin_in_cache = static_cast<uint32_t>(apin_in_cache);
+ c->pin_in_cache = static_cast<uint32_t>(pin_in_cache);
{
CACHE_TRY_LOCK(lock, c->stripe->mutex, cont->mutex->thread_holding);
@@ -745,41 +763,61 @@ CacheVConnection::CacheVConnection() :
VConnection(nullptr) {}
// if generic_host_rec.stripes == nullptr, what do we do???
StripeSM *
-Cache::key_to_stripe(const CacheKey *key, std::string_view hostname) const
+Cache::key_to_stripe(const CacheKey *key, std::string_view hostname, const
CacheHostRecord *volume_host_rec) const
{
ReplaceablePtr<CacheHostTable>::ScopedReader hosttable(&this->hosttable);
- uint32_t h = (key->slice32(2) >> DIR_TAG_WIDTH) %
STRIPE_HASH_TABLE_SIZE;
- unsigned short *hash_table = hosttable->gen_host_rec.vol_hash_table;
- const CacheHostRecord *host_rec = &hosttable->gen_host_rec;
+ uint32_t h = (key->slice32(2) >> DIR_TAG_WIDTH)
% STRIPE_HASH_TABLE_SIZE;
+ const CacheHostRecord *host_rec = hosttable->getGenHostRec();
+ unsigned short *hash_table = host_rec->vol_hash_table;
+ StripeSM *selected_stripe = nullptr;
+ bool remap_selection = false;
- if (hosttable->m_numEntries > 0 && !hostname.empty()) {
+ // Priority 1: @volume directive (highest priority)
+ if (volume_host_rec && volume_host_rec->vol_hash_table) {
+ selected_stripe =
volume_host_rec->stripes[volume_host_rec->vol_hash_table[h]];
+ remap_selection = true;
+ Dbg(dbg_ctl_cache_hosting, "@volume directive: using volume hash table for
stripe selection");
+ }
+ // Priority 2: Normal hostname-based volume selection (from hosting.config)
+ if (!selected_stripe && hosttable->getNumEntries() > 0 && !hostname.empty())
{
CacheHostResult res;
+
hosttable->Match(hostname, &res);
if (res.record) {
unsigned short *host_hash_table = res.record->vol_hash_table;
+
if (host_hash_table) {
- if (dbg_ctl_cache_hosting.on()) {
- char format_str[50];
- snprintf(format_str, sizeof(format_str), "Volume: %%xd for host:
%%.%ds", static_cast<int>(hostname.length()));
- Dbg(dbg_ctl_cache_hosting, format_str, res.record, hostname.data());
- }
- return res.record->stripes[host_hash_table[h]];
+ Dbg(dbg_ctl_cache_hosting, "Volume: %p for host: %.*s", res.record,
static_cast<int>(hostname.length()), hostname.data());
+ selected_stripe = res.record->stripes[host_hash_table[h]];
}
}
}
- if (hash_table) {
- if (dbg_ctl_cache_hosting.on()) {
- char format_str[50];
- snprintf(format_str, sizeof(format_str), "Generic volume: %%xd for host:
%%.%ds", static_cast<int>(hostname.length()));
- Dbg(dbg_ctl_cache_hosting, format_str, host_rec, hostname.data());
+
+ // Priority 3: Global default volumes from proxy.config.cache.default_volumes
+ if (!selected_stripe && default_volumes_host_rec &&
default_volumes_host_rec->vol_hash_table) {
+ selected_stripe =
default_volumes_host_rec->stripes[default_volumes_host_rec->vol_hash_table[h]];
+ Dbg(dbg_ctl_cache_hosting, "Using default_volumes for stripe selection");
+ }
+
+ // Priority 4: Generic/default volume selection (fallback)
+ if (!selected_stripe) {
+ if (hash_table) {
+ selected_stripe = host_rec->stripes[hash_table[h]];
+ } else {
+ selected_stripe = host_rec->stripes[0];
}
- return host_rec->stripes[hash_table[h]];
- } else {
- return host_rec->stripes[0];
}
-}
+ if (dbg_ctl_cache_hosting.on() && selected_stripe &&
selected_stripe->cache_vol) {
+ Dbg(dbg_ctl_cache_hosting, "Cache volume selected: %d (%s) for
key=%08x%08x hostname='%.*s' %s",
+ selected_stripe->cache_vol->vol_number,
+ selected_stripe->cache_vol->ramcache_enabled ? "ramcache_enabled" :
"ramcache_disabled", key->slice32(0), key->slice32(1),
+ static_cast<int>(hostname.length()), hostname.data(), remap_selection
? "(remap)" : "(calculated)");
+ }
+
+ return selected_stripe;
+}
int
FragmentSizeUpdateCb(const char * /* name ATS_UNUSED */, RecDataT /* data_type
ATS_UNUSED */, RecData data,
void * /* cookie ATS_UNUSED */)
diff --git a/src/iocore/cache/CacheHosting.cc b/src/iocore/cache/CacheHosting.cc
index c8f7d5f679..9f24fb1a91 100644
--- a/src/iocore/cache/CacheHosting.cc
+++ b/src/iocore/cache/CacheHosting.cc
@@ -38,6 +38,7 @@ namespace
DbgCtl dbg_ctl_cache_hosting{"cache_hosting"};
DbgCtl dbg_ctl_matcher{"matcher"};
+constexpr static int MAX_VOLUME_IDX = 255;
} // end anonymous namespace
/*************************************************************
@@ -652,6 +653,8 @@ ConfigVolumes::BuildListFromString(char *config_file_path,
char *file_buf)
bool ramcache_enabled = true;
int avg_obj_size = -1; // Defaults
int fragment_size = -1;
+ int64_t ram_cache_size = -1; // -1 means use shared allocation
+ int64_t ram_cache_cutoff = -1; // -1 means use global cutoff
while (true) {
// skip all blank spaces at beginning of line
@@ -696,7 +699,7 @@ ConfigVolumes::BuildListFromString(char *config_file_path,
char *file_buf)
break;
}
- if (volume_number < 1 || volume_number > 255) {
+ if (volume_number < 1 || volume_number > MAX_VOLUME_IDX) {
err = "Bad Volume Number";
break;
}
@@ -740,17 +743,33 @@ ConfigVolumes::BuildListFromString(char
*config_file_path, char *file_buf)
in_percent = 0;
}
} else if (strcasecmp(tmp, "avg_obj_size") == 0) { // match avg_obj_size
- tmp += 13;
- avg_obj_size = atoi(tmp);
+ tmp += 13;
+ if (!ParseRules::is_digit(*tmp)) {
+ err = "Invalid avg_obj_size value (must start with a number, e.g.,
64K)";
+ break;
+ }
+ avg_obj_size = static_cast<int>(ink_atoi64(tmp));
- while (ParseRules::is_digit(*tmp)) {
+ if (avg_obj_size < 0) {
+ err = "Invalid avg_obj_size value (must be >= 0)";
+ break;
+ }
+ while (*tmp && (ParseRules::is_digit(*tmp) || strchr("KMGT", *tmp))) {
tmp++;
}
} else if (strcasecmp(tmp, "fragment_size") == 0) { // match
fragment_size
- tmp += 14;
- fragment_size = atoi(tmp);
+ tmp += 14;
+ if (!ParseRules::is_digit(*tmp)) {
+ err = "Invalid fragment_size value (must start with a number, e.g.,
1M)";
+ break;
+ }
+ fragment_size = static_cast<int>(ink_atoi64(tmp));
- while (ParseRules::is_digit(*tmp)) {
+ if (fragment_size < 0) {
+ err = "Invalid fragment_size value (must be >= 0)";
+ break;
+ }
+ while (*tmp && (ParseRules::is_digit(*tmp) || strchr("KMGT", *tmp))) {
tmp++;
}
} else if (strcasecmp(tmp, "ramcache") == 0) { // match ramcache
@@ -765,11 +784,42 @@ ConfigVolumes::BuildListFromString(char
*config_file_path, char *file_buf)
err = "Unexpected end of line";
break;
}
+ } else if (strcasecmp(tmp, "ram_cache_size") == 0) { // match
ram_cache_size
+ tmp += 15;
+ if (!ParseRules::is_digit(*tmp)) {
+ err = "Invalid ram_cache_size value (must start with a number, e.g.,
10G)";
+ break;
+ }
+ ram_cache_size = ink_atoi64(tmp);
+
+ if (ram_cache_size < 0) {
+ err = "Invalid ram_cache_size value (must be >= 0)";
+ break;
+ }
+ // Note: ram_cache_size=0 disables RAM cache for this volume, same as
ramcache=false
+ while (*tmp && (ParseRules::is_digit(*tmp) || strchr("KMGT", *tmp))) {
+ tmp++;
+ }
+ } else if (strcasecmp(tmp, "ram_cache_cutoff") == 0) { // match
ram_cache_cutoff
+ tmp += 17;
+ if (!ParseRules::is_digit(*tmp)) {
+ err = "Invalid ram_cache_cutoff value (must start with a number,
e.g., 5M)";
+ break;
+ }
+ ram_cache_cutoff = ink_atoi64(tmp);
+
+ if (ram_cache_cutoff < 0) {
+ err = "Invalid ram_cache_cutoff value (must be >= 0)";
+ break;
+ }
+ while (*tmp && (ParseRules::is_digit(*tmp) || strchr("KMGT", *tmp))) {
+ tmp++;
+ }
}
// ends here
if (end < line_end) {
- tmp++;
+ tmp = line_end;
}
}
@@ -791,6 +841,8 @@ ConfigVolumes::BuildListFromString(char *config_file_path,
char *file_buf)
configp->size = size;
configp->avg_obj_size = avg_obj_size;
configp->fragment_size = fragment_size;
+ configp->ram_cache_size = ram_cache_size;
+ configp->ram_cache_cutoff = ram_cache_cutoff;
configp->cachep = nullptr;
configp->ramcache_enabled = ramcache_enabled;
cp_queue.enqueue(configp);
@@ -800,8 +852,10 @@ ConfigVolumes::BuildListFromString(char *config_file_path,
char *file_buf)
} else {
ink_release_assert(!"Unexpected non-HTTP cache volume");
}
- Dbg(dbg_ctl_cache_hosting, "added volume=%d, scheme=%d, size=%d
percent=%d, ramcache enabled=%d", volume_number,
- static_cast<int>(scheme), size, in_percent, ramcache_enabled);
+ Dbg(dbg_ctl_cache_hosting,
+ "added volume=%d, scheme=%d, size=%d percent=%d, ramcache
enabled=%d, "
+ "ram_cache_size=%" PRId64 ", ram_cache_cutoff=%" PRId64,
+ volume_number, static_cast<int>(scheme), size, in_percent,
ramcache_enabled, ram_cache_size, ram_cache_cutoff);
}
tmp = bufTok.iterNext(&i_state);
@@ -809,3 +863,49 @@ ConfigVolumes::BuildListFromString(char *config_file_path,
char *file_buf)
return;
}
+
+// Wrapper function for deleting CacheHostRecord from outside the cache module.
+void
+destroyCacheHostRecord(CacheHostRecord *rec)
+{
+ delete rec;
+}
+
+// Build a CacheHostRecord for @volume= directive with comma-separated volumes
+// This reuses CacheHostRecord::Init() by constructing a minimal matcher_line
+CacheHostRecord *
+createCacheHostRecord(const char *volume_str, char *errbuf, size_t errbufsize)
+{
+ if (!volume_str || !*volume_str) {
+ snprintf(errbuf, errbufsize, "Empty volume specification");
+ return nullptr;
+ }
+
+ CacheHostRecord *host_rec = new CacheHostRecord();
+
+ // Build a minimal matcher_line structure with just the volume= directive
+ char volume_key[] = "volume";
+ matcher_line ml;
+ memset(&ml, 0, sizeof(ml));
+
+ ml.line[0][0] = volume_key;
+ ml.line[1][0] = const_cast<char *>(volume_str);
+ ml.num_el = 1;
+ ml.dest_entry = -1;
+ ml.line_num = 0;
+ ml.type = MATCH_NONE;
+ ml.next = nullptr;
+
+ int result = host_rec->Init(&ml, CacheType::HTTP);
+
+ if (result != 0) {
+ delete host_rec;
+ snprintf(errbuf, errbufsize, "Failed to initialize volume record (check
volume.config)");
+ return nullptr;
+ }
+
+ Dbg(dbg_ctl_cache_hosting, "Created remap volume record with %d volumes, %d
stripes", host_rec->num_cachevols,
+ host_rec->num_vols);
+
+ return host_rec;
+}
diff --git a/src/iocore/cache/CacheProcessor.cc
b/src/iocore/cache/CacheProcessor.cc
index 8d1faf6871..14d7732097 100644
--- a/src/iocore/cache/CacheProcessor.cc
+++ b/src/iocore/cache/CacheProcessor.cc
@@ -411,16 +411,16 @@ CacheProcessor::lookup(Continuation *cont, const
HttpCacheKey *key, CacheFragTyp
Action *
CacheProcessor::open_read(Continuation *cont, const HttpCacheKey *key,
CacheHTTPHdr *request, const HttpConfigAccessor *params,
- CacheFragType type)
+ CacheFragType frag_type, const CacheHostRecord
*volume_host_rec)
{
- return caches[type]->open_read(cont, &key->hash, request, params, type,
key->hostname);
+ return caches[frag_type]->open_read(cont, &key->hash, request, params,
frag_type, key->hostname, volume_host_rec);
}
Action *
CacheProcessor::open_write(Continuation *cont, const HttpCacheKey *key,
CacheHTTPInfo *old_info, time_t pin_in_cache,
- CacheFragType type)
+ CacheFragType frag_type, const CacheHostRecord
*volume_host_rec)
{
- return caches[type]->open_write(cont, &key->hash, old_info, pin_in_cache,
type, key->hostname);
+ return caches[frag_type]->open_write(cont, &key->hash, old_info,
pin_in_cache, frag_type, key->hostname, volume_host_rec);
}
//----------------------------------------------------------------------------
@@ -483,7 +483,7 @@ CacheProcessor::mark_storage_offline(CacheDisk *d, ///<
Target disk
} else { // check cache types specifically
if (theCache) {
ReplaceablePtr<CacheHostTable>::ScopedReader
hosttable(&theCache->hosttable);
- if (!hosttable->gen_host_rec.vol_hash_table) {
+ if (!hosttable->getGenHostRec()->vol_hash_table) {
unsigned int caches_ready = 0;
caches_ready = caches_ready | (1 <<
CACHE_FRAG_TYPE_HTTP);
caches_ready = caches_ready | (1 <<
CACHE_FRAG_TYPE_NONE);
@@ -506,8 +506,8 @@ void
rebuild_host_table(Cache *cache)
{
ReplaceablePtr<CacheHostTable>::ScopedWriter hosttable(&cache->hosttable);
- build_vol_hash_table(&hosttable->gen_host_rec);
- if (hosttable->m_numEntries != 0) {
+ build_vol_hash_table(const_cast<CacheHostRecord
*>(hosttable->getGenHostRec()));
+ if (hosttable->getNumEntries() != 0) {
CacheHostMatcher *hm = hosttable->getHostMatcher();
CacheHostRecord *h_rec = hm->getDataArray();
int h_rec_len = hm->getNumElements();
@@ -1168,6 +1168,8 @@ cplist_update()
cp->ramcache_enabled = config_vol->ramcache_enabled;
cp->avg_obj_size = config_vol->avg_obj_size;
cp->fragment_size = config_vol->fragment_size;
+ cp->ram_cache_size = config_vol->ram_cache_size;
+ cp->ram_cache_cutoff = config_vol->ram_cache_cutoff;
config_vol->cachep = cp;
} else {
/* delete this volume from all the disks */
@@ -1440,9 +1442,32 @@ CacheProcessor::cacheInitialized()
}
}
+ // Calculate total private RAM allocations from per-volume configurations
int64_t http_ram_cache_size = 0;
+ int64_t total_private_ram = 0;
+
+ if (cache_config_ram_cache_size != AUTO_SIZE_RAM_CACHE) {
+ CacheVol *cp = cp_list.head;
+
+ for (; cp; cp = cp->link.next) {
+ if (cp->ram_cache_size > 0 && !cp->ramcache_enabled) {
+ Warning("Volume %d has ram_cache_size=%" PRId64 " but
ramcache=false, ignoring ram_cache_size", cp->vol_number,
+ cp->ram_cache_size);
+ cp->ram_cache_size = -1;
+ }
+ if (cp->ram_cache_size > 0) {
+ total_private_ram += cp->ram_cache_size;
+ Dbg(dbg_ctl_cache_init, "Volume %d has private RAM allocation: %"
PRId64 " bytes (%" PRId64 " MB)", cp->vol_number,
+ cp->ram_cache_size, cp->ram_cache_size / (1024 * 1024));
+ }
+ }
+
+ if (total_private_ram > 0) {
+ Dbg(dbg_ctl_cache_init, "Total private RAM allocations: %" PRId64 "
bytes (%" PRId64 " MB)", total_private_ram,
+ total_private_ram / (1024 * 1024));
+ }
+ }
- // let us calculate the Size
if (cache_config_ram_cache_size == AUTO_SIZE_RAM_CACHE) {
Dbg(dbg_ctl_cache_init, "cache_config_ram_cache_size ==
AUTO_SIZE_RAM_CACHE");
} else {
@@ -1450,12 +1475,24 @@ CacheProcessor::cacheInitialized()
// TODO, should we check the available system memories, or you will
// OOM or swapout, that is not a good situation for the server
Dbg(dbg_ctl_cache_init, "%" PRId64 " != AUTO_SIZE_RAM_CACHE",
cache_config_ram_cache_size);
- http_ram_cache_size =
- static_cast<int64_t>((static_cast<double>(theCache->cache_size) /
total_size) * cache_config_ram_cache_size);
+
+ // Calculate shared pool: global RAM cache size minus private
allocations
+ int64_t shared_pool = cache_config_ram_cache_size - total_private_ram;
+
+ if (shared_pool < 0) {
+ Fatal("Total private RAM cache allocations (%" PRId64 " bytes)
exceed global ram_cache.size (%" PRId64 " bytes). "
+ "Increase proxy.config.cache.ram_cache.size or reduce
per-volume ram_cache_size allocations.",
+ total_private_ram, cache_config_ram_cache_size);
+ } else if (total_private_ram > 0) {
+ Dbg(dbg_ctl_cache_init, "Shared RAM cache pool (after private
allocations): %" PRId64 " bytes (%" PRId64 " MB)",
+ shared_pool, shared_pool / (1024 * 1024));
+ }
+
+ http_ram_cache_size =
static_cast<int64_t>((static_cast<double>(theCache->cache_size) / total_size) *
shared_pool);
Dbg(dbg_ctl_cache_init, "http_ram_cache_size = %" PRId64 " = %" PRId64
"Mb", http_ram_cache_size,
http_ram_cache_size / (1024 * 1024));
- int64_t stream_ram_cache_size = cache_config_ram_cache_size -
http_ram_cache_size;
+ int64_t stream_ram_cache_size = shared_pool - http_ram_cache_size;
Dbg(dbg_ctl_cache_init, "stream_ram_cache_size = %" PRId64 " = %"
PRId64 "Mb", stream_ram_cache_size,
stream_ram_cache_size / (1024 * 1024));
@@ -1468,23 +1505,55 @@ CacheProcessor::cacheInitialized()
uint64_t total_cache_bytes = 0; // bytes that can used in total_size
uint64_t total_direntries = 0; // all the direntries in the cache
uint64_t used_direntries = 0; // and used
- uint64_t total_ram_cache_bytes = 0;
+ uint64_t total_ram_cache_bytes = 0; // Total RAM cache size across all
volumes
+ uint64_t shared_cache_size = 0; // Total cache size of volumes
without explicit RAM allocations
+
+ // Calculate total cache size of volumes without explicit RAM allocations
+ if (http_ram_cache_size > 0) {
+ for (int i = 0; i < gnstripes; i++) {
+ if (gstripes[i]->cache_vol->ram_cache_size <= 0) {
+ shared_cache_size += (gstripes[i]->len >> STORE_BLOCK_SHIFT);
+ }
+ }
+ Dbg(dbg_ctl_cache_init, "Shared cache size (for RAM pool
distribution): %" PRId64 " blocks", shared_cache_size);
+ }
for (int i = 0; i < gnstripes; i++) {
StripeSM *stripe = gstripes[i];
int64_t ram_cache_bytes = 0;
- if (stripe->cache_vol->ramcache_enabled) {
- if (http_ram_cache_size == 0) {
+ // If RAM cache enabled, check if this volume has a private RAM cache
allocation
+ if (stripe->cache_vol->ramcache_enabled &&
stripe->cache_vol->ram_cache_size != 0) {
+ if (stripe->cache_vol->ram_cache_size > 0) {
+ int64_t volume_stripe_count = 0;
+
+ for (int j = 0; j < gnstripes; j++) {
+ if (gstripes[j]->cache_vol == stripe->cache_vol) {
+ volume_stripe_count++;
+ }
+ }
+
+ if (volume_stripe_count > 0) {
+ ram_cache_bytes = stripe->cache_vol->ram_cache_size /
volume_stripe_count;
+ Dbg(dbg_ctl_cache_init, "Volume %d stripe %d using private RAM
allocation: %" PRId64 " bytes (%" PRId64 " MB)",
+ stripe->cache_vol->vol_number, i, ram_cache_bytes,
ram_cache_bytes / (1024 * 1024));
+ }
+ } else if (http_ram_cache_size == 0) {
// AUTO_SIZE_RAM_CACHE
ram_cache_bytes = stripe->dirlen() * DEFAULT_RAM_CACHE_MULTIPLIER;
} else {
- ink_assert(stripe->cache != nullptr);
-
- double factor =
static_cast<double>(static_cast<int64_t>(stripe->len >> STORE_BLOCK_SHIFT)) /
theCache->cache_size;
- Dbg(dbg_ctl_cache_init, "factor = %f", factor);
-
- ram_cache_bytes = static_cast<int64_t>(http_ram_cache_size *
factor);
+ // Use shared pool allocation - distribute only among volumes
without explicit allocations
+ if (shared_cache_size > 0) {
+ ink_assert(stripe->cache != nullptr);
+ double factor =
static_cast<double>(static_cast<int64_t>(stripe->len >> STORE_BLOCK_SHIFT)) /
shared_cache_size;
+
+ Dbg(dbg_ctl_cache_init, "factor = %f (divisor = %" PRId64 ")",
factor, shared_cache_size);
+ ram_cache_bytes = static_cast<int64_t>(http_ram_cache_size *
factor);
+ } else {
+ ram_cache_bytes = 0;
+ Dbg(dbg_ctl_cache_init, "Volume %d stripe has no explicit RAM
allocation, but shared pool is empty",
+ stripe->cache_vol->vol_number);
+ }
}
stripe->ram_cache->init(ram_cache_bytes, stripe);
@@ -1495,19 +1564,19 @@ CacheProcessor::cacheInitialized()
ram_cache_bytes, ram_cache_bytes / (1024 * 1024));
}
- uint64_t vol_total_cache_bytes = stripe->len - stripe->dirlen();
- total_cache_bytes += vol_total_cache_bytes;
+ uint64_t vol_total_cache_bytes = stripe->len - stripe->dirlen();
+ uint64_t vol_total_direntries = stripe->directory.entries();
+ uint64_t vol_used_direntries = stripe->directory.entries_used();
+
+ total_cache_bytes += vol_total_cache_bytes;
ts::Metrics::Gauge::increment(stripe->cache_vol->vol_rsb.bytes_total,
vol_total_cache_bytes);
ts::Metrics::Gauge::increment(stripe->cache_vol->vol_rsb.stripes);
Dbg(dbg_ctl_cache_init, "total_cache_bytes = %" PRId64 " = %" PRId64
"Mb", total_cache_bytes,
total_cache_bytes / (1024 * 1024));
- uint64_t vol_total_direntries = stripe->directory.entries();
- total_direntries += vol_total_direntries;
+ total_direntries += vol_total_direntries;
ts::Metrics::Gauge::increment(stripe->cache_vol->vol_rsb.direntries_total,
vol_total_direntries);
-
- uint64_t vol_used_direntries = stripe->directory.entries_used();
ts::Metrics::Gauge::increment(stripe->cache_vol->vol_rsb.direntries_used,
vol_used_direntries);
used_direntries += vol_used_direntries;
}
diff --git a/src/iocore/cache/CacheVC.cc b/src/iocore/cache/CacheVC.cc
index d230463e1b..e21c7a7028 100644
--- a/src/iocore/cache/CacheVC.cc
+++ b/src/iocore/cache/CacheVC.cc
@@ -412,15 +412,17 @@ CacheVC::handleReadDone(int event, Event * /* e
ATS_UNUSED */)
// Put the request in the ram cache only if its a open_read or lookup
if (vio.op == VIO::READ && okay) {
bool cutoff_check;
+ // Determine effective cutoff: use per-volume override if set,
otherwise use global
+ int64_t effective_cutoff =
+ (stripe->cache_vol->ram_cache_cutoff > 0) ?
stripe->cache_vol->ram_cache_cutoff : cache_config_ram_cache_cutoff;
// cutoff_check :
// doc_len == 0 for the first fragment (it is set from the vector)
// The decision on the first fragment is based on
// doc->total_len
// After that, the decision is based of doc_len (doc_len != 0)
- // (cache_config_ram_cache_cutoff == 0) : no cutoffs
- cutoff_check =
- ((!doc_len && static_cast<int64_t>(doc->total_len) <
cache_config_ram_cache_cutoff) ||
- (doc_len && static_cast<int64_t>(doc_len) <
cache_config_ram_cache_cutoff) || !cache_config_ram_cache_cutoff);
+ // (effective_cutoff == 0) : no cutoffs
+ cutoff_check = ((!doc_len && static_cast<int64_t>(doc->total_len) <
effective_cutoff) ||
+ (doc_len && static_cast<int64_t>(doc_len) <
effective_cutoff) || !effective_cutoff);
if (cutoff_check && !f.doc_from_ram_cache) {
uint64_t o = dir_offset(&dir);
stripe->ram_cache->put(read_key, buf.get(), doc->len, http_copy_hdr,
o);
@@ -635,7 +637,7 @@ CacheVC::scanStripe(int /* event ATS_UNUSED */, Event * /*
e ATS_UNUSED */)
ReplaceablePtr<CacheHostTable>::ScopedReader hosttable(&theCache->hosttable);
- const CacheHostRecord *rec = &hosttable->gen_host_rec;
+ const CacheHostRecord *rec = hosttable->getGenHostRec();
if (!hostname.empty()) {
CacheHostResult res;
hosttable->Match(hostname, &res);
diff --git a/src/iocore/cache/P_CacheHosting.h
b/src/iocore/cache/P_CacheHosting.h
index d6dc8fd6c2..d36821c51e 100644
--- a/src/iocore/cache/P_CacheHosting.h
+++ b/src/iocore/cache/P_CacheHosting.h
@@ -62,7 +62,9 @@ struct CacheHostRecord {
CacheHostRecord() {}
};
-void build_vol_hash_table(CacheHostRecord *cp);
+void build_vol_hash_table(CacheHostRecord *cp);
+CacheHostRecord *createCacheHostRecord(const char *volume_str, char *errbuf,
size_t errbufsize);
+void destroyCacheHostRecord(CacheHostRecord *rec);
struct CacheHostResult {
CacheHostRecord *record = nullptr;
@@ -229,18 +231,42 @@ public:
void Match(std::string_view rdata, CacheHostResult *result) const;
void Print() const;
+ // Getters for Cache::key_to_stripe access
+ const CacheHostRecord *
+ getGenHostRec() const
+ {
+ return &gen_host_rec;
+ }
+
int
- getEntryCount() const
+ getNumEntries() const
{
return m_numEntries;
}
+
+ int
+ getGenHostRecCacheVols() const
+ {
+ return gen_host_rec.num_cachevols;
+ }
+
CacheHostMatcher *
getHostMatcher() const
{
return hostMatch.get();
}
- static int config_callback(const char *, RecDataT, RecData, void *);
+ CacheType
+ getType() const
+ {
+ return type;
+ }
+
+ Cache *
+ getCache() const
+ {
+ return cache;
+ }
void
register_config_callback(ReplaceablePtr<CacheHostTable> *p)
@@ -248,12 +274,13 @@ public:
RecRegisterConfigUpdateCb("proxy.config.cache.hosting_filename",
CacheHostTable::config_callback, (void *)p);
}
- CacheType type = CacheType::HTTP;
- Cache *cache = nullptr;
- int m_numEntries = 0;
- CacheHostRecord gen_host_rec;
-
private:
+ static int config_callback(const char *, RecDataT, RecData, void *);
+
+ CacheType type = CacheType::HTTP;
+ Cache *cache = nullptr;
+ int m_numEntries = 0;
+ CacheHostRecord gen_host_rec;
std::unique_ptr<CacheHostMatcher> hostMatch = nullptr;
const matcher_tags config_tags = {"hostname", "domain",
nullptr, nullptr, nullptr, nullptr, false};
const char *matcher_name = "unknown"; // Used for
Debug/Warning/Error messages
@@ -276,8 +303,8 @@ struct CacheHostTableConfig : public Continuation {
Cache *cache = nullptr;
{
ReplaceablePtr<CacheHostTable>::ScopedReader hosttable(ppt);
- type = hosttable->type;
- cache = hosttable->cache;
+ type = hosttable->getType();
+ cache = hosttable->getCache();
}
ppt->reset(new CacheHostTable(cache, type));
delete this;
@@ -298,6 +325,8 @@ struct ConfigVol {
int percent;
int avg_obj_size;
int fragment_size;
+ int64_t ram_cache_size; // Per-volume RAM cache size (-1 = use shared
allocation)
+ int64_t ram_cache_cutoff; // Per-volume RAM cache cutoff (-1 = use global
cutoff)
CacheVol *cachep;
LINK(ConfigVol, link);
};
diff --git a/src/iocore/cache/P_CacheInternal.h
b/src/iocore/cache/P_CacheInternal.h
index 08560ba85e..08e77291bb 100644
--- a/src/iocore/cache/P_CacheInternal.h
+++ b/src/iocore/cache/P_CacheInternal.h
@@ -108,6 +108,9 @@ struct EvacuationBlock;
extern CacheStatsBlock cache_rsb;
+// Global default volumes host record (initialized from
proxy.config.cache.default_volumes)
+extern CacheHostRecord *default_volumes_host_rec;
+
// Configuration
extern int cache_config_dir_sync_frequency;
extern int cache_config_dir_sync_delay;
@@ -485,9 +488,11 @@ struct Cache {
Action *scan(Continuation *cont, std::string_view hostname =
std::string_view{}, int KB_per_second = 2500) const;
Action *open_read(Continuation *cont, const CacheKey *key, CacheHTTPHdr
*request, const HttpConfigAccessor *params,
- CacheFragType type, std::string_view hostname =
std::string_view{}) const;
+ CacheFragType type, std::string_view hostname =
std::string_view{},
+ const CacheHostRecord *volume_host_rec = nullptr)
const;
Action *open_write(Continuation *cont, const CacheKey *key,
CacheHTTPInfo *old_info, time_t pin_in_cache = 0,
- CacheFragType type = CACHE_FRAG_TYPE_HTTP,
std::string_view hostname = std::string_view{}) const;
+ CacheFragType type = CACHE_FRAG_TYPE_HTTP,
std::string_view hostname = std::string_view{},
+ const CacheHostRecord *volume_host_rec = nullptr)
const;
static void generate_key(CryptoHash *hash, CacheURL *url);
static void generate_key(HttpCacheKey *hash, CacheURL *url, bool
ignore_query = false, cache_generation_t generation = -1);
@@ -500,7 +505,7 @@ struct Cache {
int open_done();
- StripeSM *key_to_stripe(const CacheKey *key, std::string_view hostname)
const;
+ StripeSM *key_to_stripe(const CacheKey *key, std::string_view hostname,
const CacheHostRecord *volume_host_rec = nullptr) const;
Cache() {}
};
diff --git a/src/iocore/cache/Stripe.h b/src/iocore/cache/Stripe.h
index 46ab8201a6..b99b4773fe 100644
--- a/src/iocore/cache/Stripe.h
+++ b/src/iocore/cache/Stripe.h
@@ -58,6 +58,8 @@ struct CacheVol {
int avg_obj_size = -1; // Defer to the records.config if not
overriden
int fragment_size = -1; // Defer to the records.config if not
overriden
bool ramcache_enabled = true;
+ int64_t ram_cache_size = -1; // Per-volume RAM cache size (-1 = use
shared allocation)
+ int64_t ram_cache_cutoff = -1; // Per-volume RAM cache cutoff (-1 = use
global cutoff)
StripeSM **stripes = nullptr;
DiskStripe **disk_stripes = nullptr;
LINK(CacheVol, link);
diff --git a/src/proxy/ReverseProxy.cc b/src/proxy/ReverseProxy.cc
index 341c2c7de4..8deb9d0016 100644
--- a/src/proxy/ReverseProxy.cc
+++ b/src/proxy/ReverseProxy.cc
@@ -39,6 +39,7 @@
#include "proxy/http/remap/RemapProcessor.h"
#include "proxy/http/remap/UrlRewrite.h"
#include "proxy/http/remap/UrlMapping.h"
+#include "proxy/http/remap/UrlMappingPathIndex.h"
namespace
{
@@ -49,7 +50,7 @@ DbgCtl dbg_ctl_url_rewrite{"url_rewrite"};
} // end anonymous namespace
// Global Ptrs
-UrlRewrite *rewrite_table = nullptr;
+std::atomic<UrlRewrite *> rewrite_table = nullptr;
thread_local PluginThreadContext *pluginThreadContext = nullptr;
// Tokens for the Callback function
@@ -66,12 +67,13 @@ thread_local PluginThreadContext *pluginThreadContext =
nullptr;
int
init_reverse_proxy()
{
- ink_assert(rewrite_table == nullptr);
+ ink_assert(rewrite_table.load() == nullptr);
reconfig_mutex = new_ProxyMutex();
- rewrite_table = new UrlRewrite();
+ rewrite_table.store(new UrlRewrite());
+ rewrite_table.load()->acquire();
Note("%s loading ...", ts::filename::REMAP);
- if (!rewrite_table->load()) {
+ if (!rewrite_table.load()->load()) {
Emergency("%s failed to load", ts::filename::REMAP);
} else {
Note("%s finished loading", ts::filename::REMAP);
@@ -82,9 +84,6 @@ init_reverse_proxy()
RecRegisterConfigUpdateCb("proxy.config.reverse_proxy.enabled",
url_rewrite_CB, (void *)REVERSE_CHANGED);
RecRegisterConfigUpdateCb("proxy.config.http.referer_default_redirect",
url_rewrite_CB, (void *)HTTP_DEFAULT_REDIRECT_CHANGED);
- // Hold at least one lease, until we reload the configuration
- rewrite_table->acquire();
-
return 0;
}
@@ -148,7 +147,7 @@ reloadUrlRewrite()
newTable->acquire();
// Swap configurations
- oldTable = ink_atomic_swap(&rewrite_table, newTable);
+ oldTable = rewrite_table.exchange(newTable);
ink_assert(oldTable != nullptr);
@@ -168,6 +167,65 @@ reloadUrlRewrite()
}
}
+/**
+ * Helper function to initialize volume_host_rec for a single url_mapping.
+ * This is a no-op if the mapping has no volume string or is already
initialized.
+ */
+static void
+init_mapping_volume_host_rec(url_mapping &mapping)
+{
+ char errbuf[256];
+
+ if (!mapping.initVolumeHostRec(errbuf, sizeof(errbuf))) {
+ Error("Failed to initialize volume record for @volume=%s: %s",
mapping.getVolume().c_str(), errbuf);
+ }
+}
+
+static void
+init_store_volume_host_records(UrlRewrite::MappingsStore &store)
+{
+ if (store.hash_lookup) {
+ for (auto &entry : *store.hash_lookup) {
+ UrlMappingPathIndex *path_index = entry.second;
+
+ if (path_index) {
+ path_index->foreach_mapping(init_mapping_volume_host_rec);
+ }
+ }
+ }
+
+ for (UrlRewrite::RegexMapping *reg_map = store.regex_list.head; reg_map;
reg_map = reg_map->link.next) {
+ if (reg_map->url_map) {
+ init_mapping_volume_host_rec(*reg_map->url_map);
+ }
+ }
+}
+
+// This is called after the cache is initialized, since we may need the
volume_host_records.
+// Must only be called during startup before any remap reload can occur.
+void
+init_remap_volume_host_records()
+{
+ UrlRewrite *table = rewrite_table.load(std::memory_order_acquire);
+
+ if (!table) {
+ return;
+ }
+
+ table->acquire();
+
+ Dbg(dbg_ctl_url_rewrite, "Initializing volume_host_rec for all remap rules
after cache init");
+
+ // Initialize for all mapping stores
+ init_store_volume_host_records(table->forward_mappings);
+ init_store_volume_host_records(table->reverse_mappings);
+ init_store_volume_host_records(table->permanent_redirects);
+ init_store_volume_host_records(table->temporary_redirects);
+ init_store_volume_host_records(table->forward_mappings_with_recv_port);
+
+ table->release();
+}
+
int
url_rewrite_CB(const char * /* name ATS_UNUSED */, RecDataT /* data_type
ATS_UNUSED */, RecData data, void *cookie)
{
@@ -175,7 +233,7 @@ url_rewrite_CB(const char * /* name ATS_UNUSED */, RecDataT
/* data_type ATS_UNU
switch (my_token) {
case REVERSE_CHANGED:
- rewrite_table->SetReverseFlag(data.rec_int);
+ rewrite_table.load()->SetReverseFlag(data.rec_int);
break;
case TSNAME_CHANGED:
diff --git a/src/proxy/http/HttpCacheSM.cc b/src/proxy/http/HttpCacheSM.cc
index 7893fda4b1..1739f25491 100644
--- a/src/proxy/http/HttpCacheSM.cc
+++ b/src/proxy/http/HttpCacheSM.cc
@@ -323,12 +323,20 @@ HttpCacheSM::_schedule_read_retry()
Action *
HttpCacheSM::do_cache_open_read(const HttpCacheKey &key)
{
+ Action *action_handle = nullptr;
+
open_read_tries++;
ink_assert(pending_action == nullptr);
// Initialising read-while-write-inprogress flag
this->readwhilewrite_inprogress = false;
- Action *action_handle = cacheProcessor.open_read(this, &key,
this->read_request_hdr, &http_params);
+
+ if (master_sm && master_sm->t_state.cache_info.volume_host_rec) {
+ action_handle = cacheProcessor.open_read(this, &key,
this->read_request_hdr, &http_params, CACHE_FRAG_TYPE_HTTP,
+
master_sm->t_state.cache_info.volume_host_rec);
+ } else {
+ action_handle = cacheProcessor.open_read(this, &key,
this->read_request_hdr, &http_params);
+ }
if (action_handle != ACTION_RESULT_DONE) {
pending_action = action_handle;
@@ -422,9 +430,15 @@ HttpCacheSM::open_write(const HttpCacheKey *key, URL *url,
HTTPHdr *request, Cac
return ACTION_RESULT_DONE;
}
- // INKqa11166
CacheHTTPInfo *info = allow_multiple ?
reinterpret_cast<CacheHTTPInfo *>(CACHE_ALLOW_MULTIPLE_WRITES) : old_info;
- Action *action_handle = cacheProcessor.open_write(this, key, info,
pin_in_cache);
+ Action *action_handle = nullptr;
+
+ if (master_sm && master_sm->t_state.cache_info.volume_host_rec) {
+ action_handle =
+ cacheProcessor.open_write(this, key, info, pin_in_cache,
CACHE_FRAG_TYPE_HTTP, master_sm->t_state.cache_info.volume_host_rec);
+ } else {
+ action_handle = cacheProcessor.open_write(this, key, info, pin_in_cache);
+ }
if (action_handle != ACTION_RESULT_DONE) {
pending_action = action_handle;
diff --git a/src/proxy/http/HttpSM.cc b/src/proxy/http/HttpSM.cc
index 8b22d69fa3..0aa731a54f 100644
--- a/src/proxy/http/HttpSM.cc
+++ b/src/proxy/http/HttpSM.cc
@@ -332,7 +332,7 @@ HttpSM::init(bool from_early_data)
t_state.http_config_param = HttpConfig::acquire();
// Acquire a lease on the global remap / rewrite table (stupid global name
...)
- m_remap = rewrite_table->acquire();
+ m_remap = rewrite_table.load()->acquire();
// Simply point to the global config for the time being, no need to copy this
// entire struct if nothing is going to change it.
diff --git a/src/proxy/http/remap/RemapConfig.cc
b/src/proxy/http/remap/RemapConfig.cc
index 64d29d37c6..95d51af39c 100644
--- a/src/proxy/http/remap/RemapConfig.cc
+++ b/src/proxy/http/remap/RemapConfig.cc
@@ -36,6 +36,9 @@
#include "tscore/Filenames.h"
#include "proxy/IPAllow.h"
#include "proxy/http/remap/PluginFactory.h"
+#include "iocore/cache/Cache.h"
+
+extern CacheHostRecord *createCacheHostRecord(const char *volume_str, char
*errbuf, size_t errbufsize);
using namespace std::literals;
@@ -48,6 +51,7 @@ namespace
DbgCtl dbg_ctl_url_rewrite{"url_rewrite"};
DbgCtl dbg_ctl_remap_plugin{"remap_plugin"};
DbgCtl dbg_ctl_url_rewrite_regex{"url_rewrite_regex"};
+
} // end anonymous namespace
/**
@@ -829,6 +833,14 @@ remap_check_option(const char *const *argv, int argc,
unsigned long findmode, in
*argptr = &argv[i][9];
}
ret_flags |= REMAP_OPTFLG_STRATEGY;
+ } else if (!strncasecmp(argv[i], "volume=", 7)) {
+ if ((findmode & REMAP_OPTFLG_VOLUME) != 0) {
+ idx = i;
+ }
+ if (argptr) {
+ *argptr = &argv[i][7];
+ }
+ ret_flags |= REMAP_OPTFLG_VOLUME;
} else {
Warning("ignoring invalid remap option '%s'", argv[i]);
}
@@ -1197,12 +1209,74 @@ remap_parse_config_bti(const char *path,
BUILD_TABLE_INFO *bti)
if ((bti->remap_optflg & REMAP_OPTFLG_MAP_ID) != 0) {
int idx = 0;
int ret = remap_check_option(bti->argv, bti->argc, REMAP_OPTFLG_MAP_ID,
&idx);
+
if (ret & REMAP_OPTFLG_MAP_ID) {
- char *c = strchr(bti->argv[idx], static_cast<int>('='));
+ char *c = strchr(bti->argv[idx], static_cast<int>('='));
+
new_mapping->map_id = static_cast<unsigned int>(atoi(++c));
}
}
+ // Parse @volume= option with comma-separated syntax (@volume=3,4)
+ for (int i = 0; i < bti->argc; i++) {
+ if (!strncasecmp(bti->argv[i], "volume=", 7)) {
+ const char *volume_str = &bti->argv[i][7];
+
+ if (!volume_str || !*volume_str) {
+ snprintf(errStrBuf, sizeof(errStrBuf), "Empty @volume= directive at
line %d", cln + 1);
+ errStr = errStrBuf;
+ goto MAP_ERROR;
+ }
+
+ {
+ swoc::TextView vol_list{volume_str};
+
+ if (vol_list.back() == ',') {
+ snprintf(errStrBuf, sizeof(errStrBuf), "Invalid @volume=%s at line
%d (trailing comma)", volume_str, cln + 1);
+ errStr = errStrBuf;
+ goto MAP_ERROR;
+ }
+ while (!vol_list.empty()) {
+ swoc::TextView span;
+ swoc::TextView token{vol_list.take_prefix_at(',')};
+ auto n = swoc::svtoi(token, &span);
+
+ if (span.size() != token.size() || token.empty()) {
+ snprintf(errStrBuf, sizeof(errStrBuf), "Invalid @volume=%s at
line %d (expected comma-separated numbers 1-255)",
+ volume_str, cln + 1);
+ errStr = errStrBuf;
+ goto MAP_ERROR;
+ } else if (n < 1 || n > 255) {
+ snprintf(errStrBuf, sizeof(errStrBuf), "Volume number %jd out of
range (1-255) in @volume=%s at line %d", n,
+ volume_str, cln + 1);
+ errStr = errStrBuf;
+ goto MAP_ERROR;
+ }
+ }
+ }
+
+ // Check if cache is ready (will be true during config reload,
possibly false during initial startup)
+ if (CacheProcessor::IsCacheEnabled() == CacheInitState::INITIALIZED) {
+ char volume_errbuf[256];
+ CacheHostRecord *rec = createCacheHostRecord(volume_str,
volume_errbuf, sizeof(volume_errbuf));
+
+ if (!rec) {
+ snprintf(errStrBuf, sizeof(errStrBuf), "Failed to build volume
record for @volume=%s at line %d: %s", volume_str,
+ cln + 1, volume_errbuf);
+ errStr = errStrBuf;
+ goto MAP_ERROR;
+ }
+ new_mapping->volume_host_rec.store(rec, std::memory_order_release);
+ Dbg(dbg_ctl_url_rewrite, "[BuildTable] Cache volume directive built:
@volume=%s", volume_str);
+ } else {
+ // Store the volume string for lazy initialization after cache is
ready
+ new_mapping->setVolume(volume_str);
+ Dbg(dbg_ctl_url_rewrite, "[BuildTable] Cache volume directive stored
(deferred): @volume=%s", volume_str);
+ }
+ break;
+ }
+ }
+
map_from = bti->paramv[1];
length = UrlWhack(map_from, &origLength);
diff --git a/src/proxy/http/remap/RemapProcessor.cc
b/src/proxy/http/remap/RemapProcessor.cc
index e14f8c1217..550af9360e 100644
--- a/src/proxy/http/remap/RemapProcessor.cc
+++ b/src/proxy/http/remap/RemapProcessor.cc
@@ -171,6 +171,9 @@ RemapProcessor::finish_remap(HttpTransact::State *s,
UrlRewrite *table)
return false;
}
+ // Pass the volume_host_rec to the transaction state
+ s->cache_info.volume_host_rec = map->getVolumeHostRec();
+
// Do fast ACL filtering (it is safe to check map here)
table->PerformACLFiltering(s, map);
diff --git a/src/proxy/http/remap/UrlMapping.cc
b/src/proxy/http/remap/UrlMapping.cc
index 0a3a85236e..887625c91a 100644
--- a/src/proxy/http/remap/UrlMapping.cc
+++ b/src/proxy/http/remap/UrlMapping.cc
@@ -27,6 +27,11 @@
#include "records/RecCore.h"
#include "tscore/ink_cap.h"
+// Avoid including private header files
+struct CacheHostRecord;
+extern void destroyCacheHostRecord(CacheHostRecord *rec);
+extern CacheHostRecord *createCacheHostRecord(const char *volume_str, char
*errbuf, size_t errbufsize);
+
namespace
{
DbgCtl dbg_ctl_url_rewrite{"url_rewrite"};
@@ -84,11 +89,39 @@ url_mapping::~url_mapping()
delete afr;
}
+ // Destroy any volume hosting records
+ destroyCacheHostRecord(volume_host_rec.load(std::memory_order_acquire));
+
// Destroy the URLs
fromURL.destroy();
toURL.destroy();
}
+bool
+url_mapping::initVolumeHostRec(char *errbuf, size_t errbufsize)
+{
+ if (_volume_str.empty() || volume_host_rec.load(std::memory_order_acquire)
!= nullptr) {
+ return true;
+ }
+
+ CacheHostRecord *new_rec = createCacheHostRecord(_volume_str.c_str(),
errbuf, errbufsize);
+
+ if (!new_rec) {
+ return false;
+ }
+
+ CacheHostRecord *expected = nullptr;
+
+ if (volume_host_rec.compare_exchange_strong(expected, new_rec,
std::memory_order_acq_rel)) {
+ Dbg(dbg_ctl_url_rewrite, "Initialized volume_host_rec for @volume=%s",
_volume_str.c_str());
+ return true;
+ } else {
+ // Another thread beat us to it, destroy our copy
+ destroyCacheHostRecord(new_rec);
+ return true;
+ }
+}
+
void
url_mapping::Print() const
{
diff --git a/src/records/RecordsConfig.cc b/src/records/RecordsConfig.cc
index 215b6ffd6a..4ea75df726 100644
--- a/src/records/RecordsConfig.cc
+++ b/src/records/RecordsConfig.cc
@@ -82,6 +82,8 @@ static constexpr RecordElement RecordsConfig[] =
,
{RECT_CONFIG, "proxy.config.cache.persist_bad_disks", RECD_INT, "0",
RECU_RESTART_TS, RR_NULL, RECC_INT, "[0-1]", RECA_NULL}
,
+ {RECT_CONFIG, "proxy.config.cache.default_volumes", RECD_STRING, "",
RECU_RESTART_TS, RR_NULL, RECC_NULL, nullptr, RECA_NULL}
+ ,
{RECT_CONFIG, "proxy.config.output.logfile.name", RECD_STRING,
"traffic.out", RECU_RESTART_TS, RR_REQUIRED, RECC_NULL, nullptr,
RECA_NULL}
,
diff --git a/src/traffic_server/traffic_server.cc
b/src/traffic_server/traffic_server.cc
index 9a24991059..39b3074ec8 100644
--- a/src/traffic_server/traffic_server.cc
+++ b/src/traffic_server/traffic_server.cc
@@ -831,6 +831,10 @@ CB_After_Cache_Init()
start = ink_atomic_swap(&delay_listen_for_cache, -1);
emit_fully_initialized_message();
+ // Initialize volume_host_rec for any remap rules with @volume= directives
+ // that were deferred during startup because cache wasn't ready yet
+ init_remap_volume_host_records();
+
if (1 == start) {
// The delay_listen_for_cache value was 1, therefore the main function
// delayed the call to start_HttpProxyServer until we got here. We must
diff --git a/tests/gold_tests/autest-site/ats_replay.test.ext
b/tests/gold_tests/autest-site/ats_replay.test.ext
index 913257e7c3..976dc73b3d 100644
--- a/tests/gold_tests/autest-site/ats_replay.test.ext
+++ b/tests/gold_tests/autest-site/ats_replay.test.ext
@@ -53,6 +53,15 @@ def configure_ats(obj: 'TestRun', server: 'Process',
ats_config: dict, dns: Opti
if logging_yaml != None:
ts.Disk.logging_yaml.AddLines(yaml.dump(logging_yaml).split('\n'))
+ # Configure volume.config if specified.
+ volume_config = ats_config.get('volume_config', [])
+ for vol in volume_config:
+ parts = [f"volume={vol['volume']}", f"scheme={vol['scheme']}",
f"size={vol['size']}"]
+ for opt_key in ('ramcache', 'ram_cache_size', 'ram_cache_cutoff',
'avg_obj_size', 'fragment_size'):
+ if opt_key in vol:
+ parts.append(f"{opt_key}={vol[opt_key]}")
+ ts.Disk.volume_config.AddLine(' '.join(parts))
+
remap_config = ats_config.get('remap_config', [])
for remap_entry in remap_config:
if isinstance(remap_entry, str):
@@ -63,11 +72,14 @@ def configure_ats(obj: 'TestRun', server: 'Process',
ats_config: dict, dns: Opti
to_url = to_url.replace('{SERVER_HTTP_PORT}',
str(server.Variables.http_port))
to_url = to_url.replace('{SERVER_HTTPS_PORT}',
str(server.Variables.https_port))
plugins = remap_entry.get('plugins', [])
+ options = remap_entry.get('options', [])
line = f'map {from_url} {to_url}'
for plugin in plugins:
line += f' @plugin={plugin["name"]}'
for arg in plugin["args"]:
line += f' @pparam={arg}'
+ for option in options:
+ line += f' {option}'
ts.Disk.remap_config.AddLine(line)
if dns:
ts.Disk.records_config.update(
diff --git a/tests/gold_tests/cache/cache_volume_defaults.replay.yaml
b/tests/gold_tests/cache/cache_volume_defaults.replay.yaml
new file mode 100644
index 0000000000..7f90c7efc9
--- /dev/null
+++ b/tests/gold_tests/cache/cache_volume_defaults.replay.yaml
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+meta:
+ version: "1.0"
+
+# Test that proxy.config.cache.default_volumes and @volume= work correctly.
+# Uses a single volume at 100% so it fits in the 256MB min_cfg storage
+# (128MB stripe rounding requires at least 128MB per volume).
+autest:
+ description: "Test proxy.config.cache.default_volumes configuration"
+
+ dns:
+ name: "dns"
+
+ server:
+ name: "server"
+
+ client:
+ name: "client"
+
+ ats:
+ name: "ts"
+
+ records_config:
+ proxy.config.diags.debug.enabled: 1
+ proxy.config.diags.debug.tags: "cache|cache_hosting|cache_init"
+ proxy.config.http.insert_response_via_str: 0
+ proxy.config.cache.enable_read_while_writer: 0
+ # Set default volumes to volume 1
+ proxy.config.cache.default_volumes: "1"
+
+ volume_config:
+ # Single volume — multi-param line proves parsing works
+ - volume: 1
+ scheme: "http"
+ size: "100%"
+
+ remap_config:
+ # Default volume selection (uses default_volumes -> volume 1)
+ - from: "http://default.example.com/";
+ to: "http://backend.ex:{SERVER_HTTP_PORT}/";
+
+ # Explicit @volume=1 selection
+ - from: "http://volume1.example.com/";
+ to: "http://backend.ex:{SERVER_HTTP_PORT}/";
+ options:
+ - "@volume=1"
+
+# All transactions in a single session for sequential execution.
+sessions:
+- transactions:
+
+ # Test 1: Request using default_volumes
+ - client-request:
+ method: "GET"
+ version: "1.1"
+ url: "/default_test"
+ headers:
+ fields:
+ - [Host, default.example.com]
+ - [X-Test-ID, "default-volumes-test"]
+ - [uuid, default-miss]
+
+ server-response:
+ status: 200
+ reason: OK
+ headers:
+ fields:
+ - [Content-Type, "text/plain"]
+ - [Cache-Control, "max-age=3600"]
+ - [X-Default-Volumes-Test, "success"]
+ content:
+ data: "Content using default_volumes"
+
+ proxy-response:
+ status: 200
+ headers:
+ fields:
+ - [X-Default-Volumes-Test, { value: "success", as: equal }]
+
+ # Test 2: Request with explicit @volume=1
+ - client-request:
+ method: "GET"
+ version: "1.1"
+ url: "/volume1_test"
+ headers:
+ fields:
+ - [Host, volume1.example.com]
+ - [X-Test-ID, "volume1-override-test"]
+ - [uuid, volume1-miss]
+
+ server-response:
+ status: 200
+ reason: OK
+ headers:
+ fields:
+ - [Content-Type, "text/plain"]
+ - [Cache-Control, "max-age=3600"]
+ - [X-Volume-Override, "volume1"]
+ content:
+ data: "Content explicitly on volume 1"
+
+ proxy-response:
+ status: 200
+ headers:
+ fields:
+ - [X-Volume-Override, { value: "volume1", as: equal }]
+
+ # Test 3: Cache hit - verify content cached via default_volumes is served
+ - client-request:
+ delay: 100ms
+ method: "GET"
+ version: "1.1"
+ url: "/default_test"
+ headers:
+ fields:
+ - [Host, default.example.com]
+ - [X-Test-ID, "cache-hit-verify"]
+ - [uuid, default-hit]
+
+ # Server should not be contacted for a cache hit
+ server-response:
+ status: 404
+ reason: "Should not reach server"
+
+ proxy-response:
+ status: 200
+ content:
+ data: "Content using default_volumes"
+ verify: { as: equal }
diff --git a/tests/gold_tests/cache/cache_volume_defaults.test.py
b/tests/gold_tests/cache/cache_volume_defaults.test.py
new file mode 100644
index 0000000000..8e6cde2f85
--- /dev/null
+++ b/tests/gold_tests/cache/cache_volume_defaults.test.py
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+Test.Summary = '''
+Test proxy.config.cache.default_volumes configuration:
+- Verify that default_volumes is used as fallback when no other volume
selection applies
+- Verify that @volume= directive takes priority over default_volumes
+- Verify that hosting.config takes priority over default_volumes
+'''
+
+Test.ATSReplayTest(replay_file="cache_volume_defaults.replay.yaml")
diff --git a/tests/gold_tests/cache/cache_volume_features.replay.yaml
b/tests/gold_tests/cache/cache_volume_features.replay.yaml
new file mode 100644
index 0000000000..6c7c815427
--- /dev/null
+++ b/tests/gold_tests/cache/cache_volume_features.replay.yaml
@@ -0,0 +1,141 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+meta:
+ version: "1.0"
+
+# This test proves that multi-parameter volume.config lines are parsed
+# correctly. The generated volume.config lines are:
+#
+# volume=1 scheme=http size=50% ram_cache_size=32M ram_cache_cutoff=8K
+# volume=2 scheme=http size=50%
+#
+# All 5 key=value pairs on line 1 must be parsed from the single line.
+# If the inner loop's "tmp = line_end" advancement were broken (as claimed
+# in a PR review), only "volume=1" would be parsed, scheme and size would
+# be missing, and the volume would be rejected -- disabling the cache.
+autest:
+ description: "Test cache volume features: per-volume RAM cache and @volume=
directive"
+
+ dns:
+ name: "dns"
+
+ server:
+ name: "server"
+
+ client:
+ name: "client"
+
+ ats:
+ name: "ts"
+
+ records_config:
+ proxy.config.diags.debug.enabled: 1
+ proxy.config.diags.debug.tags: "cache|cache_hosting|ram_cache"
+ proxy.config.cache.ram_cache.size: 128M
+ proxy.config.cache.ram_cache_cutoff: 4K
+ proxy.config.http.insert_response_via_str: 0
+ proxy.config.cache.enable_read_while_writer: 0
+
+ volume_config:
+ # Volume 1 with all new parameters — 5 key=value pairs on one line
+ - volume: 1
+ scheme: "http"
+ size: "50%"
+ ram_cache_size: "32M"
+ ram_cache_cutoff: "8K"
+ # Volume 2 — simple line, no extra params
+ - volume: 2
+ scheme: "http"
+ size: "50%"
+
+ log_validation:
+ traffic_out:
+ excludes:
+ # No volume.config lines should be discarded
+ - expression: "discarding.*entry at line"
+ description: "No volume.config entries should be discarded"
+
+ remap_config:
+ - from: "http://volume1.example.com/";
+ to: "http://backend.ex:{SERVER_HTTP_PORT}/";
+ options:
+ - "@volume=1"
+ - from: "http://volume2.example.com/";
+ to: "http://backend.ex:{SERVER_HTTP_PORT}/";
+ options:
+ - "@volume=2"
+
+# All transactions in a single session for sequential execution.
+sessions:
+- transactions:
+
+ # Test 1: Request through volume 1 with per-volume RAM cache settings
+ - client-request:
+ method: "GET"
+ version: "1.1"
+ url: "/test1"
+ headers:
+ fields:
+ - [Host, volume1.example.com]
+ - [X-Test-ID, "volume1-ramcache-test"]
+ - [uuid, 1]
+
+ server-response:
+ status: 200
+ reason: OK
+ headers:
+ fields:
+ - [Content-Type, "text/plain"]
+ - [Cache-Control, "max-age=3600"]
+ - [X-Volume-Test, "1"]
+ content:
+ data: "Content for volume 1 with custom RAM cache"
+
+ proxy-response:
+ status: 200
+ headers:
+ fields:
+ - [X-Volume-Test, { value: "1", as: equal }]
+
+ # Test 2: Request through volume 2 (simple volume, no extra params)
+ - client-request:
+ method: "GET"
+ version: "1.1"
+ url: "/test2"
+ headers:
+ fields:
+ - [Host, volume2.example.com]
+ - [X-Test-ID, "volume2-test"]
+ - [uuid, 2]
+
+ server-response:
+ status: 200
+ reason: OK
+ headers:
+ fields:
+ - [Content-Type, "text/plain"]
+ - [Cache-Control, "max-age=3600"]
+ - [X-Volume-Test, "2"]
+ content:
+ data: "Content for volume 2"
+
+ proxy-response:
+ status: 200
+ headers:
+ fields:
+ - [X-Volume-Test, { value: "2", as: equal }]
+
diff --git a/tests/gold_tests/cache/cache_volume_features.test.py
b/tests/gold_tests/cache/cache_volume_features.test.py
new file mode 100644
index 0000000000..c52575ef56
--- /dev/null
+++ b/tests/gold_tests/cache/cache_volume_features.test.py
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+Test.Summary = '''
+Comprehensive test suite for cache volume features:
+- Per-volume RAM cache configuration (ram_cache_size, ram_cache_cutoff)
+- @volume= directive in remap.config for volume selection
+- Integration between both features
+'''
+# TODO: hosting.config + @volume= priority interaction test requires
ats_replay.test.ext to support volume_config/hosting_config.
+
+Test.ATSReplayTest(replay_file="cache_volume_features.replay.yaml")
