This is an automated email from the ASF dual-hosted git repository.
masaori335 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/trafficserver.git
The following commit(s) were added to refs/heads/master by this push:
new 48fc842914 slice: purge every block of an object, not just those
before a gap (#13475)
48fc842914 is described below
commit 48fc8429140dc9da2bc237c7fb890fb4a0bbd54d
Author: Masaori Koshiba <[email protected]>
AuthorDate: Fri Aug 7 11:41:14 2026 +0900
slice: purge every block of an object, not just those before a gap (#13475)
* slice: purge every block of an object, not just those before a gap
A PURGE is meant to discard the object, but the block walk stopped at the
first block that was not in cache, so any object whose cached blocks were
not a contiguous run from block 0 was only partially purged, and the client
still got a 200. A gap in the middle left every block behind it cached; an
uncached first block purged nothing and relayed that block's 404; and a
"bytes=-N" purge deleted the head while leaving the tail it had named.
The stop was load-bearing. The walk's only other terminator needs the object
length, which slice only ever learned from a 206's Content-Range, and a
PURGE
response has none. So the core now reports the removed object's extent as
X-Purged-Content-Range on a PURGE cache hit, and the walk learns where the
object ends from the blocks it is already deleting. It is not Content-Range
itself, since that header on a 200 is meaningless under RFC 9110 and
cache_range_requests reads the pair as a stored 206 and rewrites the status.
PURGE gets its own state machine in the plugin, so it no longer routes
through handleFirstServerHeader, whose double duty as "form and emit the
client response" is what leaked the 404. A 404 for a block is stepped over,
nothing is written downstream until the walk finishes, and the response is
then synthesized: 200 if any block was removed, 404 if none was. The extent
is taken as a maximum rather than the first value seen, since blocks of one
object disagree when the origin object was replaced in place.
Until some block reports an extent the walk has no end but a miss bound, so
add --purge-probe-blocks, default 8, capping consecutive uncached blocks. It
never limits how many blocks a purge removes. A per-request override named
by
--purge-probe-header, default X-Slice-Purge-Probe, lets an operator who
knows
the object size widen it. A suffix range names its blocks by distance from
an
end slice does not know yet, so such a purge is widened to the whole object,
a superset of what was asked. A PURGE whose Range cannot be parsed is
refused
with a 400 rather than guessing which blocks were meant.
Tests cover the traversal over gaps, an uncached first block, both
open-ended
range forms, blocks that disagree about the object length, the miss bound
and
its override, and the refusal. They measure on the origin rather than the
response body, since a purged block and a surviving block are
indistinguishable
to the client. Two further tests reproduce the client-visible failures of an
origin object replaced in place under a child/parent hierarchy, which is how
this problem was found.
* Doc: Fix example of HTTP/1.1 messages
* slice: pace the two PURGE request-validation error logs
Both values are client supplied, so a bad one repeats as fast as
requests arrive. Route them through Config::canLogError() like the
other slice error paths.
* slice: let a PURGE range bound the walk before any extent is known
The requested range end comes from the client's Range header, but the
walk only consulted it once some block had reported the object's extent,
which only a block that was actually removed can do. A closed-range
PURGE whose leading blocks were uncached therefore ran past its range
end and removed blocks the client never named.
* slice: do not report a partial PURGE as a success
A block PURGE answering neither 200 nor 404 was read as "already
absent", so a 403 from ip_allow or a 502 counted as a miss and the walk
carried on to answer 200 on the strength of blocks it had removed
earlier, telling the client the object was gone while part of it was
still cached. Such a status says nothing about the blocks behind it
either, so the walk now stops there and reports it.
---
doc/admin-guide/plugins/slice.en.rst | 99 ++-
doc/admin-guide/storage/index.en.rst | 17 +-
plugins/slice/Config.cc | 26 +-
plugins/slice/Config.h | 6 +
plugins/slice/Data.h | 23 +-
plugins/slice/HttpHeader.cc | 26 +
plugins/slice/HttpHeader.h | 11 +
plugins/slice/client.cc | 70 ++
plugins/slice/response.cc | 15 +
plugins/slice/response.h | 6 +
plugins/slice/server.cc | 213 ++++-
plugins/slice/server.h | 11 +
plugins/slice/util.cc | 8 +-
src/proxy/http/HttpTransact.cc | 19 +
.../replay/slice_purge_gaps_client.replay.yaml | 895 +++++++++++++++++++++
.../replay/slice_purge_gaps_server.replay.yaml | 586 ++++++++++++++
.../slice_stale_generation_client.replay.yaml | 206 +++++
.../slice_stale_generation_server.replay.yaml | 286 +++++++
.../slice/rules/purge_block_failure.conf | 41 +
.../pluginTest/slice/slice_purge_gaps.test.py | 346 ++++++++
.../slice/slice_stale_generation.test.py | 327 ++++++++
21 files changed, 3206 insertions(+), 31 deletions(-)
diff --git a/doc/admin-guide/plugins/slice.en.rst
b/doc/admin-guide/plugins/slice.en.rst
index 40d606129c..21edd7eb20 100644
--- a/doc/admin-guide/plugins/slice.en.rst
+++ b/doc/admin-guide/plugins/slice.en.rst
@@ -186,7 +186,21 @@ The slice plugin supports the following options::
that causes `cache_range_requests` to be bypassed in such requests, and
allow ATS to handle those range requests internally.
-
+ --purge-probe-blocks=<int> (optional)
+ Default is 8
+ How many consecutive uncached slice blocks a PURGE walks, before any
block
+ has reported the object's extent, until it concludes that nothing
about the
+ object is cached. May be overridden per request with the header named
by
+ ``--purge-probe-header``. See `Purge Requests`_.
+ -q for short
+
+ --purge-probe-header=<string> (optional)
+ Default is X-Slice-Purge-Probe
+ Name of the request header a PURGE may use to override
+ ``--purge-probe-blocks`` for that request. A malformed value is
ignored
+ in favour of the configured default. Slice strips this header from the
+ block requests it issues.
+ -H for short
Examples::
@@ -338,17 +352,78 @@ requests may end up being served from temporally
different assets.
Purge Requests
--------------
-The slice plugin supports PURGE requests, discarding the requested object from
cache.
-If a range is given in the client request, only the slice blocks from the
-requested range will be purged (if in cache). If not, all of the blocks will
be discarded
-from the cache.
-
-If a block receives a 404, indicating the requested block to be purged is not
in the cache,
-slice will not continue to purge the following blocks.
-
-The functionality works with `--ref-relative` both enabled and disabled. If
`--ref-relative` is
-disabled (using slice 0 as the reference block), requesting to PURGE a block
that does not have
-slice 0 in its range will still PURGE the slice 0 block, as the reference
block is always processed.
+The slice plugin supports PURGE requests, discarding the requested object from
+cache. Without a range every block is discarded; with a range, the blocks that
+range covers are. Two cases below purge more than the range names: a suffix
range,
+and block 0 when ``--ref-relative`` is disabled.
+
+Slice issues one PURGE per block and walks every block it was asked for,
whether
+or not each one is currently cached. A block that is already absent answers 404
+internally; that is simply noted and the walk continues, so a gap left by
+per-block eviction cannot leave the blocks behind it in cache.
+
+Slice learns where the object ends from the blocks it removes. PURGE is a
Traffic
+Server extension, so a successful block purge reports the removed object's
extent
+in a ``X-Purged-Content-Range`` header, and the walk continues to the last
block that
+extent implies. Blocks of one object can disagree about its length when the
origin
+object has been replaced in place; slice takes the largest extent any block
+reports, so the longer generation's tail is not left behind.
+
+Until some block has reported an extent, a walk over an open-ended range has no
+end but the miss bound: it stops after ``--purge-probe-blocks`` consecutive
+uncached blocks and reports that nothing was found. That is what bounds a PURGE
+for a URL which is not cached at all.
+
+An operator often knows more about the object than the plugin does, since the
+block count is just the object's size divided by the block size. That count
can be
+supplied per request with the header named by ``--purge-probe-header``, default
+``X-Slice-Purge-Probe``::
+
+ PURGE /obj HTTP/1.1
+ X-Slice-Purge-Probe: 64
+
+This only changes how long the walk keeps going without having found anything;
it
+never limits how many blocks are purged once an extent is known.
+
+The bound has to be able to span a whole object, because in the worst case only
+the object's last block is still cached, so the value an operator wants is the
+object's size divided by the block size: a 10 GB object in 1 MB blocks needs
+10240. There is no ceiling on it beyond that, since reaching the bound costs
one
+internal cache lookup per block and PURGE is already restricted by
+:file:`ip_allow.yaml`. A malformed value is ignored in favour of the configured
+default, and slice strips the header from the block requests it issues.
+
+If the bound is reached, slice logs that it gave up and reports ``404`` even
though
+later blocks may still be cached. Raise ``--purge-probe-blocks``, or send the
+override, for objects whose leading blocks are routinely absent.
+
+A client range that is already closed, such as ``bytes=0-6399999999``, bounds
the
+walk directly, and is clamped against the object's extent as soon as some block
+reports one. An over-estimate therefore costs no extra block PURGEs beyond the
end
+of the object, and if no block is cached at all the miss bound stops the walk.
+
+A suffix range, ``bytes=-<n>``, names its blocks by their distance from an end
+slice does not know yet, and purging is the only way it could find out. Rather
+than guess at the start, such a purge is widened to the whole object: a
superset of
+what was asked for, so the named blocks certainly go. A ``GET`` with the same
header
+is unaffected and still returns exactly the last *n* bytes.
+
+The response is sent once the walk is complete: ``200`` if at least one block
was
+removed, ``404`` if none was found. This matches what Traffic Server reports
for a
+PURGE of an object that is not sliced.
+
+A block whose PURGE returns neither ``200`` nor ``404`` — a ``403`` from
+:file:`ip_allow.yaml`, for instance, or a ``502`` — says nothing about whether
that
+block was cached, and nothing about the blocks behind it. The walk stops there
and that
+status is reported in place of ``200``, so the two success statuses keep
meaning what
+they say: ``200`` that the object is gone and ``404`` that it was not there,
never that
+this proxy could not tell. Blocks the walk had already removed stay removed,
and blocks
+behind the failing one are left cached, so such a PURGE is worth repeating once
+whatever refused it has been dealt with.
+
+The functionality works with ``--ref-relative`` both enabled and disabled.
With it
+disabled, block 0 is always the first block walked, so a PURGE whose range
does not
+cover block 0 still purges it.
Conditional Slicing
-------------------
diff --git a/doc/admin-guide/storage/index.en.rst
b/doc/admin-guide/storage/index.en.rst
index aa3754df5e..af6d23de06 100644
--- a/doc/admin-guide/storage/index.en.rst
+++ b/doc/admin-guide/storage/index.en.rst
@@ -304,7 +304,7 @@ from any other IP, we connect to the daemon via localhost:
::
> Host: example.com
> Accept: */*
>
- < HTTP/1.1 200 Ok
+ < HTTP/1.1 200 OK
< Date: Thu, 08 Jan 2010 20:32:07 GMT
< Connection: keep-alive
@@ -312,6 +312,21 @@ The next time Traffic Server receives a request for the
removed object,
it will contact the origin server to retrieve a new copy, which will replace
the previously cached version in Traffic Server.
+If the removed object was stored as a partial response, that is if it carried a
+``Content-Range``, then the ``200 OK`` also reports that range back in a
+``X-Purged-Content-Range`` header::
+
+ < HTTP/1.1 200 OK
+ < X-Purged-Content-Range: bytes 0-1048575/9437184
+
+This lets a caller that holds one piece of a larger resource learn the whole
+resource's extent without a second lookup. It is what allows the
+:ref:`admin-plugins-slice` plugin to purge an object block by block and know
which
+block is the last one. The range is reported under its own header name rather
than
+as ``Content-Range``, because ``Content-Range`` on a ``200`` response has no
+meaning under :rfc:`9110` and is read by other components as a sign that a
stored
+partial response is being served.
+
This procedure only removes the index to the object from a specific Traffic
Server
cache. While the object remains on disk, Traffic Server will no longer able to
find
the object. The next request for that object will result in a fresh copy of the
diff --git a/plugins/slice/Config.cc b/plugins/slice/Config.cc
index b95ace3bab..b5e19f7189 100644
--- a/plugins/slice/Config.cc
+++ b/plugins/slice/Config.cc
@@ -27,8 +27,9 @@
namespace
{
-constexpr std::string_view DefaultSliceSkipHeader = {"X-Slicer-Info"};
-constexpr std::string_view DefaultCrrIdentHeader = {"X-Crr-Ident"};
+constexpr std::string_view DefaultSliceSkipHeader = {"X-Slicer-Info"};
+constexpr std::string_view DefaultCrrIdentHeader = {"X-Crr-Ident"};
+constexpr std::string_view DefaultPurgeProbeHeader = {"X-Slice-Purge-Probe"};
} // namespace
Config::~Config()
@@ -121,13 +122,15 @@ Config::fromArgs(int const argc, char const *const argv[])
{const_cast<char *>("minimum-size"), required_argument, nullptr,
'm'},
{const_cast<char *>("metadata-cache-size"), required_argument, nullptr,
'z'},
{const_cast<char *>("stats-prefix"), required_argument, nullptr,
'x'},
+ {const_cast<char *>("purge-probe-blocks"), required_argument, nullptr,
'q'},
+ {const_cast<char *>("purge-probe-header"), required_argument, nullptr,
'H'},
{nullptr, 0, nullptr, 0
},
};
// getopt assumes args start at '1' so this hack is needed
char *const *argvp = (const_cast<char *const *>(argv) - 1);
for (;;) {
- int const opt = getopt_long(argc + 1, argvp, "b:de:g:i:lm:p:r:s:t:x:z:",
longopts, nullptr);
+ int const opt = getopt_long(argc + 1, argvp,
"b:de:g:H:i:lm:p:q:r:s:t:x:z:", longopts, nullptr);
if (-1 == opt) {
break;
}
@@ -248,6 +251,19 @@ Config::fromArgs(int const argc, char const *const argv[])
stat_prefix = optarg;
DEBUG_LOG("Stat prefix: %s", stat_prefix.c_str());
} break;
+ case 'q': {
+ int const blocksread = atoi(optarg);
+ if (0 < blocksread) {
+ m_purge_probe_blocks = blocksread;
+ DEBUG_LOG("Using purge probe blocks %d", m_purge_probe_blocks);
+ } else {
+ ERROR_LOG("Invalid purge-probe-blocks: %s", optarg);
+ }
+ } break;
+ case 'H': {
+ m_purge_probe_header.assign(optarg);
+ DEBUG_LOG("Using purge probe header %s", optarg);
+ } break;
default:
break;
}
@@ -275,6 +291,10 @@ Config::fromArgs(int const argc, char const *const argv[])
m_skip_header = DefaultSliceSkipHeader;
DEBUG_LOG("Using default slice skip header %s", m_skip_header.c_str());
}
+ if (m_purge_probe_header.empty()) {
+ m_purge_probe_header = DefaultPurgeProbeHeader;
+ DEBUG_LOG("Using default purge probe header %s",
m_purge_probe_header.c_str());
+ }
if (m_min_size_to_slice > 0) {
if (m_oscache.has_value()) {
diff --git a/plugins/slice/Config.h b/plugins/slice/Config.h
index 4fd15a2506..f9888e6640 100644
--- a/plugins/slice/Config.h
+++ b/plugins/slice/Config.h
@@ -36,6 +36,8 @@ struct Config {
static constexpr int64_t const blockbytesmax = 1024 * 1024 * 128; //
128MB
static constexpr int64_t const blockbytesdefault = 1024 * 1024; // 1MB
+ static constexpr int const purgeprobeblocksdefault = 8;
+
int64_t m_blockbytes{blockbytesdefault};
std::string m_remaphost; // remap host to use for loopback slice GET
std::string m_regexstr; // regex string for things to slice (default all)
@@ -49,8 +51,12 @@ struct Config {
bool m_head_strip_range{false}; // strip range header for head requests
uint64_t m_min_size_to_slice{0}; // Only strip objects larger than this
+ // consecutive uncached blocks a purge tolerates before giving up on the
object
+ int m_purge_probe_blocks{purgeprobeblocksdefault};
+
std::string m_skip_header;
std::string m_crr_ident_header;
+ std::string m_purge_probe_header; // request header overriding
m_purge_probe_blocks
// Convert optarg to bytes
static int64_t bytesFrom(char const *const valstr);
diff --git a/plugins/slice/Data.h b/plugins/slice/Data.h
index 0c19dfaafa..a3fd8e1cf4 100644
--- a/plugins/slice/Data.h
+++ b/plugins/slice/Data.h
@@ -77,6 +77,11 @@ struct Data {
int64_t m_blockskip{0}; // number of bytes to skip in this block
int64_t m_blockconsumed{0}; // body bytes consumed
+ int64_t m_purge_hits{0}; // blocks a purge actually
removed
+ int m_purge_misses{0}; // consecutive uncached
blocks the walk has seen
+ int m_purge_miss_bound{0}; // from the config or the
request header
+ TSHttpStatus m_purge_error{TS_HTTP_STATUS_NONE}; // block failure that ended
the walk
+
BlockState m_blockstate{Pending}; // is there an active slice block
int64_t m_bytestosend{0}; // header + content bytes to send
@@ -109,11 +114,25 @@ struct Data {
memset(&m_client_ip, 0, sizeof(m_client_ip));
}
- // Check if response only expects header
+ // HEAD only; a purge sends just a header too but never reaches the transfer
path
bool
onlyHeader() const
{
- return (m_method_type == TS_HTTP_METHOD_HEAD || m_method_type ==
TS_HTTP_METHOD_PURGE);
+ return m_method_type == TS_HTTP_METHOD_HEAD;
+ }
+
+ bool
+ is_purge() const
+ {
+ return m_method_type == TS_HTTP_METHOD_PURGE;
+ }
+
+ // The purge range, closed against the object length once known. m_req_range
+ // stays as sent so a longer extent can widen the walk; a clamp could only
shrink.
+ Range
+ purge_range() const
+ {
+ return (m_contentlen < 0) ? m_req_range :
m_req_range.intersectedWith(Range(0, m_contentlen));
}
~Data()
diff --git a/plugins/slice/HttpHeader.cc b/plugins/slice/HttpHeader.cc
index d08e76db7a..e075036a24 100644
--- a/plugins/slice/HttpHeader.cc
+++ b/plugins/slice/HttpHeader.cc
@@ -326,6 +326,32 @@ HttpHeader::toString() const
/////// HdrMgr
+bool
+HdrMgr::create_response(TSHttpStatus const status)
+{
+ resetHeader();
+
+ if (nullptr == m_buffer) {
+ m_buffer = TSMBufferCreate();
+ }
+
+ m_lochdr = TSHttpHdrCreate(m_buffer);
+ if (nullptr == m_lochdr) {
+ return false;
+ }
+
+ TSHttpHdrTypeSet(m_buffer, m_lochdr, TS_HTTP_TYPE_RESPONSE);
+ TSHttpHdrVersionSet(m_buffer, m_lochdr, TS_HTTP_VERSION(1, 1));
+ TSHttpHdrStatusSet(m_buffer, m_lochdr, status);
+
+ char const *const reason = TSHttpHdrReasonLookup(status);
+ if (nullptr != reason) {
+ TSHttpHdrReasonSet(m_buffer, m_lochdr, reason, strlen(reason));
+ }
+
+ return true;
+}
+
TSParseResult
HdrMgr::populateFrom(TSHttpParser const http_parser, TSIOBufferReader const
reader, HeaderParseFunc const parsefunc,
int64_t *const bytes)
diff --git a/plugins/slice/HttpHeader.h b/plugins/slice/HttpHeader.h
index c52738e5d1..0fd50c779a 100644
--- a/plugins/slice/HttpHeader.h
+++ b/plugins/slice/HttpHeader.h
@@ -39,6 +39,10 @@
constexpr std::string_view SLICE_CRR_HEADER = {"Slice-Crr-Status"};
constexpr std::string_view SLICE_CRR_VAL = "1";
+// extent of the object a PURGE removed, reported by ATS on a successful purge.
+// Emitted by HttpTransact::delete_all_document_alternates_and_return.
+constexpr std::string_view PURGED_CONTENT_RANGE = {"X-Purged-Content-Range"};
+
/**
Designed to be a cheap throwaway struct which allows a
consumer to make various calls to manipulate headers.
@@ -207,6 +211,13 @@ struct HdrMgr {
}
}
+ /** Create an owned HTTP/1.1 response header with the given status.
+ *
+ * For a response slice forms itself, with no server response to relay. An
+ * intercept is an HTTP/1.x channel, so the version is not negotiable.
+ */
+ bool create_response(TSHttpStatus const status);
+
void
resetHeader()
{
diff --git a/plugins/slice/client.cc b/plugins/slice/client.cc
index 1494ffa38a..5c5933cd10 100644
--- a/plugins/slice/client.cc
+++ b/plugins/slice/client.cc
@@ -19,9 +19,49 @@
#include "client.h"
#include "Config.h"
+#include "server.h"
#include "util.h"
+#include "swoc/TextView.h"
+
+#include <cctype>
#include <cinttypes>
+#include <limits>
+
+namespace
+{
+// Miss bound for this purge: the request header when usable, else the config
value.
+int
+purge_miss_bound(HttpHeader const &header, Config *const conf)
+{
+ char probestr[64];
+ int probelen = sizeof(probestr);
+
+ if (!header.valueForKey(conf->m_purge_probe_header.data(),
conf->m_purge_probe_header.size(), probestr, &probelen)) {
+ return conf->m_purge_probe_blocks;
+ }
+
+ swoc::TextView value{probestr, static_cast<size_t>(probelen)};
+ // isspace is only defined for values representable as unsigned char
+ value.trim_if([](char c) { return 0 != isspace(static_cast<unsigned
char>(c)); });
+
+ swoc::TextView parsed;
+ intmax_t const blocks = swoc::svtoi(value, &parsed, 10);
+
+ // parsed must cover the whole value: "8abc" is a mistake, not eight blocks
+ if (parsed.size() != value.size() || blocks <= 0 ||
std::numeric_limits<int>::max() < blocks) {
+ // paced: the value is client supplied, so a bad one repeats as fast as
requests arrive
+ if (conf->canLogError()) {
+ ERROR_LOG("Ignoring invalid %.*s value '%.*s'",
static_cast<int>(conf->m_purge_probe_header.size()),
+ conf->m_purge_probe_header.data(), probelen, probestr);
+ }
+ return conf->m_purge_probe_blocks;
+ }
+
+ return static_cast<int>(blocks);
+}
+
+} // namespace
// this is called once per transaction when the client sends a req header
bool
@@ -94,6 +134,31 @@ handle_client_req(TSCont contp, TSEvent event, Data *const
data)
data->m_req_range = rangebe;
+ if (data->is_purge()) {
+ // The substituted range covers block 0, so walking it would delete the
head
+ if (TS_HTTP_STATUS_REQUESTED_RANGE_NOT_SATISFIABLE ==
data->m_statustype) {
+ // paced: the range is client supplied, so a bad one repeats as fast
as requests arrive
+ if (data->m_config->canLogError()) {
+ ERROR_LOG("Refusing PURGE with an unparseable range");
+ }
+ finish_purge(contp, data, TS_HTTP_STATUS_BAD_REQUEST);
+ return true;
+ }
+
+ data->m_purge_miss_bound = purge_miss_bound(header, data->m_config);
+ DEBUG_LOG("%p Purge miss bound %d block(s)", data,
data->m_purge_miss_bound);
+
+ // A suffix range cannot know its start block, so purge a superset:
everything
+ if (data->m_req_range.isEndBytes()) {
+ data->m_req_range = Range(0, Range::maxval);
+ data->m_blocknum = 0;
+ DEBUG_LOG("%p Purge suffix range widened to the whole object", data);
+ }
+
+ // The miss bound is for this proxy to act on, not to propagate
+ header.removeKey(conf->m_purge_probe_header.data(),
conf->m_purge_probe_header.size());
+ }
+
// remove ATS keys to avoid 404 loop
header.removeKey(TS_MIME_FIELD_VIA, TS_MIME_LEN_VIA);
header.removeKey(TS_MIME_FIELD_X_FORWARDED_FOR,
TS_MIME_LEN_X_FORWARDED_FOR);
@@ -126,6 +191,11 @@ handle_client_resp(TSCont contp, TSEvent event, Data
*const data)
{
switch (event) {
case TS_EVENT_VCONN_WRITE_READY: {
+ // finish_purge writes the whole response at once; nothing to throttle or
pull
+ if (data->is_purge()) {
+ break;
+ }
+
switch (data->m_blockstate) {
case BlockState::Fail:
case BlockState::PendingRef:
diff --git a/plugins/slice/response.cc b/plugins/slice/response.cc
index 94081cf7b2..a34ef3a6cb 100644
--- a/plugins/slice/response.cc
+++ b/plugins/slice/response.cc
@@ -86,6 +86,21 @@ string502(int const httpver)
return msg;
}
+// Form the response to a sliced PURGE, once every block has been walked
+bool
+form_purge_response(HdrMgr &hdrmgr, TSHttpStatus const status)
+{
+ if (!hdrmgr.create_response(status)) {
+ return false;
+ }
+
+ // The core adds Date, Age, Server and Connection to an intercept's response
+ HttpHeader header(hdrmgr.m_buffer, hdrmgr.m_lochdr);
+ header.setKeyVal(TS_MIME_FIELD_CONTENT_LENGTH, TS_MIME_LEN_CONTENT_LENGTH,
"0", 1);
+
+ return true;
+}
+
void
form416HeaderAndBody(HttpHeader &header, int64_t const contentlen, std::string
const &bodystr)
{
diff --git a/plugins/slice/response.h b/plugins/slice/response.h
index c116d22e21..569af1120b 100644
--- a/plugins/slice/response.h
+++ b/plugins/slice/response.h
@@ -25,4 +25,10 @@ std::string string502(int const httpver);
std::string const &bodyString416();
+/** Fill hdrmgr with the response to a sliced PURGE, for the caller to print.
+ *
+ * The header is owned by hdrmgr and destroyed with it.
+ */
+bool form_purge_response(HdrMgr &hdrmgr, TSHttpStatus const status);
+
void form416HeaderAndBody(HttpHeader &header, int64_t const contentlen,
std::string const &bodystr);
diff --git a/plugins/slice/server.cc b/plugins/slice/server.cc
index 7105d5157b..2b25f22c01 100644
--- a/plugins/slice/server.cc
+++ b/plugins/slice/server.cc
@@ -32,7 +32,7 @@
namespace
{
ContentRange
-contentRangeFrom(HttpHeader const &header)
+content_range_for_key(HttpHeader const &header, char const *const key, int
const keylen)
{
ContentRange bcr;
@@ -42,22 +42,25 @@ contentRangeFrom(HttpHeader const &header)
char rangestr[1024];
int rangelen = sizeof(rangestr);
- // look for expected Content-Range field
- bool const hasContentRange(header.valueForKey(TS_MIME_FIELD_CONTENT_RANGE,
TS_MIME_LEN_CONTENT_RANGE, rangestr, &rangelen));
-
- if (!hasContentRange) {
- DEBUG_LOG("invalid response header, no Content-Range");
+ if (!header.valueForKey(key, keylen, rangestr, &rangelen)) {
+ DEBUG_LOG("invalid response header, no %.*s", keylen, key);
} else {
// ensure null termination
rangestr[rangelen] = '\0';
if (!bcr.fromStringClosed(rangestr)) {
- DEBUG_LOG("invalid response header, malformed Content-Range, %s",
rangestr);
+ DEBUG_LOG("invalid response header, malformed %.*s, %s", keylen, key,
rangestr);
}
}
return bcr;
}
+ContentRange
+contentRangeFrom(HttpHeader const &header)
+{
+ return content_range_for_key(header, TS_MIME_FIELD_CONTENT_RANGE,
TS_MIME_LEN_CONTENT_RANGE);
+}
+
int64_t
contentLengthFrom(HttpHeader const &header)
{
@@ -136,7 +139,7 @@ handleFirstServerHeader(Data *const data, TSCont const
contp)
int64_t const hlen = TSHttpHdrLengthGet(header.m_buffer, header.m_lochdr);
int64_t const clen = contentLengthFrom(header);
if (TS_HTTP_STATUS_OK == header.status() && data->onlyHeader()) {
- DEBUG_LOG("HEAD/PURGE request stripped Range header: expects 200");
+ DEBUG_LOG("HEAD request stripped Range header: expects 200");
data->m_bytestosend = hlen;
data->m_blockexpected = 0;
TSVIONBytesSet(output_vio, hlen);
@@ -499,12 +502,201 @@ handleNextServerHeader(Data *const data)
return true;
}
+// Take the largest extent any block reports: blocks disagree when the origin
+// object was replaced in place, and the shorter one would leave a tail cached.
+void
+note_purge_extent(Data *const data, int64_t const length)
+{
+ if (length <= data->m_contentlen) {
+ return;
+ }
+
+ data->m_contentlen = length;
+ DEBUG_LOG("purge extent now %" PRId64 ", walking through block %" PRId64,
length,
+ data->purge_range().lastBlockFor(data->m_config->m_blockbytes));
+}
+
+// A block that could not be purged is not a block that was absent, so the
client must
+// not be told the object is gone. The walk stops here: the rest of the object
is
+// unknown, and a 5xx often means the cache is in no state to be asked again.
+void
+note_purge_failure(Data *const data, TSHttpStatus const status)
+{
+ data->m_purge_error = status;
+
+ // paced: a config refusing PURGE would otherwise log once per purge request
+ if (data->m_config->canLogError()) {
+ ERROR_LOG("Purge of block %" PRId64 " failed (%d), the object may be left
partly cached", data->m_blocknum, status);
+ }
+}
+
+// Record what the block response said, without answering the client.
+void
+note_purge_block_result(Data *const data)
+{
+ HttpHeader const header(data->m_resp_hdrmgr.m_buffer,
data->m_resp_hdrmgr.m_lochdr);
+ DEBUG_LOG("Purge block header\n%s", header.toString().c_str());
+
+ TSHttpStatus const status = header.status();
+
+ if (TS_HTTP_STATUS_OK == status) {
+ ++data->m_purge_hits;
+ data->m_purge_misses = 0;
+
+ // Not Content-Range: cache_range_requests reads that on a 200 as a stored
206
+ // and rewrites the status
+ ContentRange const purgedcr = content_range_for_key(header,
PURGED_CONTENT_RANGE.data(), PURGED_CONTENT_RANGE.size());
+ if (purgedcr.isValid() && 0 < purgedcr.m_length) {
+ note_purge_extent(data, purgedcr.m_length);
+ } else {
+ DEBUG_LOG("Purged block %" PRId64 " reported no usable extent",
data->m_blocknum);
+ }
+ } else if (TS_HTTP_STATUS_NOT_FOUND == status) {
+ // Already absent. The walk used to stop here, leaving every later block
cached.
+ ++data->m_purge_misses;
+ DEBUG_LOG("Purge block %" PRId64 " was not cached", data->m_blocknum);
+ } else {
+ // Anything else is a refusal or a failure, which says nothing about the
block
+ note_purge_failure(data, status);
+ }
+}
+
+// Issue the next purge, or answer the client if the walk is over.
+void
+advance_purge(TSCont const contp, Data *const data)
+{
+ // A block that could not be purged says nothing about the ones behind it
+ if (TS_HTTP_STATUS_NONE != data->m_purge_error) {
+ finish_purge(contp, data);
+ return;
+ }
+
+ int64_t const blockbytes = data->m_config->m_blockbytes;
+ Range const range = data->purge_range();
+
+ ++data->m_blocknum;
+ int64_t const firstblock = range.firstBlockFor(blockbytes);
+ if (data->m_blocknum < firstblock) {
+ data->m_blocknum = firstblock;
+ }
+
+ // The requested range bounds the walk whether or not the extent is known yet
+ if (!range.blockIsInside(blockbytes, data->m_blocknum)) {
+ finish_purge(contp, data);
+ return;
+ }
+
+ // An open ended range has no such bound until some block reports an extent
+ if (data->m_contentlen < 0 && data->m_purge_miss_bound <=
data->m_purge_misses) {
+ DEBUG_LOG("purge gave up after %d consecutive uncached block(s)",
data->m_purge_misses);
+ finish_purge(contp, data);
+ return;
+ }
+
+ data->m_blockstate = BlockState::Pending;
+ if (!request_block(contp, data)) {
+ note_purge_failure(data, TS_HTTP_STATUS_INTERNAL_SERVER_ERROR);
+ finish_purge(contp, data);
+ }
+}
+
} // namespace
+// Answer the client once every block has been walked. Nothing is written
+// downstream before this, so one uncached block cannot leak a 404 to the
client.
+// A non-NONE status overrides the outcome of the walk.
+void
+finish_purge(TSCont const contp, Data *const data, TSHttpStatus const status)
+{
+ data->m_upstream.close();
+ data->m_blockstate = BlockState::Done;
+
+ // A block that could not be purged outranks the hits: 200 has to keep
meaning that
+ // the object is gone, and 404 that it was never there
+ TSHttpStatus const reply = (TS_HTTP_STATUS_NONE != status) ?
status :
+ (TS_HTTP_STATUS_NONE != data->m_purge_error) ?
data->m_purge_error :
+ (0 < data->m_purge_hits) ?
TS_HTTP_STATUS_OK :
+
TS_HTTP_STATUS_NOT_FOUND;
+
+ DEBUG_LOG("purge removed %" PRId64 " block(s), answering %d",
data->m_purge_hits, reply);
+
+ if (!data->m_dnstream.isOpen()) {
+ shutdown(contp, data);
+ return;
+ }
+
+ HdrMgr synthmgr;
+ if (!form_purge_response(synthmgr, reply)) {
+ ERROR_LOG("Failed forming the purge response");
+ shutdown(contp, data);
+ return;
+ }
+
+ HttpHeader const synth(synthmgr.m_buffer, synthmgr.m_lochdr);
+ int const hlen = synth.byteSize();
+
+ data->m_dnstream.setupVioWrite(contp, hlen);
+ TSHttpHdrPrint(synthmgr.m_buffer, synthmgr.m_lochdr,
data->m_dnstream.m_write.m_iobuf);
+ data->m_bytessent = hlen;
+ TSVIOReenable(data->m_dnstream.m_write.m_vio);
+}
+
+// A purge walks blocks instead of transferring them, so it runs its own
machine
+void
+handle_purge_resp(TSCont const contp, TSEvent const event, Data *const data)
+{
+ switch (event) {
+ case TS_EVENT_VCONN_READ_READY:
+ case TS_EVENT_VCONN_READ_COMPLETE: {
+ if (!data->m_server_block_header_parsed) {
+ int64_t consumed = 0;
+ TSIOBufferReader const reader = data->m_upstream.m_read.m_reader;
+ TSVIO const input_vio = data->m_upstream.m_read.m_vio;
+ TSParseResult const res =
data->m_resp_hdrmgr.populateFrom(data->m_http_parser, reader,
TSHttpHdrParseResp, &consumed);
+
+ TSVIONDoneSet(input_vio, TSVIONDoneGet(input_vio) + consumed);
+
+ if (TS_PARSE_CONT == res) {
+ return;
+ }
+
+ data->m_server_block_header_parsed = true;
+ note_purge_block_result(data);
+ }
+
+ // No block PURGE response has a body worth reading, but drop whatever
arrives
+ // so the upstream read cannot stall on a full buffer
+ data->m_upstream.m_read.drainReader();
+ } break;
+
+ case TS_EVENT_VCONN_EOS: {
+ if (!data->m_server_block_header_parsed) {
+ // No response at all is not evidence the block was absent
+ note_purge_failure(data, TS_HTTP_STATUS_BAD_GATEWAY);
+ DEBUG_LOG("Purge block %" PRId64 " ended with no response header",
data->m_blocknum);
+ }
+
+ // The next block cannot be requested while this one holds the upstream
+ data->m_upstream.close();
+ advance_purge(contp, data);
+ } break;
+
+ default: {
+ DEBUG_LOG("%p handle_purge_resp unhandled event: %s", data,
TSHttpEventNameLookup(event));
+ } break;
+ }
+}
+
// this is called every time the server has data for us
void
handle_server_resp(TSCont contp, TSEvent event, Data *const data)
{
+ // A purge never transfers content, so it gets its own state machine
+ if (data->is_purge()) {
+ handle_purge_resp(contp, event, data);
+ return;
+ }
+
switch (event) {
case TS_EVENT_VCONN_READ_READY: {
if (data->m_blockstate == BlockState::Passthru) {
@@ -691,10 +883,7 @@ handle_server_resp(TSCont contp, TSEvent event, Data
*const data)
// isn't keeping up
bool start_next_block = false;
- if (data->m_method_type == TS_HTTP_METHOD_PURGE) {
- // for PURGE requests, clients won't request more data (no body
content)
- start_next_block = true;
- } else if (data->m_dnstream.m_write.isOpen()) {
+ if (data->m_dnstream.m_write.isOpen()) {
// check throttle condition
TSVIO const output_vio = data->m_dnstream.m_write.m_vio;
int64_t const output_done = TSVIONDoneGet(output_vio);
diff --git a/plugins/slice/server.h b/plugins/slice/server.h
index c0d77e4803..667b854875 100644
--- a/plugins/slice/server.h
+++ b/plugins/slice/server.h
@@ -35,3 +35,14 @@
*/
void handle_server_resp(TSCont contp, TSEvent event, Data *const data);
+
+/** Walk the object's slice blocks issuing a PURGE for each.
+ *
+ * A purge transfers no content, so it runs a separate state machine: it walks
+ * every block whether or not each is cached, taking the object's extent from
the
+ * ones it removes, and answers the client only once the walk is done.
+ */
+void handle_purge_resp(TSCont contp, TSEvent event, Data *const data);
+
+// Answer a purge: 200 if any block was removed, 404 if none was, or status if
given
+void finish_purge(TSCont contp, Data *const data, TSHttpStatus const status =
TS_HTTP_STATUS_NONE);
diff --git a/plugins/slice/util.cc b/plugins/slice/util.cc
index 2f3a578629..96b6060f8b 100644
--- a/plugins/slice/util.cc
+++ b/plugins/slice/util.cc
@@ -138,7 +138,7 @@ request_block(TSCont contp, Data *const data)
}
header.removeKey(SLICE_CRR_HEADER.data(), SLICE_CRR_HEADER.size());
- if (data->m_config->m_prefetchcount > 0 && data->m_req_range.m_beg >= 0 &&
+ if (!data->is_purge() && data->m_config->m_prefetchcount > 0 &&
data->m_req_range.m_beg >= 0 &&
data->m_blocknum ==
data->m_req_range.firstBlockFor(data->m_config->m_blockbytes)) {
header.setKeyVal(SLICE_CRR_HEADER.data(), SLICE_CRR_HEADER.size(),
SLICE_CRR_VAL.data(), SLICE_CRR_VAL.size());
}
@@ -232,6 +232,12 @@ request_block(TSCont contp, Data *const data)
bool
reader_avail_more_than(TSIOBufferReader const reader, int64_t bytes)
{
+ // A purge refused before opening an upstream has no reader, and
TSIOBufferReaderStart
+ // does not tolerate a null one
+ if (nullptr == reader) {
+ return false;
+ }
+
TSIOBufferBlock block = TSIOBufferReaderStart(reader);
if (nullptr == block) {
diff --git a/src/proxy/http/HttpTransact.cc b/src/proxy/http/HttpTransact.cc
index 4aaa8b81a5..09b16decb5 100644
--- a/src/proxy/http/HttpTransact.cc
+++ b/src/proxy/http/HttpTransact.cc
@@ -7472,6 +7472,25 @@
HttpTransact::delete_all_document_alternates_and_return(State *s, bool cache_hit
build_response(s, &s->hdr_info.client_response,
s->client_info.http_version,
(cache_hit == true) ? HTTPStatus::OK :
HTTPStatus::NOT_FOUND);
+ // Report what was removed, so a caller holding one piece of a larger
resource
+ // can learn its extent without a second lookup. Not Content-Range
itself: on
+ // a 200 that is meaningless per RFC 9110, and cache_range_requests
reads the
+ // pair as a stored 206 being served as 200 and rewrites the status.
+ if (cache_hit == true && s->method == HTTP_WKSIDX_PURGE &&
s->cache_info.object_read != nullptr) {
+ // read by the slice plugin as PURGED_CONTENT_RANGE in
plugins/slice/HttpHeader.h
+ static constexpr std::string_view
PURGED_CONTENT_RANGE{"X-Purged-Content-Range"};
+ HTTPHdr *const cached_response =
s->cache_info.object_read->response_get();
+
+ if (cached_response != nullptr) {
+ auto
value{cached_response->value_get(static_cast<std::string_view>(MIME_FIELD_CONTENT_RANGE))};
+ if (!value.empty()) {
+ s->hdr_info.client_response.value_set(PURGED_CONTENT_RANGE, value);
+ TxnDbg(dbg_ctl_http_trans, "PURGE reporting
X-Purged-Content-Range: %.*s", static_cast<int>(value.length()),
+ value.data());
+ }
+ }
+ }
+
return true;
} else {
if (valid_max_forwards) {
diff --git
a/tests/gold_tests/pluginTest/slice/replay/slice_purge_gaps_client.replay.yaml
b/tests/gold_tests/pluginTest/slice/replay/slice_purge_gaps_client.replay.yaml
new file mode 100644
index 0000000000..18b768f37a
--- /dev/null
+++
b/tests/gold_tests/pluginTest/slice/replay/slice_purge_gaps_client.replay.yaml
@@ -0,0 +1,895 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Client side of slice_purge_gaps.test.py. Each transaction is one phase,
selected
+# by uuid with the verifier-client --keys option; several phases share a run
where
+# they are independent. The uuid also picks which origin transaction the block
+# requests slice derives from will match.
+#
+# Blocks are 10 bytes: block 0 is "a", 1 is "b", 2 is "c", 4 is "e". Every
proxy
+# runs --ref-relative, so a ranged GET touches only the blocks its range covers
+# and a fill phase can leave a chosen block uncached.
+#
+
+meta:
+ version: "1.0"
+
+sessions:
+- transactions:
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /hole
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, hole-fill-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /hole
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, hole-fill-2]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 20-29/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'cccccccccc'
+ verify: {as: equal}
+
+ - client-request:
+ method: PURGE
+ version: "1.1"
+ url: /hole
+ headers:
+ fields:
+ - [Host, slice]
+ - [uuid, hole-purge]
+ proxy-response:
+ status: 200
+ headers:
+ fields:
+ - [Content-Length, {value: '0', as: equal}]
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /hole
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, hole-check-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /hole
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, hole-check-2]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 20-29/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'cccccccccc'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /nofirst
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=10-19]
+ - [uuid, nofirst-fill-1]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 10-19/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'bbbbbbbbbb'
+ verify: {as: equal}
+
+ - client-request:
+ method: PURGE
+ version: "1.1"
+ url: /nofirst
+ headers:
+ fields:
+ - [Host, slice]
+ - [uuid, nofirst-purge]
+ proxy-response:
+ status: 200
+ headers:
+ fields:
+ - [Content-Length, {value: '0', as: equal}]
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /nofirst
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=10-19]
+ - [uuid, nofirst-check-1]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 10-19/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'bbbbbbbbbb'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, mixed-fill-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=10-19]
+ - [uuid, mixed-fill-1]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 10-19/50, as: equal}]
+ content:
+ encoding: plain
+ data: 'bbbbbbbbbb'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=40-49]
+ - [uuid, mixed-fill-4]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 40-49/50, as: equal}]
+ content:
+ encoding: plain
+ data: 'eeeeeeeeee'
+ verify: {as: equal}
+
+ - client-request:
+ method: PURGE
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [uuid, mixed-purge]
+ proxy-response:
+ status: 200
+ headers:
+ fields:
+ - [Content-Length, {value: '0', as: equal}]
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=40-49]
+ - [uuid, mixed-check-4]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 40-49/50, as: equal}]
+ content:
+ encoding: plain
+ data: 'eeeeeeeeee'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, mixed-check-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=10-19]
+ - [uuid, mixed-check-1]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 10-19/50, as: equal}]
+ content:
+ encoding: plain
+ data: 'bbbbbbbbbb'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /ranged
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, ranged-fill-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /ranged
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, ranged-fill-2]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 20-29/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'cccccccccc'
+ verify: {as: equal}
+
+ - client-request:
+ method: PURGE
+ version: "1.1"
+ url: /ranged
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-29]
+ - [uuid, ranged-purge]
+ proxy-response:
+ status: 200
+ headers:
+ fields:
+ - [Content-Length, {value: '0', as: equal}]
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /ranged
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, ranged-check-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /ranged
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, ranged-check-2]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 20-29/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'cccccccccc'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /openend
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, openend-fill-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /openend
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, openend-fill-2]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 20-29/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'cccccccccc'
+ verify: {as: equal}
+
+ - client-request:
+ method: PURGE
+ version: "1.1"
+ url: /openend
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-]
+ - [uuid, openend-purge]
+ proxy-response:
+ status: 200
+ headers:
+ fields:
+ - [Content-Length, {value: '0', as: equal}]
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /openend
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, openend-check-2]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 20-29/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'cccccccccc'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /openend
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, openend-check-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /outside
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=10-19]
+ - [uuid, outside-fill-1]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 10-19/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'bbbbbbbbbb'
+ verify: {as: equal}
+
+ - client-request:
+ method: PURGE
+ version: "1.1"
+ url: /outside
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, outside-purge]
+ proxy-response:
+ status: 404
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /outside
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=10-19]
+ - [uuid, outside-check-1]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 10-19/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'bbbbbbbbbb'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /endbytes
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, endbytes-fill-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /endbytes
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, endbytes-fill-2]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 20-29/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'cccccccccc'
+ verify: {as: equal}
+
+ - client-request:
+ method: PURGE
+ version: "1.1"
+ url: /endbytes
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=-10]
+ - [uuid, endbytes-purge]
+ proxy-response:
+ status: 200
+ headers:
+ fields:
+ - [Content-Length, {value: '0', as: equal}]
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /endbytes
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, endbytes-check-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /endbytes
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, endbytes-check-2]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 20-29/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'cccccccccc'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /sparse
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=40-49]
+ - [uuid, sparse-fill-4]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 40-49/50, as: equal}]
+ content:
+ encoding: plain
+ data: 'eeeeeeeeee'
+ verify: {as: equal}
+
+ - client-request:
+ method: PURGE
+ version: "1.1"
+ url: /sparse
+ headers:
+ fields:
+ - [Host, slice]
+ - [X-Slice-Purge-Probe, not-a-number]
+ - [uuid, sparse-purge-narrow]
+ proxy-response:
+ status: 404
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /sparse
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=40-49]
+ - [uuid, sparse-check-alive]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 40-49/50, as: equal}]
+ content:
+ encoding: plain
+ data: 'eeeeeeeeee'
+ verify: {as: equal}
+
+ - client-request:
+ method: PURGE
+ version: "1.1"
+ url: /sparse
+ headers:
+ fields:
+ - [Host, slice]
+ - [X-Slice-Purge-Probe, '8']
+ - [uuid, sparse-purge-wide]
+ proxy-response:
+ status: 200
+ headers:
+ fields:
+ - [Content-Length, {value: '0', as: equal}]
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /sparse
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=40-49]
+ - [uuid, sparse-check-gone]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 40-49/50, as: equal}]
+ content:
+ encoding: plain
+ data: 'eeeeeeeeee'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /badrange
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, badrange-fill-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: PURGE
+ version: "1.1"
+ url: /badrange
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=not-a-range]
+ - [uuid, badrange-purge]
+ proxy-response:
+ status: 400
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /badrange
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, badrange-check-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /failblock
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, failblock-fill-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/40, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /failblock
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=30-39]
+ - [uuid, failblock-fill-3]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 30-39/40, as: equal}]
+ content:
+ encoding: plain
+ data: 'dddddddddd'
+ verify: {as: equal}
+
+ - client-request:
+ method: PURGE
+ version: "1.1"
+ url: /failblock
+ headers:
+ fields:
+ - [Host, slice]
+ - [uuid, failblock-purge]
+ proxy-response:
+ status: 500
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /failblock
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, failblock-check-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/40, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /failblock
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=30-39]
+ - [uuid, failblock-check-3]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 30-39/40, as: equal}]
+ content:
+ encoding: plain
+ data: 'dddddddddd'
+ verify: {as: equal}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /denied
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, denied-fill-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
+
+ - client-request:
+ method: PURGE
+ version: "1.1"
+ url: /denied
+ headers:
+ fields:
+ - [Host, slice]
+ - [uuid, denied-purge]
+ proxy-response:
+ status: 403
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /denied
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, denied-check-0]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 0-9/30, as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaa'
+ verify: {as: equal}
diff --git
a/tests/gold_tests/pluginTest/slice/replay/slice_purge_gaps_server.replay.yaml
b/tests/gold_tests/pluginTest/slice/replay/slice_purge_gaps_server.replay.yaml
new file mode 100644
index 0000000000..b60525e8f8
--- /dev/null
+++
b/tests/gold_tests/pluginTest/slice/replay/slice_purge_gaps_server.replay.yaml
@@ -0,0 +1,586 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Origin side of slice_purge_gaps.test.py.
+#
+# Keyed on "{url}{field.range}{field.uuid}", so one transaction answers one
block
+# of one phase. Blocks are 10 bytes: block 0 is "a", 1 is "b", 2 is "c", 4 is
"e".
+#
+# PURGE is answered by ATS and the plugin issues no other request kind, so
neither
+# appears here.
+#
+# A check phase for a block expected to SURVIVE its purge is deliberately
absent:
+# if such a block were wrongly purged, ATS would ask for an unregistered key
and
+# the client's own expectation would fail too.
+#
+
+meta:
+ version: "1.0"
+
+sessions:
+- transactions:
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /hole
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, hole-fill-0]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-9/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'aaaaaaaaaa', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /hole
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, hole-fill-2]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 20-29/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'cccccccccc', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /hole
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, hole-check-0]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-9/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'aaaaaaaaaa', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /hole
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, hole-check-2]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 20-29/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'cccccccccc', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /nofirst
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=10-19]
+ - [uuid, nofirst-fill-1]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 10-19/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'bbbbbbbbbb', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /nofirst
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=10-19]
+ - [uuid, nofirst-check-1]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 10-19/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'bbbbbbbbbb', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, mixed-fill-0]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-9/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'aaaaaaaaaa', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=10-19]
+ - [uuid, mixed-fill-1]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 10-19/50]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'bbbbbbbbbb', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=40-49]
+ - [uuid, mixed-fill-4]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 40-49/50]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'eeeeeeeeee', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=40-49]
+ - [uuid, mixed-check-4]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 40-49/50]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'eeeeeeeeee', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /ranged
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, ranged-fill-0]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-9/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'aaaaaaaaaa', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /ranged
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, ranged-fill-2]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 20-29/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'cccccccccc', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /ranged
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, ranged-check-0]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-9/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'aaaaaaaaaa', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /ranged
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, ranged-check-2]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 20-29/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'cccccccccc', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /openend
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, openend-fill-0]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-9/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'aaaaaaaaaa', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /openend
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, openend-fill-2]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 20-29/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'cccccccccc', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /openend
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, openend-check-2]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 20-29/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'cccccccccc', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /outside
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=10-19]
+ - [uuid, outside-fill-1]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 10-19/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'bbbbbbbbbb', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /endbytes
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, endbytes-fill-0]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-9/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'aaaaaaaaaa', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /endbytes
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, endbytes-fill-2]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 20-29/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'cccccccccc', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /endbytes
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, endbytes-check-0]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-9/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'aaaaaaaaaa', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /endbytes
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=20-29]
+ - [uuid, endbytes-check-2]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 20-29/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'cccccccccc', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /sparse
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=40-49]
+ - [uuid, sparse-fill-4]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 40-49/50]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'eeeeeeeeee', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /sparse
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=40-49]
+ - [uuid, sparse-check-gone]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 40-49/50]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'eeeeeeeeee', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /badrange
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, badrange-fill-0]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-9/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'aaaaaaaaaa', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /failblock
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, failblock-fill-0]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-9/40]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'aaaaaaaaaa', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /failblock
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=30-39]
+ - [uuid, failblock-fill-3]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 30-39/40]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'dddddddddd', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /failblock
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, failblock-check-0]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-9/40]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'aaaaaaaaaa', size: 10}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /denied
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-9]
+ - [uuid, denied-fill-0]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-9/30]
+ - [Content-Length, '10']
+ - [Cache-Control, 'public, max-age=86400']
+ content: {encoding: plain, data: 'aaaaaaaaaa', size: 10}
diff --git
a/tests/gold_tests/pluginTest/slice/replay/slice_stale_generation_client.replay.yaml
b/tests/gold_tests/pluginTest/slice/replay/slice_stale_generation_client.replay.yaml
new file mode 100644
index 0000000000..b000ceda3b
--- /dev/null
+++
b/tests/gold_tests/pluginTest/slice/replay/slice_stale_generation_client.replay.yaml
@@ -0,0 +1,206 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Client side of slice_stale_generation.test.py. Each transaction is one phase,
+# selected with the verifier-client --keys option, and its uuid also selects
+# which generation the origin serves for the block requests that slice derives
+# from it.
+#
+# Generation 1 is 32 bytes of "a" with ETag "v1". Generation 2, which replaces
+# it under the same URL once the cache is filled, is 64 bytes of "b" with ETag
+# "v2".
+#
+
+meta:
+ version: "1.0"
+
+sessions:
+- transactions:
+
+ #
+ # Fill the cache while the origin holds generation 1.
+ #
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /obj
+ headers:
+ fields:
+ - [Host, slice]
+ - [uuid, fill]
+ proxy-response:
+ status: 200
+ headers:
+ fields:
+ - [Content-Length, {value: '32', as: equal}]
+ - [ETag, {value: '"v1"', as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
+ verify: {as: equal}
+
+ #
+ # The bug. Bytes 16-47 are entirely present in the current 64 byte object,
but
+ # the cached reference block still reports the old 32 byte length, so the
range
+ # is clipped to the stale object's end and served with the stale ETag as a
+ # fresh hit. Correct would be: 206, bytes 16-47/64, ETag "v2", 32 bytes of
"b".
+ #
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /obj
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=16-47]
+ - [x-debug, x-cache]
+ - [uuid, clipped]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 16-31/32, as: equal}]
+ - [Content-Length, {value: '16', as: equal}]
+ - [ETag, {value: '"v1"', as: equal}]
+ - [X-Cache, {value: hit-fresh, as: prefix}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaaaaaaaa'
+ verify: {as: equal}
+
+ #
+ # Worse: a range that starts past the stale length is refused outright, even
+ # though those bytes exist in the current object. Correct would be: 206,
+ # bytes 32-63/64.
+ #
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /obj
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=32-63]
+ - [uuid, unsatisfiable]
+ proxy-response:
+ status: 416
+ headers:
+ fields:
+ - [Content-Range, {value: '*/32', as: equal}]
+ - [ETag, {as: absent}]
+
+ #
+ # Control: a path first fetched after the object was replaced is served
+ # correctly, so the two phases above are measuring the stale cached
generation
+ # and not a broken plugin or harness.
+ #
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /fresh
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=16-47]
+ - [x-debug, x-cache]
+ - [uuid, control]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 16-47/64, as: equal}]
+ - [Content-Length, {value: '32', as: equal}]
+ - [ETag, {value: '"v2"', as: equal}]
+ content:
+ encoding: plain
+ data: 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'
+ verify: {as: equal}
+
+ #
+ # Cache an interior block of /mixed at generation 1, for
+ # SliceMixedGenerationTest. The reference block is cached with a one second
+ # lifetime and the interior block with a day, so only the reference block
+ # revalidates later.
+ #
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=16-31]
+ - [uuid, fill-interior]
+ proxy-response:
+ status: 206
+ headers:
+ fields:
+ - [Content-Range, {value: bytes 16-31/32, as: equal}]
+ - [Content-Length, {value: '16', as: equal}]
+ - [ETag, {value: '"v1"', as: equal}]
+ content:
+ encoding: plain
+ data: 'aaaaaaaaaaaaaaaa'
+ verify: {as: equal}
+
+ #
+ # The mixed generation failure. The reference block revalidates to generation
+ # 2 while the cached interior block is still generation 1, so slice
advertises
+ # the current object correctly and then cannot deliver it: the interior
block's
+ # Content-Range disagrees, the self heal refetches the interior and then the
+ # reference and gets the same blocks back, and the transaction is aborted
with
+ # the response header already on the wire. The client is left holding a well
+ # formed 206 that promises 16 bytes and delivers none.
+ #
+ # This is the production symptom behind the block walk's "curl exit 18":
+ # a correct looking Content-Range of .../7031250004 followed by a body that
+ # stops early.
+ #
+ # The client is left with nothing parsable: slice aborts the transaction, and
+ # the response header it had already formed from the reference block never
+ # reaches the wire. There is deliberately no proxy-response node here,
because
+ # no response arrives to verify. The test asserts the failure on the
+ # verifier-client's own output and in diags.log instead.
+ #
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=16-31]
+ - [x-debug, x-cache]
+ - [uuid, mixed]
+
+ # A second child with a completely cold cache, pointed at the same parent. It
+ # never saw the previous generation, yet it fails identically, because the
two
+ # blocks it fetches come from the parent and the parent is holding one of
each.
+ # This is the incident's shape: the mixed set existed in exactly one place,
the
+ # parent, and every child node inherited it. It also shows slice cannot
evict:
+ # the earlier abort left the mixed pair on the parent untouched.
+ #
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=16-31]
+ - [x-debug, x-cache]
+ - [uuid, cold-child]
diff --git
a/tests/gold_tests/pluginTest/slice/replay/slice_stale_generation_server.replay.yaml
b/tests/gold_tests/pluginTest/slice/replay/slice_stale_generation_server.replay.yaml
new file mode 100644
index 0000000000..21d778a1f9
--- /dev/null
+++
b/tests/gold_tests/pluginTest/slice/replay/slice_stale_generation_server.replay.yaml
@@ -0,0 +1,286 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# Origin side of slice_stale_generation.test.py.
+#
+# The server is keyed on "{url}{field.range}{field.uuid}", so each slice block
+# request selects a transaction by the block's byte range and by the phase uuid
+# that the client propagated through the plugin. The phase uuid is what
replaces
+# the origin object: the fill phase is answered with generation 1 (32 bytes,
+# ETag "v1"), every later phase with generation 2 (64 bytes, ETag "v2").
+#
+# For the two phases that read the already cached object there is deliberately
+# only one transaction each, the reference block. Those exist so the origin
+# really does hold the new object, but the test asserts the server never
+# receives them: the cached blocks are fresh for a day, so nothing revalidates.
+# If slice did go upstream it would either pick up generation 2, failing the
+# client side assertions, or ask for an unregistered block and get a 404,
+# failing the diags.log assertion.
+#
+
+meta:
+ version: "1.0"
+
+sessions:
+- transactions:
+
+ #
+ # Phase fill: the origin holds generation 1, cached in 16 byte blocks with a
+ # day of freshness.
+ #
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /obj
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-15]
+ - [uuid, fill]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-15/32]
+ - [Content-Length, '16']
+ - [ETag, '"v1"']
+ - [Last-Modified, 'Mon, 01 Jun 2026 00:00:00 GMT']
+ - [Cache-Control, 'public, max-age=86400']
+ - [Accept-Ranges, bytes]
+ content: {encoding: plain, data: 'aaaaaaaaaaaaaaaa', size: 16}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /obj
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=16-31]
+ - [uuid, fill]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 16-31/32]
+ - [Content-Length, '16']
+ - [ETag, '"v1"']
+ - [Last-Modified, 'Mon, 01 Jun 2026 00:00:00 GMT']
+ - [Cache-Control, 'public, max-age=86400']
+ - [Accept-Ranges, bytes]
+ content: {encoding: plain, data: 'aaaaaaaaaaaaaaaa', size: 16}
+
+ #
+ # The object is replaced here. Generation 2 is longer, with a new ETag and
+ # Last-Modified, under the same URL. The reference block is registered for
+ # each phase that reads the cached object, so the origin genuinely holds the
+ # new object, but the test asserts these keys are never requested.
+ #
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /obj
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-15]
+ - [uuid, clipped]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-15/64]
+ - [Content-Length, '16']
+ - [ETag, '"v2"']
+ - [Last-Modified, 'Wed, 08 Jul 2026 23:54:41 GMT']
+ - [Cache-Control, 'public, max-age=86400']
+ - [Accept-Ranges, bytes]
+ content: {encoding: plain, data: 'bbbbbbbbbbbbbbbb', size: 16}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /obj
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-15]
+ - [uuid, unsatisfiable]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-15/64]
+ - [Content-Length, '16']
+ - [ETag, '"v2"']
+ - [Last-Modified, 'Wed, 08 Jul 2026 23:54:41 GMT']
+ - [Cache-Control, 'public, max-age=86400']
+ - [Accept-Ranges, bytes]
+ content: {encoding: plain, data: 'bbbbbbbbbbbbbbbb', size: 16}
+
+ #
+ # Phase control: a path first requested after the object was replaced, so
+ # nothing about it is cached and the client must see generation 2.
+ #
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /fresh
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-15]
+ - [uuid, control]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-15/64]
+ - [Content-Length, '16']
+ - [ETag, '"v2"']
+ - [Last-Modified, 'Wed, 08 Jul 2026 23:54:41 GMT']
+ - [Cache-Control, 'public, max-age=86400']
+ - [Accept-Ranges, bytes]
+ content: {encoding: plain, data: 'bbbbbbbbbbbbbbbb', size: 16}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /fresh
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=16-31]
+ - [uuid, control]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 16-31/64]
+ - [Content-Length, '16']
+ - [ETag, '"v2"']
+ - [Last-Modified, 'Wed, 08 Jul 2026 23:54:41 GMT']
+ - [Cache-Control, 'public, max-age=86400']
+ - [Accept-Ranges, bytes]
+ content: {encoding: plain, data: 'bbbbbbbbbbbbbbbb', size: 16}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /fresh
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=32-47]
+ - [uuid, control]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 32-47/64]
+ - [Content-Length, '16']
+ - [ETag, '"v2"']
+ - [Last-Modified, 'Wed, 08 Jul 2026 23:54:41 GMT']
+ - [Cache-Control, 'public, max-age=86400']
+ - [Accept-Ranges, bytes]
+ content: {encoding: plain, data: 'bbbbbbbbbbbbbbbb', size: 16}
+
+ #
+ # Phase fill-interior, used by SliceMixedGenerationTest: /mixed at generation
+ # 1. The reference block is given a one second freshness lifetime so that
+ # later only it revalidates. In production the reference block was evicted
and
+ # refetched after the replacement while the interior blocks survived; a short
+ # lifetime reaches the same end state without depending on eviction.
+ #
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-15]
+ - [uuid, fill-interior]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-15/32]
+ - [Content-Length, '16']
+ - [ETag, '"v1"']
+ - [Last-Modified, 'Mon, 01 Jun 2026 00:00:00 GMT']
+ - [Cache-Control, 'public, max-age=1']
+ - [Accept-Ranges, bytes]
+ content: {encoding: plain, data: 'aaaaaaaaaaaaaaaa', size: 16}
+
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=16-31]
+ - [uuid, fill-interior]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 16-31/32]
+ - [Content-Length, '16']
+ - [ETag, '"v1"']
+ - [Last-Modified, 'Mon, 01 Jun 2026 00:00:00 GMT']
+ - [Cache-Control, 'public, max-age=86400']
+ - [Accept-Ranges, bytes]
+ content: {encoding: plain, data: 'aaaaaaaaaaaaaaaa', size: 16}
+
+ #
+ # Phase mixed: the object has been replaced. Only the reference block is
+ # stale, so only it revalidates, and it comes back as generation 2 while the
+ # cached interior block is still generation 1. That is the disagreement slice
+ # cannot heal.
+ #
+ - client-request:
+ method: GET
+ version: "1.1"
+ url: /mixed
+ headers:
+ fields:
+ - [Host, slice]
+ - [Range, bytes=0-15]
+ - [uuid, mixed]
+ server-response:
+ status: 206
+ reason: Partial Content
+ headers:
+ fields:
+ - [Content-Range, bytes 0-15/64]
+ - [Content-Length, '16']
+ - [ETag, '"v2"']
+ - [Last-Modified, 'Wed, 08 Jul 2026 23:54:41 GMT']
+ - [Cache-Control, 'public, max-age=86400']
+ - [Accept-Ranges, bytes]
+ content: {encoding: plain, data: 'bbbbbbbbbbbbbbbb', size: 16}
diff --git a/tests/gold_tests/pluginTest/slice/rules/purge_block_failure.conf
b/tests/gold_tests/pluginTest/slice/rules/purge_block_failure.conf
new file mode 100644
index 0000000000..1737d1dede
--- /dev/null
+++ b/tests/gold_tests/pluginTest/slice/rules/purge_block_failure.conf
@@ -0,0 +1,41 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Make a block PURGE fail, which the core will not do on its own.
+#
+# slice re-issues every block as its own PURGE through this same remap rule,
adding
+# Range and its skip header on the way, so a rule can single out one block or
all of
+# them without ever matching the client's own PURGE. set-status fires at the
remap
+# pseudo-hook, before any cache lookup, so it applies whether or not the block
is
+# cached and leaves a cached one where it was.
+
+# One block mid-walk, behind a block that will be removed and in front of one
that must
+# not be. Gated on the method: this object's fill and check GETs are ranged
too.
+cond %{REMAP_PSEUDO_HOOK} [AND]
+cond %{METHOD} =PURGE [AND]
+cond %{CLIENT-URL:PATH} /failblock/ [AND]
+cond %{CLIENT-HEADER:Range} ="bytes=10-19"
+ set-status 500
+
+# Every block, as ip_allow.yaml refusing PURGE would. The walk stops at the
first, so
+# only block 0 is ever asked for, and nothing is removed at all.
+cond %{REMAP_PSEUDO_HOOK} [AND]
+cond %{METHOD} =PURGE [AND]
+cond %{CLIENT-URL:PATH} /denied/ [AND]
+cond %{CLIENT-HEADER:X-Slicer-Info} ="" [NOT]
+ set-status 403
diff --git a/tests/gold_tests/pluginTest/slice/slice_purge_gaps.test.py
b/tests/gold_tests/pluginTest/slice/slice_purge_gaps.test.py
new file mode 100644
index 0000000000..a8ff70c046
--- /dev/null
+++ b/tests/gold_tests/pluginTest/slice/slice_purge_gaps.test.py
@@ -0,0 +1,346 @@
+"""Verify a PURGE traverses every slice block, not just the ones before a
gap."""
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+Test.Summary = __doc__
+
+Test.SkipUnless(
+ Condition.PluginExists('slice.so'),
+ Condition.PluginExists('cache_range_requests.so'),
+ Condition.PluginExists('header_rewrite.so'),
+)
+Test.ContinueOnFail = True
+
+
+class SlicePurgeGapsTest:
+ """Verify a PURGE removes every block it was asked for.
+
+ Slice turns one client PURGE into one PURGE per block. The walk used to
stop
+ at the first block that was not cached, because it had no other end
+ condition: m_contentlen was only ever set from a 206's Content-Range, a
PURGE
+ response carried none, and so m_req_range.blockIsInside() was true for
every
+ block number. The 404 stood in for a length the plugin never learned.
+
+ ATS now reports the removed object's extent as X-Purged-Content-Range, so
the
+ walk learns where the object ends from the blocks it is already deleting,
and
+ a 404 is merely noted and stepped over. The requested range bounds the walk
+ throughout; until some block reports an extent, an open ended one is
bounded
+ only by a limit on consecutive misses. A block response that is neither
200 nor
+ 404 is a failure rather than an absence, and says nothing about the blocks
+ behind it, so the walk stops there and reports that status.
+
+ Each object below exercises one thing, and each is purged exactly once,
+ because a purge consumes the state it is measured against.
+
+ /hole blocks 0 and 2 cached, block 1 absent. A 404 mid-walk must not
end
+ it, so block 2 goes too.
+ /nofirst only block 1 cached. The walk steps over block 0's 404, picks
the
+ extent up from block 1, and the client must not be handed that
404.
+ /mixed blocks reporting a 30 byte and a 50 byte object, as happens when
+ the origin object is replaced in place. The walk must follow the
+ largest extent reported, not the first.
+ /ranged same gap as /hole, purged with a closed range. Already bounded,
so
+ this isolates the 404 handling from the extent discovery.
+ /openend purged with "bytes=20-". The start is stated, so only block 2
may
+ go and block 0 must survive.
+ /outside only block 1 cached, purged for block 0 alone. No block is
removed,
+ so no extent is ever reported and only the range can end the
walk
+ before it reaches a block the client did not name.
+ /endbytes purged with "bytes=-10". A suffix range cannot know which block
it
+ starts at, so it is widened to the whole object and every block
+ goes.
+ /sparse only block 4 cached, on a proxy whose miss bound is 2, so the
+ default walk cannot reach it. Covers the bound, its per-request
+ override, and the fallback when the override is malformed.
+ /badrange purged with an unparseable range, which must be refused rather
+ than silently applied to block 0.
+ /failblock blocks 0 and 3 cached, with a 500 injected for block 1's PURGE.
Block
+ 0 is removed, the walk stops at block 1, block 3 is left alone,
and
+ the client hears the 500 rather than the 200 block 0 earned.
+ /denied block 0 cached, with a 403 injected for every block PURGE, as
+ ip_allow refusing PURGE would. The walk stops at its first
block, so
+ nothing is removed and the client must not be told 404.
+
+ Whether a block was purged is measured on the origin, not on the response
+ body: the origin serves each check phase exactly what the matching fill
phase
+ served, so a purged block and a surviving block are indistinguishable to
the
+ client, and the only difference is whether the origin was asked again.
+ """
+
+ _client_replay: str = 'replay/slice_purge_gaps_client.replay.yaml'
+ _server_replay: str = 'replay/slice_purge_gaps_server.replay.yaml'
+
+ # The core answers a block PURGE with nothing but 200 or 404, so a failure
has to
+ # be injected to be tested at all.
+ _fail_rules: str = 'purge_block_failure.conf'
+
+ _block_bytes: int = 10
+
+ # Low enough that the default walk cannot reach /sparse's block 4, which is
+ # what makes the per-request override observable.
+ _low_miss_bound: int = 2
+
+ # Keyed on the block range as well as the phase uuid, so one origin
+ # transaction answers one block of one phase.
+ _origin_key_format: str = '--format "{url}{field.range}{field.uuid}"'
+
+ def __init__(self) -> None:
+ """Declare the origin and the two proxies."""
+ self._started = False
+ self._configure_origin()
+ self._ts = self._make_ts('ts', fail_rules=True)
+ self._ts_bound = self._make_ts('ts-bound',
miss_bound=self._low_miss_bound)
+
+ self._ts.Disk.traffic_out.Content = Testers.ContainsExpression(
+ 'Purge suffix range widened to the whole object',
+ 'A suffix range purge should be widened rather than guessing at
its start.')
+
+ self._ts_bound.Disk.traffic_out.Content = Testers.ContainsExpression(
+ f'gave up after {self._low_miss_bound} consecutive uncached block',
+ 'The walk should stop at its configured miss bound rather than
scanning the whole range.')
+ self._ts_bound.Disk.diags_log.Content = Testers.ContainsExpression(
+ 'Ignoring invalid X-Slice-Purge-Probe', 'A malformed override
should be rejected, not acted on.')
+
+ # A purge issues nothing but block PURGEs. request_block logs every
request
+ # header it builds at debug, so an only-if-cached here would mean a
+ # read-only length probe had been reintroduced.
+ for ts in (self._ts, self._ts_bound):
+ ts.Disk.traffic_out.Content += Testers.ExcludesExpression(
+ 'only-if-cached', 'A purge should not issue a read-only length
probe.')
+
+ def _configure_origin(self) -> None:
+ """Configure the origin."""
+ self._origin = Test.MakeVerifierServerProcess('origin',
self._server_replay, other_args=self._origin_key_format)
+
+ # ATS answers PURGE itself and the plugin issues no other request
kind, so
+ # neither may ever be seen upstream.
+ self._origin.Streams.stdout += Testers.ExcludesExpression(
+ 'PURGE', 'A PURGE should be answered by ATS and never forwarded to
the origin.')
+ self._origin.Streams.stdout += Testers.ExcludesExpression('HEAD /', 'A
purge should never issue a HEAD upstream.')
+
+ def _make_ts(self, label: str, miss_bound: int = None, fail_rules: bool =
False) -> 'Process':
+ """Create a proxy that slices in front of cache_range_requests.
+
+ --ref-relative keeps a ranged GET from dragging block 0 in as a
reference
+ block, which is what lets a fill phase leave a chosen block uncached.
+
+ :param label: process name suffix.
+ :param miss_bound: --purge-probe-blocks value, or None for the default.
+ :param fail_rules: load the header_rewrite rules that fail a block
PURGE.
+ """
+ ts = Test.MakeATSProcess(label, enable_cache=True)
+
+ rules = ''
+ if fail_rules:
+ ts.Setup.CopyAs(f'rules/{self._fail_rules}', Test.RunDirectory)
+ rules = f' @plugin=header_rewrite.so
@pparam={Test.RunDirectory}/{self._fail_rules}'
+
+ bound = '' if miss_bound is None else f'
@pparam=--purge-probe-blocks={miss_bound}'
+ ts.Disk.remap_config.AddLine(
+ f'map http://slice/
http://127.0.0.1:{self._origin.Variables.http_port}/'
+ f'{rules} @plugin=slice.so
@pparam=--blockbytes-test={self._block_bytes} @pparam=--ref-relative{bound}'
+ ' @plugin=cache_range_requests.so')
+ ts.Disk.records_config.update(
+ {
+ 'proxy.config.diags.debug.enabled': 1,
+ 'proxy.config.diags.debug.tags': 'slice|cache_range_requests',
+ })
+ return ts
+
+ def _run(self, summary: str, phases: str, ts: 'Process' = None) -> None:
+ """Add a TestRun replaying one or more phases, in replay file order.
+
+ :param summary: TestRun description.
+ :param phases: space separated verifier-client keys.
+ :param ts: proxy to replay against, defaulting to the ordinary one.
+ """
+ ts = self._ts if ts is None else ts
+ tr = Test.AddTestRun(summary)
+ if not self._started:
+ tr.Processes.Default.StartBefore(self._origin)
+ tr.Processes.Default.StartBefore(self._ts)
+ tr.Processes.Default.StartBefore(self._ts_bound)
+ self._started = True
+ tr.AddVerifierClientProcess(
+ f"client-{phases.replace(' ', '-')}", self._client_replay,
http_ports=[ts.Variables.port], keys=phases)
+ tr.StillRunningAfter = ts
+
+ def _origin_saw(self, phase: str, block: str, why: str) -> None:
+ """Assert the origin was asked for a block under a given phase."""
+ self._origin.Streams.stdout += Testers.ContainsExpression(f'request
with key /{block}{phase}', why)
+
+ def _fill(self, summary: str, url: str, blocks: list, ts: 'Process' =
None) -> None:
+ """Cache the given blocks of an object, and prove each fill happened.
+
+ The fill assertions carry weight: a check phase only observes that the
+ origin was asked, which is equally true of a block that was purged and
one
+ that was never cached at all. Asserting the fill reached the origin is
what
+ makes the later check mean "removed" rather than merely "absent".
+ """
+ self._run(summary, ' '.join(f'{url}-fill-{block}' for block in
blocks), ts)
+ for block in blocks:
+ self._origin_saw(
+ f'{url}-fill-{block}', f'{url}bytes={block * 10}-{block * 10 +
9}',
+ f'Block {block} of /{url} should have been fetched and
cached.')
+
+ def _purged(self, phase: str, block: str, why: str) -> None:
+ """Assert a phase's block request reached the origin, so it was
purged."""
+ self._origin_saw(phase, block, why)
+
+ def _survived(self, phase: str, block: str, why: str) -> None:
+ """Assert a phase's block request never reached the origin, so it
survived.
+
+ The origin has no transaction registered for such a phase either, so a
+ wrongly purged block fails the client's own expectation as well.
+ """
+ self._origin.Streams.stdout += Testers.ExcludesExpression(f'request
with key /{block}{phase}', why)
+
+ def _gap_mid_walk(self) -> None:
+ """A 404 in the middle of the walk must not end it."""
+ self._fill('Cache blocks 0 and 2 of /hole, leaving block 1 uncached',
'hole', [0, 2])
+ self._run('PURGE the whole /hole object', 'hole-purge')
+ self._run('Both cached blocks of /hole were purged', 'hole-check-0
hole-check-2')
+ self._purged('hole-check-0', 'holebytes=0-9', 'Block 0 is in front of
the gap, so it should be purged.')
+ self._purged('hole-check-2', 'holebytes=20-29', 'A PURGE should
traverse blocks behind an uncached one.')
+
+ def _uncached_first_block(self) -> None:
+ """A miss on the first block must not stop the purge before it
starts."""
+ self._fill('Cache only block 1 of /nofirst', 'nofirst', [1])
+ self._run('PURGE /nofirst, whose block 0 is not cached',
'nofirst-purge')
+ self._run('Block 1 of /nofirst was purged', 'nofirst-check-1')
+ self._purged('nofirst-check-1', 'nofirstbytes=10-19', 'An uncached
first block should not stop the purge.')
+
+ def _largest_extent_wins(self) -> None:
+ """A block reporting a longer object widens the walk."""
+ self._fill('Cache blocks 0, 1 and 4 of /mixed, disagreeing about its
length', 'mixed', [0, 1, 4])
+ self._run('PURGE /mixed', 'mixed-purge')
+ self._run('Block 4 of /mixed was purged, so the walk took the longer
extent', 'mixed-check-4')
+ self._purged('mixed-check-4', 'mixedbytes=40-49', 'The walk should
follow the largest extent any block reports.')
+
+ def _closed_range(self) -> None:
+ """A closed range bounds the walk itself, isolating the 404
step-over."""
+ self._fill('Cache blocks 0 and 2 of /ranged, leaving block 1
uncached', 'ranged', [0, 2])
+ self._run('PURGE /ranged with a closed range spanning the whole
object', 'ranged-purge')
+ self._run('Both cached blocks of /ranged were purged', 'ranged-check-0
ranged-check-2')
+ self._purged('ranged-check-0', 'rangedbytes=0-9', 'A closed range
purge should remove the blocks it covers.')
+ self._purged('ranged-check-2', 'rangedbytes=20-29', 'A 404 should not
end a closed range purge either.')
+
+ def _open_ended_range(self) -> None:
+ """A "bytes=N-" purge states its start, so it purges only what it
names."""
+ self._fill('Cache blocks 0 and 2 of /openend', 'openend', [0, 2])
+ self._run('PURGE /openend from byte 20 on', 'openend-purge')
+ self._run('Block 2 of /openend went and block 0 stayed',
'openend-check-2 openend-check-0')
+ self._purged('openend-check-2', 'openendbytes=20-29', 'The block
covering the range should be purged.')
+ self._survived('openend-check-0', 'openendbytes=0-9', 'A purge should
not remove blocks before its stated start.')
+
+ def _range_bounds_the_walk(self) -> None:
+ """A range ends the walk even before any block has reported an extent.
+
+ /outside has only block 1 cached and is purged for block 0 alone.
Nothing
+ is ever removed, so no block reports an extent, and the miss bound
would
+ otherwise carry the walk past the end of the named range and into a
block
+ the client never asked to remove.
+ """
+ self._fill('Cache only block 1 of /outside', 'outside', [1])
+ self._run('PURGE /outside for block 0 only, which is not cached',
'outside-purge')
+ self._run('Block 1 of /outside survived a purge that did not name it',
'outside-check-1')
+ self._survived(
+ 'outside-check-1', 'outsidebytes=10-19', 'A purge should stop at
the end of its range, not at the miss bound.')
+
+ def _suffix_range(self) -> None:
+ """A "bytes=-N" purge is widened to the whole object."""
+ self._fill('Cache blocks 0 and 2 of /endbytes', 'endbytes', [0, 2])
+ self._run('PURGE the last 10 bytes of /endbytes', 'endbytes-purge')
+ self._run('Every cached block of /endbytes went, not just the named
tail', 'endbytes-check-0 endbytes-check-2')
+ self._purged('endbytes-check-2', 'endbytesbytes=20-29', 'The block
covering the suffix range must be purged.')
+ self._purged(
+ 'endbytes-check-0', 'endbytesbytes=0-9',
+ 'A widened suffix purge removes the whole object, which is a
superset of what was named.')
+
+ def _miss_bound_and_override(self) -> None:
+ """The miss bound stops a walk that has found nothing, and is
overridable.
+
+ /sparse has only block 4 of five cached, out of reach of this proxy's
+ configured bound of two. Nothing about the remap changes between the
two
+ purges below; only the request header does.
+ """
+ ts = self._ts_bound
+ self._fill('Cache only block 4 of /sparse, out of reach of the
configured bound', 'sparse', [4], ts)
+
+ self._run('A purge with a malformed override falls back to the
configured bound', 'sparse-purge-narrow', ts)
+ self._run('Block 4 of /sparse survived the too-narrow purge',
'sparse-check-alive', ts)
+ self._survived('sparse-check-alive', 'sparsebytes=40-49', 'A walk that
gave up before block 4 should not have purged it.')
+
+ self._run('PURGE /sparse with an override wide enough to reach block
4', 'sparse-purge-wide', ts)
+ self._run('Block 4 of /sparse was purged once the bound reached it',
'sparse-check-gone', ts)
+ self._purged('sparse-check-gone', 'sparsebytes=40-49', 'A request
supplied bound should let the walk reach block 4.')
+
+ def _unparseable_range(self) -> None:
+ """A purge whose range cannot be parsed is refused, not guessed at.
+
+ An unparseable range leaves the plugin's range covering block 0 only,
so
+ walking it would delete the head of the object and report success. A
purge
+ is destructive, so it is rejected instead.
+ """
+ self._fill('Cache block 0 of /badrange', 'badrange', [0])
+ self._run('A PURGE with an unparseable range is refused',
'badrange-purge')
+ self._run('Block 0 of /badrange survived the refused purge',
'badrange-check-0')
+ self._survived('badrange-check-0', 'badrangebytes=0-9', 'A refused
purge must not have removed anything.')
+ self._ts.Disk.diags_log.Content = Testers.ContainsExpression(
+ 'Refusing PURGE with an unparseable range', 'The refusal should be
visible in the error log.')
+
+ def _block_failure(self) -> None:
+ """A block that could not be purged is not a block that was absent.
+
+ Only a 404 says the block was not cached. Any other status says the
walk could
+ not tell, and says nothing about the blocks behind it either, so it
stops there
+ and the client hears that status rather than the 200 the earlier
blocks earned.
+ Both failures are injected with header_rewrite, since the core answers
a block
+ PURGE with nothing but 200 or 404 on its own.
+ """
+ self._fill('Cache blocks 0 and 3 of /failblock', 'failblock', [0, 3])
+ self._run('PURGE /failblock, whose block 1 answers 500',
'failblock-purge')
+ self._run('/failblock was purged up to the failing block only',
'failblock-check-0 failblock-check-3')
+ self._purged('failblock-check-0', 'failblockbytes=0-9', 'A block
removed before the failure should stay removed.')
+ self._survived(
+ 'failblock-check-3', 'failblockbytes=30-39',
+ 'The walk should stop at the failing block, leaving what is behind
it cached.')
+ self._ts.Disk.diags_log.Content += Testers.ContainsExpression(
+ 'Purge of block 1 failed', 'The block that ended the walk should
be logged.')
+
+ self._fill('Cache block 0 of /denied', 'denied', [0])
+ self._run('PURGE /denied, whose every block answers 403',
'denied-purge')
+ self._run('Block 0 of /denied survived the refused purge',
'denied-check-0')
+ self._survived('denied-check-0', 'deniedbytes=0-9', 'A purge refused
at its first block must not remove anything.')
+ self._ts.Disk.diags_log.Content += Testers.ContainsExpression(
+ 'Purge of block 0 failed', 'A purge refused outright should be
logged rather than reported as a 404.')
+
+ def run(self) -> None:
+ """Configure the test runs."""
+ self._gap_mid_walk()
+ self._uncached_first_block()
+ self._largest_extent_wins()
+ self._closed_range()
+ self._open_ended_range()
+ self._range_bounds_the_walk()
+ self._suffix_range()
+ self._unparseable_range()
+ self._miss_bound_and_override()
+ self._block_failure()
+
+
+SlicePurgeGapsTest().run()
diff --git a/tests/gold_tests/pluginTest/slice/slice_stale_generation.test.py
b/tests/gold_tests/pluginTest/slice/slice_stale_generation.test.py
new file mode 100644
index 0000000000..0538127939
--- /dev/null
+++ b/tests/gold_tests/pluginTest/slice/slice_stale_generation.test.py
@@ -0,0 +1,327 @@
+"""Verify slice serves a stale object identity after the origin object
changes."""
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+Test.Summary = __doc__
+
+Test.SkipUnless(
+ Condition.PluginExists('slice.so'),
+ Condition.PluginExists('cache_range_requests.so'),
+ Condition.PluginExists('xdebug.so'),
+)
+Test.ContinueOnFail = False
+
+
+class SliceHierarchyTest:
+ """Build the child/parent hierarchy the incident ran on.
+
+ Both tiers load slice with the same block size, and both put
+ cache_range_requests behind it, as the affected property does.
+
+ The parent only behaves as a slicing proxy for requests that arrive without
+ slice's skip header. The child stamps that header onto every block request
it
+ issues (client.cc:66), so slice returns immediately on the parent
+ (slice.cc:48) and a child block request is handled there by
+ cache_range_requests alone: look up this exact Range, forward it on a miss,
+ store whatever 206 comes back. The parent therefore holds N independent
+ per-Range objects with no shared identity, and cannot notice, refuse or
+ reconcile a version mix. That is where the mixed set lived in the incident.
+
+ A client hitting the parent directly is a different path: the parent does
+ slice that request, forms its own reference block and clamps against it.
+ """
+
+ _server_replay: str = 'replay/slice_stale_generation_server.replay.yaml'
+ _client_replay: str = 'replay/slice_stale_generation_client.replay.yaml'
+
+ _block_bytes: int = 16
+
+ _origin_key_format: str = '--format "{url}{field.range}{field.uuid}"'
+
+ def __init__(self, name: str) -> None:
+ """Declare the origin, the parent and the child.
+
+ :param name: suffix distinguishing this hierarchy's processes.
+ """
+ self._name = name
+ self._configure_dns()
+ self._configure_origin()
+ self._configure_parent()
+ self._configure_child()
+
+ def _configure_dns(self) -> None:
+ """Configure a DNS server so neither tier consults resolv.conf."""
+ self._dns = Test.MakeDNServer(f'dns-{self._name}', default='127.0.0.1')
+
+ def _configure_origin(self) -> None:
+ """Configure the origin.
+
+ The server is keyed on the block's byte range and on the phase uuid
that
+ slice propagates from the client request, so one replay file answers
+ every block request of every phase and can replace the object between
+ phases without holding any state.
+ """
+ self._origin = Test.MakeVerifierServerProcess(
+ f'origin-{self._name}', self._server_replay,
other_args=self._origin_key_format)
+
+ def _slice_remap(self, source: str, upstream: str) -> str:
+ """Build a remap rule carrying slice in front of
cache_range_requests."""
+ return (
+ f'map {source} {upstream}'
+ f' @plugin=slice.so @pparam=--blockbytes-test={self._block_bytes}'
+ ' @plugin=cache_range_requests.so')
+
+ def _records(self, ts: 'Process', debug: int) -> None:
+ """Apply the records.yaml settings common to both tiers."""
+ ts.Disk.records_config.update(
+ {
+ 'proxy.config.diags.debug.enabled': debug,
+ 'proxy.config.diags.debug.tags': 'slice|cache_range_requests',
+ 'proxy.config.dns.nameservers':
f'127.0.0.1:{self._dns.Variables.Port}',
+ 'proxy.config.dns.resolv_conf': 'NULL',
+ 'proxy.config.http.parent_proxy.self_detect': 0,
+ })
+
+ def _configure_parent(self) -> None:
+ """Configure the parent, which the child's block requests reach."""
+ self._parent = Test.MakeATSProcess(f'ts-parent-{self._name}')
+ self._parent.Disk.remap_config.AddLine(
+ self._slice_remap('http://origin.test/',
f'http://127.0.0.1:{self._origin.Variables.http_port}/'))
+ self._parent.Disk.plugin_config.AddLine('xdebug.so --enable=x-cache')
+ self._records(self._parent, debug=1)
+
+ # slice is loaded on the parent, but every block request the child
sends
+ # carries the skip header, so slice returns immediately and the request
+ # is handled by cache_range_requests alone. Plugin debug output lands
in
+ # traffic.out, not diags.log.
+ self._parent.Disk.traffic_out.Content = Testers.ContainsExpression(
+ 'slice passing GET or HEAD request through to next plugin',
+ "The child's block requests should bypass the parent's slice.")
+ self._parent.Disk.traffic_out.Content += Testers.ExcludesExpression(
+ 'slice accepting and slicing', 'The parent should never slice a
child block request.')
+
+ def _make_child(self, label: str) -> 'Process':
+ """Create a child tier that slices and forwards to the parent."""
+ ts = Test.MakeATSProcess(f'ts-{label}-{self._name}')
+ ts.Disk.remap_config.AddLine(self._slice_remap('http://slice/',
'http://origin.test/'))
+ ts.Disk.parent_config.AddLine(
+ f'dest_domain=. parent=127.0.0.1:{self._parent.Variables.port}'
+ ' round_robin=consistent_hash go_direct=false')
+ ts.Disk.plugin_config.AddLine('xdebug.so --enable=x-cache')
+ self._records(ts, debug=1)
+ return ts
+
+ def _configure_child(self) -> None:
+ """Configure the child, which slices and forwards to the parent."""
+ self._child = self._make_child('child')
+
+ def _start_hierarchy(self, tr: 'TestRun') -> None:
+ """Bring up origin, parent and child for the first TestRun."""
+ tr.Processes.Default.StartBefore(self._dns)
+ tr.Processes.Default.StartBefore(self._origin)
+ tr.Processes.Default.StartBefore(self._parent)
+ tr.Processes.Default.StartBefore(self._child)
+
+ def _replay_phase(self, tr: 'TestRun', phase: str, ts: 'Process' = None)
-> 'Process':
+ """Replay the client transaction for one phase against a child."""
+ ts = self._child if ts is None else ts
+ return tr.AddVerifierClientProcess(
+ f'client-{phase}-{self._name}', self._client_replay,
http_ports=[ts.Variables.port], keys=phase)
+
+ def _still_running(self, tr: 'TestRun') -> None:
+ """Assert both tiers survive the TestRun."""
+ tr.StillRunningAfter = self._child
+ tr.StillRunningAfter = self._parent
+
+
+class SliceStaleGenerationTest(SliceHierarchyTest):
+ """Verify a cached reference block pins a stale object identity.
+
+ The plugin takes the whole object length from the reference block's
+ Content-Range and clips the client range to it, so the cached reference
+ block, not the origin, defines the object's identity for every request::
+
+ server.cc handleFirstServerHeader:
+ data->m_contentlen = blockcr.m_length;
+ data->m_req_range.m_end = std::min(data->m_contentlen,
data->m_req_range.m_end);
+
+ Replacing the object under the same URL therefore forks every cache into
one
+ that filled before the replacement and one that filled after, for as long
as
+ the reference block stays fresh. On the stale side, ranges that exist in
the
+ current object are answered against the stale length with the stale ETag:
+ clipped short, or refused with a 416. Neither path logs a block stitch
error,
+ because handleNextServerHeader only complains when blocks disagree with
each
+ other and here they are uniformly stale.
+
+ Modelled on an incident where a versioned, year-cacheable object was
replaced
+ in place. Two edges seven hours apart on either side of the replacement
served
+ object lengths 4043309056 and 7031250004 for the same URL, the stale one
+ clipping a 64 MiB range request down to 16 MiB.
+ """
+
+ # The reference block the origin holds at the new generation for each phase
+ # that reads the cached object. The test asserts the origin never gets
these.
+ _unreachable_keys = ('/objbytes=0-15clipped',
'/objbytes=0-15unsatisfiable')
+
+ def __init__(self) -> None:
+ """Declare the hierarchy and its assertions."""
+ super().__init__('stale')
+
+ for key in self._unreachable_keys:
+ self._origin.Streams.stdout += Testers.ExcludesExpression(
+ f'request with key {key}', 'The stale object should never be
refetched after the origin object changed.')
+
+ # Debug is enabled on the child, so Config::canLogError cannot
suppress a
+ # block stitch error by pacing. The stale response is served with none.
+ self._child.Disk.diags_log.Content = Testers.ExcludesExpression(
+ 'logSliceError', 'The stale response should be served with no
block stitch error.')
+ self._child.Disk.diags_log.Content += Testers.ExcludesExpression(
+ 'Mismatch/Bad block Content-Range', 'The stale blocks agree with
each other, so nothing should mismatch.')
+
+ def _fill_cache(self) -> None:
+ """Cache the whole object while the origin holds the first
generation."""
+ tr = Test.AddTestRun('Cache the object while the origin holds the
first generation')
+ self._start_hierarchy(tr)
+ self._replay_phase(tr, 'fill')
+ self._still_running(tr)
+
+ def _verify_clipped_range(self) -> None:
+ """A range inside the current object is clipped to the stale length."""
+ tr = Test.AddTestRun('A range inside the current object is clipped to
the stale length')
+ self._replay_phase(tr, 'clipped')
+ self._still_running(tr)
+
+ def _verify_unsatisfiable_range(self) -> None:
+ """A range past the stale length is refused with a 416."""
+ tr = Test.AddTestRun('A range past the stale length is refused with a
416')
+ self._replay_phase(tr, 'unsatisfiable')
+ self._still_running(tr)
+
+ def _verify_uncached_object(self) -> None:
+ """An object first fetched after the replacement is served
correctly."""
+ tr = Test.AddTestRun('An object first fetched after the replacement is
served correctly')
+ self._replay_phase(tr, 'control')
+ self._still_running(tr)
+
+ def run(self) -> None:
+ """Configure the test runs."""
+ self._fill_cache()
+ self._verify_clipped_range()
+ self._verify_unsatisfiable_range()
+ self._verify_uncached_object()
+
+
+class SliceMixedGenerationTest(SliceHierarchyTest):
+ """Verify the parent stores a version mix and the child cannot recover.
+
+ The other failure mode from the same origin object replacement, and the one
+ the parent's per-Range cache makes possible. Only the reference block is
+ refetched after the replacement, so the parent ends up holding two blocks
of
+ one object at two different generations, both fresh, with nothing to relate
+ them. It serves each on request without complaint.
+
+ The child is the only tier that compares blocks, and only against its own
+ reference block. It forms the client response header from the reference
block,
+ which is correct for the current object, and only then discovers that the
+ interior block belongs to the previous one::
+
+ server.cc handleNextServerHeader:
+ if (!blockcr.isValid() || blockcr.m_length != data->m_contentlen) {
+ logSliceError("Mismatch/Bad block Content-Range", data, header);
+
+ The self heal refetches the reference block, which is already the newest
one
+ the parent holds, so the same block comes back and the interior block still
+ disagrees with it. The second mismatch is where slice gives up.
+
+ The upstream is aborted. Slice can abort but cannot evict, so the mixed
pair
+ on the parent survives. The final TestRun proves where the damage actually
+ lives: a second child with a completely cold cache, pointed at the same
+ parent, fails identically. It never saw the previous generation; it simply
+ inherits the mix from the one place that holds it. That is the incident's
+ shape, where all 105 blocks hashed to a single parent and every one of the
32
+ child nodes served the same broken object.
+ """
+
+ # The reference block is cached with a one second lifetime, so let it
expire.
+ _expiry_wait: int = 2
+
+ def __init__(self) -> None:
+ """Declare the hierarchy and its assertions."""
+ super().__init__('mixed')
+ self._cold_child = self._make_child('cold-child')
+
+ # Unlike the stale case, the child does report this one: first the
interior
+ # block against the reference block, then the refetch against the
interior.
+ self._child.Disk.diags_log.Content = Testers.ContainsExpression(
+ 'Mismatch/Bad block
Content-Range.*blk_range="16-31".*etag_got="%22v1%22"',
+ 'The interior block should disagree with the reference block.')
+ self._child.Disk.diags_log.Content += Testers.ContainsExpression(
+ 'Mismatch/Bad block
Content-Range.*blk_range="0-15".*etag_got="%22v2%22"',
+ 'The refetched reference block should disagree in turn, leaving no
way out.')
+
+ # The parent never compares blocks, so it never complains about the mix
+ # it is storing and serving to the child.
+ self._parent.Disk.diags_log.Content += Testers.ExcludesExpression(
+ 'logSliceError', 'The parent should not notice the version mix it
holds.')
+ self._parent.Disk.diags_log.Content += Testers.ExcludesExpression(
+ 'Mismatch/Bad block Content-Range', 'The parent should not compare
blocks at all.')
+
+ def _fill_interior_block(self) -> None:
+ """Cache an interior block at the first generation, on both tiers."""
+ tr = Test.AddTestRun('Cache an interior block at the first generation')
+ self._start_hierarchy(tr)
+ self._replay_phase(tr, 'fill-interior')
+ self._still_running(tr)
+
+ def _verify_aborted_response(self) -> None:
+ """The reference block moves on and the transaction cannot be
completed."""
+ tr = Test.AddTestRun('A mixed generation object cannot be delivered
and the request fails')
+ client = self._replay_phase(tr, 'mixed')
+ # Let the reference block go stale so that only it is revalidated.
+ tr.Processes.Default.Command = f'sleep {self._expiry_wait}; ' +
tr.Processes.Default.Command
+ # Slice aborts the transaction, so the client never reads a response at
+ # all: not a short body, no response header. verifier-client exits 1.
+ client.ReturnCode = 1
+ client.Streams.stdout += Testers.ContainsExpression(
+ 'Failed to find a well-formed, completed HTTP response:
PARSE_INCOMPLETE',
+ 'The client should not receive a parsable response.')
+ client.Streams.stdout += Testers.ContainsExpression(
+ 'Failed HTTP/1 transaction with key: mixed', 'The transaction
should fail.')
+ self._still_running(tr)
+
+ def _verify_mix_is_on_the_parent(self) -> None:
+ """A cold child fails identically, because the mix lives on the
parent."""
+ tr = Test.AddTestRun('A second child with a cold cache inherits the
mix from the parent')
+ tr.Processes.Default.StartBefore(self._cold_child)
+ client = self._replay_phase(tr, 'cold-child', ts=self._cold_child)
+ client.ReturnCode = 1
+ client.Streams.stdout += Testers.ContainsExpression(
+ 'Failed HTTP/1 transaction with key: cold-child', 'A node that
never saw the old generation should fail the same way.')
+ self._cold_child.Disk.diags_log.Content = Testers.ContainsExpression(
+ 'Mismatch/Bad block Content-Range', 'The cold child should hit the
same mismatch.')
+ tr.StillRunningAfter = self._cold_child
+ self._still_running(tr)
+
+ def run(self) -> None:
+ """Configure the test runs."""
+ self._fill_interior_block()
+ self._verify_aborted_response()
+ self._verify_mix_is_on_the_parent()
+
+
+SliceStaleGenerationTest().run()
+SliceMixedGenerationTest().run()
