Andrey Yarovoy created HDDS-16943:
-------------------------------------
Summary: Avoid the extra ListStatus RPC at the end of every file
system listing
Key: HDDS-16943
URL: https://issues.apache.org/jira/browse/HDDS-16943
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Manager
Reporter: Andrey Yarovoy
h2. Problem
The file system clients keep listing until OM returns a page that adds no new
entry:
{code:java}
// BasicRootedOzoneFileSystem.listStatusAdapter (same loop in
BasicOzoneFileSystem)
do {
tmpStatusList = adapter.listStatus(pathToKey(f), false, startPath,
numEntries, ...);
// add entries, dropping a repeated first entry
...
} while (entriesAdded > 0);
{code}
So every \{{FileSystem.listStatus}} costs at least two OM listing RPCs, even
when the whole directory fits in the
first page. The second call returns only the repeated start key (OM includes
\{{startKey}} in the result) and adds
nothing. The \{{listStatusIterator}} path (\{{hasNextNoFilter}}) has the same
rule: it fetches another page whenever
the current one was not empty.
In a benchmark of 2000 \{{ofs}} \{{listStatus}} calls on directories of 10
entries (\{{TestOfsListStatusRpcBenchmark}},
HDDS-16760), every call made exactly two ListStatusLight RPCs. One of them is
not needed.
h2. Why the client cannot stop on a short page
Before HDDS-11696 the loop stopped when a page was shorter than
\{{numEntries}}. HDDS-11696 changed it to stop on an
empty page, because OM caps the page size on its own:
* \{{OzoneManagerRequestHandler.limitListSize}}: \{{min(numEntries,
ozone.om.server.list.max.size)}} (default *1000*);
* \{{OmMetadataReader.listStatus}} / \{{listStatusLight}}: \{{min(n,
ozone.fs.listing.page.size.max)}} (default 1024),
raised to at least 2 by \{{OzoneConfigUtil.limitValue}}.
The client asks for \{{ozone.fs.listing.page.size}} (default *1024*), so with
default settings a full page of 1000
entries is shorter than the request. The client cannot know OM's caps, which
are server configuration, so a
client-only "stop on a short page" rule would silently cut off any directory
with more than 1000 entries. This
must not be reintroduced.
h2. Proposal
Let OM say whether the listing may continue, as \{{ListKeysResponse}} /
\{{ListKeysLightResponse}} already do with
{\{isTruncated}}.
# *Proto.* Add \{{optional bool isTruncated}} to \{{ListStatusResponse}} and
\{{ListStatusLightResponse}}.
# *OM.* Set it to \{{returnedCount >= effectiveLimit}}, where
\{{effectiveLimit}} is the page size OM actually used
after both caps above. Compare with that value, not with the requested
\{{numEntries}}.
Both listing paths return \{{min(effectiveLimit, remaining live entries)}}:
** FSO, \{{OzoneListStatusHelper.listStatusFSO}}: a single loop {{while
(map.size() < numEntries &&
heapIterator.hasNext())}}; cache tombstones and DB keys shadowed by the
cache never reach the heap, so they do
not use up the limit.
** LEGACY/OBS, \{{KeyManagerImpl.listStatus}}: entries are counted only when a
live status is added, and the final
loop stops at \{{numEntries}}.
So \{{isTruncated = false}} always means the listing is complete. When
exactly \{{effectiveLimit}} entries remain,
the flag is \{{true}} and the client makes one more call, as it does today. A
look-ahead
(\{{heapIterator.hasNext()}} on FSO, remaining map/iterator entries on
LEGACY) could avoid that last call, but it is
not needed for correctness.
# *Client.* Stop after a page whose response has \{{isTruncated}} set to
\{{false}}. If the field is absent (an older
OM), keep the current loop. Field presence distinguishes the two, so no
\{{OzoneManagerVersion}} gate is needed.
# *Plumbing.* The listing calls return a plain \{{List}} today through
\{{OzoneManagerProtocol}},
\{{ClientProtocol}} / \{{RpcClient}}, \{{OzoneClientAdapter}} and the two
file systems. The flag needs a way up,
for example a small result type holding the entries and the flag, used by the
file system listing loops and by
\{{hasNextNoFilter}}. Existing \{{List}}-returning methods can stay for other
callers.
Both \{{ofs}} (\{{BasicRootedOzoneFileSystem}}) and \{{o3fs}}
(\{{BasicOzoneFileSystem}}) benefit.
h2. Expected effect
A listing that fits in one page goes from two OM listing RPCs to one. On top of
HDDS-16760, an \{{ofs}}
{\{listStatus}} of such a directory goes from 2 OM RPCs to 1; before HDDS-16760
it was 4.
h2. Compatibility
* One new optional field in two response messages; no new request field, result
code or configuration.
* New client, old OM: the field is absent, so the client keeps the current loop.
* Old client, new OM: the field is ignored; behavior unchanged.
* Listing results are unchanged; only the number of calls changes.
h2. Testing
* OM unit tests for both \{{ListStatus}} and \{{ListStatusLight}}, FSO and
LEGACY: \{{isTruncated}} is \{{false}} when
fewer than the effective limit entries are returned and \{{true}} on a full
page, with the effective limit set by
each cap (\{{ozone.om.server.list.max.size}} below and above
\{{ozone.fs.listing.page.size.max}}, and a request of
size 1, which OM raises to 2).
* A full page whose only remaining entries are cache tombstones still ends the
listing on the next call.
* Client unit tests: the loop stops after one call when \{{isTruncated}} is
\{{false}}, and keeps the current loop
when the field is absent.
* Integration: a directory larger than \{{ozone.om.server.list.max.size}} with
the client page size above it, listed
in full through \{{ofs}} and \{{o3fs}}, by \{{listStatus}} and
\{{listStatusIterator}}. This is the regression guard
for HDDS-11696.
* \{{TestOfsListStatusRpcBenchmark}}: ListStatus RPCs per call drop from 2 to 1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
