Andrey Yarovoy created HDDS-17731:
-------------------------------------
Summary: OM GetFileStatus should take block pipelines from the
container location cache instead of calling SCM on every request
Key: HDDS-17731
URL: https://issues.apache.org/jira/browse/HDDS-17731
Project: Apache Ozone
Issue Type: Improvement
Components: OM
Reporter: Andrey Yarovoy
h3. Problem
For a file, a GetFileStatus that is not a head op refreshes the pipelines of
its blocks with
{\{KeyManagerImpl.refresh}}. That method calls SCM
\{{getContainerWithPipelineBatch}} synchronously, in the OM
handler thread, on every request, in both \{{getOzoneFileStatus}} and
\{{getOzoneFileStatusFSO}}.
The other read paths take pipelines from the container location cache in
\{{ScmClient}} (HDDS-7229, 100,000
containers, 6 hour TTL):
* \{{getKeyInfo}}, which clients read files with, uses
\{{refreshPipelineFromCache}}. A client that finds a location
stale retries it with \{{forceUpdateContainerCacheFromSCM}}.
* \{{listStatus}} always uses the cache.
After HDDS-17714, ofs and o3fs \{{getFileStatus}} send a head op, so a full
GetFileStatus only serves
{\{getFileBlockLocations}}. Its block locations are scheduling hints, not used
to read data. Server-side callers of
{\{KeyManagerImpl.getFileStatus}} (\{{OzonePrefixPathImpl}},
\{{TrashOzoneFileSystem}}) do not read with them either.
The SCM call costs:
* an OM to SCM round trip per request;
* SCM handler time;
* an OM handler thread blocked on SCM for that time.
h3. Proposal
In \{{KeyManagerImpl.getFileStatus}}, take the pipelines from the cache, as
\{{listStatus}} does:
{\{refreshPipelineFromCache(keyInfo, false)}}. A container missing from the
cache is loaded from SCM in one batch,
as for \{{getKeyInfo}}.
{\{lookupFile}} shares \{{getOzoneFileStatus}} and \{{getOzoneFileStatusFSO}}
and keeps calling SCM. It is a read
path:
* clients older than \{{OzoneManagerVersion.OPTIMIZED_GET_KEY_INFO}} open files
with it;
* \{{OzoneClientUtils}} computes file checksums with it;
* those clients retry a failed read with \{{lookupFile}} again, with no way to
force a cache refresh.
The patch adds a flag to the two shared methods. \{{getFileStatus}} passes
\{{true}}; \{{lookupFile}} and the
{\{listStatus}} FSO helper pass \{{false}}, so they are unchanged. No protocol
or client change.
h3. Side effects
A \{{getFileBlockLocations}} result can name the datanodes of a pipeline that
changed less than the cache TTL ago, as
{\{listStatus}} and \{{listLocatedStatus}} already can. Such a location is a
less well placed task, not a failed read.
h3. Benchmark
Prototype attached
(\{{HDDS-XXXXX-getfilestatus-location-cache-prototype.patch}}).
Setup:
* mini cluster with 5 datanodes;
* 8 files of 24 MB each: EC rs-3-2 (4 block groups) and Ratis/THREE (12 blocks);
* 4,000 operations per run, sequential and with 10 threads;
* the HDDS-17714 client in both arms;
* OM calling SCM (master) vs the cache (patch), alternating, 3 runs each.
Values are medians.
* "Other server CPU" is the CPU of the non-OM RPC handler threads in the JVM,
which is SCM.
||operation||file||threads||mean latency (µs)||p99 (µs)||throughput (ops/s)||OM
handler CPU per op (µs)||other server CPU per op (µs)||
|getFileBlockLocations(path)|EC|1|268 → 154|363 → 229|3,733 → 6,501 (1.74x)|105
→ 74|48 → 3|
|getFileBlockLocations(path)|EC|10|738 → 396|1,075 → 652|13,461 → 24,353
(1.81x)|280 → 156|112 → 0.3|
|getFileBlockLocations(path)|Ratis|1|295 → 187|410 → 281|3,384 → 5,351
(1.58x)|113 → 84|45 → 7|
|getFileBlockLocations(path)|Ratis|10|761 → 433|1,108 → 734|13,056 → 22,944
(1.76x)|275 → 158|100 → 0.6|
|getFileStatus + getFileBlockLocations(status)|EC|1|390 → 276|525 → 392|2,561 →
3,622 (1.41x)|171 → 140|51 → 10|
|getFileStatus + getFileBlockLocations(status)|EC|10|1,116 → 797|1,478 →
1,172|8,864 → 12,467 (1.41x)|436 → 312|106 → 1|
|getFileStatus + getFileBlockLocations(status)|Ratis|1|423 → 306|625 →
429|2,366 → 3,271 (1.38x)|182 → 151|47 → 10|
|getFileStatus + getFileBlockLocations(status)|Ratis|10|1,143 → 826|1,517 →
1,268|8,697 → 12,047 (1.39x)|436 → 319|98 → 1|
|getFileStatus (head op, control)|EC / Ratis|1 / 10|unchanged (0.97x to 1.07x)|
| | | |
* getFileBlockLocations(path) is 1.6x to 1.8x faster, and its SCM handler CPU
drops to almost zero.
* getFileStatus followed by getFileBlockLocations(status) takes 2 OM calls
since HDDS-17714 (1 before). With this
change it costs 276 to 306 µs sequential, about what the single call cost
before HDDS-17714 (273 to 298 µs).
* On a real cluster the saved OM to SCM round trip crosses the network, so the
gain should be larger than in this
single JVM mini cluster.
h3. Tests (for the patch)
* \{{TestKeyManagerImpl}} (or a new unit test with a mock \{{ScmClient}}):
** a full \{{getFileStatus}} for a file in an OBS/LEGACY bucket and in an FSO
bucket calls
{\{ScmClient.getContainerLocations(ids, false)}} and not
\{{getContainerWithPipelineBatch}};
** \{{lookupFile}} still calls \{{getContainerWithPipelineBatch}};
** a head op calls neither.
* \{{TestOmContainerLocationCache}}: \{{getFileStatus}} does not call SCM for a
cached container, and loads a container
that is not cached.
* \{{TestOFS}} / \{{TestO3FS}}: \{{getFileBlockLocations}} returns the
datanodes of the blocks.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
