[
https://issues.apache.org/jira/browse/HDDS-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-17731:
----------------------------------
Attachment: HDDS-XXXXX-getfilestatus-location-cache-prototype.patch
> OM GetFileStatus should take block pipelines from the container location
> cache instead of calling SCM on every request
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17731
> URL: https://issues.apache.org/jira/browse/HDDS-17731
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
> Attachments: HDDS-XXXXX-getfilestatus-location-cache-prototype.patch
>
>
> h3. Problem
> For a file, a GetFileStatus that is not a head op refreshes the pipelines of
> its blocks with
> {\{KeyManagerImpl.refresh}}. That method calls SCM
> \{{getContainerWithPipelineBatch}} synchronously, in the OM
> handler thread, on every request, in both \{{getOzoneFileStatus}} and
> \{{getOzoneFileStatusFSO}}.
> The other read paths take pipelines from the container location cache in
> \{{ScmClient}} (HDDS-7229, 100,000
> containers, 6 hour TTL):
> * \{{getKeyInfo}}, which clients read files with, uses
> \{{refreshPipelineFromCache}}. A client that finds a location
> stale retries it with \{{forceUpdateContainerCacheFromSCM}}.
> * \{{listStatus}} always uses the cache.
> After HDDS-17714, ofs and o3fs \{{getFileStatus}} send a head op, so a full
> GetFileStatus only serves
> {\{getFileBlockLocations}}. Its block locations are scheduling hints, not
> used to read data. Server-side callers of
> {\{KeyManagerImpl.getFileStatus}} (\{{OzonePrefixPathImpl}},
> \{{TrashOzoneFileSystem}}) do not read with them either.
> The SCM call costs:
> * an OM to SCM round trip per request;
> * SCM handler time;
> * an OM handler thread blocked on SCM for that time.
> h3. Proposal
> In \{{KeyManagerImpl.getFileStatus}}, take the pipelines from the cache, as
> \{{listStatus}} does:
> {\{refreshPipelineFromCache(keyInfo, false)}}. A container missing from the
> cache is loaded from SCM in one batch,
> as for \{{getKeyInfo}}.
> {\{lookupFile}} shares \{{getOzoneFileStatus}} and \{{getOzoneFileStatusFSO}}
> and keeps calling SCM. It is a read
> path:
> * clients older than \{{OzoneManagerVersion.OPTIMIZED_GET_KEY_INFO}} open
> files with it;
> * \{{OzoneClientUtils}} computes file checksums with it;
> * those clients retry a failed read with \{{lookupFile}} again, with no way
> to force a cache refresh.
> The patch adds a flag to the two shared methods. \{{getFileStatus}} passes
> \{{true}}; \{{lookupFile}} and the
> {\{listStatus}} FSO helper pass \{{false}}, so they are unchanged. No
> protocol or client change.
> h3. Side effects
> A \{{getFileBlockLocations}} result can name the datanodes of a pipeline that
> changed less than the cache TTL ago, as
> {\{listStatus}} and \{{listLocatedStatus}} already can. Such a location is a
> less well placed task, not a failed read.
> h3. Benchmark
> Prototype attached
> (\{{HDDS-XXXXX-getfilestatus-location-cache-prototype.patch}}).
> Setup:
> * mini cluster with 5 datanodes;
> * 8 files of 24 MB each: EC rs-3-2 (4 block groups) and Ratis/THREE (12
> blocks);
> * 4,000 operations per run, sequential and with 10 threads;
> * the HDDS-17714 client in both arms;
> * OM calling SCM (master) vs the cache (patch), alternating, 3 runs each.
> Values are medians.
> * "Other server CPU" is the CPU of the non-OM RPC handler threads in the JVM,
> which is SCM.
> ||operation||file||threads||mean latency (µs)||p99 (µs)||throughput
> (ops/s)||OM handler CPU per op (µs)||other server CPU per op (µs)||
> |getFileBlockLocations(path)|EC|1|268 → 154|363 → 229|3,733 → 6,501
> (1.74x)|105 → 74|48 → 3|
> |getFileBlockLocations(path)|EC|10|738 → 396|1,075 → 652|13,461 → 24,353
> (1.81x)|280 → 156|112 → 0.3|
> |getFileBlockLocations(path)|Ratis|1|295 → 187|410 → 281|3,384 → 5,351
> (1.58x)|113 → 84|45 → 7|
> |getFileBlockLocations(path)|Ratis|10|761 → 433|1,108 → 734|13,056 → 22,944
> (1.76x)|275 → 158|100 → 0.6|
> |getFileStatus + getFileBlockLocations(status)|EC|1|390 → 276|525 → 392|2,561
> → 3,622 (1.41x)|171 → 140|51 → 10|
> |getFileStatus + getFileBlockLocations(status)|EC|10|1,116 → 797|1,478 →
> 1,172|8,864 → 12,467 (1.41x)|436 → 312|106 → 1|
> |getFileStatus + getFileBlockLocations(status)|Ratis|1|423 → 306|625 →
> 429|2,366 → 3,271 (1.38x)|182 → 151|47 → 10|
> |getFileStatus + getFileBlockLocations(status)|Ratis|10|1,143 → 826|1,517 →
> 1,268|8,697 → 12,047 (1.39x)|436 → 319|98 → 1|
> |getFileStatus (head op, control)|EC / Ratis|1 / 10|unchanged (0.97x to
> 1.07x)| | | | |
> * getFileBlockLocations(path) is 1.6x to 1.8x faster, and its SCM handler CPU
> drops to almost zero.
> * getFileStatus followed by getFileBlockLocations(status) takes 2 OM calls
> since HDDS-17714 (1 before). With this
> change it costs 276 to 306 µs sequential, about what the single call cost
> before HDDS-17714 (273 to 298 µs).
> * On a real cluster the saved OM to SCM round trip crosses the network, so
> the gain should be larger than in this
> single JVM mini cluster.
> h3. Tests (for the patch)
> * \{{TestKeyManagerImpl}} (or a new unit test with a mock \{{ScmClient}}):
> ** a full \{{getFileStatus}} for a file in an OBS/LEGACY bucket and in an FSO
> bucket calls
> {\{ScmClient.getContainerLocations(ids, false)}} and not
> \{{getContainerWithPipelineBatch}};
> ** \{{lookupFile}} still calls \{{getContainerWithPipelineBatch}};
> ** a head op calls neither.
> * \{{TestOmContainerLocationCache}}: \{{getFileStatus}} does not call SCM for
> a cached container, and loads a container
> that is not cached.
> * \{{TestOFS}} / \{{TestO3FS}}: \{{getFileBlockLocations}} returns the
> datanodes of the blocks.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
