Andrey Yarovoy created HDDS-17351:
-------------------------------------
Summary: Avoid starting the Hadoop metrics system in short-lived
Ozone client processes
Key: HDDS-17351
URL: https://issues.apache.org/jira/browse/HDDS-17351
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Client
Reporter: Andrey Yarovoy
*Problem*
The first datanode client an Ozone client process creates calls
`XceiverClientManager.getXceiverClientMetrics()`,
which calls `XceiverClientMetrics.create()`, which runs
`DefaultMetricsSystem.initialize("XceiverClientMetrics")`.
That starts a full `MetricsSystemImpl` and registers JMX MBeans for the system
and each source:
- it looks for `hadoop-metrics2-xceiverclientmetrics.properties` and
`hadoop-metrics2.properties`;
- it builds the metrics config and starts its timer;
- it registers `XceiverClientMetrics`, and in the `XceiverClientManager`
constructor `CacheMetrics` too (and
`ContainerClientMetrics` on writes).
The client never reads these metrics itself and never sends them to OM or SCM.
They're only useful where something
collects them: a metrics sink configured for the client process (the reason
HDDS-1721 added the `initialize`
call), or the `/jmx` and Prometheus endpoints of a long-running service that
embeds the client (S3 Gateway, HttpFS,
Recon, datanode EC reconstruction).
Short-lived clients pay the start-up cost and nobody ever reads the result.
That covers every YARN
`ContainerLocalizer` JVM, MapReduce task, CLI command and job submitter (eg.
`hadoop jar`, `spark-submit`). In a fresh JVM on a secure compose
cluster (Kerberos, gRPC TLS), timed with step markers and DEBUG logs:
| step of a cold client | total | of which metrics2 |
|---|---|---|
| first read (first datanode client) | ~480 ms | **~100 ms**:
`MetricsSystemImpl` init, `MetricsConfig` file lookups, `MBeans` registration |
| client creation (`FileSystem.get`) | ~600 ms | tens of ms: `CacheMetrics`
registration |
A MapReduce job on Ozone starts about four cold clients one after another on
its critical path: the AppMaster's
localizer, the AppMaster, a map container's localizer and the map task. So this
is about 0.4–0.6 s of every job's
fixed cost.
*Proposed change*
1. **Make registration conditional.** Add a client setting, eg.
`ozone.client.metrics.mode` in `OzoneClientConfig`:
- `ALWAYS` (default): today's behaviour.
- `AUTO`: start the metrics system and register sources only if (a) the process
has already initialized
`DefaultMetricsSystem`, which is the case when the client runs inside an Ozone
service or another
instrumented service; or (b) a sink is configured for the client's metrics
prefix. For (b), check for a
`hadoop-metrics2[-<prefix>].properties` resource defining `*.sink.*` or
`<prefix>.sink.*` keys, which is
cheap and doesn't start metrics2.
- `NEVER`: never register.
2. **Keep the metrics objects either way.** `XceiverClientMetrics`,
`CacheMetrics` and `ContainerClientMetrics` are
created as plain, unregistered metric objects when registration is off. The
code paths that increment them, and
the tests that read them, stay unchanged, and no `MetricsSystemImpl`, MBean or
timer is created. Registration,
when it happens, should be done once per process, not per
`XceiverClientManager` (HDDS-2284 noted the current
duplicate creation).
3. **Default.** Keep `ALWAYS` in this change. Whether `AUTO` can become the
default later is for discussion: it
would preserve HDDS-1721's sink case and the embedded-service case, and only
lose ad-hoc JMX browsing of a
short-lived client JVM.
*Compatibility*
There is no change with the default. Services that embed the client keep their
client metrics under `AUTO` too,
because they initialize `DefaultMetricsSystem` before creating clients. Only
someone who switches to `AUTO` and still wants to
browse a short-lived client JVM over JMX without a configured sink would lose
its client metrics; they keep `ALWAYS`.
*Tests*
- Unit tests per mode: whether `DefaultMetricsSystem` is initialized and the
sources are registered, and that the
counters still work when registration is off.
- An `AUTO` test with a test-only `hadoop-metrics2` file that configures a
sink: metrics are registered.
- An `AUTO` test where the service initialized the metrics system first:
metrics are registered.
- Optional: the cold-client probe (create client, getFileStatus, read 64 KB,
close) shows the ~100 ms drop in
first-read time under `NEVER`/`AUTO`.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
