Andrey Yarovoy created HDDS-17779:
-------------------------------------

             Summary: make the OpenTelemetry SDK and OTLP exporter optional in 
the filesystem jar
                 Key: HDDS-17779
                 URL: https://issues.apache.org/jira/browse/HDDS-17779
             Project: Apache Ozone
          Issue Type: Improvement
          Components: Ozone Client
            Reporter: Andrey Yarovoy


h3. Problem

The shaded \{{ozone-filesystem-hadoop3}} jar bundles the whole client tracing 
stack: the OpenTelemetry API, the SDK, the OTLP exporter, and the exporter's 
HTTP client with its runtime. It comes from \{{hdds-common}}, which declares 
\{{opentelemetry-exporter-otlp}} and \{{opentelemetry-sdk*}} as compile 
dependencies:
{noformat}
ozone-filesystem-shaded
\- ozone-filesystem-common
+- opentelemetry-api 1.67.0 (+ context, common)
\- hdds-common
+- opentelemetry-exporter-otlp 1.67.0
| +- opentelemetry-sdk-metrics, opentelemetry-sdk-logs
| +- opentelemetry-exporter-otlp-common, opentelemetry-exporter-common
| \- opentelemetry-exporter-sender-okhttp
| \- okhttp-jvm 5.5.0
| +- kotlin-stdlib 1.9.25 (+ org.jetbrains:annotations)
| \- okio-jvm 3.18.1
+- opentelemetry-sdk, opentelemetry-sdk-common, opentelemetry-sdk-trace
{noformat}

Compressed size in the jar (everything is relocated under 
\{{org/apache/ozone/shaded/}}):

||what||size||needed when||
|\{{kotlin}}|1.59 MB|tracing is exported (okhttp runtime)|
|\{{okhttp3}}|0.91 MB|tracing is exported|
|\{{okio}}|0.38 MB|tracing is exported|
|\{{io/opentelemetry/sdk}}: metrics 308 KB, trace 119 KB, common 65 KB, logs 59 
KB, resources 18 KB|0.58 MB|tracing is exported; metrics and logs never (Ozone 
uses only traces)|
|\{{io/opentelemetry/exporter}}|0.37 MB|tracing is exported|
|*to move out*|*3.83 MB, about 4 MB on disk*| |
|\{{io/opentelemetry/api}}, \{{context}}, \{{common}}|0.21 MB|always: Ozone's 
code calls the API; it stays|

A filesystem client only needs the SDK and exporter when 
\{{ozone.tracing.enabled=true}}, which is off by default. That's about 6% of 
the 71 MB jar after HDDS-17772, shipped with every job that uses \{{-libjars}} 
or \{{spark.jars}}.

h3. Proposed change

# *Optional SDK in \{{TracingUtil}} (hdds-common).* \{{TracingUtil}} declares 
fields of the SDK types \{{SdkTracerProvider}} and \{{BatchSpanProcessor}}, and 
builds \{{OtlpGrpcSpanExporter}}, \{{OpenTelemetrySdk}} and \{{Resource}} 
directly. \{{initTracing}} catches \{{Exception}}, but a missing class raises 
\{{NoClassDefFoundError}}, so without the SDK on the classpath a client with 
tracing turned on would fail instead of degrading.
#* Move all SDK and exporter code into one package-private class (e.g. 
\{{OzoneTracingSdk}}): building the provider and exporter, registering the 
global, flushing and shutting down. \{{TracingUtil}} keeps only API types, and 
holds that class through an API-typed handle.
#* Call into it inside \{{try ... catch (LinkageError e)}}. Then log once at 
WARN: "ozone.tracing.enabled is true but the OpenTelemetry SDK is not on the 
classpath; add ozone-filesystem-hadoop3-tracing", and keep the no-op tracer.
#* Same for the application-aware path. On master it builds Ozone's SDK 
whenever no application global is set, even with tracing off. HDDS-17350 makes 
that build lazy, so default clients never reach the SDK. If HDDS-17350 hasn't 
landed when this change does, the fallback above covers it.
# *Shaded jar:* exclude from the \{{ozonefs-shaded}} artifact set:
#* \{{opentelemetry-sdk*}} and \{{opentelemetry-exporter-*}};
#* \{{okhttp-jvm}}, \{{okio-jvm}}, \{{kotlin-stdlib}} and 
\{{org.jetbrains:annotations}}.
#* Keep \{{opentelemetry-api}}, \{{-context}} and \{{-common}}.
# *New artifact \{{ozone-filesystem-hadoop3-tracing}}:* a small shaded jar with 
the excluded artifacts only, relocated with the same \{{ozone.shaded.prefix}} 
so the relocated Ozone code finds them. It is published to Maven, shipped in 
the dist next to the filesystem jar, and added by users who want client spans, 
e.g. \{{-libjars 
ozone-filesystem-hadoop3.jar,ozone-filesystem-hadoop3-tracing.jar}}. The same 
applies to \{{ozone-filesystem-hadoop2}} if it shades the same stack.
# *Docs:* the tracing page says which jar a filesystem client needs for 
\{{ozone.tracing.enabled=true}}.

*Considered and rejected:*
* *Asking users to add the stock OpenTelemetry jars.* The relocated Ozone code 
would never see them.
* *Replacing okhttp with the JDK \{{HttpClient}} sender 
(\{{opentelemetry-exporter-sender-jdk}}).* That would save okhttp, okio and 
Kotlin (2.9 MB) and keep tracing in the main jar. But it needs Java 11, and the 
filesystem jar targets Java 8 (\{{ozone.java.version.client}}). Revisit when 
the client moves to Java 11.

*Smaller, independent step:* \{{opentelemetry-sdk-metrics}} and 
\{{opentelemetry-sdk-logs}} (367 KB) come in only as dependencies of the OTLP 
exporter artifact, and Ozone exports only traces. Excluding them from the 
shaded jar needs a check that the trace exporter never loads them.

h3. Expected result

||jar||before||after||
|\{{ozone-filesystem-hadoop3}} with HDDS-17772|71 MB|about 67 MB|
|\{{ozone-filesystem-hadoop3}} on master today|155 MB|about 151 MB|
|\{{ozone-filesystem-hadoop3-tracing}} (new)|—|about 4 MB|

Together with HDDS-14949 (second gRPC stack, about 15 MB) and the httpclient 
removal (about 1 MB), the jar goes from 71 to about 51 MB.

h3. Compatibility

* *Default configuration:* no change. Tracing is off by default.
* *Application-aware mode (the default):* in the shaded jar it loses nothing. 
Because \{{io.opentelemetry}} is relocated, the application's 
\{{GlobalOpenTelemetry}} and current span are invisible to Ozone, so no parent 
span is ever valid, and the mode never produces spans. *This needs to be 
confirmed with a test before filing.*
* *Clients with \{{ozone.tracing.enabled=true}}:* must add 
\{{ozone-filesystem-hadoop3-tracing}}. Without it they get the WARN line and no 
spans; the client keeps working.
* *Servers and the unshaded \{{ozone-client}}:* not affected. The OM, SCM, 
datanode and S3 Gateway classpaths keep the SDK.
* No wire, proto, config or on-disk changes.

h3. Testing

* *Unit:*
** \{{TracingUtil}} with the SDK hidden: a class loader that refuses 
\{{io.opentelemetry.sdk}} and \{{io.opentelemetry.exporter}}. Check that 
\{{initTracing}} with tracing enabled and with application-aware mode returns, 
logs the WARN once, and that \{{executeInNewSpan}}, \{{createProxy}} and 
\{{flushTracing}} work as no-ops.
** \{{TracingUtil}} with the SDK present: the existing tests are unchanged.
* *Jar contents:*
** \{{ozone-filesystem-hadoop3}} has no 
\{{org/apache/ozone/shaded/io/opentelemetry/sdk}}, \{{.../exporter}}, 
\{{okhttp3}}, \{{okio}} or \{{kotlin}};
** the tracing jar has exactly those, under the same prefix.
* *End to end:* an ofs client with \{{ozone.tracing.enabled=true}} against the 
compose cluster's collector (\{{compose/ozone/monitoring.yaml}}):
** with both jars, spans arrive;
** with only the filesystem jar, there is one WARN line and the operations 
succeed.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to