This is an automated email from the ASF dual-hosted git repository.
pnowojski pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit ddfb5a3a0292c29a5d8df3e3ab408893ab363e63
Author: Piotr Nowojski <piotr.nowoj...@gmail.com>
AuthorDate: Tue Oct 24 17:36:15 2023 +0200
[FLINK-33708][docs] Document Span and TraceReporter concepts
---
docs/content.zh/docs/deployment/config.md | 9 +++
docs/content.zh/docs/deployment/security/_index.md | 4 +-
docs/content.zh/docs/deployment/trace_reporters.md | 75 ++++++++++++++++++++++
docs/content.zh/docs/ops/metrics.md | 2 +-
docs/content.zh/docs/ops/traces.md | 71 ++++++++++++++++++++
docs/content/docs/deployment/config.md | 9 +++
docs/content/docs/deployment/security/_index.md | 4 +-
docs/content/docs/deployment/trace_reporters.md | 75 ++++++++++++++++++++++
docs/content/docs/ops/metrics.md | 2 +-
docs/content/docs/ops/traces.md | 71 ++++++++++++++++++++
.../shortcodes/generated/trace_configuration.html | 36 +++++++++++
.../generated/trace_reporters_section.html | 30 +++++++++
12 files changed, 382 insertions(+), 6 deletions(-)
diff --git a/docs/content.zh/docs/deployment/config.md
b/docs/content.zh/docs/deployment/config.md
index b68d87afecf..34d04f733e5 100644
--- a/docs/content.zh/docs/deployment/config.md
+++ b/docs/content.zh/docs/deployment/config.md
@@ -287,6 +287,15 @@ Enabling RocksDB's native metrics may cause degraded
performance and should be s
----
----
+# Traces
+
+Please refer to the [tracing system documentation]({{< ref "docs/ops/traces"
>}}) for background on Flink's tracing infrastructure.
+
+{{< generated/trace_configuration >}}
+
+----
+----
+
# History Server
The history server keeps the information of completed jobs (graphs, runtimes,
statistics). To enable it, you have to enable "job archiving" in the JobManager
(`jobmanager.archive.fs.dir`).
diff --git a/docs/content.zh/docs/deployment/security/_index.md
b/docs/content.zh/docs/deployment/security/_index.md
index d098fefc680..8a2a87a398e 100644
--- a/docs/content.zh/docs/deployment/security/_index.md
+++ b/docs/content.zh/docs/deployment/security/_index.md
@@ -1,7 +1,7 @@
---
title: Security
bookCollapseSection: true
-weight: 8
+weight: 9
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -20,4 +20,4 @@ software distributed under the License is distributed on an
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
--->
\ No newline at end of file
+-->
diff --git a/docs/content.zh/docs/deployment/trace_reporters.md
b/docs/content.zh/docs/deployment/trace_reporters.md
new file mode 100644
index 00000000000..eaab7d4ce3d
--- /dev/null
+++ b/docs/content.zh/docs/deployment/trace_reporters.md
@@ -0,0 +1,75 @@
+---
+title: "Trace Reporters"
+weight: 7
+type: docs
+aliases:
+ - /deployment/trace_reporters.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Trace Reporters
+
+Flink allows reporting traces to external systems.
+For more information about Flink's tracing system go to the [tracing system
documentation]({{< ref "docs/ops/traces" >}}).
+
+Traces can be exposed to an external system by configuring one or several
reporters in `conf/flink-conf.yaml`. These
+reporters will be instantiated on each job and task manager when they are
started.
+
+Below is a list of parameters that are generally applicable to all reporters.
+All properties are configured by setting
`traces.reporter.<reporter_name>.<property>` in the configuration.
+Reporters may additionally offer implementation-specific parameters, which are
documented in the respective reporter's section.
+
+{{< include_reporter_config
"layouts/shortcodes/generated/trace_reporters_section.html" >}}
+
+All reporter configurations must contain the `factory.class` property.
+
+Example reporter configuration that specifies multiple reporters:
+
+```yaml
+traces.reporters: otel,my_other_otel
+
+traces.reporter.otel.factory.class:
org.apache.flink.common.metrics.OpenTelemetryTraceReporterFactory
+traces.reporter.otel.exporter.endpoint: http://127.0.0.1:1337
+traces.reporter.otel.scope.variables.additional:
region:eu-west-1,environment:local-pnowojski-test,flink_runtime:1.17.1
+
+traces.reporter.my_other_otel.factory.class:
org.apache.flink.common.metrics.OpenTelemetryTraceReporterFactory
+traces.reporter.my_other_otel.exporter.endpoint: http://196.168.0.1:31337
+```
+
+**Important:** The jar containing the reporter must be accessible when Flink
is started.
+ Reporters are loaded as [plugins]({{< ref
"docs/deployment/filesystems/plugins" >}}).
+ All reporters documented on this page are available by default.
+
+You can write your own `Reporter` by implementing the
`org.apache.flink.traces.reporter.TraceReporter` and
`org.apache.flink.traces.reporter.TraceReporterFactory` interfaces.
+Be careful that all the method must not block for a significant amount of
time, and any reporter needing more time should instead run the operation
asynchronously.
+
+## Reporters
+
+The following sections list the supported reporters.
+
+### Slf4j
+#### (org.apache.flink.traces.slf4j.Slf4jTraceReporter)
+
+Example configuration:
+
+```yaml
+traces.reporter.slf4j.factory.class:
org.apache.flink.traces.slf4j.Slf4jTraceReporterFactory
+```
+{{< top >}}
diff --git a/docs/content.zh/docs/ops/metrics.md
b/docs/content.zh/docs/ops/metrics.md
index d16d5e6d3eb..959f0f0e700 100644
--- a/docs/content.zh/docs/ops/metrics.md
+++ b/docs/content.zh/docs/ops/metrics.md
@@ -1,6 +1,6 @@
---
title: "指标"
-weight: 6
+weight: 5
type: docs
aliases:
- /zh/ops/metrics.html
diff --git a/docs/content.zh/docs/ops/traces.md
b/docs/content.zh/docs/ops/traces.md
new file mode 100644
index 00000000000..427ab8d6cc3
--- /dev/null
+++ b/docs/content.zh/docs/ops/traces.md
@@ -0,0 +1,71 @@
+---
+title: "Traces"
+weight: 6
+type: docs
+aliases:
+ - /ops/traces.html
+ - /apis/traces.html
+ - /monitoring/traces.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Traces
+
+Flink exposes a tracing system that allows gathering and exposing traces to
external systems.
+
+## Reporting traces
+
+You can access the tracing system from any user function that extends
[RichFunction]({{< ref "docs/dev/datastream/user_defined_functions"
>}}#rich-functions) by calling `getRuntimeContext().getMetricGroup()`.
+This method returns a `MetricGroup` object via which you can report a new
single span trace.
+
+### Reporting single Span
+
+
+A `Span` represents something that happened in Flink at certain point of time,
that will be reported to a `TraceReporter`.
+To report a `Span` you can use the `MetricGroup#addSpan(SpanBuilder)` method.
+
+Currently we don't support traces with multiple spans. Each `Span` is
self-contained and represents things like a checkpoint or recovery.
+{{< tabs "9612d275-bdda-4322-a01f-ae6da805e917" >}}
+{{< tab "Java" >}}
+```java
+public class MyClass {
+ void doSomething() {
+ // (...)
+ metricGroup.addSpan(
+ Span.builder(MyClass.class, "SomeAction")
+ .setStartTsMillis(startTs) // Optional
+ .setEndTsMillis(endTs) // Optional
+ .setAttribute("foo", "bar");
+ }
+}
+```
+{{< /tab >}}
+{{< tab "Python" >}}
+```python
+Currently reporting Spans from Python is not supported.
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+## Reporter
+
+For information on how to set up Flink's trace reporters please take a look at
the [trace reporters documentation]({{< ref "docs/deployment/trace_reporters"
>}}).
+
+{{< top >}}
diff --git a/docs/content/docs/deployment/config.md
b/docs/content/docs/deployment/config.md
index 4fb5023d15e..cbdc4f25a77 100644
--- a/docs/content/docs/deployment/config.md
+++ b/docs/content/docs/deployment/config.md
@@ -289,6 +289,15 @@ Enabling RocksDB's native metrics may cause degraded
performance and should be s
----
----
+# Traces
+
+Please refer to the [tracing system documentation]({{< ref "docs/ops/traces"
>}}) for background on Flink's tracing infrastructure.
+
+{{< generated/trace_configuration >}}
+
+----
+----
+
# History Server
The history server keeps the information of completed jobs (graphs, runtimes,
statistics). To enable it, you have to enable "job archiving" in the JobManager
(`jobmanager.archive.fs.dir`).
diff --git a/docs/content/docs/deployment/security/_index.md
b/docs/content/docs/deployment/security/_index.md
index d098fefc680..8a2a87a398e 100644
--- a/docs/content/docs/deployment/security/_index.md
+++ b/docs/content/docs/deployment/security/_index.md
@@ -1,7 +1,7 @@
---
title: Security
bookCollapseSection: true
-weight: 8
+weight: 9
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -20,4 +20,4 @@ software distributed under the License is distributed on an
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
--->
\ No newline at end of file
+-->
diff --git a/docs/content/docs/deployment/trace_reporters.md
b/docs/content/docs/deployment/trace_reporters.md
new file mode 100644
index 00000000000..eaab7d4ce3d
--- /dev/null
+++ b/docs/content/docs/deployment/trace_reporters.md
@@ -0,0 +1,75 @@
+---
+title: "Trace Reporters"
+weight: 7
+type: docs
+aliases:
+ - /deployment/trace_reporters.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Trace Reporters
+
+Flink allows reporting traces to external systems.
+For more information about Flink's tracing system go to the [tracing system
documentation]({{< ref "docs/ops/traces" >}}).
+
+Traces can be exposed to an external system by configuring one or several
reporters in `conf/flink-conf.yaml`. These
+reporters will be instantiated on each job and task manager when they are
started.
+
+Below is a list of parameters that are generally applicable to all reporters.
+All properties are configured by setting
`traces.reporter.<reporter_name>.<property>` in the configuration.
+Reporters may additionally offer implementation-specific parameters, which are
documented in the respective reporter's section.
+
+{{< include_reporter_config
"layouts/shortcodes/generated/trace_reporters_section.html" >}}
+
+All reporter configurations must contain the `factory.class` property.
+
+Example reporter configuration that specifies multiple reporters:
+
+```yaml
+traces.reporters: otel,my_other_otel
+
+traces.reporter.otel.factory.class:
org.apache.flink.common.metrics.OpenTelemetryTraceReporterFactory
+traces.reporter.otel.exporter.endpoint: http://127.0.0.1:1337
+traces.reporter.otel.scope.variables.additional:
region:eu-west-1,environment:local-pnowojski-test,flink_runtime:1.17.1
+
+traces.reporter.my_other_otel.factory.class:
org.apache.flink.common.metrics.OpenTelemetryTraceReporterFactory
+traces.reporter.my_other_otel.exporter.endpoint: http://196.168.0.1:31337
+```
+
+**Important:** The jar containing the reporter must be accessible when Flink
is started.
+ Reporters are loaded as [plugins]({{< ref
"docs/deployment/filesystems/plugins" >}}).
+ All reporters documented on this page are available by default.
+
+You can write your own `Reporter` by implementing the
`org.apache.flink.traces.reporter.TraceReporter` and
`org.apache.flink.traces.reporter.TraceReporterFactory` interfaces.
+Be careful that all the method must not block for a significant amount of
time, and any reporter needing more time should instead run the operation
asynchronously.
+
+## Reporters
+
+The following sections list the supported reporters.
+
+### Slf4j
+#### (org.apache.flink.traces.slf4j.Slf4jTraceReporter)
+
+Example configuration:
+
+```yaml
+traces.reporter.slf4j.factory.class:
org.apache.flink.traces.slf4j.Slf4jTraceReporterFactory
+```
+{{< top >}}
diff --git a/docs/content/docs/ops/metrics.md b/docs/content/docs/ops/metrics.md
index ac106f6e6c9..c2cddc4a6a8 100644
--- a/docs/content/docs/ops/metrics.md
+++ b/docs/content/docs/ops/metrics.md
@@ -1,6 +1,6 @@
---
title: "Metrics"
-weight: 6
+weight: 5
type: docs
aliases:
- /ops/metrics.html
diff --git a/docs/content/docs/ops/traces.md b/docs/content/docs/ops/traces.md
new file mode 100644
index 00000000000..427ab8d6cc3
--- /dev/null
+++ b/docs/content/docs/ops/traces.md
@@ -0,0 +1,71 @@
+---
+title: "Traces"
+weight: 6
+type: docs
+aliases:
+ - /ops/traces.html
+ - /apis/traces.html
+ - /monitoring/traces.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Traces
+
+Flink exposes a tracing system that allows gathering and exposing traces to
external systems.
+
+## Reporting traces
+
+You can access the tracing system from any user function that extends
[RichFunction]({{< ref "docs/dev/datastream/user_defined_functions"
>}}#rich-functions) by calling `getRuntimeContext().getMetricGroup()`.
+This method returns a `MetricGroup` object via which you can report a new
single span trace.
+
+### Reporting single Span
+
+
+A `Span` represents something that happened in Flink at certain point of time,
that will be reported to a `TraceReporter`.
+To report a `Span` you can use the `MetricGroup#addSpan(SpanBuilder)` method.
+
+Currently we don't support traces with multiple spans. Each `Span` is
self-contained and represents things like a checkpoint or recovery.
+{{< tabs "9612d275-bdda-4322-a01f-ae6da805e917" >}}
+{{< tab "Java" >}}
+```java
+public class MyClass {
+ void doSomething() {
+ // (...)
+ metricGroup.addSpan(
+ Span.builder(MyClass.class, "SomeAction")
+ .setStartTsMillis(startTs) // Optional
+ .setEndTsMillis(endTs) // Optional
+ .setAttribute("foo", "bar");
+ }
+}
+```
+{{< /tab >}}
+{{< tab "Python" >}}
+```python
+Currently reporting Spans from Python is not supported.
+```
+{{< /tab >}}
+{{< /tabs >}}
+
+## Reporter
+
+For information on how to set up Flink's trace reporters please take a look at
the [trace reporters documentation]({{< ref "docs/deployment/trace_reporters"
>}}).
+
+{{< top >}}
diff --git a/docs/layouts/shortcodes/generated/trace_configuration.html
b/docs/layouts/shortcodes/generated/trace_configuration.html
new file mode 100644
index 00000000000..eb837e317ab
--- /dev/null
+++ b/docs/layouts/shortcodes/generated/trace_configuration.html
@@ -0,0 +1,36 @@
+<table class="configuration table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Key</th>
+ <th class="text-left" style="width: 15%">Default</th>
+ <th class="text-left" style="width: 10%">Type</th>
+ <th class="text-left" style="width: 55%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>traces.reporter.<name>.<parameter></h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>Configures the parameter <parameter> for the reporter
named <name>.</td>
+ </tr>
+ <tr>
+ <td><h5>traces.reporter.<name>.factory.class</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>The reporter factory class to use for the reporter named
<name>.</td>
+ </tr>
+ <tr>
+
<td><h5>traces.reporter.<name>.scope.variables.additional</h5></td>
+ <td style="word-wrap: break-word;"></td>
+ <td>Map</td>
+ <td>The map of additional variables that should be included for
the reporter named <name>.</td>
+ </tr>
+ <tr>
+ <td><h5>traces.reporters</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>An optional list of trace reporter names. If configured, only
reporters whose name matches any of the names in the list will be started.
Otherwise, all reporters that could be found in the configuration will be
started.</td>
+ </tr>
+ </tbody>
+</table>
diff --git a/docs/layouts/shortcodes/generated/trace_reporters_section.html
b/docs/layouts/shortcodes/generated/trace_reporters_section.html
new file mode 100644
index 00000000000..a4b349690bf
--- /dev/null
+++ b/docs/layouts/shortcodes/generated/trace_reporters_section.html
@@ -0,0 +1,30 @@
+<table class="configuration table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Key</th>
+ <th class="text-left" style="width: 15%">Default</th>
+ <th class="text-left" style="width: 10%">Type</th>
+ <th class="text-left" style="width: 55%">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><h5>traces.reporter.<name>.factory.class</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>The reporter factory class to use for the reporter named
<name>.</td>
+ </tr>
+ <tr>
+
<td><h5>traces.reporter.<name>.scope.variables.additional</h5></td>
+ <td style="word-wrap: break-word;"></td>
+ <td>Map</td>
+ <td>The map of additional variables that should be included for
the reporter named <name>.</td>
+ </tr>
+ <tr>
+ <td><h5>traces.reporter.<name>.<parameter></h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>Configures the parameter <parameter> for the reporter
named <name>.</td>
+ </tr>
+ </tbody>
+</table>
