Repository: flink Updated Branches: refs/heads/master 2d56b11e7 -> d66e95c61
[FLINK-7843][metrics][docs] Add time unit and metric type to system metrics reference This closes #4869. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/d66e95c6 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/d66e95c6 Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/d66e95c6 Branch: refs/heads/master Commit: d66e95c610900103d3e9e14227eeac2f6d2fe607 Parents: 2d56b11 Author: yew1eb <[email protected]> Authored: Fri Oct 20 15:55:21 2017 +0800 Committer: zentol <[email protected]> Committed: Mon Nov 6 14:41:57 2017 +0100 ---------------------------------------------------------------------- docs/monitoring/metrics.md | 187 ++++++++++++++++++++++++++++------------ 1 file changed, 131 insertions(+), 56 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flink/blob/d66e95c6/docs/monitoring/metrics.md ---------------------------------------------------------------------- diff --git a/docs/monitoring/metrics.md b/docs/monitoring/metrics.md index 0bc17a7..bb15cec 100644 --- a/docs/monitoring/metrics.md +++ b/docs/monitoring/metrics.md @@ -512,7 +512,7 @@ metrics.reporter.slf4j.interval: 60 SECONDS By default Flink gathers several metrics that provide deep insights on the current state. This section is a reference of all these metrics. -The tables below generally feature 4 columns: +The tables below generally feature 5 columns: * The "Scope" column describes which scope format is used to generate the system scope. For example, if the cell contains "Operator" then the scope format for "metrics.scope.operator" is used. @@ -525,6 +525,8 @@ The tables below generally feature 4 columns: * The "Description" column provides information as to what a given metric is measuring. +* The "Type" column describes which metric type is used for the measurement. + Note that all dots in the infix/metric name columns are still subject to the "metrics.delimiter" setting. Thus, in order to infer the metric identifier: @@ -537,10 +539,11 @@ Thus, in order to infer the metric identifier: <table class="table table-bordered"> <thead> <tr> - <th class="text-left" style="width: 20%">Scope</th> - <th class="text-left" style="width: 25%">Infix</th> - <th class="text-left" style="width: 23%">Metrics</th> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 22%">Infix</th> + <th class="text-left" style="width: 20%">Metrics</th> <th class="text-left" style="width: 32%">Description</th> + <th class="text-left" style="width: 8%">Type</th> </tr> </thead> <tbody> @@ -549,10 +552,12 @@ Thus, in order to infer the metric identifier: <td rowspan="2">Status.JVM.CPU</td> <td>Load</td> <td>The recent CPU usage of the JVM.</td> + <td>Gauge</td> </tr> <tr> <td>Time</td> <td>The CPU time used by the JVM.</td> + <td>Gauge</td> </tr> </tbody> </table> @@ -561,10 +566,11 @@ Thus, in order to infer the metric identifier: <table class="table table-bordered"> <thead> <tr> - <th class="text-left" style="width: 20%">Scope</th> - <th class="text-left" style="width: 25%">Infix</th> - <th class="text-left" style="width: 23%">Metrics</th> - <th class="text-left" style="width: 32%">Description</th> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 22%">Infix</th> + <th class="text-left" style="width: 20%">Metrics</th> + <th class="text-left" style="width: 32%">Description</th> + <th class="text-left" style="width: 8%">Type</th> </tr> </thead> <tbody> @@ -572,51 +578,63 @@ Thus, in order to infer the metric identifier: <th rowspan="12"><strong>Job-/TaskManager</strong></th> <td rowspan="12">Status.JVM.Memory</td> <td>Heap.Used</td> - <td>The amount of heap memory currently used.</td> + <td>The amount of heap memory currently used (in bytes).</td> + <td>Gauge</td> </tr> <tr> <td>Heap.Committed</td> - <td>The amount of heap memory guaranteed to be available to the JVM.</td> + <td>The amount of heap memory guaranteed to be available to the JVM (in bytes).</td> + <td>Gauge</td> </tr> <tr> <td>Heap.Max</td> - <td>The maximum amount of heap memory that can be used for memory management.</td> + <td>The maximum amount of heap memory that can be used for memory management (in bytes).</td> + <td>Gauge</td> </tr> <tr> <td>NonHeap.Used</td> - <td>The amount of non-heap memory currently used.</td> + <td>The amount of non-heap memory currently used (in bytes).</td> + <td>Gauge</td> </tr> <tr> <td>NonHeap.Committed</td> - <td>The amount of non-heap memory guaranteed to be available to the JVM.</td> + <td>The amount of non-heap memory guaranteed to be available to the JVM (in bytes).</td> + <td>Gauge</td> </tr> <tr> <td>NonHeap.Max</td> - <td>The maximum amount of non-heap memory that can be used for memory management.</td> + <td>The maximum amount of non-heap memory that can be used for memory management (in bytes).</td> + <td>Gauge</td> </tr> <tr> <td>Direct.Count</td> <td>The number of buffers in the direct buffer pool.</td> + <td>Gauge</td> </tr> <tr> <td>Direct.MemoryUsed</td> - <td>The amount of memory used by the JVM for the direct buffer pool.</td> + <td>The amount of memory used by the JVM for the direct buffer pool (in bytes).</td> + <td>Gauge</td> </tr> <tr> <td>Direct.TotalCapacity</td> - <td>The total capacity of all buffers in the direct buffer pool.</td> + <td>The total capacity of all buffers in the direct buffer pool (in bytes).</td> + <td>Gauge</td> </tr> <tr> <td>Mapped.Count</td> <td>The number of buffers in the mapped buffer pool.</td> + <td>Gauge</td> </tr> <tr> <td>Mapped.MemoryUsed</td> - <td>The amount of memory used by the JVM for the mapped buffer pool.</td> + <td>The amount of memory used by the JVM for the mapped buffer pool (in bytes).</td> + <td>Gauge</td> </tr> <tr> <td>Mapped.TotalCapacity</td> - <td>The number of buffers in the mapped buffer pool.</td> + <td>The number of buffers in the mapped buffer pool (in bytes).</td> + <td>Gauge</td> </tr> </tbody> </table> @@ -625,10 +643,11 @@ Thus, in order to infer the metric identifier: <table class="table table-bordered"> <thead> <tr> - <th class="text-left" style="width: 20%">Scope</th> - <th class="text-left" style="width: 25%">Infix</th> - <th class="text-left" style="width: 23%">Metrics</th> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 22%">Infix</th> + <th class="text-left" style="width: 20%">Metrics</th> <th class="text-left" style="width: 32%">Description</th> + <th class="text-left" style="width: 8%">Type</th> </tr> </thead> <tbody> @@ -637,6 +656,7 @@ Thus, in order to infer the metric identifier: <td rowspan="1">Status.JVM.ClassLoader</td> <td>Threads.Count</td> <td>The total number of live threads.</td> + <td>Gauge</td> </tr> </tbody> </table> @@ -645,10 +665,11 @@ Thus, in order to infer the metric identifier: <table class="table table-bordered"> <thead> <tr> - <th class="text-left" style="width: 20%">Scope</th> - <th class="text-left" style="width: 25%">Infix</th> - <th class="text-left" style="width: 23%">Metrics</th> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 22%">Infix</th> + <th class="text-left" style="width: 20%">Metrics</th> <th class="text-left" style="width: 32%">Description</th> + <th class="text-left" style="width: 8%">Type</th> </tr> </thead> <tbody> @@ -657,10 +678,12 @@ Thus, in order to infer the metric identifier: <td rowspan="2">Status.JVM.GarbageCollector</td> <td><GarbageCollector>.Count</td> <td>The total number of collections that have occurred.</td> + <td>Gauge</td> </tr> <tr> <td><GarbageCollector>.Time</td> <td>The total time spent performing garbage collection.</td> + <td>Gauge</td> </tr> </tbody> </table> @@ -669,10 +692,11 @@ Thus, in order to infer the metric identifier: <table class="table table-bordered"> <thead> <tr> - <th class="text-left" style="width: 20%">Scope</th> - <th class="text-left" style="width: 25%">Infix</th> - <th class="text-left" style="width: 23%">Metrics</th> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 22%">Infix</th> + <th class="text-left" style="width: 20%">Metrics</th> <th class="text-left" style="width: 32%">Description</th> + <th class="text-left" style="width: 8%">Type</th> </tr> </thead> <tbody> @@ -681,10 +705,12 @@ Thus, in order to infer the metric identifier: <td rowspan="2">Status.JVM.ClassLoader</td> <td>ClassesLoaded</td> <td>The total number of classes loaded since the start of the JVM.</td> + <td>Gauge</td> </tr> <tr> <td>ClassesUnloaded</td> <td>The total number of classes unloaded since the start of the JVM.</td> + <td>Gauge</td> </tr> </tbody> </table> @@ -693,10 +719,11 @@ Thus, in order to infer the metric identifier: <table class="table table-bordered"> <thead> <tr> - <th class="text-left" style="width: 20%">Scope</th> - <th class="text-left" style="width: 25%">Infix</th> - <th class="text-left" style="width: 25%">Metrics</th> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 22%">Infix</th> + <th class="text-left" style="width: 22%">Metrics</th> <th class="text-left" style="width: 30%">Description</th> + <th class="text-left" style="width: 8%">Type</th> </tr> </thead> <tbody> @@ -705,46 +732,56 @@ Thus, in order to infer the metric identifier: <td rowspan="2">Status.Network</td> <td>AvailableMemorySegments</td> <td>The number of unused memory segments.</td> + <td>Gauge</td> </tr> <tr> <td>TotalMemorySegments</td> <td>The number of allocated memory segments.</td> + <td>Gauge</td> </tr> <tr> <th rowspan="8">Task</th> <td rowspan="4">buffers</td> <td>inputQueueLength</td> <td>The number of queued input buffers.</td> + <td>Gauge</td> </tr> <tr> <td>outputQueueLength</td> <td>The number of queued output buffers.</td> + <td>Gauge</td> </tr> <tr> <td>inPoolUsage</td> <td>An estimate of the input buffers usage.</td> + <td>Gauge</td> </tr> <tr> <td>outPoolUsage</td> <td>An estimate of the output buffers usage.</td> + <td>Gauge</td> </tr> <tr> <td rowspan="4">Network.<Input|Output>.<gate><br /> <strong>(only available if <tt>taskmanager.net.detailed-metrics</tt> config option is set)</strong></td> <td>totalQueueLen</td> <td>Total number of queued buffers in all input/output channels.</td> + <td>Gauge</td> </tr> <tr> <td>minQueueLen</td> <td>Minimum number of queued buffers in all input/output channels.</td> + <td>Gauge</td> </tr> <tr> <td>maxQueueLen</td> <td>Maximum number of queued buffers in all input/output channels.</td> + <td>Gauge</td> </tr> <tr> <td>avgQueueLen</td> <td>Average number of queued buffers in all input/output channels.</td> + <td>Gauge</td> </tr> </tbody> </table> @@ -753,9 +790,10 @@ Thus, in order to infer the metric identifier: <table class="table table-bordered"> <thead> <tr> - <th class="text-left" style="width: 20%">Scope</th> - <th class="text-left" style="width: 30%">Metrics</th> - <th class="text-left" style="width: 50%">Description</th> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 26%">Metrics</th> + <th class="text-left" style="width: 48%">Description</th> + <th class="text-left" style="width: 8%">Type</th> </tr> </thead> <tbody> @@ -763,18 +801,22 @@ Thus, in order to infer the metric identifier: <th rowspan="4"><strong>JobManager</strong></th> <td>numRegisteredTaskManagers</td> <td>The number of registered taskmanagers.</td> + <td>Gauge</td> </tr> <tr> <td>numRunningJobs</td> <td>The number of running jobs.</td> + <td>Gauge</td> </tr> <tr> <td>taskSlotsAvailable</td> <td>The number of available task slots.</td> + <td>Gauge</td> </tr> <tr> <td>taskSlotsTotal</td> <td>The total number of task slots.</td> + <td>Gauge</td> </tr> </tbody> </table> @@ -783,34 +825,39 @@ Thus, in order to infer the metric identifier: <table class="table table-bordered"> <thead> <tr> - <th class="text-left" style="width: 20%">Scope</th> - <th class="text-left" style="width: 30%">Metrics</th> - <th class="text-left" style="width: 50%">Description</th> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 26%">Metrics</th> + <th class="text-left" style="width: 48%">Description</th> + <th class="text-left" style="width: 8%">Type</th> </tr> </thead> <tbody> <tr> <th rowspan="4"><strong>Job (only available on JobManager)</strong></th> <td>restartingTime</td> - <td>The time it took to restart the job, or how long the current restart has been in progress.</td> + <td>The time it took to restart the job, or how long the current restart has been in progress (in milliseconds).</td> + <td>Gauge</td> </tr> <tr> <td>uptime</td> <td> The time that the job has been running without interruption. - <p>Returns -1 for completed jobs.</p> + <p>Returns -1 for completed jobs (in milliseconds).</p> </td> + <td>Gauge</td> </tr> <tr> <td>downtime</td> <td> For jobs currently in a failing/recovering situation, the time elapsed during this outage. - <p>Returns 0 for running jobs and -1 for completed jobs.</p> + <p>Returns 0 for running jobs and -1 for completed jobs (in milliseconds).</p> </td> + <td>Gauge</td> </tr> <tr> <td>fullRestarts</td> - <td>The total number of full restarts since this job was submitted.</td> + <td>The total number of full restarts since this job was submitted (in milliseconds).</td> + <td>Gauge</td> </tr> </tbody> </table> @@ -819,53 +866,64 @@ Thus, in order to infer the metric identifier: <table class="table table-bordered"> <thead> <tr> - <th class="text-left" style="width: 20%">Scope</th> - <th class="text-left" style="width: 30%">Metrics</th> - <th class="text-left" style="width: 50%">Description</th> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 26%">Metrics</th> + <th class="text-left" style="width: 48%">Description</th> + <th class="text-left" style="width: 8%">Type</th> </tr> </thead> <tbody> <tr> <th rowspan="9"><strong>Job (only available on JobManager)</strong></th> <td>lastCheckpointDuration</td> - <td>The time it took to complete the last checkpoint.</td> + <td>The time it took to complete the last checkpoint (in milliseconds).</td> + <td>Gauge</td> </tr> <tr> <td>lastCheckpointSize</td> - <td>The total size of the last checkpoint.</td> + <td>The total size of the last checkpoint (in bytes).</td> + <td>Gauge</td> </tr> <tr> <td>lastCheckpointExternalPath</td> <td>The path where the last external checkpoint was stored.</td> + <td>Gauge</td> </tr> <tr> <td>lastCheckpointRestoreTimestamp</td> - <td>Timestamp when the last checkpoint was restored at the coordinator.</td> + <td>Timestamp when the last checkpoint was restored at the coordinator (in milliseconds).</td> + <td>Gauge</td> </tr> <tr> <td>lastCheckpointAlignmentBuffered</td> - <td>The number of buffered bytes during alignment over all subtasks for the last checkpoint.</td> + <td>The number of buffered bytes during alignment over all subtasks for the last checkpoint (in bytes).</td> + <td>Gauge</td> </tr> <tr> <td>numberOfInProgressCheckpoints</td> <td>The number of in progress checkpoints.</td> + <td>Gauge</td> </tr> <tr> <td>numberOfCompletedCheckpoints</td> <td>The number of successfully completed checkpoints.</td> + <td>Gauge</td> </tr> <tr> <td>numberOfFailedCheckpoints</td> <td>The number of failed checkpoints.</td> + <td>Gauge</td> </tr> <tr> <td>totalNumberOfCheckpoints</td> <td>The number of total checkpoints (in progress, completed, failed).</td> + <td>Gauge</td> </tr> <tr> <th rowspan="1">Task</th> <td>checkpointAlignmentTime</td> - <td>The time in nanoseconds that the last barrier alignment took to complete, or how long the current alignment has taken so far.</td> + <td>The time in nanoseconds that the last barrier alignment took to complete, or how long the current alignment has taken so far (in nanoseconds).</td> + <td>Gauge</td> </tr> </tbody> </table> @@ -874,57 +932,69 @@ Thus, in order to infer the metric identifier: <table class="table table-bordered"> <thead> <tr> - <th class="text-left" style="width: 20%">Scope</th> - <th class="text-left" style="width: 30%">Metrics</th> - <th class="text-left" style="width: 50%">Description</th> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 26%">Metrics</th> + <th class="text-left" style="width: 48%">Description</th> + <th class="text-left" style="width: 8%">Type</th> </tr> </thead> <tbody> <tr> <th rowspan="7"><strong>Task</strong></th> <td>currentLowWatermark</td> - <td>The lowest watermark this task has received.</td> + <td>The lowest watermark this task has received (in milliseconds).</td> + <td>Gauge</td> </tr> <tr> <td>numBytesInLocal</td> <td>The total number of bytes this task has read from a local source.</td> + <td>Counter</td> </tr> <tr> <td>numBytesInLocalPerSecond</td> <td>The number of bytes this task reads from a local source per second.</td> + <td>Meter</td> </tr> <tr> <td>numBytesInRemote</td> <td>The total number of bytes this task has read from a remote source.</td> + <td>Counter</td> </tr> <tr> <td>numBytesInRemotePerSecond</td> <td>The number of bytes this task reads from a remote source per second.</td> + <td>Meter</td> </tr> <tr> <td>numBytesOut</td> <td>The total number of bytes this task has emitted.</td> + <td>Counter</td> </tr> <tr> <td>numBytesOutPerSecond</td> <td>The number of bytes this task emits per second.</td> + <td>Meter</td> </tr> <tr> <th rowspan="5"><strong>Task/Operator</strong></th> <td>numRecordsIn</td> <td>The total number of records this operator/task has received.</td> + <td>Counter</td> </tr> <tr> <td>numRecordsInPerSecond</td> <td>The number of records this operator/task receives per second.</td> + <td>Meter</td> </tr> <tr> <td>numRecordsOut</td> <td>The total number of records this operator/task has emitted.</td> + <td>Counter</td> </tr> <tr> <td>numRecordsOutPerSecond</td> <td>The number of records this operator/task sends per second.</td> + <td>Meter</td> </tr> <tr> <td>numLateRecordsDropped</td> @@ -933,11 +1003,13 @@ Thus, in order to infer the metric identifier: <tr> <th rowspan="2"><strong>Operator</strong></th> <td>latency</td> - <td>The latency distributions from all incoming sources.</td> + <td>The latency distributions from all incoming sources (in milliseconds).</td> + <td>Histogram</td> </tr> <tr> <td>numSplitsProcessed</td> <td>The total number of InputSplits this data source has processed (if the operator is a data source).</td> + <td>Gauge</td> </tr> </tbody> </table> @@ -948,9 +1020,10 @@ Thus, in order to infer the metric identifier: <table class="table table-bordered"> <thead> <tr> - <th class="text-left" style="width: 20%">Scope</th> - <th class="text-left" style="width: 30%">Metrics</th> - <th class="text-left" style="width: 50%">Description</th> + <th class="text-left" style="width: 18%">Scope</th> + <th class="text-left" style="width: 26%">Metrics</th> + <th class="text-left" style="width: 48%">Description</th> + <th class="text-left" style="width: 8%">Type</th> </tr> </thead> <tbody> @@ -958,11 +1031,13 @@ Thus, in order to infer the metric identifier: <th rowspan="1">Operator</th> <td>commitsSucceeded</td> <td>Kafka offset commit success count if Kafka commit is turned on and checkpointing is enabled.</td> + <td>Counter</td> </tr> <tr> <th rowspan="1">Operator</th> <td>commitsFailed</td> <td>Kafka offset commit failure count if Kafka commit is turned on and checkpointing is enabled.</td> + <td>Counter</td> </tr> </tbody> </table>
