Repository: kafka Updated Branches: refs/heads/0.10.2 f6226777d -> f6d07e206
MINOR: Stream metrics documentation Author: Eno Thereska <[email protected]> Reviewers: Matthias J. Sax, Guozhang Wang Closes #2542 from enothereska/minor-streams-metrics (cherry picked from commit 2e1b00f262947c4c3211df205df259ef517763f4) Signed-off-by: Guozhang Wang <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/kafka/repo Commit: http://git-wip-us.apache.org/repos/asf/kafka/commit/f6d07e20 Tree: http://git-wip-us.apache.org/repos/asf/kafka/tree/f6d07e20 Diff: http://git-wip-us.apache.org/repos/asf/kafka/diff/f6d07e20 Branch: refs/heads/0.10.2 Commit: f6d07e206799f1caabccc3e2a30eea62e4d4881d Parents: f622677 Author: Eno Thereska <[email protected]> Authored: Wed Feb 15 16:32:02 2017 -0800 Committer: Guozhang Wang <[email protected]> Committed: Wed Feb 15 16:32:12 2017 -0800 ---------------------------------------------------------------------- docs/ops.html | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 124 insertions(+), 6 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/kafka/blob/f6d07e20/docs/ops.html ---------------------------------------------------------------------- diff --git a/docs/ops.html b/docs/ops.html index a3423a7..9232f65 100644 --- a/docs/ops.html +++ b/docs/ops.html @@ -14,8 +14,8 @@ See the License for the specific language governing permissions and limitations under the License. --> - <script id="ops-template" type="text/x-handlebars-template"> + Here is some information on actually running Kafka as a production system based on usage and experience at LinkedIn. Please send us any additional tips you know of. <h3><a id="basic_ops" href="#basic_ops">6.1 Basic Kafka Operations</a></h3> @@ -842,9 +842,9 @@ </tr> </tbody></table> - <h4><a id="selector_monitoring" href="#selector_monitoring">Common monitoring metrics for producer/consumer/connect</a></h4> + <h4><a id="selector_monitoring" href="#selector_monitoring">Common monitoring metrics for producer/consumer/connect/streams</a></h4> - The following metrics are available on producer/consumer/connector instances. For specific metrics, please see following sections. + The following metrics are available on producer/consumer/connector/streams instances. For specific metrics, please see following sections. <table class="data-table"> <tbody> @@ -931,9 +931,9 @@ </tbody> </table> - <h4><a id="common_node_monitoring" href="#common_node_monitoring">Common Per-broker metrics for producer/consumer/connect</a></h4> + <h4><a id="common_node_monitoring" href="#common_node_monitoring">Common Per-broker metrics for producer/consumer/connect/streams</a></h4> - The following metrics are available on producer/consumer/connector instances. For specific metrics, please see following sections. + The following metrics are available on producer/consumer/connector/streams instances. For specific metrics, please see following sections. <table class="data-table"> <tbody> @@ -1314,7 +1314,125 @@ </tbody> </table> - <h5><a id="others_monitoring" href="#others_monitoring">Others</a></h5> + + + <h4><a id="kafka_streams_monitoring" href="#kafka_streams_monitoring">Streams Monitoring</a></h4> + + A Kafka Streams instance contains all the producer and consumer metrics as well as additional metrics specific to streams. By default Kafka Streams has metrics with two recording levels: debug and info. The debug level records all metrics, while the info level records only the thread-level metrics. Use the following configuration option to specify which metrics you want collected: +<pre>metrics.recording.level="info"</pre> + +<h5><a id="kafka_streams_thread_monitoring" href="#kafka_streams_thread_monitoring">Thread Metrics</a></h5> +All the following metrics have a recording level of ``info``: +<table class="data-table"> + <tbody> + <tr> + <th>Metric/Attribute name</th> + <th>Description</th> + <th>Mbean name</th> + </tr> + <tr> + <td>[commit | poll | process | punctuate]-latency-[avg | max]</td> + <td>The [average | maximum] execution time in ms, for the respective operation, across all running tasks of this thread.</td> + <td>kafka.streams:type=stream-metrics,thread.client-id=([-.\w]+)</td> + </tr> + <tr> + <td>[commit | poll | process | punctuate]-rate</td> + <td>The average number of respective operations per second across all tasks.</td> + <td>kafka.streams:type=stream-metrics,thread.client-id=([-.\w]+)</td> + </tr> + <tr> + <td>task-created-rate</td> + <td>The average number of newly created tasks per second.</td> + <td>kafka.streams:type=stream-metrics,thread.client-id=([-.\w]+)</td> + </tr> + <tr> + <td>task-closed-rate</td> + <td>The average number of tasks closed per second.</td> + <td>kafka.streams:type=stream-metrics,thread.client-id=([-.\w]+)</td> + </tr> + <tr> + <td>skipped-records-rate</td> + <td>The average number of skipped records per second. </td> + <td>kafka.streams:type=stream-metrics,thread.client-id=([-.\w]+)</td> + </tr> + </tbody> +</table> + +<h5><a id="kafka_streams_task_monitoring" href="#kafka_streams_task_monitoring">Task Metrics</a></h5> +All the following metrics have a recording level of ``debug``: + <table class="data-table"> + <tbody> + <tr> + <th>Metric/Attribute name</th> + <th>Description</th> + <th>Mbean name</th> + </tr> + <tr> + <td>commit-latency-[avg | max]</td> + <td>The [average | maximum] commit time in ns for this task. </td> + <td>kafka.streams:type=stream-task-metrics,streams-task-id=([-.\w]+)</td> + </tr> + <tr> + <td>commit-rate</td> + <td>The average number of commit calls per second. </td> + <td>kafka.streams:type=stream-task-metrics,streams-task-id=([-.\w]+)</td> + </tr> + </tbody> +</table> + + <h5><a id="kafka_streams_node_monitoring" href="#kafka_streams_node_monitoring">Processor Node Metrics</a></h5> +All the following metrics have a recording level of ``debug``: + <table class="data-table"> + <tbody> + <tr> + <th>Metric/Attribute name</th> + <th>Description</th> + <th>Mbean name</th> + </tr> + <tr> + <td>[process | punctuate | create | destroy]-latency-[avg | max]</td> + <td>The [average | maximum] execution time in ns, for the respective operation. </td> + <td>kafka.streams:type=stream-processor-node-metrics, processor-node-id=([-.\w]+)</td> + </tr> + <tr> + <td>[process | punctuate | create | destroy]-rate</td> + <td>The average number of respective operations per second. </td> + <td>kafka.streams:type=stream-processor-node-metrics, processor-node-id=([-.\w]+)</td> + </tr> + <tr> + <td>forward-rate</td> + <td>The average rate of records being forwarded downstream, from source nodes only, per second. </td> + <td>kafka.streams:type=stream-processor-node-metrics, processor-node-id=([-.\w]+)</td> + </tr> + </tbody> + </table> + + <h5><a id="kafka_streams_store_monitoring" href="#kafka_streams_store_monitoring">State Store Metrics</a></h5> +All the following metrics have a recording level of ``debug``: + + <table class="data-table"> + <tbody> + <tr> + <th>Metric/Attribute name</th> + <th>Description</th> + <th>Mbean name</th> + </tr> + <tr> + <td>[put | put-if-absent | get | delete | put-all | all | range | flush | restore]-latency-[avg | max]</td> + <td>The average execution time in ns, for the respective operation. </td> + <td>kafka.streams:type=stream-[store-type]-metrics</td> + </tr> + <tr> + <td>[put | put-if-absent | get | delete | put-all | all | range | flush | restore]-rate</td> + <td>The average rate of respective operations per second for this store.</td> + <td>kafka.streams:type=stream-[store-type]-metrics</td> + </tr> + + </tbody> +</table> + + + <h4><a id="others_monitoring" href="#others_monitoring">Others</a></h4> We recommend monitoring GC time and other stats and various server stats such as CPU utilization, I/O service time, etc.
