[ https://issues.apache.org/jira/browse/KAFKA-7800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Manikumar resolved KAFKA-7800. ------------------------------ Fix Version/s: 2.4.0 Resolution: Fixed > Extend Admin API to support dynamic application log levels > ---------------------------------------------------------- > > Key: KAFKA-7800 > URL: https://issues.apache.org/jira/browse/KAFKA-7800 > Project: Kafka > Issue Type: New Feature > Reporter: Stanislav Kozlovski > Assignee: Stanislav Kozlovski > Priority: Major > Fix For: 2.4.0 > > > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-412%3A+Extend+Admin+API+to+support+dynamic+application+log+levels] > Logging is a critical part of any system's infrastructure. It is the most > direct way of observing what is happening with a system. In the case of > issues, it helps us diagnose the problem quickly which in turn helps lower > the > [MTTR|http://enterprisedevops.org/article/devops-metric-mean-time-to-recovery-mttr-definition-and-reasoning]. > Kafka supports application logging via the log4j library and outputs messages > in various log levels (TRACE, DEBUG, INFO, WARN, ERROR). Log4j is a rich > library that supports fine-grained logging configurations (e.g use INFO-level > logging in {{kafka.server.ReplicaManager}} and use DEBUG-level in > {{kafka.server.KafkaApis}}). > This is statically configurable through the > [log4j.properties|https://github.com/apache/kafka/blob/trunk/config/log4j.properties] > file which gets read once at broker start-up. > A problem with this static configuration is that we cannot alter the log > levels when a problem arises. It is severely impractical to edit a properties > file and restart all brokers in order to gain visibility of a problem taking > place in production. > It would be very useful if we support dynamically altering the log levels at > runtime without needing to restart the Kafka process. > Log4j itself supports dynamically altering the log levels in a programmatic > way and Kafka exposes a [JMX > API|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/Log4jController.scala] > that lets you alter them. This allows users to change the log levels via a > GUI (jconsole) or a CLI (jmxterm) that uses JMX. > There is one problem with changing log levels through JMX that we hope to > address and that is *Ease of Use*: > * Establishing a connection - Connecting to a remote process via JMX > requires configuring and exposing multiple JMX ports to the outside world. > This is a burden on users, as most production deployments may stand behind > layers of firewalls and have policies against opening ports. This makes > opening the ports and connections in the middle of an incident even more > burdensome > * Security - JMX and tools around it support authentication and > authorization but it is an additional hassle to set up credentials for > another system. > * Manual process - Changing the whole cluster's log level requires manually > connecting to each broker. In big deployments, this is severely impractical > and forces users to build tooling around it. > h4. Proposition > Ideally, Kafka would support dynamically changing log levels and address all > of the aforementioned concerns out of the box. > We propose extending the IncrementalAlterConfig/DescribeConfig Admin API with > functionality for dynamically altering the broker's log level. > This approach would also pave the way for even finer-grained logging logic > (e.g log DEBUG level only for a certain topic) and would allow us to leverage > the existing *AlterConfigPolicy* for custom user-defined validation of > log-level changes. > These log-level changes will be *temporary* and reverted on broker restart - > we will not persist them anywhere. -- This message was sent by Atlassian Jira (v8.3.4#803005)