miguno commented on a change in pull request #334: URL: https://github.com/apache/kafka-site/pull/334#discussion_r585572446
########## File path: 27/ops.html ########## @@ -1090,7 +1090,157 @@ <h4 class="anchor-heading"><a id="georeplication-monitoring" class="anchor-link" </p> - <h3 class="anchor-heading"><a id="config" class="anchor-link"></a><a href="#config">6.4 Kafka Configuration</a></h3> + <h3 class="anchor-heading"><a id="multitenancy" class="anchor-link"></a><a href="#multitenancy">6.4 Multi-Tenancy</a></h3> + + <h4 class="anchor-heading"><a id="multitenancy-overview" class="anchor-link"></a><a href="#multitenancy-overview">Multi-Tenancy Overview</a></h4> + + <p> + As a highly scalable event streaming platform, Kafka is used by many users as their central nervous system, connecting in real-time a wide range of different systems and applications from various teams and lines of businesses. Such multi-tenant cluster environments command proper control and management to ensure the peaceful coexistence of these different needs. This section highlights features and best practices to set up such shared environments, which should help you operate clusters that meet SLAs/OLAs and that minimize potential collateral damage caused by "noisy neighbors". + </p> + + <p> + Multi-tenancy is a many-sided subject, including but not limited to: + </p> + + <ul> + <li>Creating user spaces for tenants (sometimes called namespaces)</li> + <li>Configuring topics with data retention policies and more</li> + <li>Securing topics and clusters with encryption, authentication, and authorization</li> + <li>Isolating tenants with quotas and rate limits</li> + <li>Monitoring and metering</li> + <li>Inter-cluster data sharing (cf. geo-replication)</li> + </ul> + + <h4 class="anchor-heading"><a id="multitenancy-topic-naming" class="anchor-link"></a><a href="#multitenancy-topic-naming">Creating User Spaces (Namespaces) For Tenants With Topic Naming</a></h4> + + <p> + Kafka administrators operating a multi-tenant cluster typically need to define user spaces for each tenant. For the purpose of this section, "user spaces" are a collection of topics, which are grouped together under the management of a single entity or user. + </p> + + <p> + In Kafka, the main unit of data is the topic. Users can create and name each topic. They can also delete them, but it is not possible to rename a topic directly. Instead, to rename a topic, the user must create a new topic, move the messages from the original topic to the new, and then delete the original. With this in mind, it is recommended to define logical spaces, based on an hierarchical topic naming structure. This setup can then be combined with security features, such as prefixed ACLs, to isolate different spaces and tenants, while also minimizing the administrative overhead for securing the data in the cluster. + </p> + + <p> + These logical user spaces can be grouped in different ways, and the concrete choice depends on how your organization prefers to use your Kafka clusters. The most common groupings are as follows. + </p> + + <p> + <em>By team or organizational unit:</em> Here, the team is the main aggregator. In an organization where teams are the main user of the Kafka infrastructure, this might be the best grouping. + </p> + + <p> + Example topic naming structure: + </p> + + <ul> + <li><code><organization>.<team>.<dataset>.<event-name></code><br />(e.g., "acme.infosec.telemetry.logins")</li> + </ul> + + <p> + <em>By project or product:</em> Here, a team manages more than one project. Their credentials will be different for each project, so all the controls and settings will always be project related. + </p> + + <p> + Example topic naming structure: + </p> + + <ul> + <li><code><project>.<product>.<event-name></code><br />(e.g., "mobility.payments.suspicious")</li> + </ul> + + <p> + Certain information should normally not be put in a topic name, such as information that is likely to change over time (e.g., the name of the intended consumer) or that is a technical detail or metadata that is available elsewhere (e.g., the topic's partition count and other configuration settings). + </p> + + <p> + To enforce a topic naming structure, it is useful to disable the Kafka feature to auto-create topics on demand by setting <code>auto.create.topics.enable=false</code> in the broker configuration. This stops users and applications from deliberately or inadvertently creating topics with arbitrary names, thus violating the naming structure. Then, you may want to put in place your own organizational process for controlled, yet automated creation of topics according to your naming convention, using scripting or your favorite automation toolkit. + </p> + + <h4 class="anchor-heading"><a id="multitenancy-topic-configs" class="anchor-link"></a><a href="#multitenancy-topic-configs">Configuring Topics: Data Retention And More</a></h4> + + <p> + Kafka's configuration is very flexible due to its fine granularity, and it supports a plethora of <a href="#topicconfigs">per-topic configuration settings</a> to help administrators set up multi-tenant clusters. For example, administrators often need to define data retention policies to control how much and/or for how long data will be stored in a topic, with settings such as <a href="#retention.bytes">retention.bytes</a> (size) and <a href="#retention.ms">retention.ms</a> (time). This limits storage consumption within the cluster, and helps complying with legal requirements such as GDPR. + </p> + + <h4 class="anchor-heading"><a id="multitenancy-security" class="anchor-link"></a><a href="#multitenancy-security">Securing Clusters and Topics: Authentication, Authorization, Encryption</a></h4> + + <p> + Because the documentation has a dedicated chapter on <a href="#security">security</a> that applies to any Kafka deployment, this section focuses on additional considerations for multi-tenant environments. + </p> + + <p> +Security settings for Kafka fall into three main categories, which are similar to how administrators would secure other client-server data systems, like relational databases and traditional messaging systems. + </p> + + <ol> + <li><strong>Encryption</strong> of data transferred between Kafka brokers and Kafka clients, between brokers, between brokers and ZooKeeper nodes, and between brokers and other, optional tools.</li> + <li><strong>Authentication</strong> of connections from Kafka clients and applications to Kafka brokers, as well as connections from Kafka brokers to ZooKeeper nodes.</li> + <li><strong>Authorization</strong> of client operations such as creating, deleting, and altering the configuration of topics; writing events to or reading events from a topic; creating and deleting ACLs.</li> + </ol> + + <p> + When securing a multi-tenant Kafka environment, the most common administrative task is the third category (authorization), i.e., managing the user/client permissions that grant or deny access to certain topics and thus to the data stored by users within a cluster. This task is performed predominantly through the <a href="#security_authz">setting of access control lists (ACLs)</a>. Here, administrators of multi-tenant environments in particular benefit from putting a hierarchical topic naming structure in place as described in a previous section, because they can conveniently control access to topics through prefixed ACLs (<code>--resource-pattern-type Prefixed</code>). This significantly minimizes the administrative overhead of securing topics in multi-tenant environments: administrators can make their own trade-offs between higher developer convenience (more lenient permissions, using fewer and broader ACLs) vs. tighter security (more stringent permissions, using more and narrowe r ACLs). + </p> + + <p> + In the following example, user Alice—a new member of ACME corporation's InfoSec team—is granted write permissions to all topics whose names start with "acme.infosec.", such as "acme.infosec.telemetry.logins" and "acme.infosec.syslogs.events". + </p> + +<pre class="line-numbers"><code class="language-text"># Grant permissions to user Alice +$ bin/kafka-acls.sh \ + --bootstrap-server broker1:9092 \ + --add --allow-principal User:Alice \ + --producer \ + --resource-pattern-type prefixed --topic acme.infosec. +</code></pre> + + <p> + You can similarly use this approach to isolate different customers on the same shared cluster. + </p> + + <h4 class="anchor-heading"><a id="multitenancy-isolation" class="anchor-link"></a><a href="#multitenancy-isolation">Isolating Tenants: Quotas, Rate Limiting, Throttling</a></h4> + + <p> + Multi-tenant clusters should generally be configured with <a href="#design_quotas">quotas</a>, which protect against users (tenants) eating up too many cluster resources, such as when they attempt to write or read very high volumes of data, or create requests to brokers at an excessively high rate. This may cause network saturation, monopolize broker resources, and impact other clients—all of which you want to avoid in a shared environment. + </p> + + <p> + <strong>Client quotas:</strong> Kafka supports different types of (per-user principal) client quotas. Because a client's quotas apply irrespective of which topics the client is writing to or reading from, they are a convenient and effective tool to allocate resources in a multi-tenant cluster. <a href="#design_quotascpu">Request rate quotas</a>, for example, help to limit a user's impact on broker CPU usage by limiting the time a broker spends on the <a href="/protocol.html">request handling path</a> for that user, after which throttling kicks in. In many situations, isolating users with request rate quotas has a bigger impact in multi-tenant clusters than setting incoming/outgoing network bandwidth quotas, because excessive broker CPU usage for processing requests reduces the effective bandwidth the broker can serve. Furthermore, administrators can also define <a href="#brokerconfigs_controller.quota.window.num">quotas on topic operations</a> such as create, delete, and alter t o prevent Kafka clusters from being overwhelmed by highly concurrent topic operations (see <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-599%3A+Throttle+Create+Topic%2C+Create+Partition+and+Delete+Topic+Operations";>KIP-599</a>). + </p> + + <p> + <strong>Server quotas:</strong> In addition to client-side quotas, Kafka supports different types of broker-side quotas. For example, administrators can set a limit on the rate with which the <a href="#brokerconfigs_max.connection.creation.rate">broker accepts new connections</a>, set the <a href="#brokerconfigs_max.connections">maximum number of connections per broker</a>, or set the maximum number of connections allowed <a href="#brokerconfigs_max.connections.per.ip">from a specific IP address</a>. Review comment: Ack and updated ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
