[
https://issues.apache.org/jira/browse/KAFKA-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17293722#comment-17293722
]
ASF GitHub Bot commented on KAFKA-12393:
----------------------------------------
miguno commented on a change in pull request #334:
URL: https://github.com/apache/kafka-site/pull/334#discussion_r585575690
##########
File path: 27/ops.html
##########
@@ -1090,7 +1090,157 @@ <h4 class="anchor-heading"><a
id="georeplication-monitoring" class="anchor-link"
</p>
- <h3 class="anchor-heading"><a id="config" class="anchor-link"></a><a
href="#config">6.4 Kafka Configuration</a></h3>
+ <h3 class="anchor-heading"><a id="multitenancy" class="anchor-link"></a><a
href="#multitenancy">6.4 Multi-Tenancy</a></h3>
+
+ <h4 class="anchor-heading"><a id="multitenancy-overview"
class="anchor-link"></a><a href="#multitenancy-overview">Multi-Tenancy
Overview</a></h4>
+
+ <p>
+ As a highly scalable event streaming platform, Kafka is used by many users
as their central nervous system, connecting in real-time a wide range of
different systems and applications from various teams and lines of businesses.
Such multi-tenant cluster environments command proper control and management to
ensure the peaceful coexistence of these different needs. This section
highlights features and best practices to set up such shared environments,
which should help you operate clusters that meet SLAs/OLAs and that minimize
potential collateral damage caused by "noisy neighbors".
+ </p>
+
+ <p>
+ Multi-tenancy is a many-sided subject, including but not limited to:
+ </p>
+
+ <ul>
+ <li>Creating user spaces for tenants (sometimes called namespaces)</li>
+ <li>Configuring topics with data retention policies and more</li>
+ <li>Securing topics and clusters with encryption, authentication, and
authorization</li>
+ <li>Isolating tenants with quotas and rate limits</li>
+ <li>Monitoring and metering</li>
+ <li>Inter-cluster data sharing (cf. geo-replication)</li>
+ </ul>
+
+ <h4 class="anchor-heading"><a id="multitenancy-topic-naming"
class="anchor-link"></a><a href="#multitenancy-topic-naming">Creating User
Spaces (Namespaces) For Tenants With Topic Naming</a></h4>
+
+ <p>
+ Kafka administrators operating a multi-tenant cluster typically need to
define user spaces for each tenant. For the purpose of this section, "user
spaces" are a collection of topics, which are grouped together under the
management of a single entity or user.
+ </p>
+
+ <p>
+ In Kafka, the main unit of data is the topic. Users can create and name
each topic. They can also delete them, but it is not possible to rename a topic
directly. Instead, to rename a topic, the user must create a new topic, move
the messages from the original topic to the new, and then delete the original.
With this in mind, it is recommended to define logical spaces, based on an
hierarchical topic naming structure. This setup can then be combined with
security features, such as prefixed ACLs, to isolate different spaces and
tenants, while also minimizing the administrative overhead for securing the
data in the cluster.
+ </p>
+
+ <p>
+ These logical user spaces can be grouped in different ways, and the
concrete choice depends on how your organization prefers to use your Kafka
clusters. The most common groupings are as follows.
+ </p>
+
+ <p>
+ <em>By team or organizational unit:</em> Here, the team is the main
aggregator. In an organization where teams are the main user of the Kafka
infrastructure, this might be the best grouping.
+ </p>
+
+ <p>
+ Example topic naming structure:
+ </p>
+
+ <ul>
+
<li><code><organization>.<team>.<dataset>.<event-name></code><br
/>(e.g., "acme.infosec.telemetry.logins")</li>
+ </ul>
+
+ <p>
+ <em>By project or product:</em> Here, a team manages more than one
project. Their credentials will be different for each project, so all the
controls and settings will always be project related.
+ </p>
+
+ <p>
+ Example topic naming structure:
+ </p>
+
+ <ul>
+ <li><code><project>.<product>.<event-name></code><br
/>(e.g., "mobility.payments.suspicious")</li>
+ </ul>
+
+ <p>
+ Certain information should normally not be put in a topic name, such as
information that is likely to change over time (e.g., the name of the intended
consumer) or that is a technical detail or metadata that is available elsewhere
(e.g., the topic's partition count and other configuration settings).
+ </p>
+
+ <p>
+ To enforce a topic naming structure, it is useful to disable the Kafka
feature to auto-create topics on demand by setting
<code>auto.create.topics.enable=false</code> in the broker configuration. This
stops users and applications from deliberately or inadvertently creating topics
with arbitrary names, thus violating the naming structure. Then, you may want
to put in place your own organizational process for controlled, yet automated
creation of topics according to your naming convention, using scripting or your
favorite automation toolkit.
+ </p>
+
+ <h4 class="anchor-heading"><a id="multitenancy-topic-configs"
class="anchor-link"></a><a href="#multitenancy-topic-configs">Configuring
Topics: Data Retention And More</a></h4>
+
+ <p>
+ Kafka's configuration is very flexible due to its fine granularity, and it
supports a plethora of <a href="#topicconfigs">per-topic configuration
settings</a> to help administrators set up multi-tenant clusters. For example,
administrators often need to define data retention policies to control how much
and/or for how long data will be stored in a topic, with settings such as <a
href="#retention.bytes">retention.bytes</a> (size) and <a
href="#retention.ms">retention.ms</a> (time). This limits storage consumption
within the cluster, and helps complying with legal requirements such as GDPR.
+ </p>
+
+ <h4 class="anchor-heading"><a id="multitenancy-security"
class="anchor-link"></a><a href="#multitenancy-security">Securing Clusters and
Topics: Authentication, Authorization, Encryption</a></h4>
+
+ <p>
+ Because the documentation has a dedicated chapter on <a
href="#security">security</a> that applies to any Kafka deployment, this
section focuses on additional considerations for multi-tenant environments.
+ </p>
+
+ <p>
+Security settings for Kafka fall into three main categories, which are similar
to how administrators would secure other client-server data systems, like
relational databases and traditional messaging systems.
+ </p>
+
+ <ol>
+ <li><strong>Encryption</strong> of data transferred between Kafka brokers
and Kafka clients, between brokers, between brokers and ZooKeeper nodes, and
between brokers and other, optional tools.</li>
+ <li><strong>Authentication</strong> of connections from Kafka clients and
applications to Kafka brokers, as well as connections from Kafka brokers to
ZooKeeper nodes.</li>
+ <li><strong>Authorization</strong> of client operations such as creating,
deleting, and altering the configuration of topics; writing events to or
reading events from a topic; creating and deleting ACLs.</li>
Review comment:
Yeah, I can add a note. But it's unfortunate that there's essentially
zero coverage in the AK docs on how to use these.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Document multi-tenancy considerations
> -------------------------------------
>
> Key: KAFKA-12393
> URL: https://issues.apache.org/jira/browse/KAFKA-12393
> Project: Kafka
> Issue Type: Bug
> Components: documentation
> Reporter: Michael G. Noll
> Assignee: Michael G. Noll
> Priority: Minor
>
> We should provide an overview of multi-tenancy consideration (e.g., user
> spaces, security) as the current documentation lacks such information.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
