This is an automated email from the ASF dual-hosted git repository.
bbejeck pushed a commit to branch 2.8
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/2.8 by this push:
new 40ea91a KAFKA-12393: Document multi-tenancy considerations (#334)
(#10263)
40ea91a is described below
commit 40ea91aa294534bd42eef88395a0b2b110e92f65
Author: Michael G. Noll <mig...@miguno.com>
AuthorDate: Thu Mar 4 16:47:48 2021 +0100
KAFKA-12393: Document multi-tenancy considerations (#334) (#10263)
KAFKA-12393: Document multi-tenancy considerations
Addressed review feedback by @dajac and @rajinisivaram
Ported from apache/kafka-site#334
Reviewers: Bill Bejeck <bbej...@apache.org>
---
docs/ops.html | 168 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
docs/toc.html | 21 ++++++--
2 files changed, 179 insertions(+), 10 deletions(-)
diff --git a/docs/ops.html b/docs/ops.html
index 6762b6d..5c2d911 100644
--- a/docs/ops.html
+++ b/docs/ops.html
@@ -1089,7 +1089,165 @@ checkpoint-latency-ms-avg
</p>
- <h3 class="anchor-heading"><a id="config" class="anchor-link"></a><a
href="#config">6.4 Kafka Configuration</a></h3>
+ <h3 class="anchor-heading"><a id="multitenancy" class="anchor-link"></a><a
href="#multitenancy">6.4 Multi-Tenancy</a></h3>
+
+ <h4 class="anchor-heading"><a id="multitenancy-overview"
class="anchor-link"></a><a href="#multitenancy-overview">Multi-Tenancy
Overview</a></h4>
+
+ <p>
+ As a highly scalable event streaming platform, Kafka is used by many users
as their central nervous system, connecting in real-time a wide range of
different systems and applications from various teams and lines of businesses.
Such multi-tenant cluster environments command proper control and management to
ensure the peaceful coexistence of these different needs. This section
highlights features and best practices to set up such shared environments,
which should help you operate clust [...]
+ </p>
+
+ <p>
+ Multi-tenancy is a many-sided subject, including but not limited to:
+ </p>
+
+ <ul>
+ <li>Creating user spaces for tenants (sometimes called namespaces)</li>
+ <li>Configuring topics with data retention policies and more</li>
+ <li>Securing topics and clusters with encryption, authentication, and
authorization</li>
+ <li>Isolating tenants with quotas and rate limits</li>
+ <li>Monitoring and metering</li>
+ <li>Inter-cluster data sharing (cf. geo-replication)</li>
+ </ul>
+
+ <h4 class="anchor-heading"><a id="multitenancy-topic-naming"
class="anchor-link"></a><a href="#multitenancy-topic-naming">Creating User
Spaces (Namespaces) For Tenants With Topic Naming</a></h4>
+
+ <p>
+ Kafka administrators operating a multi-tenant cluster typically need to
define user spaces for each tenant. For the purpose of this section, "user
spaces" are a collection of topics, which are grouped together under the
management of a single entity or user.
+ </p>
+
+ <p>
+ In Kafka, the main unit of data is the topic. Users can create and name
each topic. They can also delete them, but it is not possible to rename a topic
directly. Instead, to rename a topic, the user must create a new topic, move
the messages from the original topic to the new, and then delete the original.
With this in mind, it is recommended to define logical spaces, based on an
hierarchical topic naming structure. This setup can then be combined with
security features, such as pref [...]
+ </p>
+
+ <p>
+ These logical user spaces can be grouped in different ways, and the
concrete choice depends on how your organization prefers to use your Kafka
clusters. The most common groupings are as follows.
+ </p>
+
+ <p>
+ <em>By team or organizational unit:</em> Here, the team is the main
aggregator. In an organization where teams are the main user of the Kafka
infrastructure, this might be the best grouping.
+ </p>
+
+ <p>
+ Example topic naming structure:
+ </p>
+
+ <ul>
+
<li><code><organization>.<team>.<dataset>.<event-name></code><br
/>(e.g., "acme.infosec.telemetry.logins")</li>
+ </ul>
+
+ <p>
+ <em>By project or product:</em> Here, a team manages more than one
project. Their credentials will be different for each project, so all the
controls and settings will always be project related.
+ </p>
+
+ <p>
+ Example topic naming structure:
+ </p>
+
+ <ul>
+ <li><code><project>.<product>.<event-name></code><br
/>(e.g., "mobility.payments.suspicious")</li>
+ </ul>
+
+ <p>
+ Certain information should normally not be put in a topic name, such as
information that is likely to change over time (e.g., the name of the intended
consumer) or that is a technical detail or metadata that is available elsewhere
(e.g., the topic's partition count and other configuration settings).
+ </p>
+
+ <p>
+ To enforce a topic naming structure, several options are available:
+ </p>
+
+ <ul>
+ <li>Use <a href="#security_authz">prefix ACLs</a> (cf. <a
href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%3A+Support+for+Prefixed+ACLs";>KIP-290</a>)
to enforce a common prefix for topic names. For example, team A may only be
permitted to create topics whose names start with
<code>payments.teamA.</code>.</li>
+ <li>Define a custom <code>CreateTopicPolicy</code> (cf. <a
href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-108%3A+Create+Topic+Policy";>KIP-108</a>
and the setting <a
href="#brokerconfigs_create.topic.policy.class.name">create.topic.policy.class.name</a>)
to enforce strict naming patterns. These policies provide the most flexibility
and can cover complex patterns and rules to match an organization's needs.</li>
+ <li>Disable topic creation for normal users by denying it with an ACL, and
then rely on an external process to create topics on behalf of users (e.g.,
scripting or your favorite automation toolkit).</li>
+ <li>It may also be useful to disable the Kafka feature to auto-create
topics on demand by setting <code>auto.create.topics.enable=false</code> in the
broker configuration. Note that you should not rely solely on this option.</li>
+ </ul>
+
+
+ <h4 class="anchor-heading"><a id="multitenancy-topic-configs"
class="anchor-link"></a><a href="#multitenancy-topic-configs">Configuring
Topics: Data Retention And More</a></h4>
+
+ <p>
+ Kafka's configuration is very flexible due to its fine granularity, and it
supports a plethora of <a href="#topicconfigs">per-topic configuration
settings</a> to help administrators set up multi-tenant clusters. For example,
administrators often need to define data retention policies to control how much
and/or for how long data will be stored in a topic, with settings such as <a
href="#retention.bytes">retention.bytes</a> (size) and <a
href="#retention.ms">retention.ms</a> (time). Th [...]
+ </p>
+
+ <h4 class="anchor-heading"><a id="multitenancy-security"
class="anchor-link"></a><a href="#multitenancy-security">Securing Clusters and
Topics: Authentication, Authorization, Encryption</a></h4>
+
+ <p>
+ Because the documentation has a dedicated chapter on <a
href="#security">security</a> that applies to any Kafka deployment, this
section focuses on additional considerations for multi-tenant environments.
+ </p>
+
+ <p>
+Security settings for Kafka fall into three main categories, which are similar
to how administrators would secure other client-server data systems, like
relational databases and traditional messaging systems.
+ </p>
+
+ <ol>
+ <li><strong>Encryption</strong> of data transferred between Kafka brokers
and Kafka clients, between brokers, between brokers and ZooKeeper nodes, and
between brokers and other, optional tools.</li>
+ <li><strong>Authentication</strong> of connections from Kafka clients and
applications to Kafka brokers, as well as connections from Kafka brokers to
ZooKeeper nodes.</li>
+ <li><strong>Authorization</strong> of client operations such as creating,
deleting, and altering the configuration of topics; writing events to or
reading events from a topic; creating and deleting ACLs. Administrators can
also define custom policies to put in place additional restrictions, such as a
<code>CreateTopicPolicy</code> and <code>AlterConfigPolicy</code> (see <a
href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-108%3A+Create+Topic+Policy";>KIP-108</a>
and the sett [...]
+ </ol>
+
+ <p>
+ When securing a multi-tenant Kafka environment, the most common
administrative task is the third category (authorization), i.e., managing the
user/client permissions that grant or deny access to certain topics and thus to
the data stored by users within a cluster. This task is performed predominantly
through the <a href="#security_authz">setting of access control lists
(ACLs)</a>. Here, administrators of multi-tenant environments in particular
benefit from putting a hierarchical topic [...]
+ </p>
+
+ <p>
+ In the following example, user Alice—a new member of ACME corporation's
InfoSec team—is granted write permissions to all topics whose names start with
"acme.infosec.", such as "acme.infosec.telemetry.logins" and
"acme.infosec.syslogs.events".
+ </p>
+
+<pre class="line-numbers"><code class="language-text"># Grant permissions to
user Alice
+$ bin/kafka-acls.sh \
+ --bootstrap-server broker1:9092 \
+ --add --allow-principal User:Alice \
+ --producer \
+ --resource-pattern-type prefixed --topic acme.infosec.
+</code></pre>
+
+ <p>
+ You can similarly use this approach to isolate different customers on the
same shared cluster.
+ </p>
+
+ <h4 class="anchor-heading"><a id="multitenancy-isolation"
class="anchor-link"></a><a href="#multitenancy-isolation">Isolating Tenants:
Quotas, Rate Limiting, Throttling</a></h4>
+
+ <p>
+ Multi-tenant clusters should generally be configured with <a
href="#design_quotas">quotas</a>, which protect against users (tenants) eating
up too many cluster resources, such as when they attempt to write or read very
high volumes of data, or create requests to brokers at an excessively high
rate. This may cause network saturation, monopolize broker resources, and
impact other clients—all of which you want to avoid in a shared environment.
+ </p>
+
+ <p>
+ <strong>Client quotas:</strong> Kafka supports different types of (per-user
principal) client quotas. Because a client's quotas apply irrespective of which
topics the client is writing to or reading from, they are a convenient and
effective tool to allocate resources in a multi-tenant cluster. <a
href="#design_quotascpu">Request rate quotas</a>, for example, help to limit a
user's impact on broker CPU usage by limiting the time a broker spends on the
<a href="/protocol.html">request ha [...]
+ </p>
+
+ <p>
+ <strong>Server quotas:</strong> Kafka also supports different types of
broker-side quotas. For example, administrators can set a limit on the rate
with which the <a href="#brokerconfigs_max.connection.creation.rate">broker
accepts new connections</a>, set the <a
href="#brokerconfigs_max.connections">maximum number of connections per
broker</a>, or set the maximum number of connections allowed <a
href="#brokerconfigs_max.connections.per.ip">from a specific IP address</a>.
+ </p>
+
+ <p>
+ For more information, please refer to the <a href="#design_quotas">quota
overview</a> and <a href="#quotas">how to set quotas</a>.
+ </p>
+
+ <h4 class="anchor-heading"><a id="multitenancy-monitoring"
class="anchor-link"></a><a href="#multitenancy-monitoring">Monitoring and
Metering</a></h4>
+
+ <p>
+ <a href="#monitoring">Monitoring</a> is a broader subject that is covered <a
href="#monitoring">elsewhere</a> in the documentation. Administrators of any
Kafka environment, but especially multi-tenant ones, should set up monitoring
according to these instructions. Kafka supports a wide range of metrics, such
as the rate of failed authentication attempts, request latency, consumer lag,
total number of consumer groups, metrics on the quotas described in the
previous section, and many more.
+ </p>
+
+ <p>
+ For example, monitoring can be configured to track the size of
topic-partitions (with the JMX metric
<code>kafka.log.Log.Size.<TOPIC-NAME></code>), and thus the total size of
data stored in a topic. You can then define alerts when tenants on shared
clusters are getting close to using too much storage space.
+ </p>
+
+ <h4 class="anchor-heading"><a id="multitenancy-georeplication"
class="anchor-link"></a><a href="#multitenancy-georeplication">Multi-Tenancy
and Geo-Replication</a></h4>
+
+ <p>
+ Kafka lets you share data across different clusters, which may be located
in different geographical regions, data centers, and so on. Apart from use
cases such as disaster recovery, this functionality is useful when a
multi-tenant setup requires inter-cluster data sharing. See the section <a
href="#georeplication">Geo-Replication (Cross-Cluster Data Mirroring)</a> for
more information.
+ </p>
+
+ <h4 class="anchor-heading"><a id="multitenancy-more"
class="anchor-link"></a><a href="#multitenancy-more">Further
considerations</a></h4>
+
+ <p>
+ <strong>Data contracts:</strong> You may need to define data contracts
between the producers and the consumers of data in a cluster, using event
schemas. This ensures that events written to Kafka can always be read properly
again, and prevents malformed or corrupt events being written. The best way to
achieve this is to deploy a so-called schema registry alongside the cluster.
(Kafka does not include a schema registry, but there are third-party
implementations available.) A schema re [...]
+ </p>
+
+
+ <h3 class="anchor-heading"><a id="config" class="anchor-link"></a><a
href="#config">6.5 Kafka Configuration</a></h3>
<h4 class="anchor-heading"><a id="clientconfig" class="anchor-link"></a><a
href="#clientconfig">Important Client Configurations</a></h4>
@@ -1122,7 +1280,7 @@ checkpoint-latency-ms-avg
Our client configuration varies a fair amount between different use cases.
- <h3 class="anchor-heading"><a id="java" class="anchor-link"></a><a
href="#java">6.5 Java Version</a></h3>
+ <h3 class="anchor-heading"><a id="java" class="anchor-link"></a><a
href="#java">6.6 Java Version</a></h3>
Java 8 and Java 11 are supported. Java 11 performs significantly better if
TLS is enabled, so it is highly recommended (it also includes a number of other
performance improvements: G1GC, CRC32C, Compact Strings, Thread-Local
Handshakes and more).
@@ -1145,7 +1303,7 @@ checkpoint-latency-ms-avg
All of the brokers in that cluster have a 90% GC pause time of about 21ms
with less than 1 young GC per second.
- <h3 class="anchor-heading"><a id="hwandos" class="anchor-link"></a><a
href="#hwandos">6.6 Hardware and OS</a></h3>
+ <h3 class="anchor-heading"><a id="hwandos" class="anchor-link"></a><a
href="#hwandos">6.7 Hardware and OS</a></h3>
We are using dual quad-core Intel Xeon machines with 24GB of memory.
<p>
You need sufficient memory to buffer active readers and writers. You can do
a back-of-the-envelope estimate of memory needs by assuming you want to be able
to buffer for 30 seconds and compute your memory need as write_throughput*30.
@@ -1230,7 +1388,7 @@ checkpoint-latency-ms-avg
<li>delalloc: Delayed allocation means that the filesystem avoid
allocating any blocks until the physical write occurs. This allows ext4 to
allocate a large extent instead of smaller pages and helps ensure the data is
written sequentially. This feature is great for throughput. It does seem to
involve some locking in the filesystem which adds a bit of latency variance.
</ul>
- <h3 class="anchor-heading"><a id="monitoring" class="anchor-link"></a><a
href="#monitoring">6.7 Monitoring</a></h3>
+ <h3 class="anchor-heading"><a id="monitoring" class="anchor-link"></a><a
href="#monitoring">6.8 Monitoring</a></h3>
Kafka uses Yammer Metrics for metrics reporting in the server. The Java
clients use Kafka Metrics, a built-in metrics registry that minimizes
transitive dependencies pulled into client applications. Both expose metrics
via JMX and can be configured to report stats using pluggable stats reporters
to hook up to your monitoring system.
<p>
@@ -2848,7 +3006,7 @@ dropped-records-rate and dropped-records-total which have
a recording level of <
On the client side, we recommend monitoring the message/byte rate (global
and per topic), request rate/size/time, and on the consumer side, max lag in
messages among all partitions and min fetch request rate. For a consumer to
keep up, max lag needs to be less than a threshold and min fetch rate needs to
be larger than 0.
- <h3 class="anchor-heading"><a id="zk" class="anchor-link"></a><a
href="#zk">6.8 ZooKeeper</a></h3>
+ <h3 class="anchor-heading"><a id="zk" class="anchor-link"></a><a
href="#zk">6.9 ZooKeeper</a></h3>
<h4 class="anchor-heading"><a id="zkversion" class="anchor-link"></a><a
href="#zkversion">Stable version</a></h4>
The current stable branch is 3.5. Kafka is regularly updated to include the
latest release in the 3.5 series.
diff --git a/docs/toc.html b/docs/toc.html
index 8d15b2b..fa11c81 100644
--- a/docs/toc.html
+++ b/docs/toc.html
@@ -96,13 +96,24 @@
<li><a
href="#georeplication-apply-config-changes">Applying Configuration
Changes</a></li>
<li><a href="#georeplication-monitoring">Monitoring
Geo-Replication</a></li>
</ul>
- <li><a href="#config">6.4 Important Configs</a>
+ <li><a href="#multitenancy">6.4 Multi-Tenancy</a></li>
+ <ul>
+ <li><a href="#multitenancy-overview">Multi-Tenancy
Overview</a></li>
+ <li><a href="#multitenancy-topic-naming">Creating User
Spaces (Namespaces)</a></li>
+ <li><a href="#multitenancy-topic-configs">Configuring
Topics</a></li>
+ <li><a href="#multitenancy-security">Securing Clusters
and Topics</a></li>
+ <li><a href="#multitenancy-isolation">Isolating
Tenants</a></li>
+ <li><a href="#multitenancy-monitoring">Monitoring and
Metering</a></li>
+ <li><a
href="#multitenancy-georeplication">Multi-Tenancy and Geo-Replication</a></li>
+ <li><a href="#multitenancy-more">Further
considerations</a></li>
+ </ul>
+ <li><a href="#config">6.5 Important Configs</a>
<ul>
<li><a href="#clientconfig">Important Client
Configs</a>
<li><a href="#prodconfig">A Production Server
Configs</a>
</ul>
- <li><a href="#java">6.5 Java Version</a>
- <li><a href="#hwandos">6.6 Hardware and OS</a>
+ <li><a href="#java">6.6 Java Version</a>
+ <li><a href="#hwandos">6.7 Hardware and OS</a>
<ul>
<li><a href="#os">OS</a>
<li><a href="#diskandfs">Disks and Filesystems</a>
@@ -110,7 +121,7 @@
<li><a href="#linuxflush">Linux Flush Behavior</a>
<li><a href="#ext4">Ext4 Notes</a>
</ul>
- <li><a href="#monitoring">6.7 Monitoring</a>
+ <li><a href="#monitoring">6.8 Monitoring</a>
<ul>
<li><a href="#selector_monitoring">Selector
Monitoring</a></li>
<li><a href="#common_node_monitoring">Common Node
Monitoring</a></li>
@@ -120,7 +131,7 @@
<li><a href="#kafka_streams_monitoring">Streams
Monitoring</a></li>
<li><a href="#others_monitoring">Others</a></li>
</ul>
- <li><a href="#zk">6.8 ZooKeeper</a>
+ <li><a href="#zk">6.9 ZooKeeper</a>
<ul>
<li><a href="#zkversion">Stable Version</a>
<li><a href="#zkops">Operationalization</a>
