[
https://issues.apache.org/jira/browse/KAFKA-7710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mr Kafka updated KAFKA-7710:
----------------------------
Description:
I have seen many organizations run many Kafka clusters. The simplest scenario
is you may have a *kafka.dev.example.com* cluster and a
*kafka.prod.example.com* cluster. The more extreme examples is teams within an
organization may run their own individual clusters and want isolation.
When you enable Zookeeper ACLs in Kafka the ACL looks to be set to the
principal (SPN) that is used to authenticate against Zookeeper.
For example I have brokers:
* *01.kafka.dev.example.com*
* *02.kafka.dev.example.com***
* *03.kafka.dev.example.com***
On *01.kafka.dev.example.com* **I run the below the security-migration tool:
{code:java}
KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf
-Dzookeeper.sasl.clientconfig=ZkClient" zookeeper-security-migration
--zookeeper.acl=secure --zookeeper.connect=a01.zookeeper.dev.example.com:2181
{code}
I end up with ACL's in Zookeeper as below:
{code:java}
# [zk: localhost:2181(CONNECTED) 2] getAcl /cluster
# 'sasl,'kafka/01.kafka.dev.example.com@EXAMPLE
# : cdrwa
{code}
This ACL means no other broker in the cluster can access the znode in Zookeeper
except broker 01.
To resolve the issue you need to set the below properties in Zookeeper's config:
{code:java}
kerberos.removeHostFromPrincipal = true
kerberos.removeRealmFromPrincipal = true
{code}
Now when Kafka set ACL's they are stored as:
{code:java}
# [zk: localhost:2181(CONNECTED) 2] getAcl /cluster
# 'sasl,'kafka
#: cdrwa
{code}
This now means any broker in the cluster can access the ZK node.This means if I
have a dev Kafka broker it can right to a "prod.zookeeper.example.com"
zookeeper host as when it auth's based on a SPN
"kafka/01.kafka.dev.example.com" the host is dropped and we auth against the
service principal kafka.
If your organization is flexible you may be able to create different Kerberos
Realms per cluster and use:
{code:java}
kerberos.removeHostFromPrincipal = true
kerberos.removeRealmFromPrincipal = false{code}
That means acl's will be in the format "kafka/REALM" which means only brokers
in the same realm can connect. The difficulty here is your average large
organization security team willing to create adhoc realms.
*Proposal*
Kafka support setting ACLs for all known brokers in the cluster i.e ACLs on a
Znode have
{code:java}
kafka/01.kafka.dev.example.com@EXAMPLE
kafka/02.kafka.dev.example.com@EXAMPLE
kafka/03.kafka.dev.example.com@EXAMPLE{code}
With this though some kind of support will need to be added so if a new broker
joins the cluster the host ACL gets added to existing ZNodes.
was:
I have seen many organizations run many Kafka clusters. The simplest scenario
is you may have a *kafka.dev.example.com* cluster and a
*kafka.prod.example.com* cluster. The more extreme examples is teams with in an
organization may run their own individual clusters.
When you enable Zookeeper ACLs in Kafka the ACL looks to be set to the
principal (SPN) that is used to authenticate against Zookeeper.
For example I have brokers:
* *01.kafka.dev.example.com*
* *02.kafka.dev.example.com***
* *03.kafka.dev.example.com***
On *01.kafka.dev.example.com* **I run the below the security-migration tool:
{code:java}
KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf
-Dzookeeper.sasl.clientconfig=ZkClient" zookeeper-security-migration
--zookeeper.acl=secure --zookeeper.connect=a01.zookeeper.dev.example.com:2181
{code}
I end up with ACL's in Zookeeper as below:
{code:java}
# [zk: localhost:2181(CONNECTED) 2] getAcl /cluster
# 'sasl,'kafka/01.kafka.dev.example.com@EXAMPLE
# : cdrwa
{code}
This ACL means no other broker in the cluster can access the znode in Zookeeper
except broker 01.
To resolve the issue you need to set the below properties in Zookeeper's config:
{code:java}
kerberos.removeHostFromPrincipal = true
kerberos.removeRealmFromPrincipal = true
{code}
Now when Kafka set ACL's they are stored as:
{code:java}
# [zk: localhost:2181(CONNECTED) 2] getAcl /cluster
# 'sasl,'kafka
#: cdrwa
{code}
This now means any broker in the cluster can access the ZK node.This means if I
have a dev Kafka broker it can right to a "prod.zookeeper.example.com"
zookeeper host as when it auth's based on a SPN
"kafka/01.kafka.dev.example.com" the host is dropped and we auth against the
service principal kafka.
If your organization is flexible you may be able to create different Kerberos
Realms per cluster and use:
{code:java}
kerberos.removeHostFromPrincipal = true
kerberos.removeRealmFromPrincipal = false{code}
That means acl's will be in the format "kafka/REALM" which means only brokers
in the same realm can connect. The difficulty here is your average large
organization security team willing to create adhoc realms.
*Proposal*
Kafka support setting ACLs for all known brokers in the cluster i.e ACLs on a
Znode have
{code:java}
kafka/01.kafka.dev.example.com@EXAMPLE
kafka/02.kafka.dev.example.com@EXAMPLE
kafka/03.kafka.dev.example.com@EXAMPLE{code}
With this though some kind of support will need to be added so if a new broker
joins the cluster the host ACL gets added to existing ZNodes.
> Poor Zookeeper ACL management with Kerberos
> -------------------------------------------
>
> Key: KAFKA-7710
> URL: https://issues.apache.org/jira/browse/KAFKA-7710
> Project: Kafka
> Issue Type: Bug
> Reporter: Mr Kafka
> Priority: Major
>
> I have seen many organizations run many Kafka clusters. The simplest scenario
> is you may have a *kafka.dev.example.com* cluster and a
> *kafka.prod.example.com* cluster. The more extreme examples is teams within
> an organization may run their own individual clusters and want isolation.
> When you enable Zookeeper ACLs in Kafka the ACL looks to be set to the
> principal (SPN) that is used to authenticate against Zookeeper.
> For example I have brokers:
> * *01.kafka.dev.example.com*
> * *02.kafka.dev.example.com***
> * *03.kafka.dev.example.com***
> On *01.kafka.dev.example.com* **I run the below the security-migration tool:
> {code:java}
> KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf
> -Dzookeeper.sasl.clientconfig=ZkClient" zookeeper-security-migration
> --zookeeper.acl=secure --zookeeper.connect=a01.zookeeper.dev.example.com:2181
> {code}
> I end up with ACL's in Zookeeper as below:
> {code:java}
> # [zk: localhost:2181(CONNECTED) 2] getAcl /cluster
> # 'sasl,'kafka/01.kafka.dev.example.com@EXAMPLE
> # : cdrwa
> {code}
> This ACL means no other broker in the cluster can access the znode in
> Zookeeper except broker 01.
> To resolve the issue you need to set the below properties in Zookeeper's
> config:
> {code:java}
> kerberos.removeHostFromPrincipal = true
> kerberos.removeRealmFromPrincipal = true
> {code}
> Now when Kafka set ACL's they are stored as:
> {code:java}
> # [zk: localhost:2181(CONNECTED) 2] getAcl /cluster
> # 'sasl,'kafka
> #: cdrwa
> {code}
> This now means any broker in the cluster can access the ZK node.This means if
> I have a dev Kafka broker it can right to a "prod.zookeeper.example.com"
> zookeeper host as when it auth's based on a SPN
> "kafka/01.kafka.dev.example.com" the host is dropped and we auth against the
> service principal kafka.
> If your organization is flexible you may be able to create different Kerberos
> Realms per cluster and use:
> {code:java}
> kerberos.removeHostFromPrincipal = true
> kerberos.removeRealmFromPrincipal = false{code}
> That means acl's will be in the format "kafka/REALM" which means only brokers
> in the same realm can connect. The difficulty here is your average large
> organization security team willing to create adhoc realms.
> *Proposal*
> Kafka support setting ACLs for all known brokers in the cluster i.e ACLs on a
> Znode have
> {code:java}
> kafka/01.kafka.dev.example.com@EXAMPLE
> kafka/02.kafka.dev.example.com@EXAMPLE
> kafka/03.kafka.dev.example.com@EXAMPLE{code}
> With this though some kind of support will need to be added so if a new
> broker joins the cluster the host ACL gets added to existing ZNodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
