[
https://issues.apache.org/jira/browse/KAFKA-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KAFKA-3328:
-------------------------------
Description:
Currently when adding or removing an ACL with the SimpleAclAuthorizer the
following high level steps happen:
# read acls from cache
# merge with the changes acls
# update zookeeper
# add a change notification
Then the Authorizers listening for the change notification know to invalidate
their cache and get the latest value. However that takes some time. In the time
between the ACL change and the cache update, a new add or remove request could
be made. This will follow the steps listed above, and if the cache is not
correct all changes from the previous request are lost.
This can be solved on a single node, by updating the cache at the same time you
update zookeeper any time a change is made. However, because there can be
multiple instances of the Authorizer, a request could come to a separate
authorizer and overwrite the Zookeeper state again loosing changes from earlier
requests.
To solve this on multiple instances. The authorizer could always read/write
state from zookeeper (instead of the cache) for add/remove requests and only
leverage the cache for get/authorize requests. Or it could block until all the
live instances have updated their cache.
Below is a log from a failed test in the WIP [pull
request|https://github.com/apache/kafka/pull/1005] for KAFKA-3266 that shows
this behavior:
{noformat}
[2016-03-03 11:09:20,714] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Allow
permission for operations: Describe from hosts: * for Cluster:kafka-cluster
(kafka.server.KafkaApis:52)
[2016-03-03 11:09:20,726] DEBUG updatedAcls: Set(User:ANONYMOUS has Allow
permission for operations: Describe from hosts: *)
(kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,738] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Deny
permission for operations: Describe from hosts: * for Cluster:kafka-cluster
(kafka.server.KafkaApis:52)
[2016-03-03 11:09:20,739] DEBUG updatedAcls: Set(User:ANONYMOUS has Deny
permission for operations: Describe from hosts: *)
(kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,752] DEBUG Processing ACL change notification for
Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,755] DEBUG Processing ACL change notification for
Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,762] DEBUG Processing ACL change notification for
Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,768] DEBUG Processing ACL change notification for
Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,773] DEBUG Processing ACL change notification for
Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,777] DEBUG Processing ACL change notification for
Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
{noformat}
was:
Currently when adding or removing an ACL with the SimpleAclAuthorizer the
following high level steps happen:
# read acls from cache
# merge with the changes acls
# update zookeeper
# add a change notification
Then the Authorizers listening for the change notification know to invalidate
their cache and get the latest value. However that takes some time. In the time
between the ACL change and the cache update, a new add or remove request could
be made. This will follow the steps listed above, and if the cache is not
correct all changes from the previous request are lost.
This can be solved on a single node, by updating the cache at the same time you
update zookeeper any time a change is made. However, because there can be
multiple instances of the Authorizer, a request could come to a separate
authorizer and overwrite the Zookeeper state again loosing changes from earlier
requests.
Below is a log from a failed test in the WIP [pull
request|https://github.com/apache/kafka/pull/1005] for KAFKA-3266 that shows
this behavior:
{noformat}
[2016-03-03 11:09:20,714] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Allow
permission for operations: Describe from hosts: * for Cluster:kafka-cluster
(kafka.server.KafkaApis:52)
[2016-03-03 11:09:20,726] DEBUG updatedAcls: Set(User:ANONYMOUS has Allow
permission for operations: Describe from hosts: *)
(kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,738] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Deny
permission for operations: Describe from hosts: * for Cluster:kafka-cluster
(kafka.server.KafkaApis:52)
[2016-03-03 11:09:20,739] DEBUG updatedAcls: Set(User:ANONYMOUS has Deny
permission for operations: Describe from hosts: *)
(kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,752] DEBUG Processing ACL change notification for
Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,755] DEBUG Processing ACL change notification for
Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,762] DEBUG Processing ACL change notification for
Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,768] DEBUG Processing ACL change notification for
Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,773] DEBUG Processing ACL change notification for
Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
[2016-03-03 11:09:20,777] DEBUG Processing ACL change notification for
Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
operations: Describe from hosts: *) (kafka.security.auth.SimpleAclAuthorizer:52)
{noformat}
> SimpleAclAuthorizer can lose ACLs with frequent add/remove calls
> ----------------------------------------------------------------
>
> Key: KAFKA-3328
> URL: https://issues.apache.org/jira/browse/KAFKA-3328
> Project: Kafka
> Issue Type: Bug
> Reporter: Grant Henke
>
> Currently when adding or removing an ACL with the SimpleAclAuthorizer the
> following high level steps happen:
> # read acls from cache
> # merge with the changes acls
> # update zookeeper
> # add a change notification
> Then the Authorizers listening for the change notification know to invalidate
> their cache and get the latest value. However that takes some time. In the
> time between the ACL change and the cache update, a new add or remove request
> could be made. This will follow the steps listed above, and if the cache is
> not correct all changes from the previous request are lost.
> This can be solved on a single node, by updating the cache at the same time
> you update zookeeper any time a change is made. However, because there can be
> multiple instances of the Authorizer, a request could come to a separate
> authorizer and overwrite the Zookeeper state again loosing changes from
> earlier requests.
> To solve this on multiple instances. The authorizer could always read/write
> state from zookeeper (instead of the cache) for add/remove requests and only
> leverage the cache for get/authorize requests. Or it could block until all
> the live instances have updated their cache.
> Below is a log from a failed test in the WIP [pull
> request|https://github.com/apache/kafka/pull/1005] for KAFKA-3266 that shows
> this behavior:
> {noformat}
> [2016-03-03 11:09:20,714] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Allow
> permission for operations: Describe from hosts: * for Cluster:kafka-cluster
> (kafka.server.KafkaApis:52)
> [2016-03-03 11:09:20,726] DEBUG updatedAcls: Set(User:ANONYMOUS has Allow
> permission for operations: Describe from hosts: *)
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,738] DEBUG [KafkaApi-0] adding User:ANONYMOUS has Deny
> permission for operations: Describe from hosts: * for Cluster:kafka-cluster
> (kafka.server.KafkaApis:52)
> [2016-03-03 11:09:20,739] DEBUG updatedAcls: Set(User:ANONYMOUS has Deny
> permission for operations: Describe from hosts: *)
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,752] DEBUG Processing ACL change notification for
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
> operations: Describe from hosts: *)
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,755] DEBUG Processing ACL change notification for
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
> operations: Describe from hosts: *)
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,762] DEBUG Processing ACL change notification for
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
> operations: Describe from hosts: *)
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,768] DEBUG Processing ACL change notification for
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
> operations: Describe from hosts: *)
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,773] DEBUG Processing ACL change notification for
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
> operations: Describe from hosts: *)
> (kafka.security.auth.SimpleAclAuthorizer:52)
> [2016-03-03 11:09:20,777] DEBUG Processing ACL change notification for
> Cluster:kafka-cluster and Set(User:ANONYMOUS has Deny permission for
> operations: Describe from hosts: *)
> (kafka.security.auth.SimpleAclAuthorizer:52)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
