Sandor Molnar created KNOX-3531:
-----------------------------------
Summary: Add Admin API endpoints to flush the HadoopGroupProvider
group cache
Key: KNOX-3531
URL: https://issues.apache.org/jira/browse/KNOX-3531
Project: Apache Knox
Issue Type: Improvement
Components: Admin API
Reporter: Sandor Molnar
Assignee: David Han
h3. Problem
When the HadoopGroupProvider is used, Knox resolves a user's groups through
Hadoop's {{Groups}} / {{GroupMappingServiceProvider}} (e.g.
{{{}LdapGroupsMapping{}}}), configured with the {{gateway.group.config.*}}
properties in {{gateway-site.xml}} or with provider parameters. Hadoop keeps
the result in an in-memory cache controlled by:
{code:java}
hadoop.security.groups.cache.secs (default 300)
hadoop.security.groups.negative-cache.secs (default 30)
{code}
When group membership changes in the backend (LDAP, etc.), Knox keeps using the
old groups until the cache entry is refreshed, which can take up to
{{{}hadoop.security.groups.cache.secs{}}}. If backend lookups fail while the
cache is refreshing, the old groups can stay for up to 10x that value.
Authorization decisions based on groups (ACLs, virtual groups, dispatch of
group information to backends) are wrong during that window.
Today the only ways to make Knox pick up the change are:
* lowering the TTL, which only narrows the window and increases the load on
the backend;
* redeploying the topology, which is heavy and rebuilds everything just to
clear a cache.
There is no way to flush the cache on demand.
h3. Constraints
* Each HadoopGroupProvider filter instance creates its own Hadoop {{Groups}}
object, so each topology has its own private cache, separate from the Admin API.
* Hadoop's {{Groups}} cache cannot invalidate a single user. Its only public
flush operation, {{{}refresh(){}}}, clears the whole cache (including the
negative cache).
* {{hadoop.security.groups.cache.secs}} cannot be set to {{{}0{}}}; it has to
be a positive value.
h3. Proposal
Let administrators (or external systems that manage group membership) flush the
group cache on demand through the Admin API.
*1. Group cache registry (new gateway service)*
* Add a gateway-level service where group caches can be registered by topology
name.
* When it is initialized, the HadoopGroupProvider registers its cache (the
flush action of its {{Groups}} instance) under the topology it belongs to. It
unregisters when the filter is destroyed, so caches from undeployed or
redeployed topologies are not kept.
* This only applies when groups are resolved through Hadoop {{{}Groups{}}}. It
doesn't apply when the provider is set up to use the Knox LDAP service
({{{}use.ldap.service=true{}}}), because that path has no cache that lasts
across requests.
*2. New Admin API endpoints*
{code:java}
DELETE /gateway/admin/api/v1/topologies/{topology}/groupcache
DELETE /gateway/admin/api/v1/groupcache
{code}
* The first endpoint clears every group cache registered for one topology. It
returns {{404}} if the topology doesn't exist, otherwise {{{}200{}}}.
* The second endpoint clears every registered group cache in the gateway. This
is for callers that don't know which topologies use group lookup.
* Both endpoints are protected the same way as the rest of the Admin API (the
{{admin}} topology's authentication and authorization).
* Example response:
{code:java}
{
"topology": "sandbox",
"flushedCaches": 2
}
{code}
For the global endpoint, the {{topology}} field is left out.
*3. Behavior after a flush*
* The next request for any user looks up the groups in the backend again.
* Requests already in flight finish with the groups they started with.
* {{hadoop.security.groups.cache.secs}} still applies, as a safety net for
changes that aren't followed by a flush.
h3. Example
{code:java}
curl -iku admin:admin-password -X DELETE \
https://localhost:8443/gateway/admin/api/v1/topologies/sandbox/groupcache
curl -iku admin:admin-password -X DELETE \
https://localhost:8443/gateway/admin/api/v1/groupcache
{code}
h3. Notes
* The cache lives in memory in each Knox instance. In an HA / multi-instance
deployment, the flush has to be called on every instance. Flushing across the
cluster is out of scope.
* Flushing one user's entry is out of scope, because Hadoop's {{Groups}}
doesn't support it. It could be added later if Knox kept its own group cache.
* Flushing automatically when the backend changes (e.g. from LDAP change
notifications) is out of scope.
h3. Testing
* Unit tests for the registry: register, unregister, flushing one topology,
flushing everything, and checking that one failing flush doesn't stop the
others.
* Unit tests for the HadoopGroupProvider: the cache is registered on init and
unregistered on destroy, and nothing is registered when the Knox LDAP service
is used.
* Unit tests for the Admin API: both endpoints, including {{404}} for an
unknown topology.
* Manual test:
## Send a request through a topology that uses the HadoopGroupProvider, so the
user's groups get cached.
## Change the user's group membership in the backend.
## Call the flush endpoint.
## Send another request: the new groups are used immediately, without waiting
for {{{}hadoop.security.groups.cache.secs{}}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
