[
https://issues.apache.org/jira/browse/RANGER-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramachandran Krishnan updated RANGER-5655:
------------------------------------------
Description:
Ranger plugins (HDFS, Hive, Trino, etc.) send audit events to
{*}audit-ingestor{*}, which writes them to Kafka topic {{{}ranger_audits{}}}.
To isolate hot plugins, ingestor can route each plugin to dedicated Kafka
partitions; unknown plugins share a *buffer* pool.
*Today (static mode):* Plugin → partition mapping is computed once at startup
from XML. Changing assignments requires config edits, topic partition growth,
and {*}ingestor restart{*}. Early-plugin changes can *reshuffle* later plugins
because allocation uses contiguous ranges.
Plugins POST audits to {{{}POST
/api/audit/access?serviceName=<repo>&appId=<agent>{}}}. After
Kerberos/JWT/basic *authentication* (401 on failure), ingestor performs
{*}authorization{*}: the authenticated short username must appear in
{{{}ranger.audit.ingestor.service.<repo>.allowed.users{}}}.
Today that map is loaded *once at JVM startup* from
{{{}ranger-audit-ingestor-site.xml{}}}. Adding a new Ranger service repo (e.g.
{{{}dev_trino{}}}) requires XML edits and *ingestor restart* — even when
partition routing is already dynamic via the partition-plan topic.
*Proposed (dynamic mode):*
Ranger plugins POST audits to the *Audit Ingestor* ({{{}POST
/api/audit/access{}}}). This feature adds:
# *Dynamic Kafka partition routing* — layout stored in Admin
({{{}x_ranger_global_state{}}}), synced by a background poller.
# *Dynamic allow-list* — who may POST audits per service comes from
{{policy.download.auth.users}} in the partition plan.
was:
Ranger plugins (HDFS, Hive, Trino, etc.) send audit events to
{*}audit-ingestor{*}, which writes them to Kafka topic {{{}ranger_audits{}}}.
To isolate hot plugins, ingestor can route each plugin to dedicated Kafka
partitions; unknown plugins share a *buffer* pool.
*Today (static mode):* Plugin → partition mapping is computed once at startup
from XML. Changing assignments requires config edits, topic partition growth,
and {*}ingestor restart{*}. Early-plugin changes can *reshuffle* later plugins
because allocation uses contiguous ranges.
Plugins POST audits to {{{}POST
/api/audit/access?serviceName=<repo>&appId=<agent>{}}}. After
Kerberos/JWT/basic *authentication* (401 on failure), ingestor performs
{*}authorization{*}: the authenticated short username must appear in
{{{}ranger.audit.ingestor.service.<repo>.allowed.users{}}}.
Today that map is loaded *once at JVM startup* from
{{{}ranger-audit-ingestor-site.xml{}}}. Adding a new Ranger service repo (e.g.
{{{}dev_trino{}}}) requires XML edits and *ingestor restart* — even when
partition routing is already dynamic via the partition-plan topic.
*Proposed (dynamic mode):*
Ranger plugins POST audits to the *Audit Ingestor* ({{{}POST
/api/audit/access{}}}). This feature adds:
# *Dynamic Kafka partition routing* — layout stored in Admin
({{{}x_ranger_global_state{}}}), synced by a background poller.
# *Dynamic allow-list* — who may POST audits per service comes from
{{policy.download.auth.users}} in the partition plan.
# *SPIFFE workload identity* at two boundaries:
** *Plugin → Ingestor* — {{X-Spiffe-Id}} on audit POSTs
({{{}AuditHeaderAuthFilter{}}}).
** *Ingestor → Admin* — {{X-Spiffe-Id}} as {{rangerauditserver}} for plan
download (no password).
Same pattern for Hive, HDFS, Ozone, Knox, Kafka, HBase, KMS, Trino, Atlas, etc.
> Ranger Admin-Managed Kafka Audit Partition Plan & SPIFFE Auth
> -------------------------------------------------------------
>
> Key: RANGER-5655
> URL: https://issues.apache.org/jira/browse/RANGER-5655
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
> Reporter: Ramachandran Krishnan
> Assignee: Ramachandran Krishnan
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Ranger plugins (HDFS, Hive, Trino, etc.) send audit events to
> {*}audit-ingestor{*}, which writes them to Kafka topic {{{}ranger_audits{}}}.
> To isolate hot plugins, ingestor can route each plugin to dedicated Kafka
> partitions; unknown plugins share a *buffer* pool.
> *Today (static mode):* Plugin → partition mapping is computed once at startup
> from XML. Changing assignments requires config edits, topic partition growth,
> and {*}ingestor restart{*}. Early-plugin changes can *reshuffle* later
> plugins because allocation uses contiguous ranges.
> Plugins POST audits to {{{}POST
> /api/audit/access?serviceName=<repo>&appId=<agent>{}}}. After
> Kerberos/JWT/basic *authentication* (401 on failure), ingestor performs
> {*}authorization{*}: the authenticated short username must appear in
> {{{}ranger.audit.ingestor.service.<repo>.allowed.users{}}}.
> Today that map is loaded *once at JVM startup* from
> {{{}ranger-audit-ingestor-site.xml{}}}. Adding a new Ranger service repo
> (e.g. {{{}dev_trino{}}}) requires XML edits and *ingestor restart* — even
> when partition routing is already dynamic via the partition-plan topic.
> *Proposed (dynamic mode):*
> Ranger plugins POST audits to the *Audit Ingestor* ({{{}POST
> /api/audit/access{}}}). This feature adds:
> # *Dynamic Kafka partition routing* — layout stored in Admin
> ({{{}x_ranger_global_state{}}}), synced by a background poller.
> # *Dynamic allow-list* — who may POST audits per service comes from
> {{policy.download.auth.users}} in the partition plan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
