[
https://issues.apache.org/jira/browse/RANGER-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramachandran Krishnan updated RANGER-5655:
------------------------------------------
Description:
Ranger plugins (HDFS, Hive, Trino, etc.) send audit events to
{*}audit-ingestor{*}, which writes them to Kafka topic {{{}ranger_audits{}}}.
To isolate hot plugins, ingestor can route each plugin to dedicated Kafka
partitions; unknown plugins share a *buffer* pool.
*Today (static mode):* Plugin → partition mapping is computed once at startup
from XML. Changing assignments requires config edits, topic partition growth,
and {*}ingestor restart{*}. Early-plugin changes can *reshuffle* later plugins
because allocation uses contiguous ranges.
Plugins POST audits to {{{}POST
/api/audit/access?serviceName=<repo>&appId=<agent>{}}}. After
Kerberos/JWT/basic *authentication* (401 on failure), ingestor performs
{*}authorization{*}: the authenticated short username must appear in
{{{}ranger.audit.ingestor.service.<repo>.allowed.users{}}}.
Today that map is loaded *once at JVM startup* from
{{{}ranger-audit-ingestor-site.xml{}}}. Adding a new Ranger service repo (e.g.
{{{}dev_trino{}}}) requires XML edits and *ingestor restart* — even when
partition routing is already dynamic via the partition-plan topic.
*Proposed (dynamic mode):*
Ranger plugins POST audits to the *Audit Ingestor* ({{{}POST
/api/audit/access{}}}). This feature adds:
# *Dynamic Kafka partition routing* — layout stored in Admin
({{{}x_ranger_global_state{}}}), synced by a background poller.
# *Dynamic allow-list* — who may POST audits per service comes from
{{policy.download.auth.users}} in the partition plan.
# *SPIFFE workload identity* at two boundaries:
** *Plugin → Ingestor* — {{X-Spiffe-Id}} on audit POSTs
({{{}AuditHeaderAuthFilter{}}}).
** *Ingestor → Admin* — {{X-Spiffe-Id}} as {{rangerauditserver}} for plan
download (no password).
Same pattern for Hive, HDFS, Ozone, Knox, Kafka, HBase, KMS, Trino, Atlas, etc.
was:
Implement a dynamic unified ingestor registry for Ranger audit-ingestor so
operators can change Kafka partition routing and per-repo service allowlists at
runtime — without restarting ingestor pods.
The registry is stored in a Kafka compacted topic
({{ranger_audit_partition_plan}}) and managed via REST. All ingestor replicas
converge on the same versioned plan through {{PartitionPlanWatcher}};
{{AuditPartitioner}} routes audit records on the hot path from in-memory state
only.
*Feature flag (default off):*
{{ranger.audit.ingestor.kafka.partition.plan.dynamic.enabled=false}}
*Pull request:* [GitHub PR #1032|https://github.com/apache/ranger/pull/1032]
----
h2. Problem
Today (static mode), audit-ingestor loads two kinds of configuration from XML
at startup only:
|| Job || Question || Static behavior ||
| Service allowlist | May this Kerberos principal POST audits for repo R? |
{{ranger.audit.ingestor.service.<repo>.allowed.users}} in site XML |
| Partition routing | After accept, which {{ranger_audits}} partition? |
{{kafka.configured.plugins}} + per-plugin overrides in site XML |
Changing either requires editing XML and restarting every ingestor replica.
Contiguous-range static allocation can also reshuffle later plugins when an
early plugin's partition count changes.
*Goals:*
* Onboard new plugins/repos and scale hot plugins without ingestor restart
* Append-only partition growth (no reshuffle of existing plugin assignments)
* One shared source of truth across all ingestor pods
* No new infra (no Postgres / ZooKeeper for the registry)
----
h2. Solution
Introduce a unified partition plan document (versioned JSON) in Kafka topic
{{ranger_audit_partition_plan}} (1 partition, compacted). One document holds:
* {{plugins}} — dedicated partition IDs per plugin id (Kafka record key / agent
id)
* {{buffer}} — partition pool for not-yet-promoted plugins (sticky hash)
* {{services}} — per-repo {{allowedUsers}} for {{POST /api/audit/access}}
(optional {{pluginId}} for ownership)
* {{topicPartitionCount}} — must match live {{ranger_audits}} partition count
* {{version}} — optimistic locking for REST mutations
{code}
Control plane: GET/PATCH /api/audit/partition-plan → Kafka registry topic →
PartitionPlanWatcher
Data plane: POST /api/audit/access (Kerberos) → allowlist check
→ optional auto-onboard
→ AuditPartitioner → ranger_audits
{code}
Solr/HDFS dispatchers are unchanged; they consume all partitions of
{{ranger_audits}}.
h3. REST API (v2 — two public endpoints)
Plan admin endpoints are *unauthenticated* (same as health/status):
{{permitAll()}} in Spring Security and bypass in
{{AuditDelegationTokenFilter}}. Audit ingestion remains Kerberos-protected on
{{POST /api/audit/access}}.
All mutations use optimistic locking via {{expectedVersion}} (from GET). Stale
writes return *409* with the current plan body.
|| Method || Endpoint || Purpose ||
| GET | {{/api/audit/partition-plan}} | Read current plan (503 when dynamic
mode off) |
| PATCH | {{/api/audit/partition-plan}} | Partial update via
{{PartitionPlanReplacement}} JSON body |
*PATCH body semantics* ({{PartitionPlanReplacement}}):
* {{expectedVersion}} — required; stale → 409
* {{plugins}} / {{services}} — merge *new keys only* (promote/onboard entries
not already in plan)
* {{pluginScales}} — map of {{pluginId → additionalPartitions}}; append-only
tail growth
* {{buffer}} / {{topicPartitionCount}} — optional; omitted fields inherited
from current plan
*Removed (superseded by PATCH + auto-onboard):*
* {{POST /api/audit/partition-plan/plugins}}
* {{PATCH /api/audit/partition-plan/plugins/\{pluginId\}}}
h3. Auto-onboard on audit access
When dynamic mode is enabled and a Kerberos-authenticated {{POST
/api/audit/access}} includes {{appId}} for a plugin not yet in the plan:
# Allowlist check passes for {{serviceName}} + authenticated principal
# Ingestor promotes the plugin from buffer (default partition count from
config) and upserts {{services[serviceName]}} in one plan version bump
# Audit batch is then produced using the updated in-memory plan
No separate onboard REST call is required for first-time plugin discovery.
Operators can still pre-promote plugins and allowlists via PATCH.
h3. Allowlist updates
{{services}} entries in the registry drive {{POST /access}} authorization. In
dynamic mode, {{AuthToLocalRuleComposer}} recomposes {{auth_to_local}} from the
union of all {{services[*].allowedUsers}} whenever the plan changes. Removing
or tightening a repo's allowlist causes *403* on subsequent access attempts
without pod restart.
----
h2. Key deliverables (PR #1032)
* Kafka registry — {{KafkaPartitionPlanRegistry}}, bootstrap from XML on
greenfield, brownfield pre-seed support
* REST API — {{GET}} / {{PATCH}} {{/api/audit/partition-plan}} with
{{PartitionPlanReplacement}} and {{expectedVersion}} (409 on stale)
* Auto-onboard — {{PartitionPlanService.ensurePluginOnboarded()}} on {{POST
/api/audit/access}} when {{appId}} is unknown
* Dynamic partitioner — {{AuditPartitioner}} reads {{PartitionPlanHolder}};
round-robin for promoted plugins; buffer sticky hash for unknown plugins;
post-scale routing uses {{max(cluster, plan)}} when metadata lags
* Unified allowlist — {{services}} map in same registry; per-repo POST checks
use each repo's own allowlist
* Topic grow — grow {{ranger_audits}} before registry write when
promoting/scaling
* Unit tests — partition-plan, allowlist, partitioner, and bootstrap coverage
in {{audit-common}} + {{audit-ingestor}}
*Out of scope for PR #1032 (follow-up / lab validation):* Docker E2E harness,
ops runbooks, brownfield migration guide (tracked separately in dev-support /
audit-server docs).
----
h2. Configuration
{code:xml}
<property>
<name>ranger.audit.ingestor.kafka.partition.plan.dynamic.enabled</name>
<value>true</value>
</property>
<property>
<name>ranger.audit.ingestor.kafka.partition.plan.topic</name>
<value>ranger_audit_partition_plan</value>
</property>
{code}
When dynamic mode is on and the registry topic is empty, the first ingestor pod
bootstraps an initial plan from existing XML properties
({{kafka.configured.plugins}}, buffer/per-plugin counts, service allowlists).
h3. Example plan JSON
{code:json}
{
"topic": "ranger_audits",
"version": 12,
"topicPartitionCount": 48,
"plugins": {
"hdfs": { "partitions": [0, 1, 2, 3, 4, 5] },
"hiveServer2": { "partitions": [6, 7, 8, 9, 10, 11] }
},
"buffer": { "partitions": [12, 13, "..."] },
"services": {
"dev_hive": { "allowedUsers": ["hive"], "pluginId": "hiveServer2" },
"dev_ozone": { "allowedUsers": ["om", "ozone"], "pluginId": "ozone" },
"dev_hdfs": { "allowedUsers": ["hdfs", "nn"], "pluginId": "hdfs" }
}
}
{code}
h3. Example PATCH (promote + scale)
{code:json}
{
"expectedVersion": 12,
"plugins": {
"storm": { "partitions": [12, 13, 14] }
},
"services": {
"dev_storm": { "allowedUsers": ["storm"], "pluginId": "storm" }
},
"pluginScales": {
"hdfs": 2
}
}
{code}
----
h2. Testing
h3. Unit tests and quality gates
{code:bash}
mvn verify -pl audit-server/audit-common,audit-server/audit-ingestor
-Drat.skip=true
{code}
|| Gate || Result ||
| Unit tests | 104 passed (audit-ingestor module) |
| Checkstyle | Pass |
| PMD / SpotBugs | Pass (existing audit-ingestor findings unchanged) |
h3. Validated scenarios (manual + lab)
|| # || Scenario || Pass criteria ||
| 1 | Static mode (default off) | {{GET /partition-plan}} → 503; health 200;
plugin audits unchanged |
| 2 | Dynamic bootstrap | Plan topic created; watcher active; {{GET
/partition-plan}} → 200 with v≥1 |
| 3 | Public plan REST | {{GET}} / {{PATCH}} without Kerberos succeed when
dynamic mode on |
| 4 | PATCH promote/onboard | New {{plugins}} + {{services}} keys merged in one
version bump |
| 5 | PATCH {{pluginScales}} | Tail partitions appended; {{ranger_audits}}
grown; no reshuffle |
| 6 | Optimistic locking | Stale {{expectedVersion}} → 409 with current plan
body |
| 7 | Auto-onboard on {{/access}} | Unknown {{appId}} + allowlisted principal →
plan updated, audit accepted |
| 8 | Multi-pod convergence | All replicas report same version after PATCH on
one pod |
| 9 | Routing | Kafka record partition ∈ plugin assignment from plan |
| 10 | Allowlist enforcement | Allow/deny/cross-repo → 200/403; recomposition
on plan change |
----
h2. Acceptance criteria
# With {{dynamic.enabled=false}}, behavior matches existing static XML
partitioning; {{GET /partition-plan}} returns 503
# With {{dynamic.enabled=true}}, registry topic created (1 partition,
compacted); bootstrap plan published on greenfield
# {{PATCH /api/audit/partition-plan}} promotes/scales/mutates allowlists; stale
{{expectedVersion}} returns 409
# Unknown {{appId}} on authenticated {{POST /access}} auto-onboards plugin when
allowlist permits
# All ingestor replicas converge to same plan within watcher refresh interval
# Audits for promoted plugin land only on assigned {{ranger_audits}} partitions
# Service allowlists in registry drive {{/access}} authorization; unauthorized
principal returns 403
# Post-scale routing works when Kafka metadata lags plan ({{AuditPartitioner}}
bound logic)
> Ranger Admin-Managed Kafka Audit Partition Plan & SPIFFE Auth
> -------------------------------------------------------------
>
> Key: RANGER-5655
> URL: https://issues.apache.org/jira/browse/RANGER-5655
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
> Reporter: Ramachandran Krishnan
> Assignee: Ramachandran Krishnan
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Ranger plugins (HDFS, Hive, Trino, etc.) send audit events to
> {*}audit-ingestor{*}, which writes them to Kafka topic {{{}ranger_audits{}}}.
> To isolate hot plugins, ingestor can route each plugin to dedicated Kafka
> partitions; unknown plugins share a *buffer* pool.
> *Today (static mode):* Plugin → partition mapping is computed once at startup
> from XML. Changing assignments requires config edits, topic partition growth,
> and {*}ingestor restart{*}. Early-plugin changes can *reshuffle* later
> plugins because allocation uses contiguous ranges.
> Plugins POST audits to {{{}POST
> /api/audit/access?serviceName=<repo>&appId=<agent>{}}}. After
> Kerberos/JWT/basic *authentication* (401 on failure), ingestor performs
> {*}authorization{*}: the authenticated short username must appear in
> {{{}ranger.audit.ingestor.service.<repo>.allowed.users{}}}.
> Today that map is loaded *once at JVM startup* from
> {{{}ranger-audit-ingestor-site.xml{}}}. Adding a new Ranger service repo
> (e.g. {{{}dev_trino{}}}) requires XML edits and *ingestor restart* — even
> when partition routing is already dynamic via the partition-plan topic.
> *Proposed (dynamic mode):*
> Ranger plugins POST audits to the *Audit Ingestor* ({{{}POST
> /api/audit/access{}}}). This feature adds:
> # *Dynamic Kafka partition routing* — layout stored in Admin
> ({{{}x_ranger_global_state{}}}), synced by a background poller.
> # *Dynamic allow-list* — who may POST audits per service comes from
> {{policy.download.auth.users}} in the partition plan.
> # *SPIFFE workload identity* at two boundaries:
> ** *Plugin → Ingestor* — {{X-Spiffe-Id}} on audit POSTs
> ({{{}AuditHeaderAuthFilter{}}}).
> ** *Ingestor → Admin* — {{X-Spiffe-Id}} as {{rangerauditserver}} for plan
> download (no password).
> Same pattern for Hive, HDFS, Ozone, Knox, Kafka, HBase, KMS, Trino, Atlas,
> etc.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
