Ramachandran Krishnan created RANGER-5722:
---------------------------------------------
Summary: Ingestor dynamic routing, SPIFFE inbound auth, and Admin
plan poller
Key: RANGER-5722
URL: https://issues.apache.org/jira/browse/RANGER-5722
Project: Ranger
Issue Type: Sub-task
Components: Ranger
Reporter: Ramachandran Krishnan
Assignee: Ramachandran Krishnan
Fix For: 3.0.0
Extends the Audit Ingestor so it can accept SPIFFE-authenticated audit POSTs
from plugins, poll Ranger Admin for the canonical partition plan, and route
Kafka messages using Admin-managed layout and allow-lists.
This PR completes the ingestor side of the feature. Plugin outbound SPIFFE
(PR5) should land after this so callers and the ingestor agree on
authentication before rollout.
h3. Admin plan poller (dynamic mode)
When dynamic partition routing is enabled, a background poller:
* Calls Admin’s partition-plan download API on a configurable interval
(default ~30s)
* Sends {{X-Spiffe-Id}} as {{rangerauditserver}} (SPIFFE only — no password on
this path)
* Uses conditional download: 304 when unchanged, 200 + full plan JSON when
version is newer
* Installs routing into an in-memory plan holder used by the Kafka partitioner
* Installs per-service allow-lists from the plan for audit POST authorization
* Optionally refreshes effective Kafka topic partition count from cluster
metadata
Admin remains the source of truth; ingestor picks up service create/delete and
partition layout changes without restart.
h3. Dynamic Kafka routing
The audit Kafka partitioner uses the synced plan to choose a partition per
plugin/service ({{{}appId{}}}), mapping Admin’s 1-based partition IDs to
Kafka’s 0-based indices with safe fallbacks when topic metadata lags.
When dynamic mode is off ({{{}partition.plan.admin.enabled=false{}}}), behavior
falls back to existing static site-XML configuration — SPIFFE inbound auth
still works in both modes.
h3. SPIFFE inbound authentication
Adds header-based authentication on the ingestor audit REST path (RANGER-5700
trusted-proxy pattern):
* Reads {{X-Spiffe-Id}} (and optional alternate SPIFFE header names)
* Extracts the Kubernetes service account from a well-formed SPIFFE ID as the
authenticated principal
* Integrates with the existing filter chain alongside optional JWT and
Kerberos paths (each filter skips if already authenticated)
* 401 when no valid identity; 403 when identity does not match the requested
{{serviceName}} / allow-list
Typical K8s flow: plugin sends {{X-Spiffe-Id: …/sa/hive}} with
{{{}serviceName=dev_hive{}}}; ingestor validates SPIFFE → principal {{hive}} →
checks allow-list → produces to the Hive partition slot from the plan.
h3. Ingestor → Admin SPIFFE outbound
The plan-download client authenticates to Admin using the same trusted-header
SPIFFE model ({{{}rangerauditserver{}}} identity), relying on RANGER-5700 on
Admin and the {{ROLE_ADMIN_AUDITOR}} seed from DB patch 078 (PR2).
h3. Runtime authorization on audit POST
On each {{{}POST /api/audit/access{}}}:
# Authenticate via SPIFFE (or existing JWT/Kerberos if enabled)
# Check {{AllowedUsersHolder}} — populated from the synced plan’s
{{serviceAllowedUsers}} (from each repo’s {{{}policy.download.auth.users{}}})
# Route to Kafka via {{AuditPartitioner}} using the synced plan
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
