pradeepagrawal8184 opened a new pull request, #1153:
URL: https://github.com/apache/ranger/pull/1153
## What changes were proposed in this pull request?
This PR makes the Ranger Elasticsearch plugin work with Elasticsearch 7.17 +
X-Pack Security instead of relying on client-supplied identity headers. It adds
end-to-end Docker bring-up, fixes plugin packaging/classloading for ES 7.17,
and wires audits through the Ranger audit ingestor path under Kerberos.
## How was this patch tested?
The two commits were tested at **three levels**: automated unit tests
(commit 1 only), **manual Docker integration** (commit 2, and validating commit
1 in a real ES stack), and **problem reproduction** before the fix. There is
**no new CI/integration test suite** checked into the PR for the Docker or
audit paths.
---
## Commit 1 — `RANGER-5740` (X-Pack caller identity)
### Automated unit tests
Run with:
```bash
mvn -pl ranger-elasticsearch-plugin-shim verify
```
This was run repeatedly during development and passes today. It includes **5
JUnit 5 tests** in two classes:
| Test class | What it verifies |
|------------|------------------|
| `TestElasticsearchAuthenticatedUserResolver` | Verified user comes from
X-Pack `SecurityContext`; null when no user; system context skips auth
requirement |
| `TestRangerSecurityRestFilter` | **Rejects** unverified `Authorization:
Basic` (401, handler not called); **accepts** when X-Pack has set a verified
user |
These are **mock-based unit tests** — they do not start Elasticsearch or
Ranger Admin.
### Manual / planned testing
The commit message / PR test plan also listed manual checks that were **not
automated in the repo**:
- Deploy on ES 7.17 with X-Pack Security enabled
- Authenticated REST requests authorized under the **ES-authenticated** user
- Wrong Basic password must **not** grant another user’s Ranger permissions
### Pre-fix validation (motivation)
The original issue was reproduced separately on **Ranger 2.9.0** in an
isolated runtime replay: wrong Basic password still authorized as the named
user. That confirmed the bug before this fix.
---
## Commit 2 — `RANGER-5743` (Docker + full ES 7.17 / X-Pack integration)
### No new automated tests
This commit adds **no new unit or integration tests**. Validation was done
by **bringing up the Ranger Docker stack** and exercising ES manually.
Documented in `dev-support/ranger-docker/README.md`:
```bash
mvn clean package -pl distro -am -DskipTests
cp target/ranger-* dev-support/ranger-docker/dist/
cd dev-support/ranger-docker
./download-archives.sh elasticsearch
docker compose -f docker-compose.ranger.yml -f
docker-compose.ranger-solr.yml \
-f docker-compose.ranger-elasticsearch.yml up -d --build
```
Smoke tests from the README:
```bash
curl -u elastic:rangerR0cks! http://localhost:9201/test-index/_search #
expect 200 or policy 403, not 401
curl http://localhost:9201/test-index/_search #
expect 401
```
### Docker integration testing (manual, iterative)
The second commit was validated through many **rebuild / recreate** cycles
of `ranger-elasticsearch` against a running stack (Ranger Admin, KDC, Solr,
Kafka, audit ingestor, etc.). Issues found and fixed during testing included:
- Plugin tarball / `.dockerignore` / classpath /
`extended.plugins=x-pack-security`
- REST wrapper conflict with X-Pack (removed Ranger REST wrapper)
- ES Security Manager / plugin policy / classloader init
- Policy cache path and seeding (`ranger-elasticsearch-post-setup.sh`)
- Internal ES users (`_system`, `internal:*`) bypass in
`RangerSecurityActionFilter`
- Kerberos keytab + JAAS for audit posts to ingestor
- Ingestor `AuditREST` lazy allowed-users + `dev_elasticsearch` site XML
### Authorization smoke tests (verified in Docker)
Against `test-index` on port **9201**:
| Request | Expected | Observed |
|---------|----------|----------|
| Unauthenticated | 401 | **401** |
| `elastic` | 200 | **200** |
| `testuser_2` (in Ranger policy) | 200 | **200** |
| `testuser_denied` (ES role OK, not in Ranger policy) | 403 | **403** |
Test users are created offline via file realm (`roles.yml` +
`elasticsearch-users`) because Ranger blocks ES security REST APIs.
### Audit pipeline testing (partial)
Also tested manually in the same Docker stack:
- Sync audit client on ES request thread (`ElasticsearchAuditIngestorClient`)
- Kerberos SPNEGO to audit ingestor (not plain HTTP)
- ES logs showed successful posts, e.g.:
- `Audit event posted to ingestor for service=dev_elasticsearch
user=testuser_2`
- `Audit event posted to ingestor for service=dev_elasticsearch
user=testuser_denied`
**Not fully verified end-to-end:** audits appearing in **Solr** via
Kafka/dispatcher (Solr Kerberos blocked direct query verification).
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
