This is an automated email from the ASF dual-hosted git repository.
pradeepagrawal8184 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ranger.git
The following commit(s) were added to refs/heads/master by this push:
new 49f04a583 RANGER-5834: Extend functional test framework with
multi-service docker container support (#1304)
49f04a583 is described below
commit 49f04a583e6e1e0ae0b4332021ed94b6b84fe56f
Author: Chinmay Hegde <[email protected]>
AuthorDate: Mon Oct 12 13:20:13 2026 +0530
RANGER-5834: Extend functional test framework with multi-service docker
container support (#1304)
---
functional-tests/readme.md | 162 +++++++++++++++++--------------------
functional-tests/run-tests.sh | 181 +++++++++++++++++++++++++++---------------
2 files changed, 191 insertions(+), 152 deletions(-)
diff --git a/functional-tests/readme.md b/functional-tests/readme.md
index a7c6a317a..8bf786a95 100644
--- a/functional-tests/readme.md
+++ b/functional-tests/readme.md
@@ -18,7 +18,7 @@ limitations under the License.
## Pytest Functional Test Suite
-This test suite validates REST API endpoints for Apache Ranger services —
Admin (rolerest, xuserrest, servicerest,tagrest), KMS (Key Management Service),
and HDFS encryption functionalities including key management and file
operations within encryption zones.
+A Docker-based test execution framework that validates Apache Ranger service
integrations through pytest. Spins up the required containers, runs the test
suites, and tears down — all from a single script. New service modules can be
added as pytest folders without changing the script.
### Available Test Suites
@@ -33,24 +33,14 @@ This test suite validates REST API endpoints for Apache
Ranger services — Admi
| **api** | Public API v2 service and policy CRUD (admin Docker only) |
| **hive** | Hive Ranger plugin enforcement via HS2/beeline (requires
`ranger-hadoop` + `ranger-hive`) |
----
-
### Directory Structure
```text
functional-tests/
-├── ranger_test_utils/utils.py # Shared logging, HTTP asserts, Public API v2
helpers
-├── hdfs/ # Tests on HDFS encryption cycle
-├── kms/ # Tests on KMS REST API
-├── xuserrest/ # Tests on Ranger User/Group/Role REST APIs
-├── rolerest/ # Tests on Ranger Role REST APIs
-├── servicerest/ # Tests on Ranger Service REST APIs
-├── tagrest/ # Tests on Ranger Tag REST APIs
-├── api/ # Public API v2 (service / policy) tests
-├── hive/ # Hive plugin enforcement tests
-├── ranger_test_utils/hive_utils.py
-├── ranger_test_utils/access_audit_utils.py # For future access-audit asserts
+├── multiple modules containing test cases
+├── ...
│
+├── ranger_test_utils/utils.py # Shared logging, HTTP asserts, Public API v2
helpers
├── pytest.ini # Registers custom pytest markers
├── run-tests.sh # Script to automate setup and test execution
├── requirements.txt # Python dependencies
@@ -77,106 +67,95 @@ chmod +x run-tests.sh
---
-## Environment Variables
-
-These can be exported before running, or passed inline. All have sensible
defaults
-
-| Variable | Default | Description |
-|---|---|---|
-| `CLEAN_CONTAINERS` | `0` | Set to `1` to wipe all containers and force a
full rebuild from scratch |
-| `RUN_TESTS` | `1` | Set to `0` to bring up infrastructure only without
running any tests |
-| `AUDIT_INDEX_STORE` | `opensearch` | Audit backend: `opensearch`, `solr`, or
`none` to skip the audit pipeline entirely |
+## Running Tests
-### `CLEAN_CONTAINERS`
+The `run-tests.sh` script manages Docker container setup, dependency
installation, and test execution. It supports both interactive and
argument-based modes.
-By default containers **persist between runs** for fast re-execution. Use
`CLEAN_CONTAINERS=1` only when you want a completely clean rebuild (e.g. for
rebuilding the Docker image via Maven against your local source):
+### 1. Interactive Mode (Preferred)
+Run the script without arguments to be prompted for each input:
```text
-export CLEAN_CONTAINERS=1
./run-tests.sh
```
-On all subsequent re-runs (no rebuild needed):
+You will be asked **four** questions in sequence:
+
```text
-./run-tests.sh
-```
+Available DB types: postgres, mysql, oracle
+Enter DB type (press Enter to default to postgres): postgres
-### `RUN_TESTS`
+Available audit stores: opensearch, solr, none
+Enter audit store (press Enter to default to opensearch): opensearch
-Bring up the full infrastructure without executing any tests. Useful when
containers are still initializing or you want to inspect the environment first:
+Available test suites: rolerest xuserrest servicerest tagrest hdfs kms api hive
+Enter test suites space-separated (press Enter to run ALL): hdfs kms
-```text
-export RUN_TESTS=0
-./run-tests.sh
+Available containers : hadoop hive hbase knox trino ozone kafka
+ (base always starts: ranger, kdc, db, zk, kms, pdp, usersync, tagsync +
audit pipeline)
+Enter containers space-separated (press Enter or 'all' for ALL): hadoop
```
-Once containers are healthy, run tests normally (default):
-```text
-./run-tests.sh
-```
+> Press **Enter** at any prompt to accept the default. Type `all` at the
containers prompt to bring up every optional container.
-### `AUDIT_INDEX_STORE`
+### 2. Command-Line Arguments Mode
-Controls whether the audit pipeline (Kafka + ingestor + OpenSearch/Solr) is
started. Defaults to `opensearch`. Allowed values for export AUDIT_INDEX_STORE=
(solr/opensearch/none)
+Pass `db-type` and `test-suites` as arguments to skip those prompts. Use env
vars for the remaining options:
```text
-export AUDIT_INDEX_STORE=none
-./run-tests.sh
+./run-tests.sh [db-type] [test-suites...]
```
----
+- `db-type` — First positional argument. Valid values: `postgres`, `mysql`,
`oracle`.
+- `test-suites` — Remaining arguments, space-separated.
-## Running Tests
-The `run-tests.sh` script manages Docker container setup, dependency
installation, and test execution. It supports both interactive and
argument-based modes.
+The containers prompt always appears unless `SELECTED_CONTAINERS` is exported.
-1. Interactive Mode:
+**Examples:**
-### 1. Interactive Mode
-
-Run the script without arguments to be prompted for each input:
```text
-./run-tests.sh
-```
+# Run hdfs and kms tests — postgres db, no audit pipeline, hadoop container
only
+export AUDIT_INDEX_STORE=none
+export SELECTED_CONTAINERS=hadoop
+./run-tests.sh postgres hdfs kms
-You will be asked three questions in sequence:
+# Run all test suites — mysql db, opensearch audit, all optional containers
+export SELECTED_CONTAINERS=all
+./run-tests.sh mysql
-```text
-Available DB types: postgres, mysql, oracle
-Enter DB type (press Enter to default to postgres): postgres
+# Full clean rebuild — all suites, all containers
+export CLEAN_CONTAINERS=1
+export SELECTED_CONTAINERS=all
+./run-tests.sh postgres
+```
-Available audit stores: opensearch, solr, none
-Enter audit store (press Enter to default to opensearch): none
+---
-Available test suites: rolerest xuserrest servicerest tagrest hdfs kms
-Enter test suites space-separated (press Enter to run ALL): kms hdfs
-```
+## Environment Variables
-> Press **Enter** at any prompt to accept the default value.
+All variables have sensible defaults and can be exported before running or
passed inline.
-### 2. Command-Line Arguments Mode
+| Variable | Default | Description |
+|---|---|---|
+| `CLEAN_CONTAINERS` | `0` | Set to `1` to wipe all containers and force a
full rebuild from scratch |
+| `RUN_TESTS` | `1` | Set to `0` to bring up infrastructure only without
running any tests |
+| `AUDIT_INDEX_STORE` | `opensearch` | Audit backend: `opensearch`, `solr`, or
`none` to skip the audit pipeline entirely |
+| `SELECTED_CONTAINERS` | _(prompted)_ | Space-separated optional containers
to bring up, e.g. `"hadoop hive"`. Set to `"all"` to bring up every optional
container. |
-Pass `db-type` and `test-suites` directly to skip prompts. Use env vars for
the remaining options:
+### `CLEAN_CONTAINERS`
-```text
-./run-tests.sh [db-type] [test-suites...]
-```
+By default containers **persist between runs** for fast re-execution. Set to
`1` only when you need a completely clean rebuild — e.g. after a Maven source
change or to reset all data. On all subsequent re-runs, leave it unset to reuse
existing containers.
-- `db-type` — First argument. Valid values: `postgres`, `mysql`, `oracle`.
-- `test-suites` — Space-separated list: `hdfs`, `hive`, `kms`, `rolerest`,
`xuserrest`, `servicerest`, `tagrest`, `api`.
+### `RUN_TESTS`
-Examples:
+Set to `0` to bring up the full infrastructure without executing any tests.
Useful when containers are still initializing or you want to inspect the
environment first before running tests.
-```text
-# Run kms and hdfs tests with postgres, no audit pipeline
-export AUDIT_INDEX_STORE=none
-./run-tests.sh postgres kms hdfs
+### `AUDIT_INDEX_STORE`
-# Run all suites with mysql and opensearch audit
-./run-tests.sh mysql
+Controls whether the audit pipeline (Kafka + ingestor + OpenSearch/Solr) is
started. Defaults to `opensearch`. Set to `none` to skip the entire audit
pipeline — no Kafka download, no audit containers.
-# Full clean rebuild with all suites
-CLEAN_CONTAINERS=1 ./run-tests.sh postgres
-```
+### `SELECTED_CONTAINERS`
+
+Pre-select which optional containers to bring up without being prompted
interactively. Accepts a space-separated list of container names (e.g. `"hadoop
hive"`) or `all` to bring up every optional container. When not set, the script
prompts at runtime.
---
@@ -184,7 +163,7 @@ CLEAN_CONTAINERS=1 ./run-tests.sh postgres
### Always (Base Services)
-Regardless of which test suites you choose, these containers always start:
+Regardless of which test suites or containers you choose, these always start.
This matches the `ranger-core` command in `dev-support/ranger-docker/README.md`.
| Container | Role |
|---|---|
@@ -193,6 +172,9 @@ Regardless of which test suites you choose, these
containers always start:
| `ranger-<db>` | Database (`ranger-postgres`, `ranger-mysql`,
`ranger-oracle`) |
| `ranger-zk` | ZooKeeper |
| `ranger-kms` | Ranger KMS |
+| `ranger-pdp` | Ranger PDP (Policy Decision Point) |
+| `ranger-usersync` | User/group sync from LDAP or Unix |
+| `ranger-tagsync` | Tag-based policy sync |
### Audit Pipeline (when `AUDIT_INDEX_STORE != none`)
@@ -203,17 +185,22 @@ Regardless of which test suites you choose, these
containers always start:
| `ranger-opensearch` / `ranger-solr` | Audit index store |
| `ranger-audit-dispatcher-<store>` | Dispatches audit events |
-### Suite-Specific
+### Optional Containers (selected at runtime)
-| Suite | Extra Container |
-|---|---|
-| `hdfs` | `ranger-hadoop` |
-| `hive` | `ranger-hadoop`, `ranger-hive` |
-| `kms` | _(already in base)_ |
-| `rolerest`, `xuserrest`, `servicerest`, `tagrest`| _(none — use Ranger Admin
API only)_ |
+Brought up based on what you enter at the containers prompt (or
`SELECTED_CONTAINERS` env var):
----
+| Container name entered | Containers started | Notes |
+|---|---|---|
+| `hadoop` | `ranger-hadoop` | HDFS + YARN |
+| `hive` | `ranger-hadoop`, `ranger-hive` | hadoop auto-included |
+| `hbase` | `ranger-hadoop`, `ranger-hbase` | hadoop auto-included |
+| `knox` | `ranger-knox` | |
+| `trino` | `ranger-trino` | |
+| `ozone` | `ozone-om`, `ozone-scm`, `ozone-datanode` | pre-setup script runs
automatically |
+| `kafka` | `ranger-kafka` | |
+| `all` / _(Enter)_ | All of the above | |
+---
### Hive suite notes
@@ -222,4 +209,5 @@ Regardless of which test suites you choose, these
containers always start:
- Example: `./run-tests.sh postgres hive`
## Test Reports
+
After execution, an HTML report is automatically generated for each suite in
the `functional-tests/` directory:
diff --git a/functional-tests/run-tests.sh b/functional-tests/run-tests.sh
index efa7dae95..56493c276 100755
--- a/functional-tests/run-tests.sh
+++ b/functional-tests/run-tests.sh
@@ -19,12 +19,13 @@
ALL_TEST_SUITES=(rolerest xuserrest servicerest tagrest hdfs kms api hive)
-# Suites that have actual docker-compose services
-DOCKER_SERVICES=(hdfs kms hive)
-
# Test-only suites (no docker-compose file needed)
TEST_ONLY_SUITES=(rolerest xuserrest servicerest tagrest api)
+# All optional containers (on top of base services) — shown in the containers
prompt
+# Each maps to a docker-compose file and a set of archives to download.
+ALL_OPTIONAL_CONTAINERS=(hadoop hive hbase knox trino ozone kafka)
+
#handle input
DB_TYPE="${1:-}"
shift || true
@@ -76,11 +77,36 @@ if [[ "${#EXTRA_SERVICES[@]}" -eq 0 ]]; then
fi
fi
+# Prompt for optional containers to bring up (independent of test suites)
+# Base services always start: ranger, kdc, db, zk, kms, pdp, usersync, tagsync
+ audit pipeline.
+# Pre-set via env var (space-separated): export SELECTED_CONTAINERS="hadoop
hive" OR "all"
+SELECTED_CONTAINERS_ENV="${SELECTED_CONTAINERS:-}"
+SELECTED_CONTAINERS=()
+if [[ -n "${SELECTED_CONTAINERS_ENV}" ]]; then
+ if [[ "${SELECTED_CONTAINERS_ENV}" == "all" ]]; then
+ SELECTED_CONTAINERS=("${ALL_OPTIONAL_CONTAINERS[@]}")
+ else
+ read -ra SELECTED_CONTAINERS <<< "${SELECTED_CONTAINERS_ENV}"
+ fi
+else
+ echo ""
+ echo "Available containers : ${ALL_OPTIONAL_CONTAINERS[*]}"
+ echo " (base always starts: ranger, kdc, db, zk, kms, pdp, usersync,
tagsync + audit pipeline)"
+ read -rp "Enter containers space-separated (press Enter or 'all' for ALL): "
-a input_containers
+ if [[ "${#input_containers[@]}" -eq 0 || "${input_containers[0]}" == "all"
]]; then
+ echo "Bringing up ALL optional containers..."
+ SELECTED_CONTAINERS=("${ALL_OPTIONAL_CONTAINERS[@]}")
+ else
+ SELECTED_CONTAINERS=("${input_containers[@]}")
+ fi
+fi
+
echo ""
echo "DB Type : ${DB_TYPE}"
echo "Audit Store : ${AUDIT_INDEX_STORE}"
echo "Audit Destinations: ${AUDIT_DESTINATIONS}"
echo "Test Suites : ${EXTRA_SERVICES[*]}"
+echo "Containers : ${SELECTED_CONTAINERS[*]:-none (base only)}"
echo ""
echo "CLEAN_CONTAINERS=${CLEAN_CONTAINERS}"
@@ -100,26 +126,19 @@ cd "$RANGER_DOCKER_PATH" || exit 1
chmod +x scripts/**/*.sh || true
chmod +x download-archives.sh || true
-# Download archives — only for docker-backed services
+# Download archives for selected optional containers
+# 'hive' arg downloads hadoop + tez + hive (all needed by
Dockerfile.ranger-hadoop and ranger-hive)
DOCKER_BACKED=()
-for service in "${EXTRA_SERVICES[@]}"; do
- for ds in "${DOCKER_SERVICES[@]}"; do
- if [[ "$service" == "$ds" ]]; then
- case "$service" in
- hdfs|hive)
- # 'hive' arg downloads hadoop + tez (both needed by
Dockerfile.ranger-hadoop)
- # 'hadoop' arg alone does NOT download tez
- DOCKER_BACKED+=("hive")
- ;;
- kms)
- : # KMS needs no archive download — it is always in BASE_SERVICES as
a container
- ;;
- *)
- DOCKER_BACKED+=("$service")
- ;;
- esac
- fi
- done
+for container in "${SELECTED_CONTAINERS[@]}"; do
+ case "$container" in
+ hadoop) DOCKER_BACKED+=("hive") ;; # 'hive' arg downloads hadoop
+ tez (needed by Dockerfile.ranger-hadoop)
+ hive) DOCKER_BACKED+=("hive") ;;
+ hbase) DOCKER_BACKED+=("hive" "hbase") ;; # hadoop + tez (compose dep) +
hbase image
+ knox) DOCKER_BACKED+=("knox") ;;
+ trino) ;; # no archive — pulled from
Docker image directly
+ ozone) DOCKER_BACKED+=("ozone") ;;
+ kafka) DOCKER_BACKED+=("kafka") ;;
+ esac
done
# Kafka archive is needed whenever the audit pipeline is up (ranger-kafka
build)
@@ -174,11 +193,13 @@ else
echo "Admin service (${ADMIN_SERVICE}) already exists. Skipping build/up."
fi
-# Bring up basic services
+# Bring up base services — matches README ranger-core command:
+# ranger + usersync + tagsync + pdp + kms (+ audit-service when audit store !=
none)
DOCKER_FILES=(
"-f" "docker-compose.ranger.yml"
"-f" "docker-compose.ranger-usersync.yml"
"-f" "docker-compose.ranger-tagsync.yml"
+ "-f" "docker-compose.ranger-pdp.yml"
"-f" "docker-compose.ranger-kms.yml"
)
@@ -187,24 +208,38 @@ if [[ "${AUDIT_INDEX_STORE}" != "none" ]]; then
DOCKER_FILES+=("-f" "docker-compose.ranger-audit-service.yml")
fi
-# Add compose files based on requested suites
-for service in "${EXTRA_SERVICES[@]}"; do
- case "$service" in
- hdfs)
- DOCKER_FILES+=("-f" "docker-compose.ranger-hadoop.yml")
- ;;
- hive)
- DOCKER_FILES+=("-f" "docker-compose.ranger-hadoop.yml")
- DOCKER_FILES+=("-f" "docker-compose.ranger-hive.yml")
- ;;
- kms)
- # already included in base
- ;;
+# Helper: add a compose file only if not already in the list (prevents
duplicates
+# when e.g. both 'hadoop' and 'hive' are selected — both need
ranger-hadoop.yml)
+_add_compose() {
+ local file="$1"
+ local already=false
+ for ac in "${ADDED_COMPOSE[@]}"; do
+ [[ "$ac" == "$file" ]] && already=true && break
+ done
+ if ! $already; then
+ DOCKER_FILES+=("-f" "$file")
+ ADDED_COMPOSE+=("$file")
+ fi
+}
+
+# Add compose files for selected optional containers
+ADDED_COMPOSE=()
+for container in "${SELECTED_CONTAINERS[@]}"; do
+ case "$container" in
+ hadoop) _add_compose "docker-compose.ranger-hadoop.yml" ;;
+ hive) _add_compose "docker-compose.ranger-hadoop.yml" # hive
depends_on: ranger-hadoop: service_healthy
+ _add_compose "docker-compose.ranger-hive.yml" ;;
+ hbase) _add_compose "docker-compose.ranger-hadoop.yml" # hbase
depends_on: ranger-hadoop: service_healthy
+ _add_compose "docker-compose.ranger-hbase.yml" ;;
+ knox) _add_compose "docker-compose.ranger-knox.yml" ;;
+ trino) _add_compose "docker-compose.ranger-trino.yml" ;;
+ ozone) _add_compose "docker-compose.ranger-ozone.yml" ;;
+ kafka) _add_compose "docker-compose.ranger-kafka.yml" ;;
esac
done
-# Build ALL_SERVICES list — only docker-backed services get container checks
-BASE_SERVICES=(ranger ranger-kdc ranger-${RANGER_DB_TYPE} ranger-zk ranger-kms)
+# Build ALL_SERVICES list — only docker-backed services get container health
checks
+BASE_SERVICES=(ranger ranger-kdc ranger-${RANGER_DB_TYPE} ranger-zk ranger-kms
ranger-pdp)
ALL_SERVICES=("${BASE_SERVICES[@]}")
# Audit pipeline containers brought up by
docker-compose.ranger-audit-service.yml
@@ -217,53 +252,69 @@ if [[ "${AUDIT_INDEX_STORE}" != "none" ]]; then
)
fi
-for service in "${EXTRA_SERVICES[@]}"; do
- case "$service" in
- hdfs)
- ALL_SERVICES+=("ranger-hadoop")
- ;;
- hive)
- ALL_SERVICES+=("ranger-hadoop")
- ALL_SERVICES+=("ranger-hive")
- ;;
- kms)
- : # already in BASE_SERVICES
- ;;
+# Add containers for selected optional services
+for container in "${SELECTED_CONTAINERS[@]}"; do
+ case "$container" in
+ hadoop) ALL_SERVICES+=("ranger-hadoop") ;;
+ hive) ALL_SERVICES+=("ranger-hadoop" "ranger-hive") ;;
+ hbase) ALL_SERVICES+=("ranger-hadoop" "ranger-hbase") ;;
+ knox) ALL_SERVICES+=("ranger-knox") ;;
+ trino) ALL_SERVICES+=("ranger-trino") ;;
+ ozone) ALL_SERVICES+=("ozone-om" "ozone-scm" "ozone-datanode") ;;
+ kafka) ALL_SERVICES+=("ranger-kafka") ;;
esac
done
+# Deduplicate — e.g. 'hadoop' + 'hive' both selected adds ranger-hadoop twice
+ALL_SERVICES=($(printf '%s\n' "${ALL_SERVICES[@]}" | sort -u))
-# only create/build if containers do not exist
+# Run ozone pre-setup script if ozone is among selected containers
+for container in "${SELECTED_CONTAINERS[@]}"; do
+ if [[ "$container" == "ozone" ]]; then
+ echo "Running ozone pre-setup..."
+ bash scripts/ozone/ozone-plugin-docker-setup.sh
+ break
+ fi
+done
+
+
+# Determine container state: missing, stopped, or all already running
missing=false
+all_running=true
+
for container in "${ALL_SERVICES[@]}"; do
if ! docker container inspect "$container" >/dev/null 2>&1; then
missing=true
+ all_running=false
break
+ elif [[ $(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)
!= "true" ]]; then
+ all_running=false
fi
done
if [[ "${missing}" == "true" ]]; then
+ # First run or containers were wiped — build and start everything
echo "Some containers are missing. Creating services..."
if [[ -n "${AUDIT_DESTINATIONS}" ]]; then
docker compose --profile "${AUDIT_DESTINATIONS}" "${DOCKER_FILES[@]}" up
-d --build
else
docker compose "${DOCKER_FILES[@]}" up -d --build
fi
-else
- echo "All containers already exist. Starting without rebuild..."
+ echo "Waiting for containers to start..."
+ sleep 20
+elif [[ "${all_running}" == "false" ]]; then
+ # Containers exist but some are stopped — start them without rebuild
+ echo "Some containers are stopped. Starting services..."
if [[ -n "${AUDIT_DESTINATIONS}" ]]; then
docker compose --profile "${AUDIT_DESTINATIONS}" "${DOCKER_FILES[@]}" up -d
else
docker compose "${DOCKER_FILES[@]}" up -d
fi
-fi
-
-echo "Waiting for containers to start..."
-if [[ "${missing}" == "true" || "${admin_missing}" == "true" ]]; then
- sleep 20
else
- echo "No rebuild/start needed. Skipping wait."
+ # All containers already running — skip compose and sleeps entirely
+ echo "All containers already running. Skipping compose and proceeding
directly to tests..."
fi
+
echo "Checking container status..."
flag=true
@@ -296,12 +347,12 @@ else
echo "All containers are running fine!"
fi
-
-echo -e "\n\n\n\n\n\n\n\n\n\n-------------------------------------------"
-echo "Please wait while all services are fully initialized. This may take a
few minutes..."
-echo "-------------------------------------------"
-
-sleep 60 # Wait for 60 seconds to allow services to initialize
+if [[ "${missing}" == "true" || "${admin_missing}" == "true" ||
"${all_running}" == "false" ]]; then
+ echo -e "\n\n\n\n\n\n\n\n\n\n-------------------------------------------"
+ echo "Please wait while all services are fully initialized. This may take a
few minutes..."
+ echo "-------------------------------------------"
+ sleep 60 # Wait for services to initialize after a fresh start or restart
+fi
#RUN TESTS--------
