kumaab commented on code in PR #986: URL: https://github.com/apache/ranger/pull/986#discussion_r3437395076
########## dev-support/ranger-docker/README.md: ########## @@ -106,6 +106,72 @@ docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-trino.yml u ~~~ docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-opensearch.yml up -d ~~~ + +#### OpenSearch audit flow (replace Solr for access audits) + +OpenSearch can replace Solr for **audit storage and UI queries**. Ranger Admin reads audits via +`audit_store=opensearch` using a native low-level REST client (compatible with any OpenSearch version). + +**Write path:** access audits flow through audit-server ingestor, Kafka, and the Java +`ranger-audit-dispatcher-opensearch` service into the OpenSearch `ranger_audits` index. +Ranger Admin policy/admin transaction audits remain DB-backed; this is the same boundary +used by the Solr audit path. + +##### Quick start + +~~~ +# Prerequisites: build the audit-dispatcher tarball and download archives +mvn clean package -DskipTests -pl distro -am +cp target/ranger-*-audit-dispatcher.tar.gz dev-support/ranger-docker/dist/ +cd dev-support/ranger-docker +./download-archives.sh kafka opensearch hadoop + +# Run the E2E test (starts stack, tests, tears down automatically) Review Comment: Is this script included in the PR? ########## security-admin/scripts/install.properties: ########## @@ -101,6 +101,15 @@ audit_elasticsearch_password= audit_elasticsearch_index= audit_elasticsearch_bootstrap_enabled=true +# * OpenSearch audit store properties (when audit_store=opensearch) Review Comment: Are these being used given they are already present in `ranger-admin-site.xml`? ########## dev-support/ranger-docker/README.md: ########## @@ -106,6 +106,72 @@ docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-trino.yml u ~~~ docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-opensearch.yml up -d ~~~ + +#### OpenSearch audit flow (replace Solr for access audits) + +OpenSearch can replace Solr for **audit storage and UI queries**. Ranger Admin reads audits via +`audit_store=opensearch` using a native low-level REST client (compatible with any OpenSearch version). + +**Write path:** access audits flow through audit-server ingestor, Kafka, and the Java +`ranger-audit-dispatcher-opensearch` service into the OpenSearch `ranger_audits` index. +Ranger Admin policy/admin transaction audits remain DB-backed; this is the same boundary +used by the Solr audit path. + +##### Quick start + +~~~ +# Prerequisites: build the audit-dispatcher tarball and download archives +mvn clean package -DskipTests -pl distro -am +cp target/ranger-*-audit-dispatcher.tar.gz dev-support/ranger-docker/dist/ +cd dev-support/ranger-docker +./download-archives.sh kafka opensearch hadoop + +# Run the E2E test (starts stack, tests, tears down automatically) +./scripts/audit/e2e-audit-opensearch.sh + +# Or keep the stack running after the test for debugging +./scripts/audit/e2e-audit-opensearch.sh --no-teardown + +# Re-run just the test against an already-running stack +./scripts/audit/e2e-audit-opensearch.sh --skip-start +~~~ + +##### Manual setup (advanced) + +For finer control, the individual steps can be run manually: + +~~~ +export RANGER_DB_TYPE=postgres + +# 1. Start OpenSearch first (Ranger Admin's bootstrapper needs it on startup) +docker compose -f docker-compose.ranger.yml -f docker-compose.ranger-opensearch.yml \ Review Comment: Steps like this should be included in Ranger CI: https://github.com/apache/ranger/blob/master/.github/workflows/ci.yml ########## dev-support/ranger-docker/docker-compose.ranger.yml: ########## @@ -81,6 +81,9 @@ services: image: ranger-zk container_name: ranger-zk hostname: ranger-zk.rangernw + depends_on: Review Comment: I'm trying to understand why should ZK depend on KDC? ########## audit-server/audit-dispatcher/dispatcher-opensearch/src/test/java/org/apache/ranger/audit/dispatcher/TestOpenSearchDispatcherManager.java: ########## @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.ranger.audit.dispatcher; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; + +import java.util.Properties; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; +import static org.junit.jupiter.api.Assertions.assertThrows; + +public class TestOpenSearchDispatcherManager { + @AfterEach + void clearSystemProperty() { + System.clearProperty("ranger.audit.dispatcher.type"); + } + + @Test + void init_skipsWhenDispatcherTypeIsNotOpenSearch() { + System.setProperty("ranger.audit.dispatcher.type", "solr"); + + OpenSearchDispatcherManager manager = new OpenSearchDispatcherManager(); + Properties props = new Properties(); + + assertDoesNotThrow(() -> manager.init(props)); + } + + @Test + void init_throwsWhenPropsAreNull() { + OpenSearchDispatcherManager manager = new OpenSearchDispatcherManager(); + + assertThrows(RuntimeException.class, () -> manager.init(null)); + } + + @Test + void init_skipsWhenOpenSearchDestinationDisabled() { + OpenSearchDispatcherManager manager = new OpenSearchDispatcherManager(); + Properties props = new Properties(); + props.setProperty("xasecure.audit.destination.elasticsearch", "false"); Review Comment: I think the config property should refer to `opensearch`? ########## security-admin/scripts/setup.sh: ########## @@ -102,6 +102,13 @@ audit_elasticsearch_user=$(get_prop 'audit_elasticsearch_user' $PROPFILE) audit_elasticsearch_password=$(get_prop 'audit_elasticsearch_password' $PROPFILE) audit_elasticsearch_index=$(get_prop 'audit_elasticsearch_index' $PROPFILE) audit_elasticsearch_bootstrap_enabled=$(get_prop 'audit_elasticsearch_bootstrap_enabled' $PROPFILE) +audit_opensearch_urls=$(get_prop 'audit_opensearch_urls' $PROPFILE) Review Comment: Since these properties are already added in `ranger-admin-site.xml`, are they really need in this script ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
