RussellSpitzer commented on code in PR #2: URL: https://github.com/apache/polaris-tools/pull/2#discussion_r2023134652
########## benchmarks/README.md: ########## @@ -0,0 +1,242 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Polaris Benchmarks + +This repository contains benchmarks for the Polaris service using Gatling. + +## Available Benchmarks + +### Dataset Creation Benchmark + +The CreateTreeDataset benchmark creates a test dataset with a specific structure. It exists in two variants: + +- `org.apache.polaris.benchmarks.simulations.CreateTreeDatasetSequential`: Creates entities one at a time +- `org.apache.polaris.benchmarks.simulations.CreateTreeDatasetConcurrent`: Creates up to 50 entities simultaneously + +These are write-only workloads designed to populate the system for subsequent benchmarks. + +### Read/Update Benchmark + +The ReadUpdateTreeDataset benchmark tests read and update operations on an existing dataset. It exists in two variants: + +- `org.apache.polaris.benchmarks.simulations.ReadUpdateTreeDatasetSequential`: Performs read/update operations one at a time +- `org.apache.polaris.benchmarks.simulations.ReadUpdateTreeDatasetConcurrent`: Performs up to 20 read/update operations simultaneously + +These benchmarks can only be run after using CreateTreeDataset to populate the system. + +## Parameters + +All parameters are configured through the `application.conf` file located in `src/gatling/resources/`. The configuration uses the [Typesafe Config](https://github.com/lightbend/config) format. + +### Dataset Structure Parameters + +These parameters must be consistent across all benchmarks and are configured under `dataset.tree`: + +```hocon +dataset.tree { + num-catalogs = 1 # Number of catalogs to create + namespace-width = 2 # Width of the namespace tree + namespace-depth = 4 # Depth of the namespace tree + tables-per-namespace = 5 # Tables per namespace + views-per-namespace = 3 # Views per namespace + columns-per-table = 10 # Columns per table + columns-per-view = 10 # Columns per view + default-base-location = "file:///tmp/polaris" # Base location for datasets + namespace-properties = 10 # Number of properties to add to each namespace + table-properties = 10 # Number of properties to add to each table + view-properties = 10 # Number of properties to add to each view + max-tables = -1 # Maximum tables (-1 for unlimited) + max-views = -1 # Maximum views (-1 for unlimited) +} +``` + +### Connection Parameters + +Connection settings are configured under `http` and `auth`: + +```hocon +http { + base-url = "http://localhost:8181"; # Service URL +} + +auth { + client-id = null # Required: OAuth2 client ID + client-secret = null # Required: OAuth2 client secret +} +``` + +### Workload Parameters + +Workload settings are configured under `workload`: + +```hocon +workload { + read-write-ratio = 0.8 # Ratio of reads (0.0-1.0) +} +``` + +## Running the Benchmarks + +The benchmark uses [typesafe-config](https://github.com/lightbend/config) for configuration management. Default settings are in `src/gatling/resources/benchmark-defaults.conf`. This file should not be modified directly. + +To customize the benchmark settings, create your own `application.conf` file and specify it using the `-Dconfig.file` parameter. Your settings will override the default values. + +Example `application.conf`: +```hocon +auth { + client-id = "your-client-id" + client-secret = "your-client-secret" +} + +http { + base-url = "http://your-polaris-instance:8181"; +} + +workload { + read-write-ratio = 0.8 +} +``` + +Run benchmarks with your configuration: + +```bash +# Sequential dataset creation +./gradlew gatlingRun --simulation org.apache.polaris.benchmarks.simulations.CreateTreeDatasetSequential \ + -Dconfig.file=./application.conf + +# Concurrent dataset creation +./gradlew gatlingRun --simulation org.apache.polaris.benchmarks.simulations.CreateTreeDatasetConcurrent \ + -Dconfig.file=./application.conf +``` + +A message will show the location of the Gatling report: +``` +Reports generated in: ./benchmarks/build/reports/gatling/<simulation-name>/index.html +``` + +### Example Polaris server startup + +For repeated testing and benchmarking purposes it's convenient to have fixed client-ID + client-secret combinations. **The following example is ONLY for testing and benchmarking against an airgapped Polaris instance** + +```bash +# Start Polaris with the fixed client-ID/secret admin/admin +# DO NEVER EVER USE THE FOLLOWING FOR ANY NON-AIRGAPPED POLARIS INSTANCE !! +./gradlew :polaris-quarkus-server:quarkusBuild && java \ + -Dpolaris.bootstrap.credentials=POLARIS,admin,admin \ + -Djava.security.manager=allow \ + -jar quarkus/server/build/quarkus-app/quarkus-run.jar +``` + +With the above you can run the benchmarks using a configuration file with `client-id = "admin"` and `client-secret = "admin"` - meant only for convenience in a fully airgapped system. + +# Test Dataset + +The benchmarks use synthetic procedural datasets that are generated deterministically at runtime. This means that given the same input parameters, the exact same dataset structure will always be generated. This approach allows generating large volumes of test data without having to store it, while ensuring reproducible benchmark results across different runs. + +The diagrams below describe the data sets that are used in benchmarks. Note that the benchmark dataset may not cover all Polaris features. + +## Generation rules + +The dataset has a tree shape. At the root of the tree is a Polaris realm that must exist before the dataset is created. + +An arbitrary number of catalogs can be created under the realm. However, only the first catalog (`C_0`) is used for the rest of the dataset. Review Comment: realm? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
