This is an automated email from the ASF dual-hosted git repository.
gerlowskija pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/solr-sandbox.git
The following commit(s) were added to refs/heads/main by this push:
new 5f13abc Add Gatling-based "macro" benchmark module (#123)
5f13abc is described below
commit 5f13abcab7c4feed5135127ee9ef578414d786cc
Author: Jason Gerlowski <[email protected]>
AuthorDate: Mon Sep 22 20:17:29 2025 -0400
Add Gatling-based "macro" benchmark module (#123)
The main Solr repository offers a micro-benchmarking module at
solr/benchmark,
built on JMH. Though it's not always used this way, JMH shines when
evaluating
smaller chunks of code, or the impact of singular changes. It's not
typically
intended for running larger benchmarks of many HTTP requests, or tracking
longitudinal performance drift over time.
This commit introduces a benchmarking integration to tackle these larger
"macro" questions. It consists of two Gradle modules:
- gatling-data-prep - utilities for downloading and prepping the datasets
needed
for perf tests gatling-simulations - the Gatling performance tests (i.e.
"simulations" in Gatling terminology) themselves. Currently consists of a
single configurable simulation that indexes Wikipedia documents into Solr.
- gatling-simulations - the Gatling performance tests (i.e. "simulations" in
Gatling terminology) themselves. Currently consists of a single configurable
simulation that indexes Wikipedia documents into Solr.
---------
Co-authored-by: David Smiley <[email protected]>
---
.gitignore | 2 +
build.gradle | 2 -
gatling-data-prep/README.md | 60 ++++++++++
gatling-data-prep/build.gradle | 17 +++
gatling-data-prep/src/main/java/Page.java | 25 ++++
.../src/main/java/WikipediaHandler.java | 130 +++++++++++++++++++++
.../src/main/java/WikipediaJsonHandler.java | 32 +++++
.../src/main/java/WikipediaXmlHandler.java | 63 ++++++++++
.../src/main/java/WikipediaXmlToSolr.java | 50 ++++++++
gatling-simulations/README.md | 30 +++++
gatling-simulations/build.gradle | 17 +++
.../index/IndexWikipediaBatchesSimulation.java | 120 +++++++++++++++++++
.../src/gatling/java/util/GatlingUtils.java | 19 +++
.../src/gatling/java/util/SolrUtil.java | 51 ++++++++
.../resources/configSets/wikipedia-8/elevate.xml | 27 +++++
.../resources/configSets/wikipedia-8/schema.xml | 84 +++++++++++++
.../configSets/wikipedia-8/solrconfig.xml | 93 +++++++++++++++
.../resources/configSets/wikipedia-9/elevate.xml | 27 +++++
.../resources/configSets/wikipedia-9/schema.xml | 84 +++++++++++++
.../configSets/wikipedia-9/solrconfig.xml | 93 +++++++++++++++
.../configSets/wikipedia-main/elevate.xml | 27 +++++
.../resources/configSets/wikipedia-main/schema.xml | 84 +++++++++++++
.../configSets/wikipedia-main/solrconfig.xml | 93 +++++++++++++++
.../src/gatling/resources/gatling.conf | 10 ++
gradle/wrapper/gradle-wrapper.properties | 2 +-
scripts/gatling/setup_wikipedia_tests.sh | 84 +++++++++++++
settings.gradle | 4 +
27 files changed, 1327 insertions(+), 3 deletions(-)
diff --git a/.gitignore b/.gitignore
index 1070024..ee0b600 100644
--- a/.gitignore
+++ b/.gitignore
@@ -17,4 +17,6 @@ build
out
cluster
+.gatling
+
logs
diff --git a/build.gradle b/build.gradle
index 9a38422..2392593 100644
--- a/build.gradle
+++ b/build.gradle
@@ -22,10 +22,8 @@
* Learn more about Gradle by exploring our samples at
https://docs.gradle.org/6.7.1/samples
*/
-
description 'Root for Solr plugins sandbox'
-
subprojects {
group "org.apache.solr.crossdc"
group "org.apache.solr.encryption"
diff --git a/gatling-data-prep/README.md b/gatling-data-prep/README.md
new file mode 100644
index 0000000..99c4306
--- /dev/null
+++ b/gatling-data-prep/README.md
@@ -0,0 +1,60 @@
+# Gatling Data Preparation
+
+Performance benchmarks, whether query or index based, rely on input data.
+This module offers utilities to help fetch and prepare that data.
+(Currently the only supported data source is wikipedia, but other sources may
be added over time.)
+
+> [!NOTE]
+> Input data is made available to download on
[nightlies.apache.org](https://nightlies.apache.org/solr/benchmark-data).
+> Most users will find it easier to download from there, and not need to use
this module to prepare their own input data.
+
+# Wikipedia Data
+## Download
+
+Wikipedia offers "raw" dumps from
https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
+
+Since Wikipedia content changes daily, some users may prefer to use an older,
more static dump.
+Interested users can explore the options available at
https://dumps.wikimedia.org/
+
+As of writing, the compressed dump is roughly ~25GB, and expands to ~75GB when
extracted.
+
+```
+bunzip2 enwiki-latest-pages-articles.xml.bz2
+```
+
+## Solr-Documentification
+
+Of course, the Wikipedia dumps must be converted into Solr documents before
they can be ingested.
+
+To generate data suitable for indexing into Solr, we will process the raw
wikipedia XML into solr documents.
+Currently we support outputting documents as either XML or JSON, though other
formats (e.g. javabin) may be added over time.
+
+First, compile the code we use to read the raw wikipedia archives.
+```
+./gradlew :gatling-data-prep:jar
+```
+
+Next, we will use this tool to convert the data:
+
+```
+java -cp gatling-data-prep/build/libs/gatling-data-prep.jar WikipediaXmlToSolr
source-file output-dir data-type batch-size text-size
+```
+
+With the following arguments:
+
+* `source-file` is the path to the previously unzipped Wikipedia dump
+* `output-dir` is the path to a directory intended to contain the batch files
(e.g. ".gatling/batches")
+* `data-type` is one of `xml` or `json`, and determines the output format
+* `batch-size` is an integer, and will control how many documents are saved
per file. These files will later be uploaded to Solr as individual batches.
+* `text-size` is an integer, and will truncate article text longer beyond a
certain length. Lucene benchmarks run with both 1k and 4k truncations.
+
+An example invocation might look like:
+
+```
+mkdir -p .gatling/batches
+java -cp gatling-data-prep/build/libs/gatling-data-prep.jar WikipediaXmlToSolr
~/Downloads/wiki/enwiki-latest-pages-articles.xml .gatling/batches json 5000
1000
+```
+
+This will generate json files with 5000 articles per file, and each article's
text will be truncated to 1kb of data.
+The files will be placed in the `.gatling/batches` directory (a "default"
location used by the wikipedia simulations)
+The wikipedia dump contains approximately 20M pages, so this output directory
should contain a little over 4000 output files.
diff --git a/gatling-data-prep/build.gradle b/gatling-data-prep/build.gradle
new file mode 100644
index 0000000..5e28805
--- /dev/null
+++ b/gatling-data-prep/build.gradle
@@ -0,0 +1,17 @@
+plugins {
+ id 'java'
+}
+
+repositories {
+ mavenCentral()
+}
+
+dependencies {
+ implementation 'com.google.code.gson:gson:2.8.6'
+}
+
+jar {
+ from {
+ configurations.compileClasspath.collect { it.isDirectory() ? it :
zipTree(it) }
+ }
+}
diff --git a/gatling-data-prep/src/main/java/Page.java
b/gatling-data-prep/src/main/java/Page.java
new file mode 100644
index 0000000..bde82ca
--- /dev/null
+++ b/gatling-data-prep/src/main/java/Page.java
@@ -0,0 +1,25 @@
+public class Page {
+ String title;
+ String id;
+ String contributor;
+ String date;
+ String text;
+ String comment;
+
+ @Override
+ public String toString() {
+ return "Page [title=" + title + ", id=" + id + ", contributor=" +
contributor + ", date="
+ + date + ", text=" + text + ", comment=" + comment + "]";
+ }
+
+ public String[][] toFields() {
+ String[][] fields = new String[6][2];
+ fields[0] = new String[]{"id", id};
+ fields[1] = new String[]{"title", title};
+ fields[2] = new String[]{"date", date};
+ fields[3] = new String[]{"contributor", contributor};
+ fields[4] = new String[]{"comment", comment};
+ fields[5] = new String[]{"text", text};
+ return fields;
+ }
+}
diff --git a/gatling-data-prep/src/main/java/WikipediaHandler.java
b/gatling-data-prep/src/main/java/WikipediaHandler.java
new file mode 100644
index 0000000..ffb7a6c
--- /dev/null
+++ b/gatling-data-prep/src/main/java/WikipediaHandler.java
@@ -0,0 +1,130 @@
+import org.xml.sax.Attributes;
+import org.xml.sax.helpers.DefaultHandler;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.Stack;
+
+public abstract class WikipediaHandler extends DefaultHandler {
+ private final static Set<String> interestingElements = Set.of("title", "id",
"username", "comment", "text", "timestamp");
+
+ List<Page> pages;
+ Page currentPage = null;
+ private StringBuilder data = null;
+
+ protected final String outputDirectory;
+ private final int batchSize;
+
+ private final int maxTextSize;
+
+ private int flushCount = 0;
+
+ private Stack<String> location = new Stack<>();
+
+ public int offset = 0;
+
+ public WikipediaHandler(String outputDirectory, int batchSize, int
maxTextSize) {
+ this.outputDirectory = outputDirectory;
+ this.batchSize = batchSize;
+ this.maxTextSize = maxTextSize;
+ pages = new ArrayList<>(batchSize);
+ }
+
+ @Override
+ public void startElement(String uri, String localName, String qName,
Attributes attributes) {
+ if (qName.equalsIgnoreCase("page")) {
+ Page p = new Page();
+ pages.add(p);
+ currentPage = p;
+ }
+ if (interestingElements.contains(qName.toLowerCase())) {
+ data = new StringBuilder();
+ }
+ location.add(qName);
+ }
+
+ public void endElement(String uri, String localName, String qName) {
+ if (!qName.equals(location.pop())) {
+ throw new RuntimeException("Something is wrong in the stack");
+ }
+
+ if (qName.equalsIgnoreCase("page")) {
+ currentPage = null;
+ if (pages.size() == batchSize) {
+ if (Integer.parseInt(pages.get(0).id) < offset) {
+ System.out.println("Skipping " + pages.size() + " docs, starting at
" + pages.get(0).id);
+ } else {
+ flush();
+ }
+ flushCount++;
+ pages.clear();
+ }
+ } else if (qName.equalsIgnoreCase("title")) {
+ if (currentPage != null) {
+ currentPage.title = data.toString();
+ data = null;
+ }
+ } else if (qName.equalsIgnoreCase("id") &&
"page".contentEquals(location.peek())) {
+ if (currentPage != null) {
+ currentPage.id = data.toString();
+ data = null;
+ if (Integer.parseInt(currentPage.id) < offset) {
+ currentPage = null;
+ }
+ }
+ } else if (qName.equalsIgnoreCase("username")) {
+ if (currentPage != null) {
+ currentPage.contributor = data.toString();
+ data = null;
+ }
+ } else if (qName.equalsIgnoreCase("timestamp")) {
+ if (currentPage != null) {
+ currentPage.date = data.toString();
+ data = null;
+ }
+ } else if (qName.equalsIgnoreCase("text")) {
+ if (currentPage != null) {
+ if (data.length() > maxTextSize) {
+ int truncation = maxTextSize;
+
+ // Make sure we're not splitting surrogate characters
+ if (Character.isHighSurrogate(data.charAt(truncation - 1))) {
+ truncation--;
+ }
+
+ data.setLength(truncation);
+ }
+
+ currentPage.text = data.toString();
+ data = null;
+ }
+ } else if (qName.equalsIgnoreCase("comment")) {
+ if (currentPage != null) {
+ currentPage.comment = data.toString();
+ data = null;
+ }
+ }
+
+ }
+
+ @Override
+ public void endDocument() {
+ flush();
+ }
+
+ abstract void flush();
+
+ final int getFlushCount() {
+ return flushCount;
+ }
+
+ @Override
+ public void characters(char[] ch, int start, int length) {
+ if (data != null) {
+ data.append(new String(ch, start, length));
+ }
+ }
+
+}
diff --git a/gatling-data-prep/src/main/java/WikipediaJsonHandler.java
b/gatling-data-prep/src/main/java/WikipediaJsonHandler.java
new file mode 100644
index 0000000..77ad23d
--- /dev/null
+++ b/gatling-data-prep/src/main/java/WikipediaJsonHandler.java
@@ -0,0 +1,32 @@
+import com.google.gson.stream.JsonWriter;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.nio.file.Paths;
+
+public class WikipediaJsonHandler extends WikipediaHandler {
+ public WikipediaJsonHandler(String outputDirectory, int batchSize, int
maxTextSize) {
+ super(outputDirectory, batchSize, maxTextSize);
+ }
+
+ @Override
+ void flush() {
+ try (JsonWriter writer = new JsonWriter(new FileWriter(new
File(outputDirectory, "output" + getFlushCount() + ".json")))) {
+ writer.beginArray();
+ for (Page p : pages) {
+ writer.beginObject();
+ for (String[] pageField : p.toFields()) {
+ writer.name(pageField[0]).value(pageField[1]);
+ }
+ writer.endObject();
+ }
+ writer.endArray();
+ System.out.println("Flushed output" + getFlushCount() + ".json,
containing " + pages.size() + " docs, starting at " + pages.get(0).id);
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to flush on document starting
with " + pages.get(0), e);
+ }
+ }
+}
+
+
diff --git a/gatling-data-prep/src/main/java/WikipediaXmlHandler.java
b/gatling-data-prep/src/main/java/WikipediaXmlHandler.java
new file mode 100644
index 0000000..bf92af0
--- /dev/null
+++ b/gatling-data-prep/src/main/java/WikipediaXmlHandler.java
@@ -0,0 +1,63 @@
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+
+import javax.xml.parsers.DocumentBuilder;
+import javax.xml.parsers.DocumentBuilderFactory;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.transform.OutputKeys;
+import javax.xml.transform.Transformer;
+import javax.xml.transform.TransformerFactory;
+import javax.xml.transform.dom.DOMSource;
+import javax.xml.transform.stream.StreamResult;
+import java.io.File;
+import java.io.FileOutputStream;
+
+public class WikipediaXmlHandler extends WikipediaHandler {
+ public WikipediaXmlHandler(String outputDirectory, int batchSize, int
maxTextSize) {
+ super(outputDirectory, batchSize, maxTextSize);
+ }
+
+ @Override
+ void flush() {
+ Document doc;
+ DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
+ try {
+ // use factory to get an instance of document builder
+ DocumentBuilder db = dbf.newDocumentBuilder();
+ // create instance of DOM
+ doc = db.newDocument();
+
+ // create the root element
+ Element rootElement = doc.createElement("add");
+ doc.appendChild(rootElement);
+
+ for (Page p:pages) {
+ Element pageDoc = doc.createElement("doc");
+ rootElement.appendChild(pageDoc);
+ String[][] fields = p.toFields();
+ for (int i = 0; i < fields.length; i++) {
+ Element fieldElement = doc.createElement("field");
+ pageDoc.appendChild(fieldElement);
+ fieldElement.setAttribute("name", fields[i][0]);
+ fieldElement.setTextContent(fields[i][1]);
+ }
+ }
+
+ try {
+ Transformer tr =
TransformerFactory.newInstance().newTransformer();
+ tr.setOutputProperty(OutputKeys.INDENT, "yes");
+ tr.setOutputProperty(OutputKeys.METHOD, "xml");
+ tr.setOutputProperty(OutputKeys.ENCODING, "UTF-8");
+
tr.setOutputProperty("{http://xml.apache.org/xslt}indent-amount";, "4");
+ tr.transform(new DOMSource(doc),
+ new StreamResult(new FileOutputStream(new
File(outputDirectory, "output" + getFlushCount() + ".xml"))));
+ System.out.println("Flushed output" + getFlushCount() + ".xml,
containing " + pages.size() + " docs, starting at " + pages.get(0).id);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to flush on document
starting with " + pages.get(0), e);
+ }
+ } catch (ParserConfigurationException e) {
+ e.printStackTrace();
+ }
+ }
+
+}
diff --git a/gatling-data-prep/src/main/java/WikipediaXmlToSolr.java
b/gatling-data-prep/src/main/java/WikipediaXmlToSolr.java
new file mode 100644
index 0000000..b107e5d
--- /dev/null
+++ b/gatling-data-prep/src/main/java/WikipediaXmlToSolr.java
@@ -0,0 +1,50 @@
+import javax.xml.XMLConstants;
+import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.parsers.SAXParser;
+import javax.xml.parsers.SAXParserFactory;
+
+import org.xml.sax.SAXException;
+
+import java.io.File;
+import java.io.IOException;
+
+public class WikipediaXmlToSolr {
+
+ public static void main(String[] args) throws IOException {
+ if (args.length < 5 || args.length > 6) {
+ System.out.println("Expecting five positional arguments: source-file,
output-dir, output-data-type, batchSize, maxTextSize, [docOffset]");
+ System.exit(1);
+ }
+
+ final var sourceFilePath = args[0];
+ final var outputDirPath = args[1];
+ final var outputDataType = args[2];
+ final var batchSize = Integer.parseInt(args[3]);
+ final var maxTextSize = Integer.parseInt(args[4]);
+
+ WikipediaHandler handler = null;
+ if ("json".equals(outputDataType)) {
+ handler = new WikipediaJsonHandler(outputDirPath, batchSize,
maxTextSize);
+ } else if ("xml".equals(outputDataType)) {
+ handler = new WikipediaXmlHandler(outputDirPath, batchSize, maxTextSize);
+ } else {
+ System.out.println("data-type should be 'xml' or 'json'");
+ System.exit(1);
+ }
+ if (args.length > 5) {
+ handler.offset = Integer.parseInt(args[5]);
+ }
+ try {
+ File f = new File(sourceFilePath);
+ SAXParserFactory factory = SAXParserFactory.newInstance();
+ factory.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, false);
+ SAXParser parser = factory.newSAXParser();
+ parser.setProperty(XMLConstants.ACCESS_EXTERNAL_DTD, "false");
+ parser.setProperty(XMLConstants.ACCESS_EXTERNAL_SCHEMA, "false");
+ parser.parse(f, handler);
+ } catch (ParserConfigurationException | SAXException e) {
+ System.err.println(e.getMessage());
+ e.printStackTrace();
+ }
+ }
+}
diff --git a/gatling-simulations/README.md b/gatling-simulations/README.md
new file mode 100644
index 0000000..f4a8033
--- /dev/null
+++ b/gatling-simulations/README.md
@@ -0,0 +1,30 @@
+# Gatling Simulations
+
+This module supports running performance benchmarks packaged and run as
"Gatling simulations".
+
+## Supported Simulations
+
+Currently only a single simulation is supported,
"IndexWikipediaBatchesSimulation", which provides a benchmark for indexing
truncated wikipedia pages.
+(This benchmark relies on having processed data available to index.
+See [the data-prep README](../gatling-data-prep/README.md) for details on
creating the necessary data.)
+This benchmark can be can be configured using the environment-variable knobs
below:
+
+- `TESTS_WORK_DIR` - used to locate Wikipedia data (defaults to:
`$REPOSITORY_ROOT/.gatling`)
+- `BATCH_CONTENT_TYPE` - the format of the preprocessed Wikipedia data.
Options are `application/json` or `application/xml` (defaults to:
`application/json`)
+- `CONCURRENT_USERS` - the number of threads used to index data to Solr
(defaults to: 10)
+- `endpoint` - a Solr URL to send documents to (defaults to:
`"http://localhost:8983/solr"`)
+- `COLLECTION_NAME` - the collection name to index data into, created by the
simulation (defaults to: "wikipedia")
+- `NUM_SHARDS` - the number of shards for the created collection (defaults to:
1)
+- `NUM_REPLICAS` - the number of replicas for each shard of the created
collection (defaults to: 1)
+
+## Running Built-In Scenarios
+
+Built-in indexing scenarios will create a collection and delete it after the
test completes.
+
+Indexing benchmarks may be run using the command below from the repository root
+
+```
+ NUM_SHARDS=2 ./gradlew gatlingRun --simulation
index.IndexWikipediaBatchesSimulation
+```
+
+Gatling will print basic statistics on `stdout`, but a more comprehensive (and
human-friendly) HTML report is also available in
`gatling-simulations/build/reports`
diff --git a/gatling-simulations/build.gradle b/gatling-simulations/build.gradle
new file mode 100644
index 0000000..9cf2758
--- /dev/null
+++ b/gatling-simulations/build.gradle
@@ -0,0 +1,17 @@
+plugins {
+ id 'java'
+ id "io.gatling.gradle" version "3.14.3.5"
+}
+
+repositories {
+ mavenCentral()
+}
+
+gatling {
+ systemProperties = [
+ 'endpoint': System.getProperty('endpoint', 'http://localhost:8983'),
+ 'TESTS_WORK_DIR': project.rootDir.absolutePath + '/.gatling'
+ ]
+}
+
+tasks.matching { it.name.contains("gatlingRun") }.all {
it.outputs.upToDateWhen { false } }
diff --git
a/gatling-simulations/src/gatling/java/index/IndexWikipediaBatchesSimulation.java
b/gatling-simulations/src/gatling/java/index/IndexWikipediaBatchesSimulation.java
new file mode 100644
index 0000000..f5164e5
--- /dev/null
+++
b/gatling-simulations/src/gatling/java/index/IndexWikipediaBatchesSimulation.java
@@ -0,0 +1,120 @@
+package index;
+
+import io.gatling.javaapi.core.*;
+import io.gatling.javaapi.http.*;
+import util.GatlingUtils;
+import util.SolrUtil;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.file.DirectoryStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.stream.StreamSupport;
+
+import static io.gatling.javaapi.core.CoreDsl.*;
+import static io.gatling.javaapi.http.HttpDsl.*;
+
+public class IndexWikipediaBatchesSimulation extends Simulation {
+
+ private final String testWorkDir;
+ private final Path batchesDir;
+ private final int numFiles;
+ private final int atOnceUsersCount;
+ private final HttpProtocolBuilder httpProtocol;
+ private final ChainBuilder updates;
+ private final ScenarioBuilder scn;
+
+ public IndexWikipediaBatchesSimulation() {
+ atOnceUsersCount = getConfigInt("CONCURRENT_USERS", 10);
+
+ testWorkDir = getConfig("TESTS_WORK_DIR", ".gatling");
+ batchesDir = Paths.get(testWorkDir, "batches");
+ numFiles = batchesDir.toFile().list().length;
+ httpProtocol = http.baseUrl(GatlingUtils.getEndpoint());
+ updates = index(testWorkDir);
+ scn = scenario(this.getClass().getSimpleName())
+ .repeat(getIterations(numFiles, atOnceUsersCount))
+ .on(exec(updates));
+
this.setUp(scn.injectOpen(atOnceUsers(atOnceUsersCount))).protocols(httpProtocol);
+ }
+
+ public int getIterations(int numBatchesAvailable, int numAtOnceUsers) {
+ return numBatchesAvailable / numAtOnceUsers;
+ }
+
+ public static String getConfig(String key, String defaultValue) {
+ return System.getenv().getOrDefault(key, System.getProperty(key,
defaultValue));
+ }
+
+ public static int getConfigInt(String key, int defaultValue) {
+ return Integer.parseInt(getConfig(key, String.valueOf(defaultValue)));
+ }
+
+ @Override
+ public void before() {
+ setupCollection();
+ }
+
+ @Override
+ public void after() {
+ tearDownCollection();
+ }
+
+
+ public void setupCollection() {
+ String endpoint = GatlingUtils.getEndpoint();
+ String collectionName = getConfig("COLLECTION_NAME", "wikipedia");
+ int numShards = getConfigInt("NUM_SHARDS", 1);
+ int numReplicas = getConfigInt("NUM_REPLICAS", 1);
+
+ try {
+ SolrUtil.deleteCollection(endpoint + "/solr", collectionName,
false);
+ System.out.println("Creating collection " + collectionName);
+ SolrUtil.createCollection(endpoint + "/solr", collectionName,
numShards, numReplicas, "wikipedia");
+ } catch (Exception e) {
+ System.out.printf("Wikipedia collection %s could not be created.
%s: %s%n", collectionName, e.getClass(), e.getMessage());
+ }
+
+ }
+
+ public void tearDownCollection() {
+ String endpoint = GatlingUtils.getEndpoint();
+ String collectionName = getConfig("COLLECTION_NAME", "wikipedia");
+ System.out.println("Deleting collection " + collectionName);
+ try {
+ SolrUtil.deleteCollection(endpoint + "/solr", collectionName,
true);
+ } catch (Exception e) {
+ System.out.printf("Wikipedia Collection %s could not be deleted.
%s: %s%n", collectionName, e.getClass(), e.getMessage());
+ }
+ }
+
+ public ChainBuilder index(String workDir) {
+ String collectionName = getConfig("COLLECTION_NAME", "wikipedia");
+ String contentType = getConfig("BATCH_CONTENT_TYPE",
"application/json");
+ Map<String, String> headers = Collections.singletonMap("Content-Type",
contentType);
+ Path batchesDir = Paths.get(workDir, "batches");
+
+ Iterator<Map<String, Object>> documents =
StreamSupport.stream(newDirectoryStream(batchesDir).spliterator(), false)
+ .map(path -> Collections.singletonMap("file", (Object) path))
+ .iterator();
+
+ HttpRequestActionBuilder updateReq = http("updates")
+ .post("/solr/" + collectionName + "/update")
+ .headers(headers);
+
+ return feed(documents).exec(updateReq.body(RawFileBody("#{file}")));
+ }
+
+ private static DirectoryStream<Path> newDirectoryStream(Path dir) {
+ try {
+ return Files.newDirectoryStream(dir);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+}
\ No newline at end of file
diff --git a/gatling-simulations/src/gatling/java/util/GatlingUtils.java
b/gatling-simulations/src/gatling/java/util/GatlingUtils.java
new file mode 100644
index 0000000..9659cf5
--- /dev/null
+++ b/gatling-simulations/src/gatling/java/util/GatlingUtils.java
@@ -0,0 +1,19 @@
+package util;
+
+public class GatlingUtils {
+
+ public static String getEndpoint() {
+ String endpoint = System.getProperty("endpoint");
+ //val endpoint = "http://localhost:8983";;
+ if (endpoint == null) {
+ throw new IllegalArgumentException("Missing endpoint. Please
provide \"endpoint\" system property");
+ }
+ if (endpoint.endsWith("/solr")) {
+ throw new IllegalArgumentException("Endpoint should not include
Solr path");
+ }
+ if (endpoint.endsWith("/")) {
+ throw new IllegalArgumentException("Endpoint should not end with
trailing /");
+ }
+ return endpoint;
+ }
+}
\ No newline at end of file
diff --git a/gatling-simulations/src/gatling/java/util/SolrUtil.java
b/gatling-simulations/src/gatling/java/util/SolrUtil.java
new file mode 100644
index 0000000..a2ec1ae
--- /dev/null
+++ b/gatling-simulations/src/gatling/java/util/SolrUtil.java
@@ -0,0 +1,51 @@
+package util;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.HttpURLConnection;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+
+public class SolrUtil {
+ private static final Logger logger =
LoggerFactory.getLogger(SolrUtil.class);
+
+ public static void createCollection(String endpoint, String
collectionName, int numShards, int numReplicas, String configName) throws
IOException {
+ URL url = new URL(endpoint +
"/admin/collections?action=create&maxShardsPerNode=-1&name="+ collectionName
+ + "&numShards=" + numShards
+ + "&nrtReplicas=" + numReplicas
+ + "&waitForFinalState=true"
+ + "&collection.configName=" + configName);
+ sendGet(url, "create", true);
+ }
+
+ public static void deleteCollection(String endpoint, String
collectionName) throws IOException {
+ deleteCollection(endpoint, collectionName, true);
+ }
+
+ public static void deleteCollection(String endpoint, String
collectionName, boolean logError) throws IOException {
+ URL url = new URL(endpoint + "/admin/collections?action=delete&name="
+ collectionName);
+ sendGet(url, "delete", logError);
+ }
+
+ private static void sendGet(URL url, String action, boolean logError)
throws IOException {
+ HttpURLConnection con = (HttpURLConnection) url.openConnection();
+ con.setRequestMethod("GET"); // This is default method, but be
explicit anyway
+ con.setConnectTimeout(10 * 1000);
+ con.setReadTimeout(600 * 1000);
+ int responseCode = con.getResponseCode();
+ if (logError && responseCode != HttpURLConnection.HTTP_OK) {
+ try (InputStream err = con.getErrorStream()) {
+ String response = "";
+ if (err != null) { // code >= 400
+ response = new String(err.readAllBytes(),
StandardCharsets.UTF_8);
+ }
+ logger.error("Unable to {} collection; {}", action, response);
+ }
+ throw new RuntimeException("Unable to " + action + " collection:
Response Code:" + responseCode);
+ }
+ con.disconnect();
+ }
+}
diff --git
a/gatling-simulations/src/gatling/resources/configSets/wikipedia-8/elevate.xml
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-8/elevate.xml
new file mode 100644
index 0000000..f33f42a
--- /dev/null
+++
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-8/elevate.xml
@@ -0,0 +1,27 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!-- If this file is found in the config directory, it will only be
+ loaded once at startup. If it is found in Solr's data
+ directory, it will be re-loaded every commit.
+
+ See http://wiki.apache.org/solr/QueryElevationComponent for more info
+
+-->
+<elevate>
+</elevate>
diff --git
a/gatling-simulations/src/gatling/resources/configSets/wikipedia-8/schema.xml
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-8/schema.xml
new file mode 100644
index 0000000..99abcba
--- /dev/null
+++
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-8/schema.xml
@@ -0,0 +1,84 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<schema name="example" version="1.5">
+ <field name="id" required="true" type="string" indexed="true"
stored="true" docValues="true" multiValued="false" />
+ <field name="_version_" type="plong" indexed="true"
stored="true" docValues="true" multiValued="false" />
+
+ <field name="title" type="text_general" indexed="true" stored="true"/>
+ <field name="date" type="pdate" indexed="true" stored="true"
docValues="true"/>
+ <field name="contributor" type="string" indexed="true" stored="true"
docValues="true"/>
+ <field name="comment" type="text_general" indexed="true" stored="true"
multiValued="false"/>
+ <field name="text" type="text_general" indexed="true" stored="true"
multiValued="false"/>
+
+ <!-- Field to use to determine and enforce document uniqueness.
+ Unless this field is marked with required="false", it will be a
required field
+ -->
+ <uniqueKey>id</uniqueKey>
+ <!-- The StrField type is not analyzed, but indexed/stored verbatim. -->
+ <fieldType name="string" class="solr.StrField" sortMissingLast="true"
docValues="true" />
+ <fieldType name="strings" class="solr.StrField" sortMissingLast="true"
multiValued="true" docValues="true" />
+
+ <!-- boolean type: "true" or "false" -->
+ <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true"/>
+ <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true"
multiValued="true"/>
+
+ <!--
+ Numeric field types that index values using KD-trees.
+ Point fields don't support FieldCache, so they must have
docValues="true" if needed for sorting, faceting, functions, etc.
+ -->
+ <fieldType name="pint" class="solr.IntPointField" docValues="true"/>
+ <fieldType name="pfloat" class="solr.FloatPointField" docValues="true"/>
+ <fieldType name="plong" class="solr.LongPointField" docValues="true"/>
+ <fieldType name="pdouble" class="solr.DoublePointField"
docValues="true"/>
+
+ <fieldType name="pints" class="solr.IntPointField" docValues="true"
multiValued="true"/>
+ <fieldType name="pfloats" class="solr.FloatPointField" docValues="true"
multiValued="true"/>
+ <fieldType name="plongs" class="solr.LongPointField" docValues="true"
multiValued="true"/>
+ <fieldType name="pdoubles" class="solr.DoublePointField"
docValues="true" multiValued="true"/>
+ <fieldType name="random" class="solr.RandomSortField" indexed="true"/>
+
+ <!-- since fields of this type are by default not stored or indexed,
+ any data added to them will be ignored outright. -->
+ <fieldType name="ignored" stored="false" indexed="false"
multiValued="true" class="solr.StrField" />
+
+ <!-- KD-tree versions of date fields -->
+ <fieldType name="pdate" class="solr.DatePointField" docValues="true"/>
+ <fieldType name="pdates" class="solr.DatePointField" docValues="true"
multiValued="true"/>
+
+ <!--Binary data type. The data should be sent/retrieved in as Base64
encoded Strings -->
+ <fieldType name="binary" class="solr.BinaryField"/>
+
+ <!-- A general text field that has reasonable, generic
+ cross-language defaults: it tokenizes with StandardTokenizer,
+ removes stop words from case-insensitive "stopwords.txt"
+ (empty by default), and down cases. At query time only, it
+ also applies synonyms.
+ -->
+ <fieldType name="text_general" class="solr.TextField"
positionIncrementGap="100" multiValued="true">
+ <analyzer type="index">
+ <tokenizer class="solr.StandardTokenizerFactory"/>
+ <filter class="solr.LowerCaseFilterFactory"/>
+ </analyzer>
+ <analyzer type="query">
+ <tokenizer class="solr.StandardTokenizerFactory"/>
+ <filter class="solr.LowerCaseFilterFactory"/>
+ </analyzer>
+ </fieldType>
+
+</schema>
diff --git
a/gatling-simulations/src/gatling/resources/configSets/wikipedia-8/solrconfig.xml
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-8/solrconfig.xml
new file mode 100644
index 0000000..f4959a9
--- /dev/null
+++
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-8/solrconfig.xml
@@ -0,0 +1,93 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!--
+ For more details about configurations options that may appear in
+ this file, see http://wiki.apache.org/solr/SolrConfigXml.
+-->
+<config>
+ <luceneMatchVersion>8.6.0</luceneMatchVersion>
+ <dataDir>${solr.data.dir:}</dataDir>
+ <directoryFactory name="DirectoryFactory"
+
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
+
+ <codecFactory class="solr.SchemaCodecFactory"/>
+
+ <indexConfig>
+ <lockType>${solr.lock.type:native}</lockType>
+ </indexConfig>
+ <jmx />
+ <updateHandler class="solr.DirectUpdateHandler2">
+
+ <updateLog>
+ <str name="dir">${solr.ulog.dir:}</str>
+ <int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
+ </updateLog>
+
+ <autoCommit>
+ <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
+ <openSearcher>false</openSearcher>
+ </autoCommit>
+
+ <autoSoftCommit>
+ <maxTime>${solr.autoSoftCommit.maxTime:30000}</maxTime>
+ </autoSoftCommit>
+ </updateHandler>
+ <query>
+ <maxBooleanClauses>${solr.max.booleanClauses:1024}</maxBooleanClauses>
+
+ <filterCache size="0"
+ initialSize="0"
+ autowarmCount="0"/>
+
+ <queryResultCache size="0"
+ initialSize="0"
+ autowarmCount="0"/>
+
+ <documentCache size="0"
+ initialSize="0"
+ autowarmCount="0"/>
+
+ <enableLazyFieldLoading>true</enableLazyFieldLoading>
+
+ <queryResultWindowSize>20</queryResultWindowSize>
+
+ <queryResultMaxDocsCached>200</queryResultMaxDocsCached>
+
+ <useColdSearcher>false</useColdSearcher>
+
+ </query>
+
+ <requestDispatcher>
+ <httpCaching never304="true" />
+ </requestDispatcher>
+ <requestHandler name="/select" class="solr.SearchHandler">
+ <lst name="defaults">
+ <str name="echoParams">explicit</str>
+ <int name="rows">10</int>
+ <str name="df">text</str>
+ </lst>
+ </requestHandler>
+
+ <initParams path="/update/**,/query,/select,/spell">
+ <lst name="defaults">
+ <str name="df">text</str>
+ </lst>
+ </initParams>
+
+</config>
diff --git
a/gatling-simulations/src/gatling/resources/configSets/wikipedia-9/elevate.xml
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-9/elevate.xml
new file mode 100644
index 0000000..f33f42a
--- /dev/null
+++
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-9/elevate.xml
@@ -0,0 +1,27 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!-- If this file is found in the config directory, it will only be
+ loaded once at startup. If it is found in Solr's data
+ directory, it will be re-loaded every commit.
+
+ See http://wiki.apache.org/solr/QueryElevationComponent for more info
+
+-->
+<elevate>
+</elevate>
diff --git
a/gatling-simulations/src/gatling/resources/configSets/wikipedia-9/schema.xml
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-9/schema.xml
new file mode 100644
index 0000000..d587b33
--- /dev/null
+++
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-9/schema.xml
@@ -0,0 +1,84 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<schema name="example" version="1.5">
+ <field name="id" required="true" type="string" indexed="true"
stored="true" docValues="true" multiValued="false" />
+ <field name="_version_" type="plong" indexed="true"
stored="true" docValues="true" multiValued="false" />
+
+ <field name="title" type="text_general" indexed="true" stored="true"/>
+ <field name="date" type="pdate" indexed="true" stored="true"
docValues="true"/>
+ <field name="contributor" type="string" indexed="true" stored="true"
docValues="true"/>
+ <field name="comment" type="text_general" indexed="true" stored="true"
multiValued="false"/>
+ <field name="text" type="text_general" indexed="true" stored="true"
multiValued="false"/>
+
+ <!-- Field to use to determine and enforce document uniqueness.
+ Unless this field is marked with required="false", it will be a
required field
+ -->
+ <uniqueKey>id</uniqueKey>
+ <!-- The StrField type is not analyzed, but indexed/stored verbatim. -->
+ <fieldType name="string" class="solr.StrField" sortMissingLast="true"
docValues="true" />
+ <fieldType name="strings" class="solr.StrField" sortMissingLast="true"
multiValued="true" docValues="true" />
+
+ <!-- boolean type: "true" or "false" -->
+ <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true"/>
+ <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true"
multiValued="true"/>
+
+ <!--
+ Numeric field types that index values using KD-trees.
+ Point fields don't support FieldCache, so they must have
docValues="true" if needed for sorting, faceting, functions, etc.
+ -->
+ <fieldType name="pint" class="solr.IntPointField" docValues="true"/>
+ <fieldType name="pfloat" class="solr.FloatPointField" docValues="true"/>
+ <fieldType name="plong" class="solr.LongPointField" docValues="true"/>
+ <fieldType name="pdouble" class="solr.DoublePointField"
docValues="true"/>
+
+ <fieldType name="pints" class="solr.IntPointField" docValues="true"
multiValued="true"/>
+ <fieldType name="pfloats" class="solr.FloatPointField" docValues="true"
multiValued="true"/>
+ <fieldType name="plongs" class="solr.LongPointField" docValues="true"
multiValued="true"/>
+ <fieldType name="pdoubles" class="solr.DoublePointField"
docValues="true" multiValued="true"/>
+ <fieldType name="random" class="solr.RandomSortField" indexed="true"/>
+
+ <!-- since fields of this type are by default not stored or indexed,
+ any data added to them will be ignored outright. -->
+ <fieldType name="ignored" stored="false" indexed="false"
multiValued="true" class="solr.StrField" />
+
+ <!-- KD-tree versions of date fields -->
+ <fieldType name="pdate" class="solr.DatePointField" docValues="true"/>
+ <fieldType name="pdates" class="solr.DatePointField" docValues="true"
multiValued="true"/>
+
+ <!--Binary data type. The data should be sent/retrieved in as Base64
encoded Strings -->
+ <fieldType name="binary" class="solr.BinaryField"/>
+
+ <!-- A general text field that has reasonable, generic
+ cross-language defaults: it tokenizes with StandardTokenizer,
+ removes stop words from case-insensitive "stopwords.txt"
+ (empty by default), and down cases. At query time only, it
+ also applies synonyms.
+ -->
+ <fieldType name="text_general" class="solr.TextField"
positionIncrementGap="100" multiValued="true">
+ <analyzer type="index">
+ <tokenizer name="standard"/>
+ <filter name="lowercase"/>
+ </analyzer>
+ <analyzer type="query">
+ <tokenizer name="standard"/>
+ <filter name="lowercase"/>
+ </analyzer>
+ </fieldType>
+
+</schema>
diff --git
a/gatling-simulations/src/gatling/resources/configSets/wikipedia-9/solrconfig.xml
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-9/solrconfig.xml
new file mode 100644
index 0000000..c3782a0
--- /dev/null
+++
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-9/solrconfig.xml
@@ -0,0 +1,93 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!--
+ For more details about configurations options that may appear in
+ this file, see http://wiki.apache.org/solr/SolrConfigXml.
+-->
+<config>
+ <luceneMatchVersion>LATEST</luceneMatchVersion>
+ <dataDir>${solr.data.dir:}</dataDir>
+ <directoryFactory name="DirectoryFactory"
+
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
+
+ <codecFactory class="solr.SchemaCodecFactory"/>
+
+ <indexConfig>
+ <lockType>${solr.lock.type:native}</lockType>
+ </indexConfig>
+ <jmx />
+ <updateHandler class="solr.DirectUpdateHandler2">
+
+ <updateLog>
+ <str name="dir">${solr.ulog.dir:}</str>
+ <int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
+ </updateLog>
+
+ <autoCommit>
+ <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
+ <openSearcher>false</openSearcher>
+ </autoCommit>
+
+ <autoSoftCommit>
+ <maxTime>${solr.autoSoftCommit.maxTime:30000}</maxTime>
+ </autoSoftCommit>
+ </updateHandler>
+ <query>
+ <maxBooleanClauses>${solr.max.booleanClauses:1024}</maxBooleanClauses>
+
+ <filterCache size="0"
+ initialSize="0"
+ autowarmCount="0"/>
+
+ <queryResultCache size="0"
+ initialSize="0"
+ autowarmCount="0"/>
+
+ <documentCache size="0"
+ initialSize="0"
+ autowarmCount="0"/>
+
+ <enableLazyFieldLoading>true</enableLazyFieldLoading>
+
+ <queryResultWindowSize>20</queryResultWindowSize>
+
+ <queryResultMaxDocsCached>200</queryResultMaxDocsCached>
+
+ <useColdSearcher>false</useColdSearcher>
+
+ </query>
+
+ <requestDispatcher>
+ <httpCaching never304="true" />
+ </requestDispatcher>
+ <requestHandler name="/select" class="solr.SearchHandler">
+ <lst name="defaults">
+ <str name="echoParams">explicit</str>
+ <int name="rows">10</int>
+ <str name="df">text</str>
+ </lst>
+ </requestHandler>
+
+ <initParams path="/update/**,/query,/select,/spell">
+ <lst name="defaults">
+ <str name="df">text</str>
+ </lst>
+ </initParams>
+
+</config>
diff --git
a/gatling-simulations/src/gatling/resources/configSets/wikipedia-main/elevate.xml
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-main/elevate.xml
new file mode 100644
index 0000000..f33f42a
--- /dev/null
+++
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-main/elevate.xml
@@ -0,0 +1,27 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!-- If this file is found in the config directory, it will only be
+ loaded once at startup. If it is found in Solr's data
+ directory, it will be re-loaded every commit.
+
+ See http://wiki.apache.org/solr/QueryElevationComponent for more info
+
+-->
+<elevate>
+</elevate>
diff --git
a/gatling-simulations/src/gatling/resources/configSets/wikipedia-main/schema.xml
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-main/schema.xml
new file mode 100644
index 0000000..d587b33
--- /dev/null
+++
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-main/schema.xml
@@ -0,0 +1,84 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<schema name="example" version="1.5">
+ <field name="id" required="true" type="string" indexed="true"
stored="true" docValues="true" multiValued="false" />
+ <field name="_version_" type="plong" indexed="true"
stored="true" docValues="true" multiValued="false" />
+
+ <field name="title" type="text_general" indexed="true" stored="true"/>
+ <field name="date" type="pdate" indexed="true" stored="true"
docValues="true"/>
+ <field name="contributor" type="string" indexed="true" stored="true"
docValues="true"/>
+ <field name="comment" type="text_general" indexed="true" stored="true"
multiValued="false"/>
+ <field name="text" type="text_general" indexed="true" stored="true"
multiValued="false"/>
+
+ <!-- Field to use to determine and enforce document uniqueness.
+ Unless this field is marked with required="false", it will be a
required field
+ -->
+ <uniqueKey>id</uniqueKey>
+ <!-- The StrField type is not analyzed, but indexed/stored verbatim. -->
+ <fieldType name="string" class="solr.StrField" sortMissingLast="true"
docValues="true" />
+ <fieldType name="strings" class="solr.StrField" sortMissingLast="true"
multiValued="true" docValues="true" />
+
+ <!-- boolean type: "true" or "false" -->
+ <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true"/>
+ <fieldType name="booleans" class="solr.BoolField" sortMissingLast="true"
multiValued="true"/>
+
+ <!--
+ Numeric field types that index values using KD-trees.
+ Point fields don't support FieldCache, so they must have
docValues="true" if needed for sorting, faceting, functions, etc.
+ -->
+ <fieldType name="pint" class="solr.IntPointField" docValues="true"/>
+ <fieldType name="pfloat" class="solr.FloatPointField" docValues="true"/>
+ <fieldType name="plong" class="solr.LongPointField" docValues="true"/>
+ <fieldType name="pdouble" class="solr.DoublePointField"
docValues="true"/>
+
+ <fieldType name="pints" class="solr.IntPointField" docValues="true"
multiValued="true"/>
+ <fieldType name="pfloats" class="solr.FloatPointField" docValues="true"
multiValued="true"/>
+ <fieldType name="plongs" class="solr.LongPointField" docValues="true"
multiValued="true"/>
+ <fieldType name="pdoubles" class="solr.DoublePointField"
docValues="true" multiValued="true"/>
+ <fieldType name="random" class="solr.RandomSortField" indexed="true"/>
+
+ <!-- since fields of this type are by default not stored or indexed,
+ any data added to them will be ignored outright. -->
+ <fieldType name="ignored" stored="false" indexed="false"
multiValued="true" class="solr.StrField" />
+
+ <!-- KD-tree versions of date fields -->
+ <fieldType name="pdate" class="solr.DatePointField" docValues="true"/>
+ <fieldType name="pdates" class="solr.DatePointField" docValues="true"
multiValued="true"/>
+
+ <!--Binary data type. The data should be sent/retrieved in as Base64
encoded Strings -->
+ <fieldType name="binary" class="solr.BinaryField"/>
+
+ <!-- A general text field that has reasonable, generic
+ cross-language defaults: it tokenizes with StandardTokenizer,
+ removes stop words from case-insensitive "stopwords.txt"
+ (empty by default), and down cases. At query time only, it
+ also applies synonyms.
+ -->
+ <fieldType name="text_general" class="solr.TextField"
positionIncrementGap="100" multiValued="true">
+ <analyzer type="index">
+ <tokenizer name="standard"/>
+ <filter name="lowercase"/>
+ </analyzer>
+ <analyzer type="query">
+ <tokenizer name="standard"/>
+ <filter name="lowercase"/>
+ </analyzer>
+ </fieldType>
+
+</schema>
diff --git
a/gatling-simulations/src/gatling/resources/configSets/wikipedia-main/solrconfig.xml
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-main/solrconfig.xml
new file mode 100644
index 0000000..c3782a0
--- /dev/null
+++
b/gatling-simulations/src/gatling/resources/configSets/wikipedia-main/solrconfig.xml
@@ -0,0 +1,93 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!--
+ For more details about configurations options that may appear in
+ this file, see http://wiki.apache.org/solr/SolrConfigXml.
+-->
+<config>
+ <luceneMatchVersion>LATEST</luceneMatchVersion>
+ <dataDir>${solr.data.dir:}</dataDir>
+ <directoryFactory name="DirectoryFactory"
+
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
+
+ <codecFactory class="solr.SchemaCodecFactory"/>
+
+ <indexConfig>
+ <lockType>${solr.lock.type:native}</lockType>
+ </indexConfig>
+ <jmx />
+ <updateHandler class="solr.DirectUpdateHandler2">
+
+ <updateLog>
+ <str name="dir">${solr.ulog.dir:}</str>
+ <int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
+ </updateLog>
+
+ <autoCommit>
+ <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
+ <openSearcher>false</openSearcher>
+ </autoCommit>
+
+ <autoSoftCommit>
+ <maxTime>${solr.autoSoftCommit.maxTime:30000}</maxTime>
+ </autoSoftCommit>
+ </updateHandler>
+ <query>
+ <maxBooleanClauses>${solr.max.booleanClauses:1024}</maxBooleanClauses>
+
+ <filterCache size="0"
+ initialSize="0"
+ autowarmCount="0"/>
+
+ <queryResultCache size="0"
+ initialSize="0"
+ autowarmCount="0"/>
+
+ <documentCache size="0"
+ initialSize="0"
+ autowarmCount="0"/>
+
+ <enableLazyFieldLoading>true</enableLazyFieldLoading>
+
+ <queryResultWindowSize>20</queryResultWindowSize>
+
+ <queryResultMaxDocsCached>200</queryResultMaxDocsCached>
+
+ <useColdSearcher>false</useColdSearcher>
+
+ </query>
+
+ <requestDispatcher>
+ <httpCaching never304="true" />
+ </requestDispatcher>
+ <requestHandler name="/select" class="solr.SearchHandler">
+ <lst name="defaults">
+ <str name="echoParams">explicit</str>
+ <int name="rows">10</int>
+ <str name="df">text</str>
+ </lst>
+ </requestHandler>
+
+ <initParams path="/update/**,/query,/select,/spell">
+ <lst name="defaults">
+ <str name="df">text</str>
+ </lst>
+ </initParams>
+
+</config>
diff --git a/gatling-simulations/src/gatling/resources/gatling.conf
b/gatling-simulations/src/gatling/resources/gatling.conf
new file mode 100644
index 0000000..b25cda6
--- /dev/null
+++ b/gatling-simulations/src/gatling/resources/gatling.conf
@@ -0,0 +1,10 @@
+gatling {
+ charting {
+ indicators {
+ percentile1 = 25
+ percentile2 = 50
+ percentile3 = 90
+ percentile4 = 95
+ }
+ }
+}
diff --git a/gradle/wrapper/gradle-wrapper.properties
b/gradle/wrapper/gradle-wrapper.properties
index f398c33..707e499 100644
--- a/gradle/wrapper/gradle-wrapper.properties
+++ b/gradle/wrapper/gradle-wrapper.properties
@@ -1,6 +1,6 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-7.6-bin.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-8.10-bin.zip
networkTimeout=10000
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
diff --git a/scripts/gatling/setup_wikipedia_tests.sh
b/scripts/gatling/setup_wikipedia_tests.sh
new file mode 100755
index 0000000..566eae1
--- /dev/null
+++ b/scripts/gatling/setup_wikipedia_tests.sh
@@ -0,0 +1,84 @@
+#!/usr/bin/env bash
+
+function print_help {
+ echo "Prepares a Solr environment to run Gatling simulations based on
wikipedia data"
+ echo "Options:"
+ echo " h: Print this help"
+ echo " s: The Solr endpoint to use. 'localhost:8983' is the default"
+ echo " m: The major version of Solr in use. i.e. 'main' or '9' or '8'..."
+ echo "Example:"
+ echo "scripts/gatling/setup_wikipedia_tests.sh -s http://localhost:8983";
+}
+
+SOLR_ENDPOINT="localhost:8983"
+# This should be either "main" or a released major version ("9", "8", etc.)
+SOLR_MAJOR_VERSION=${SOLR_MAJOR_VERSION:-main}
+
+TESTS_WORK_DIR=${TESTS_WORK_DIR:-".gatling"}
+
+
+while getopts ":s:m:h" opt; do
+ case $opt in
+ s)
+ SOLR_ENDPOINT=$OPTARG
+ ;;
+ m)
+ SOLR_MAJOR_VERSION=$OPTARG
+ ;;
+ h)
+ print_help
+ exit 1
+ ;;
+ \?)
+ echo "Invalid option: -$OPTARG" >&2
+ exit 1
+ ;;
+ :)
+ echo "Option -$OPTARG requires an argument." >&2
+ exit 1
+ ;;
+ esac
+done
+
+CONF="wikipedia-${SOLR_MAJOR_VERSION}"
+
+
+echo "Pinging $SOLR_ENDPOINT"
+curl -s "$SOLR_ENDPOINT/solr/admin/cores" > /dev/null
+
+if [ $? -eq 0 ]; then
+ echo "Solr endpoint OK"
+else
+ echo "Couldn't talk to $SOLR_ENDPOINT"
+ exit 1
+fi
+
+set -e
+
+if [ ! -d ./scripts ]; then
+ echo "Run this script from the root of the project"
+ exit 1
+fi
+
+mkdir -p $TESTS_WORK_DIR/results
+
+if [ -e "$TESTS_WORK_DIR/batches" ]; then
+ echo "Found uncompressed wikipedia batches locally."
+else
+ # TODO It's probably beyond the purposes of this script to download a wiki
dump and preprocess it into
+ # batch-files for the caller, particularly since that involves some
decision-making on their part
+ # (batch-size, max-text-size, data-format), but maybe we could let them
specify a tarball URL of pre-
+ # prepared data that we can download and unpack?
+ echo "Data-batches not found in ${TESTS_WORK_DIR}/batches; create batch
files using the instructions in gatling-data-prep/README.md"
+ exit 1
+fi
+
+echo "Uploading latest configset"
+rm -f $TESTS_WORK_DIR/wikipedia-conf.zip
+( cd gatling-simulations/src/gatling/resources/configSets/$CONF ; zip -r
wikipedia-conf.zip * )
+mv
gatling-simulations/src/gatling/resources/configSets/$CONF/wikipedia-conf.zip
$TESTS_WORK_DIR/wikipedia-conf.zip
+
+curl -s -XPOST -T "$TESTS_WORK_DIR/wikipedia-conf.zip" -H "Content-Type:
application/octet-stream"
"$SOLR_ENDPOINT/solr/admin/configs?action=UPLOAD&name=wikipedia&overwrite=true&cleanup=true"
&& echo "...Configset uploaded"
+
+echo "Done! This script doesn't create the Collection, that's expected to be
done inside the simulation"
+echo " Example: ./gradlew gatlingRun --simulation
index.IndexWikipediaBatchesSimulation"
diff --git a/settings.gradle b/settings.gradle
index f0d9385..39abf37 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -12,4 +12,8 @@ rootProject.name = 'solr-sandbox'
include 'crossdc-consumer'
include 'crossdc-producer'
include 'crossdc-commons'
+
include 'encryption'
+
+include 'gatling-data-prep'
+include 'gatling-simulations'
