IGNITE-4212 Ignite Benchmarking Simplification and Automation / IGNITE-4478 Fixing documentation and resolving lack of usability
Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/605d946a Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/605d946a Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/605d946a Branch: refs/heads/ignite-3477-merge2.0 Commit: 605d946ada5ca4e562e256fa0edb29cc25dea7dc Parents: e1c3dda Author: oleg-ostanin <oosta...@gridgain.com> Authored: Thu Feb 9 20:08:57 2017 +0300 Committer: Anton Vinogradov <a...@apache.org> Committed: Thu Feb 9 20:08:57 2017 +0300 ---------------------------------------------------------------------- modules/yardstick/DEVNOTES-standalone.txt | 15 +++ modules/yardstick/DEVNOTES.txt | 20 ++-- modules/yardstick/README.txt | 119 +++++++++++++------ .../config/benchmark-sample.properties | 62 ++++++++++ modules/yardstick/pom.xml | 4 +- pom.xml | 1 + 6 files changed, 172 insertions(+), 49 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/ignite/blob/605d946a/modules/yardstick/DEVNOTES-standalone.txt ---------------------------------------------------------------------- diff --git a/modules/yardstick/DEVNOTES-standalone.txt b/modules/yardstick/DEVNOTES-standalone.txt new file mode 100644 index 0000000..d14dde2 --- /dev/null +++ b/modules/yardstick/DEVNOTES-standalone.txt @@ -0,0 +1,15 @@ +Building from standalone sources +================================ + +Run + +mvn clean package + +Artifacts can be found in `/target/assembly directory.` + + +Writing Ignite Benchmarks +========================= + +All benchmarks extend `AbstractBenchmark` class. A new benchmark should also extend this abstract class +and implement `test` method. This is the method which actually tests performance. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ignite/blob/605d946a/modules/yardstick/DEVNOTES.txt ---------------------------------------------------------------------- diff --git a/modules/yardstick/DEVNOTES.txt b/modules/yardstick/DEVNOTES.txt index 6921243..b8b1f53 100644 --- a/modules/yardstick/DEVNOTES.txt +++ b/modules/yardstick/DEVNOTES.txt @@ -1,16 +1,20 @@ -Yardstick Ignite Maven Build Instructions +Building from Ignite sources ========================================= -Yardstick can be build from standalone sources using following maven command: +Run -mvn clean package +mvn clean package -Pyardstick -pl modules/yardstick -am -DskipTests -Artifacts can be found in /target/assembly directory. +in the Apache Ignite root directory. -To build yardstick from Apache Ignite sources use: +This command will compile the project and also unpack scripts from `yardstick-resources.zip` file to +`modules/yardstick/target/assembly/bin` directory. -mvn clean package -Pyardstick -pl modules/yardstick -am -DskipTests +Artifacts can be found in `modules/yardstick/target/assembly` directory. + +Writing Ignite Benchmarks +========================= -in Apache Ignite root directory +All benchmarks extend `AbstractBenchmark` class. A new benchmark should also extend this abstract class +and implement `test` method. This is the method which actually tests performance. -Artifacts can be found in modules/yardstick/target/assembly directory. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ignite/blob/605d946a/modules/yardstick/README.txt ---------------------------------------------------------------------- diff --git a/modules/yardstick/README.txt b/modules/yardstick/README.txt index 353ddd8..dcfe0e6 100644 --- a/modules/yardstick/README.txt +++ b/modules/yardstick/README.txt @@ -1,55 +1,92 @@ Yardstick Ignite Benchmarks =========================== -Yardstick Ignite is a set of Ignite Grid (http://ignite.apache.org/) benchmarks written on top of Yardstick framework. +Apache Ignite benchmarks are written on top of Yardstick Framework (https://github.com/gridgain/yardstick) and allow +you to measure performance of various Apache Ignite components and modules. -Yardstick Framework -=================== -Visit Yardstick Repository (https://github.com/gridgain/yardstick) for detailed information on how to run Yardstick benchmarks and how to generate graphs. +The documentation below describes how to execute and configure already assembled benchmarks. If you need to add new +benchmarks or build existing one then refer to instruction from `DEVNOTES.txt` in source directory. -The documentation below describes configuration parameters in addition to standard Yardstick parameters. +Visit Yardstick Repository (https://github.com/gridgain/yardstick) for detailed information in regards resulting graphs +generation and how the frameworks works. -Building from Ignite sources -============ -1. Create a local clone of Ignite repository -2. Run -mvn package +Running Ignite Benchmarks Locally +========================================== -command for Yardstick Ignite POM +The simplest way to start with benchmarking is to use one of the executable scripts available under `benchmarks\bin` directory: -Building from standalone sources -===================== -Run +./bin/benchmark-run-all.sh config/benchmark-atomic-put.properties -mvn package +The command above will benchmark the cache put operation for a distributed atomic cache. The results of the benchmark will be added +to auto-generated `results-{DATE-TIME}` directory. -command for Yardstick Ignite POM +If `./bin/benchmark-run-all.sh` command is executed as is without any parameters and modifications in configurations files then +all the available benchmarks will be executed on a local machine using `config/benchmark.properties` configuration. -Provided Benchmarks -=================== -The following benchmarks are provided: +To get more information about available benchmarks and configuration parameters refer to âProvided Benchmarksâ and +âProperties And Command Line Argumentsâ sections below. + + +Running Ignite Benchmarks Remotely +========================================= + +For running Ignite benchmarks on remote hosts you need to upload Ignite-Yardstick to each one of your remote hosts. + +NOTE: The path to the uploaded Ignite-Yardstick should be exactly the same on each host. + +Then you need to make some changes in config files: -1. `GetBenchmark` - benchmarks atomic distributed cache get operation -2. `PutBenchmark` - benchmarks atomic distributed cache put operation -3. `PutGetBenchmark` - benchmarks atomic distributed cache put and get operations together -4. `PutTxBenchmark` - benchmarks transactional distributed cache put operation -5. `PutGetTxBenchmark` - benchmarks transactional distributed cache put and get operations together -6. `SqlQueryBenchmark` - benchmarks distributed SQL query over cached data -7. `SqlQueryJoinBenchmark` - benchmarks distributed SQL query with a Join over cached data -8. `SqlQueryPutBenchmark` - benchmarks distributed SQL query with simultaneous cache updates +1. You need to comment or delete the + <property name="localHost" value="127.0.0.1"/> +line in `config/ignite-localhost-config.xml` file. -Writing Ignite Benchmarks -========================= -All benchmarks extend `AbstractBenchmark` class. A new benchmark should also extend this abstract class -and implement `test` method. This is the method that is actually benchmarked. +2. You need to replace all the `127.0.0.1` addresses in `ignite-localhost-config.xml` file by actual IPs of your remote +servers. You can add or delete lines with IP addresses if you want to run benchmarks on different number of hosts. +There must be at least one IP address in the list. -Running Ignite Benchmarks -========================= -Before running Ignite benchmarks, run: +3. You need to replace the `localhost` strings by actual IP of your servers in the lines +SERVERS='localhost,localhost' +DRIVERS='localhost' +in `config/benchmark-atomic-put.properties` file. + +Then use the following command: + +./bin/benchmark-run-all.sh config/benchmark-atomic-put.properties + + +It is recommended to create some copies of original config files, edit these copies and then use as a +parameter for `./bin/benchmark-run-all.sh` script. + + +Provided Benchmarks +=================== +The following benchmarks are provided: -mvn package +1. `GetBenchmark` - benchmarks atomic distributed cache get operation +2. `PutBenchmark` - benchmarks atomic distributed cache put operation +3. `PutGetBenchmark` - benchmarks atomic distributed cache put and get operations together +4. `PutTxBenchmark` - benchmarks transactional distributed cache put operation +5. `PutGetTxBenchmark` - benchmarks transactional distributed cache put and get operations together +6. `SqlQueryBenchmark` - benchmarks distributed SQL query over cached data +7. `SqlQueryJoinBenchmark` - benchmarks distributed SQL query with a Join over cached data +8. `SqlQueryPutBenchmark` - benchmarks distributed SQL query with simultaneous cache updates +9. `AffinityCallBenchmark` - benchmarks affinity call operation +10. `ApplyBenchmark` - benchmarks apply operation +11. `BroadcastBenchmark` - benchmarks broadcast operations +12. `ExecuteBenchmark` - benchmarks execute operations +13. `RunBenchmark` - benchmarks running task operations +14. `PutGetOffHeapBenchmark` - benchmarks atomic distributed cache put and get operations together off heap +15. `PutGetOffHeapValuesBenchmark` - benchmarks atomic distributed cache put value operations off heap +16. `PutOffHeapBenchmark` - benchmarks atomic distributed cache put operations off heap +17. `PutOffHeapValuesBenchmark` - benchmarks atomic distributed cache get value operations off heap +18. `PutTxOffHeapBenchmark` - benchmarks transactional distributed cache put operation off heap +19. `PutTxOffHeapValuesBenchmark` - benchmarks transactional distributed cache put value operation off heap +20. `SqlQueryOffHeapBenchmark` -benchmarks distributed SQL query over cached data off heap +21. `SqlQueryJoinOffHeapBenchmark` - benchmarks distributed SQL query with a Join over cached data off heap +22. `SqlQueryPutOffHeapBenchmark` - benchmarks distributed SQL query with simultaneous cache updates off heap +23. `PutAllBenchmark` - benchmarks atomic distributed cache batch put operation +24. `PutAllTxBenchmark` - benchmarks transactional distributed cache batch put operation -command. This command will compile the project and also will unpack scripts from `yardstick-resources.zip` file to `bin` directory. Properties And Command Line Arguments ===================================== @@ -67,15 +104,19 @@ The following Ignite benchmark properties can be defined in the benchmark config * `-cs` or `--cacheStore` - Enable or disable cache store readThrough, writeThrough * `-cl` or `--client` - Client flag * `-nc` or `--nearCache` - Near cache flag -* `-nn <num>` or `--nodeNumber <num>` - Number of nodes (automatically set in `benchmark.properties`), used to wait for the specified number of nodes to start +* `-nn <num>` or `--nodeNumber <num>` - Number of nodes (automatically set in `benchmark.properties`), used to wait for + the specified number of nodes to start * `-sm <mode>` or `-syncMode <mode>` - Synchronization mode (defined in `CacheWriteSynchronizationMode`) * `-ot` or `--offheapTiered` - Flag indicating whether tiered off-heap mode is on * `-ov` or `--offheapValuesOnly` - Flag indicating whether off-heap mode is on and only cache values are stored off-heap * `-r <num>` or `--range` - Range of keys that are randomly generated for cache operations +* `-rd or --restartdelay` - Restart delay in seconds +* `-rs or --restartsleep` - Restart sleep in seconds * `-rth <host>` or `--restHost <host>` - REST TCP host * `-rtp <num>` or `--restPort <num>` - REST TCP port, indicates that a Ignite node is ready to process Ignite Clients * `-ss` or `--syncSend` - Flag indicating whether synchronous send is used in `TcpCommunicationSpi` -* `-txc <value>` or `--txConcurrency <value>` - Cache transaction concurrency control, either `OPTIMISTIC` or `PESSIMISTIC` (defined in `CacheTxConcurrency`) +* `-txc <value>` or `--txConcurrency <value>` - Cache transaction concurrency control, either `OPTIMISTIC` or + `PESSIMISTIC` (defined in `CacheTxConcurrency`) * `-txi <value>` or `--txIsolation <value>` - Cache transaction isolation (defined in `CacheTxIsolation`) * `-wb` or `--writeBehind` - Enable or disable writeBehind for cache store * `-wom <mode>` or `--writeOrderMode <mode>` - Write order mode for ATOMIC caches (defined in `CacheAtomicWriteOrderMode`) @@ -91,7 +132,7 @@ SERVER_HOSTS=localhost,localhost Note that -dn and -sn, which stand for data node and server node, are native Yardstick parameters and are documented in Yardstick framework. =========================================================================================================================================== -CONFIGS="-b 1 -sm PRIMARY_SYNC -dn PutBenchmark -sn IgniteNode" +CONFIGS="-b 1 -sm PRIMARY_SYNC -dn PutBenchmark`IgniteNode" ``` Issues http://git-wip-us.apache.org/repos/asf/ignite/blob/605d946a/modules/yardstick/config/benchmark-sample.properties ---------------------------------------------------------------------- diff --git a/modules/yardstick/config/benchmark-sample.properties b/modules/yardstick/config/benchmark-sample.properties new file mode 100644 index 0000000..fee7cbb --- /dev/null +++ b/modules/yardstick/config/benchmark-sample.properties @@ -0,0 +1,62 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Contains benchmarks for ATOMIC cache. +# + +# JVM options. +JVM_OPTS=${JVM_OPTS}" -DIGNITE_QUIET=false" + +# Uncomment to enable concurrent garbage collection (GC) if you encounter long GC pauses. +# JVM_OPTS=${JVM_OPTS}" \ +# -XX:+UseParNewGC \ +# -XX:+UseConcMarkSweepGC \ +# -XX:+UseTLAB \ +# -XX:NewSize=128m \ +# -XX:MaxNewSize=128m \ +# -XX:MaxTenuringThreshold=0 \ +# -XX:SurvivorRatio=1024 \ +# -XX:+UseCMSInitiatingOccupancyOnly \ +# -XX:CMSInitiatingOccupancyFraction=60 \ +#" + +# List of default probes. +# Add DStatProbe or VmStatProbe if your OS supports it (e.g. if running on Linux). +BENCHMARK_DEFAULT_PROBES=ThroughputLatencyProbe,PercentileProbe + +# Packages where the specified benchmark is searched by reflection mechanism. +BENCHMARK_PACKAGES=org.yardstickframework,org.apache.ignite.yardstick + +# Probe point writer class name. +# BENCHMARK_WRITER= + +# Comma-separated list of the hosts to run BenchmarkServers on. 2 nodes on local host are enabled by default. +SERVER_HOSTS=localhost,localhost + +# Comma-separated list of the hosts to run BenchmarkDrivers on. 1 node on local host is enabled by default. +DRIVER_HOSTS=localhost + +# Remote username. +# REMOTE_USER= + +# Number of nodes, used to wait for the specified number of nodes to start. +nodesNum=$((`echo ${SERVER_HOSTS} | tr ',' '\n' | wc -l` + `echo ${DRIVER_HOSTS} | tr ',' '\n' | wc -l`)) + +# Run configuration. +# Note that each benchmark is set to run for 300 seconds (5 mins) with warm-up set to 60 seconds (1 minute). +CONFIGS="\ +-cfg ${SCRIPT_DIR}/../config/ignite-localhost-config.xml -nn ${nodesNum} -b 1 -w 60 -d 300 -t 64 -sm PRIMARY_SYNC -dn IgnitePutBenchmark -sn IgniteNode -ds atomic-put-1-backup,\ +" http://git-wip-us.apache.org/repos/asf/ignite/blob/605d946a/modules/yardstick/pom.xml ---------------------------------------------------------------------- diff --git a/modules/yardstick/pom.xml b/modules/yardstick/pom.xml index 4c4d138..d96fcc6 100644 --- a/modules/yardstick/pom.xml +++ b/modules/yardstick/pom.xml @@ -35,7 +35,7 @@ <url>http://ignite.apache.org</url> <properties> - <yardstick.version>0.8.0</yardstick.version> + <yardstick.version>0.8.2</yardstick.version> <spring.version>4.1.0.RELEASE</spring.version> </properties> @@ -56,7 +56,7 @@ <groupId>org.apache.ignite</groupId> <artifactId>ignite-indexing</artifactId> <version>${project.version}</version> - </dependency> + </dependency> <dependency> <groupId>org.apache.ignite</groupId> http://git-wip-us.apache.org/repos/asf/ignite/blob/605d946a/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index 790cd4b..ea76053 100644 --- a/pom.xml +++ b/pom.xml @@ -506,6 +506,7 @@ <fileset dir="${basedir}/target/release-package/benchmarks/config/"> <include name="*.*" /> <exclude name="benchmark.properties"/> + <exclude name="benchmark-sample.properties"/> <exclude name="benchmark-multicast.properties"/> <exclude name="ignite-base-config.xml"/> <exclude name="ignite-localhost-config.xml"/>