Hey, I didn't try your script but it's not quite clear how it's gathering statistics. just once at each end of the test?
I'd left some test suites from my blog posts in the test-suites/ directory of mc-crusher, but they weren't very clean or easy to modify... Found another one I'd done later with a refactoring that's a lot better: https://github.com/memcached/mc-crusher/blob/master/test-suites/test-suite-example just cleaned up some bits and wrote a few notes into it. I don't have the if tooling I was using to process the data into json blobs (maybe I'll add it you want it :P) There are a bunch of parameters to tune, or tests you can outright delete. Any mc-crusher configuration file can be represented in there (see the "get_*_conf" subs). This will grab a bunch of data _while_ the perf test runs, which you can then plot or examine for consistency. It also runs a scaled set of tests, so you can find the performance cliffs. I don't believe any benchmark is valid unless you find the test to failure point and show what latency looks like on your way there. So this will run a bunch of tests and scale up the mc-crusher config. You get: - stats sample output (see sample_args()). You get the rate every N seconds for X periods, and at the end it'll have an average from the full run. You can look into this for spikes/dips/consistency. - the stats sampling starts _after_ the benchmark starts, so you don't end up weighting ramp up time into the average. - latency sampled output (along with some summary information). You can examine these files to see when the benchmark throughput has exceeded your latency budget. - Some other crap I probably forget. Anyway this is how I test stuff for those blog posts with all the graphs. You mostly tweak it a bit and ignore it for a while, then tweak and try again. On Wed, 8 Apr 2020, Martin Grigorov wrote: > Hi, > > Does it make sense to have a command 'flush_stats' ? > 'flush_all' resets the caches but the stats stay, e.g. cmd_get value is not > reset. > > I'd find it useful for performance measurements: > 1) run random warm-up > 2) reset_stats > 3) run perf test > 4) collect stats > > Right now I do: > 1) run random warm-up > 2) collect stats > 3) run perf test > 4) collect stats > 5) substract 2) from 4) and store the result > > Regards, > Martin > > On Tue, Apr 7, 2020 at 3:29 PM Martin Grigorov <martin.grigo...@gmail.com> > wrote: > Hi Dormando, > > This is a continuation of the mail thread with subject "Is ARM64 > officially supported ?" [1] > > I've refreshed my Perl coding skills and came up with this wrapper around > mc-crusher: > > ================================================= > #!/usr/bin/env perl > > # sudo cpan Cache::Memcached > > use strict; > use warnings; > > use Cache::Memcached; > use Data::Dumper; > use Time::Piece; > > my $TESTBED_HOME = $ENV{'TESTBED_HOME'}; > die "Env variable 'TESTBED_HOME' is not set!" if ! defined($TESTBED_HOME); > my $MC_CRUSHER_HOME = $ENV{'MC_CRUSHER_HOME'}; > die "Env variable 'MC_CRUSHER_HOME' is not set!" if ! > defined($MC_CRUSHER_HOME); > > if (scalar(@ARGV) < 2) { > die "Usage: mc-crusher.pl <conf> <host> <port> <timeoutInSecs>. For > example: mc-crusher.pl cmd_get a.b.c.d 11211 60" > } > > my $config = shift @ARGV; > my $host = shift @ARGV; > my $port = shift @ARGV || 8080; > my $duration = shift @ARGV || 5 * 60; # 5 mins > > system("$MC_CRUSHER_HOME/mc-crusher --conf > $TESTBED_HOME/etc/memcached/mc-crusher/conf/$config.conf --ip $host --port > $port --timeout $duration"); > > print "MC Crusher status: $!\n" if $!; > > my $serverAddr = "$host:$port"; > print "Server: $serverAddr\n"; > my @servers = ($serverAddr); > > my $memcached = new Cache::Memcached; > $memcached->set_servers(\@servers); > > my $stats = $memcached->stats(['misc'])->{'hosts'}->{$serverAddr}->{'misc'}; > warn Dumper($stats); > > my $timestamp = localtime->datetime(); > my $cmd_per_sec = int($stats->{$config}) / $duration; > my $bytes_written = $stats->{'bytes_written'}; > my $bytes_read = $stats->{'bytes_read'}; > my $time_system = $stats->{'rusage_system'}; > my $time_user = $stats->{'rusage_user'}; > > my $today = localtime->ymd(); > my $folder = "$TESTBED_HOME/etc/memcached/$today"; > system("mkdir -p $folder"); > print "Cannot create '$folder': $!\n" if $!; > my $filename = "$folder/memcached-mc-crusher-report-$host-$config.csv"; > > my $headerPrefix = "${host}_${config}"; > open(my $fh, '>', $filename) or die "Could not open file '$filename': $!"; > print $fh > "timeStamp,${headerPrefix}_per_sec,${headerPrefix}_bytes_written,${headerPrefix}_bytes_read," > . > "${headerPrefix}_time_system,${headerPrefix}_time_user,config,host\n"; > print $fh > "$timestamp,$cmd_per_sec,$bytes_written,$bytes_read,$time_system,$time_user,$config,$host\n"; > close $fh; > print "done\n"; > ========================================== > > Basically it executes mc-crusher with different configurations against > different servers (x64 vs arm64) and stores the stats in CSV files, which > later I visualize in Kibana. > > So far I've ran with these configs: > > cmd_get.conf: > send=ascii_get,recv=blind_read,conns=50,key_prefix=foobar,key_prealloc=0,pipelines=8 > send=ascii_set,recv=blind_read,conns=10,key_prefix=foobar,key_prealloc=0,pipelines=4,stop_after=200000,usleep=1000,value_size=10 > > cmd_set.conf: > send=ascii_set,recv=blind_read,conns=5,key_prefix=foobar,value_size=2,value=hi,key_prealloc=1 > > > I always run those against freshly started Memcached server (started with: > ./memcached -p 8080). > After mc-crusher finishes I get the value of the current command, e.g. > cmd_get, and divide it by > the duration, to get the cmds per second. > > And here are the numbers I have so far: > > gets/s: > - x64: ~450 000 > - arm64: ~310 000 > > sets/s: > - x64: ~25 000 > - arm64: ~63 000 > > I.e. x64 is faster for GET and arm64 is faster for SET. > Those numbers are averages from multiple runs. > > Do you see anything wrong in my Perl script ? Or any ideas how to improve it. > Any hints how to profile the server to see why/where it is slower on > different platforms ? > > 1. https://groups.google.com/forum/#!topic/memcached/8hT2BT9cgEc > > > Regards, > Martin > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to memcached+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/memcached/CAMomwMqQwxDYchMFde6vDDpB6hAqRsnznYcUc5HP2RkKUW-v_Q%40mail.gmail.com. > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.2005011830170.7684%40dskull.
