Hey,
I didn't try your script but it's not quite clear how it's gathering
statistics. just once at each end of the test?
I'd left some test suites from my blog posts in the test-suites/ directory
of mc-crusher, but they weren't very clean or easy to modify...
Found another one I'd done later with a refactoring that's a lot better:
https://github.com/memcached/mc-crusher/blob/master/test-suites/test-suite-example
just cleaned up some bits and wrote a few notes into it. I don't have the
if tooling I was using to process the data into json blobs (maybe I'll add
it you want it :P)
There are a bunch of parameters to tune, or tests you can outright delete.
Any mc-crusher configuration file can be represented in there (see the
"get_*_conf" subs).
This will grab a bunch of data _while_ the perf test runs, which you can
then plot or examine for consistency. It also runs a scaled set of tests,
so you can find the performance cliffs. I don't believe any benchmark is
valid unless you find the test to failure point and show what latency
looks like on your way there.
So this will run a bunch of tests and scale up the mc-crusher config. You
get:
- stats sample output (see sample_args()). You get the rate every N
seconds for X periods, and at the end it'll have an average from the full
run. You can look into this for spikes/dips/consistency.
- the stats sampling starts _after_ the benchmark starts, so you don't end
up weighting ramp up time into the average.
- latency sampled output (along with some summary information). You can
examine these files to see when the benchmark throughput has exceeded your
latency budget.
- Some other crap I probably forget.
Anyway this is how I test stuff for those blog posts with all the graphs.
You mostly tweak it a bit and ignore it for a while, then tweak and try
again.
On Wed, 8 Apr 2020, Martin Grigorov wrote:
> Hi,
>
> Does it make sense to have a command 'flush_stats' ?
> 'flush_all' resets the caches but the stats stay, e.g. cmd_get value is not
> reset.
>
> I'd find it useful for performance measurements:
> 1) run random warm-up
> 2) reset_stats
> 3) run perf test
> 4) collect stats
>
> Right now I do:
> 1) run random warm-up
> 2) collect stats
> 3) run perf test
> 4) collect stats
> 5) substract 2) from 4) and store the result
>
> Regards,
> Martin
>
> On Tue, Apr 7, 2020 at 3:29 PM Martin Grigorov
> wrote:
> Hi Dormando,
>
> This is a continuation of the mail thread with subject "Is ARM64
> officially supported ?" [1]
>
> I've refreshed my Perl coding skills and came up with this wrapper around
> mc-crusher:
>
> =
> #!/usr/bin/env perl
>
> # sudo cpan Cache::Memcached
>
> use strict;
> use warnings;
>
> use Cache::Memcached;
> use Data::Dumper;
> use Time::Piece;
>
> my $TESTBED_HOME = $ENV{'TESTBED_HOME'};
> die "Env variable 'TESTBED_HOME' is not set!" if ! defined($TESTBED_HOME);
> my $MC_CRUSHER_HOME = $ENV{'MC_CRUSHER_HOME'};
> die "Env variable 'MC_CRUSHER_HOME' is not set!" if !
> defined($MC_CRUSHER_HOME);
>
> if (scalar(@ARGV) < 2) {
> die "Usage: mc-crusher.pl. For
> example: mc-crusher.pl cmd_get a.b.c.d 11211 60"
> }
>
> my $config = shift @ARGV;
> my $host = shift @ARGV;
> my $port = shift @ARGV || 8080;
> my $duration = shift @ARGV || 5 * 60; # 5 mins
>
> system("$MC_CRUSHER_HOME/mc-crusher --conf
> $TESTBED_HOME/etc/memcached/mc-crusher/conf/$config.conf --ip $host --port
> $port --timeout $duration");
>
> print "MC Crusher status: $!\n" if $!;
>
> my $serverAddr = "$host:$port";
> print "Server: $serverAddr\n";
> my @servers = ($serverAddr);
>
> my $memcached = new Cache::Memcached;
> $memcached->set_servers(\@servers);
>
> my $stats = $memcached->stats(['misc'])->{'hosts'}->{$serverAddr}->{'misc'};
> warn Dumper($stats);
>
> my $timestamp = localtime->datetime();
> my $cmd_per_sec = int($stats->{$config}) / $duration;
> my $bytes_written = $stats->{'bytes_written'};
> my $bytes_read = $stats->{'bytes_read'};
> my $time_system = $stats->{'rusage_system'};
> my $time_user = $stats->{'rusage_user'};
>
> my $today = localtime->ymd();
> my $folder = "$TESTBED_HOME/etc/memcached/$today";
> system("mkdir -p $folder");
> print "Cannot create '$folder': $!\n" if $!;
> my $filename = "$folder/memcached-mc-crusher-report-$host-$config.csv";
>
> my $headerPrefix = "${host}_${config}";
> open(my $fh, '>', $filename) or die "Could not open file '$filename': $!";
> print $fh
> "timeStamp,${headerPrefix}_per_sec,${headerPrefix}_bytes_written,${headerPrefix}_bytes_read,"
> .
> "${headerPrefix}_time_system,${headerPrefix}_time_user,config,host\n";
> print $fh
> "$timestamp,$cmd_per_sec,$bytes_written,$bytes_read,$time_system,$time_user,$config,$host\n";
> close $fh;
> print "done\n";
> ==
>
> Basically it executes mc-crusher with different configurations against
> different servers (x64 vs arm64) and stores the stats in CSV files, which
> l
