Re: Load testing with mc-crusher
Hey,
I didn't try your script but it's not quite clear how it's gathering
statistics. just once at each end of the test?
I'd left some test suites from my blog posts in the test-suites/ directory
of mc-crusher, but they weren't very clean or easy to modify...
Found another one I'd done later with a refactoring that's a lot better:
https://github.com/memcached/mc-crusher/blob/master/test-suites/test-suite-example
just cleaned up some bits and wrote a few notes into it. I don't have the
if tooling I was using to process the data into json blobs (maybe I'll add
it you want it :P)
There are a bunch of parameters to tune, or tests you can outright delete.
Any mc-crusher configuration file can be represented in there (see the
"get_*_conf" subs).
This will grab a bunch of data _while_ the perf test runs, which you can
then plot or examine for consistency. It also runs a scaled set of tests,
so you can find the performance cliffs. I don't believe any benchmark is
valid unless you find the test to failure point and show what latency
looks like on your way there.
So this will run a bunch of tests and scale up the mc-crusher config. You
get:
- stats sample output (see sample_args()). You get the rate every N
seconds for X periods, and at the end it'll have an average from the full
run. You can look into this for spikes/dips/consistency.
- the stats sampling starts _after_ the benchmark starts, so you don't end
up weighting ramp up time into the average.
- latency sampled output (along with some summary information). You can
examine these files to see when the benchmark throughput has exceeded your
latency budget.
- Some other crap I probably forget.
Anyway this is how I test stuff for those blog posts with all the graphs.
You mostly tweak it a bit and ignore it for a while, then tweak and try
again.
On Wed, 8 Apr 2020, Martin Grigorov wrote:
> Hi,
>
> Does it make sense to have a command 'flush_stats' ?
> 'flush_all' resets the caches but the stats stay, e.g. cmd_get value is not
> reset.
>
> I'd find it useful for performance measurements:
> 1) run random warm-up
> 2) reset_stats
> 3) run perf test
> 4) collect stats
>
> Right now I do:
> 1) run random warm-up
> 2) collect stats
> 3) run perf test
> 4) collect stats
> 5) substract 2) from 4) and store the result
>
> Regards,
> Martin
>
> On Tue, Apr 7, 2020 at 3:29 PM Martin Grigorov
> wrote:
> Hi Dormando,
>
> This is a continuation of the mail thread with subject "Is ARM64
> officially supported ?" [1]
>
> I've refreshed my Perl coding skills and came up with this wrapper around
> mc-crusher:
>
> =
> #!/usr/bin/env perl
>
> # sudo cpan Cache::Memcached
>
> use strict;
> use warnings;
>
> use Cache::Memcached;
> use Data::Dumper;
> use Time::Piece;
>
> my $TESTBED_HOME = $ENV{'TESTBED_HOME'};
> die "Env variable 'TESTBED_HOME' is not set!" if ! defined($TESTBED_HOME);
> my $MC_CRUSHER_HOME = $ENV{'MC_CRUSHER_HOME'};
> die "Env variable 'MC_CRUSHER_HOME' is not set!" if !
> defined($MC_CRUSHER_HOME);
>
> if (scalar(@ARGV) < 2) {
> die "Usage: mc-crusher.pl. For
> example: mc-crusher.pl cmd_get a.b.c.d 11211 60"
> }
>
> my $config = shift @ARGV;
> my $host = shift @ARGV;
> my $port = shift @ARGV || 8080;
> my $duration = shift @ARGV || 5 * 60; # 5 mins
>
> system("$MC_CRUSHER_HOME/mc-crusher --conf
> $TESTBED_HOME/etc/memcached/mc-crusher/conf/$config.conf --ip $host --port
> $port --timeout $duration");
>
> print "MC Crusher status: $!\n" if $!;
>
> my $serverAddr = "$host:$port";
> print "Server: $serverAddr\n";
> my @servers = ($serverAddr);
>
> my $memcached = new Cache::Memcached;
> $memcached->set_servers(\@servers);
>
> my $stats = $memcached->stats(['misc'])->{'hosts'}->{$serverAddr}->{'misc'};
> warn Dumper($stats);
>
> my $timestamp = localtime->datetime();
> my $cmd_per_sec = int($stats->{$config}) / $duration;
> my $bytes_written = $stats->{'bytes_written'};
> my $bytes_read = $stats->{'bytes_read'};
> my $time_system = $stats->{'rusage_system'};
> my $time_user = $stats->{'rusage_user'};
>
> my $today = localtime->ymd();
> my $folder = "$TESTBED_HOME/etc/memcached/$today";
> system("mkdir -p $folder");
> print "Cannot create '$folder': $!\n" if $!;
> my $filename = "$folder/memcached-mc-crusher-report-$host-$config.csv";
>
> my $headerPrefix = "${host}_${config}";
> open(my $fh, '>', $filename) or die "Could not open file '$filename': $!";
> print $fh
> "timeStamp,${headerPrefix}_per_sec,${headerPrefix}_bytes_written,${headerPrefix}_bytes_read,"
> .
> "${headerPrefix}_time_system,${headerPrefix}_time_user,config,host\n";
> print $fh
> "$timestamp,$cmd_per_sec,$bytes_written,$bytes_read,$time_system,$time_user,$config,$host\n";
> close $fh;
> print "done\n";
> ==
>
> Basically it executes mc-crusher with different configurations against
> different servers (x64 vs arm64) and stores the stats in CSV files, which
>
Re: Load testing with mc-crusher
Hola Martin,
El mar., 7 abr. 2020 a las 15:29, Martin Grigorov (<
martin.grigo...@gmail.com>) escribió:
> Hi Dormando,
>
> This is a continuation of the mail thread with subject "Is ARM64
> officially supported ?" [1]
>
> I've refreshed my Perl coding skills and came up with this wrapper around
> mc-crusher:
>
> =
> #!/usr/bin/env perl
>
> # sudo cpan Cache::Memcached
>
> use strict;
> use warnings;
>
> use Cache::Memcached;
> use Data::Dumper;
> use Time::Piece;
>
> my $TESTBED_HOME = $ENV{'TESTBED_HOME'};
> die "Env variable 'TESTBED_HOME' is not set!" if ! defined($TESTBED_HOME);
> my $MC_CRUSHER_HOME = $ENV{'MC_CRUSHER_HOME'};
> die "Env variable 'MC_CRUSHER_HOME' is not set!" if !
> defined($MC_CRUSHER_HOME);
>
> if (scalar(@ARGV) < 2) {
> die "Usage: mc-crusher.pl.
> For example: mc-crusher.pl cmd_get a.b.c.d 11211 60"
> }
>
> my $config = shift @ARGV;
> my $host = shift @ARGV;
> my $port = shift @ARGV || 8080;
> my $duration = shift @ARGV || 5 * 60; # 5 mins
>
> system("$MC_CRUSHER_HOME/mc-crusher --conf
> $TESTBED_HOME/etc/memcached/mc-crusher/conf/$config.conf --ip $host --port
> $port --timeout $duration");
>
> print "MC Crusher status: $!\n" if $!;
>
> my $serverAddr = "$host:$port";
> print "Server: $serverAddr\n";
> my @servers = ($serverAddr);
>
> my $memcached = new Cache::Memcached;
> $memcached->set_servers(\@servers);
>
> my $stats =
> $memcached->stats(['misc'])->{'hosts'}->{$serverAddr}->{'misc'};
> warn Dumper($stats);
>
> my $timestamp = localtime->datetime();
> my $cmd_per_sec = int($stats->{$config}) / $duration;
> my $bytes_written = $stats->{'bytes_written'};
> my $bytes_read = $stats->{'bytes_read'};
> my $time_system = $stats->{'rusage_system'};
> my $time_user = $stats->{'rusage_user'};
>
> my $today = localtime->ymd();
> my $folder = "$TESTBED_HOME/etc/memcached/$today";
> system("mkdir -p $folder");
> print "Cannot create '$folder': $!\n" if $!;
> my $filename = "$folder/memcached-mc-crusher-report-$host-$config.csv";
>
> my $headerPrefix = "${host}_${config}";
> open(my $fh, '>', $filename) or die "Could not open file '$filename': $!";
> print $fh
> "timeStamp,${headerPrefix}_per_sec,${headerPrefix}_bytes_written,${headerPrefix}_bytes_read,"
> .
> "${headerPrefix}_time_system,${headerPrefix}_time_user,config,host\n";
> print $fh
> "$timestamp,$cmd_per_sec,$bytes_written,$bytes_read,$time_system,$time_user,$config,$host\n";
> close $fh;
> print "done\n";
> ==
>
> Basically it executes mc-crusher with different configurations against
> different servers (x64 vs arm64) and stores the stats in CSV files, which
> later I visualize in Kibana.
>
> So far I've ran with these configs:
>
> cmd_get.conf:
>
> send=ascii_get,recv=blind_read,conns=50,key_prefix=foobar,key_prealloc=0,pipelines=8
>
> send=ascii_set,recv=blind_read,conns=10,key_prefix=foobar,key_prealloc=0,pipelines=4,stop_after=20,usleep=1000,value_size=10
>
> cmd_set.conf:
>
> send=ascii_set,recv=blind_read,conns=5,key_prefix=foobar,value_size=2,value=hi,key_prealloc=1
>
>
> I always run those against freshly started Memcached server (started with:
> ./memcached -p 8080).
> After mc-crusher finishes I get the value of the current command, e.g.
> cmd_get, and divide it by
> the duration, to get the cmds per second.
>
> And here are the numbers I have so far:
>
> gets/s:
> - x64: ~450 000
> - arm64: ~310 000
>
> sets/s:
> - x64: ~25 000
> - arm64: ~63 000
>
> I.e. x64 is faster for GET and arm64 is faster for SET.
> Those numbers are averages from multiple runs.
>
> Do you see anything wrong in my Perl script ? Or any ideas how to improve
> it.
> Any hints how to profile the server to see why/where it is slower on
> different platforms ?
>
I was able to use your Perl script to run some tests here. It took me a
while to figure out how to provide the configurations but I managed :-)
I am experiencing a similar behavior on my machines - set command is faster
on ARM64 and get command on x86_64.
With a colleague of mine we are trying to find out the reason for these
differences with the help of 'perf' tool.
http://www.brendangregg.com/perf.html is a good resource how to use it.
Gracias!
Emilio
>
> 1. https://groups.google.com/forum/#!topic/memcached/8hT2BT9cgEc
>
>
> Regards,
> Martin
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to memcached+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/memcached/CAMomwMrES-%2BsTxYQrK9rQNaNCPRjA2FZUOYKR_rSgRUCiz5TiA%40mail.gmail.com
>
> .
>
--
---
You received this message because you are subscribed to the Google Groups
Re: Load testing with mc-crusher
Hi,
Does it make sense to have a command 'flush_stats' ?
'flush_all' resets the caches but the stats stay, e.g. cmd_get value is not
reset.
I'd find it useful for performance measurements:
1) run random warm-up
2) reset_stats
3) run perf test
4) collect stats
Right now I do:
1) run random warm-up
2) collect stats
3) run perf test
4) collect stats
5) substract 2) from 4) and store the result
Regards,
Martin
On Tue, Apr 7, 2020 at 3:29 PM Martin Grigorov
wrote:
> Hi Dormando,
>
> This is a continuation of the mail thread with subject "Is ARM64
> officially supported ?" [1]
>
> I've refreshed my Perl coding skills and came up with this wrapper around
> mc-crusher:
>
> =
> #!/usr/bin/env perl
>
> # sudo cpan Cache::Memcached
>
> use strict;
> use warnings;
>
> use Cache::Memcached;
> use Data::Dumper;
> use Time::Piece;
>
> my $TESTBED_HOME = $ENV{'TESTBED_HOME'};
> die "Env variable 'TESTBED_HOME' is not set!" if ! defined($TESTBED_HOME);
> my $MC_CRUSHER_HOME = $ENV{'MC_CRUSHER_HOME'};
> die "Env variable 'MC_CRUSHER_HOME' is not set!" if !
> defined($MC_CRUSHER_HOME);
>
> if (scalar(@ARGV) < 2) {
> die "Usage: mc-crusher.pl.
> For example: mc-crusher.pl cmd_get a.b.c.d 11211 60"
> }
>
> my $config = shift @ARGV;
> my $host = shift @ARGV;
> my $port = shift @ARGV || 8080;
> my $duration = shift @ARGV || 5 * 60; # 5 mins
>
> system("$MC_CRUSHER_HOME/mc-crusher --conf
> $TESTBED_HOME/etc/memcached/mc-crusher/conf/$config.conf --ip $host --port
> $port --timeout $duration");
>
> print "MC Crusher status: $!\n" if $!;
>
> my $serverAddr = "$host:$port";
> print "Server: $serverAddr\n";
> my @servers = ($serverAddr);
>
> my $memcached = new Cache::Memcached;
> $memcached->set_servers(\@servers);
>
> my $stats =
> $memcached->stats(['misc'])->{'hosts'}->{$serverAddr}->{'misc'};
> warn Dumper($stats);
>
> my $timestamp = localtime->datetime();
> my $cmd_per_sec = int($stats->{$config}) / $duration;
> my $bytes_written = $stats->{'bytes_written'};
> my $bytes_read = $stats->{'bytes_read'};
> my $time_system = $stats->{'rusage_system'};
> my $time_user = $stats->{'rusage_user'};
>
> my $today = localtime->ymd();
> my $folder = "$TESTBED_HOME/etc/memcached/$today";
> system("mkdir -p $folder");
> print "Cannot create '$folder': $!\n" if $!;
> my $filename = "$folder/memcached-mc-crusher-report-$host-$config.csv";
>
> my $headerPrefix = "${host}_${config}";
> open(my $fh, '>', $filename) or die "Could not open file '$filename': $!";
> print $fh
> "timeStamp,${headerPrefix}_per_sec,${headerPrefix}_bytes_written,${headerPrefix}_bytes_read,"
> .
> "${headerPrefix}_time_system,${headerPrefix}_time_user,config,host\n";
> print $fh
> "$timestamp,$cmd_per_sec,$bytes_written,$bytes_read,$time_system,$time_user,$config,$host\n";
> close $fh;
> print "done\n";
> ==
>
> Basically it executes mc-crusher with different configurations against
> different servers (x64 vs arm64) and stores the stats in CSV files, which
> later I visualize in Kibana.
>
> So far I've ran with these configs:
>
> cmd_get.conf:
>
> send=ascii_get,recv=blind_read,conns=50,key_prefix=foobar,key_prealloc=0,pipelines=8
>
> send=ascii_set,recv=blind_read,conns=10,key_prefix=foobar,key_prealloc=0,pipelines=4,stop_after=20,usleep=1000,value_size=10
>
> cmd_set.conf:
>
> send=ascii_set,recv=blind_read,conns=5,key_prefix=foobar,value_size=2,value=hi,key_prealloc=1
>
>
> I always run those against freshly started Memcached server (started with:
> ./memcached -p 8080).
> After mc-crusher finishes I get the value of the current command, e.g.
> cmd_get, and divide it by
> the duration, to get the cmds per second.
>
> And here are the numbers I have so far:
>
> gets/s:
> - x64: ~450 000
> - arm64: ~310 000
>
> sets/s:
> - x64: ~25 000
> - arm64: ~63 000
>
> I.e. x64 is faster for GET and arm64 is faster for SET.
> Those numbers are averages from multiple runs.
>
> Do you see anything wrong in my Perl script ? Or any ideas how to improve
> it.
> Any hints how to profile the server to see why/where it is slower on
> different platforms ?
>
> 1. https://groups.google.com/forum/#!topic/memcached/8hT2BT9cgEc
>
>
> Regards,
> Martin
>
>
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to memcached+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/memcached/CAMomwMqQwxDYchMFde6vDDpB6hAqRsnznYcUc5HP2RkKUW-v_Q%40mail.gmail.com.
Load testing with mc-crusher
Hi Dormando,
This is a continuation of the mail thread with subject "Is ARM64 officially
supported ?" [1]
I've refreshed my Perl coding skills and came up with this wrapper around
mc-crusher:
=
#!/usr/bin/env perl
# sudo cpan Cache::Memcached
use strict;
use warnings;
use Cache::Memcached;
use Data::Dumper;
use Time::Piece;
my $TESTBED_HOME = $ENV{'TESTBED_HOME'};
die "Env variable 'TESTBED_HOME' is not set!" if ! defined($TESTBED_HOME);
my $MC_CRUSHER_HOME = $ENV{'MC_CRUSHER_HOME'};
die "Env variable 'MC_CRUSHER_HOME' is not set!" if !
defined($MC_CRUSHER_HOME);
if (scalar(@ARGV) < 2) {
die "Usage: mc-crusher.pl. For
example: mc-crusher.pl cmd_get a.b.c.d 11211 60"
}
my $config = shift @ARGV;
my $host = shift @ARGV;
my $port = shift @ARGV || 8080;
my $duration = shift @ARGV || 5 * 60; # 5 mins
system("$MC_CRUSHER_HOME/mc-crusher --conf
$TESTBED_HOME/etc/memcached/mc-crusher/conf/$config.conf --ip $host --port
$port --timeout $duration");
print "MC Crusher status: $!\n" if $!;
my $serverAddr = "$host:$port";
print "Server: $serverAddr\n";
my @servers = ($serverAddr);
my $memcached = new Cache::Memcached;
$memcached->set_servers(\@servers);
my $stats = $memcached->stats(['misc'])->{'hosts'}->{$serverAddr}->{'misc'};
warn Dumper($stats);
my $timestamp = localtime->datetime();
my $cmd_per_sec = int($stats->{$config}) / $duration;
my $bytes_written = $stats->{'bytes_written'};
my $bytes_read = $stats->{'bytes_read'};
my $time_system = $stats->{'rusage_system'};
my $time_user = $stats->{'rusage_user'};
my $today = localtime->ymd();
my $folder = "$TESTBED_HOME/etc/memcached/$today";
system("mkdir -p $folder");
print "Cannot create '$folder': $!\n" if $!;
my $filename = "$folder/memcached-mc-crusher-report-$host-$config.csv";
my $headerPrefix = "${host}_${config}";
open(my $fh, '>', $filename) or die "Could not open file '$filename': $!";
print $fh
"timeStamp,${headerPrefix}_per_sec,${headerPrefix}_bytes_written,${headerPrefix}_bytes_read,"
.
"${headerPrefix}_time_system,${headerPrefix}_time_user,config,host\n";
print $fh
"$timestamp,$cmd_per_sec,$bytes_written,$bytes_read,$time_system,$time_user,$config,$host\n";
close $fh;
print "done\n";
==
Basically it executes mc-crusher with different configurations against
different servers (x64 vs arm64) and stores the stats in CSV files, which
later I visualize in Kibana.
So far I've ran with these configs:
cmd_get.conf:
send=ascii_get,recv=blind_read,conns=50,key_prefix=foobar,key_prealloc=0,pipelines=8
send=ascii_set,recv=blind_read,conns=10,key_prefix=foobar,key_prealloc=0,pipelines=4,stop_after=20,usleep=1000,value_size=10
cmd_set.conf:
send=ascii_set,recv=blind_read,conns=5,key_prefix=foobar,value_size=2,value=hi,key_prealloc=1
I always run those against freshly started Memcached server (started with:
./memcached -p 8080).
After mc-crusher finishes I get the value of the current command, e.g.
cmd_get, and divide it by
the duration, to get the cmds per second.
And here are the numbers I have so far:
gets/s:
- x64: ~450 000
- arm64: ~310 000
sets/s:
- x64: ~25 000
- arm64: ~63 000
I.e. x64 is faster for GET and arm64 is faster for SET.
Those numbers are averages from multiple runs.
Do you see anything wrong in my Perl script ? Or any ideas how to improve
it.
Any hints how to profile the server to see why/where it is slower on
different platforms ?
1. https://groups.google.com/forum/#!topic/memcached/8hT2BT9cgEc
Regards,
Martin
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to memcached+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/memcached/CAMomwMrES-%2BsTxYQrK9rQNaNCPRjA2FZUOYKR_rSgRUCiz5TiA%40mail.gmail.com.
