Yo, Fwiw, I use mc-crusher as the "official" benchmark: https://github.com/memcached/mc-crusher tho I need to update the README slightly. will do that in a bit.
I also test on hardware uninterrupted by other things with turbo disabled :) testing on laptops can be really tough since you'll get different frequencies minute to minute. You have to interleave test runs A/B/A/B/A/B then average to get through the noise. Also, make sure to test both binprot/ascii. with ascii multiget you can deeply pipeline requests and get more hashing vs syscalls. Also, test short vs long keys. a for loop and some scripting should get you there. :) I don't really want to add a ton of algo's. More or less a "best average" is fine. Given the syscall overhead, hashing is barely measurable for a lot of workloads. When I switched from jenkins to murmur I did a somewhat comprehensive set of tests then swapped default + left old one in just in case. I highly value good defaults! Libmemcached ended up kitchen-sinking hash algo's and I think that didn't work out well in the long run. I did also test hash chain bucket depth a bit. Finally, loading up different counts of keys (1m, 10m, 100m, etc) and re-running uniform random benchmarks since fairness will affect the bucket depth and thus latency. Sorry if that's a pain in the ass, but this thing is quite widely used and there aren't really beta testers :) Have to be thorough these days. -Dormando On Sun, 17 Mar 2019, 'Eamonn Nugent' via memcached wrote: > Reporting back with very preliminary benchmarks. Somehow, xxh64 is actually > faster than xxh3 on my machine. One thing I forgot to > mention before - I also looked at latencies with xxh32/64, and saw the 99th > percentile latency lowered by about half compared to mmh3. > So it could be beneficial in that sense. Latencies with xxh3 are in the 3.6ms > 99% range, xxh64 go down to about 3.0 (I saw 2.5 > yesterday, maybe testing on a laptop with about a billion chrome tabs open > isn't a brilliant idea), and mmh3 were in the 4.xms range. > All of this with modern options, but with non-modern, xxh64 shone quite a > bit. I was doing my testing there yesterday. > I used the following memtier_benchmark command to stress test: > > ./memtier_benchmark -P memcache_binary -p 11211 --key-prefix="" > --key-maximum=9999999999999999999 > > A lot of this seems to be very architecture dependent. Maybe it would make > sense to include a lot of hash algos long term, and let power > users figure out which they feel like using? Not sure, though, and you're the > expert here :P > > Thanks, > > Eamonn > > On Sun, Mar 17, 2019 at 2:59 PM Eamonn Nugent <eamonn.nug...@demilletech.net> > wrote: > Hiya, > Last night, I was running memtier_benchmark on my laptop (mid-2015 15" MBP, > 2.5GHz 4c i7) and achieved about a 10-15% throughput > improvement on both modern and non-modern settings on the 64 bit variant. 32 > bit variant was about equal in performance (the > results showed them to be within about 3% of each other, but most of the > difference was probably just typical entropy). I was able > to solve the 32/64 bit compile time problem by adding in a wrapper and some > compile-time declarations, so I'd say that's about 50% > solved for x86-based systems. But yeah, with ARM, it could turn interesting. > > As a next-ish step, I'm going to attempt to drop in xxh3, but since it's > still in active development, it's probably not good as > anything more than a tech demo. I'm happy, if it would help, just to go nuts > adding a dozen different algos into hash.c, though > (cityhash/farmhash, as you mentioned). In xxhash's implementation, though, I > played with some compile-time flags to make it a bit > faster, and I've been toying with the idea of modifying it so no seed logic > ever occurs, to maybe gain a couple cycles of speed > increase. I'm also looking into seeing if I can find a pure assembly version > to squeeze a bit more speed out of x86 and ARM > versions. I should probably get one of my ARM systems running and test the > difference... > > But hey, thanks for humoring me. Maybe next I'll take a look at the reading & > processing command steps, and see if there's > anything I can do. Or maybe parallelizing rotl... Hm. I'll keep on with > trying it out :) > > Thanks, > > Eamonn > > > On Sun, Mar 17, 2019 at 2:46 PM dormando <dorma...@rydia.net> wrote: > Hey, > > What exact test did you do? > > Well to be honest I've been wanting to swap in xxhash for a long time, > but > in my own profiling other things show up higher than murmur so I keep > deprioritizing it :) > > One big problem with the hash algo is mc keys can be short and are > hashed one at a time. xxhash is more optimized for longer data > (kilobytes > to megabytes). The original author tries to address this with an updated > algorithm: > https://fastcompression.blogspot.com/2019/03/presenting-xxh3.html > > xxhash makes significant use of instruction parallelism, such that if a > key is 8 bytes or less you could end up waiting for the pipeline more > than murmur. Other algos like cityhash/farmhash are better at short keys > IIRC. Also xx's 32bit algo is a bit slower on 64bit machines... so if I > wanted to use it I was going to test both 32bit and 64bit hashes and > then > have to do compile time testing to figure out which to use. It's also > heavily x86 optimized so we might have to default something else for > ARM. > > Sorry, not debated on the list, just in my own head :) It's not quite as > straightforward as just dropping it in. If you're willing to get all the > conditions tested go nuts! :) > > -Dormando > > On Sat, 16 Mar 2019, eamonn.nugent via memcached wrote: > > > Hi there, > > I started using memcached in prod a week or two ago, and am loving > it. I wanted to give back, and took a look > through the issues board, > > but most of them looked solved. So, in my usual "it's never fast > enough" style, I went and profiled its performance, > and had some fun. > > > > After seeing that MurmurHash3 was taking a good amount of the > execution time, I decided to run a test integrating > one of my old favorite > > hash functions, xxhash. My guess is that Memcached could benefit from > using the hash function, as it is faster than > MMH3 and has several > > native variants. I ran some of my own tests, and found roughly equal > performance, but with no tuning performed on > xxhash. For example, > > using an assembly (x86/arm/etc) version could likely speed up > hashing, along with properly implementing it in > memcached. However, I was > > also running this on a much older Nehalem CPU, so there could be > unseen advantages to one or both of the algorithms > by running them on a > > newer CPU. I'm in the process of fighting with my newer systems to > get libevent installed properly, so I'll report > back with more > > up-to-date tests later. > > > > I did a cursory search, but didn't find any mention of the algo in > the mailing list. If this has been discussed, > though, apologies for > > bringing it up again. On the other hand, I would be happy to write a > PR to add it, using the `hash_algorithm` CLI > arg. > > > > Thanks, > > Eamonn > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, > send an email to > memcached+unsubscr...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "memcached" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/memcached/Y02zPF-WTKg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > memcached+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to memcached+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
