> more things in reference test suite... increasing the load until latency > hits... operations and measures... test matrix... checking in complete > cassandra.yaml... different hardware... different tests... All great things. For v2+. :)
Perf testing is a deep, deep rabbit hole. What's tripped us up in the past has (IMO) predominantly been due to us biting off more than we could chew to consensus. I immediately agree at face value with most of the things you've asked about in your reply but I think we'll need to build up to that and/or include some of that in the "community benchmarks" rather than "reference benchmarks" as outlined in the doc. ~Josh On Tue, Jan 3, 2023, at 12:57 PM, German Eichberger via dev wrote: > All, > > This is a great idea and I am looking forward to it. > > Having dedicated consistent hardware is a good way to find regressions in > the code but orthogonal to that is "certifying" new hardware to run with > Cassandra, e.g. is there a performance regression when running on AMD? ARM64? > What about more RAM? faster SSD? > > What has limited us in perf testing in the past was some "representative" > benchmark with clear recommendations so I am hoping that this work will > produce a reference test suite with at least some hardware recommendation for > the machine running the tests to make things more comparable. Additionally, > some perf tests keep increasing the load until latency hits a certain > threshold and others do some operations and measure how long it took. What > types of tests where you aiming for? > > The proposal also doesn't talk much about the test matrix. Will all supported > Cassandra versions be tested with the same tests or will there be version > specific tests? > > I understand that we need to account for variances in configuration hardware > but I am wondering if we can have more than just the sha. For example the > complete cassandra.yaml for a test should be checked in as well - also we > shoudl encourage people not to change too much from the reference test. > Different hardware, different cassandra.yaml, and different tests will just > create numbers which are hard to make sense of. > > Really excited about this - thanks for the great work, > German > > > > *From:* Josh McKenzie <jmckenzll,i...@apache.org> > *Sent:* Friday, December 30, 2022 7:41 AM > *To:* dev <dev@cassandra.apache.org> > *Subject:* [EXTERNAL] [DISCUSS] Taking another(other(other)) stab at > performance testing > > There was a really interesting presentation from the Lucene folks at > ApacheCon about how they're doing perf regression testing. That combined with > some recent contributors wanting to get involved on some performance work and > not having much direction or clarity on how to get involved led some of us to > come together and riff on what we might be able to take away from that > presentation and context. > > Lucene presentation: "Learning from 11+ years of Apache Lucene benchmarks": > https://docs.google.com/presentation/d/1Tix2g7W5YoSFK8jRNULxOtqGQTdwQH3dpuBf4Kp4ouY/edit#slide=id.p > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fpresentation%2Fd%2F1Tix2g7W5YoSFK8jRNULxOtqGQTdwQH3dpuBf4Kp4ouY%2Fedit%23slide%3Did.p&data=05%7C01%7CGerman.Eichberger%40microsoft.com%7C53bc172f6ff44b7b0f7008daea7c5724%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638080117008027125%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ghifEezYq4XqP%2FDG4lm8ztUD41Ud%2Fzn3%2BC7M%2FDaUmYE%3D&reserved=0> > > Their nightly indexing benchmark site: > https://home.apache.org/~mikemccand/lucenebench/indexing.html > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~mikemccand%2Flucenebench%2Findexing.html&data=05%7C01%7CGerman.Eichberger%40microsoft.com%7C53bc172f6ff44b7b0f7008daea7c5724%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638080117008027125%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Cs5A2UaMEI6pPC0AUEkNqBsm7LDMiK%2FzF0fENFgIzm4%3D&reserved=0> > > I've checked in with a handful of performance minded contributors in early > December and we came up with a first draft, then some others of us met on an > adhoc call on the 12/9 (which was recorded; ping on this thread if you'd like > that linked - I believe Joey Lynch has that). > > Here's where we landed after the discussions earlier this month (1st page, > estimated reading time 5 minutes): > https://docs.google.com/document/d/1X5C0dQdl6-oGRr9mXVPwAJTPjkS8lyt2Iz3hWTI4yIk/edit# > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1X5C0dQdl6-oGRr9mXVPwAJTPjkS8lyt2Iz3hWTI4yIk%2Fedit%23&data=05%7C01%7CGerman.Eichberger%40microsoft.com%7C53bc172f6ff44b7b0f7008daea7c5724%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638080117008027125%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=L2kbzpDaVa8TOKpo7PmSyuTxYWzPwtjNb7zAOGN%2BflQ%3D&reserved=0> > > Curious to hear what other perspectives there are out there on the topic. > > Early Happy New Years everyone! > > ~Josh >
