All, This is a great idea and I am looking forward to it.
Having dedicated consistent hardware is a good way to find regressions in the code but orthogonal to that is "certifying" new hardware to run with Cassandra, e.g. is there a performance regression when running on AMD? ARM64? What about more RAM? faster SSD? What has limited us in perf testing in the past was some "representative" benchmark with clear recommendations so I am hoping that this work will produce a reference test suite with at least some hardware recommendation for the machine running the tests to make things more comparable. Additionally, some perf tests keep increasing the load until latency hits a certain threshold and others do some operations and measure how long it took. What types of tests where you aiming for? The proposal also doesn't talk much about the test matrix. Will all supported Cassandra versions be tested with the same tests or will there be version specific tests? I understand that we need to account for variances in configuration hardware but I am wondering if we can have more than just the sha. For example the complete cassandra.yaml for a test should be checked in as well - also we shoudl encourage people not to change too much from the reference test. Different hardware, different cassandra.yaml, and different tests will just create numbers which are hard to make sense of. Really excited about this - thanks for the great work, German ________________________________ From: Josh McKenzie <jmckenzll,i...@apache.org> Sent: Friday, December 30, 2022 7:41 AM To: dev <dev@cassandra.apache.org> Subject: [EXTERNAL] [DISCUSS] Taking another(other(other)) stab at performance testing There was a really interesting presentation from the Lucene folks at ApacheCon about how they're doing perf regression testing. That combined with some recent contributors wanting to get involved on some performance work and not having much direction or clarity on how to get involved led some of us to come together and riff on what we might be able to take away from that presentation and context. Lucene presentation: "Learning from 11+ years of Apache Lucene benchmarks": https://docs.google.com/presentation/d/1Tix2g7W5YoSFK8jRNULxOtqGQTdwQH3dpuBf4Kp4ouY/edit#slide=id.p<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fpresentation%2Fd%2F1Tix2g7W5YoSFK8jRNULxOtqGQTdwQH3dpuBf4Kp4ouY%2Fedit%23slide%3Did.p&data=05%7C01%7CGerman.Eichberger%40microsoft.com%7C53bc172f6ff44b7b0f7008daea7c5724%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638080117008027125%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ghifEezYq4XqP%2FDG4lm8ztUD41Ud%2Fzn3%2BC7M%2FDaUmYE%3D&reserved=0> Their nightly indexing benchmark site: https://home.apache.org/~mikemccand/lucenebench/indexing.html<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhome.apache.org%2F~mikemccand%2Flucenebench%2Findexing.html&data=05%7C01%7CGerman.Eichberger%40microsoft.com%7C53bc172f6ff44b7b0f7008daea7c5724%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638080117008027125%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Cs5A2UaMEI6pPC0AUEkNqBsm7LDMiK%2FzF0fENFgIzm4%3D&reserved=0> I've checked in with a handful of performance minded contributors in early December and we came up with a first draft, then some others of us met on an adhoc call on the 12/9 (which was recorded; ping on this thread if you'd like that linked - I believe Joey Lynch has that). Here's where we landed after the discussions earlier this month (1st page, estimated reading time 5 minutes): https://docs.google.com/document/d/1X5C0dQdl6-oGRr9mXVPwAJTPjkS8lyt2Iz3hWTI4yIk/edit#<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1X5C0dQdl6-oGRr9mXVPwAJTPjkS8lyt2Iz3hWTI4yIk%2Fedit%23&data=05%7C01%7CGerman.Eichberger%40microsoft.com%7C53bc172f6ff44b7b0f7008daea7c5724%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638080117008027125%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=L2kbzpDaVa8TOKpo7PmSyuTxYWzPwtjNb7zAOGN%2BflQ%3D&reserved=0> Curious to hear what other perspectives there are out there on the topic. Early Happy New Years everyone! ~Josh
