Yes, it’s a new Gradle module called benchmark. I’ll likely commit the base early tomorrow. It’s been working through pre commit checks.
There are currently only two benchmarks to start, but I have more that I’ll be adding. Once I have a reasonable number in, I’ll run some comparisons with the 8x branch. Eventually, I’ll do some comparisons with the ref branch as well, but probably not that soon. I also have a subproject for the module that builds an Apache Zeppelin interpreter, which allows creating a notebook of parametrized benchmarks which can be versioned and allows for organizing various run plans, charting, and various other things. You can turn on a variety of profilers via command line, gc, jfr, and the async profiler being the most common I’ve used. I’ve been using it on both relatively small runs as well as runs against up to 150 GB of index. I’ve also used it in docker and will add more support I have to make that simple option. My main motivation there being to be able to control and vary hardware resources. Mark On Sun, Aug 1, 2021 at 2:01 PM Jason Gerlowski <gerlowsk...@gmail.com> wrote: > Just clarifyng, but the "Solr Benchmark Module" you're referring to > here is your work from SOLR-15428? Or something else? > > Jason > > On Sun, Aug 1, 2021 at 12:16 AM Mark Miller <markrmil...@gmail.com> wrote: > > > > I’m about ready to commit the first iteration of the Solr benchmark > module. > > > > It is meant to target both micro and macro benchmarks, though it is > additive to, not a replacement for, Gatling and a full performance cluster. > > > > The inner workings of Solr and SolrCloud have always been something of a > mystery to me. Benchmarking has been as well. Not that I ever spent any > time thinking clearly about that. > > > > If I had, I wouldn’t have had an alternative plan to rectify it. And it > didn’t matter. It didn’t affect me getting work. It didn’t affect my bonus > from the boss. > > > > Over the past few years I did start to learn something about these > mysteries though. Not with a genius plan of attack. Not with a strategy I > can write down on the wiki and successfully share with you. I did it by > attacking everything in sight. And then improving my sight. > > > > If some genius computer God once said “don’t do this”, I did and found > out why not. If something looked huge effort for unlikely reasonable > return, I did it. And maybe scrapped it. If something took literally 16 > hours just to manually process the code changes with 0 thought the whole > time and repetitive pain and loud expletives accompanying the final hours, > I did it. And sometimes maybe scrapped it later. If there was a rabbit > hole, I went down it. > > > > I used the tests to chase features and code and surface area I’d never > have touched or even known existed. I spent hundreds of hours or more > building tools and hundreds more coopting existing tools to expand my grasp > and view. I went after other code bases with a similar attack and less > depth to raise my vantage point. > > > > And I could go on, except that illustrates my point and there is little > value in doing so. > > > > So I learned a couple things on that journey. And I found an answer or > two. Formed an opinion or three. And I’ve had to think. Think about how I > can turn that into some value for Apache Solr. I chose to do that work, but > I was also paid during that time. Paid for work that is supposed to end up > returning value. The basic employer / employee contract. > > > > I will never march down that path again. The destination was never > really the point. No sane developer would or could join the full trip. > > > > I have to use that journey to plot a new one. > > > > Thought one: there was huge value in playing around with the system. > Trying a wide range of things simply. Getting valuable and low effort > feedback and introspection easily. > > > > Thought two: I did not play around or explore much before, or see it > done, because it was high effort to explore even a small surface area. Even > more effort to properly vet or ensure getting quality results or > information from it. > > > > Thought three: Continuing on thought two, setting up good experiments is > very difficult. Collecting results and evaluating the quality of those > results is very difficult. More difficult than many developers who would > immediately agree with those statements even know. In the way that Elon > Musk knew fully self driving cars would be difficult. But he didn’t know it > would be “that” difficult. Of course a smaller percentage of developers do > know the extent of it. > > > > Thought four: When the above was even attempted, it was generally by > developers working in isolation. And climbing on their own scaffolding that > was not peer reviewed and either tossed out, abandoned, reconstructed, or > maybe eventually reused by one. > > > > Thought five: Building something that allows for exploration and > experimentation essentially always reduces to some kind of benchmark type > framework. And benchmarks are notoriously and ridiculously difficult. See > above. Any project that wants to truly benefit from them needs to work on > them together. And retain them. And improve them. And retain and improve > the knowledge behind them. > > > > And so we come to the Solr benchmark module. I’ve poured some of my > knowledge and experience into standing up an initial framework. I will > document it. I will share a video explaining some of the what and why and > how. And I will make it so easy to join in on that the only reason a > developer will not join the effort is because they have no interest in > understanding or improving the system and their changes. > > > > So I will make a commit next week. And then I will continue to move it > forward. I encourage you to take a look and evaluate what return for what > effort you might get from joining in. > > > > MRM > > > > > > -- > > - Mark > > > > http://about.me/markrmiller > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > For additional commands, e-mail: dev-h...@solr.apache.org > > -- - Mark http://about.me/markrmiller
