I’m about ready to commit the first iteration of the Solr benchmark module.
It is meant to target both micro and macro benchmarks, though it is additive to, not a replacement for, Gatling and a full performance cluster. The inner workings of Solr and SolrCloud have always been something of a mystery to me. Benchmarking has been as well. Not that I ever spent any time thinking clearly about that. If I had, I wouldn’t have had an alternative plan to rectify it. And it didn’t matter. It didn’t affect me getting work. It didn’t affect my bonus from the boss. Over the past few years I did start to learn something about these mysteries though. Not with a genius plan of attack. Not with a strategy I can write down on the wiki and successfully share with you. I did it by attacking everything in sight. And then improving my sight. If some genius computer God once said “don’t do this”, I did and found out why not. If something looked huge effort for unlikely reasonable return, I did it. And maybe scrapped it. If something took literally 16 hours just to manually process the code changes with 0 thought the whole time and repetitive pain and loud expletives accompanying the final hours, I did it. And sometimes maybe scrapped it later. If there was a rabbit hole, I went down it. I used the tests to chase features and code and surface area I’d never have touched or even known existed. I spent hundreds of hours or more building tools and hundreds more coopting existing tools to expand my grasp and view. I went after other code bases with a similar attack and less depth to raise my vantage point. And I could go on, except that illustrates my point and there is little value in doing so. So I learned a couple things on that journey. And I found an answer or two. Formed an opinion or three. And I’ve had to think. Think about how I can turn that into some value for Apache Solr. I chose to do that work, but I was also paid during that time. Paid for work that is supposed to end up returning value. The basic employer / employee contract. I will never march down that path again. The destination was never really the point. No sane developer would or could join the full trip. I have to use that journey to plot a new one. Thought one: there was huge value in playing around with the system. Trying a wide range of things simply. Getting valuable and low effort feedback and introspection easily. Thought two: I did not play around or explore much before, or see it done, because it was high effort to explore even a small surface area. Even more effort to properly vet or ensure getting quality results or information from it. Thought three: Continuing on thought two, setting up good experiments is very difficult. Collecting results and evaluating the quality of those results is very difficult. More difficult than many developers who would immediately agree with those statements even know. In the way that Elon Musk knew fully self driving cars would be difficult. But he didn’t know it would be “that” difficult. Of course a smaller percentage of developers do know the extent of it. Thought four: When the above was even attempted, it was generally by developers working in isolation. And climbing on their own scaffolding that was not peer reviewed and either tossed out, abandoned, reconstructed, or maybe eventually reused by one. Thought five: Building something that allows for exploration and experimentation essentially always reduces to some kind of benchmark type framework. And benchmarks are notoriously and ridiculously difficult. See above. Any project that wants to truly benefit from them needs to work on them together. And retain them. And improve them. And retain and improve the knowledge behind them. And so we come to the Solr benchmark module. I’ve poured some of my knowledge and experience into standing up an initial framework. I will document it. I will share a video explaining some of the what and why and how. And I will make it so easy to join in on that the only reason a developer will not join the effort is because they have no interest in understanding or improving the system and their changes. So I will make a commit next week. And then I will continue to move it forward. I encourage you to take a look and evaluate what return for what effort you might get from joining in. MRM -- - Mark http://about.me/markrmiller
