I'd like to raise an issue that I'm having with Solr 9.3 that might make a good case for getting another release (9.3.1 or 9.4) out soon.
I've had a 9.3 cluster of 180 nodes up and running for about a week. After the weekend, I noticed that metrics from the cluster were failing to ship. Upon investigation, I found that the payload coming back from `/solr/admin/metrics` was over a GB in size. I also saw query responses had ballooned in size to tens of MB and the UI was slow. I captured query responses in the browser, with curl and in a packet capture. In all cases, there were a large amount of space characters. This can be seen in the brower's network inspector when clicking around the UI, as well (make sure to disable pretty print). This all lead me to SOLR-16925. I have a fix in place for query results but the admin endpoints like `/solr/admin/metrics` are still experiencing the issue. You can simply pass `&indent=off` to the endpoints and the whitespace issue goes away. However, I can't figure out a way to configure Solr to use this default param with the CoreAdminHandler. The Solr Prom Exporter could be reconfigured to include this param but that's a lot of configuration to add. There's no way to reconfigure the UI. Here's a good illustration of the issue: ``` ❯ curl --user 'foo:bar' "http://localhost:8983/solr/admin/metrics"; > /tmp/metrics_indented.json ❯ curl --user 'foo:bar' "http://localhost:8983/solr/admin/metrics?indent=off"; > /tmp/metrics_not_indented.json ❯ ls -lh /tmp | grep metrics -rw-rw-r-- 1 foo bar 1.4G Sep 28 08:57 metrics_indented.json -rw-rw-r-- 1 foo bar 177K Sep 28 08:59 metrics_not_indented.json ❯ grep -o ' ' /tmp/metrics_indented.json | wc -l 1449499825 ``` In my opinion, Solr 9.3 can't be monitored and is therefore unfit to run. I'll be AFK for the next few days but am happy to lend a hand with a new release next week. I'd need a lot of help, though. Thanks, Chris
