Re: Collection API for performance monitoring?

.org> Date: Tuesday, November 15, 2016 at 8:31 AM To: "dev@lucene.apache.org" <dev@lucene.apache.org> Subject: Re: Collection API for performance monitoring? To calculate percentiles we need all the data points. If there is a lot of data, it could be sampled. Average can be calcula

Re: Collection API for performance monitoring?

If you only need query/update performance you could aggregate the logs too. If you need more information, I like what was proposed in SOLR-9641, that would allow you do collect and aggregate metrics for internal components too. Tomás On Tue, Nov 15, 2016 at 8:31 AM, Walter Underwood

Re: Collection API for performance monitoring?

To calculate percentiles we need all the data points. If there is a lot of data, it could be sampled. Average can be calculated with the total time and the number of requests. Snapshots of those two values allow snapshots of averages. But averages are the wrong metric for a one-sided

Re: Collection API for performance monitoring?

I haven't tried for 95th percentile, but generally with those collection start stats you would monitor based on calculated deltas. You can figure out the average response time for any given window of time not smaller than your snapshot polling interval. I don't see why 95th percentile would be

Re: Collection API for performance monitoring?

Because the current stats are not usable. They really should be removed from the code. They calculate percentiles since the last collection load. We need to know 95th percentile during the peak hour last night, not the 95th for the last month. Right now, we run eleven collections in our Solr 4

Re: Collection API for performance monitoring?

Point taken, and thanks for the link. The stats I'm referring to in this thread are available now, and would (I think) be a quick win. I don't have a huge amount of investment in it though, more "why didn't we think of this before?" followed by "maybe there's a very good reason not to bother".

Re: Collection API for performance monitoring?

I’m not fond of polling for performance stats. I’d rather have the app report them. We could integrate existing Jetty monitoring: http://metrics.dropwizard.io/3.1.0/manual/jetty/ From our experience with a similar approach, we might need some

Collection API for performance monitoring?

What do people think about exposing a Collections API call (name TBD, but the sense is PERFORMANCESTATS) that would simply issue the admin/mbeans call to each replica of a collection and report them back. This would give operations monitors the ability to see, say, anomalous replicas that had poor