subject:"\[jira\] \[Commented\] \(SOLR\-1972\) Need additional query stats in admin interface \- median, 95th and 99th percentile"

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539544#comment-13539544
]

Shawn Heisey commented on SOLR-1972:

This is really useful to me, I'm very sad to learn it's causing problems. I
haven't noticed anything in my own running of branch_4x, which runs for days at
a time. I'm attempting to get some heap dumps so I can compare them. Based on
what I know from my own experience, I don't think this is actually a leak, but
even a minimal test JVM ends up with a lot of request handlers, each of which
has these additional objects. As for seeing what looks like a leak with each
query, after 1024 queries, the growth in non-garbage objects should stop,
because it throws away an entry to make room for the new one.

I wonder if there might be an easy way to make these new statistics optional in
the request handler, so they do not cause memory problems with minimal test
configs. Although it's really cool to be able to see detailed query statistics
on the SolrInfoMBeanHandler, it's unnecessary.

If it does get removed, I guess I'll have to go back to the previous version of
the patch that only has additional statistics on qtime. I depend on these
statistics - the average values otherwise available are nearly useless.

I will attempt a patch. No guarantees about success!

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch,
elyograg-1972-trunk.patch, elyograg-1972-trunk.patch,
SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch,
SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch,
SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-12-26 Thread Robert Muir (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539546#comment-13539546
]

Robert Muir commented on SOLR-1972:
---

{quote}
This commit causes a huge memory leak in Solr
...
We should revert this and investigate how to remove the com.yammer.metrics
package dependency (or make the stats cleaner). To me it looks like every query
to solr creates a new entry in those huge maps, causing them to grow and grow
and grow...
{quote}

Maybe the stats have to be implemented by hand or something. I don't understand
why huge maps or string interning is necessary here.

Lets back out the change and implement it in a non-leaky way.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-12-26 Thread Mark Miller (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539553#comment-13539553
 ] 

Mark Miller commented on SOLR-1972:
---

bq. I would revert this patch for now

+1

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1, 5.0

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-12-26 Thread Uwe Schindler (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539552#comment-13539552
 ] 

Uwe Schindler commented on SOLR-1972:
-

bq. As for seeing what looks like a leak with each query, after 1024 queries, 
the growth in non-garbage objects should stop, because it throws away an entry 
to make room for the new one.

That might be correct, the problem here is a leak, that might also affect 
people reloading Solr in their App-Server. The problem is that somehow on 
CoreShutdown the MBeans are not destroyed. Maybe this is fine how it is 
implemented, but when the Solr core shuts down, all statistics data have to be 
removed and nulled out or whatever.

I would revert this patch for now, until there is a better solution. Once you 
have a new patch, that does not leak those objects on core shutdowns.

The horrible thing is that all those huge maps contains stupid strings, all 
starting with some solr class name (a RequestHandler), followed by an id and 
various other information. There are (in the dump analyzed with visualvm) 
millions of String looking like that (OQL executed on heap dump):

{code:sql}
select count(heap.objects(java.lang.String), 
'it.toString().startsWith(org.apache.solr.handler)')
{code}

This looks like a huge waste!

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1, 5.0

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-12-26 Thread Robert Muir (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539554#comment-13539554
 ] 

Robert Muir commented on SOLR-1972:
---

I wonder if the permgen leak is related to the thread leaks caused by this 
metrics package (see SolrIgnoredThreadsFilter.class, it ignores threads leakd 
by the metrics package).

In general we should try to prune this list and keep it minimal... at the 
moment i'm pretty suspicious of this third party library though: leaking 
threads, interning strings, etc.

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1, 5.0

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-12-26 Thread Uwe Schindler (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539558#comment-13539558
 ] 

Uwe Schindler commented on SOLR-1972:
-

[~romseygeek] shood revert, if he is not available for the next days, I will do 
it. OK? (I dont want to be again the bad guy reverting other's commits)

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1, 5.0

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539575#comment-13539575
]

Shawn Heisey commented on SOLR-1972:

Robert, Alan did try creating a registry for each handler. It caused OOM errors
when running Solr tests. See comments on 2012/10/26.

Perhaps we could create a single static registry object that gets used by every
handler to create the individual objects. I'll use your patch and attempt that.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch,
elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, leak.patch,
SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch,
SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch,
SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-12-26 Thread Robert Muir (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539576#comment-13539576
]

Robert Muir commented on SOLR-1972:
---

I dont think we should have a single static: its no better than whats happening
right now! its the problem!

But for now, again I feel the right solution is to back out the change. there
are too many resource leaks here.

Besides the permgen leak, this thing leaks threads like a sieve. and this is
currently just being ignored.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-12-26 Thread Uwe Schindler (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539586#comment-13539586
 ] 

Uwe Schindler commented on SOLR-1972:
-

MetricsRegistry *is* the singleton: With VisualVM you can get the retained size 
(Retained size of an object is its shallow size plus the shallow sizes of the 
objects that are accessible, directly or indirectly, only from this object. In 
other words, the retained size represents the amount of memory that will be 
freed by the garbage collector when this object is collected.) - o holy 
x-mas! One instance hold references to all metrics ever created. This is the 
leak, together with the threads spawned in its lifetime.

Whenever you reload yur Solr instance you leak more objects (which is *the* 
classic classloader leak).



 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1, 5.0

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, leak.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539623#comment-13539623
 ] 

Shawn Heisey commented on SOLR-1972:


I've been trying to find evidence of the leak, and can't.  I completely trust 
that you guys do know what you're talking about, so this is probably just my 
inexperience.

What I did find:

Before any additional patching, looking at jconsole, I have two metrics 
threads.  Looking at MBeans in jconsole, there are 290 metric scopes under 
RequestHandlerBase with names from metrics-scope-0 to metrics-scope-289.

Taking Robert's patch and fixing it to create one static MetricsRegistry, the 
additional objects under MBeans disappear, and the number of threads named 
metric-* goes from 2 to 1.  Through some additional debugs, I have files 
containing stacktraces from every time RequestHandlerBase is called, so I know 
it is still being called 290 times during startup.

Examining those stacktraces, I have determined that each of my 16 cores has 18 
request handlers in common, with two additional server-wide handlers - 
CollectionsHandler and CoreAdminHandler - for a total of 290.  I'm attaching an 
archive with all the stacktraces and a summary - the third line from each 
stacktrace, sorted.

If we made the metrics package optional on a per-handler basis, I would drop 
from 290 scopes to 48 - each of my 16 cores has four search handlers, but I 
really only need full metrics in three of them.


 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1, 5.0

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, leak.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539628#comment-13539628
 ] 

Shawn Heisey commented on SOLR-1972:


Some things just clicked and I think I might have figured it out, tell me if I 
get this wrong.  The metrics objects get created during initialization, but 
then if you reload a core (or possibly if you reload the entire app in your 
container) then the old metrics objects are never garbage collected because the 
registry object holds onto them forever.

I do see that the registry has the ability to remove metrics.  To make this 
possible, RequestHandlerBase would probably need a close() method, and it would 
need to be properly called when required.  I am assuming this is not a trivial 
undertaking?


 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1, 5.0

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, leak.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch, stacktraces.tar.gz


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-12-26 Thread Lance Norskog (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539779#comment-13539779
 ] 

Lance Norskog commented on SOLR-1972:
-

The OnlineSummary class in Mahout does the calculations you want. One little 
class you can steal. No dependencies necessary.

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1, 5.0

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, leak-closeable.patch, 
 leak.patch, SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch, stacktraces.tar.gz


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-12-06 Thread Alan Woodward (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511261#comment-13511261
 ] 

Alan Woodward commented on SOLR-1972:
-

Hi Mark, I've marked this as fixed for 5.0 as well - was it just the tags that 
made you think it was only on 4x or is it giving you trouble merging?

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1, 5.0

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-12-06 Thread Mark Miller (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511405#comment-13511405
 ] 

Mark Miller commented on SOLR-1972:
---

Yeah, I had a merge conflict in changes and didn't see the entry in 5x changes. 

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1, 5.0

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-12-05 Thread Mark Miller (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511106#comment-13511106
 ] 

Mark Miller commented on SOLR-1972:
---

Hey guys, any reason this is not on 5x? That's going to make it difficult in 
the future...

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-12-05 Thread Shawn Heisey (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511149#comment-13511149
]

Shawn Heisey commented on SOLR-1972:

bq. Hey guys, any reason this is not on 5x? That's going to make it difficult
in the future...

It looks like r1403555 added this to trunk on 2012/10/29.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-11-26 Thread Commit Tag Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504138#comment-13504138
 ] 

Commit Tag Bot commented on SOLR-1972:
--

[branch_4x commit] Alan Woodward
http://svn.apache.org/viewvc?view=revisionrevision=1413797

SOLR-1972: Add extra query stats to RequestHandler



 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-11-08 Thread Shawn Heisey (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13493476#comment-13493476
 ] 

Shawn Heisey commented on SOLR-1972:


Now that metrics is in trunk, there are a zillion potential uses for it for 
improving visibility into Solr performance.  Would it be better to have one big 
Jira for adding these (or for depending on additional issues) or would it be 
better to just go ahead and create an individual issue each time I think of 
one?  At this moment, I only have one candidate, but I'm sure I'll think of 
more later.


 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-11-08 Thread Adrien Grand (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13493574#comment-13493574
 ] 

Adrien Grand commented on SOLR-1972:


Hi Shawn, I personally prefer small issues. They make it easier to get stuff 
committed.

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-29 Thread Adrien Grand (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486007#comment-13486007
 ] 

Adrien Grand commented on SOLR-1972:


+1

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-29 Thread Lance Norskog (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486272#comment-13486272
 ] 

Lance Norskog commented on SOLR-1972:
-

The 5th percentile is really useful. There is always a maximum query time of 
30s just because of a garbage collection failure, and people look at that 
number and freak out. For query times, the 5th percentile shows what is 
repeatedly too slow. 

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-29 Thread Alan Woodward (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486280#comment-13486280
 ] 

Alan Woodward commented on SOLR-1972:
-

Hi Lance, in this case, that's represented by the 95th percentile - lower 
numbers mean faster query times.

I'll commit this later this evening if no-one objects.

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-29 Thread Alan Woodward (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486423#comment-13486423
 ] 

Alan Woodward commented on SOLR-1972:
-

Committed to trunk, r1403555.  I'll let it sit here for a bit before 
back-porting.

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 solr1972-metricsregistry-branch4x-failure.log, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-28 Thread Adrien Grand (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485613#comment-13485613
 ] 

Adrien Grand commented on SOLR-1972:


In that case could we use something else than the string representation as a 
scope? (like a randomly-generated string that is unique across 
RequestHandlerBase instances?)

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, solr1972-metricsregistry-branch4x-failure.log, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-28 Thread Alan Woodward (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485639#comment-13485639
 ] 

Alan Woodward commented on SOLR-1972:
-

The toString() is sort of a randomly-generated string, really.  It's the 
default Object toString, ie a class name and a memory address.

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, solr1972-metricsregistry-branch4x-failure.log, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-28 Thread Adrien Grand (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485641#comment-13485641
 ] 

Adrien Grand commented on SOLR-1972:


Can it break if people write custom request handlers that override toString?

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, solr1972-metricsregistry-branch4x-failure.log, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-28 Thread Alan Woodward (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485643#comment-13485643
 ] 

Alan Woodward commented on SOLR-1972:
-

I suppose if we had two custom request handlers that both overrode it with the 
same String it could break.

How about using super.toString() instead of this.toString()?  That way it's 
always the parent Object method that's called.

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, solr1972-metricsregistry-branch4x-failure.log, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-28 Thread Adrien Grand (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485645#comment-13485645
 ] 

Adrien Grand commented on SOLR-1972:


After reading the documentation of {{Object.toString}} 
http://docs.oracle.com/javase/6/docs/api/java/lang/Object.html#toString() it 
seems to me that it can still break if the hashCode is overriden with an impl 
that is likely to return the same value for 2 different instances. I know this 
is very unlikely but on the other hand the bug would be very hard to track... 
so I think we should rather generate unique strings (unique a static atomic 
counter for example) to make sure timers and counters are never shared?

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, solr1972-metricsregistry-branch4x-failure.log, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-27 Thread Alan Woodward (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485432#comment-13485432
]

Alan Woodward commented on SOLR-1972:
-

Eek, yes, I get that as well. Must have been unlucky on my first test run.

Looks like the previous patch is the way to go.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch,
elyograg-1972-trunk.patch, elyograg-1972-trunk.patch,
SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch,
SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972_metrics.patch, solr1972-metricsregistry-branch4x-failure.log,
SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch,
SOLR-1972-url_pattern.patch

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-26 Thread Adrien Grand (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13484945#comment-13484945
 ] 

Adrien Grand commented on SOLR-1972:


Hi Alan,

This is great to have all these statistics available through JMX, this would be 
really helpful for Solr monitoring! Instead of relying on 
{{RequestHandlerBase.toString}} to return a distinct instance per request 
handler, maybe each request handler should have its own {{MetricsRegistry}} (or 
maybe use the request handler path as the scope, but this looks harder to do)?

I have no objection to adding the metrics jar to the dependencies of solr-core.

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-26 Thread Alan Woodward (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13484964#comment-13484964
]

Alan Woodward commented on SOLR-1972:
-

Yeah, the toString hack is quite ... hacky. The problem with using the request
handler path as the scope (which I agree would be logical) is that this isn't
available until init() is called, and both getStatistics() and handleRequest()
can get called before init().

Having one MetricsRegistry per request handler looks a lot more sensible
though. I'll put a patch together. Thanks!

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch,
elyograg-1972-trunk.patch, elyograg-1972-trunk.patch,
SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch,
SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch,
SOLR-1972-url_pattern.patch

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-26 Thread Shawn Heisey (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13485280#comment-13485280
 ] 

Shawn Heisey commented on SOLR-1972:


I ran into some trouble with tests failing, so I checked out a fresh branch_4x, 
applied this patch, and tried running solr tests.  There are major explosions.  
I'll be attaching the full putty log for review.  It appears that the first 
exception is an out of memory error:

[junit4:junit4]   2 7800 T177 
ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING 
Uncaught exception in thread: 
Thread[metrics-meter-tick-thread-1,5,TGRP-TestRandomFaceting] 
java.lang.OutOfMemoryError: unable to create new native thread
[junit4:junit4]   2at 
__randomizedtesting.SeedInfo.seed([2191B18B87EDCD66]:0)
[junit4:junit4]   2at java.lang.Thread.start0(Native Method)
[junit4:junit4]   2at java.lang.Thread.start(Thread.java:691)
[junit4:junit4]   2at 
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
[junit4:junit4]   2at 
java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:992)
[junit4:junit4]   2at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[junit4:junit4]   2at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[junit4:junit4]   2at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2

If I edit out the MetricsRegistry and go back to creating the objects directly 
from Metrics and using this.toString() as the scope, the tests all pass.

The Metrics documentation talks about MetricsRegistry as being something you 
create on a per-application basis.  That suggests that it's a very heavy 
object, and even a barebones Solr install probably has at least a dozen 
requestHandlers defined.  I don't know how many are defined in the JVMs used 
for testing.

On my test branch_4x installation, I see 29 handlers between QUERYHANDLER and 
UPDATEHANDLER in the stats visible in the gui, and that's just on one core.  
I've got 16 cores defined.


 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-25 Thread Alan Woodward (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483981#comment-13483981
 ] 

Alan Woodward commented on SOLR-1972:
-

Thanks for testing it Shawn.

I think I'd like some more eyes on it before I commit - it adds a dependency to 
solr-core, which is a pretty big change.  Anyone else have an opinion?

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-24 Thread Alan Woodward (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483091#comment-13483091
 ] 

Alan Woodward commented on SOLR-1972:
-

Updated patch, using this.toString() as the scope identifier.  Your 
handlerCount solution wouldn't have been thread-safe, Shawn, but thanks for 
finding the right method to use!

Also adds a test to check that different handlers have different statistics.

[~otis] as Shawn says, this is just extending the existing mbeans, so it's 
already available through JMX.  Metrics also exposes everything through JMX by 
default anyway, so you can get the stats either way.

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-24 Thread Shawn Heisey (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483277#comment-13483277
]

Shawn Heisey commented on SOLR-1972:

I wanted to double-check that the new test would fail with the old constructor,
so I installed your new patch and removed this.toString() from the parameter
list on those statements.

The updated RequestHandlerTests didn't fail like I'd hoped. I suspect that's
because the terms handler and the update handler are different classes, so
this.getClass() was apparently different. Before the fix, I did have different
statistics in the update handler versus the search handlers. It was only
handlers of the same type that were the same -- my four search handlers.

I did however get a rather spectacular (and repeatable) failure in
MBeansHandlerTest. That failure went away when I restored the constructor to
the current patch state. When I make it into the office, I will do some
additional testing to make sure I did everything right.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch,
elyograg-1972-trunk.patch, elyograg-1972-trunk.patch,
SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch,
SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972.patch,
SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-24 Thread Shawn Heisey (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483445#comment-13483445
 ] 

Shawn Heisey commented on SOLR-1972:


With this as the contructor:

{code}
  public RequestHandlerBase() {
//numRequests = Metrics.newCounter(RequestHandlerBase.class, numRequests, 
this.toString());
//numErrors = Metrics.newCounter(RequestHandlerBase.class, numErrors, 
this.toString());
//numTimeouts = Metrics.newCounter(RequestHandlerBase.class, numTimeouts, 
this.toString());
//requestTimes = Metrics.newTimer(RequestHandlerBase.class, requestTimes, 
this.toString());
numRequests = Metrics.newCounter(RequestHandlerBase.class, numRequests);
numErrors = Metrics.newCounter(RequestHandlerBase.class, numErrors);
numTimeouts = Metrics.newCounter(RequestHandlerBase.class, numTimeouts);
requestTimes = Metrics.newTimer(RequestHandlerBase.class, requestTimes);
  }
{code}

I get the following as the failure.  I suppose I should be glad that it fails, 
but RequestsHandlerTests, which is the test that was modified, continues to 
pass.

{code}
[junit4:junit4] Suite: org.apache.solr.handler.admin.MBeansHandlerTest
[junit4:junit4]   2 0 T297 oas.SolrTestCaseJ4.initCore initCore
[junit4:junit4]   2 Creating dataDir: 
/index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087
[junit4:junit4]   2 26 T297 oasc.SolrResourceLoader.init new 
SolrResourceLoader for directory: 
'/index/src/branch_4x/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2 27 T297 oasc.SolrResourceLoader.replaceClassLoader Adding 
'file:/index/src/branch_4x/solr/build/solr-core/test-files/solr/collection1/lib/classes/'
 to classloader
[junit4:junit4]   2 27 T297 oasc.SolrResourceLoader.replaceClassLoader Adding 
'file:/index/src/branch_4x/solr/build/solr-core/test-files/solr/collection1/lib/README'
 to classloader
[junit4:junit4]   2 59 T297 oasc.SolrConfig.init Using Lucene MatchVersion: 
LUCENE_41
[junit4:junit4]   2 98 T297 oasc.SolrConfig.init Loaded SolrConfig: 
solrconfig.xml
[junit4:junit4]   2 98 T297 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2 105 T297 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2 414 T297 oass.OpenExchangeRatesOrgProvider.init 
Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2 420 T297 oass.IndexSchema.readSchema default search field 
in schema is text
[junit4:junit4]   2 422 T297 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2 430 T297 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
[junit4:junit4]   2 432 T297 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
[junit4:junit4]   2 434 T297 oass.OpenExchangeRatesOrgProvider.reload 
Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2 435 T297 oass.OpenExchangeRatesOrgProvider.reload 
Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2 436 T297 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
[junit4:junit4]   2 436 T297 oasc.SolrResourceLoader.locateSolrHome using 
system property solr.solr.home: 
/index/src/branch_4x/solr/build/solr-core/test-files/solr
[junit4:junit4]   2 436 T297 oasc.SolrResourceLoader.init new 
SolrResourceLoader for directory: 
'/index/src/branch_4x/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2 442 T297 oasc.CoreContainer.init New CoreContainer 
1897850152
[junit4:junit4]   2 442 T297 oasc.SolrCore.init [collection1] Opening new 
SolrCore at 
/index/src/branch_4x/solr/build/solr-core/test-files/solr/collection1/, 
dataDir=/index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087/
[junit4:junit4]   2 443 T297 oasc.JmxMonitoredMap.init JMX monitoring is 
enabled. Adding Solr mbeans to JMX Server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@201a970
[junit4:junit4]   2 443 T297 oasc.SolrCore.getNewIndexDir New index directory 
detected: old=null 
new=/index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087/index/
[junit4:junit4]   2 444 T297 oasc.SolrCore.initIndex WARNING [collection1] 
Solr index directory 
'/index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087/index'
 doesn't exist. Creating new index...
[junit4:junit4]   2 460 T297 oasc.CachingDirectoryFactory.get return new 
directory for 
/index/src/branch_4x/solr/build/solr-core/test/J0/./solrtest-MBeansHandlerTest-1351102494087/index
 forceNew:false
[junit4:junit4]   2 462 T297 oasc.SolrDeletionPolicy.onCommit 
SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2
commit{dir=MockDirWrapper(org.apache.lucene.store.SimpleFSDirectory@/index/src/branch_4x/solr/build/solr-core/test/J0/index4103741039tmp

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-24 Thread Alan Woodward (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483449#comment-13483449
 ] 

Alan Woodward commented on SOLR-1972:
-

There's a test bug in there - it's comparing the NamedList objects when it 
should be comparing their values.  Will put up a patch in a bit...

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Assignee: Alan Woodward
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-24 Thread Shawn Heisey (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13483614#comment-13483614
]

Shawn Heisey commented on SOLR-1972:

You probably already knew this, but now the updated test fails properly for me
when the constructor doesn't set the scope. I still get the failure in
MBeansHandlerTest, which without knowledge of how all this stuff works
internally, is really really mystifying. Since all solr tests pass when I
don't monkey with the constructor, I guess it's not a big deal.

Not that my vote really counts, but +1 for committing to 4x and trunk from me.
Take out those enemy death cannons.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch,
elyograg-1972-trunk.patch, elyograg-1972-trunk.patch,
SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch,
SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch,
SOLR-1972-url_pattern.patch

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482921#comment-13482921
 ] 

Shawn Heisey commented on SOLR-1972:


I never know when things are threadsafe, although it does seem to work.  Does a 
static class member variable 'automatically' become threadsafe, or would 
volatile be required?  Was protected the right way to do that?


 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
Affects Versions: 1.4
Reporter: Shawn Heisey
Priority: Minor
 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-23 Thread Otis Gospodnetic (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482948#comment-13482948
]

Otis Gospodnetic commented on SOLR-1972:

[~romseygeek] are these metrics being published in JMX? They should, and
Code's Metrics should make that easy. I had 2.5 second look at the patch and
didn't find any mentions of jmx.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

Key: SOLR-1972
URL: https://issues.apache.org/jira/browse/SOLR-1972
Project: Solr
Issue Type: Improvement
Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Priority: Minor
Fix For: 4.1

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482960#comment-13482960
 ] 

Shawn Heisey commented on SOLR-1972:


Solr handler statistics were already published in JMX, this just added some 
entries.  I pulled up jconsole and connected to my patched solr server and the 
new stats were there.

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
  Components: web gui
Affects Versions: 1.4
Reporter: Shawn Heisey
Priority: Minor
 Fix For: 4.1

 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972_metrics.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482171#comment-13482171
]

Shawn Heisey commented on SOLR-1972:

After poking around a lot looking for a way to bump the reservoir size, I
finally came across the paper on reservoir sampling by Vitter. After even more
poking around, I think I get it now. Their small reservoir apparently really
does give statistically relevant results over millions or billions of total
samples. If it didn't give them numbers they could use, they would have
already made it larger.

Do you think it's worthwhile to give people the ability to customize the
percentile list -- turn some of the standard percentiles off, and/or add custom
ones? As soon as we conclude that including the full predefined set won't
present a performance problem because it only gets calculated when the admin
GUI is accessed, there'll be someone who has created hundreds of request
handlers and polls the statistics for all of them once a minute. I can also
see someone wanting to see the 12th and 87th percentiles for some reason
neither of us can fathom, but makes perfect sense to them.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

Key: SOLR-1972
URL: https://issues.apache.org/jira/browse/SOLR-1972
Project: Solr
Issue Type: Improvement
Affects Versions: 1.4
Reporter: Shawn Heisey
Priority: Minor
Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch,
elyograg-1972-trunk.patch, elyograg-1972-trunk.patch,
SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch,
SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972.patch,
SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-23 Thread Alan Woodward (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482415#comment-13482415
 ] 

Alan Woodward commented on SOLR-1972:
-

I think we'd probably want to keep the percentile list limited to start with.  
People can always ask for improvements later if they need them.

I think this is ready to go in?

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
Affects Versions: 1.4
Reporter: Shawn Heisey
Priority: Minor
 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-23 Thread Stefan Matheis (steffkes) (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482431#comment-13482431
]

Stefan Matheis (steffkes) commented on SOLR-1972:
-

Hey, i applied to Patch to see where the Output goes and how it looks in the
admin GUI .. right now, it's listed in the stats-section, as a table with all
the other given attributes. how helpful would it be to have some kind of graph
here? perhaps one like we have already on the dashboard to see the
memory-usage? Let me know what you think about it :)

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

Key: SOLR-1972
URL: https://issues.apache.org/jira/browse/SOLR-1972
Project: Solr
Issue Type: Improvement
Affects Versions: 1.4
Reporter: Shawn Heisey
Priority: Minor
Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch,
elyograg-1972-trunk.patch, elyograg-1972-trunk.patch,
SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch,
SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch,
SOLR-1972-url_pattern.patch

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-23 Thread Alan Woodward (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482459#comment-13482459
 ] 

Alan Woodward commented on SOLR-1972:
-

Hey Stefan, yeah I've been thinking about graphs, pictures are always good.  
For this sort of information, though, I think you generally want time-series 
representations, and for that you want proper monitoring software (something 
like graphite).  So I think the best thing we can do here is just expose the 
data, and let people plug in their own monitors.  Unless you have a better idea 
for how to represent it?

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
Affects Versions: 1.4
Reporter: Shawn Heisey
Priority: Minor
 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482470#comment-13482470
 ] 

Shawn Heisey commented on SOLR-1972:


This is lightyears beyond what I could have hoped for when I first opened this 
issue.  I do have one more picky note: the extreme floating point precision of 
the output.  Here's what I am getting:

handlerStart: 1351011529717
requests: 14
errors:   0
timeouts: 0
totalTime:53483.059463
avgTimePerRequest:  3820.2185330714287
avgRequestsPerSecond:   0.1592740558481214
5minRateReqsPerSecond:  0.6393213424767414
15minRateReqsPerSecond: 0.7422605686168207
medianRequestTime:  2537.401157
75thPcRequestTime:  7728.151086
95thPcRequestTime:  8963.867643
99thPcRequestTime:  8963.867643
999thPcRequestTime: 8963.867643

Here's what I would like to see instead ... lower precision, and rounded up 
when the first eliminated digit is 5 or higher.

handlerStart: 1351011529717
requests: 14
errors:   0
timeouts: 0
totalTime:53483
avgTimePerRequest:  3820
avgRequestsPerSecond:   0.16
5minRateReqsPerSecond:  0.64
15minRateReqsPerSecond: 0.74
medianRequestTime:  2537.40
75thPcRequestTime:  7728.15
95thPcRequestTime:  8963.87
99thPcRequestTime:  8963.87
999thPcRequestTime: 8963.87


 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
Affects Versions: 1.4
Reporter: Shawn Heisey
Priority: Minor
 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482475#comment-13482475
 ] 

Shawn Heisey commented on SOLR-1972:


I would also move avgTimePerRequest to right before medianRequestTime so 
similar numbers are all together.  I was going to suggest giving it a new name 
to jive with the additions, but there are likely a lot of existing customer 
scripts that rely on that name -- including some of mine.


 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
Affects Versions: 1.4
Reporter: Shawn Heisey
Priority: Minor
 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482501#comment-13482501
]

Shawn Heisey commented on SOLR-1972:

I have now discovered what a real problem. All of my search handlers have the
exact same statistics. I have defined /lbcheck, /ncdismax, /search, and
/select.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

Key: SOLR-1972
URL: https://issues.apache.org/jira/browse/SOLR-1972
Project: Solr
Issue Type: Improvement
Affects Versions: 1.4
Reporter: Shawn Heisey
Priority: Minor
Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch,
elyograg-1972-trunk.patch, elyograg-1972-trunk.patch,
SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch,
SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch,
SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch,
SOLR-1972-url_pattern.patch

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-23 Thread Alan Woodward (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13482647#comment-13482647
 ] 

Alan Woodward commented on SOLR-1972:
-

Hm, OK.  I'm creating the various Metrics objects in the base class 
constructor, and registering them by class using this.getClass().  Only problem 
here is that in a super constructor, getClass() returns the superclass.  Oops.

If I move the object creation to init() I get other errors, because 
RequestHandlers are registered with JMX before init() is called, and JMX calls 
getStatistics() to get all the various measurement names and register them.

Maybe put a guard in getStatistics to check if the counters are null, and if 
they are, instantiate them?  Seems a bit hacky though.  Let me have a think 
about this.

In re the precision of the measurements, the jscript in the front end could 
presumably round them to 2 sig figs - that way they look prettier in the UI, 
but are still precise for any client that wants to use it.

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
Affects Versions: 1.4
Reporter: Shawn Heisey
Priority: Minor
 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972_metrics.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, 
 SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-22 Thread Alan Woodward (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481655#comment-13481655
]

Alan Woodward commented on SOLR-1972:
-

Here's a patch that uses the metrics library. It doesn't include Eric's regex
matching or anything at the moment - it basically just takes what's currently
in trunk, refactors it to use metrics' Counter and Timer objects, and adds the
rolling average data.

Cons:
- it adds another dependency to solr-core. It's a useful dependency, IMO,
but still.
- tests don't pass at the moment, as metrics spawns extra threads which the
test runner doesn't know how to deal with

Pros:
- it's a purpose-designed stats and metrics library, so we don't need to
worry about the maths or sampling algorithms
- it adds the functionality of the original ticket/patch in a much simpler
way.

The ideal solution would be a component of some kind, I think, but this at
least improves on what's in trunk at the moment.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481751#comment-13481751
]

Shawn Heisey commented on SOLR-1972:

Awesome, Alan! What options might we have to prevent long-running handlers
from accumulating huge metrics histories and chewing up tons of RAM? Is there
a get75thpercentile method? With the old patch, I do 75, 95, and 99. I would
also like to add 99.9, but the old patch uses ints so that wasn't possible.

When I have a moment, I will attempt to look at the javadocs for the package
and answer my own questions. Unless you get to it first, I will also attempt
to mod the patch to expose any memory-limiting options.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-22 Thread Alan Woodward (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481767#comment-13481767
]

Alan Woodward commented on SOLR-1972:
-

Hi Shawn,

Metrics uses reservoir sampling to maintain its measurements, so the history is
actually always a fixed size. This is configurable, but defaults to 1024
entries. There's more information at
http://metrics.codahale.com/manual/core/#histograms and
http://www.johndcook.com/standard_deviation.html.

There are get75thpercentile and get999thpercentile methods out of the box, and
you can also ask for values at arbitrary percentiles using getValue().

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481797#comment-13481797
]

Shawn Heisey commented on SOLR-1972:

I have answers to some of my questions. There is a 75th percentile. I added
the 75th and 999th to what you had, and it seems to display the stats page a
lot faster than my patch did. We'll see what happens when it gets a few
thousand queries under its belt, though. I was running the old patch with
16384 samples, and I put the stats on three handlers, so it was having to copy
arrays of 16384 longs a total of six times every time I refreshed the stats
page. I may also add the 98th percentile. It may be a good idea to make each
percentile point configurable in solrconfig.xml. So far I have not yet figured
out whether it is possible to limit the number of samples stored, or anything
else which can limit the amount of memory required.

The names for the average req/s over the last 5 and 15 minutes are REALLY long.
Unless you have a high res display (1920 pixels wide) and maximize the window,
the names overlap the values. If I think of a reasonable way to shorten those,
I will. I ran into it myself when making my branch_4x patch.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

[
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481809#comment-13481809
]

Shawn Heisey commented on SOLR-1972:

I didn't see your reply about the reservoir size until after I'd already
submitted mine. If I want to increase/decrease that size, how do I do that?
So far poking around the javadocs and using google hasn't turned anything up.

Need additional query stats in admin interface - median, 95th and 99th
percentile
-

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile

2012-10-22 Thread Alan Woodward (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13481842#comment-13481842
 ] 

Alan Woodward commented on SOLR-1972:
-

It looks like the sample size for Timer objects is hard-wired.

It's only 8kB per Timer, though, which is pretty much a rounding error when 
compared with Solr's memory usage.

 Need additional query stats in admin interface - median, 95th and 99th 
 percentile
 -

 Key: SOLR-1972
 URL: https://issues.apache.org/jira/browse/SOLR-1972
 Project: Solr
  Issue Type: Improvement
Affects Versions: 1.4
Reporter: Shawn Heisey
Priority: Minor
 Attachments: elyograg-1972-3.2.patch, elyograg-1972-3.2.patch, 
 elyograg-1972-trunk.patch, elyograg-1972-trunk.patch, 
 SOLR-1972-branch3x-url_pattern.patch, SOLR-1972-branch4x.patch, 
 SOLR-1972-branch4x.patch, SOLR-1972_metrics.patch, SOLR-1972.patch, 
 SOLR-1972.patch, SOLR-1972.patch, SOLR-1972.patch, SOLR-1972-url_pattern.patch


 I would like to see more detailed query statistics from the admin GUI.  This 
 is what you can get now:
 requests : 809
 errors : 0
 timeouts : 0
 totalTime : 70053
 avgTimePerRequest : 86.59209
 avgRequestsPerSecond : 0.8148785 
 I'd like to see more data on the time per request - median, 95th percentile, 
 99th percentile, and any other statistical function that makes sense to 
 include.  In my environment, the first bunch of queries after startup tend to 
 take several seconds each.  I find that the average value tends to be useless 
 until it has several thousand queries under its belt and the caches are 
 thoroughly warmed.  The statistical functions I have mentioned would quickly 
 eliminate the influence of those initial slow queries.
 The system will have to store individual data about each query.  I don't know 
 if this is something Solr does already.  It would be nice to have a 
 configurable count of how many of the most recent data points are kept, to 
 control the amount of memory the feature uses.  The default value could be 
 something like 1024 or 4096.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1972) Need additional query stats in admin interface - median, 95th and 99th percentile