[jira] [Commented] (OAK-3478) Provide JMX Beans for Oak that can be monitored by external tooling.
[
https://issues.apache.org/jira/browse/OAK-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15012786#comment-15012786
]
Chetan Mehrotra commented on OAK-3478:
--
{quote}
The bean "Oak Repository Statistics-simple" produces the right type of stats,
but I think the the "per second" approach wont work as the sequence of numbers
from the existing TimeSeries follows the pattern
0,0,0,0,0,0,0,76100,0,0,0,0,0,0,0 etc. If the monitoring tool happens to
query either side of the 71600 value, then it get 0. To get 76100 it
has to query at exactly the correct time. You could use the per minute
value but it would be better not to re-invent the research in this area and
look at what others have already proven to work in production.
{quote}
Makes sense. Opened OAK-3654 to determine effort required to provide a metric
based implementation. Would close this issue as Wont Fix then
> Provide JMX Beans for Oak that can be monitored by external tooling.
>
>
> Key: OAK-3478
> URL: https://issues.apache.org/jira/browse/OAK-3478
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, jcr, lucene, query
>Affects Versions: 1.3.7
>Reporter: Ian Boston
>Assignee: Chetan Mehrotra
> Fix For: 1.3.11
>
> Attachments: OAK-3478-v1.patch
>
>
> The Current JMX beans, while ok in the Sling Web Console are hard if not
> impossible to monitor with external tooling, as external tooling will poll
> for current values, ideally from named attributes containing primitive types.
> Those values containing timers, counters or gauges. Timers timing an
> operation. Counters counting an operation. Guages measuring an instantaneous
> value.
> The request is to provide a small number of JMX beans that can be configured
> into an external monitoring tool like AppDynamics, Ganglia, NewRelic, Splunk
> etc etc, which in turn will provide long term time series and statistics.
> Primitive values of this form can also be graphed with ease in JConsole,
> VisualVM etc. A improvement for the Sling Web Console might be to add a
> Console that can maintain a TimeSeries graph of any JMX bean by object name
> in the same way Ganglia, AppDynamics does, however that may be duplicating
> existing functionality.
> The Metrics Library could be considered to provide the above functionality
> for all JMX beans and monitoring, although its footprint at 111K might be
> considered too big as an additional dependency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3478) Provide JMX Beans for Oak that can be monitored by external tooling.
[ https://issues.apache.org/jira/browse/OAK-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15010908#comment-15010908 ] Ian Boston commented on OAK-3478: - @Chetan Mehrotra The bean "Oak Repository Statistics-simple" produces the right type of stats, but I think the the "per second" approach wont work as the sequence of numbers from the existing TimeSeries follows the pattern 0,0,0,0,0,0,0,76100,0,0,0,0,0,0,0 etc. If the monitoring tool happens to query either side of the 71600 value, then it get 0. To get 76100 it has to query at exactly the correct time. You could use the per minute value but it would be better not to re-invent the research in this area and look at what others have already proven to work in production. Most Metrics gathering systems use some form of moving average eg http://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average or a pure counter to let the monitoring tool do the work. Pure counters are not very good as they tend to hit overflow problems, Moving averages of some form are better, provided the Windows or reservoirs can be implemented efficiently. Since averages are not always that useful when identifying performance most metrics tools also provide live histogram of the metric to produce 50%, 99.99%. 99.99% becomes important if the operation is performed thousands of times where each operation correlates with others, as a slow operation 0.001 % of the time will impact all operations with absolute certainty. Oak has plenty of places where this is a characteristic, starting with queues. To produce those sort of stats you may need to implement http://www.cs.umd.edu/~samir/498/vitter.pdf. The link was taken from Codehale Metrics which has various implementations testing in production. If Oak is not prepared to use a third party library for metrics support it should learn from what others have used successfully in production. > Provide JMX Beans for Oak that can be monitored by external tooling. > > > Key: OAK-3478 > URL: https://issues.apache.org/jira/browse/OAK-3478 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, jcr, lucene, query >Affects Versions: 1.3.7 >Reporter: Ian Boston >Assignee: Chetan Mehrotra > Fix For: 1.3.11 > > Attachments: OAK-3478-v1.patch > > > The Current JMX beans, while ok in the Sling Web Console are hard if not > impossible to monitor with external tooling, as external tooling will poll > for current values, ideally from named attributes containing primitive types. > Those values containing timers, counters or gauges. Timers timing an > operation. Counters counting an operation. Guages measuring an instantaneous > value. > The request is to provide a small number of JMX beans that can be configured > into an external monitoring tool like AppDynamics, Ganglia, NewRelic, Splunk > etc etc, which in turn will provide long term time series and statistics. > Primitive values of this form can also be graphed with ease in JConsole, > VisualVM etc. A improvement for the Sling Web Console might be to add a > Console that can maintain a TimeSeries graph of any JMX bean by object name > in the same way Ganglia, AppDynamics does, however that may be duplicating > existing functionality. > The Metrics Library could be considered to provide the above functionality > for all JMX beans and monitoring, although its footprint at 111K might be > considered too big as an additional dependency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3478) Provide JMX Beans for Oak that can be monitored by external tooling.
[ https://issues.apache.org/jira/browse/OAK-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15010755#comment-15010755 ] Ian Boston commented on OAK-3478: - @Michael Dürig SLING-4849 might do what you need if you want to decouple monitoring. > Provide JMX Beans for Oak that can be monitored by external tooling. > > > Key: OAK-3478 > URL: https://issues.apache.org/jira/browse/OAK-3478 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, jcr, lucene, query >Affects Versions: 1.3.7 >Reporter: Ian Boston >Assignee: Chetan Mehrotra > Fix For: 1.3.11 > > Attachments: OAK-3478-v1.patch > > > The Current JMX beans, while ok in the Sling Web Console are hard if not > impossible to monitor with external tooling, as external tooling will poll > for current values, ideally from named attributes containing primitive types. > Those values containing timers, counters or gauges. Timers timing an > operation. Counters counting an operation. Guages measuring an instantaneous > value. > The request is to provide a small number of JMX beans that can be configured > into an external monitoring tool like AppDynamics, Ganglia, NewRelic, Splunk > etc etc, which in turn will provide long term time series and statistics. > Primitive values of this form can also be graphed with ease in JConsole, > VisualVM etc. A improvement for the Sling Web Console might be to add a > Console that can maintain a TimeSeries graph of any JMX bean by object name > in the same way Ganglia, AppDynamics does, however that may be duplicating > existing functionality. > The Metrics Library could be considered to provide the above functionality > for all JMX beans and monitoring, although its footprint at 111K might be > considered too big as an additional dependency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3478) Provide JMX Beans for Oak that can be monitored by external tooling.
[
https://issues.apache.org/jira/browse/OAK-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15010669#comment-15010669
]
Michael Dürig commented on OAK-3478:
I think in general we should get rid of the time series and leave recording
tasks to monitoring tools. However, as this is not possible for all cases the
suggested approach looks like a good start to me. Instead of baking this into
Oak I would rather like to decouple it. Couldn't we make this into a separate
utility for monitoring that could be deployed and configured OSGi style? The
implementation could live e.g. in {{jcr-commons}} (like {{TimeSeries}}) and
default configuration/provisioning could happen from upstream projects like
Sling/Granite/AEM.
> Provide JMX Beans for Oak that can be monitored by external tooling.
>
>
> Key: OAK-3478
> URL: https://issues.apache.org/jira/browse/OAK-3478
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, jcr, lucene, query
>Affects Versions: 1.3.7
>Reporter: Ian Boston
>Assignee: Chetan Mehrotra
> Fix For: 1.3.11
>
> Attachments: OAK-3478-v1.patch
>
>
> The Current JMX beans, while ok in the Sling Web Console are hard if not
> impossible to monitor with external tooling, as external tooling will poll
> for current values, ideally from named attributes containing primitive types.
> Those values containing timers, counters or gauges. Timers timing an
> operation. Counters counting an operation. Guages measuring an instantaneous
> value.
> The request is to provide a small number of JMX beans that can be configured
> into an external monitoring tool like AppDynamics, Ganglia, NewRelic, Splunk
> etc etc, which in turn will provide long term time series and statistics.
> Primitive values of this form can also be graphed with ease in JConsole,
> VisualVM etc. A improvement for the Sling Web Console might be to add a
> Console that can maintain a TimeSeries graph of any JMX bean by object name
> in the same way Ganglia, AppDynamics does, however that may be duplicating
> existing functionality.
> The Metrics Library could be considered to provide the above functionality
> for all JMX beans and monitoring, although its footprint at 111K might be
> considered too big as an additional dependency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3478) Provide JMX Beans for Oak that can be monitored by external tooling.
[ https://issues.apache.org/jira/browse/OAK-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15008455#comment-15008455 ] Chetan Mehrotra commented on OAK-3478: -- [~ianeboston] I get the issue with use of CompositeData and its widespread use. I see two approaches. # One we go for most important MBean and expose both variants of the data # OR Use some generic approach say based on DynamicmMBean to expose simple variants for the time series data So far I was planning for #1 hence the ask for the important MBean to target first. Would try to prototype an approach for #2 though > Provide JMX Beans for Oak that can be monitored by external tooling. > > > Key: OAK-3478 > URL: https://issues.apache.org/jira/browse/OAK-3478 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, jcr, lucene, query >Affects Versions: 1.3.7 >Reporter: Ian Boston >Assignee: Chetan Mehrotra > > The Current JMX beans, while ok in the Sling Web Console are hard if not > impossible to monitor with external tooling, as external tooling will poll > for current values, ideally from named attributes containing primitive types. > Those values containing timers, counters or gauges. Timers timing an > operation. Counters counting an operation. Guages measuring an instantaneous > value. > The request is to provide a small number of JMX beans that can be configured > into an external monitoring tool like AppDynamics, Ganglia, NewRelic, Splunk > etc etc, which in turn will provide long term time series and statistics. > Primitive values of this form can also be graphed with ease in JConsole, > VisualVM etc. A improvement for the Sling Web Console might be to add a > Console that can maintain a TimeSeries graph of any JMX bean by object name > in the same way Ganglia, AppDynamics does, however that may be duplicating > existing functionality. > The Metrics Library could be considered to provide the above functionality > for all JMX beans and monitoring, although its footprint at 111K might be > considered too big as an additional dependency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3478) Provide JMX Beans for Oak that can be monitored by external tooling.
[
https://issues.apache.org/jira/browse/OAK-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15008260#comment-15008260
]
Ian Boston commented on OAK-3478:
-
[~chetanm]
Any bean that uses org.apache.jackrabbit.api.stats.TimeSeries is expressing
time series data in a way that can't be consumed.
Possible classes include
{code}
./oak-core/src/main/java/org/apache/jackrabbit/oak/stats/RepositoryStats.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/index/AsyncIndexUpdate.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStoreGCMonitor.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/stats/StatisticManager.java
./oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ChangeProcessor.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/IndexStatsMBean.java
{code}
Most beans that expose their data via javax.management.openmbean.CompositeData
Only those beans that expose data containing information that needs to be
monitored are an issue. Although most JMX tools are slow and hard to use when
confronted with deeply nested data inside Composite or Tabular data, a human
can deal with them.
Potential classes are below
{code}
/oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/IndexStatsMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/RepositoryManagementMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/RepositoryStatsMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/CheckpointMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/ConsolidatedCacheStatsMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/api/jmx/RepositoryManagementMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/cache/ConsolidatedCacheStats.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/management/ManagementOperation.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/management/RepositoryManager.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/backup/FileStoreBackupRestore.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/backup/FileStoreBackupRestoreMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/blob/BlobGC.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/blob/BlobGCMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/blob/migration/BlobMigration.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/blob/migration/BlobMigrationMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentCheckpointMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentCheckpointMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/index/AsyncIndexUpdate.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/index/property/jmx/PropertyIndexAsyncReindex.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/index/property/jmx/PropertyIndexAsyncReindexMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStoreGCMonitor.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/GCMonitorMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentCheckpointMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/spi/state/RevisionGC.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/spi/state/RevisionGCMBean.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/stats/RepositoryStats.java
./oak-core/src/main/java/org/apache/jackrabbit/oak/util/AbstractCheckpointMBean.java
./oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ConsolidatedListenerMBeanImpl.java
./oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/IndexCopier.java
./oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/LuceneIndexMBeanImpl.java
{code}
> Provide JMX Beans for Oak that can be monitored by external tooling.
>
>
> Key: OAK-3478
> URL: https://issues.apache.org/jira/browse/OAK-3478
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, jcr, lucene, query
>Affects Versions: 1.3.7
>Reporter: Ian Boston
>Assignee: Chetan Mehrotra
>
> The Current JMX beans, while ok in the Sling Web Console are hard if not
> impossible to monitor with external tooling, as external tooling will poll
> for current values, ideally from named attributes containing primitive types.
> Those values containing timers, counters or gauges. Timers timing an
> operation. Counters counting an operation. Guages measuring an instantaneous
> value.
> The request is to provide a small number of JMX beans that can be configured
> into an external monitoring
[jira] [Commented] (OAK-3478) Provide JMX Beans for Oak that can be monitored by external tooling.
[ https://issues.apache.org/jira/browse/OAK-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15008102#comment-15008102 ] Chetan Mehrotra commented on OAK-3478: -- [~ianeboston] Do you have list of MBean in mind which we should simplify? That would make the ask here more specific > Provide JMX Beans for Oak that can be monitored by external tooling. > > > Key: OAK-3478 > URL: https://issues.apache.org/jira/browse/OAK-3478 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, jcr, lucene, query >Affects Versions: 1.3.7 >Reporter: Ian Boston > > The Current JMX beans, while ok in the Sling Web Console are hard if not > impossible to monitor with external tooling, as external tooling will poll > for current values, ideally from named attributes containing primitive types. > Those values containing timers, counters or gauges. Timers timing an > operation. Counters counting an operation. Guages measuring an instantaneous > value. > The request is to provide a small number of JMX beans that can be configured > into an external monitoring tool like AppDynamics, Ganglia, NewRelic, Splunk > etc etc, which in turn will provide long term time series and statistics. > Primitive values of this form can also be graphed with ease in JConsole, > VisualVM etc. A improvement for the Sling Web Console might be to add a > Console that can maintain a TimeSeries graph of any JMX bean by object name > in the same way Ganglia, AppDynamics does, however that may be duplicating > existing functionality. > The Metrics Library could be considered to provide the above functionality > for all JMX beans and monitoring, although its footprint at 111K might be > considered too big as an additional dependency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
