Re: [DISCUSS] Which I/O statistics should the FileStore expose?
Thanks Chetan and Julian for the pointers. On one hand I would like to do something similar - in spirit - to DocumentStoreStats. On the other hand, I don't see the advantage of the Metrics framework over a hand-rolled IOMonitorMBean with two AtomicLong fields and two volatile variables. IIUC there is no automatic way of exposing Metrics via an MBean. In fact, the DocumentNodeStoreService registers the DocumentStoreStatsMBean in the same way I would register the IOMonitorMBean from the SegmentNodeStoreService. What could be gained by adding Metrics to the trivial implementation of IOMonitorMBean described above? It is also unclear to me if it's really appropriate to digest some statistics before exposing them to the outside world. In example, how the methods in DocumentStoreStats returning time series as CompositeData play with other monitoring solutions like ElasticSearch/Kibana? How easy is to consume and aggregate this data for a monitoring solution? 2017-02-14 5:13 GMT+01:00 Chetan Mehrotra : > Hi Francesco, > > As Julian mentioned it would be good to collects stats as Metrics. > Have a look at DocumentStoreStats which collects some stats around > operations being performed by DocumentStore implementations > Chetan Mehrotra > > > On Tue, Feb 14, 2017 at 12:37 AM, Julian Sedding wrote: >> Hi Francesco >> >> I believe you should implement an IOMonitor using the metrics in the >> org.apache.jackrabbit.oak.stats package. These can be backed by >> swappable StatisticsProvider implementations. I believe by default >> it's a NOOP implementation. However, I believe that if the >> MetricStatisticsProvider implementation is used, it automatically >> exposes the metrics via JMX. So all you need to do is feed the correct >> data into a suitable metric. I believe Chetan contributed these, so he >> will know more about the details. >> >> Regards >> Julian >> >> >> On Mon, Feb 13, 2017 at 6:21 PM, Francesco Mari >> wrote: >>> Hi all, >>> >>> The recently introduced IOMonitor allows the FileStore to trigger I/O >>> events. Callback methods from IOMonitor can be implemented to receive >>> information about segment reads and writes. >>> >>> A trivial implementation of IOMonitor is able to track the following raw >>> data. >>> >>> - The number of segments read and write operations. >>> - The duration in nanoseconds of every read and write. >>> - The number of bytes read or written by each operation. >>> >>> We are about to expose this kind of information from an MBean - for >>> the sake of discussion, let's call it IOMonitorMBean. I'm currently in >>> favour of starting small and exposing the following statistics: >>> >>> - The duration of the latest write (long). >>> - The duration of the latest read (long). >>> - The number of write operations (long). >>> - The number of read operations (long). >>> >>> I would like your opinion about what's the most useful way to present >>> this data through an MBean. Should just raw data be exposed? Is it >>> appropriate for IOMonitorMBean to perform some kind of aggregation, >>> like sum and average? Should richer data be returned from the MBean, >>> like tabular data? >>> >>> Please keep in mind that this data is supposed to be consumed by a >>> monitoring solution, and not a by human reader.
[Oak origin/trunk] Apache Jackrabbit Oak matrix - Build # 1422 - Still Failing
The Apache Jenkins build system has built Apache Jackrabbit Oak matrix (build
#1422)
Status: Still Failing
Check console output at
https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1422/ to view
the results.
Changes:
[frm] OAK-5631 - Expose segment read time from IOMonitor
[adulceanu] OAK-5600 - Test coverage for CheckCommand Added tests for checking a
[mreutegg] OAK-5633: Builds on travis-ci time out
[frm] OAK-5632 - Expose segment write statistics from IOMonitor
[frm] OAK-5637 - Increase time granularity in IOMonitor
[adulceanu] OAK-5620 - Simplify consistency check
[stefanegli] OAK-5619 : fixing withIncludeAncestorsRemove which reported
/unrelated
Test results:
3 tests failed.
FAILED: org.apache.jackrabbit.oak.segment.standby.StandbyTestIT.testSyncLoop
Error Message:
expected: org.apache.jackrabbit.oak.segment.SegmentNodeState<{ checkpoints = {
... }, root = { ... } }> but was:
org.apache.jackrabbit.oak.segment.SegmentNodeState<{ checkpoints = { ... },
root = { ... } }>
Stack Trace:
java.lang.AssertionError: expected:
org.apache.jackrabbit.oak.segment.SegmentNodeState<{ checkpoints = { ... },
root = { ... } }> but was: org.apache.jackrabbit.oak.segment.SegmentNodeState<{
checkpoints = { ... }, root = { ... } }>
at
org.apache.jackrabbit.oak.segment.standby.StandbyTestIT.testSyncLoop(StandbyTestIT.java:126)
FAILED: org.apache.jackrabbit.oak.segment.standby.StandbyTestIT.testSyncLoop
Error Message:
expected: org.apache.jackrabbit.oak.segment.SegmentNodeState<{ checkpoints = {
... }, root = { ... } }> but was:
org.apache.jackrabbit.oak.segment.SegmentNodeState<{ checkpoints = { ... },
root = { ... } }>
Stack Trace:
java.lang.AssertionError: expected:
org.apache.jackrabbit.oak.segment.SegmentNodeState<{ checkpoints = { ... },
root = { ... } }> but was: org.apache.jackrabbit.oak.segment.SegmentNodeState<{
checkpoints = { ... }, root = { ... } }>
at
org.apache.jackrabbit.oak.segment.standby.StandbyTestIT.testSyncLoop(StandbyTestIT.java:126)
FAILED:
org.apache.jackrabbit.oak.plugins.document.DocumentDiscoveryLiteServiceIT.testLargeStartStopFiesta
Error Message:
unable to create new native thread
Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.(DocumentNodeStore.java:599)
at
org.apache.jackrabbit.oak.plugins.document.DocumentMK$Builder.getNodeStore(DocumentMK.java:839)
at
org.apache.jackrabbit.oak.plugins.document.DocumentMK.(DocumentMK.java:144)
at
org.apache.jackrabbit.oak.plugins.document.DocumentMK$Builder.open(DocumentMK.java:1138)
at
org.apache.jackrabbit.oak.plugins.document.BaseDocumentDiscoveryLiteServiceTest.createMK(BaseDocumentDiscoveryLiteServiceTest.java:716)
at
org.apache.jackrabbit.oak.plugins.document.BaseDocumentDiscoveryLiteServiceTest.createMK(BaseDocumentDiscoveryLiteServiceTest.java:710)
at
org.apache.jackrabbit.oak.plugins.document.BaseDocumentDiscoveryLiteServiceTest.createNodeStore(BaseDocumentDiscoveryLiteServiceTest.java:589)
at
org.apache.jackrabbit.oak.plugins.document.BaseDocumentDiscoveryLiteServiceTest.createInstance(BaseDocumentDiscoveryLiteServiceTest.java:608)
at
org.apache.jackrabbit.oak.plugins.document.BaseDocumentDiscoveryLiteServiceTest.doStartStopFiesta(BaseDocumentDiscoveryLiteServiceTest.java:760)
at
org.apache.jackrabbit.oak.plugins.document.DocumentDiscoveryLiteServiceIT.testLargeStartStopFiesta(DocumentDiscoveryLiteServiceIT.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJ
[Oak origin/1.0] Apache Jackrabbit Oak matrix - Build # 1421 - Still Failing
The Apache Jenkins build system has built Apache Jackrabbit Oak matrix (build #1421) Status: Still Failing Check console output at https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1421/ to view the results. Changes: [mreutegg] OAK-5608: Test failure: org.apache.jackrabbit.mk.util.CommitGateIT.test Test results: 28 tests failed. FAILED: org.apache.jackrabbit.oak.security.authentication.ldap.LdapDefaultLoginModuleTest.org.apache.jackrabbit.oak.security.authentication.ldap.LdapDefaultLoginModuleTest Error Message: ERR_171 Failed to bind an LDAP service (1,024) to the service registry. Stack Trace: org.apache.directory.api.ldap.model.exception.LdapConfigurationException: ERR_171 Failed to bind an LDAP service (1,024) to the service registry. at org.apache.directory.server.ldap.LdapServer.startNetwork(LdapServer.java:695) at org.apache.directory.server.ldap.LdapServer.start(LdapServer.java:547) at org.apache.jackrabbit.oak.security.authentication.ldap.AbstractServer.setUp(AbstractServer.java:227) at org.apache.jackrabbit.oak.security.authentication.ldap.InternalLdapServer.setUp(InternalLdapServer.java:33) at org.apache.jackrabbit.oak.security.authentication.ldap.LdapLoginTestBase.beforeClass(LdapLoginTestBase.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:463) at sun.nio.ch.Net.bind(Net.java:455) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) at org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) at org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) FAILED: org.apache.jackrabbit.oak.security.authentication.ldap.LdapProviderTest.testNullIntermediatePath Error Message: ERR_171 Failed to bind an LDAP service (1,024) to the service registry. Stack Trace: org.apache.directory.api.ldap.model.exception.LdapConfigurationException: ERR_171 Failed to bind an LDAP service (1,024) to the service registry. at org.apache.directory.server.ldap.LdapServer.startNetwork(LdapServer.java:695) at org.apache.directory.server.ldap.LdapServ
[Oak origin/1.2] Apache Jackrabbit Oak matrix - Build # 1420 - Still Failing
The Apache Jenkins build system has built Apache Jackrabbit Oak matrix (build #1420) Status: Still Failing Check console output at https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1420/ to view the results. Changes: [mreutegg] OAK-5608: Test failure: org.apache.jackrabbit.mk.util.CommitGateIT.test Test results: 1 tests failed. FAILED: org.apache.jackrabbit.oak.jcr.LargeOperationIT.initializationError Error Message: unable to create new native thread Stack Trace: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at com.mongodb.Mongo.(Mongo.java:326) at com.mongodb.Mongo.(Mongo.java:304) at com.mongodb.MongoClient.(MongoClient.java:253) at org.apache.jackrabbit.oak.plugins.document.util.MongoConnection.(MongoConnection.java:45) at org.apache.jackrabbit.oak.jcr.NodeStoreFixture$DocumentFixture.createNodeStore(NodeStoreFixture.java:180) at org.apache.jackrabbit.oak.jcr.NodeStoreFixture$DocumentFixture.isAvailable(NodeStoreFixture.java:211) at org.apache.jackrabbit.oak.jcr.LargeOperationIT.fixtures(LargeOperationIT.java:146) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.runners.Parameterized.allParameters(Parameterized.java:280) at org.junit.runners.Parameterized.(Parameterized.java:248) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104) at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86) at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59) at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26) at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59) at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:250) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Re: [DISCUSS] Which I/O statistics should the FileStore expose?
Hi Francesco, As Julian mentioned it would be good to collects stats as Metrics. Have a look at DocumentStoreStats which collects some stats around operations being performed by DocumentStore implementations Chetan Mehrotra On Tue, Feb 14, 2017 at 12:37 AM, Julian Sedding wrote: > Hi Francesco > > I believe you should implement an IOMonitor using the metrics in the > org.apache.jackrabbit.oak.stats package. These can be backed by > swappable StatisticsProvider implementations. I believe by default > it's a NOOP implementation. However, I believe that if the > MetricStatisticsProvider implementation is used, it automatically > exposes the metrics via JMX. So all you need to do is feed the correct > data into a suitable metric. I believe Chetan contributed these, so he > will know more about the details. > > Regards > Julian > > > On Mon, Feb 13, 2017 at 6:21 PM, Francesco Mari > wrote: >> Hi all, >> >> The recently introduced IOMonitor allows the FileStore to trigger I/O >> events. Callback methods from IOMonitor can be implemented to receive >> information about segment reads and writes. >> >> A trivial implementation of IOMonitor is able to track the following raw >> data. >> >> - The number of segments read and write operations. >> - The duration in nanoseconds of every read and write. >> - The number of bytes read or written by each operation. >> >> We are about to expose this kind of information from an MBean - for >> the sake of discussion, let's call it IOMonitorMBean. I'm currently in >> favour of starting small and exposing the following statistics: >> >> - The duration of the latest write (long). >> - The duration of the latest read (long). >> - The number of write operations (long). >> - The number of read operations (long). >> >> I would like your opinion about what's the most useful way to present >> this data through an MBean. Should just raw data be exposed? Is it >> appropriate for IOMonitorMBean to perform some kind of aggregation, >> like sum and average? Should richer data be returned from the MBean, >> like tabular data? >> >> Please keep in mind that this data is supposed to be consumed by a >> monitoring solution, and not a by human reader.
Re: [DISCUSS] Which I/O statistics should the FileStore expose?
Hi Francesco I believe you should implement an IOMonitor using the metrics in the org.apache.jackrabbit.oak.stats package. These can be backed by swappable StatisticsProvider implementations. I believe by default it's a NOOP implementation. However, I believe that if the MetricStatisticsProvider implementation is used, it automatically exposes the metrics via JMX. So all you need to do is feed the correct data into a suitable metric. I believe Chetan contributed these, so he will know more about the details. Regards Julian On Mon, Feb 13, 2017 at 6:21 PM, Francesco Mari wrote: > Hi all, > > The recently introduced IOMonitor allows the FileStore to trigger I/O > events. Callback methods from IOMonitor can be implemented to receive > information about segment reads and writes. > > A trivial implementation of IOMonitor is able to track the following raw data. > > - The number of segments read and write operations. > - The duration in nanoseconds of every read and write. > - The number of bytes read or written by each operation. > > We are about to expose this kind of information from an MBean - for > the sake of discussion, let's call it IOMonitorMBean. I'm currently in > favour of starting small and exposing the following statistics: > > - The duration of the latest write (long). > - The duration of the latest read (long). > - The number of write operations (long). > - The number of read operations (long). > > I would like your opinion about what's the most useful way to present > this data through an MBean. Should just raw data be exposed? Is it > appropriate for IOMonitorMBean to perform some kind of aggregation, > like sum and average? Should richer data be returned from the MBean, > like tabular data? > > Please keep in mind that this data is supposed to be consumed by a > monitoring solution, and not a by human reader.
[DISCUSS] Which I/O statistics should the FileStore expose?
Hi all, The recently introduced IOMonitor allows the FileStore to trigger I/O events. Callback methods from IOMonitor can be implemented to receive information about segment reads and writes. A trivial implementation of IOMonitor is able to track the following raw data. - The number of segments read and write operations. - The duration in nanoseconds of every read and write. - The number of bytes read or written by each operation. We are about to expose this kind of information from an MBean - for the sake of discussion, let's call it IOMonitorMBean. I'm currently in favour of starting small and exposing the following statistics: - The duration of the latest write (long). - The duration of the latest read (long). - The number of write operations (long). - The number of read operations (long). I would like your opinion about what's the most useful way to present this data through an MBean. Should just raw data be exposed? Is it appropriate for IOMonitorMBean to perform some kind of aggregation, like sum and average? Should richer data be returned from the MBean, like tabular data? Please keep in mind that this data is supposed to be consumed by a monitoring solution, and not a by human reader.
Re: Clone Not implemented
Hi, > The repository description might be wrong here. It doesn't look like it though [0]. Maybe an error on the client side reading/displaying the values? alex [0] https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/core/ContentRepositoryImpl.java#L233 On Mon, Feb 6, 2017 at 9:23 AM, Michael Dürig wrote: > > Hi, > > Shareable nodes are not supported by Oak and there is currently no > discussions for implementing this feature. The repository description might > be wrong here. Please report an issue at https://issues.apache.org/jira > /browse/OAK if you think this is a bug. Attaching a test case for the > problem helps getting this addressed. > > Michael > > > > On 3.2.17 4:05 , Valentina Marioli wrote: > >> Hi, I'm Valentina Marioli, >> I'm working with Oak and I get confused using "clone" method. >> >> Querying the repository descriptor table with >> "Repository.OPTION_SHAREABLE_NODES_SUPPORTED" , I get "true", so I >> aspect that shareable nodes are supported, but when I try to use the >> method "clone", I get "Not implemented". >> Is it right? Is "clone" operation not supported yet? >> When are you planning to release this feature? >> >> Regards, >> Valentina Marioli >> >
