[jira] [Commented] (OAK-2596) more (jmx) instrumentation for observation queue
[ https://issues.apache.org/jira/browse/OAK-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358265#comment-14358265 ] Marcel Reutegger commented on OAK-2596: --- bq. Maybe there are ways to expose such information from the DocumentMK diff chache There are cache stats available already for the diff cache, which includes e.g. a hit rate. But it doesn't provide historical numbers like the our TimeSeries. more (jmx) instrumentation for observation queue Key: OAK-2596 URL: https://issues.apache.org/jira/browse/OAK-2596 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.0.12 Reporter: Stefan Egli Assignee: Michael Dürig Labels: monitoring, observation Fix For: 1.0.13 While debugging issues with the observation queue it would be handy to have more detailed information available. At the moment you can only see one value wrt length of the queue: that is the maximum of all queues. It is unclear if the queue is that long for only one or many listeners. And it is unclear from that if the listener is slow or the engine that produces the events for the listener. So I'd suggest to add the following details - possible exposed via JMX? : # add queue length details to each of the observation listeners # have a history of the last, eg 1000 events per listener showing a) how long the event took to be created/generated and b) how long the listener took to process. Sometimes averages are not detailed enough so such a in-depth information might become useful. (Not sure about the feasibility of '1000' here - maybe that could be configurable though - just putting the idea out here). # have some information about whether a listener is currently 'reading events from the cache' or whether it has to go to eg mongo # maybe have a 'top 10' listeners that have the largest queue at the moment to easily allow navigation instead of having to go through all (eg 200) listeners manually each time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-2582) RDB: improve memory cache handling
[ https://issues.apache.org/jira/browse/OAK-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke resolved OAK-2582. - Resolution: Fixed RDB: improve memory cache handling -- Key: OAK-2582 URL: https://issues.apache.org/jira/browse/OAK-2582 Project: Jackrabbit Oak Issue Type: Bug Components: rdbmk Affects Versions: 1.0.11, 1.1.6 Reporter: Julian Reschke Assignee: Julian Reschke Fix For: 1.1.8, 1.0.13 Improve memory cache handling: - invalidateCache should just mark cache entries as to be revalidated - to-be revalidated cache entries can be used for conditional retrieval from DB -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-2464) Optimize read of known non-existing children
[
https://issues.apache.org/jira/browse/OAK-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356901#comment-14356901
]
Julian Reschke edited comment on OAK-2464 at 3/11/15 2:18 PM:
--
This change causes a reproducible failure on my systems:
addNodeGetNode(org.apache.jackrabbit.oak.plugins.document.SimpleTest) Time
elapsed: 0.453 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.junit.Assert.assertNotNull(Assert.java:537)
at
org.apache.jackrabbit.oak.plugins.document.SimpleTest.addNodeGetNode(SimpleTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
[~chetanm] removed fix version 1.0.13 and backing out the change for now
was (Author: reschke):
This change causes a reproducible failure on my systems:
addNodeGetNode(org.apache.jackrabbit.oak.plugins.document.SimpleTest) Time
elapsed: 0.453 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.junit.Assert.assertNotNull(Assert.java:537)
at
org.apache.jackrabbit.oak.plugins.document.SimpleTest.addNodeGetNode(SimpleTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
Optimize read of known non-existing children
Key: OAK-2464
URL: https://issues.apache.org/jira/browse/OAK-2464
Project: Jackrabbit Oak
Issue Type: Improvement
Reporter: Vikas Saurabh
Assignee: Chetan Mehrotra
Priority: Minor
Fix For: 1.1.6
Attachments:
0001-OAK-2464-Optimize-read-of-known-non-existing-childre.patch,
OAK-2464-Sort order test case.patch, OAK-2464-Take2.patch
Reading a non-existing node always queries down to {{DocumentStore}}.
{{DocumentNodeStore}} should consider {{nodeChildrenCache}} to see if it
already knows about non-existence and avoid a look up into {{DocumentStore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2464) Optimize read of known non-existing children
[
https://issues.apache.org/jira/browse/OAK-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356978#comment-14356978
]
Chetan Mehrotra commented on OAK-2464:
--
Thanks Julian for spotting this. I saw this failure but somehow in my mind
assumed that SimpleTest is some custom test of mine I have in branch code but
not checked in! Digging this further indicates that failure is due to OAK-2448
not being merged to branch.
[~mreutegg] Would it be fine to merge OAK-2448 to branch?
Optimize read of known non-existing children
Key: OAK-2464
URL: https://issues.apache.org/jira/browse/OAK-2464
Project: Jackrabbit Oak
Issue Type: Improvement
Reporter: Vikas Saurabh
Assignee: Chetan Mehrotra
Priority: Minor
Fix For: 1.1.6
Attachments:
0001-OAK-2464-Optimize-read-of-known-non-existing-childre.patch,
OAK-2464-Sort order test case.patch, OAK-2464-Take2.patch
Reading a non-existing node always queries down to {{DocumentStore}}.
{{DocumentNodeStore}} should consider {{nodeChildrenCache}} to see if it
already knows about non-existence and avoid a look up into {{DocumentStore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2570) Open indexes in IndexTracker non blocking way
[
https://issues.apache.org/jira/browse/OAK-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356986#comment-14356986
]
Chetan Mehrotra commented on OAK-2570:
--
I have [increased the timeout to 30 sec|http://svn.apache.org/r1665893]. Async
index gets triggered every 5 sec so it might miss out picking up changes for 2
cycle but should be able to do this 6 cycles. Let see how it behaves
Open indexes in IndexTracker non blocking way
-
Key: OAK-2570
URL: https://issues.apache.org/jira/browse/OAK-2570
Project: Jackrabbit Oak
Issue Type: Improvement
Components: oak-lucene
Reporter: Chetan Mehrotra
Assignee: Chetan Mehrotra
Fix For: 1.1.8, 1.0.13
Attachments: OAK-2570.patch
Currently {{IndexTracker}} opens the indexes in same thread where the commit
happens as it acts as an {{Observer}}. If opening the index takes long time
it would block the commit thread and might introduce delay in other operation
on same thread.
For example {{DocumentNodeStore}} invokes the {{Observer}} in
{{backgroundRead}} and that call would not complete unless the {{Observer}}
returns back. For a system to be responsive the {{backgroundRead}} has to be
quick as it holds the write lock. Due to current implementation at time
system turn unresponsive.
Looking at the threaddump from one such unresponsive system it appears that
slowness is due to Lucene index taking time to reopen.
Thread which hold the {{backgroundOperationLock}} for {{read}} is
{noformat}
pool-9-thread-1 Id=101 in RUNNABLE
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.init(FileInputStream.java:146)
at
org.apache.jackrabbit.core.data.LazyFileInputStream.open(LazyFileInputStream.java:104)
at
org.apache.jackrabbit.core.data.LazyFileInputStream.read(LazyFileInputStream.java:163)
at com.google.common.io.ByteStreams.read(ByteStreams.java:828)
at com.google.common.io.ByteStreams.readFully(ByteStreams.java:695)
at
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.loadBlob(OakDirectory.java:218)
at
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.readBytes(OakDirectory.java:264)
at
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.readBytes(OakDirectory.java:350)
at
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.readByte(OakDirectory.java:356)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:84)
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:126)
at org.apache.lucene.util.fst.FST.init(FST.java:318)
at org.apache.lucene.util.fst.FST.init(FST.java:304)
at
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader.init(BlockTreeTermsReader.java:484)
at
org.apache.lucene.codecs.BlockTreeTermsReader.init(BlockTreeTermsReader.java:176)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:437)
at
org.apache.lucene.index.SegmentCoreReaders.init(SegmentCoreReaders.java:116)
at org.apache.lucene.index.SegmentReader.init(SegmentReader.java:96)
at
org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:62)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:843)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:66)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.init(IndexNode.java:94)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:62)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker$1.leave(IndexTracker.java:96)
at
org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:153)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeState.dispatch(DocumentNodeState.java:433)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeState.compareAgainstBaseState(DocumentNodeState.java:260)
at
org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeState.dispatch(DocumentNodeState.java:433)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeState.compareAgainstBaseState(DocumentNodeState.java:260)
at
org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.update(IndexTracker.java:105)
- locked
org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker@6243264e
at
[jira] [Resolved] (OAK-2413) Clarify Editor.childNodeChanged()
[
https://issues.apache.org/jira/browse/OAK-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angela resolved OAK-2413.
-
Resolution: Fixed
Fix Version/s: (was: 1.2)
1.1.8
Clarify Editor.childNodeChanged()
-
Key: OAK-2413
URL: https://issues.apache.org/jira/browse/OAK-2413
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core
Reporter: Marcel Reutegger
Assignee: angela
Priority: Minor
Fix For: 1.1.8
The current contract for {{Editor.childNodeChanged()}} does not specify if
this method may also be called when the child node did not actually change.
The method {{NodeStateDiff.childNodeChanged()}} explicitly states that there
may be such calls. Looking at the implementation connecting the two classes,
{{EditorDiff.childNodeChange()}} simply calls the editor without checking
whether the child node did in fact change.
I think we either have to change the {{EditorDiff}} or update the contract
for the Editor and adjust implementations. E.g. right now, PrivilegeValidator
(implements Editor), assumes a call to {{childNodeChange()}} indeed means the
child node changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2413) Clarify Editor.childNodeChanged()
[
https://issues.apache.org/jira/browse/OAK-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355280#comment-14355280
]
angela commented on OAK-2413:
-
fixed {{PrivilegeValidator}} at revision 1665630 (including a test for both
before and after state being equal or not). [~mreutegg] thanks for spotting!
Clarify Editor.childNodeChanged()
-
Key: OAK-2413
URL: https://issues.apache.org/jira/browse/OAK-2413
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core
Reporter: Marcel Reutegger
Assignee: angela
Priority: Minor
Fix For: 1.1.8
The current contract for {{Editor.childNodeChanged()}} does not specify if
this method may also be called when the child node did not actually change.
The method {{NodeStateDiff.childNodeChanged()}} explicitly states that there
may be such calls. Looking at the implementation connecting the two classes,
{{EditorDiff.childNodeChange()}} simply calls the editor without checking
whether the child node did in fact change.
I think we either have to change the {{EditorDiff}} or update the contract
for the Editor and adjust implementations. E.g. right now, PrivilegeValidator
(implements Editor), assumes a call to {{childNodeChange()}} indeed means the
child node changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2596) more (jmx) instrumentation for observation queue
[
https://issues.apache.org/jira/browse/OAK-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358269#comment-14358269
]
Chetan Mehrotra commented on OAK-2596:
--
bq. maybe have a 'top 10' listeners that have the largest queue at the moment
to easily allow navigation instead of having to go through all (eg 200)
listeners manually each time.
This can be done similar to {{ConsolidatedCacheStatsMBean}} which collate the
cache related stats and provides a unified view
more (jmx) instrumentation for observation queue
Key: OAK-2596
URL: https://issues.apache.org/jira/browse/OAK-2596
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core
Affects Versions: 1.0.12
Reporter: Stefan Egli
Assignee: Michael Dürig
Labels: monitoring, observation
Fix For: 1.0.13
While debugging issues with the observation queue it would be handy to have
more detailed information available. At the moment you can only see one value
wrt length of the queue: that is the maximum of all queues. It is unclear if
the queue is that long for only one or many listeners. And it is unclear from
that if the listener is slow or the engine that produces the events for the
listener.
So I'd suggest to add the following details - possible exposed via JMX? :
# add queue length details to each of the observation listeners
# have a history of the last, eg 1000 events per listener showing a) how long
the event took to be created/generated and b) how long the listener took to
process. Sometimes averages are not detailed enough so such a in-depth
information might become useful. (Not sure about the feasibility of '1000'
here - maybe that could be configurable though - just putting the idea out
here).
# have some information about whether a listener is currently 'reading events
from the cache' or whether it has to go to eg mongo
# maybe have a 'top 10' listeners that have the largest queue at the moment
to easily allow navigation instead of having to go through all (eg 200)
listeners manually each time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-1450) Run repair tool if repo cannot start (automatically or explicitly)
[ https://issues.apache.org/jira/browse/OAK-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-1450: --- Fix Version/s: (was: 1.2) Run repair tool if repo cannot start (automatically or explicitly) -- Key: OAK-1450 URL: https://issues.apache.org/jira/browse/OAK-1450 Project: Jackrabbit Oak Issue Type: Improvement Components: segmentmk Reporter: Michael Marth Priority: Minor Labels: production, resilience, tools If the repo does not come up should we automatically run the repair tool (OAK-1446) and then retry...? Not sure about this one. Maybe starting the repo again with an explicit flag --repair would be better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-2602) [Solr] Cost calculation takes time with solr pings even when not fulfilling query
[ https://issues.apache.org/jira/browse/OAK-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommaso Teofili resolved OAK-2602. -- Resolution: Fixed removed the ping calls in r1665892 (trunk) and r1665895 (branch 1.0) [Solr] Cost calculation takes time with solr pings even when not fulfilling query - Key: OAK-2602 URL: https://issues.apache.org/jira/browse/OAK-2602 Project: Jackrabbit Oak Issue Type: Improvement Components: oak-solr Affects Versions: 1.0.12, 1.1.7 Reporter: Amit Jain Assignee: Tommaso Teofili Fix For: 1.1.8, 1.0.13 Cost calculation for queries which are fired quite often [1] and which are not going to fulfilled by solr take time due to which the overall cost of the operation is high. [1] SELECT * FROM [nt:base] WHERE PROPERTY([rep:members], 'WeakReference') = $uuid SELECT * FROM [nt:base] WHERE [jcr:uuid] = $id -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-2590) IndexCopier Error occurred while removing deleted files from Local
[
https://issues.apache.org/jira/browse/OAK-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-2590.
--
Resolution: Fixed
As a fix now a info level message would logged. When index gets updated again
then next cycle of opening a new directory would again make an attempt to
delete such files. Hence such files would never get accumulated and created
garbage for long time
Fixed with http://svn.apache.org/r1666102
IndexCopier Error occurred while removing deleted files from Local
--
Key: OAK-2590
URL: https://issues.apache.org/jira/browse/OAK-2590
Project: Jackrabbit Oak
Issue Type: Improvement
Components: oak-lucene
Reporter: Chetan Mehrotra
Assignee: Chetan Mehrotra
Priority: Minor
Fix For: 1.1.8
Under windows, with copy on read mode enabled for the lucene indexes,
following WARN logs were seen:
{code:java}
03.02.2015 17:40:58.952 *WARN* [pool-5-thread-2]
org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier Error occurred
while removing deleted files from Local
MMapDirectory@D:\repository\index\e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293\1
lockFactory=NativeFSLockFactory@D:\repository\index\e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293\1,
Remote OakDirectory@232fe713
lockFactory=org.apache.lucene.store.NoLockFactory@696097fc
java.io.IOException: Cannot delete
D:\repository\index\e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293\1\_wq4.cfs
at org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:273)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.removeDeletedFiles(IndexCopier.java:274)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.access$1000(IndexCopier.java:113)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory$2.run(IndexCopier.java:247)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
{code}
As the files are memory mapped and the handle is yet not GC its possible that
windows would not allow to delete such file. In such a case the log message
should be improved and possibly have a way where attempt is made to delete
such files later
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2605) Support for additional encodings needed in ReversedLinesFileReader
[ https://issues.apache.org/jira/browse/OAK-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357003#comment-14357003 ] Alex Parvulescu commented on OAK-2605: -- fyi I added a missing license header with http://svn.apache.org/r1665898 Support for additional encodings needed in ReversedLinesFileReader -- Key: OAK-2605 URL: https://issues.apache.org/jira/browse/OAK-2605 Project: Jackrabbit Oak Issue Type: Bug Components: segmentmk Affects Versions: 1.1.7 Environment: Windows 2012 R2 Japanese Windows 2012 R2 Korean Windows 2012 R2 Simplified Chinese Windows 2012 R2 Traditional Chinese Reporter: Leandro Reis Assignee: Michael Dürig Fix For: 1.1.8 Attachments: OAK-2605.patch I'm working on a product that uses Commons IO via Jackrabbit Oak. In the process of testing the launch of such product on Japanese Windows 2012 Server R2, I came across the following exception: (java.io.UnsupportedEncodingException: Encoding windows-31j is not supported yet (feel free to submit a patch)) windows-31j is the IANA name for Windows code page 932 (Japanese), and is returned by Charset.defaultCharset(), used in org.apache.commons.io.input.ReversedLinesFileReader [0]. A patch for this issue was provided in https://issues.apache.org/jira/browse/IO-471 . It also includes changes needed to support Chinese Simplified, Chinese Traditional and Korean. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-2612) Findbugs plugin version should depend on JDK version
Tommaso Teofili created OAK-2612: Summary: Findbugs plugin version should depend on JDK version Key: OAK-2612 URL: https://issues.apache.org/jira/browse/OAK-2612 Project: Jackrabbit Oak Issue Type: Bug Components: parent Affects Versions: 1.1.7 Reporter: Tommaso Teofili Assignee: Tommaso Teofili Fix For: 1.1.8 Looking at the current CI builds the JDK 8 one is always failing because _maven-findbugs-plugin-2.5.3_ doesn't work with it, on the other hand upgrading to 3.0.0 will make the JDK 6 build failing, therefore the version used has to be dependent on the used JDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2612) Findbugs plugin version should depend on JDK version
[ https://issues.apache.org/jira/browse/OAK-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358377#comment-14358377 ] Tommaso Teofili commented on OAK-2612: -- in the meantime I've fixed this issue by using _maven-findbugs-plugin-2.5.5_ for JDK6 and _3.0.0_ for JDK 7+. Findbugs plugin version should depend on JDK version Key: OAK-2612 URL: https://issues.apache.org/jira/browse/OAK-2612 Project: Jackrabbit Oak Issue Type: Bug Components: parent Affects Versions: 1.1.7 Reporter: Tommaso Teofili Assignee: Tommaso Teofili Fix For: 1.1.8 Looking at the current CI builds the JDK 8 one is always failing because _maven-findbugs-plugin-2.5.3_ doesn't work with it, on the other hand upgrading to 3.0.0 will make the JDK 6 build failing, therefore the version used has to be dependent on the used JDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2287) ConflictHandler for merging jcr:lastModified
[ https://issues.apache.org/jira/browse/OAK-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358325#comment-14358325 ] Alex Parvulescu commented on OAK-2287: -- from where I'm standing the patch is good enough. The only remaining question is if it makes sense to refactor it to be used for any date value that might conflict in a similar way (ie. custom:lastModified). I can take care of this if it is interesting to have. ConflictHandler for merging jcr:lastModified Key: OAK-2287 URL: https://issues.apache.org/jira/browse/OAK-2287 Project: Jackrabbit Oak Issue Type: Improvement Components: core, jcr Reporter: Alex Parvulescu Assignee: Alex Parvulescu Priority: Minor Fix For: 1.2 Attachments: OAK-2287-2.patch, OAK-2287.patch As discussed on oak-dev, 'jcr:lastModified' is a date property and it represents the last modification date. It makes sense to merge concurrent updates by picking the newer date of the 2 options. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2464) Optimize read of known non-existing children
[
https://issues.apache.org/jira/browse/OAK-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357103#comment-14357103
]
Chetan Mehrotra commented on OAK-2464:
--
Remerged the fix to branch. All testcase pass locally
Optimize read of known non-existing children
Key: OAK-2464
URL: https://issues.apache.org/jira/browse/OAK-2464
Project: Jackrabbit Oak
Issue Type: Improvement
Reporter: Vikas Saurabh
Assignee: Chetan Mehrotra
Priority: Minor
Fix For: 1.1.6, 1.0.13
Attachments:
0001-OAK-2464-Optimize-read-of-known-non-existing-childre.patch,
OAK-2464-Sort order test case.patch, OAK-2464-Take2.patch
Reading a non-existing node always queries down to {{DocumentStore}}.
{{DocumentNodeStore}} should consider {{nodeChildrenCache}} to see if it
already knows about non-existence and avoid a look up into {{DocumentStore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2611) Lucene suggester should only be updated if the index is used for suggestions
Tommaso Teofili created OAK-2611: Summary: Lucene suggester should only be updated if the index is used for suggestions Key: OAK-2611 URL: https://issues.apache.org/jira/browse/OAK-2611 Project: Jackrabbit Oak Issue Type: Bug Components: oak-lucene Affects Versions: 1.1.7 Reporter: Tommaso Teofili Assignee: Tommaso Teofili Fix For: 1.1.8 As mentioned on [oak-dev@|http://markmail.org/message/vp6qg7v5j3oxev73] Lucene suggester should be updated only in indexes that are used for suggestions (_useForSuggestions = true_ in index definition). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2611) Lucene suggester should only be updated if the index is used for suggestions
[
https://issues.apache.org/jira/browse/OAK-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358350#comment-14358350
]
Tommaso Teofili commented on OAK-2611:
--
improved suggester test configuration to avoid continuously update suggester
index other than in {{SuggestTest}} in r1666100
Lucene suggester should only be updated if the index is used for suggestions
Key: OAK-2611
URL: https://issues.apache.org/jira/browse/OAK-2611
Project: Jackrabbit Oak
Issue Type: Bug
Components: oak-lucene
Affects Versions: 1.1.7
Reporter: Tommaso Teofili
Assignee: Tommaso Teofili
Fix For: 1.1.8
As mentioned on [oak-dev@|http://markmail.org/message/vp6qg7v5j3oxev73]
Lucene suggester should be updated only in indexes that are used for
suggestions (_useForSuggestions = true_ in index definition).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358376#comment-14358376
]
Stefan Egli commented on OAK-2557:
--
Discussed offline with [~mreutegg]: created OAK-2613 to follow this up
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-2612) Findbugs plugin version should depend on JDK version
[ https://issues.apache.org/jira/browse/OAK-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommaso Teofili resolved OAK-2612. -- Resolution: Fixed fixed in r1666107 Findbugs plugin version should depend on JDK version Key: OAK-2612 URL: https://issues.apache.org/jira/browse/OAK-2612 Project: Jackrabbit Oak Issue Type: Bug Components: parent Affects Versions: 1.1.7 Reporter: Tommaso Teofili Assignee: Tommaso Teofili Fix For: 1.1.8 Looking at the current CI builds the JDK 8 one is always failing because _maven-findbugs-plugin-2.5.3_ doesn't work with it, on the other hand upgrading to 3.0.0 will make the JDK 6 build failing, therefore the version used has to be dependent on the used JDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-2609) Thread.interrupt seems to stop repository
Carsten Ziegeler created OAK-2609: - Summary: Thread.interrupt seems to stop repository Key: OAK-2609 URL: https://issues.apache.org/jira/browse/OAK-2609 Project: Jackrabbit Oak Issue Type: Bug Affects Versions: 1.1.7 Environment: Java 7 Reporter: Carsten Ziegeler Priority: Critical We have a sporadic problem with Sling's JCR installer 3.3.8 and Oak (tar mk). It seems to timing related: the JCR installer does a Thread#interrupt at one point and sometimes this brings the hole instance to stop. Nothing else is going on any more. While of course, a workaround is to remove the Thread.interrupt call in the JCR installer (which we did, see SLING-4477), I have the fear that this can happen with any code that is using the repository and gets interrupted. This error is hard to reproduce, however with three people testing we could see this several times happening -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2612) Findbugs plugin version should depend on JDK version
[ https://issues.apache.org/jira/browse/OAK-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358369#comment-14358369 ] Alex Parvulescu commented on OAK-2612: -- _will make the JDK 6 build failing_ which off course begs the question about dropping JDK6 support :) Findbugs plugin version should depend on JDK version Key: OAK-2612 URL: https://issues.apache.org/jira/browse/OAK-2612 Project: Jackrabbit Oak Issue Type: Bug Components: parent Affects Versions: 1.1.7 Reporter: Tommaso Teofili Assignee: Tommaso Teofili Fix For: 1.1.8 Looking at the current CI builds the JDK 8 one is always failing because _maven-findbugs-plugin-2.5.3_ doesn't work with it, on the other hand upgrading to 3.0.0 will make the JDK 6 build failing, therefore the version used has to be dependent on the used JDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2613) Do versionGC more frequently and at adjusted speed
[
https://issues.apache.org/jira/browse/OAK-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Egli updated OAK-2613:
-
Description:
This is a follow-up ticket from
[here|https://issues.apache.org/jira/browse/OAK-2557?focusedCommentId=14355322page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14355322]
mixed with an offline discussion with [~mreutegg]:
* we could make the VersionGC play nicely to existing load on the system: it
could progress slower if the load is higher and vice-verca. One simple measure
could be: if the observation queue is small (eg below 10) then the load is low
and it could progress full-speed. Otherwise it could add some artificial
sleeping in between.
* we could run versionGC more regularly than once a day but instead kind of
'continuously' let it run in the background. While the speed of the gc would be
adjusted to the load - it also would have to be assured that it doesn't run too
slow (and would never finish if instance is under some constant load)
Note that 'adjusted speed' would also imply some intelligence about the system
load, as pointed out by [~chetanm] on OAK-2557:
{quote}Version GC currently ensures that query fired is made against the
Secondary (if present). However having some throttling in such background task
would be good thing to have. But first we need to have some SystemLoadIndicator
notion in Oak which can be provide details say in percentage 1..100 about
system load. We can then expose configurable threshold which VersionGC would
listen for and adjust its working accordingly.
It can be a JMX bean which emits notification and we have our components listen
to those notification (or use OSGi SR/Events). That can be used in other places
like Observation processing, Blob GC etc
{quote}
was:
This is a follow-up ticket from
[here|https://issues.apache.org/jira/browse/OAK-2557?focusedCommentId=14355322page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14355322]
mixed with an offline discussion with [~mreutegg]:
* we could make the VersionGC play nicely to existing load on the system: it
could progress slower if the load is higher and vice-verca. One simple measure
could be: if the observation queue is small (eg below 10) then the load is low
and it could progress full-speed. Otherwise it could add some artificial
sleeping in between.
* we could run versionGC more regularly than once a day but instead kind of
'continuously' let it run in the background. While the speed of the gc would be
adjusted to the load - it also would have to be assured that it doesn't run too
slow (and would never finish if instance is under some constant load)
Do versionGC more frequently and at adjusted speed
--
Key: OAK-2613
URL: https://issues.apache.org/jira/browse/OAK-2613
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core, mongomk
Affects Versions: 1.0.12
Reporter: Stefan Egli
Fix For: 1.2
This is a follow-up ticket from
[here|https://issues.apache.org/jira/browse/OAK-2557?focusedCommentId=14355322page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14355322]
mixed with an offline discussion with [~mreutegg]:
* we could make the VersionGC play nicely to existing load on the system: it
could progress slower if the load is higher and vice-verca. One simple
measure could be: if the observation queue is small (eg below 10) then the
load is low and it could progress full-speed. Otherwise it could add some
artificial sleeping in between.
* we could run versionGC more regularly than once a day but instead kind of
'continuously' let it run in the background. While the speed of the gc would
be adjusted to the load - it also would have to be assured that it doesn't
run too slow (and would never finish if instance is under some constant load)
Note that 'adjusted speed' would also imply some intelligence about the
system load, as pointed out by [~chetanm] on OAK-2557:
{quote}Version GC currently ensures that query fired is made against the
Secondary (if present). However having some throttling in such background
task would be good thing to have. But first we need to have some
SystemLoadIndicator notion in Oak which can be provide details say in
percentage 1..100 about system load. We can then expose configurable
threshold which VersionGC would listen for and adjust its working accordingly.
It can be a JMX bean which emits notification and we have our components
listen to those notification (or use OSGi SR/Events). That can be used in
other places like Observation processing, Blob GC etc
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2612) Findbugs plugin version should depend on JDK version
[ https://issues.apache.org/jira/browse/OAK-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358374#comment-14358374 ] Tommaso Teofili commented on OAK-2612: -- bq. begs the question about dropping JDK6 suppor :) absolutely, I'd be generally happy to drop it. Findbugs plugin version should depend on JDK version Key: OAK-2612 URL: https://issues.apache.org/jira/browse/OAK-2612 Project: Jackrabbit Oak Issue Type: Bug Components: parent Affects Versions: 1.1.7 Reporter: Tommaso Teofili Assignee: Tommaso Teofili Fix For: 1.1.8 Looking at the current CI builds the JDK 8 one is always failing because _maven-findbugs-plugin-2.5.3_ doesn't work with it, on the other hand upgrading to 3.0.0 will make the JDK 6 build failing, therefore the version used has to be dependent on the used JDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-2613) Do versionGC more frequently and at adjusted speed
Stefan Egli created OAK-2613: Summary: Do versionGC more frequently and at adjusted speed Key: OAK-2613 URL: https://issues.apache.org/jira/browse/OAK-2613 Project: Jackrabbit Oak Issue Type: Improvement Components: core, mongomk Affects Versions: 1.0.12 Reporter: Stefan Egli Fix For: 1.2 This is a follow-up ticket from [here|https://issues.apache.org/jira/browse/OAK-2557?focusedCommentId=14355322page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14355322] mixed with an offline discussion with [~mreutegg]: * we could make the VersionGC play nicely to existing load on the system: it could progress slower if the load is higher and vice-verca. One simple measure could be: if the observation queue is small (eg below 10) then the load is low and it could progress full-speed. Otherwise it could add some artificial sleeping in between. * we could run versionGC more regularly than once a day but instead kind of 'continuously' let it run in the background. While the speed of the gc would be adjusted to the load - it also would have to be assured that it doesn't run too slow (and would never finish if instance is under some constant load) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-2612) Findbugs plugin version should depend on JDK version
[ https://issues.apache.org/jira/browse/OAK-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358424#comment-14358424 ] Michael Dürig edited comment on OAK-2612 at 3/12/15 10:05 AM: -- I think we should drop it after 1.2. Let's take the discussion up again for 1.3.x was (Author: mduerig): I think we should drop it after 1.2. Let's the discussion up again for 1.3.x Findbugs plugin version should depend on JDK version Key: OAK-2612 URL: https://issues.apache.org/jira/browse/OAK-2612 Project: Jackrabbit Oak Issue Type: Bug Components: parent Affects Versions: 1.1.7 Reporter: Tommaso Teofili Assignee: Tommaso Teofili Fix For: 1.1.8 Looking at the current CI builds the JDK 8 one is always failing because _maven-findbugs-plugin-2.5.3_ doesn't work with it, on the other hand upgrading to 3.0.0 will make the JDK 6 build failing, therefore the version used has to be dependent on the used JDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2500) checkDeepHistory/fixDeepHistory/prepareDeepHistory for oak-mongo.js
[ https://issues.apache.org/jira/browse/OAK-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger updated OAK-2500: -- Fix Version/s: (was: 1.0.13) 1.1.8 We do not maintain the version in the 1.0 branch and the [documentation|http://jackrabbit.apache.org/oak/docs/command_line.html] refers to the trunk version. checkDeepHistory/fixDeepHistory/prepareDeepHistory for oak-mongo.js --- Key: OAK-2500 URL: https://issues.apache.org/jira/browse/OAK-2500 Project: Jackrabbit Oak Issue Type: Improvement Components: run Affects Versions: 1.0.8 Reporter: Stefan Egli Assignee: Marcel Reutegger Fix For: 1.1.8 Attachments: oak-mongo-mod.js The oak-mongo.js currently contains checkHistory/fixHistory which are cleaning up 'dangling revisions/split-documents' on a particular path. Now it would be good to have a command that goes through the entire repository and checks/fixes these dangling revisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-2557:
-
Attachment: OAK-2557-3.patch
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358420#comment-14358420 ] Tommaso Teofili commented on OAK-2476: -- after fixing OAK-2612, also the JDK 8 build starts ok. I've set each build to poll SVN once a day to at least have an execution for each day, I'm now trying to set up the different fixtures and, hopefully, the matrix. Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358443#comment-14358443 ] Tommaso Teofili commented on OAK-2476: -- matrix job created with [~mirko]'s help (thanks! :) ) at https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/ Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358517#comment-14358517
]
Chetan Mehrotra commented on OAK-2557:
--
Thanks for the detailed review [~mreutegg]. Attached is the [updated
patch|^OAK-2557-3.patch]
bq. Maybe even promote NodeDocIdCollector to a top level class to avoid
breaking encapsulation?
That makes more sense as its a generic class not bound to Document logic. I
have refactored and moved this class to oak-commons under sort package.
[~tmueller] Can you have a quick look as with this it becomes part of Oak API
bq. NodeDocIdCollector.sort() uses a hard coded Comparator when sorting in
memory instead of the instance passed in the constructor.
Fixed that
bq. NodeDocIdCollector.flushToFile() uses PrintWriter.println(). This method
does not throw an IOException when the write fails. I think it would be better
to use BufferedWriter directly.
Did not knew that. Refactored to use {{BufferedWriter}}
bq. VersionGCState.close() deletes the the directory before resources are
closed. I think this will fail on Windows based machines.
Thanks for catching that. Would have missed this completely!
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2614) Timeout for MongoDB query
Marcel Reutegger created OAK-2614: - Summary: Timeout for MongoDB query Key: OAK-2614 URL: https://issues.apache.org/jira/browse/OAK-2614 Project: Jackrabbit Oak Issue Type: Improvement Components: core, mongomk Reporter: Marcel Reutegger Assignee: Marcel Reutegger Fix For: 1.1.8 MongoDB 2.6 introduced a query timeout which allows a client to specify a time how long a query is allowed to execute. I'd like to introduce this as a configurable feature to better control resource usage in MongoDB and stop too expensive queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2596) more (jmx) instrumentation for observation queue
[ https://issues.apache.org/jira/browse/OAK-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358533#comment-14358533 ] Michael Dürig commented on OAK-2596: The gust of the implementation for this will be in the Jackrabbit code base. more (jmx) instrumentation for observation queue Key: OAK-2596 URL: https://issues.apache.org/jira/browse/OAK-2596 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.0.12 Reporter: Stefan Egli Assignee: Michael Dürig Labels: monitoring, observation Fix For: 1.1.8, 1.0.13 While debugging issues with the observation queue it would be handy to have more detailed information available. At the moment you can only see one value wrt length of the queue: that is the maximum of all queues. It is unclear if the queue is that long for only one or many listeners. And it is unclear from that if the listener is slow or the engine that produces the events for the listener. So I'd suggest to add the following details - possible exposed via JMX? : # add queue length details to each of the observation listeners # have a history of the last, eg 1000 events per listener showing a) how long the event took to be created/generated and b) how long the listener took to process. Sometimes averages are not detailed enough so such a in-depth information might become useful. (Not sure about the feasibility of '1000' here - maybe that could be configurable though - just putting the idea out here). # have some information about whether a listener is currently 'reading events from the cache' or whether it has to go to eg mongo # maybe have a 'top 10' listeners that have the largest queue at the moment to easily allow navigation instead of having to go through all (eg 200) listeners manually each time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358591#comment-14358591
]
Marcel Reutegger commented on OAK-2557:
---
Looks better now. [~tmueller], WDYT?
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358595#comment-14358595
]
Marcel Reutegger commented on OAK-2557:
---
Forgot one thing: the patch contains a change in the logback-test.xml. I assume
this is unintentional and will not be committed.
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2615) oak-pojosr tests fail
Michael Dürig created OAK-2615:
--
Summary: oak-pojosr tests fail
Key: OAK-2615
URL: https://issues.apache.org/jira/browse/OAK-2615
Project: Jackrabbit Oak
Issue Type: Bug
Components: pojosr
Reporter: Michael Dürig
Assignee: Michael Dürig
The build fails for me in {{oak-pojosr}} with many failures like:
{noformat}
Running org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.142 sec
FAILURE!
testNodeStoreWithBlobStore_OAK_1676(org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest)
Time elapsed: 10.138 sec FAILURE!
java.lang.AssertionError: No service found for
org.apache.jackrabbit.oak.spi.state.NodeStore. Expression: sr. Values: sr = null
at
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:400)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:646)
at
org.apache.jackrabbit.oak.run.osgi.AbstractRepositoryFactoryTest.getServiceWithWait(AbstractRepositoryFactoryTest.groovy:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at
org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest.testNodeStoreWithBlobStore_OAK_1676(NodeStoreConfigTest.groovy:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Running org.apache.jackrabbit.oak.run.osgi.OakOSGiRepositoryFactoryTest
Tests run: 1,
[jira] [Updated] (OAK-2615) oak-pojosr tests fail
[
https://issues.apache.org/jira/browse/OAK-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-2615:
---
Assignee: Chetan Mehrotra (was: Michael Dürig)
oak-pojosr tests fail
-
Key: OAK-2615
URL: https://issues.apache.org/jira/browse/OAK-2615
Project: Jackrabbit Oak
Issue Type: Bug
Components: pojosr
Reporter: Michael Dürig
Assignee: Chetan Mehrotra
Fix For: 1.1.8
The build fails for me in {{oak-pojosr}} with many failures like:
{noformat}
Running org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.142 sec
FAILURE!
testNodeStoreWithBlobStore_OAK_1676(org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest)
Time elapsed: 10.138 sec FAILURE!
java.lang.AssertionError: No service found for
org.apache.jackrabbit.oak.spi.state.NodeStore. Expression: sr. Values: sr =
null
at
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:400)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:646)
at
org.apache.jackrabbit.oak.run.osgi.AbstractRepositoryFactoryTest.getServiceWithWait(AbstractRepositoryFactoryTest.groovy:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at
org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest.testNodeStoreWithBlobStore_OAK_1676(NodeStoreConfigTest.groovy:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
[jira] [Updated] (OAK-2615) oak-pojosr tests fail
[
https://issues.apache.org/jira/browse/OAK-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-2615:
---
Fix Version/s: 1.1.8
oak-pojosr tests fail
-
Key: OAK-2615
URL: https://issues.apache.org/jira/browse/OAK-2615
Project: Jackrabbit Oak
Issue Type: Bug
Components: pojosr
Reporter: Michael Dürig
Assignee: Chetan Mehrotra
Fix For: 1.1.8
The build fails for me in {{oak-pojosr}} with many failures like:
{noformat}
Running org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.142 sec
FAILURE!
testNodeStoreWithBlobStore_OAK_1676(org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest)
Time elapsed: 10.138 sec FAILURE!
java.lang.AssertionError: No service found for
org.apache.jackrabbit.oak.spi.state.NodeStore. Expression: sr. Values: sr =
null
at
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:400)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:646)
at
org.apache.jackrabbit.oak.run.osgi.AbstractRepositoryFactoryTest.getServiceWithWait(AbstractRepositoryFactoryTest.groovy:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at
org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest.testNodeStoreWithBlobStore_OAK_1676(NodeStoreConfigTest.groovy:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
[jira] [Commented] (OAK-2596) more (jmx) instrumentation for observation queue
[ https://issues.apache.org/jira/browse/OAK-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358626#comment-14358626 ] Michael Dürig commented on OAK-2596: I also removed the 1.0.13 fix version as I think we should only merge this into the branch when we have a hard requirement. more (jmx) instrumentation for observation queue Key: OAK-2596 URL: https://issues.apache.org/jira/browse/OAK-2596 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.0.12 Reporter: Stefan Egli Assignee: Michael Dürig Labels: monitoring, observation Fix For: 1.1.9 While debugging issues with the observation queue it would be handy to have more detailed information available. At the moment you can only see one value wrt length of the queue: that is the maximum of all queues. It is unclear if the queue is that long for only one or many listeners. And it is unclear from that if the listener is slow or the engine that produces the events for the listener. So I'd suggest to add the following details - possible exposed via JMX? : # add queue length details to each of the observation listeners # have a history of the last, eg 1000 events per listener showing a) how long the event took to be created/generated and b) how long the listener took to process. Sometimes averages are not detailed enough so such a in-depth information might become useful. (Not sure about the feasibility of '1000' here - maybe that could be configurable though - just putting the idea out here). # have some information about whether a listener is currently 'reading events from the cache' or whether it has to go to eg mongo # maybe have a 'top 10' listeners that have the largest queue at the moment to easily allow navigation instead of having to go through all (eg 200) listeners manually each time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358655#comment-14358655 ] Michael Dürig commented on OAK-2476: To add to that we should have 1.0 vs trunk as an further dimension. Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358704#comment-14358704
]
Thomas Mueller commented on OAK-2557:
-
I made some code coverage tests, and it looks very good to me. What is not
covered by deleteLargeNumber are:
{noformat}
collectDeletedDocuments
..
docIdsToDelete.add(prevDoc.getId()); this
..
if (docIdsToDelete.isEmpty()){
return; this
}
..
if (log.isDebugEnabled() docIdsToDelete.getSize() 1000) {
.. this
}
{noformat}
The log part has a side effect (docIdsToDelete.getIds(), which might open new
files)
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358632#comment-14358632
]
Thomas Mueller commented on OAK-2557:
-
I assume the document ids can not contain strange characters (newlines), right?
I noticed with debug level logging, the removed entries are logged, but only up
to 1000. Maybe it's better to log in all cases? The list is iterated over twice
in this case (for logging and for removing), but the logic looks good to me.
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358639#comment-14358639
]
Thomas Mueller commented on OAK-2557:
-
[~egli], I think createTempFile can not create directories (which is what we
need here).
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2617) Provide config parameter for timing out longrunning/inefficient queries
[ https://issues.apache.org/jira/browse/OAK-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2617: --- Assignee: Stefan Egli Provide config parameter for timing out longrunning/inefficient queries --- Key: OAK-2617 URL: https://issues.apache.org/jira/browse/OAK-2617 Project: Jackrabbit Oak Issue Type: Improvement Components: mongomk Affects Versions: 1.0.12 Reporter: Stefan Egli Assignee: Stefan Egli Fix For: 1.0.13 Attachments: OAK-2617.patch There have been cases where mongo queries where running for a very long time due to inefficient use of indexes or wrong indexes - mostly due to resulting traversals of loads of child nodes. While a proper fix for this is likely to be found in properly hinting at the right index (or by not hinting at all and letting mongo decide) - it might nevertheless be a good fallback plan to let queries not run for an infinite amount of time in case 'something went wrong/inefficient'. For that, mongo provides a timeout for queries, see [here|http://mongodb.github.io/node-mongodb-native/driver-articles/anintroductionto1_4_and_2_6.html#maxtimems]. The suggestion is to make use of this by providing a system property that would set this. I'll come up with a patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2617) Provide config parameter for timing out longrunning/inefficient queries
[ https://issues.apache.org/jira/browse/OAK-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Egli updated OAK-2617: - Attachment: OAK-2617.patch Attached [^OAK-2617.patch] with a suggested implementation. [~chetanm], [~mreutegg], could you pls have a look? thx! Provide config parameter for timing out longrunning/inefficient queries --- Key: OAK-2617 URL: https://issues.apache.org/jira/browse/OAK-2617 Project: Jackrabbit Oak Issue Type: Improvement Components: mongomk Affects Versions: 1.0.12 Reporter: Stefan Egli Assignee: Stefan Egli Fix For: 1.0.13 Attachments: OAK-2617.patch There have been cases where mongo queries where running for a very long time due to inefficient use of indexes or wrong indexes - mostly due to resulting traversals of loads of child nodes. While a proper fix for this is likely to be found in properly hinting at the right index (or by not hinting at all and letting mongo decide) - it might nevertheless be a good fallback plan to let queries not run for an infinite amount of time in case 'something went wrong/inefficient'. For that, mongo provides a timeout for queries, see [here|http://mongodb.github.io/node-mongodb-native/driver-articles/anintroductionto1_4_and_2_6.html#maxtimems]. The suggestion is to make use of this by providing a system property that would set this. I'll come up with a patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[
https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358637#comment-14358637
]
Davide Giannella commented on OAK-2476:
---
+1 for changing the variable if needed.
In terms of profiles as we discussed off ticket I would suggest to
- unittesting. Run once against all the fixtures it never gave any problem.
{{-Punittesting -Dns-fixtures=}}
- split {{pedantic}} and {{integrationTesting}} if not an hassle.
- having the the IT split as following
- Segment_MK
- Document_MK, Document_NS
- Document_RDB.
Move our CI to Jenkins
--
Key: OAK-2476
URL: https://issues.apache.org/jira/browse/OAK-2476
Project: Jackrabbit Oak
Issue Type: Task
Reporter: Tommaso Teofili
Assignee: Tommaso Teofili
Priority: Critical
Labels: CI, build, infrastructure
Fix For: 1.1.8, 1.2
We should strive for stabilization of our CI setup, as of now we had Buildbot
and Travis.
It seems ASF Jenkins can perform jobs on different environments (*nix,
Windows and others) so we can evaluate that and check if it better address
our needs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358666#comment-14358666 ] Tommaso Teofili commented on OAK-2476: -- for that I'd probably opt for creating a different job (the same with different SVN path), to keep separate build histories, in the long run I expect branch 1.0 to be more stable and that should be reflected by the builds history (hopefully :) ). Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-2616) GCMonitorMBean#getRepositorySize reports incorrect values
Michael Dürig created OAK-2616: -- Summary: GCMonitorMBean#getRepositorySize reports incorrect values Key: OAK-2616 URL: https://issues.apache.org/jira/browse/OAK-2616 Project: Jackrabbit Oak Issue Type: Bug Components: core Reporter: Michael Dürig Assignee: Michael Dürig Fix For: 1.1.8, 1.0.13 The values of said method are of as the counter does not reset each seconds while at the same time the values reported to the counter are accumulated instead of sampled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-2617) Provide config parameter for timing out longrunning/inefficient queries
Stefan Egli created OAK-2617: Summary: Provide config parameter for timing out longrunning/inefficient queries Key: OAK-2617 URL: https://issues.apache.org/jira/browse/OAK-2617 Project: Jackrabbit Oak Issue Type: Improvement Components: mongomk Affects Versions: 1.0.12 Reporter: Stefan Egli Fix For: 1.0.13 There have been cases where mongo queries where running for a very long time due to inefficient use of indexes or wrong indexes - mostly due to resulting traversals of loads of child nodes. While a proper fix for this is likely to be found in properly hinting at the right index (or by not hinting at all and letting mongo decide) - it might nevertheless be a good fallback plan to let queries not run for an infinite amount of time in case 'something went wrong/inefficient'. For that, mongo provides a timeout for queries, see [here|http://mongodb.github.io/node-mongodb-native/driver-articles/anintroductionto1_4_and_2_6.html#maxtimems]. The suggestion is to make use of this by providing a system property that would set this. I'll come up with a patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-2617) Provide config parameter for timing out longrunning/inefficient queries
[ https://issues.apache.org/jira/browse/OAK-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger resolved OAK-2617. --- Resolution: Duplicate Fix Version/s: (was: 1.0.13) Thanks for the patch. I already created an issue earlier today: OAK-2614. Provide config parameter for timing out longrunning/inefficient queries --- Key: OAK-2617 URL: https://issues.apache.org/jira/browse/OAK-2617 Project: Jackrabbit Oak Issue Type: Improvement Components: mongomk Affects Versions: 1.0.12 Reporter: Stefan Egli Assignee: Stefan Egli Attachments: OAK-2617.patch There have been cases where mongo queries where running for a very long time due to inefficient use of indexes or wrong indexes - mostly due to resulting traversals of loads of child nodes. While a proper fix for this is likely to be found in properly hinting at the right index (or by not hinting at all and letting mongo decide) - it might nevertheless be a good fallback plan to let queries not run for an infinite amount of time in case 'something went wrong/inefficient'. For that, mongo provides a timeout for queries, see [here|http://mongodb.github.io/node-mongodb-native/driver-articles/anintroductionto1_4_and_2_6.html#maxtimems]. The suggestion is to make use of this by providing a system property that would set this. I'll come up with a patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-2618) Improve performance of queries with ORDER BY and multiple OR filters
Amit Jain created OAK-2618: -- Summary: Improve performance of queries with ORDER BY and multiple OR filters Key: OAK-2618 URL: https://issues.apache.org/jira/browse/OAK-2618 Project: Jackrabbit Oak Issue Type: Improvement Components: query Reporter: Amit Jain Fix For: 1.4 When multiple OR constraints are specified in the XPATH query, itis broken up into union of multiple clauses. If query includes an order by clause, the sorting in this case is done by traversing the result set in memory leading to slow query performance. Possible improvements could include: * For indexes which can support multiple filters (like lucene, solr) such queries should be efficient and the query engine can pass-thru the query as is. ** Possibly also needed for other cases also. So, we can have some sort of capability advertiser for indexes which can hint the query engine and/or * Batched merging of the sorted iterators returned for the multiple union queries (possible externally). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358743#comment-14358743 ] Davide Giannella commented on OAK-2476: --- keep in mind that for 1.0 we don't have the concept of fixtures in the build. It was not backported. So for 1.0 it will be a simple mvn clean verify/install -PintegrationTesting,pedantic (to be double-checked) Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358678#comment-14358678 ] Michael Dürig commented on OAK-2476: +1 Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358718#comment-14358718
]
Stefan Egli commented on OAK-2557:
--
bq. [~tmueller] I think createTempFile can not create directories (which is
what we need here).
[right|http://stackoverflow.com/questions/617414/create-a-temporary-directory-in-java],
I see
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-2616) GCMonitorMBean#getRepositorySize reports incorrect values
[ https://issues.apache.org/jira/browse/OAK-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig resolved OAK-2616. Resolution: Fixed Fixed in trunk at http://svn.apache.org/r1666177 and in the 1.0 branch at http://svn.apache.org/r1666179 GCMonitorMBean#getRepositorySize reports incorrect values - Key: OAK-2616 URL: https://issues.apache.org/jira/browse/OAK-2616 Project: Jackrabbit Oak Issue Type: Bug Components: core Reporter: Michael Dürig Assignee: Michael Dürig Fix For: 1.1.8, 1.0.13 The values of said method are of as the counter does not reset each seconds while at the same time the values reported to the counter are accumulated instead of sampled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358649#comment-14358649 ] Tommaso Teofili commented on OAK-2476: -- thanks [~edivad], I'll try to setup what you suggest. In the meanwhile I've changed the _ns-fixtures_ variable to _nsfixtures_ in r1666181. Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358742#comment-14358742
]
Michael Dürig commented on OAK-2557:
You can do
{code}
directory = File.createTempFile();
directory.delete();
directory.mkdir();
{code}
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358694#comment-14358694
]
Chetan Mehrotra commented on OAK-2557:
--
bq. I noticed with debug level logging, the removed entries are logged, but
only up to 1000
Can do that. But like in current case the list might be pretty big 12M. So
avoided flooding the logs
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358699#comment-14358699
]
Thomas Mueller commented on OAK-2557:
-
I think large log files are acceptable, maybe it could help us see the progress?
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2623) Add test for GC of previous docs of a deleted document
Chetan Mehrotra created OAK-2623: Summary: Add test for GC of previous docs of a deleted document Key: OAK-2623 URL: https://issues.apache.org/jira/browse/OAK-2623 Project: Jackrabbit Oak Issue Type: Task Components: mongomk Reporter: Chetan Mehrotra Assignee: Chetan Mehrotra Priority: Minor Fix For: 1.2 As [noted|https://issues.apache.org/jira/browse/OAK-2557?focusedCommentId=14358704#comment-14358704] by [~tmueller] we do not have a test coverage for case when a deleted document has previous documents and Version GC should also remove those previous documents -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-2615) oak-pojosr tests fail
[
https://issues.apache.org/jira/browse/OAK-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-2615.
--
Resolution: Fixed
oak-pojosr tests fail
-
Key: OAK-2615
URL: https://issues.apache.org/jira/browse/OAK-2615
Project: Jackrabbit Oak
Issue Type: Bug
Components: pojosr
Reporter: Michael Dürig
Assignee: Chetan Mehrotra
Fix For: 1.1.8
The build fails for me in {{oak-pojosr}} with many failures like:
{noformat}
Running org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.142 sec
FAILURE!
testNodeStoreWithBlobStore_OAK_1676(org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest)
Time elapsed: 10.138 sec FAILURE!
java.lang.AssertionError: No service found for
org.apache.jackrabbit.oak.spi.state.NodeStore. Expression: sr. Values: sr =
null
at
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:400)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:646)
at
org.apache.jackrabbit.oak.run.osgi.AbstractRepositoryFactoryTest.getServiceWithWait(AbstractRepositoryFactoryTest.groovy:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at
org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest.testNodeStoreWithBlobStore_OAK_1676(NodeStoreConfigTest.groovy:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
[jira] [Commented] (OAK-2615) oak-pojosr tests fail
[
https://issues.apache.org/jira/browse/OAK-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359947#comment-14359947
]
Chetan Mehrotra commented on OAK-2615:
--
Checked the unit-test.log which indicate the dependency on commons-io. So yes
adding that as a dependency would be the proper fix
{noformat}
13.03.2015 10:35:47.095 *ERROR* [main] org.apache.jackrabbit.oak-core
[org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService(18)] The
activate method has thrown an exception (java.lang.NoClassDefFoundError:
org/apache/commons/io/Charsets)
java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
at
org.apache.jackrabbit.oak.plugins.segment.file.ReversedLinesFileReader.init(ReversedLinesFileReader.java:150)
~[oak-core-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at
org.apache.jackrabbit.oak.plugins.segment.file.ReversedLinesFileReader.init(ReversedLinesFileReader.java:67)
~[oak-core-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at
org.apache.jackrabbit.oak.plugins.segment.file.JournalReader.init(JournalReader.java:40)
~[oak-core-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.init(FileStore.java:368)
~[oak-core-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.init(FileStore.java:82)
~[oak-core-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore$Builder.create(FileStore.java:288)
~[oak-core-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.registerSegmentStore(SegmentNodeStoreService.java:322)
~[oak-core-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.registerNodeStore(SegmentNodeStoreService.java:236)
~[oak-core-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.activate(SegmentNodeStoreService.java:231)
~[oak-core-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
{noformat}
oak-pojosr tests fail
-
Key: OAK-2615
URL: https://issues.apache.org/jira/browse/OAK-2615
Project: Jackrabbit Oak
Issue Type: Bug
Components: pojosr
Reporter: Michael Dürig
Assignee: Chetan Mehrotra
Fix For: 1.1.8
The build fails for me in {{oak-pojosr}} with many failures like:
{noformat}
Running org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.142 sec
FAILURE!
testNodeStoreWithBlobStore_OAK_1676(org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest)
Time elapsed: 10.138 sec FAILURE!
java.lang.AssertionError: No service found for
org.apache.jackrabbit.oak.spi.state.NodeStore. Expression: sr. Values: sr =
null
at
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:400)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:646)
at
org.apache.jackrabbit.oak.run.osgi.AbstractRepositoryFactoryTest.getServiceWithWait(AbstractRepositoryFactoryTest.groovy:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:207)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at
org.apache.jackrabbit.oak.run.osgi.NodeStoreConfigTest.testNodeStoreWithBlobStore_OAK_1676(NodeStoreConfigTest.groovy:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359898#comment-14359898
]
Chetan Mehrotra commented on OAK-2557:
--
Minor nit pick there :)
Above approach creates a potential security risk as per [Guava
Files|http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/io/Files.html#createTempDir%28%29].
Though those risks do not apply in our usecase!
{quote}
Use this method instead of File.createTempFile(String, String) when you wish to
create a directory, not a regular file. A common pitfall is to call
createTempFile, delete the file and create a directory in its place, but this
leads a race condition which can be exploited to create security
vulnerabilities, especially when executable files are to be written into the
directory.
{quote}
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.1.8, 1.0.13
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359896#comment-14359896
]
Chetan Mehrotra commented on OAK-2557:
--
bq. maybe it could help us see the progress
That would be useful. Changed the impl a bit with
http://svn.apache.org/r1666352. Would that meet the requirements
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.1.8, 1.0.13
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359903#comment-14359903
]
Chetan Mehrotra commented on OAK-2557:
--
bq. docIdsToDelete.add(prevDoc.getId()); this
Thats important and should be covered by a test. Opened OAK-2623 for that
bq. The log part has a side effect (docIdsToDelete.getIds(), which might open
new files)
Changed the log part to not have that side effect
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.1.8, 1.0.13
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OAK-1826) Empty directories not cleaned up when gc run on FileDataStore
[ https://issues.apache.org/jira/browse/OAK-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Jain reassigned OAK-1826: -- Assignee: Amit Jain Empty directories not cleaned up when gc run on FileDataStore - Key: OAK-1826 URL: https://issues.apache.org/jira/browse/OAK-1826 Project: Jackrabbit Oak Issue Type: Bug Components: blob Reporter: Amit Jain Assignee: Amit Jain Priority: Minor Fix For: 1.1.9 The garbage collection only deletes the particular files identified as garbage. Any empty directories remaining after this operation are not cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-1826) Empty directories not cleaned up when gc run on FileDataStore
[ https://issues.apache.org/jira/browse/OAK-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Jain updated OAK-1826: --- Fix Version/s: 1.1.9 Empty directories not cleaned up when gc run on FileDataStore - Key: OAK-1826 URL: https://issues.apache.org/jira/browse/OAK-1826 Project: Jackrabbit Oak Issue Type: Bug Components: blob Reporter: Amit Jain Assignee: Amit Jain Priority: Minor Fix For: 1.1.9 The garbage collection only deletes the particular files identified as garbage. Any empty directories remaining after this operation are not cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-2614) Timeout for MongoDB query
[
https://issues.apache.org/jira/browse/OAK-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger resolved OAK-2614.
---
Resolution: Fixed
Introduced a system property as suggested by [~egli] in OAK-2617.
The system property is named {{oak.mongo.maxQueryTimeMS}} and is a timeout in
milliseconds.
Done in trunk: http://svn.apache.org/r1666218
Timeout for MongoDB query
-
Key: OAK-2614
URL: https://issues.apache.org/jira/browse/OAK-2614
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core, mongomk
Reporter: Marcel Reutegger
Assignee: Marcel Reutegger
Fix For: 1.1.8
MongoDB 2.6 introduced a query timeout which allows a client to specify a
time how long a query is allowed to execute.
I'd like to introduce this as a configurable feature to better control
resource usage in MongoDB and stop too expensive queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger resolved OAK-2557.
---
Resolution: Fixed
Fix Version/s: (was: 1.2)
1.1.8
Chetan committed the patch to trunk: http://svn.apache.org/r1666220
and merged it to 1.0: http://svn.apache.org/r1666221
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.1.8, 1.0.13
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358854#comment-14358854 ] Tommaso Teofili commented on OAK-2476: -- I've had a couple of additional issues with Jenkins not being able to couple with parenthesis and commas in axis values, anyway the basic configuration should be ok. For the record during the [currently running build|https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/11/] at lest one job running on Windows failed because of the Jenkins slave disk being full, while the one(s) running on Ubuntu seem to be going ok. Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358898#comment-14358898 ] Tommaso Teofili commented on OAK-2476: -- Also one more thing to add is trigger notifications to oak-dev@ as soon as this looks stable and reliable enough to start bothering us each day :) Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358854#comment-14358854 ] Tommaso Teofili edited comment on OAK-2476 at 3/12/15 3:53 PM: --- I've had a couple of additional issues with Jenkins not being able to cope with parenthesis and commas in axis values, anyway the basic configuration should be ok. For the record during the [currently running build|https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/11/] at lest one job running on Windows failed because of the Jenkins slave disk being full, while the one(s) running on Ubuntu seem to be going ok. Once the build configuration has proved to be ok, I'll remove the previously created Jenkins jobs (one for each JDK). was (Author: teofili): I've had a couple of additional issues with Jenkins not being able to cope with parenthesis and commas in axis values, anyway the basic configuration should be ok. For the record during the [currently running build|https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/11/] at lest one job running on Windows failed because of the Jenkins slave disk being full, while the one(s) running on Ubuntu seem to be going ok. Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358854#comment-14358854 ] Tommaso Teofili edited comment on OAK-2476 at 3/12/15 3:51 PM: --- I've had a couple of additional issues with Jenkins not being able to cope with parenthesis and commas in axis values, anyway the basic configuration should be ok. For the record during the [currently running build|https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/11/] at lest one job running on Windows failed because of the Jenkins slave disk being full, while the one(s) running on Ubuntu seem to be going ok. was (Author: teofili): I've had a couple of additional issues with Jenkins not being able to couple with parenthesis and commas in axis values, anyway the basic configuration should be ok. For the record during the [currently running build|https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/11/] at lest one job running on Windows failed because of the Jenkins slave disk being full, while the one(s) running on Ubuntu seem to be going ok. Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[
https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358954#comment-14358954
]
Michael Dürig commented on OAK-2476:
I just realised that the Maven profile configuration didn't work as expected.
The builds would haven been started with {{mvn clean install
-Dprofile=unittesting}} instead of {{mvn clean install -Punittesting}}.
Together with [~mirko] we hopefully figured it out, changed the configuration
and restarted the job.
Move our CI to Jenkins
--
Key: OAK-2476
URL: https://issues.apache.org/jira/browse/OAK-2476
Project: Jackrabbit Oak
Issue Type: Task
Reporter: Tommaso Teofili
Assignee: Tommaso Teofili
Priority: Critical
Labels: CI, build, infrastructure
Fix For: 1.1.8, 1.2
We should strive for stabilization of our CI setup, as of now we had Buildbot
and Travis.
It seems ASF Jenkins can perform jobs on different environments (*nix,
Windows and others) so we can evaluate that and check if it better address
our needs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2619) Support merging content during upgrade
Julian Sedding created OAK-2619: --- Summary: Support merging content during upgrade Key: OAK-2619 URL: https://issues.apache.org/jira/browse/OAK-2619 Project: Jackrabbit Oak Issue Type: New Feature Components: upgrade Affects Versions: 1.1.7 Reporter: Julian Sedding Priority: Minor When upgrading from Jackrabbit 2 to Oak there are several scenarios that could benefit from the ability to merge content rather than overwrite it. Especially in combination with OAK-2586, i.e. support to include/exclude selected paths from the copy operation, merging can become very useful. # Start vanilla product with an empty repo that writes some initial content, then copy content from a Jackrabbit 2 repo into this instance # Unify content from several Jackrabbit 2 repositories into a single Oak repo # Copy all content 1 week before the actual migration, then merge in the diff on migration day -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358967#comment-14358967 ] Tommaso Teofili commented on OAK-2476: -- thanks Mirko and Michael for fixing it, I had thought that would have been behaving the same way. Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2287) ConflictHandler for merging jcr:lastModified
[ https://issues.apache.org/jira/browse/OAK-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2287: --- Attachment: OAK-2287-3.patch Updated patch: [^OAK-2287-3.patch]: * A bit more test coverage and a bit of cleanup * I removed the conflict resolution for the said cases where the conflict is not between actual values but between deletion and values. I think this case is best left at the default behaviour. ConflictHandler for merging jcr:lastModified Key: OAK-2287 URL: https://issues.apache.org/jira/browse/OAK-2287 Project: Jackrabbit Oak Issue Type: Improvement Components: core, jcr Reporter: Alex Parvulescu Assignee: Alex Parvulescu Priority: Minor Fix For: 1.2 Attachments: OAK-2287-2.patch, OAK-2287-3.patch, OAK-2287.patch As discussed on oak-dev, 'jcr:lastModified' is a date property and it represents the last modification date. It makes sense to merge concurrent updates by picking the newer date of the 2 options. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-2622) dynamic cache allocation
Stefan Egli created OAK-2622: Summary: dynamic cache allocation Key: OAK-2622 URL: https://issues.apache.org/jira/browse/OAK-2622 Project: Jackrabbit Oak Issue Type: Improvement Components: mongomk Affects Versions: 1.0.12 Reporter: Stefan Egli Fix For: 1.4 At the moment mongoMk's various caches are configurable (OAK-2546) but other than that static in terms of size. Different use-cases might require different allocations of the sub caches though. And it might not always be possible to find a good configuration upfront for all use cases. We might be able to come up with a dynamically allocating the overall cache size to the different sub-caches, based on which cache is how heavily loaded for example. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2546) allow configuring subcaches of documentnodestore individually
[ https://issues.apache.org/jira/browse/OAK-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359426#comment-14359426 ] Stefan Egli commented on OAK-2546: -- Created follow-up idea 'dynamic cache allocation' in OAK-2622 allow configuring subcaches of documentnodestore individually - Key: OAK-2546 URL: https://issues.apache.org/jira/browse/OAK-2546 Project: Jackrabbit Oak Issue Type: Improvement Components: mongomk Affects Versions: 1.0.11, 1.1.6 Reporter: Stefan Egli Assignee: Marcel Reutegger Fix For: 1.0.12, 1.1.7 Currently there's one config for the DocumentNodeStore's (actually DocumentMk) caches (Documents, Diff, DocChildren, NodeChildren, NodeState) as a whole, and it distributes to each of those subcaches in a [hard-coded way|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentMK.java#L732]. It would be good to be able to configure the individual sub-caches individually (be it in percentage or absolute) The reason I bring this up is where I think it might be useful to try out giving the NodeChildren relatively more cache as it turned out during a heavy-load test that observation queues were starting to grow at the same point in time when the NodeChildren cache started to perform badly. While the problem there is relative bad performance of one or a few observation listeners, it could nevertheless help bridge a situation when the NodeChildren cache contains a longer 'history' so that the slow listener could catch up. PS: Leaving it up to discussion as to whether this goes into 1.0 branch or not -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2622) dynamic cache allocation
[ https://issues.apache.org/jira/browse/OAK-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Egli updated OAK-2622: - Description: At the moment mongoMk's various caches are configurable (OAK-2546) but other than that static in terms of size. Different use-cases might require different allocations of the sub caches though. And it might not always be possible to find a good configuration upfront for all use cases. We might be able to come up with dynamically allocating the overall cache size to the different sub-caches, based on which cache is how heavily loaded for example. was: At the moment mongoMk's various caches are configurable (OAK-2546) but other than that static in terms of size. Different use-cases might require different allocations of the sub caches though. And it might not always be possible to find a good configuration upfront for all use cases. We might be able to come up with a dynamically allocating the overall cache size to the different sub-caches, based on which cache is how heavily loaded for example. dynamic cache allocation Key: OAK-2622 URL: https://issues.apache.org/jira/browse/OAK-2622 Project: Jackrabbit Oak Issue Type: Improvement Components: mongomk Affects Versions: 1.0.12 Reporter: Stefan Egli Fix For: 1.4 At the moment mongoMk's various caches are configurable (OAK-2546) but other than that static in terms of size. Different use-cases might require different allocations of the sub caches though. And it might not always be possible to find a good configuration upfront for all use cases. We might be able to come up with dynamically allocating the overall cache size to the different sub-caches, based on which cache is how heavily loaded for example. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2622) dynamic cache allocation
[ https://issues.apache.org/jira/browse/OAK-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Egli updated OAK-2622: - Description: At the moment mongoMk's various caches are configurable (OAK-2546) but other than that static in terms of size. Different use-cases might require different allocations of the sub caches though. And it might not always be possible to find a good configuration upfront for all use cases. We might be able to come up with dynamically allocating the overall cache size to the different sub-caches, based on which cache is how heavily loaded or how well performing for example. was: At the moment mongoMk's various caches are configurable (OAK-2546) but other than that static in terms of size. Different use-cases might require different allocations of the sub caches though. And it might not always be possible to find a good configuration upfront for all use cases. We might be able to come up with dynamically allocating the overall cache size to the different sub-caches, based on which cache is how heavily loaded for example. dynamic cache allocation Key: OAK-2622 URL: https://issues.apache.org/jira/browse/OAK-2622 Project: Jackrabbit Oak Issue Type: Improvement Components: mongomk Affects Versions: 1.0.12 Reporter: Stefan Egli Fix For: 1.4 At the moment mongoMk's various caches are configurable (OAK-2546) but other than that static in terms of size. Different use-cases might require different allocations of the sub caches though. And it might not always be possible to find a good configuration upfront for all use cases. We might be able to come up with dynamically allocating the overall cache size to the different sub-caches, based on which cache is how heavily loaded or how well performing for example. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2614) Timeout for MongoDB query
[ https://issues.apache.org/jira/browse/OAK-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger updated OAK-2614: -- Fix Version/s: 1.0.13 Merged into 1.0 branch: http://svn.apache.org/r1666223 Timeout for MongoDB query - Key: OAK-2614 URL: https://issues.apache.org/jira/browse/OAK-2614 Project: Jackrabbit Oak Issue Type: Improvement Components: core, mongomk Reporter: Marcel Reutegger Assignee: Marcel Reutegger Fix For: 1.1.8, 1.0.13 MongoDB 2.6 introduced a query timeout which allows a client to specify a time how long a query is allowed to execute. I'd like to introduce this as a configurable feature to better control resource usage in MongoDB and stop too expensive queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (OAK-1981) Implement full scale Revision GC for DocumentNodeStore
[ https://issues.apache.org/jira/browse/OAK-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger reassigned OAK-1981: - Assignee: Marcel Reutegger Implement full scale Revision GC for DocumentNodeStore -- Key: OAK-1981 URL: https://issues.apache.org/jira/browse/OAK-1981 Project: Jackrabbit Oak Issue Type: New Feature Components: mongomk Reporter: Chetan Mehrotra Assignee: Marcel Reutegger Fix For: 1.1.9 So far we have implemented garbage collection in some form with OAK-1341. Those approaches help us remove quite a bit of garbage (mostly due to deleted nodes) but till some part is left However full GC is still not performed due to which some of the old revision related data cannot be GCed like * Revision info present in revision maps of various commit roots * Revision related to unmerged branches (OAK-1926) * Revision data created to property being modified by different cluster nodes So having a tool which can perform above GC would be helpful. For start we can have an implementation which takes a brute force approach and scans whole repo (would take quite a bit of time) and later we can evolve it. Or allow system admins to determine to what level GC has to be done -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-1981) Implement full scale Revision GC for DocumentNodeStore
[ https://issues.apache.org/jira/browse/OAK-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger updated OAK-1981: -- Fix Version/s: (was: 1.2) 1.1.9 Implement full scale Revision GC for DocumentNodeStore -- Key: OAK-1981 URL: https://issues.apache.org/jira/browse/OAK-1981 Project: Jackrabbit Oak Issue Type: New Feature Components: mongomk Reporter: Chetan Mehrotra Fix For: 1.1.9 So far we have implemented garbage collection in some form with OAK-1341. Those approaches help us remove quite a bit of garbage (mostly due to deleted nodes) but till some part is left However full GC is still not performed due to which some of the old revision related data cannot be GCed like * Revision info present in revision maps of various commit roots * Revision related to unmerged branches (OAK-1926) * Revision data created to property being modified by different cluster nodes So having a tool which can perform above GC would be helpful. For start we can have an implementation which takes a brute force approach and scans whole repo (would take quite a bit of time) and later we can evolve it. Or allow system admins to determine to what level GC has to be done -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-2621) Too many reads for child nodes
Marcel Reutegger created OAK-2621:
-
Summary: Too many reads for child nodes
Key: OAK-2621
URL: https://issues.apache.org/jira/browse/OAK-2621
Project: Jackrabbit Oak
Issue Type: Bug
Affects Versions: 1.0
Reporter: Marcel Reutegger
Priority: Minor
Fix For: 1.1.9
The DocumentNodeStore issues a lot of reads when sibling nodes are deleted,
which are also index with a property index.
The following calls will become a hotspot:
{noformat}
at
org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:406)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.readChildDocs(DocumentNodeStore.java:846)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.readChildren(DocumentNodeStore.java:788)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.getChildren(DocumentNodeStore.java:753)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeState.getChildNodeCount(DocumentNodeState.java:194)
at
org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.getChildNodeCount(ModifiedNodeState.java:198)
at
org.apache.jackrabbit.oak.plugins.memory.MutableNodeState.getChildNodeCount(MutableNodeState.java:265)
at
org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.getChildNodeCount(MemoryNodeBuilder.java:293)
at
org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.prune(ContentMirrorStoreStrategy.java:456)
{noformat}
I think the code triggering this issue is in
{{ModifiedNodeState.getChildNodeCount()}}. It keeps track of already deleted
children and requests {{max += deleted}}. The actual {{max}} is always 1 as
requested from {{ContentMirrorStoreStrategy.prune()}}, but as more nodes get
deleted, the higher {{max}} gets passed to
{{DocumentNodeState.getChildNodeCount()}}. The DocumentNodeStore then checks if
it has the children in the cache, only to find out the cache entry has too few
entries and it needs to fetch one more.
It would be best to have a minimum number of child nodes to fetch from MongoDB
in this case. E.g. when NodeState.getChildNodeEntries() is called, the
DocumentNodeState fetches 100 children.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2596) more (jmx) instrumentation for observation queue
[ https://issues.apache.org/jira/browse/OAK-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358624#comment-14358624 ] Michael Dürig commented on OAK-2596: Initial implementation at my Github fork: https://github.com/mduerig/jackrabbit-oak/tree/OAK-2596. You need a recent Jackrabbit 2.9-SNAPSHOT build for this. As this depends on the Jackrabbit 2.9.2 release I bump the fix version to 1.1.9 more (jmx) instrumentation for observation queue Key: OAK-2596 URL: https://issues.apache.org/jira/browse/OAK-2596 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.0.12 Reporter: Stefan Egli Assignee: Michael Dürig Labels: monitoring, observation Fix For: 1.0.13, 1.1.9 While debugging issues with the observation queue it would be handy to have more detailed information available. At the moment you can only see one value wrt length of the queue: that is the maximum of all queues. It is unclear if the queue is that long for only one or many listeners. And it is unclear from that if the listener is slow or the engine that produces the events for the listener. So I'd suggest to add the following details - possible exposed via JMX? : # add queue length details to each of the observation listeners # have a history of the last, eg 1000 events per listener showing a) how long the event took to be created/generated and b) how long the listener took to process. Sometimes averages are not detailed enough so such a in-depth information might become useful. (Not sure about the feasibility of '1000' here - maybe that could be configurable though - just putting the idea out here). # have some information about whether a listener is currently 'reading events from the cache' or whether it has to go to eg mongo # maybe have a 'top 10' listeners that have the largest queue at the moment to easily allow navigation instead of having to go through all (eg 200) listeners manually each time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2557) VersionGC uses way too much memory if there is a large pile of garbage
[
https://issues.apache.org/jira/browse/OAK-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358628#comment-14358628
]
Stefan Egli commented on OAK-2557:
--
Only a small comment/question: is there a reason you're doing the temp file
creation manually vs using {{java.io.File.createTempFile}} ?
VersionGC uses way too much memory if there is a large pile of garbage
--
Key: OAK-2557
URL: https://issues.apache.org/jira/browse/OAK-2557
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.0.11
Reporter: Stefan Egli
Assignee: Chetan Mehrotra
Priority: Blocker
Fix For: 1.0.13, 1.2
Attachments: OAK-2557-2.patch, OAK-2557-3.patch, OAK-2557.patch
It has been noticed that on a system where revision-gc
(VersionGarbageCollector of mongomk) did not run for a few days (due to not
interfering with some tests/large bulk operations) that there was such a
large pile of garbage accumulating, that the following code
{code}
VersionGarbageCollector.collectDeletedDocuments
{code}
in the for loop, creates such a large list of NodeDocuments to delete
(docIdsToDelete) that it uses up too much memory, causing the JVM's GC to
constantly spin in Full-GCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OAK-2500) checkDeepHistory/fixDeepHistory/prepareDeepHistory for oak-mongo.js
[ https://issues.apache.org/jira/browse/OAK-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger reassigned OAK-2500: - Assignee: Marcel Reutegger checkDeepHistory/fixDeepHistory/prepareDeepHistory for oak-mongo.js --- Key: OAK-2500 URL: https://issues.apache.org/jira/browse/OAK-2500 Project: Jackrabbit Oak Issue Type: Improvement Components: run Affects Versions: 1.0.8 Reporter: Stefan Egli Assignee: Marcel Reutegger Fix For: 1.0.13 Attachments: oak-mongo-mod.js The oak-mongo.js currently contains checkHistory/fixHistory which are cleaning up 'dangling revisions/split-documents' on a particular path. Now it would be good to have a command that goes through the entire repository and checks/fixes these dangling revisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2476) Move our CI to Jenkins
[ https://issues.apache.org/jira/browse/OAK-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358428#comment-14358428 ] Michael Dürig commented on OAK-2476: Thanks Tommaso! Feel free to contact [~mirko] if needed as he has a lot of Jenkins experience. Move our CI to Jenkins -- Key: OAK-2476 URL: https://issues.apache.org/jira/browse/OAK-2476 Project: Jackrabbit Oak Issue Type: Task Reporter: Tommaso Teofili Assignee: Tommaso Teofili Priority: Critical Labels: CI, build, infrastructure Fix For: 1.1.8, 1.2 We should strive for stabilization of our CI setup, as of now we had Buildbot and Travis. It seems ASF Jenkins can perform jobs on different environments (*nix, Windows and others) so we can evaluate that and check if it better address our needs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2612) Findbugs plugin version should depend on JDK version
[ https://issues.apache.org/jira/browse/OAK-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358424#comment-14358424 ] Michael Dürig commented on OAK-2612: I think we should drop it after 1.2. Let's the discussion up again for 1.3.x Findbugs plugin version should depend on JDK version Key: OAK-2612 URL: https://issues.apache.org/jira/browse/OAK-2612 Project: Jackrabbit Oak Issue Type: Bug Components: parent Affects Versions: 1.1.7 Reporter: Tommaso Teofili Assignee: Tommaso Teofili Fix For: 1.1.8 Looking at the current CI builds the JDK 8 one is always failing because _maven-findbugs-plugin-2.5.3_ doesn't work with it, on the other hand upgrading to 3.0.0 will make the JDK 6 build failing, therefore the version used has to be dependent on the used JDK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-1588) Create more tests/validation to LDAP integration
[
https://issues.apache.org/jira/browse/OAK-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angela updated OAK-1588:
Fix Version/s: (was: 1.2)
Create more tests/validation to LDAP integration
Key: OAK-1588
URL: https://issues.apache.org/jira/browse/OAK-1588
Project: Jackrabbit Oak
Issue Type: Bug
Components: core
Reporter: angela
Assignee: Tobias Bocanegra
Labels: test
this is a follow up issue for the remaining tasks as mentioned by [~tripod]
in OAK-516:
{quote}
[...] needs more testing, validation [...]
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2403) Improve monitoring capabilities for TarMk revision gc
[ https://issues.apache.org/jira/browse/OAK-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2403: --- Fix Version/s: (was: 1.0.13) Improve monitoring capabilities for TarMk revision gc - Key: OAK-2403 URL: https://issues.apache.org/jira/browse/OAK-2403 Project: Jackrabbit Oak Issue Type: Task Components: segmentmk Reporter: Michael Dürig Assignee: Michael Dürig Labels: gc, monitoring Fix For: 1.2 Container devoted to improving monitoring of the TarMk revision garbage collection process. The overall goal is to make it more transparent what revision gc does, how it performs, why it failed etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2408) Investigate ways to make revision gc more precise
[ https://issues.apache.org/jira/browse/OAK-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2408: --- Fix Version/s: (was: 1.0.13) Investigate ways to make revision gc more precise -- Key: OAK-2408 URL: https://issues.apache.org/jira/browse/OAK-2408 Project: Jackrabbit Oak Issue Type: Task Components: segmentmk Reporter: Michael Dürig Assignee: Michael Dürig Labels: gc Fix For: 1.2 Current approaches to revision garbage collection tend to be too conservative (too little space reclaimed, e.g. OAK-2045) or too aggressive (removing segments still being used, e.g. OAK-2384). This issue is to explore ways to make revision gc on TarMk more precise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2596) more (jmx) instrumentation for observation queue
[ https://issues.apache.org/jira/browse/OAK-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2596: --- Labels: monitoring observation (was: ) more (jmx) instrumentation for observation queue Key: OAK-2596 URL: https://issues.apache.org/jira/browse/OAK-2596 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.0.12 Reporter: Stefan Egli Assignee: Michael Dürig Labels: monitoring, observation Fix For: 1.0.13 While debugging issues with the observation queue it would be handy to have more detailed information available. At the moment you can only see one value wrt length of the queue: that is the maximum of all queues. It is unclear if the queue is that long for only one or many listeners. And it is unclear from that if the listener is slow or the engine that produces the events for the listener. So I'd suggest to add the following details - possible exposed via JMX? : * add queue length details to each of the observation listeners * have a history of the last, eg 1000 events per listener showing a) how long the event took to be created/generated and b) how long the listener took to process. Sometimes averages are not detailed enough so such a in-depth information might become useful. (Not sure about the feasibility of '1000' here - maybe that could be configurable though - just putting the idea out here). ** have some information about whether a listener is currently 'reading events from the cache' or whether it has to go to eg mongo * maybe have a 'top 10' listeners that have the largest queue at the moment to easily allow navigation instead of having to go through all (eg 200) listeners manually each time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-1410) Allow automatic propagation of oak-whiteboard when adding security manager
[ https://issues.apache.org/jira/browse/OAK-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355087#comment-14355087 ] angela commented on OAK-1410: - [~tripod], are you still planning to work on this or can this be resolved? Allow automatic propagation of oak-whiteboard when adding security manager -- Key: OAK-1410 URL: https://issues.apache.org/jira/browse/OAK-1410 Project: Jackrabbit Oak Issue Type: Bug Components: core Reporter: Tobias Bocanegra Assignee: Tobias Bocanegra Fix For: 1.2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2413) Clarify Editor.childNodeChanged()
[
https://issues.apache.org/jira/browse/OAK-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355136#comment-14355136
]
angela commented on OAK-2413:
-
[~mduerig], you are right... it actually assumes that there are changes. is
there an easy way to identify if there are changes? and how would the test look
like that illustrates the issue (such that i could verify that a potential fix
actually addresses the problem)?
Clarify Editor.childNodeChanged()
-
Key: OAK-2413
URL: https://issues.apache.org/jira/browse/OAK-2413
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core
Reporter: Marcel Reutegger
Assignee: Michael Dürig
Priority: Minor
Fix For: 1.2
The current contract for {{Editor.childNodeChanged()}} does not specify if
this method may also be called when the child node did not actually change.
The method {{NodeStateDiff.childNodeChanged()}} explicitly states that there
may be such calls. Looking at the implementation connecting the two classes,
{{EditorDiff.childNodeChange()}} simply calls the editor without checking
whether the child node did in fact change.
I think we either have to change the {{EditorDiff}} or update the contract
for the Editor and adjust implementations. E.g. right now, PrivilegeValidator
(implements Editor), assumes a call to {{childNodeChange()}} indeed means the
child node changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-1458) Out-of-process text extraction for better protection agains JVM/memory/CPU problems
[ https://issues.apache.org/jira/browse/OAK-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-1458: --- Fix Version/s: (was: 1.2) 1.3.0 Out-of-process text extraction for better protection agains JVM/memory/CPU problems --- Key: OAK-1458 URL: https://issues.apache.org/jira/browse/OAK-1458 Project: Jackrabbit Oak Issue Type: Improvement Components: oak-lucene, query Reporter: Michael Marth Labels: production, resilience Fix For: 1.3.0 This is a tracking / collection bug for solving problems with text extraction of documents (very large, broken, malicious, etc), causing JVM crashes, memory problems, excessive CPU usage. The basic TIKA feature to enable this fix is TIKA-416 [1] [1] https://issues.apache.org/jira/browse/TIKA-416 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-2587) observation processing too eager/unfair under load
[ https://issues.apache.org/jira/browse/OAK-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig resolved OAK-2587. Resolution: Fixed Fixed at http://svn.apache.org/r1665604 Let's create a separate issue for merging into 1.0 when necessary. observation processing too eager/unfair under load -- Key: OAK-2587 URL: https://issues.apache.org/jira/browse/OAK-2587 Project: Jackrabbit Oak Issue Type: Improvement Components: core Affects Versions: 1.0.12 Reporter: Stefan Egli Assignee: Michael Dürig Priority: Critical Labels: observation Fix For: 1.1.8 Attachments: OAK-2587.patch The current implementation of oak's observation event processing is too eager and thus unfair under load scenarios. Consider having many (eg 200) Eventlisteners but only a relatively small threadpool (eg 5 as is the default in sling) backing them. When processing changes for a particular BackgroundObserver, that one (in BackgroundObserver.completionHandler.call) currently processes *all changes irrespective of how many there are* - ie it is *eager*. Only once that BackgroundObserver processed all changes will it let go and 'pass the thread' to the next BackgroundObserver. Now if for some reason changes (ie commits) are coming in while a BackgroundObserver is busy processing an earlier change, this will lengthen that while loop. As a result the remaining (eg 195) *EventListeners will have to wait for a potentially long time* until it's their turn - thus *unfair*. Now combine the above pattern with a scenario where mongo is used as the underlying store. In that case in order to remain highly performant it is important that the diffs (for compareAgainstBaseState) are served from the MongoDiffCache for as many cases as possible to avoid doing a round-trip to mongoD. The unfairness in the BackgroundObservers can now result in a large delay between the 'first' observers getting the event and the 'last' one (of those 200). When this delay increases due to a burst in the load, there is a risk of the diffs to no longer be in the cache - those last observers are basically kicked out of the (diff) cache. Once this happens, *the situation gets even worse*, since now you have yet new commits coming in and old changes still having to be processed - all of which are being processed through in 'stripes of 5 listeners' before the next one gets a chance. This at some point results in a totally inefficient cache behavior, or in other words, at some point all diffs have to be read from mongoD. To avoid this there are probably a number of options - a few one that come to mind: * increase thread-pool to match or be closer to the number of listeners (but this has other disadvantages, eg cost of thread-switching) * make BackgroundObservers fairer by limiting the number of changes they process before they give others a chance to be served by the pool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-1962) Fix and Improve Locking
[ https://issues.apache.org/jira/browse/OAK-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-1962: --- Fix Version/s: (was: 1.1.8) 1.4 Fix and Improve Locking --- Key: OAK-1962 URL: https://issues.apache.org/jira/browse/OAK-1962 Project: Jackrabbit Oak Issue Type: Bug Components: core, jcr Affects Versions: 1.0 Reporter: angela Fix For: 1.4 as discussed during our biweekly planning session, we are having the following locking related use case in our product: a page (which is an aggregation of nodes and properties) must be freezed (prevented from being edited/modified/deleted) during the period of a dedicated review (which in our case is triggered by a workflow) before the page then is either published or reopened for further changes. for this feature locking would be a perfect match but with the current implementation in oak it's not possible to remove the lock token from a given session and make sure it's no longer recognizes as lock owner; in particular with the simple-locking feature which we introduced in order to cope with the situation that in our product sessions are living just for the time of a single request. is was therefore thinking about ways to address this, keeping the simple-locking requirement in mind while at the same time improving compliance with the JCR specification. my suggestion as follows: - use a dedicated hidden and write protected location the repository to store the lock information independent of the protected properties defined by mix:lockable which are just for information purpose (that would be the replacement for the lock-related file we had in jackrabbit 2.x) - format: simple lookup consisting of userId + associated lock tokens - in case of simple locking LockManager#getLockTokens would make use of that map even if the lock tokens have NOT been explicitly added to the Session either upon LockManager#lock or LockManager#addLockToken - in the default (non-simple case) LockManager#getLockTokens would work as specified and the lookup would only be used for maintaining and enforcing the lock related information) - LockManager#removeLockToken removes the token from the lookup thus effectively revoking the lock ownership from the given Session and thus preventing it from performing further editing... even in the simple-locking case - LockManager#addLockToken results in a modification of the lookup table as well depending on the API contract (i.e. verifying if the lock token would be shared... i don't remember exactly if that's accepted by the specification) - LockManager#isLockOwner no longer needs to perform a cheap hack comparing the jcr:lockOwner property with the sessions userId but could really enforce ownership based on the internal lock information stored in the repository in which case getLockTokens would really reflect the ownership for open-scoped locks; furthermore the 'jcr:lockOwner' information could be to what is specified by the API consumer upon LockManager#lock as it is no longer used for enforcing the locks. in addition: - we could generated proper lock tokens instead of just using the node id and again keep those in a hidden (or otherwise protected) location in the system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2602) [Solr] Cost calculation takes time with solr pings even when not fulfilling query
[ https://issues.apache.org/jira/browse/OAK-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommaso Teofili updated OAK-2602: - Fix Version/s: 1.0.13 1.1.8 [Solr] Cost calculation takes time with solr pings even when not fulfilling query - Key: OAK-2602 URL: https://issues.apache.org/jira/browse/OAK-2602 Project: Jackrabbit Oak Issue Type: Bug Components: oak-solr Affects Versions: 1.0.12, 1.1.7 Reporter: Amit Jain Assignee: Tommaso Teofili Fix For: 1.1.8, 1.0.13 Cost calculation for queries which are fired quite often [1] and which are not going to fulfilled by solr take time due to which the overall cost of the operation is high. [1] SELECT * FROM [nt:base] WHERE PROPERTY([rep:members], 'WeakReference') = $uuid SELECT * FROM [nt:base] WHERE [jcr:uuid] = $id -- This message was sent by Atlassian JIRA (v6.3.4#6332)
