[jira] [Resolved] (OAK-2655) Test failure: OrderableNodesTest.testAddNode
[
https://issues.apache.org/jira/browse/OAK-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke resolved OAK-2655.
-
Resolution: Fixed
Fix Version/s: (was: 1.4)
1.3.12
Removing the test exclusion for now. I currently can not reproduce the issue,
and there have been quite a few changes that may have fixed this.
trunk: http://svn.apache.org/r1716901
> Test failure: OrderableNodesTest.testAddNode
>
>
> Key: OAK-2655
> URL: https://issues.apache.org/jira/browse/OAK-2655
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: rdbmk
>Affects Versions: 1.0.12, 1.2
> Environment: Jenkins, Ubuntu:
> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/
>Reporter: Michael Dürig
>Assignee: Julian Reschke
> Labels: CI, Jenkins
> Fix For: 1.3.12
>
> Attachments: 2655.diff
>
>
> {{org.apache.jackrabbit.oak.jcr.OrderableNodesTest.testAddNode}} fails on
> Jenkins when running the {{DOCUMENT_RDB}} fixture.
> Failure seen at builds: 30, 44, 57, 58, 69, 78, 115, 121, 130, 132, 142, 152
> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/30/jdk=jdk-1.6u45,label=Ubuntu,nsfixtures=DOCUMENT_RDB,profile=unittesting/testReport/junit/org.apache.jackrabbit.oak.jcr/OrderableNodesTest/testAddNode_0_/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-3692) java.lang.NoClassDefFoundError: org/apache/lucene/index/sorter/Sorter$DocComparator
Vikas Saurabh created OAK-3692:
--
Summary: java.lang.NoClassDefFoundError:
org/apache/lucene/index/sorter/Sorter$DocComparator
Key: OAK-3692
URL: https://issues.apache.org/jira/browse/OAK-3692
Project: Jackrabbit Oak
Issue Type: Bug
Components: lucene
Reporter: Vikas Saurabh
Assignee: Tommaso Teofili
Priority: Blocker
I'm getting following exception while trying to include oak trunk build into
AEM:
{noformat}
27.11.2015 20:41:25.946 *ERROR* [oak-lucene-2]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider Uncaught
exception in
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider@36ba558b
java.lang.NoClassDefFoundError:
org/apache/lucene/index/sorter/Sorter$DocComparator
at
org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:108)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker$1.leave(IndexTracker.java:98)
at
org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:153)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:444)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:487)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord.compareBranch(MapRecord.java:565)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:470)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:436)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at
org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:403)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:444)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:487)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:436)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:394)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at
org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.update(IndexTracker.java:108)
at
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider.contentChanged(LuceneIndexProvider.java:73)
at
org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:131)
at
org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:125)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException:
org.apache.lucene.index.sorter.Sorter$DocComparator not found by
org.apache.jackrabbit.oak-lucene [95]
at
org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1573)
at
org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:79)
at
org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:2018)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 27 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3692) java.lang.NoClassDefFoundError: org/apache/lucene/index/sorter/Sorter$DocComparator
[
https://issues.apache.org/jira/browse/OAK-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh updated OAK-3692:
---
Fix Version/s: 1.3.12
> java.lang.NoClassDefFoundError:
> org/apache/lucene/index/sorter/Sorter$DocComparator
> ---
>
> Key: OAK-3692
> URL: https://issues.apache.org/jira/browse/OAK-3692
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Vikas Saurabh
>Assignee: Tommaso Teofili
>Priority: Blocker
> Fix For: 1.3.12
>
>
> I'm getting following exception while trying to include oak trunk build into
> AEM:
> {noformat}
> 27.11.2015 20:41:25.946 *ERROR* [oak-lucene-2]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider Uncaught
> exception in
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider@36ba558b
> java.lang.NoClassDefFoundError:
> org/apache/lucene/index/sorter/Sorter$DocComparator
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:108)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker$1.leave(IndexTracker.java:98)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:153)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:444)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:487)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compareBranch(MapRecord.java:565)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:470)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:436)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:403)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:444)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:487)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:436)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:394)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.update(IndexTracker.java:108)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider.contentChanged(LuceneIndexProvider.java:73)
> at
> org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:131)
> at
> org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:125)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.lucene.index.sorter.Sorter$DocComparator not found by
> org.apache.jackrabbit.oak-lucene [95]
> at
> org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1573)
> at
> org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:79)
> at
> org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:2018)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 27 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3691) RDBDocumentStore: refactor update logic
[ https://issues.apache.org/jira/browse/OAK-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke resolved OAK-3691. - Resolution: Fixed trunk: http://svn.apache.org/r1716883 1.2: http://svn.apache.org/r1716884 1.0: http://svn.apache.org/r1716890 > RDBDocumentStore: refactor update logic > --- > > Key: OAK-3691 > URL: https://issues.apache.org/jira/browse/OAK-3691 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: rdbmk >Affects Versions: 1.3.11, 1.2.8, 1.0.24 >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Fix For: 1.2.9, 1.0.25, 1.3.12 > > > The update logic currently conflates condition checks, counter maintenance > and application of updates. Untangle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3681) SegmentSizeTest.testAccessControlNodes() and testNodeSize() fail
[
https://issues.apache.org/jira/browse/OAK-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-3681:
---
Description:
Failed on travis: https://travis-ci.org/apache/jackrabbit-oak/builds/93310633
and also reproduces in my local checkout.
{code}
testAccessControlNodes(org.apache.jackrabbit.oak.plugins.segment.SegmentSizeTest)
Time elapsed: 0.003 sec <<< FAILURE!
junit.framework.AssertionFailedError: expected:<112> but was:<96>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentSizeTest.testAccessControlNodes(SegmentSizeTest.java:121)
testNodeSize(org.apache.jackrabbit.oak.plugins.segment.SegmentSizeTest) Time
elapsed: 0.001 sec <<< FAILURE!
junit.framework.AssertionFailedError: expected:<112> but was:<96>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentSizeTest.testNodeSize(SegmentSizeTest.java:43)
{code}
The test succeeds when I revert this commit: http://svn.apache.org/r1716472
was:
Failed on travis: https://travis-ci.org/apache/jackrabbit-oak/builds/93310633
and also reproduces in my local checkout.
The test succeeds when I revert this commit: http://svn.apache.org/r1716472
> SegmentSizeTest.testAccessControlNodes() and testNodeSize() fail
>
>
> Key: OAK-3681
> URL: https://issues.apache.org/jira/browse/OAK-3681
> Project: Jackrabbit Oak
> Issue Type: Test
> Components: core, segmentmk
> Environment: Oracle Java 1.8.0_65
>Reporter: Marcel Reutegger
>Assignee: Michael Dürig
>Priority: Trivial
>
> Failed on travis: https://travis-ci.org/apache/jackrabbit-oak/builds/93310633
> and also reproduces in my local checkout.
> {code}
> testAccessControlNodes(org.apache.jackrabbit.oak.plugins.segment.SegmentSizeTest)
> Time elapsed: 0.003 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<112> but was:<96>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:234)
> at junit.framework.Assert.assertEquals(Assert.java:241)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentSizeTest.testAccessControlNodes(SegmentSizeTest.java:121)
> testNodeSize(org.apache.jackrabbit.oak.plugins.segment.SegmentSizeTest) Time
> elapsed: 0.001 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<112> but was:<96>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:234)
> at junit.framework.Assert.assertEquals(Assert.java:241)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentSizeTest.testNodeSize(SegmentSizeTest.java:43)
> {code}
> The test succeeds when I revert this commit: http://svn.apache.org/r1716472
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3688) Provide and use a default set of bundle filters
[ https://issues.apache.org/jira/browse/OAK-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-3688. -- Resolution: Fixed Done with 1716835 > Provide and use a default set of bundle filters > --- > > Key: OAK-3688 > URL: https://issues.apache.org/jira/browse/OAK-3688 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: pojosr >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.3.12 > > > OAK-3194 enabled to control the set of bundles which need to be started while > running Oak. Given that a basic default set of bundle would always be > required it would be good to provide a sensible default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-3689) OakOSGiRepositoryFactory shutting down the repository twice
[ https://issues.apache.org/jira/browse/OAK-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-3689. -- Resolution: Fixed Done with 1716840,1716845 > OakOSGiRepositoryFactory shutting down the repository twice > --- > > Key: OAK-3689 > URL: https://issues.apache.org/jira/browse/OAK-3689 > Project: Jackrabbit Oak > Issue Type: Bug > Components: pojosr >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.3.12 > > > With OAK-3375 the repository instance would be shutdown by the > RepositoryManager itself. Currently OakOSGiRepositoryFactory also invokes the > shutdown method. > As the instance is owned by RepositoryManager it should not be shutdown in > OakOSGiRepositoryFactory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3303) FileStore flush thread can get stuck
[ https://issues.apache.org/jira/browse/OAK-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029858#comment-15029858 ] Thomas Mueller commented on OAK-3303: - An alternative explanation of the stuck thread: currently, the priority is set as follows: "setPriority(MIN_PRIORITY)". As it's used with a long delay anyway, I would probably not set the priority, but use the default. > stuck for a while Do you remember if the thread name was just "TarMK ... thread [...]", or that plus "active since" or plus "avg ... max ..."? If the thread name did contain "active since" or "avg ... max ...", then interval was not 0. But, to rule out the "interval" is not yet set, maybe the thread name could be set before the first iteration. > FileStore flush thread can get stuck > > > Key: OAK-3303 > URL: https://issues.apache.org/jira/browse/OAK-3303 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segmentmk >Reporter: Alex Parvulescu >Assignee: Alex Parvulescu > Fix For: 1.3.12 > > > In some very rare circumstances the flush thread was seen as possibly stuck > for a while following a restart of the system. This results in data loss on > restart (the system will roll back to the latest persisted revision on > restart), and worse off there's no way of extracting the latest head revision > using the tar files, so recovery is not (yet) possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-2833) Refactor TarMK
[
https://issues.apache.org/jira/browse/OAK-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14560848#comment-14560848
]
Michael Dürig edited comment on OAK-2833 at 11/27/15 1:21 PM:
--
We need to be careful going forward here in order not to break anything as this
is pretty much open heart surgery. Also we need take to be careful wrt. to
maintainability of our branches. The suggest approach is to break this apart
into smaller incremental improvements where each improvement would first
increase test coverage for the refactored part, do the refactoring and then
merge it back to the branches.
IMO a good place to start is to break up the {{SegmentWriter}} (see OAK-1828).
Based on the experience gathered we should then follow up with introducing dual
concepts for reading from segments instead of doing this ad-hoc and inline all
over the places.
cc [~alex.parvulescu], [~egli]
was (Author: mduerig):
We need to be careful going forward here in order not to break anything as this
is pretty much open hear surgery. Also we need take to be careful wrt. to
maintainability of our branches. The suggest approach is to break this apart
into smaller incremental improvements where each improvement would first
increase test coverage for the refactored part, do the refactoring and then
merge it back to the branches.
IMO a good place to start is to break up the {{SegmentWriter}} (see OAK-1828).
Based on the experience gathered we should then follow up with introducing dual
concepts for reading from segments instead of doing this ad-hoc and inline all
over the places.
cc [~alex.parvulescu], [~egli]
> Refactor TarMK
> --
>
> Key: OAK-2833
> URL: https://issues.apache.org/jira/browse/OAK-2833
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: technical_debt
> Fix For: 1.4
>
>
> Container issue for refactoring the TarMK to make it more testable,
> maintainable, extensible and less entangled.
> For example the segment format should be readable, writeable through
> standalone means so tests, tools and production code can share this code.
> Currently there is a lot of code duplication involved here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-1828) Improved SegmentWriter
[ https://issues.apache.org/jira/browse/OAK-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig resolved OAK-1828. Resolution: Fixed Resolving as fixed as with we addressed the bulk of what this issue was about. Created OAK-3690 to follow up on a remaining issue.. > Improved SegmentWriter > -- > > Key: OAK-1828 > URL: https://issues.apache.org/jira/browse/OAK-1828 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: segmentmk >Reporter: Jukka Zitting >Assignee: Alex Parvulescu >Priority: Minor > Labels: technical_debt > Fix For: 1.3.12 > > Attachments: record-writers-v0.patch, record-writers-v1.patch, > record-writers-v2.patch > > > At about 1kLOC and dozens of methods, the SegmentWriter class currently a bit > too complex for one of the key components of the TarMK. It also uses a > somewhat non-obvious mix of synchronized and unsynchronized code to > coordinate multiple concurrent threads that may be writing content at the > same time. The synchronization blocks are also broader than what really would > be needed, which in some cases causes unnecessary lock contention in > concurrent write loads. > To improve the readability and maintainability of the code, and to increase > performance of concurrent writes, it would be useful to split part of the > SegmentWriter functionality to a separate RecordWriter class that would be > responsible for writing individual records into a segment. The > SegmentWriter.prepare() method would return a new RecordWriter instance, and > the higher-level SegmentWriter methods would use the returned instance for > all the work that's currently guarded in synchronization blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3690) Decouple SegmentBufferWriter from SegmentStore
Michael Dürig created OAK-3690:
--
Summary: Decouple SegmentBufferWriter from SegmentStore
Key: OAK-3690
URL: https://issues.apache.org/jira/browse/OAK-3690
Project: Jackrabbit Oak
Issue Type: Technical task
Components: segmentmk
Reporter: Michael Dürig
Currently {{SegmentBufferWriter.flush()}} directly calls
{{SegmentStore.writeSegment()}} once the current segment does not have enough
space for the next record. We should try to cut this dependency as
{{SegmentBufferWriter}} should only be concerned with providing buffers for
segments. Actually writing these to the store should be handled by a higher
level component.
A number of deadlock (e.g. (OAK-2560, OAK-3179, OAK-3264) we have seen is one
manifestation of this troublesome dependency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3690) Decouple SegmentBufferWriter from SegmentStore
[
https://issues.apache.org/jira/browse/OAK-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029894#comment-15029894
]
Michael Dürig commented on OAK-3690:
One way of decoupling would be to hand a 'queue' to the {{SegmentBufferWriter}}
instead of the {{SegmentStore}}. In a first step that 'queue' would be directly
backed by the {{SegmentStore}}, thus apart from the abstraction noting changes.
In a next step that 'queue' could be replaced by a real producer/consumer queue
pushing segments from the {{SegmentBufferWritter}} to the {{SegmentStore}} on a
background thread (e.g. the flush thread).
> Decouple SegmentBufferWriter from SegmentStore
> --
>
> Key: OAK-3690
> URL: https://issues.apache.org/jira/browse/OAK-3690
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: segmentmk
>Reporter: Michael Dürig
> Labels: technical_debt
> Fix For: 1.4
>
>
> Currently {{SegmentBufferWriter.flush()}} directly calls
> {{SegmentStore.writeSegment()}} once the current segment does not have enough
> space for the next record. We should try to cut this dependency as
> {{SegmentBufferWriter}} should only be concerned with providing buffers for
> segments. Actually writing these to the store should be handled by a higher
> level component.
> A number of deadlock (e.g. (OAK-2560, OAK-3179, OAK-3264) we have seen is one
> manifestation of this troublesome dependency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-3691) RDBDocumentStore: refactor update logic
Julian Reschke created OAK-3691: --- Summary: RDBDocumentStore: refactor update logic Key: OAK-3691 URL: https://issues.apache.org/jira/browse/OAK-3691 Project: Jackrabbit Oak Issue Type: Technical task Components: rdbmk Affects Versions: 1.0.24, 1.2.8, 1.3.11 Reporter: Julian Reschke Assignee: Julian Reschke Priority: Minor The update logic currently conflates condition checks, counter maintenance and application of updates. Untangle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-3681) SegmentSizeTest.testAccessControlNodes() and testNodeSize() fail
[
https://issues.apache.org/jira/browse/OAK-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-3681.
Resolution: Fixed
Fix Version/s: 1.3.12
Fixed at http://svn.apache.org/viewvc?rev=1716870=rev
> SegmentSizeTest.testAccessControlNodes() and testNodeSize() fail
>
>
> Key: OAK-3681
> URL: https://issues.apache.org/jira/browse/OAK-3681
> Project: Jackrabbit Oak
> Issue Type: Test
> Components: core, segmentmk
> Environment: Oracle Java 1.8.0_65
>Reporter: Marcel Reutegger
>Assignee: Michael Dürig
>Priority: Trivial
> Fix For: 1.3.12
>
>
> Failed on travis: https://travis-ci.org/apache/jackrabbit-oak/builds/93310633
> and also reproduces in my local checkout.
> {code}
> testAccessControlNodes(org.apache.jackrabbit.oak.plugins.segment.SegmentSizeTest)
> Time elapsed: 0.003 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<112> but was:<96>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:234)
> at junit.framework.Assert.assertEquals(Assert.java:241)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentSizeTest.testAccessControlNodes(SegmentSizeTest.java:121)
> testNodeSize(org.apache.jackrabbit.oak.plugins.segment.SegmentSizeTest) Time
> elapsed: 0.001 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<112> but was:<96>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:234)
> at junit.framework.Assert.assertEquals(Assert.java:241)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentSizeTest.testNodeSize(SegmentSizeTest.java:43)
> {code}
> The test succeeds when I revert this commit: http://svn.apache.org/r1716472
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-3692) java.lang.NoClassDefFoundError: org/apache/lucene/index/sorter/Sorter$DocComparator
[
https://issues.apache.org/jira/browse/OAK-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15030306#comment-15030306
]
Vikas Saurabh edited comment on OAK-3692 at 11/27/15 11:15 PM:
---
Added dependency to lucene-misc for oak-lucene/pom.xml on trunk at
[r1716931|http://svn.apache.org/r1716931].
was (Author: catholicon):
Added dependency to lucene-misc for oak-lucene/pom.xml on trunk at
[r1716931|svn.apache.org/r1716931].
> java.lang.NoClassDefFoundError:
> org/apache/lucene/index/sorter/Sorter$DocComparator
> ---
>
> Key: OAK-3692
> URL: https://issues.apache.org/jira/browse/OAK-3692
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Blocker
> Fix For: 1.3.12
>
>
> I'm getting following exception while trying to include oak trunk build into
> AEM:
> {noformat}
> 27.11.2015 20:41:25.946 *ERROR* [oak-lucene-2]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider Uncaught
> exception in
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider@36ba558b
> java.lang.NoClassDefFoundError:
> org/apache/lucene/index/sorter/Sorter$DocComparator
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:108)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker$1.leave(IndexTracker.java:98)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:153)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:444)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:487)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compareBranch(MapRecord.java:565)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:470)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:436)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:403)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:444)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:487)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:436)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:394)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.update(IndexTracker.java:108)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider.contentChanged(LuceneIndexProvider.java:73)
> at
> org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:131)
> at
> org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:125)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.lucene.index.sorter.Sorter$DocComparator not found by
> org.apache.jackrabbit.oak-lucene [95]
> at
> org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1573)
> at
> org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:79)
> at
> org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:2018)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
[jira] [Resolved] (OAK-3692) java.lang.NoClassDefFoundError: org/apache/lucene/index/sorter/Sorter$DocComparator
[
https://issues.apache.org/jira/browse/OAK-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh resolved OAK-3692.
Resolution: Fixed
Assignee: Vikas Saurabh (was: Tommaso Teofili)
Added dependency to lucene-misc for oak-lucene/pom.xml on trunk at
[r1716931|svn.apache.org/r1716931].
> java.lang.NoClassDefFoundError:
> org/apache/lucene/index/sorter/Sorter$DocComparator
> ---
>
> Key: OAK-3692
> URL: https://issues.apache.org/jira/browse/OAK-3692
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Blocker
> Fix For: 1.3.12
>
>
> I'm getting following exception while trying to include oak trunk build into
> AEM:
> {noformat}
> 27.11.2015 20:41:25.946 *ERROR* [oak-lucene-2]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider Uncaught
> exception in
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider@36ba558b
> java.lang.NoClassDefFoundError:
> org/apache/lucene/index/sorter/Sorter$DocComparator
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:108)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker$1.leave(IndexTracker.java:98)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:153)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:444)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:487)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compareBranch(MapRecord.java:565)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:470)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:436)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:403)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:444)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:487)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:436)
> at
> org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:394)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.update(IndexTracker.java:108)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider.contentChanged(LuceneIndexProvider.java:73)
> at
> org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:131)
> at
> org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:125)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.lucene.index.sorter.Sorter$DocComparator not found by
> org.apache.jackrabbit.oak-lucene [95]
> at
> org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1573)
> at
> org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:79)
> at
> org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:2018)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 27 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3691) RDBDocumentStore: refactor update logic
[ https://issues.apache.org/jira/browse/OAK-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-3691: Fix Version/s: 1.3.12 > RDBDocumentStore: refactor update logic > --- > > Key: OAK-3691 > URL: https://issues.apache.org/jira/browse/OAK-3691 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: rdbmk >Affects Versions: 1.3.11, 1.2.8, 1.0.24 >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Fix For: 1.3.12 > > > The update logic currently conflates condition checks, counter maintenance > and application of updates. Untangle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3691) RDBDocumentStore: refactor update logic
[ https://issues.apache.org/jira/browse/OAK-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-3691: Fix Version/s: 1.2.9 > RDBDocumentStore: refactor update logic > --- > > Key: OAK-3691 > URL: https://issues.apache.org/jira/browse/OAK-3691 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: rdbmk >Affects Versions: 1.3.11, 1.2.8, 1.0.24 >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Fix For: 1.2.9, 1.3.12 > > > The update logic currently conflates condition checks, counter maintenance > and application of updates. Untangle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3691) RDBDocumentStore: refactor update logic
[ https://issues.apache.org/jira/browse/OAK-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-3691: Fix Version/s: (was: 1.2.9) 1.0.25 1.29 > RDBDocumentStore: refactor update logic > --- > > Key: OAK-3691 > URL: https://issues.apache.org/jira/browse/OAK-3691 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: rdbmk >Affects Versions: 1.3.11, 1.2.8, 1.0.24 >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Fix For: 1.0.25, 1.3.12, 1.29 > > > The update logic currently conflates condition checks, counter maintenance > and application of updates. Untangle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3691) RDBDocumentStore: refactor update logic
[ https://issues.apache.org/jira/browse/OAK-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-3691: Fix Version/s: 1.2.9 > RDBDocumentStore: refactor update logic > --- > > Key: OAK-3691 > URL: https://issues.apache.org/jira/browse/OAK-3691 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: rdbmk >Affects Versions: 1.3.11, 1.2.8, 1.0.24 >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Fix For: 1.2.9, 1.0.25, 1.3.12 > > > The update logic currently conflates condition checks, counter maintenance > and application of updates. Untangle this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3649) Extract node document cache from Mongo and RDB document stores
[ https://issues.apache.org/jira/browse/OAK-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029681#comment-15029681 ] Julian Reschke commented on OAK-3649: - For some operations, RDBDocumentStore also just sends the update to the DB (see the "appending" methods). WRT increasing modcount: this is just a readability issue. To make things easier to follow in both implementations, we may want to introduce (in both) a "applyCounters(UpdateOp)" method. In RDB, it would also increment the (experimental) CMODCOUNT. I'll look into refactoring this part in RDB; and maybe that can be used as guidance for Mongo. > Extract node document cache from Mongo and RDB document stores > -- > > Key: OAK-3649 > URL: https://issues.apache.org/jira/browse/OAK-3649 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: documentmk, mongomk, rdbmk >Reporter: Tomek Rękawek >Priority: Minor > Labels: candidate_oak_1_0, candidate_oak_1_2 > Fix For: 1.3.12 > > > MongoDocumentStore and RDBDocumentStore contains copy & pasted methods > responsible for handling node document cache. Extract these into a new > NodeDocumentCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2683) the "hitting the observation queue limit" problem
[ https://issues.apache.org/jira/browse/OAK-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029683#comment-15029683 ] Michael Dürig commented on OAK-2683: I implemented [something along this line|https://issues.apache.org/jira/browse/OAK-1368?focusedCommentId=14092732=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14092732] in OAK-1368 a long time ago but so far nobody dared to look ;-) Specifically: multiple listeners share the same diff (fewer diffs). Listeners falling behind too much will be forked off to a different queue fed by a separate diff. I think this approach could further be generalised to "only one observer per instance". > the "hitting the observation queue limit" problem > - > > Key: OAK-2683 > URL: https://issues.apache.org/jira/browse/OAK-2683 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, mongomk, segmentmk >Reporter: Stefan Egli > Labels: observation, resilience > > There are several tickets in this area: > * OAK-2587: threading with observation being too eagar causing observation > queue to grow > * OAK-2669: avoiding diffing from mongo by using persistent cache instead. > * OAK-2349: which might be a duplicate or at least similar to 2669.. > * OAK-2562: diffcache is inefficient > Yet I think it makes sense to create this summarizing ticket, about > describing again what happens when the observation queue hits the limit - and > eventually about how this can be improved > Consider the following scenario (also compare with OAK-2587 - but that one > focused more on eagerness of threading): > * rate of incoming commits is large and starts to generate many changes into > the observation queues, hence those queue become somewhat filled/loaded > * depending on the underlying nodestore used the calculation of diffs is more > or less expensive - but at least for mongomk it is important that the diff > can be served from the cache > ** in case of mongomk it can happen that diffs are no longer found in the > cache and thus require a round-trip to mongo - which is magnitudes slower > than via cache of course. this would result in the queue to start increasing > even faster as dequeuing becomes slower now. > ** not sure about tarmk - I believe it should always be fast there > * so based on the above, there can be a situation where the queue grows and > hits the configured limit > * if this limit is reached, the current mechanism is to collapse any > subsequent change into one-big-marked-as-external-event change, lets call > this a collapsed-change. > * this collapsed-change now becomes part of the normal queue and eventually > would 'walk down the queue' and be processed normally - hence opening a high > chance that yet a new collapsed-change is created should the queue just hit > the limit again. and this game can now be played for a while, resulting in > the queue to contain many/mostly such collapse-changes. > * there is now an additional assumption in that the diffing of such collapses > is more expensive than normal diffing - plus it is almost guaranteed that the > diff cannot for example be shared between observation listeners, since the > exact 'collapse borders' depends on timing of each of the listeners' queues - > ie the collapse diffs are unique thus not cachable.. > * so as a result: once you have those collapse-diffs you can almost not get > rid of them - they are heavy to process - hence dequeuing is very slow > * at the same time, there is always likely some commits happening in a > typical system, eg with sling on top you have sling discovery which does > heartbeats every now and then. So there's always new commits that add to the > load. > * this will hence create a situation where quite a small additional commit > rate can keep all the queues filled - due to the fact that the queue is full > with 'heavy collapse diffs' that have to be calculated for each and every > listener (of which you could have eg 150-200) individually. > So again, possible solutions for this: > * OAK-2669: tune diffing via persistent cache > * OAK-2587: have more threads to remain longer 'in the cache zone' > * tune your input speed explicitly to avoid filling the observation queues > (this would be specific to your use-case of course, but can be seen as > explicitly throttling on the input side) > * increase the relevant caches to the max > * but I think we will come up with yet a broader improvement of this > observation queue limit problem by either > ** doing flow control - eg via the commit rate limiter (also see OAK-1659) > ** moving out handling of observation changes to a messaging subsystem - be > it to handle local events only (since handling external events makes the > system problematic wrt
[jira] [Commented] (OAK-1828) Improved SegmentWriter
[ https://issues.apache.org/jira/browse/OAK-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029706#comment-15029706 ] Michael Dürig commented on OAK-1828: Committed the name change at http://svn.apache.org/viewvc?rev=1716816=rev > Improved SegmentWriter > -- > > Key: OAK-1828 > URL: https://issues.apache.org/jira/browse/OAK-1828 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: segmentmk >Reporter: Jukka Zitting >Assignee: Alex Parvulescu >Priority: Minor > Labels: technical_debt > Fix For: 1.3.12 > > Attachments: record-writers-v0.patch, record-writers-v1.patch, > record-writers-v2.patch > > > At about 1kLOC and dozens of methods, the SegmentWriter class currently a bit > too complex for one of the key components of the TarMK. It also uses a > somewhat non-obvious mix of synchronized and unsynchronized code to > coordinate multiple concurrent threads that may be writing content at the > same time. The synchronization blocks are also broader than what really would > be needed, which in some cases causes unnecessary lock contention in > concurrent write loads. > To improve the readability and maintainability of the code, and to increase > performance of concurrent writes, it would be useful to split part of the > SegmentWriter functionality to a separate RecordWriter class that would be > responsible for writing individual records into a segment. The > SegmentWriter.prepare() method would return a new RecordWriter instance, and > the higher-level SegmentWriter methods would use the returned instance for > all the work that's currently guarded in synchronization blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2683) the "hitting the observation queue limit" problem
[ https://issues.apache.org/jira/browse/OAK-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029640#comment-15029640 ] Carsten Ziegeler commented on OAK-2683: --- With SLING-4751 we have a new way for resource observation in Sling (not completely finished yet). With this we can get rid off the "/" listener and only register specific listeners based on what clients are interested in. > the "hitting the observation queue limit" problem > - > > Key: OAK-2683 > URL: https://issues.apache.org/jira/browse/OAK-2683 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, mongomk, segmentmk >Reporter: Stefan Egli > Labels: observation, resilience > > There are several tickets in this area: > * OAK-2587: threading with observation being too eagar causing observation > queue to grow > * OAK-2669: avoiding diffing from mongo by using persistent cache instead. > * OAK-2349: which might be a duplicate or at least similar to 2669.. > * OAK-2562: diffcache is inefficient > Yet I think it makes sense to create this summarizing ticket, about > describing again what happens when the observation queue hits the limit - and > eventually about how this can be improved > Consider the following scenario (also compare with OAK-2587 - but that one > focused more on eagerness of threading): > * rate of incoming commits is large and starts to generate many changes into > the observation queues, hence those queue become somewhat filled/loaded > * depending on the underlying nodestore used the calculation of diffs is more > or less expensive - but at least for mongomk it is important that the diff > can be served from the cache > ** in case of mongomk it can happen that diffs are no longer found in the > cache and thus require a round-trip to mongo - which is magnitudes slower > than via cache of course. this would result in the queue to start increasing > even faster as dequeuing becomes slower now. > ** not sure about tarmk - I believe it should always be fast there > * so based on the above, there can be a situation where the queue grows and > hits the configured limit > * if this limit is reached, the current mechanism is to collapse any > subsequent change into one-big-marked-as-external-event change, lets call > this a collapsed-change. > * this collapsed-change now becomes part of the normal queue and eventually > would 'walk down the queue' and be processed normally - hence opening a high > chance that yet a new collapsed-change is created should the queue just hit > the limit again. and this game can now be played for a while, resulting in > the queue to contain many/mostly such collapse-changes. > * there is now an additional assumption in that the diffing of such collapses > is more expensive than normal diffing - plus it is almost guaranteed that the > diff cannot for example be shared between observation listeners, since the > exact 'collapse borders' depends on timing of each of the listeners' queues - > ie the collapse diffs are unique thus not cachable.. > * so as a result: once you have those collapse-diffs you can almost not get > rid of them - they are heavy to process - hence dequeuing is very slow > * at the same time, there is always likely some commits happening in a > typical system, eg with sling on top you have sling discovery which does > heartbeats every now and then. So there's always new commits that add to the > load. > * this will hence create a situation where quite a small additional commit > rate can keep all the queues filled - due to the fact that the queue is full > with 'heavy collapse diffs' that have to be calculated for each and every > listener (of which you could have eg 150-200) individually. > So again, possible solutions for this: > * OAK-2669: tune diffing via persistent cache > * OAK-2587: have more threads to remain longer 'in the cache zone' > * tune your input speed explicitly to avoid filling the observation queues > (this would be specific to your use-case of course, but can be seen as > explicitly throttling on the input side) > * increase the relevant caches to the max > * but I think we will come up with yet a broader improvement of this > observation queue limit problem by either > ** doing flow control - eg via the commit rate limiter (also see OAK-1659) > ** moving out handling of observation changes to a messaging subsystem - be > it to handle local events only (since handling external events makes the > system problematic wrt scalability if not done right) - also see > [corresponding suggestion on dev > list|http://markmail.org/message/b5trr6csyn4zzuj7] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2683) the "hitting the observation queue limit" problem
[ https://issues.apache.org/jira/browse/OAK-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029663#comment-15029663 ] Vikas Saurabh commented on OAK-2683: Nice.. didn't remember about that one. But, I still think that an observer requiring big diff's - say looking for all changes under /content or /var might not be trivial to remove. And we go down black-listing, then it would just make it harder of projects/products/customer-code downstream to absorb it - or just disable it by configuration if we allow for that. > the "hitting the observation queue limit" problem > - > > Key: OAK-2683 > URL: https://issues.apache.org/jira/browse/OAK-2683 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, mongomk, segmentmk >Reporter: Stefan Egli > Labels: observation, resilience > > There are several tickets in this area: > * OAK-2587: threading with observation being too eagar causing observation > queue to grow > * OAK-2669: avoiding diffing from mongo by using persistent cache instead. > * OAK-2349: which might be a duplicate or at least similar to 2669.. > * OAK-2562: diffcache is inefficient > Yet I think it makes sense to create this summarizing ticket, about > describing again what happens when the observation queue hits the limit - and > eventually about how this can be improved > Consider the following scenario (also compare with OAK-2587 - but that one > focused more on eagerness of threading): > * rate of incoming commits is large and starts to generate many changes into > the observation queues, hence those queue become somewhat filled/loaded > * depending on the underlying nodestore used the calculation of diffs is more > or less expensive - but at least for mongomk it is important that the diff > can be served from the cache > ** in case of mongomk it can happen that diffs are no longer found in the > cache and thus require a round-trip to mongo - which is magnitudes slower > than via cache of course. this would result in the queue to start increasing > even faster as dequeuing becomes slower now. > ** not sure about tarmk - I believe it should always be fast there > * so based on the above, there can be a situation where the queue grows and > hits the configured limit > * if this limit is reached, the current mechanism is to collapse any > subsequent change into one-big-marked-as-external-event change, lets call > this a collapsed-change. > * this collapsed-change now becomes part of the normal queue and eventually > would 'walk down the queue' and be processed normally - hence opening a high > chance that yet a new collapsed-change is created should the queue just hit > the limit again. and this game can now be played for a while, resulting in > the queue to contain many/mostly such collapse-changes. > * there is now an additional assumption in that the diffing of such collapses > is more expensive than normal diffing - plus it is almost guaranteed that the > diff cannot for example be shared between observation listeners, since the > exact 'collapse borders' depends on timing of each of the listeners' queues - > ie the collapse diffs are unique thus not cachable.. > * so as a result: once you have those collapse-diffs you can almost not get > rid of them - they are heavy to process - hence dequeuing is very slow > * at the same time, there is always likely some commits happening in a > typical system, eg with sling on top you have sling discovery which does > heartbeats every now and then. So there's always new commits that add to the > load. > * this will hence create a situation where quite a small additional commit > rate can keep all the queues filled - due to the fact that the queue is full > with 'heavy collapse diffs' that have to be calculated for each and every > listener (of which you could have eg 150-200) individually. > So again, possible solutions for this: > * OAK-2669: tune diffing via persistent cache > * OAK-2587: have more threads to remain longer 'in the cache zone' > * tune your input speed explicitly to avoid filling the observation queues > (this would be specific to your use-case of course, but can be seen as > explicitly throttling on the input side) > * increase the relevant caches to the max > * but I think we will come up with yet a broader improvement of this > observation queue limit problem by either > ** doing flow control - eg via the commit rate limiter (also see OAK-1659) > ** moving out handling of observation changes to a messaging subsystem - be > it to handle local events only (since handling external events makes the > system problematic wrt scalability if not done right) - also see > [corresponding suggestion on dev > list|http://markmail.org/message/b5trr6csyn4zzuj7] -- This message was sent
[jira] [Commented] (OAK-3649) Extract node document cache from Mongo and RDB document stores
[ https://issues.apache.org/jira/browse/OAK-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029665#comment-15029665 ] Tomek Rękawek commented on OAK-3649: [~reschke], thanks for your comments. I removed the UpdateOps from the API and now the cache doesn't depend on the DocumentStore - good tip! It also allowed to remove two methods from NodeDocumentCache - and the less code we have the better :) Regarding the applyChanges, I think the reason of this difference is that RDB requires the whole new Document object to perform the DB update, while the Mongo handles the update with a special DBObject containing just diff, rather than the whole doc. In MongoDocumentStore, the applyChanges is invoked after the change is actually committed, just to put new doc into cache. Do you have in mind some better place to increase the modcount in MongoDS? > Extract node document cache from Mongo and RDB document stores > -- > > Key: OAK-3649 > URL: https://issues.apache.org/jira/browse/OAK-3649 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: documentmk, mongomk, rdbmk >Reporter: Tomek Rękawek >Priority: Minor > Labels: candidate_oak_1_0, candidate_oak_1_2 > Fix For: 1.3.12 > > > MongoDocumentStore and RDBDocumentStore contains copy & pasted methods > responsible for handling node document cache. Extract these into a new > NodeDocumentCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3672) SegmentDiscoveryLiteService does not persist clusterView.id
[
https://issues.apache.org/jira/browse/OAK-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029667#comment-15029667
]
Michael Dürig commented on OAK-3672:
That one is an instance id that is not persisted and not guaranteed to be
stable across restarts: its contract is: "... will return a unique number per
instance across the cluster. It will only make its best effort to preserve the
same number across restarts but it must be unique across the cluster."
The main reason for not making it stable across restarts where the troublesome
semantics that go along with it: what about cloning/backing up/restoring a
repository?
> SegmentDiscoveryLiteService does not persist clusterView.id
> ---
>
> Key: OAK-3672
> URL: https://issues.apache.org/jira/browse/OAK-3672
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
>Affects Versions: 1.3.10
>Reporter: Stefan Egli
> Fix For: 1.4
>
>
> The discovery-lite-descriptor introduced with OAK-2844 has a property {{id}}
> that uniquely and persistently identifies a cluster. However, the
> {{SegmentDiscoveryLiteService}} creates this id upon each instance restart
> (by setting {{runtimeClusterId}}).
> This should be fixed to have this {{id}} persisted somehow.
> Note that the consequences of this id changing upon each restart is that the
> corresponding presumed-to-be-persistent {{ClusterView.id}} of the
> discovery.oak will also change upon restart. Which is a violation of the
> discovery API and upper level applications might thus misbehave in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3674) Search Excerpt highlighting is not correct
[
https://issues.apache.org/jira/browse/OAK-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029631#comment-15029631
]
Thomas Mueller commented on OAK-3674:
-
This is a limitation of the current SimpleExcerptProvider: it ignores word
boundaries.
Could you try quoting the search tokens? That is, use double quotes inside the
single quotes:
{noformat}
//*[jcr:contains(., '"conflict of interest"')]/rep:excerpt(.)
{noformat}
I think ultimately, it doesn't make sense to try to improve
SimpleExcerptProvider. Instead, we should probably use the Lucene highlighter
instead, which is available in OAK-3580. So I will close this issue as
duplicate of OAK-3580.
> Search Excerpt highlighting is not correct
> --
>
> Key: OAK-3674
> URL: https://issues.apache.org/jira/browse/OAK-3674
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: query
>Affects Versions: 1.0.18, 1.0.23
>Reporter: Srijan Bhatnagar
>
> We have the following text on a jcr:content node :
> {code}A state agency’s Conflict of Interest Code must reflect the current
> structure of the organization and properly identify officials
> andemployees{code}
> On executing the following query :
> {code}//*[jcr:contains(., 'conflict of interest')]/rep:excerpt(.){code}
> we get a row whose excerpt value is having wrong placement of
> tags.
> Observed result:
> {code}pA state agency’s Conflict of
> Interest Code must reflect the current structure of the
> organization and properly identify officials
> andemployees/p{code}
> I don't think it is expected to have {quote}officials{quote}
> in the excerpt.
> We get the excerpt value in the following manner :
> org.apache.jackrabbit.oak.jcr.query.RowImpl#getValue("rep:excerpt(" +
> nodePath + ")")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3674) Search Excerpt highlighting is not correct
[
https://issues.apache.org/jira/browse/OAK-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Mueller resolved OAK-3674.
-
Resolution: Duplicate
> Search Excerpt highlighting is not correct
> --
>
> Key: OAK-3674
> URL: https://issues.apache.org/jira/browse/OAK-3674
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: query
>Affects Versions: 1.0.18, 1.0.23
>Reporter: Srijan Bhatnagar
>
> We have the following text on a jcr:content node :
> {code}A state agency’s Conflict of Interest Code must reflect the current
> structure of the organization and properly identify officials
> andemployees{code}
> On executing the following query :
> {code}//*[jcr:contains(., 'conflict of interest')]/rep:excerpt(.){code}
> we get a row whose excerpt value is having wrong placement of
> tags.
> Observed result:
> {code}pA state agency’s Conflict of
> Interest Code must reflect the current structure of the
> organization and properly identify officials
> andemployees/p{code}
> I don't think it is expected to have {quote}officials{quote}
> in the excerpt.
> We get the excerpt value in the following manner :
> org.apache.jackrabbit.oak.jcr.query.RowImpl#getValue("rep:excerpt(" +
> nodePath + ")")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2410) [sonar]Some statements not being closed in RDBDocumentStore
[ https://issues.apache.org/jira/browse/OAK-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029723#comment-15029723 ] Julian Reschke commented on OAK-2410: - FWIW, not explicitly closing the ResultSet shouldn't be an issue as long as the Statement is closed. > [sonar]Some statements not being closed in RDBDocumentStore > --- > > Key: OAK-2410 > URL: https://issues.apache.org/jira/browse/OAK-2410 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: rdbmk >Affects Versions: 1.2.2, 1.3.0, 1.0.15 >Reporter: Chetan Mehrotra >Assignee: Julian Reschke >Priority: Minor > Labels: technical_debt > Fix For: 1.3.1, 1.2.3, 1.0.16 > > > The Sonar analysis for Oak [1] report shows some warnings for > RDBDocumentStore [2]. The critical ones are related to not closing of > statement and resultset instances. > [1] > https://analysis.apache.org/dashboard/index/org.apache.jackrabbit:jackrabbit-oak?did=2 > [2] > https://analysis.apache.org/resource/index/179726?display_title=true=violations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
