date:20161020


[ 
https://issues.apache.org/jira/browse/OAK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15594145#comment-15594145
 ] 

Chetan Mehrotra commented on OAK-1312:
--

Thanks [~julian.resc...@gmx.de]. Overall numbers are much better compared to 
Mongo (and to an extent similar to Tar) :). Can you provide some hardware/setup 
details. Was DB local or remote 

> Bundle nodes into a document
> 
>
> Key: OAK-1312
> URL: https://issues.apache.org/jira/browse/OAK-1312
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: core, documentmk
>Reporter: Marcel Reutegger
>Assignee: Chetan Mehrotra
>  Labels: docs-impacting, performance
> Fix For: 1.6, 1.5.13
>
> Attachments: OAK-1312-meta-prop-handling.patch, 
> OAK-1312-review-v1.diff, OAK-1312-review-v2.diff, benchmark-result-db2.txt, 
> benchmark-result-postgres.txt, benchmark-results.txt, run-benchmark.sh
>
>
> For very fine grained content with many nodes and only few properties per 
> node it would be more efficient to bundle multiple nodes into a single 
> MongoDB document. Mostly reading would benefit because there are less 
> roundtrips to the backend. At the same time storage footprint would be lower 
> because metadata overhead is per document.
> Feature branch - 
> https://github.com/chetanmeh/jackrabbit-oak/compare/trunk...chetanmeh:OAK-1312



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-1312) Bundle nodes into a document


 [ 
https://issues.apache.org/jira/browse/OAK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chetan Mehrotra updated OAK-1312:
-
Issue Type: New Feature  (was: Improvement)

> Bundle nodes into a document
> 
>
> Key: OAK-1312
> URL: https://issues.apache.org/jira/browse/OAK-1312
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: core, documentmk
>Reporter: Marcel Reutegger
>Assignee: Chetan Mehrotra
>  Labels: docs-impacting, performance
> Fix For: 1.6, 1.5.13
>
> Attachments: OAK-1312-meta-prop-handling.patch, 
> OAK-1312-review-v1.diff, OAK-1312-review-v2.diff, benchmark-result-db2.txt, 
> benchmark-result-postgres.txt, benchmark-results.txt, run-benchmark.sh
>
>
> For very fine grained content with many nodes and only few properties per 
> node it would be more efficient to bundle multiple nodes into a single 
> MongoDB document. Mostly reading would benefit because there are less 
> roundtrips to the backend. At the same time storage footprint would be lower 
> because metadata overhead is per document.
> Feature branch - 
> https://github.com/chetanmeh/jackrabbit-oak/compare/trunk...chetanmeh:OAK-1312



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4966) Re-introduce a blocker for compaction based on available heap

2016-10-20 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/OAK-4966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-4966:
---
Fix Version/s: Segment Tar 0.0.18

> Re-introduce a blocker for compaction based on available heap
> -
>
> Key: OAK-4966
> URL: https://issues.apache.org/jira/browse/OAK-4966
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Alex Parvulescu
> Fix For: Segment Tar 0.0.18
>
>
> As seen in a local test, running compaction on a tight heap can lead to 
> OOMEs. There used to be a best effort barrier against this situation 'not 
> enough heap for compaction', but we removed it with the compaction maps.
> I think it makes sense to add it again based on the max size of some of the 
> caches: segment cache {{256MB}} by default [0] and some writer caches which 
> can go up to {{2GB}} all combined [1] and probably others I missed.
> [0] 
> https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java#L48
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/WriterCacheManager.java#L50



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


[ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591870#comment-15591870
 ] 

Julian Reschke edited comment on OAK-4964 at 10/20/16 4:11 PM:
---

trunk: [r1765817|http://svn.apache.org/r1765817]
1.4: [r1765823|http://svn.apache.org/r1765823]
1.2: [r1765835|http://svn.apache.org/r1765835]
1.0: [r1765845|http://svn.apache.org/r1765845]





was (Author: reschke):
trunk: [r1765817|http://svn.apache.org/r1765817]
1.4: [r1765823|http://svn.apache.org/r1765823]
1.2: [r1765835|http://svn.apache.org/r1765835]



> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
> Fix For: 1.6, 1.0.35, 1.4.9, 1.2.21, 1.5.13
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


 [ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4964:

Fix Version/s: 1.0.35

> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
> Fix For: 1.6, 1.0.35, 1.4.9, 1.2.21, 1.5.13
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


 [ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4964:

Labels:   (was: candidate_oak_1_0)

> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
> Fix For: 1.6, 1.0.35, 1.4.9, 1.2.21, 1.5.13
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


 [ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4964:

Fix Version/s: 1.2.21

> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_0
> Fix For: 1.6, 1.4.9, 1.2.21, 1.5.13
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


[ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591870#comment-15591870
 ] 

Julian Reschke edited comment on OAK-4964 at 10/20/16 3:34 PM:
---

trunk: [r1765817|http://svn.apache.org/r1765817]
1.4: [r1765823|http://svn.apache.org/r1765823]
1.2: [r1765835|http://svn.apache.org/r1765835]




was (Author: reschke):
trunk: [r1765817|http://svn.apache.org/r1765817]
1.4: [r1765823|http://svn.apache.org/r1765823]


> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_0
> Fix For: 1.6, 1.4.9, 1.2.21, 1.5.13
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


 [ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4964:

Labels: candidate_oak_1_0  (was: candidate_oak_1_0 candidate_oak_1_2)

> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_0
> Fix For: 1.6, 1.4.9, 1.2.21, 1.5.13
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4958) Test failure: BrokenNetworkTest

2016-10-20 Thread Francesco Mari (JIRA)


[ 
https://issues.apache.org/jira/browse/OAK-4958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592116#comment-15592116
 ] 

Francesco Mari commented on OAK-4958:
-

I took a closer look at NetworkErrorProxy at r1765831. The implementation now 
lives in its own package and has been slightly simplified. I took care that 
channels and groups are correctly initialized and finalized.

> Test failure: BrokenNetworkTest
> ---
>
> Key: OAK-4958
> URL: https://issues.apache.org/jira/browse/OAK-4958
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Francesco Mari
>  Labels: test-failure
> Fix For: Segment Tar 0.0.18
>
> Attachments: BrokenNetworkTest-logs.zip
>
>
> On some machines the {{BrokenNetworkTest}} fails:
> {noformat}
> Failed tests:
>   BrokenNetworkTest.testProxyFlippedEndByteSSL:103->useProxy:146 expected:<{ 
> root = { ... } }> but was:<{ root : { } }>
>   BrokenNetworkTest.testProxyFlippedIntermediateByte:88->useProxy:146 
> expected:<{ root = { ... } }> but was:<{ root : { } }>
>   BrokenNetworkTest.testProxyFlippedIntermediateByteSSL:93->useProxy:146 
> expected:<{ root = { ... } }> but was:<{ root : { } }>
>   BrokenNetworkTest.testProxyFlippedStartByte:78->useProxy:146 expected:<{ 
> root = { ... } }> but was:<{ root : { } }>
>   BrokenNetworkTest.testProxySSLSkippedBytes:63->useProxy:113->useProxy:146 
> expected:<{ root = { ... } }> but was:<{ root : { } }>
>   
> BrokenNetworkTest.testProxySSLSkippedBytesIntermediateChange:73->useProxy:113->useProxy:146
>  expected:<{ root = { ... } }> but was:<{ root : { } }>
>   
> BrokenNetworkTest.testProxySkippedBytesIntermediateChange:68->useProxy:113->useProxy:146
>  expected:<{ root = { ... } }> but was:<{ root : { } }>
> {noformat}
> Stack traces are all similar to 
> {noformat}
> testProxySkippedBytesIntermediateChange(org.apache.jackrabbit.oak.segment.standby.BrokenNetworkTest)
>   Time elapsed: 5.577 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<{ root = { ... } }> but was:<{ root : { } 
> }>
>   at 
> org.apache.jackrabbit.oak.segment.standby.BrokenNetworkTest.useProxy(BrokenNetworkTest.java:146)
>   at 
> org.apache.jackrabbit.oak.segment.standby.BrokenNetworkTest.useProxy(BrokenNetworkTest.java:113)
>   at 
> org.apache.jackrabbit.oak.segment.standby.BrokenNetworkTest.testProxySkippedBytesIntermediateChange(BrokenNetworkTest.java:68)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4861) AsyncIndexUpdate consuming constantly 1 CPU core

2016-10-20 Thread Wim Symons (JIRA)


[ 
https://issues.apache.org/jira/browse/OAK-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592094#comment-15592094
 ] 

Wim Symons commented on OAK-4861:
-

We also have a very fast object allocation rate (about 2 GB/sec) in the JVM 
causing our ParNewGC collector to go nuts. Our JVM settings are  {{-server 
-Xmx8192m -Xms8192m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode}} and we 
are running on Oracle Java 1.8.0_51

> AsyncIndexUpdate consuming constantly 1 CPU core
> 
>
> Key: OAK-4861
> URL: https://issues.apache.org/jira/browse/OAK-4861
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>Affects Versions: 1.0.31
>Reporter: Jörg Hoh
>
> The AsyncIndexUpdate thread takes a long time on some of our environments to 
> update the index for a few nodes:
> {noformat}
> 28.09.2016 18:23:49.405 *DEBUG* [pool-11-thread-1] 
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate AsyncIndex (async) 
> update run completed in 3,369 min. Indexed 14 nodes
> {noformat}
> I set the log to DEBUG for investigation and during the time I saw constantly 
> such times; this instance is a test instance and there was no activity from 
> the outside during this time. I also can confirm such times from other 
> environments.
> The immediately visible problem is that this instance (and others as well) 
> constantly consume at least 1 CPU core, no matter what activity happens on 
> the system. This only happens on the master node of our DocumentNodeStore 
> clusters. 
> I did a number of threaddumps during investigation and a common stacktrace 
> for the AsyncIndexUpdate was like this:
> {noformat}
> 3XMTHREADINFO "pool-11-thread-1" J9VMThread:0x01E69C00, 
> j9thread_t:0x7FFFBE101CB0, java/lang/Thread:0x0001077FE938, state:R, 
> prio=5
> 3XMJAVALTHREAD (java/lang/Thread getId:0x10A, isDaemon:false)
> 3XMTHREADINFO1 (native thread ID:0x3C781, native priority:0x5, native 
> policy:UNKNOWN, vmstate:CW, vm thread flags:0x0001)
> 3XMTHREADINFO2 (native stack address range from:0x7FFFC696A000, 
> to:0x7FFFC69AB000, size:0x41000)
> 3XMCPUTIME CPU usage total: 992425.397213554 secs, current 
> category="Application"
> 3XMHEAPALLOC Heap bytes allocated since last GC cycle=735893672 (0x2BDCD8A8)
> 3XMTHREADINFO3 Java callstack:
> 4XESTACKTRACE at 
> com/google/common/collect/Lists.newArrayList(Lists.java:128(Compiled Code))
> 4XESTACKTRACE at 
> org/apache/jackrabbit/oak/plugins/index/lucene/OakDirectory$OakIndexFile.(OakDirectory.java:255(Compiled
>  Code))
> 4XESTACKTRACE at 
> org/apache/jackrabbit/oak/plugins/index/lucene/OakDirectory$OakIndexFile.(OakDirectory.java:201(Compiled
>  Code))
> 4XESTACKTRACE at 
> org/apache/jackrabbit/oak/plugins/index/lucene/OakDirectory$OakIndexInput.(OakDirectory.java:400(Compiled
>  Code))
> 4XESTACKTRACE at 
> org/apache/jackrabbit/oak/plugins/index/lucene/OakDirectory$OakIndexInput.clone(OakDirectory.java:408(Compiled
>  Code))
> 4XESTACKTRACE at 
> org/apache/jackrabbit/oak/plugins/index/lucene/OakDirectory$OakIndexInput.clone(OakDirectory.java:384(Compiled
>  Code))
> 4XESTACKTRACE at 
> org/apache/lucene/store/Directory$SlicedIndexInput.clone(Directory.java:288(Compiled
>  Code))
> 4XESTACKTRACE at 
> org/apache/lucene/store/Directory$SlicedIndexInput.clone(Directory.java:269(Compiled
>  Code))
> 4XESTACKTRACE at 
> org/apache/lucene/codecs/BlockTreeTermsReader$FieldReader.(BlockTreeTermsReader.java:481(Compiled
>  Code))
> 4XESTACKTRACE at 
> org/apache/lucene/codecs/BlockTreeTermsReader.(BlockTreeTermsReader.java:176(Compiled
>  Code))
> 4XESTACKTRACE at 
> org/apache/lucene/codecs/lucene41/Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:437(Compiled
>  Code))
> 4XESTACKTRACE at 
> org/apache/lucene/index/SegmentCoreReaders.(SegmentCoreReaders.java:116(Compiled
>  Code))
> 4XESTACKTRACE at 
> org/apache/lucene/index/SegmentReader.(SegmentReader.java:96(Compiled 
> Code))
> 4XESTACKTRACE at 
> org/apache/lucene/index/ReadersAndUpdates.getReader(ReadersAndUpdates.java:141(Compiled
>  Code))
> 4XESTACKTRACE at 
> org/apache/lucene/index/BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:279(Compiled
>  Code))
> 5XESTACKTRACE (entered lock: 
> org/apache/lucene/index/BufferedUpdatesStream@0x0002245B4100, entry 
> count: 1)
> 4XESTACKTRACE at 
> org/apache/lucene/index/IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3191(Compiled
>  Code))
> 5XESTACKTRACE (entered lock: 
> org/apache/lucene/index/IndexWriter@0x0002245B4050, entry count: 3)
> 4XESTACKTRACE at 
> org/apache/lucene/index/IndexWriter.maybeApplyDeletes(IndexWriter.java:3182(Compiled
>  Code))
> 5XESTACKTRACE (entered lock: 
> org/apache/lucene/index/IndexWriter@0x0002245B4050, entry count: 2)
> 4XESTACKTRACE at 
>

[jira] [Comment Edited] (OAK-4861) AsyncIndexUpdate consuming constantly 1 CPU core

2016-10-20 Thread Wim Symons (JIRA)


[ 
https://issues.apache.org/jira/browse/OAK-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592080#comment-15592080
 ] 

Wim Symons edited comment on OAK-4861 at 10/20/16 3:11 PM:
---

Hi we have exactly the same issue as you have. Except it uses almost all of our 
CPU.

Stacktrace:

{code}
"pool-7-thread-4" - Thread t@114
java.lang.Thread.State: RUNNABLE
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.(OakDirectory.java:255)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.(OakDirectory.java:205)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.(OakDirectory.java:404)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.clone(OakDirectory.java:412)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.clone(OakDirectory.java:388)
at 
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader.(BlockTreeTermsReader.java:481)
at 
org.apache.lucene.codecs.BlockTreeTermsReader.(BlockTreeTermsReader.java:176)
at 
org.apache.lucene.codecs.lucene41.Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:437)
at 
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:116)
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:96)
at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:141)
at 
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:279)
- locked <7e04344b> (a org.apache.lucene.index.BufferedUpdatesStream)
at 
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3191)
- locked <50c64b3b> (a org.apache.lucene.index.IndexWriter)
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3182)
- locked <50c64b3b> (a org.apache.lucene.index.IndexWriter)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3155)
- locked <50c64b3b> (a org.apache.lucene.index.IndexWriter)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3123)
at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:988)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:932)
- locked <6c1af558> (a java.lang.Object)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:894)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext.closeWriter(LuceneIndexEditorContext.java:250)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.leave(LuceneIndexEditor.java:214)
at 
org.apache.jackrabbit.oak.plugins.index.IndexUpdate.leave(IndexUpdate.java:252)
at 
org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:63)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:56)
at 
org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:491)
at 
org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:433)
at 
org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:325)
- locked <5472cf2> (a org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate)
at 
org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:115)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- locked <788eb307> (a java.util.concurrent.ThreadPoolExecutor$Worker)
{code}

We have it on oak-lucene version 1.2.16 on AEM 6.1 SP1.

What I also did is to attach a Java debugger to the AEM process and set a 
logging breakpoint at OakDirectory.java:255. If I enable the breakpoint, CPU 
drops to 2-3%. If I disable the breakpoint CPU rises again to 50%.

So I think it is a concurrency issue. I also have a trace level output as 
mentioned above, but nothing to see there.

To be complete, we are using a segment datastore (TarMK), CopyOnRead is 
enabled, CopyOnWrite is disabled and the number of threads is 5.

Did you find a solution for this already? At the moment all our production 
instances have this issue. We opened a P1 ticket at Daycare, but so far no luck.


was (Author: wim.symons):
Hi we have exactly the same issue as you have. Except it uses almost all of our 
CPU.

Stacktrace:

"pool-7-thread-4" - Thread t@114
java.lang.Thread.State: RUNNABLE
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.(OakDirectory.java:255)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.(OakDirectory.java:205)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.(OakDirectory.java:404)
at

[jira] [Commented] (OAK-4861) AsyncIndexUpdate consuming constantly 1 CPU core

2016-10-20 Thread Wim Symons (JIRA)


[ 
https://issues.apache.org/jira/browse/OAK-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592080#comment-15592080
 ] 

Wim Symons commented on OAK-4861:
-

Hi we have exactly the same issue as you have. Except it uses almost all of our 
CPU.

Stacktrace:

"pool-7-thread-4" - Thread t@114
java.lang.Thread.State: RUNNABLE
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.(OakDirectory.java:255)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.(OakDirectory.java:205)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.(OakDirectory.java:404)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.clone(OakDirectory.java:412)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.clone(OakDirectory.java:388)
at 
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader.(BlockTreeTermsReader.java:481)
at 
org.apache.lucene.codecs.BlockTreeTermsReader.(BlockTreeTermsReader.java:176)
at 
org.apache.lucene.codecs.lucene41.Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:437)
at 
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:116)
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:96)
at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:141)
at 
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:279)
- locked <7e04344b> (a org.apache.lucene.index.BufferedUpdatesStream)
at 
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3191)
- locked <50c64b3b> (a org.apache.lucene.index.IndexWriter)
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3182)
- locked <50c64b3b> (a org.apache.lucene.index.IndexWriter)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3155)
- locked <50c64b3b> (a org.apache.lucene.index.IndexWriter)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3123)
at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:988)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:932)
- locked <6c1af558> (a java.lang.Object)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:894)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext.closeWriter(LuceneIndexEditorContext.java:250)
at 
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.leave(LuceneIndexEditor.java:214)
at 
org.apache.jackrabbit.oak.plugins.index.IndexUpdate.leave(IndexUpdate.java:252)
at 
org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:63)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:56)
at 
org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:491)
at 
org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:433)
at 
org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:325)
- locked <5472cf2> (a org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate)
at 
org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:115)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- locked <788eb307> (a java.util.concurrent.ThreadPoolExecutor$Worker)

We have it on oak-lucene version 1.2.16 on AEM 6.1 SP1.

What I also did is to attach a Java debugger to the AEM process and set a 
logging breakpoint at OakDirectory.java:255. If I enable the breakpoint, CPU 
drops to 2-3%. If I disable the breakpoint CPU rises again to 50%.

So I think it is a concurrency issue. I also have a trace level output as 
mentioned above, but nothing to see there.

To be complete, we are using a segment datastore (TarMK), CopyOnRead is 
enabled, CopyOnWrite is disabled and the number of threads is 5.

Did you find a solution for this already? At the moment all our production 
instances have this issue. We opened a P1 ticket at Daycare, but so far no luck.

> AsyncIndexUpdate consuming constantly 1 CPU core
> 
>
> Key: OAK-4861
> URL: https://issues.apache.org/jira/browse/OAK-4861
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>Affects Versions: 1.0.31
>Reporter: Jörg Hoh
>
> The AsyncIndexUpdate thread takes a long time on some of our environments to 
> update the index for a few nodes:
> {noformat}
> 28.09.2016 18:23:49.405 *DEBUG* [pool-11-thread-1] 
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate AsyncIndex (async) 
> update run completed in 3,369 min. Indexed 14

[jira] [Assigned] (OAK-4969) ColdStandby does not fetch missing blobs


 [ 
https://issues.apache.org/jira/browse/OAK-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothee Maret reassigned OAK-4969:
---

Assignee: Timothee Maret

> ColdStandby does not fetch missing blobs
> 
>
> Key: OAK-4969
> URL: https://issues.apache.org/jira/browse/OAK-4969
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Affects Versions: Segment Tar 0.0.10
>Reporter: Timothee Maret
>Assignee: Timothee Maret
> Fix For: Segment Tar 0.0.16
>
>
> 1. In a setup composed of two instances (primary, standby) configured with a 
> custom blob store (File blob store).
> 2. On the primary instance, set/update a BINARY property of an existing 
> resource with > 2MB binary.
> 3. Observe that the standby instance does not fetch the binary and instead, 
> enters a loop detecting the missing binary upon comparing node states.
> For example, the following stack trace would be printed every 5 seconds on 
> the standby (the polling time is 5sec). 
> {code}
> 19.10.2016 16:22:38.035 *DEBUG* [nioEventLoopGroup-1005-1] 
> org.apache.jackrabbit.oak.segment.standby.codec.ResponseDecoder Decoding 'get 
> head' response
> 19.10.2016 16:22:38.038 *DEBUG* [sling-default-81-Registered Service.607] 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyClient Channel closed
> 19.10.2016 16:22:40.241 *DEBUG* [sling-default-81-Registered Service.607] 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyClient Group shut down
> 19.10.2016 16:22:40.242 *ERROR* [sling-default-81-Registered Service.607] 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync Failed 
> synchronizing state.
> java.lang.RuntimeException: Error occurred while obtaining InputStream for 
> blobId [4dfc748c91d518c9221031ec6115fd7ac04fe27b#10]
>   at 
> org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:49)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:252)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:87)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.getInput(AbstractBlob.java:45)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.getInput(AbstractBlob.java:42)
>   at com.google.common.io.ByteStreams$3.openStream(ByteStreams.java:907)
>   at com.google.common.io.ByteSource.contentEquals(ByteSource.java:301)
>   at com.google.common.io.ByteStreams.equal(ByteStreams.java:661)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:68)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentBlob.equals(SegmentBlob.java:193)
>   at com.google.common.base.Objects.equal(Objects.java:55)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:53)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentPropertyState.equals(SegmentPropertyState.java:249)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:622)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:516)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:415)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:457)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord$2.childNodeChanged(MapRecord.java:401)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord$3.childNodeChanged(MapRecord.java:442)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:485)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:433)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:391)
>   at 
>

[jira] [Issue Comment Deleted] (OAK-4969) ColdStandby does not fetch missing blobs


 [ 
https://issues.apache.org/jira/browse/OAK-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothee Maret updated OAK-4969:

Comment: was deleted

(was: Looking at the oak-segment-tar code, the client (standby) should fetch 
blobs by sending {{o.a.j.o.s.s.c.GetBlobRequest}} messages to the primary 
instance.

The code that is support to fetch blobs is in the code base 
({{StandbyDiff#binaryCheck(Blob, java.lang.String)}} to 
{{StandbyClient#getBlob}}), however it is never called.
)

> ColdStandby does not fetch missing blobs
> 
>
> Key: OAK-4969
> URL: https://issues.apache.org/jira/browse/OAK-4969
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Affects Versions: Segment Tar 0.0.10
>Reporter: Timothee Maret
> Fix For: Segment Tar 0.0.16
>
>
> 1. In a setup composed of two instances (primary, standby) configured with a 
> custom blob store (File blob store).
> 2. On the primary instance, set/update a BINARY property of an existing 
> resource with > 2MB binary.
> 3. Observe that the standby instance does not fetch the binary and instead, 
> enters a loop detecting the missing binary upon comparing node states.
> For example, the following stack trace would be printed every 5 seconds on 
> the standby (the polling time is 5sec). 
> {code}
> 19.10.2016 16:22:38.035 *DEBUG* [nioEventLoopGroup-1005-1] 
> org.apache.jackrabbit.oak.segment.standby.codec.ResponseDecoder Decoding 'get 
> head' response
> 19.10.2016 16:22:38.038 *DEBUG* [sling-default-81-Registered Service.607] 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyClient Channel closed
> 19.10.2016 16:22:40.241 *DEBUG* [sling-default-81-Registered Service.607] 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyClient Group shut down
> 19.10.2016 16:22:40.242 *ERROR* [sling-default-81-Registered Service.607] 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync Failed 
> synchronizing state.
> java.lang.RuntimeException: Error occurred while obtaining InputStream for 
> blobId [4dfc748c91d518c9221031ec6115fd7ac04fe27b#10]
>   at 
> org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:49)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:252)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:87)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.getInput(AbstractBlob.java:45)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.getInput(AbstractBlob.java:42)
>   at com.google.common.io.ByteStreams$3.openStream(ByteStreams.java:907)
>   at com.google.common.io.ByteSource.contentEquals(ByteSource.java:301)
>   at com.google.common.io.ByteStreams.equal(ByteStreams.java:661)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:68)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentBlob.equals(SegmentBlob.java:193)
>   at com.google.common.base.Objects.equal(Objects.java:55)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:53)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentPropertyState.equals(SegmentPropertyState.java:249)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:622)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:516)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:415)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:457)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord$2.childNodeChanged(MapRecord.java:401)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord$3.childNodeChanged(MapRecord.java:442)
>   at 
>

[jira] [Commented] (OAK-4969) ColdStandby does not fetch missing blobs


[ 
https://issues.apache.org/jira/browse/OAK-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592028#comment-15592028
 ] 

Timothee Maret commented on OAK-4969:
-

Looking at the oak-segment-tar code, the client (standby) should fetch blobs by 
sending {{o.a.j.o.s.s.c.GetBlobRequest}} messages to the primary instance.

The code that is support to fetch blobs is in the code base 
({{StandbyDiff#binaryCheck(Blob, java.lang.String)}} to 
{{StandbyClient#getBlob}}), however it is never called.


> ColdStandby does not fetch missing blobs
> 
>
> Key: OAK-4969
> URL: https://issues.apache.org/jira/browse/OAK-4969
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Affects Versions: Segment Tar 0.0.10
>Reporter: Timothee Maret
> Fix For: Segment Tar 0.0.16
>
>
> 1. In a setup composed of two instances (primary, standby) configured with a 
> custom blob store (File blob store).
> 2. On the primary instance, set/update a BINARY property of an existing 
> resource with > 2MB binary.
> 3. Observe that the standby instance does not fetch the binary and instead, 
> enters a loop detecting the missing binary upon comparing node states.
> For example, the following stack trace would be printed every 5 seconds on 
> the standby (the polling time is 5sec). 
> {code}
> 19.10.2016 16:22:38.035 *DEBUG* [nioEventLoopGroup-1005-1] 
> org.apache.jackrabbit.oak.segment.standby.codec.ResponseDecoder Decoding 'get 
> head' response
> 19.10.2016 16:22:38.038 *DEBUG* [sling-default-81-Registered Service.607] 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyClient Channel closed
> 19.10.2016 16:22:40.241 *DEBUG* [sling-default-81-Registered Service.607] 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyClient Group shut down
> 19.10.2016 16:22:40.242 *ERROR* [sling-default-81-Registered Service.607] 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync Failed 
> synchronizing state.
> java.lang.RuntimeException: Error occurred while obtaining InputStream for 
> blobId [4dfc748c91d518c9221031ec6115fd7ac04fe27b#10]
>   at 
> org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:49)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:252)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:87)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.getInput(AbstractBlob.java:45)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.getInput(AbstractBlob.java:42)
>   at com.google.common.io.ByteStreams$3.openStream(ByteStreams.java:907)
>   at com.google.common.io.ByteSource.contentEquals(ByteSource.java:301)
>   at com.google.common.io.ByteStreams.equal(ByteStreams.java:661)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:68)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentBlob.equals(SegmentBlob.java:193)
>   at com.google.common.base.Objects.equal(Objects.java:55)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:53)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentPropertyState.equals(SegmentPropertyState.java:249)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:622)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:516)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:415)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:457)
>   at 
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
>   at 
> org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord$2.childNodeChanged(MapRecord.java:401)
>   at 
> org.apache.jackrabbit.oak.segment.MapRecord$3.childNodeChanged(MapRecord.java:442)
>   at 
>

[jira] [Comment Edited] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


[ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591870#comment-15591870
 ] 

Julian Reschke edited comment on OAK-4964 at 10/20/16 2:46 PM:
---

trunk: [r1765817|http://svn.apache.org/r1765817]
1.4: [r1765823|http://svn.apache.org/r1765823]



was (Author: reschke):
trunk: [r1765817|http://svn.apache.org/r1765817]

> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.6, 1.4.9, 1.5.13
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4969) ColdStandby does not fetch missing blobs


 [ 
https://issues.apache.org/jira/browse/OAK-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothee Maret updated OAK-4969:

Description: 
1. In a setup composed of two instances (primary, standby) configured with a 
custom blob store (File blob store).
2. On the primary instance, set/update a BINARY property of an existing 
resource with > 2MB binary.
3. Observe that the standby instance does not fetch the binary and instead, 
enters a loop detecting the missing binary upon comparing node states.

For example, the following stack trace would be printed every 5 seconds on the 
standby (the polling time is 5sec). 
{code}

19.10.2016 16:22:38.035 *DEBUG* [nioEventLoopGroup-1005-1] 
org.apache.jackrabbit.oak.segment.standby.codec.ResponseDecoder Decoding 'get 
head' response
19.10.2016 16:22:38.038 *DEBUG* [sling-default-81-Registered Service.607] 
org.apache.jackrabbit.oak.segment.standby.client.StandbyClient Channel closed
19.10.2016 16:22:40.241 *DEBUG* [sling-default-81-Registered Service.607] 
org.apache.jackrabbit.oak.segment.standby.client.StandbyClient Group shut down
19.10.2016 16:22:40.242 *ERROR* [sling-default-81-Registered Service.607] 
org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync Failed 
synchronizing state.
java.lang.RuntimeException: Error occurred while obtaining InputStream for 
blobId [4dfc748c91d518c9221031ec6115fd7ac04fe27b#10]
at 
org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:49)
at 
org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:252)
at 
org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:87)
at 
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.getInput(AbstractBlob.java:45)
at 
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.getInput(AbstractBlob.java:42)
at com.google.common.io.ByteStreams$3.openStream(ByteStreams.java:907)
at com.google.common.io.ByteSource.contentEquals(ByteSource.java:301)
at com.google.common.io.ByteStreams.equal(ByteStreams.java:661)
at 
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:68)
at 
org.apache.jackrabbit.oak.segment.SegmentBlob.equals(SegmentBlob.java:193)
at com.google.common.base.Objects.equal(Objects.java:55)
at 
org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:53)
at 
org.apache.jackrabbit.oak.segment.SegmentPropertyState.equals(SegmentPropertyState.java:249)
at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:622)
at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:516)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
at 
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:415)
at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
at 
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:457)
at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
at 
org.apache.jackrabbit.oak.segment.MapRecord$2.childNodeChanged(MapRecord.java:401)
at 
org.apache.jackrabbit.oak.segment.MapRecord$3.childNodeChanged(MapRecord.java:442)
at 
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:485)
at 
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:433)
at 
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:391)
at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
at 
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:485)
at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
at

[jira] [Updated] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


 [ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4964:

Labels: candidate_oak_1_0 candidate_oak_1_2  (was: candidate_oak_1_0 
candidate_oak_1_2 candidate_oak_1_4)

> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.6, 1.4.9, 1.5.13
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


 [ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4964:

Fix Version/s: 1.4.9

> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.6, 1.4.9, 1.5.13
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4969) ColdStandby does not fetch missing blobs

Timothee Maret created OAK-4969:
---

 Summary: ColdStandby does not fetch missing blobs
 Key: OAK-4969
 URL: https://issues.apache.org/jira/browse/OAK-4969
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: segment-tar
Affects Versions: Segment Tar 0.0.10
Reporter: Timothee Maret
 Fix For: Segment Tar 0.0.16


1. In a setup composed of two instances (primary, standdy) configured with a 
custom blob store (File blob store).
2. On the primary instance, set/update a BINARY property of an existing 
resource with > 2MB binary.
3. Observe that the standby instance does not fetch the binary and instead, 
enters a loop detecting the missing binary upon comparing node states.

For example, the following stack trace would be printed every 5 seconds on the 
standby (the polling time is 5sec). 
{code}

19.10.2016 16:22:38.035 *DEBUG* [nioEventLoopGroup-1005-1] 
org.apache.jackrabbit.oak.segment.standby.codec.ResponseDecoder Decoding 'get 
head' response
19.10.2016 16:22:38.038 *DEBUG* [sling-default-81-Registered Service.607] 
org.apache.jackrabbit.oak.segment.standby.client.StandbyClient Channel closed
19.10.2016 16:22:40.241 *DEBUG* [sling-default-81-Registered Service.607] 
org.apache.jackrabbit.oak.segment.standby.client.StandbyClient Group shut down
19.10.2016 16:22:40.242 *ERROR* [sling-default-81-Registered Service.607] 
org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync Failed 
synchronizing state.
java.lang.RuntimeException: Error occurred while obtaining InputStream for 
blobId [4dfc748c91d518c9221031ec6115fd7ac04fe27b#10]
at 
org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:49)
at 
org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:252)
at 
org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:87)
at 
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.getInput(AbstractBlob.java:45)
at 
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.getInput(AbstractBlob.java:42)
at com.google.common.io.ByteStreams$3.openStream(ByteStreams.java:907)
at com.google.common.io.ByteSource.contentEquals(ByteSource.java:301)
at com.google.common.io.ByteStreams.equal(ByteStreams.java:661)
at 
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:68)
at 
org.apache.jackrabbit.oak.segment.SegmentBlob.equals(SegmentBlob.java:193)
at com.google.common.base.Objects.equal(Objects.java:55)
at 
org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:53)
at 
org.apache.jackrabbit.oak.segment.SegmentPropertyState.equals(SegmentPropertyState.java:249)
at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:622)
at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:516)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
at 
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:415)
at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
at 
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:457)
at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
at 
org.apache.jackrabbit.oak.segment.MapRecord$2.childNodeChanged(MapRecord.java:401)
at 
org.apache.jackrabbit.oak.segment.MapRecord$3.childNodeChanged(MapRecord.java:442)
at 
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:485)
at 
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:433)
at 
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:391)
at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:609)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:216)
at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:186)
at

[jira] [Created] (OAK-4968) Query engine: sort order is calculated multiple times unnecessarily

Thomas Mueller created OAK-4968:
---

 Summary: Query engine: sort order is calculated multiple times 
unnecessarily
 Key: OAK-4968
 URL: https://issues.apache.org/jira/browse/OAK-4968
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: query
Reporter: Thomas Mueller
Assignee: Thomas Mueller
 Fix For: 1.5.13


In QueryImpl.getBestSelectorExecutionPlan, the sort order is calculated for 
each index, which is unnecessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


[ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591870#comment-15591870
 ] 

Julian Reschke commented on OAK-4964:
-

trunk: [r1765817|http://svn.apache.org/r1765817]

> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.6, 1.5.13
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4968) Query engine: sort order is calculated multiple times unnecessarily


 [ 
https://issues.apache.org/jira/browse/OAK-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller resolved OAK-4968.
-
Resolution: Fixed

https://svn.apache.org/r1765818 (trunk)

> Query engine: sort order is calculated multiple times unnecessarily
> ---
>
> Key: OAK-4968
> URL: https://issues.apache.org/jira/browse/OAK-4968
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Fix For: 1.5.13
>
>
> In QueryImpl.getBestSelectorExecutionPlan, the sort order is calculated for 
> each index, which is unnecessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


 [ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke resolved OAK-4964.
-
   Resolution: Fixed
Fix Version/s: 1.5.13

> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.6, 1.5.13
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


 [ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4964:

Labels: candidate_oak_1_0 candidate_oak_1_2 candidate_oak_1_4  (was: )

> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.6
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id


 [ 
https://issues.apache.org/jira/browse/OAK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4964:

Attachment: OAK-4964.diff

Proposed patch.

> UpdateOp.set("_id", ...) should do a sanity check on the id
> ---
>
> Key: OAK-4964
> URL: https://issues.apache.org/jira/browse/OAK-4964
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.6
>
> Attachments: OAK-4964.diff
>
>
> ...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4967) Offload the node deduplication cache offheap partially

2016-10-20 Thread Alex Parvulescu (JIRA)

Alex Parvulescu created OAK-4967:


 Summary: Offload the node deduplication cache offheap partially
 Key: OAK-4967
 URL: https://issues.apache.org/jira/browse/OAK-4967
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: segment-tar
Reporter: Alex Parvulescu


The node deduplication cache sits by default at {{8388608}} by default [0] 
which means it can grow up to {{1.6GB}} on its own. It would be interesting to 
look into offloading some it the items offheap by configuration, to reduce the 
effect a full compaction might have on a running instance (and possibly in 
general reduce the on-heap footprint).



[0] 
https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/WriterCacheManager.java#L70



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4966) Re-introduce a blocker for compaction based on available heap

2016-10-20 Thread Alex Parvulescu (JIRA)

Alex Parvulescu created OAK-4966:


 Summary: Re-introduce a blocker for compaction based on available 
heap
 Key: OAK-4966
 URL: https://issues.apache.org/jira/browse/OAK-4966
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: segment-tar
Reporter: Alex Parvulescu


As seen in a local test, running compaction on a tight heap can lead to OOMEs. 
There used to be a best effort barrier against this situation 'not enough heap 
for compaction', but we removed it with the compaction maps.
I think it makes sense to add it again based on the max size of some of the 
caches: segment cache {{256MB}} by default [0] and some writer caches which can 
go up to {{2GB}} all combined [1] and probably others I missed.


[0] 
https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentCache.java#L48
[1] 
https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/WriterCacheManager.java#L50



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4965) Cold standby logs SNFE ERROR

2016-10-20 Thread Andrei Dulceanu (JIRA)

Andrei Dulceanu created OAK-4965:


 Summary: Cold standby logs SNFE ERROR
 Key: OAK-4965
 URL: https://issues.apache.org/jira/browse/OAK-4965
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: segment-tar
Affects Versions: Segment Tar 0.0.14
Reporter: Andrei Dulceanu
Assignee: Andrei Dulceanu
 Fix For: Segment Tar 0.0.18


On coldstandby, there are a lot of occurences of SNFE:
{code}
200766 04.10.2016 14:29:52.657 *ERROR* [sling-default-16-Registered 
Service.577] org.apache.jackrabbit.oak.segment.SegmentId Segment not found: 
19d493e3-8bad-4124-a962-5388d91f560e. SegmentId age=0ms
200767 org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 
19d493e3-8bad-4124-a962-5388d91f560e not found
200768 at 
org.apache.jackrabbit.oak.segment.file.FileStore$14.call(FileStore.java:1345)
200769 at 
org.apache.jackrabbit.oak.segment.file.FileStore$14.call(FileStore.java:1285)
200770 at 
org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1013)
200771 at 
org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:974)
200772 at 
org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:285)
200773 at 
org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:92)
200774 at 
org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1285)
200775 at 
org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:123)
200776 at 
org.apache.jackrabbit.oak.segment.Record.getSegment(Record.java:70)
200777 at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.getStableIdBytes(SegmentNodeState.java:139)
200778 at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.getStableId(SegmentNodeState.java:122)
200779 at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.fastEquals(SegmentNodeState.java:633)
200780 at 
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:459)
200781 at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:100)
200782 at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:80)
200783 at 
org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:143)
200784 at 
org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:118)
200785 at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
200786 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
200787 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
200788 at java.lang.Thread.run(Thread.java:745)
200789 04.10.2016 14:29:52.657 *INFO* [sling-default-16-Registered Service.577] 
org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution 
Found missing segment 19d493e3-8bad-4124-a962-5388d91f560e
200790 04.10.2016 14:29:52.657 *INFO* [sling-default-16-Registered Service.577] 
org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution 
Loading segment 19d493e3-8bad-4124-a962-5388d91f560e
200791 04.10.2016 14:29:52.657 *DEBUG* [nioEventLoopGroup-208-1] 
org.apache.jackrabbit.oak.segment.standby.codec.GetSegmentRequestEncoder 
Sending request from client qastandby1 for segment 
19d493e3-8bad-4124-a962-5388d91f560e
{code}

While these are false positives (the segment is found later), we need to find a 
way to avoid logging the errors. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4964) UpdateOp.set("_id", ...) should do a sanity check on the id

Julian Reschke created OAK-4964:
---

 Summary: UpdateOp.set("_id", ...) should do a sanity check on the 
id
 Key: OAK-4964
 URL: https://issues.apache.org/jira/browse/OAK-4964
 Project: Jackrabbit Oak
  Issue Type: Technical task
  Components: documentmk
Reporter: Julian Reschke
Assignee: Julian Reschke
Priority: Minor


...checking that it is consistent with the {{id}} member field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4949) SegmentWriter buffers child node list changes

2016-10-20 Thread Alex Parvulescu (JIRA)


[ 
https://issues.apache.org/jira/browse/OAK-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591781#comment-15591781
 ] 

Alex Parvulescu commented on OAK-4949:
--

I think the change here is that compaction itself is a very large transaction 
and might introduce this problem. I'm wondering if we could only approach a 
single code path for optimization, the 'persisting a new node with a very large 
child node list' scenario.
This is in any case an improvement analysis, not something we need to urgently 
fix.

> SegmentWriter buffers child node list changes
> -
>
> Key: OAK-4949
> URL: https://issues.apache.org/jira/browse/OAK-4949
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
>
> The {{SegmentWriter}} currently buffers the list of child nodes changed on a 
> nodestate update [0] (new node or updated node). This can be problematic in a 
> scenario where there are a large number of children added to a node (ie. 
> unique index size seen to spike above {{10MM}} in one case).
> To have a reference for the impact of this, at the {{SegmentWriter}} level, 
> for a list of map entries of almost {{3MM}} items, I saw it take up around 
> {{245MB}} heap.
> This issue serves to track a possible improvement here in how we handle this 
> update scenario.
> [0] 
> https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentWriter.java#L516



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4951) UpdateOp without set operation for _id: clarify, add unit tests, fix DocumentStore instances


 [ 
https://issues.apache.org/jira/browse/OAK-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4951:

Attachment: OAK-4951.diff

Started work by removing the {{set("_id", ...)}} calls in 
{{BasicDocumentStoreTest}}.

{{MemoryDocumentStore}} and {{RDBDocumentStore}} can easily be fixed with a 
minor change in {{UpdateUtils}}. However, {{MongoDocumentStore}} has many 
failures.

> UpdateOp without set operation for _id: clarify, add unit tests, fix 
> DocumentStore instances
> 
>
> Key: OAK-4951
> URL: https://issues.apache.org/jira/browse/OAK-4951
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: documentmk
>Affects Versions: 1.0.34, 1.4.8, 1.2.20, 1.5.12
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Minor
>  Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.6
>
> Attachments: OAK-4951.diff
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-1312) Bundle nodes into a document


 [ 
https://issues.apache.org/jira/browse/OAK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-1312:

Attachment: benchmark-result-postgres.txt
benchmark-result-db2.txt

RDB results for PostgreSQL and DB2.

> Bundle nodes into a document
> 
>
> Key: OAK-1312
> URL: https://issues.apache.org/jira/browse/OAK-1312
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: core, documentmk
>Reporter: Marcel Reutegger
>Assignee: Chetan Mehrotra
>  Labels: docs-impacting, performance
> Fix For: 1.6, 1.5.13
>
> Attachments: OAK-1312-meta-prop-handling.patch, 
> OAK-1312-review-v1.diff, OAK-1312-review-v2.diff, benchmark-result-db2.txt, 
> benchmark-result-postgres.txt, benchmark-results.txt, run-benchmark.sh
>
>
> For very fine grained content with many nodes and only few properties per 
> node it would be more efficient to bundle multiple nodes into a single 
> MongoDB document. Mostly reading would benefit because there are less 
> roundtrips to the backend. At the same time storage footprint would be lower 
> because metadata overhead is per document.
> Feature branch - 
> https://github.com/chetanmeh/jackrabbit-oak/compare/trunk...chetanmeh:OAK-1312



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation

2016-10-20 Thread Stefan Egli (JIRA)

[
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588993#comment-15588993
]

Stefan Egli edited comment on OAK-4581 at 10/20/16 9:55 AM:

h4. Prototype based on a SwitchingObserver
[pushed a
prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7]
to my github fork (in [this
branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581]) which
tries to scetch an approach that would use a SwitchingObserer in front of
either a BackgroundObserver/ChangeProcessor pair or a
PersistingChangeProcessor/PersistingEventQueue pair. The disadvantage of this
approach is that there are quite a few new classes in play. The advantage is
that it clearly distinguishes between normal (live) mode - by using the
exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a new
persistent mode. Switching between these modes is delicate and is thus
explicitly handled with extra switching modes. This conceptually works fine -
added a test that illustrates the switch.

h4. Prototype based on persistence embedded in the ChangeProcessor
An alternative to the above would be to embedd the persistence directly in the
ChangeProcessor. The contentChanged method there would inspect the queue size
and if too large, not deliver events to the listener anymore, but instead go
via a persistent queue. The advantage is that it's more enclosed in the
ChangeProcessor. The disadvantage is that if a listener would block it could
not prevent the queue from growing (but maybe supporting an indefinitely
blocking listener is not required). I'll look into how such an approach could
be implemented next.
UPDATE: Added a prototype of this second variant too (in [this
branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581-type2]).

h4. Comparison of the approaches
The two approaches basically represent _'head of queue persisting'_
(PersistingEventListener) versus _'tail of queue persisting'_
(SwitchingObserver).
h5. Falling out of the documentMk's diff cache
* 'tail of queue persisting' has the advantage that it supports any type of
storm coming in. It would immediately start persisting newly enqueued commits.
(At the moment those diffs already in the queue are worked off 'slowly' at
onEvent time, but this could be changed to be force-persisted too. EDIT: but
basically 'tail of queue persisting' is a fail-safe for newly added commits
which would allow the listener to have infinite time for working off the
inmemory queue)
* 'head of queue persisting' is less favorable when falling out of the diff
cache, as it would basically start persisting the head at max speed (ie without
onEvent cost), but perhaps that max speed is slower than new commits coming in.

h4. Conclusion
While the head-of-queue-persisting/PersistingEventListener seems to be the more
KISS/elegant solution, it has downsides under heavy load. It seems thus better
to do tail-of-queue-persisting with eg the SwitchingObserver.

What do people think? /cc [~chetanm], [~mduerig], [~mreutegg], [~catholicon],
[~tmueller]

was (Author: egli):
h4. Prototype based on a SwitchingObserver
[pushed a
prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7]
to my github fork (in [this
branch|https://github.com/stefan-egli/jackrabbit-oak/tree/OAK-4581]) which
tries to scetch an approach that would use a SwitchingObserer in front of
either a BackgroundObserver/ChangeProcessor pair or a
PersistingChangeProcessor/PersistingEventQueue pair. The disadvantage of this
approach is that there are quite a few new classes in play. The advantage is
that it clearly distinguishes between normal (live) mode - by using the
exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a new
persistent mode. Switching between these modes is delicate and is thus
explicitly handled with extra switching modes. This conceptually works fine -
added a test that illustrates the switch.

[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation

2016-10-20 Thread Stefan Egli (JIRA)

[
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588993#comment-15588993
]

Stefan Egli edited comment on OAK-4581 at 10/20/16 9:50 AM:

h4. Comparison of the approaches
The two approaches basically represent _'head of queue persisting'_
(PersistingEventListener) versus _'tail of queue persisting'_
(SwitchingObserver).
h5. Falling out of the documentMk's diff cache
* 'tail of queue persisting' has the advantage that it supports any type of
storm coming in. It would immediately start persisting newly enqueued commits.
(At the moment those diffs already in the queue are worked off 'slowly' at
onEvent time, but this could be changed to be force-persisted too)
* 'head of queue persisting' is less favorable when falling out of the diff
cache, as it would basically start persisting the head at max speed (ie without
onEvent cost), but perhaps that max speed is slower than new commits coming in.

What do people think? /cc [~chetanm], [~mduerig], [~mreutegg], [~catholicon],
[~tmueller]

[jira] [Comment Edited] (OAK-4581) Persistent local journal for more reliable event generation

2016-10-20 Thread Stefan Egli (JIRA)

[
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588993#comment-15588993
]

Stefan Egli edited comment on OAK-4581 at 10/20/16 9:49 AM:

h4. Comparison of the approaches
The two approaches basically represent _'head of queue persisting'_
(PersistingEventListener) versus _'tail of queue persisting'_
(SwitchingObserver).
h5. Falling out of the documentMk's diff cache
* 'tail of queue persisting' has the advantage that it supports any type of
storm coming in. It would immediately start persisting newly enqueued commits.
(At the moment those diffs already in the queue are worked off 'slowly' at
onEvent time, but this could be changed to be force-persisted too)
* 'head of queue persisting' is less favorable when falling out of the diff
cache, as it would basically start persisting the head at max speed (ie without
onEvent cost), but perhaps that max speed is slower than new commits coming in.

h5. Storm of commits coming in
* 'tail of queue persisting' again is deterministic for this case as it just
routes incoming commits to persistence for good - no possibility of further
queue growth. And the remaining missing feature of not-yet-force-flushing the
inmemory queue is not a problem here as we remain in the cache.
* 'head of queue persisting' would likely be overwhelmed here: it depends on
how rapidly it can filter/generate/persist off the queue vs how quickly new
commits come in.
h4. Conclusion
While the head-of-queue-persisting/PersistingEventListener seems to be the more
KISS/elegant solution, it has downsides under heavy load. It seems thus better
to do tail-of-queue-persisting with eg the SwitchingObserver.

What do people think? /cc [~chetanm], [~mduerig], [~mreutegg], [~catholicon],
[~tmueller]

was (Author: egli):
h4. Prototype based on a SwitchingObserver
[pushed a
prototype|https://github.com/stefan-egli/jackrabbit-oak/commit/a521599e89b62ed8d40af5ae0a987cb4a9b546b7]
to my github fork which tries to scetch an approach that would use a
SwitchingObserer in front of either a BackgroundObserver/ChangeProcessor pair
or a PersistingChangeProcessor/PersistingEventQueue pair. The disadvantage of
this approach is that there are quite a few new classes in play. The advantage
is that it clearly distinguishes between normal (live) mode - by using the
exiting, unchanged BackgroundObserver/ChangeProcessor pair - and a new
persistent mode. Switching between these modes is delicate and is thus
explicitly handled with extra switching modes. This conceptually works fine -
added a test that illustrates the switch.

[jira] [Comment Edited] (OAK-4887) Query cost estimation: ordering by an unindexed property not reflected


[ 
https://issues.apache.org/jira/browse/OAK-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591350#comment-15591350
 ] 

Thomas Mueller edited comment on OAK-4887 at 10/20/16 9:39 AM:
---

[~alexander.klimetschek] could you provide an example (query, index 
definitions) so that I can understand what is the most pressing issue? Would it 
help in your case if we add "query option" mechanism as follows, so that the 
query engine knows what is more important?

Example (about matchin 2000 rows)

{noformat}
//element(*, 'ec:Product')[@color = 'red' and @size = 'L'] order by @popularity 
desc
-> expectes the whole result to be read
-> will favor an index on color, size

//element(*, 'ec:Product')[@color = 'red' and @size = 'L'] order by @popularity 
desc option(fast 10)
-> will favor an index on ec:Product + popularity, 
so that the first 10 rows can be returned quickly
{noformat}


was (Author: tmueller):
[~alexander.klimetschek] could you provide an example (query, index 
definitions) so that I can understand what is the most pressing issue? Would it 
help in your case if we add "query option" mechanism as follows, so that the 
query engine knows what is more important?

Example (about 1 rows with 

{noformat}
//element(*, 'ec:Product')[@color = 'red' and @size = 'L'] order by @popularity 
desc
-> expectes the whole result to be read
-> will favor an index on color, size

//element(*, 'ec:Product')[@color = 'red' and @size = 'L'] order by @popularity 
desc option(fast 10)
-> will favor an index on ec:Product + popularity, 
so that the first 10 rows can be returned quickly
{noformat}

> Query cost estimation: ordering by an unindexed property not reflected
> --
>
> Key: OAK-4887
> URL: https://issues.apache.org/jira/browse/OAK-4887
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Affects Versions: 1.4.2
>Reporter: Alexander Klimetschek
>Assignee: Thomas Mueller
> Fix For: 1.6
>
>
> A query that orders by an unindexed property seems to have no effect on the 
> cost estimation, compared to the same query without the order by, although it 
> has a big impact on the execution performance for larger results/indexes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4887) Query cost estimation: ordering by an unindexed property not reflected


[ 
https://issues.apache.org/jira/browse/OAK-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591350#comment-15591350
 ] 

Thomas Mueller commented on OAK-4887:
-

[~alexander.klimetschek] could you provide an example (query, index 
definitions) so that I can understand what is the most pressing issue? Would it 
help in your case if we add "query option" mechanism as follows, so that the 
query engine knows what is more important?

Example (about 1 rows with 

{noformat}
//element(*, 'ec:Product')[@color = 'red' and @size = 'L'] order by @popularity 
desc
-> expectes the whole result to be read
-> will favor an index on color, size

//element(*, 'ec:Product')[@color = 'red' and @size = 'L'] order by @popularity 
desc option(fast 10)
-> will favor an index on ec:Product + popularity, 
so that the first 10 rows can be returned quickly
{noformat}

> Query cost estimation: ordering by an unindexed property not reflected
> --
>
> Key: OAK-4887
> URL: https://issues.apache.org/jira/browse/OAK-4887
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Affects Versions: 1.4.2
>Reporter: Alexander Klimetschek
>Assignee: Thomas Mueller
> Fix For: 1.6
>
>
> A query that orders by an unindexed property seems to have no effect on the 
> cost estimation, compared to the same query without the order by, although it 
> has a big impact on the execution performance for larger results/indexes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-1312) Bundle nodes into a document


 [ 
https://issues.apache.org/jira/browse/OAK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chetan Mehrotra updated OAK-1312:
-
Labels: docs-impacting performance  (was: performance)

> Bundle nodes into a document
> 
>
> Key: OAK-1312
> URL: https://issues.apache.org/jira/browse/OAK-1312
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: core, documentmk
>Reporter: Marcel Reutegger
>Assignee: Chetan Mehrotra
>  Labels: docs-impacting, performance
> Fix For: 1.6, 1.5.13
>
> Attachments: OAK-1312-meta-prop-handling.patch, 
> OAK-1312-review-v1.diff, OAK-1312-review-v2.diff, benchmark-results.txt, 
> run-benchmark.sh
>
>
> For very fine grained content with many nodes and only few properties per 
> node it would be more efficient to bundle multiple nodes into a single 
> MongoDB document. Mostly reading would benefit because there are less 
> roundtrips to the backend. At the same time storage footprint would be lower 
> because metadata overhead is per document.
> Feature branch - 
> https://github.com/chetanmeh/jackrabbit-oak/compare/trunk...chetanmeh:OAK-1312



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-1312) Bundle nodes into a document


 [ 
https://issues.apache.org/jira/browse/OAK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chetan Mehrotra resolved OAK-1312.
--
   Resolution: Fixed
Fix Version/s: 1.5.13

Resolving this as done. Specific issue can be open later on as per needs. 
Benchmark updates can continue to be added

> Bundle nodes into a document
> 
>
> Key: OAK-1312
> URL: https://issues.apache.org/jira/browse/OAK-1312
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: core, documentmk
>Reporter: Marcel Reutegger
>Assignee: Chetan Mehrotra
>  Labels: performance
> Fix For: 1.6, 1.5.13
>
> Attachments: OAK-1312-meta-prop-handling.patch, 
> OAK-1312-review-v1.diff, OAK-1312-review-v2.diff, benchmark-results.txt, 
> run-benchmark.sh
>
>
> For very fine grained content with many nodes and only few properties per 
> node it would be more efficient to bundle multiple nodes into a single 
> MongoDB document. Mostly reading would benefit because there are less 
> roundtrips to the backend. At the same time storage footprint would be lower 
> because metadata overhead is per document.
> Feature branch - 
> https://github.com/chetanmeh/jackrabbit-oak/compare/trunk...chetanmeh:OAK-1312



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-1312) Bundle nodes into a document


[ 
https://issues.apache.org/jira/browse/OAK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588423#comment-15588423
 ] 

Chetan Mehrotra edited comment on OAK-1312 at 10/20/16 9:09 AM:


h4. Benchmark - Result with bundling enabled

Ran a benchmark using [script|^run-benchmark.sh] with 
[results|^benchmark-results.txt]. Script also dumps Mongo DB stats, Metrics 
stats etc. Results are also summarized 
[here|https://docs.google.com/spreadsheets/d/1lzwDjwS-HSL0WazYBx9Wtx2ZI3J-fGl-EJ08-rxdAE8]

{noformat}
+--+---+-+-+-+--+--+-++-+-+++--+++--+
| Fixtues  | C | min | 10% | 50% | 90%  | max  | N   | Reader | Mutator | 
Assets# | Mongo Doc# | Mongo Size | Idx Size | Find#  | Query# | Comment
  |
+--+---+-+-+-+--+--+-++-+-+++--+++--+
| Oak-Mongo-DS | 5 | 360 | 483 | 710 | 1509 | 2843 | 350 | 75251  | 2504| 
3680| 56966  | 58 | 43   | 44387  | 2808   | #default   
  |
| Oak-Mongo-DS | 5 | 346 | 477 | 787 | 1508 | 2498 | 336 | 41805  | 1798| 
3480| 8710   | 36 | 5| 5105   | 1906   | #bundling,ALL  
  |
| Oak-Mongo-DS | 5 | 312 | 469 | 746 | 1491 | 2630 | 339 | 67085  | 2268| 
3550| 30162  | 58 | 22   | 26655  | 12008  | 
#bundling,EXCLUDE_RENDITIONS |
+--+---+-+-+-+--+--+-++-+-+++--+++--+

{noformat}

*Environment details*
{noformat}
$ uname -a
Linux xxx 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 
x86_64 x86_64 GNU/Linux

$ java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

$ mongo -version
MongoDB shell version: 2.6.4
{noformat}

*Legend*
* Mongo Doc# - number of Mongo documents across all collections
* Mongo Size - Size in MB of Mongo DB
* Idx Size - Size of all indexes in Mongo (MB)
* ALL - It uses bundling pattern {{jcr:content, jcr:content/metadata, 
jcr:content/renditions/**}}
* EXCLUDE_RENDITIONS -  It uses bundling pattern {{jcr:content, 
jcr:content/metadata}}


*Highlights*
* With ALL bundling there is a significant reduction in 
** Mongo docs - 56966 -> 8710
** Index size - 43 -> 5
** Calls to mongo for find
* BUT there is a decrease in read/write also
** Reads 75251 -> 41805
** Updates 2504 -> 1798
* Changing the bundling pattern helps in improving reads 

So bundling leads to very signification savings in Mongo level storage. However 
has some adverse impacts on read and updates. 

*Next Steps*
* Merge current branch to trunk - As shown in previous comment if bundling is 
disabled there is no perf imapct. So its safe in disabled state
* Analyze why reads have reduced - Given that access should involve lesser 
number of remote calls we need to see why reads are slow
* benchmark in more real world scenarios where the read access pattern is more 
real
* Benchmark on RDB - [~reschke] Can you run it against any DB setup you have 
once I have done the merge to trunk
* Benchmark with Mongo 3.x - [~mreutegg] Can you try it against Wired Tiger

/cc [~mreutegg] [~catholicon] [~ianeboston] [~alexxx]  [~mmarth] [~tmueller]


was (Author: chetanm):
h4. Benchmark - Result with bundling enabled

Ran a benchmark using [script|^run-benchmark.sh] with 
[results|^benchmark-results.txt]. Script also dumps Mongo DB stats, Metrics 
stats etc. Results are also summarized 
[here|https://docs.google.com/spreadsheets/d/1lzwDjwS-HSL0WazYBx9Wtx2ZI3J-fGl-EJ08-rxdAE8]

{noformat}
+--+---+-+-+-+--+--+-++-+-+++--+++--+
| Fixtues  | C | min | 10% | 50% | 90%  | max  | N   | Reader | Mutator | 
Assets# | Mongo Doc# | Mongo Size | Idx Size | Find#  | Query# | Comment
  |
+--+---+-+-+-+--+--+-++-+-+++--+++--+
| Oak-Mongo-DS | 5 | 360 | 483 | 710 | 1509 | 2843 | 350 | 75251  | 2504| 
3680| 56966  | 58 | 43   | 44387  | 2808   | #default   
  |
| Oak-Mongo-DS | 5 | 346 | 477 | 787 | 1508 | 2498 | 336 | 41805  | 1798| 
3480| 8710   | 36 | 5| 5105   | 1906   | #bundling,ALL  
  |
| Oak-Mongo-DS | 5 | 312 | 469 | 746 | 1491 | 2630 | 339 | 67085  | 2268| 
3550| 30162  | 58 | 22   | 26655  | 12008  |

[jira] [Updated] (OAK-4959) Review the security aspect of bundling configuration