[jira] [Commented] (OAK-4581) Persistent local journal for more reliable event generation

[Chetan Mehrotra (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550982#comment-15550982
 ] 

Chetan Mehrotra commented on OAK-4581:
--

Well I am still bit sceptical around serializing the whole event stuff but lets 
try. I think feature like this might need multiple implementations as poc, done 
in branch. And then we run certain scenarios or benchmark test to see the 
effectiveness of the approach. 

So its fine to start with one approach in a branch and measure the impact

> Persistent local journal for more reliable event generation
> ---
>
> Key: OAK-4581
> URL: https://issues.apache.org/jira/browse/OAK-4581
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: core
>Reporter: Chetan Mehrotra
>Assignee: Stefan Egli
>  Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4581.v0.patch
>
>
> As discussed in OAK-2683 "hitting the observation queue limit" has multiple 
> drawbacks. Quite a bit of work is done to make diff generation faster. 
> However there are still chances of event queue getting filled up. 
> This issue is meant to implement a persistent event journal. Idea here being
> # NodeStore would push the diff into a persistent store via a synchronous 
> observer
> # Observors which are meant to handle such events in async way (by virtue of 
> being wrapped in BackgroundObserver) would instead pull the events from this 
> persisted journal
> h3. A - What is persisted
> h4. 1 - Serialized Root States and CommitInfo
> In this approach we just persist the root states in serialized form. 
> * DocumentNodeStore - This means storing the root revision vector
> * SegmentNodeStore - {color:red}Q1 - What does serialized form of 
> SegmentNodeStore root state looks like{color} - Possible the RecordId of 
> "root" state
> Note that with OAK-4528 DocumentNodeStore can rely on persisted remote 
> journal to determine the affected paths. Which reduces the need for 
> persisting complete diff locally.
> Event generation logic would then "deserialize" the persisted root states and 
> then generate the diff as currently done via NodeState comparison
> h4. 2 - Serialized commit diff and CommitInfo
> In this approach we can save the diff in JSOP form. The diff only contains 
> information about affected path. Similar to what is current being stored in 
> DocumentNodeStore journal
> h4. CommitInfo
> The commit info would also need to be serialized. So it needs to be ensure 
> whatever is stored there can be serialized or re calculated
> h3. B - How it is persisted
> h4. 1 - Use a secondary segment NodeStore
> OAK-4180 makes use of SegmentNodeStore as a secondary store for caching. 
> [~mreutegg] suggested that for persisted local journal we can also utilize a 
> SegmentNodeStore instance. Care needs to be taken for compaction. Either via 
> generation approach or relying on online compaction
> h4. 2- Make use of write ahead log implementations
> [~ianeboston] suggested that we can make use of some write ahead log 
> implementation like [1], [2] or [3]
> h3. C - How changes get pulled
> Some points to consider for event generation logic
> # Would need a way to keep pointers to journal entry on per listener basis. 
> This would allow each Listener to "pull" content changes and generate diff as 
> per its speed and keeping in memory overhead low
> # The journal should survive restarts
> [1] http://www.mapdb.org/javadoc/latest/mapdb/org/mapdb/WriteAheadLog.html
> [2] 
> https://github.com/apache/activemq/tree/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/journal
> [3] 
> https://github.com/elastic/elasticsearch/tree/master/core/src/main/java/org/elasticsearch/index/translog



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4796) filter events before adding to ChangeProcessor's queue

[Chetan Mehrotra (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550976#comment-15550976
 ] 

Chetan Mehrotra commented on OAK-4796:
--

bq. Thing now is that for external and overflow entries the CommitInfo is null. 
So I'd say, as long as that's the case it's very hard to implement correctly 
switching the filter.

Opened OAK-4898 to discuss that. 

> filter events before adding to ChangeProcessor's queue
> --
>
> Key: OAK-4796
> URL: https://issues.apache.org/jira/browse/OAK-4796
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: jcr
>Affects Versions: 1.5.9
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>  Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4796.changeSet.patch, OAK-4796.patch
>
>
> Currently the 
> [ChangeProcessor.contentChanged|https://github.com/apache/jackrabbit-oak/blob/f4f4e01dd8f708801883260481d37fdcd5868deb/oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ChangeProcessor.java#L335]
>  is in charge of doing the event diffing and filtering and does so in a 
> pooled Thread, ie asynchronously, at a later stage independent from the 
> commit. This has the advantage that the commit is fast, but has the following 
> potentially negative effects:
> # events (in the form of ContentChange Objects) occupy a slot of the queue 
> even if the listener is not interested in it - any commit lands on any 
> listener's queue. This reduces the capacity of the queue for 'actual' events 
> to be delivered. It therefore increases the risk that the queue fills - and 
> when full has various consequences such as loosing the CommitInfo etc.
> # each event==ContentChange later on must be evaluated, and for that a diff 
> must be calculated. Depending on runtime behavior that diff might be 
> expensive if no longer in the cache (documentMk specifically).
> As an improvement, this diffing+filtering could be done at an earlier stage 
> already, nearer to the commit, and in case the filter would ignore the event, 
> it would not have to be put into the queue at all, thus avoiding occupying a 
> slot and later potentially slower diffing.
> The suggestion is to implement this via the following algorithm:
> * During the commit, in a {{Validator}} the listener's filters are evaluated 
> - in an as-efficient-as-possible manner (Reason for doing it in a Validator 
> is that this doesn't add overhead as oak already goes through all changes for 
> other Validators). As a result a _list of potentially affected observers_ is 
> added to the {{CommitInfo}} (false positives are fine).
> ** Note that the above adds cost to the commit and must therefore be 
> carefully done and measured
> ** One potential measure could be to only do filtering when listener's queues 
> are larger than a certain threshold (eg 10)
> * The ChangeProcessor in {{contentChanged}} (in the one created in 
> [createObserver|https://github.com/apache/jackrabbit-oak/blob/f4f4e01dd8f708801883260481d37fdcd5868deb/oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ChangeProcessor.java#L224])
>  then checks the new commitInfo's _potentially affected observers_ list and 
> if it's not in the list, adds a {{NOOP}} token at the end of the queue. If 
> there's already a NOOP there, the two are collapsed (this way when a filter 
> is not affected it would have a NOOP at the end of the queue). If later on a 
> no-NOOP item is added, the NOOP's {{root}} is used as the {{previousRoot}} 
> for the newly added {{ContentChange}} obj.
> ** To achieve that, the ContentChange obj is extended to not only have the 
> "to" {{root}} pointer, but also the "from" {{previousRoot}} pointer which 
> currently is implicitly maintained.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4898) Allow for external changes to have a CommitInfo attached

[Chetan Mehrotra (JIRA)]

Chetan Mehrotra created OAK-4898:


 Summary: Allow for external changes to have a CommitInfo attached
 Key: OAK-4898
 URL: https://issues.apache.org/jira/browse/OAK-4898
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: core
Reporter: Chetan Mehrotra
 Fix For: 1.6


Currently the observation logic relies on fact that CommitInfo being null means 
that changes are from other cluster node i.e. external changes. 

We should change this semantic and provide a different way to indicate that 
changes are external. This would allow a NodeStore implementation to still pass 
in a CommitInfo which captures useful information about commit like brief 
summary on what got changed which can be used for pre filtering (OAK-4796)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4796) filter events before adding to ChangeProcessor's queue

[Chetan Mehrotra (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550962#comment-15550962
 ] 

Chetan Mehrotra commented on OAK-4796:
--

Overall approach look fine. Below is bit high level feedback on current state 
of patch. 

If possible break this task in 2 sub task. 
* Change collection - Work being done in 
{{PrefilterCollectorValidatorProvider}} is not limited to pre filtering i.e. it 
does not depend on observation logic. So we can have it as independent task
* Prefiltering in ChangeProcessor - This make use of previous work to do pre 
filtering

h4. Collecting Changes in a commit
   
* {{PrefilterCollectorValidatorProvider}}
** Use OSGi config instead of using system properties
** MAX_PREFILTER_PATH_DEPTH - When I referred to limit on paths it was only for 
limiting the set of changed paths upto certain depth. Intention was to limit in 
memory cost of changed path set. However other data points should still be 
collected irrespective of depth. For those data points we can have different 
limits. So {{childNodeXXX}} method should always traverse. Also it should be 
fine to not overflow on paths as its always possible to use whatever 
information is collected

*Naming*
PrefilterCollectorValidatorProvider name looks bit confusing i.e. collector <=> 
validator are two different concerns. Its fine to piggy back on Validator 
extension point but lets name it different like {{ChangeCollectorProvider}} 
which returns an instance of {{ChangeCollector}} which sets a {{ChangeSet}} 
instance in {{CommitContext}}. This {{ChangeSet}} gets used in pre filtering is 
secondary i.e. in future we might use this information with any other feature 
as its not limited to prefiltering.

h4. ChangeProcessor and Observation

* {{CommitInfo.NOOP_CHANGE}} - This is very specific to observation. May be 
define this in {{BackgroundObserver}} and not introduce it globally
* {{Prefilter}} - Have a default noop instance which returns false. Use that in 
{{FilterBuilder}} to avoid null check. 
{code}
@Override
public boolean excludeCommit(PrefilterChangeSet prefilterChangeSet) 
{
if (prefilter != null) {
return prefilter.excludeCommit(prefilterChangeSet);
} else {
return false;
}
}
{code}
* {{PrefilterImpl}} - Not reviewed the filtering logic completely so far 
however move the test setup in main to a proper testclass. It would be 
important to have decent coverage for this class 
** Probably we can keep things simple here and focus on included paths - If any 
of change paths (upto level n in ChangeSet) is ancestor then its go otherwise 
no go
** Try avoid null checks by having instance initialized with empty variants of 
collection. Reduces noise!
* {{ChangeProcessor}} - 
** Not sure on PREFILTERING_LIMIT. Hard work is already done in change 
collection. So if pre filtering feature is enabled then using that should be 
fine as whole pre filtering logic is not costly compared to actual diffing
** {{isExcludedByPrefiltering}} 
*** Have it return an enum with 3 states. Use of nullable Boolean is confusing
*** {{fp.getFilter}} is marked as NonNull so this check is redundant. So no 
need to trigger EventFilter building for prefiltering case 
{code}
EventFilter filter = fp.getFilter(before, after);
if (filter == null) {
// if the filter provider doesn't return a filter this 
corresponds
// to filtering-out
return true;
}
{code}
*** info map is non null (we should add annotation in CommitInfo for that). So 
this check can be skipped
{code}
Map m = info.getInfo();
if (m == null) {
return null;
}
{code}
* BackgroundObserver - Need unit test for changes being done here as semantics 
of noop handling are subtle here

May be we can introduce a new interface {{PrefilterAwareObserver}} which can be 
passed extra params like pre filtering result. Have {{ChangeProcessor}} provide 
an impl for that to {{BackgroundObserver}}. This would avoid need to 
recalculate prefiltering result in {{ChangeProcessor}} as being done for test 
code

> filter events before adding to ChangeProcessor's queue
> --
>
> Key: OAK-4796
> URL: https://issues.apache.org/jira/browse/OAK-4796
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: jcr
>Affects Versions: 1.5.9
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>  Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4796.changeSet.patch, OAK-4796.patch
>
>
> Currently the 
> 

[jira] [Resolved] (OAK-3753) Test failure: HeavyWriteIT

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig resolved OAK-3753.

Resolution: Not A Problem

Fixed with OAK-4675

> Test failure: HeavyWriteIT
> --
>
> Key: OAK-3753
> URL: https://issues.apache.org/jira/browse/OAK-3753
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segmentmk
> Environment: Jenkins, Ubuntu: 
> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Critical
>  Labels: ci, jenkins
> Fix For: 1.6
>
> Attachments: build-1462283631.log.gz, build-1463134483.log.gz, 
> build-1467622598.log.gz, build-1467624994.log.gz, build-1468934799.log.gz, 
> build-1470736039.log.gz
>
>
> {{org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT}} failed on Jenkins:
> {noformat}
> heavyWrite[usePersistedMap: 
> false](org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT)  Time elapsed: 
> 106.519 sec  <<< ERROR!
> java.lang.IllegalStateException
>   at 
> com.google.common.base.Preconditions.checkState(Preconditions.java:134)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.Segment.(Segment.java:214)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.Segment.(Segment.java:198)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:1177)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.readSegment(SegmentTracker.java:224)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:149)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.RecordId.getSegment(RecordId.java:88)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.Segment.readTemplate(Segment.java:506)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getTemplate(SegmentNodeState.java:79)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getChildNode(SegmentNodeState.java:381)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$UnconnectedHead.update(MemoryNodeBuilder.java:651)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$ConnectedHead.update(MemoryNodeBuilder.java:729)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.head(MemoryNodeBuilder.java:171)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.access$300(MemoryNodeBuilder.java:88)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$UnconnectedHead.update(MemoryNodeBuilder.java:650)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$ConnectedHead.update(MemoryNodeBuilder.java:729)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.head(MemoryNodeBuilder.java:171)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.exists(MemoryNodeBuilder.java:273)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setProperty(MemoryNodeBuilder.java:506)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setProperty(MemoryNodeBuilder.java:515)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT.createProperties(HeavyWriteIT.java:156)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT.createNodes(HeavyWriteIT.java:148)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT.createNodes(HeavyWriteIT.java:149)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT.heavyWrite(HeavyWriteIT.java:129)
> {noformat}
> Seen at build 597



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4741) MemoryNodeStore constructor should only allow MemoryNodeState instances as base

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig resolved OAK-4741.

   Resolution: Not A Problem
Fix Version/s: (was: 1.6)

The fix from OAK-4847 fixes this issue

> MemoryNodeStore constructor should only allow MemoryNodeState instances as 
> base
> ---
>
> Key: OAK-4741
> URL: https://issues.apache.org/jira/browse/OAK-4741
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Minor
>
> {{MemoryNodeStore(NodeState)}} is overly generic and it turns out that 
> {{MemoryNodeStore}} cannot cope with other {{NodeState}} instances than 
> {{MemoryNodeState}} as its base: merging a {{NodeBuilder}} acquired through 
> such a store will throw a {{IllegalArgumentException}}.
> We need to adjust the signature accordingly.
> See http://markmail.org/message/mnhhwujowygc4m2p



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4823) Upgrade Oak Segment Tar dependency to 0.0.12

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-4823:
---
Fix Version/s: (was: 1.6)
   1.5.12

> Upgrade Oak Segment Tar dependency to 0.0.12
> 
>
> Key: OAK-4823
> URL: https://issues.apache.org/jira/browse/OAK-4823
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: parent
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Fix For: 1.5.12
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-3613) Backport TarMK revision gc related issues

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig resolved OAK-3613.

Resolution: Won't Fix

> Backport TarMK revision gc related issues
> -
>
> Key: OAK-3613
> URL: https://issues.apache.org/jira/browse/OAK-3613
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: compaction, gc
> Fix For: 1.0.35
>
> Attachments: 
> 0001-OAK-2870-Introduce-a-SegmentNodeStoreBuilder-to-help.patch, 
> 1.2-OAK-2870.patch, OAK-1995-1.0.patch, OAK-1995-1.2.zip, OsgiUtil-1.0.patch, 
> OsgiUtil-1.2.patch
>
>
> Some of the issues related to TarMK revision gc should be back ported to the 
> branches. This issue is for keeping track of which issues and which svn 
> revisions we consider for back porting. The task consists of the following 
> steps:
> # Identify issue to back port
> # Merge the respective commits into a private forks of the 1.0 and 1.2 
> branches
> # Run tests on builds from the private forks
> # On success merge the private forks to the 1.0 and 1.2 branches and update 
> the fix versions of the respective issues. 
> * Update the svn merge info with the respective merged svn revisions. 
> * Update the fix versions of the affected issues.
> [~dhasler]: FYI
> [~alex.parvulescu], [~frm]: please refrain from merging potential conflicting 
> changes into the branches in the meanwhile. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4828) oak-ugrade tests fail with segment tar 0.0.12 (on Windows)

[JIRA]

[ 
https://issues.apache.org/jira/browse/OAK-4828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549767#comment-15549767
 ] 

Michael Dürig commented on OAK-4828:


[~tomek.rekawek], any idea what could be causing this?

> oak-ugrade tests fail with segment tar 0.0.12 (on Windows)
> --
>
> Key: OAK-4828
> URL: https://issues.apache.org/jira/browse/OAK-4828
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Affects Versions: 1.5.10
>Reporter: Julian Reschke
> Fix For: Segment Tar 0.0.16
>
>
> When the dependency for segment tar is changed from 0.0.10 to 0.0.12 (see 
> OAK-4823), test cases fail due to files being open and thus not deleted on 
> Windows:
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.781 sec <<< 
> FAILURE!
> upgradeFrom10(org.apache.jackrabbit.oak.upgrade.UpgradeOldSegmentTest)  Time 
> elapsed: 0.781 sec  <<< ERROR!
> java.io.IOException: Unable to delete file: 
> target\UpgradeOldSegmentTest\test-repo-new\segmentstore\data0a.tar
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at 
> org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at 
> org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at 
> org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at 
> org.apache.jackrabbit.oak.upgrade.UpgradeOldSegmentTest.upgradeFrom10(UpgradeOldSegmentTest.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4873) Avoid running GC too frequently

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-4873:
---
Fix Version/s: (was: Segment Tar 0.0.14)
   Segment Tar 0.0.16

> Avoid running GC too frequently
> ---
>
> Key: OAK-4873
> URL: https://issues.apache.org/jira/browse/OAK-4873
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Affects Versions: Segment Tar 0.0.16
>Reporter: Michael Dürig
>  Labels: compaction, gc
> Fix For: Segment Tar 0.0.16
>
>
> As {{RevisionGCMBean.startRevisionGC()}} can be used to manually invoke a gc 
> cycle, there is the danger of running into a {{SNFE}} when gc is run multiple 
> times in quick succession (due to the retention time being based on number of 
> generations). We should come up with a mechanism to prevent this scenario. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4873) Avoid running GC too frequently

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-4873:
---
Fix Version/s: Segment Tar 0.0.14

> Avoid running GC too frequently
> ---
>
> Key: OAK-4873
> URL: https://issues.apache.org/jira/browse/OAK-4873
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Affects Versions: Segment Tar 0.0.16
>Reporter: Michael Dürig
>  Labels: compaction, gc
> Fix For: Segment Tar 0.0.14
>
>
> As {{RevisionGCMBean.startRevisionGC()}} can be used to manually invoke a gc 
> cycle, there is the danger of running into a {{SNFE}} when gc is run multiple 
> times in quick succession (due to the retention time being based on number of 
> generations). We should come up with a mechanism to prevent this scenario. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4828) oak-ugrade tests fail with segment tar 0.0.12 (on Windows)

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-4828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-4828:
---
Fix Version/s: (was: Segment Tar 0.0.14)
   Segment Tar 0.0.16

> oak-ugrade tests fail with segment tar 0.0.12 (on Windows)
> --
>
> Key: OAK-4828
> URL: https://issues.apache.org/jira/browse/OAK-4828
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Affects Versions: 1.5.10
>Reporter: Julian Reschke
> Fix For: Segment Tar 0.0.16
>
>
> When the dependency for segment tar is changed from 0.0.10 to 0.0.12 (see 
> OAK-4823), test cases fail due to files being open and thus not deleted on 
> Windows:
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.781 sec <<< 
> FAILURE!
> upgradeFrom10(org.apache.jackrabbit.oak.upgrade.UpgradeOldSegmentTest)  Time 
> elapsed: 0.781 sec  <<< ERROR!
> java.io.IOException: Unable to delete file: 
> target\UpgradeOldSegmentTest\test-repo-new\segmentstore\data0a.tar
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at 
> org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at 
> org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at 
> org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at 
> org.apache.jackrabbit.oak.upgrade.UpgradeOldSegmentTest.upgradeFrom10(UpgradeOldSegmentTest.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4828) oak-ugrade tests fail with segment tar 0.0.12 (on Windows)

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-4828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-4828:
---
Fix Version/s: Segment Tar 0.0.14

> oak-ugrade tests fail with segment tar 0.0.12 (on Windows)
> --
>
> Key: OAK-4828
> URL: https://issues.apache.org/jira/browse/OAK-4828
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Affects Versions: 1.5.10
>Reporter: Julian Reschke
> Fix For: Segment Tar 0.0.14
>
>
> When the dependency for segment tar is changed from 0.0.10 to 0.0.12 (see 
> OAK-4823), test cases fail due to files being open and thus not deleted on 
> Windows:
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.781 sec <<< 
> FAILURE!
> upgradeFrom10(org.apache.jackrabbit.oak.upgrade.UpgradeOldSegmentTest)  Time 
> elapsed: 0.781 sec  <<< ERROR!
> java.io.IOException: Unable to delete file: 
> target\UpgradeOldSegmentTest\test-repo-new\segmentstore\data0a.tar
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at 
> org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at 
> org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at 
> org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at 
> org.apache.jackrabbit.oak.upgrade.UpgradeOldSegmentTest.upgradeFrom10(UpgradeOldSegmentTest.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4897) The cold standby client should store segments in the right order

[Francesco Mari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Francesco Mari resolved OAK-4897.
-
Resolution: Fixed

Fixed at r1763480.

> The cold standby client should store segments in the right order
> 
>
> Key: OAK-4897
> URL: https://issues.apache.org/jira/browse/OAK-4897
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Reporter: Francesco Mari
>Assignee: Francesco Mari
>Priority: Blocker
> Fix For: Segment Tar 0.0.14
>
>
> The algorithm currently employed by the cold standby client doesn't store 
> segments in the order expected for the cleanup algorithm to work correctly. 
> In particular, if segment A references segment B, A is persisted after B. 
> Instead, if A references B, then B should be persisted before A in the 
> segment store.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4897) The cold standby client should store segments in the right order

[Francesco Mari (JIRA)]

Francesco Mari created OAK-4897:
---

 Summary: The cold standby client should store segments in the 
right order
 Key: OAK-4897
 URL: https://issues.apache.org/jira/browse/OAK-4897
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: segment-tar
Reporter: Francesco Mari
Assignee: Francesco Mari
Priority: Blocker
 Fix For: Segment Tar 0.0.14


The algorithm currently employed by the cold standby client doesn't store 
segments in the order expected for the cleanup algorithm to work correctly. In 
particular, if segment A references segment B, A is persisted after B. Instead, 
if A references B, then B should be persisted before A in the segment store.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4896) OSGi config for the size based estimation is ignored

[Alex Parvulescu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Parvulescu resolved OAK-4896.
--
Resolution: Fixed

fixed with http://svn.apache.org/viewvc?rev=1763473=rev

> OSGi config for the size based estimation is ignored
> 
>
> Key: OAK-4896
> URL: https://issues.apache.org/jira/browse/OAK-4896
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Affects Versions: Segment Tar 0.0.12
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
>Priority: Blocker
> Fix For: Segment Tar 0.0.14
>
>
> Followup of OAK-4729, setting the config value is not possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4896) OSGi config for the size based estimation is ignored

[Alex Parvulescu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Parvulescu updated OAK-4896:
-
Priority: Major  (was: Blocker)

> OSGi config for the size based estimation is ignored
> 
>
> Key: OAK-4896
> URL: https://issues.apache.org/jira/browse/OAK-4896
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Affects Versions: Segment Tar 0.0.12
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
> Fix For: Segment Tar 0.0.14
>
>
> Followup of OAK-4729, setting the config value is not possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4896) OSGi config for the size based estimation is ignored

[Alex Parvulescu (JIRA)]

Alex Parvulescu created OAK-4896:


 Summary: OSGi config for the size based estimation is ignored
 Key: OAK-4896
 URL: https://issues.apache.org/jira/browse/OAK-4896
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: segment-tar
Affects Versions: Segment Tar 0.0.12
Reporter: Alex Parvulescu
Assignee: Alex Parvulescu
Priority: Blocker
 Fix For: Segment Tar 0.0.14


Followup of OAK-4729, setting the config value is not possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4894) Potential NPE in Commit.apply()

[Marcel Reutegger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcel Reutegger resolved OAK-4894.
---
   Resolution: Fixed
Fix Version/s: 1.5.12

Fixed in trunk: http://svn.apache.org/r1763465

> Potential NPE in Commit.apply()
> ---
>
> Key: OAK-4894
> URL: https://issues.apache.org/jira/browse/OAK-4894
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, documentmk
>Affects Versions: 1.5.6, 1.4.8
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
>  Labels: candidate_oak_1_4
> Fix For: 1.6, 1.5.12
>
>
> May happen when a branch commit fails and {{Commit#b}} is not initialized. 
> The code should rather use getBranch(), which is responsible for lazy 
> initializing the field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4895) FileStore cleanup should not leak out file handles

[JIRA]

[ 
https://issues.apache.org/jira/browse/OAK-4895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549165#comment-15549165
 ] 

Michael Dürig commented on OAK-4895:


Patch at 
https://github.com/mduerig/jackrabbit-oak/commit/50a6b5f3be72a17121d755f6e04e612a51bea026.
 Note, this is on top of the changes I did for OAK-4138.

cc [~frm]

> FileStore cleanup should not leak out file handles
> --
>
> Key: OAK-4895
> URL: https://issues.apache.org/jira/browse/OAK-4895
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: technical_debt
> Fix For: Segment Tar 0.0.14
>
>
> {{FileStore.cleanup()}} currently returns a list of {{File}} instances 
> relying on the caller to remove those files. This breaks encapsulation as the 
> file store is the sole owner of these files and only the file store should be 
> removing them.
> I suggest to replace the current cleanup method with one that returns 
> {{void}}. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4894) Potential NPE in Commit.apply()

[Marcel Reutegger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcel Reutegger updated OAK-4894:
--
Labels: candidate_oak_1_4  (was: )

> Potential NPE in Commit.apply()
> ---
>
> Key: OAK-4894
> URL: https://issues.apache.org/jira/browse/OAK-4894
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, documentmk
>Affects Versions: 1.5.6, 1.4.8
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
>  Labels: candidate_oak_1_4
> Fix For: 1.6
>
>
> May happen when a branch commit fails and {{Commit#b}} is not initialized. 
> The code should rather use getBranch(), which is responsible for lazy 
> initializing the field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4894) Potential NPE in Commit.apply()

[Marcel Reutegger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcel Reutegger updated OAK-4894:
--
Affects Version/s: 1.5.6
   1.4.8

> Potential NPE in Commit.apply()
> ---
>
> Key: OAK-4894
> URL: https://issues.apache.org/jira/browse/OAK-4894
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, documentmk
>Affects Versions: 1.5.6, 1.4.8
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.6
>
>
> May happen when a branch commit fails and {{Commit#b}} is not initialized. 
> The code should rather use getBranch(), which is responsible for lazy 
> initializing the field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4894) Potential NPE in Commit.apply()

[Marcel Reutegger (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549122#comment-15549122
 ] 

Marcel Reutegger commented on OAK-4894:
---

The issue was introduced with OAK-4556.

> Potential NPE in Commit.apply()
> ---
>
> Key: OAK-4894
> URL: https://issues.apache.org/jira/browse/OAK-4894
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.6
>
>
> May happen when a branch commit fails and {{Commit#b}} is not initialized. 
> The code should rather use getBranch(), which is responsible for lazy 
> initializing the field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4895) FileStore cleanup should not leak out file handles

[JIRA]

Michael Dürig created OAK-4895:
--

 Summary: FileStore cleanup should not leak out file handles
 Key: OAK-4895
 URL: https://issues.apache.org/jira/browse/OAK-4895
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: segment-tar
Reporter: Michael Dürig
Assignee: Michael Dürig
 Fix For: Segment Tar 0.0.14


{{FileStore.cleanup()}} currently returns a list of {{File}} instances relying 
on the caller to remove those files. This breaks encapsulation as the file 
store is the sole owner of these files and only the file store should be 
removing them.

I suggest to replace the current cleanup method with one that returns {{void}}. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4581) Persistent local journal for more reliable event generation

[Stefan Egli (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549104#comment-15549104
 ] 

Stefan Egli commented on OAK-4581:
--

Summarizing those numerous, recent threaded comments I believe we have 
consensus that the goal is to get rid of external use of BackgroundObserver 
(not scope of this ticket, but a consequence) and that we do the *persistence 
on the oak-jcr/ChangeProcessor level*. This makes the persistence independent 
from cache and GC problems, works fine with filter changes, but has the 
downside that it means larger temporary storage required (as events are 
'exploded'). The actual file format of the stored events is somewhat of an 
orthogonal/detail question, but can be something like a flat file.
Unless I hear objections I'm looking at following up on these assumption in the 
next days.
/cc [~chetanm], [~mduerig], [~mreutegg], [~reschke], [~catholicon]

> Persistent local journal for more reliable event generation
> ---
>
> Key: OAK-4581
> URL: https://issues.apache.org/jira/browse/OAK-4581
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: core
>Reporter: Chetan Mehrotra
>Assignee: Stefan Egli
>  Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4581.v0.patch
>
>
> As discussed in OAK-2683 "hitting the observation queue limit" has multiple 
> drawbacks. Quite a bit of work is done to make diff generation faster. 
> However there are still chances of event queue getting filled up. 
> This issue is meant to implement a persistent event journal. Idea here being
> # NodeStore would push the diff into a persistent store via a synchronous 
> observer
> # Observors which are meant to handle such events in async way (by virtue of 
> being wrapped in BackgroundObserver) would instead pull the events from this 
> persisted journal
> h3. A - What is persisted
> h4. 1 - Serialized Root States and CommitInfo
> In this approach we just persist the root states in serialized form. 
> * DocumentNodeStore - This means storing the root revision vector
> * SegmentNodeStore - {color:red}Q1 - What does serialized form of 
> SegmentNodeStore root state looks like{color} - Possible the RecordId of 
> "root" state
> Note that with OAK-4528 DocumentNodeStore can rely on persisted remote 
> journal to determine the affected paths. Which reduces the need for 
> persisting complete diff locally.
> Event generation logic would then "deserialize" the persisted root states and 
> then generate the diff as currently done via NodeState comparison
> h4. 2 - Serialized commit diff and CommitInfo
> In this approach we can save the diff in JSOP form. The diff only contains 
> information about affected path. Similar to what is current being stored in 
> DocumentNodeStore journal
> h4. CommitInfo
> The commit info would also need to be serialized. So it needs to be ensure 
> whatever is stored there can be serialized or re calculated
> h3. B - How it is persisted
> h4. 1 - Use a secondary segment NodeStore
> OAK-4180 makes use of SegmentNodeStore as a secondary store for caching. 
> [~mreutegg] suggested that for persisted local journal we can also utilize a 
> SegmentNodeStore instance. Care needs to be taken for compaction. Either via 
> generation approach or relying on online compaction
> h4. 2- Make use of write ahead log implementations
> [~ianeboston] suggested that we can make use of some write ahead log 
> implementation like [1], [2] or [3]
> h3. C - How changes get pulled
> Some points to consider for event generation logic
> # Would need a way to keep pointers to journal entry on per listener basis. 
> This would allow each Listener to "pull" content changes and generate diff as 
> per its speed and keeping in memory overhead low
> # The journal should survive restarts
> [1] http://www.mapdb.org/javadoc/latest/mapdb/org/mapdb/WriteAheadLog.html
> [2] 
> https://github.com/apache/activemq/tree/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/journal
> [3] 
> https://github.com/elastic/elasticsearch/tree/master/core/src/main/java/org/elasticsearch/index/translog



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4894) Potential NPE in Commit.apply()

[Marcel Reutegger (JIRA)]

Marcel Reutegger created OAK-4894:
-

 Summary: Potential NPE in Commit.apply()
 Key: OAK-4894
 URL: https://issues.apache.org/jira/browse/OAK-4894
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: core, documentmk
Reporter: Marcel Reutegger
Assignee: Marcel Reutegger
Priority: Minor
 Fix For: 1.6


May happen when a branch commit fails and {{Commit#b}} is not initialized. The 
code should rather use getBranch(), which is responsible for lazy initializing 
the field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4522) Improve CommitRateLimiter to optionally block some commits

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548855#comment-15548855
 ] 

Thomas Mueller commented on OAK-4522:
-

> this patch only works with documentMk
> revisit the nature of the CommitRateLimiter being a CommitHook
> OAK-4122

Yes. I think that's a limitation we can live with at the moment. If we need to 
support Tar storage, more changes would be needed, which would be harder to 
backport (if backporting is needed). And I'm not aware of problems when using 
the Tar storage.

> high-water-mark/low-water-mark as definition as to when you start/end blocking

I think it would make the code more complicated, while not providing an 
additional feature. The current patch is to just ensure the CommitRateLimiter 
doesn't shoot itself in the foot. It's not designed to be fully optimized for 
performance.

> Improve CommitRateLimiter to optionally block some commits
> --
>
> Key: OAK-4522
> URL: https://issues.apache.org/jira/browse/OAK-4522
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: jcr
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
>  Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4522-b.patch, OAK-4522-c.patch, OAK-4522.patch
>
>
> The CommitRateLimiter of OAK-1659 can delay commits, but doesn't currently 
> block them, and delays even those commits that are part of handling events. 
> Because of that, the queue can still get full, and possibly delaying commits 
> while handling events can make the situation even worse.
> In Jackrabbit 2.x, we had a similar feature: JCR-2402. Also related is 
> JCR-2746.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-4796) filter events before adding to ChangeProcessor's queue

[Stefan Egli (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548765#comment-15548765
 ] 

Stefan Egli edited comment on OAK-4796 at 10/5/16 2:14 PM:
---

bq. This is wrong. We need to either do prefiltering or postfiltering and can't 
mix the two. Therefore for prefiltering it's essential to pass around the 
applied filter in the ContentChange obj and use that later at delivery time.
Coming back to this point, there seems to be some issues with this based on the 
current design: Prior to prefiltering we had only postfiltering. And changing 
the FilterProvider was applied immediately - basically on all elements in the 
queue. With prefiltering this is, as pointed out, not correct: those elements 
in the queue already have gone through prefiltering, so postfiltering should be 
done with the same FilterProvider. Which means, the ChangeProcessor - which is 
in charge of postfiltering - should not use the FilterProvider set on its 
instance, but use the same that was used for prefiltering. Therefore the 
ChangeProcessor needs to be given the FilterProvider for each change that it 
processes. The way it receives changes though is via the 
Observer.contentChanged. Therefore about the only feasible place to pass the 
FilterProvider from BackgroundObserver to ChangeProcessor is via the CommitInfo.

Thing now is that for external and overflow entries the CommitInfo is null. So 
I'd say, as long as that's the case it's very hard to implement correctly 
switching the filter.

Unless this switch is done correctly, the only thing that can be said is that: 
when a filter is changed and the queue is not empty, then both filters are 
applied. However the listener doesn't know anything about the queue internas, 
so cannot make any conclusions based on that.


was (Author: egli):
bq. This is wrong. We need to either do prefiltering or postfiltering and can't 
mix the two. Therefore for prefiltering it's essential to pass around the 
applied filter in the ContentChange obj and use that later at delivery time.
Coming back to this point, there seems to be some issues with this based on the 
current design: Prior to prefiltering we had only postfiltering. And changing 
the FilterProvider was applied immediately - basically on all elements in the 
queue. With prefiltering this is, as pointed out, not correct: those elements 
in the queue already have gone through prefiltering, so postfiltering should be 
done with the same FilterProvider. Which means, the ChangeProcessor - which is 
in charge of postfiltering - should not use the FilterProvider set on its 
instance, but use the same that was used for prefiltering. Therefore the 
ChangeProcessor needs to be given the FilterProvider for each change that it 
processes. The way it receives changes though is via the 
Observer.contentChanged. Therefore about the only feasible place to pass the 
FilterProvider from BackgroundObserver to ChangeProcessor is via the CommitInfo.

Thing now is that for external and overflow entries the CommitInfo is null. So 
I'd say, as long as that's the case it's very hard to implement correctly 
switching the filter.

Unless this switch is done correctly, the only thing that can be said is that: 
when a filter is changed it is undefined for which changes both filters are 
applied (if the queue is not empty when switching).

> filter events before adding to ChangeProcessor's queue
> --
>
> Key: OAK-4796
> URL: https://issues.apache.org/jira/browse/OAK-4796
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: jcr
>Affects Versions: 1.5.9
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>  Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4796.changeSet.patch, OAK-4796.patch
>
>
> Currently the 
> [ChangeProcessor.contentChanged|https://github.com/apache/jackrabbit-oak/blob/f4f4e01dd8f708801883260481d37fdcd5868deb/oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ChangeProcessor.java#L335]
>  is in charge of doing the event diffing and filtering and does so in a 
> pooled Thread, ie asynchronously, at a later stage independent from the 
> commit. This has the advantage that the commit is fast, but has the following 
> potentially negative effects:
> # events (in the form of ContentChange Objects) occupy a slot of the queue 
> even if the listener is not interested in it - any commit lands on any 
> listener's queue. This reduces the capacity of the queue for 'actual' events 
> to be delivered. It therefore increases the risk that the queue fills - and 
> when full has various consequences such as loosing the CommitInfo etc.
> # each event==ContentChange later on must be evaluated, and for that a diff 
> must be calculated. Depending on 

[jira] [Commented] (OAK-4816) Property index: cost estimate with path restriction is too optimistic

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548822#comment-15548822
 ] 

Thomas Mueller commented on OAK-4816:
-

http://svn.apache.org/r1763455 (trunk)

> Property index: cost estimate with path restriction is too optimistic
> -
>
> Key: OAK-4816
> URL: https://issues.apache.org/jira/browse/OAK-4816
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Fix For: 1.5.12
>
>
> The property index cost estimation is too optimistic in case there is a 
> property restriction plus a path restriction. The current algorithm, as 
> documented in 
> http://jackrabbit.apache.org/oak/docs/query/property-index.html#Cost_Estimation
>  , assumes that matching entries are evenly distributed over the whole 
> repository. In many cases, this is not the case. In extreme cases, _all_ 
> entries that match the property restriction are in the subtree that matches 
> the path restriction. Example: 
> * 10'000 nodes with property color "red".
> * 1 million nodes in the repository
> * 10'000 nodes in the subtree /content
> * query {{/jcr:root/content//\*[@color = 'red']}}
> Currently, the cost estimate is about 100, there are about 10'000 entries for 
> "red", and "/content" contains 1% of all nodes. But in reality, there might 
> be 10'000 entries with color "red" in that subtree (that is, all of them).
> The cost estimation should take that into account, and assume that at least 
> 80% of the matching nodes are in that subtree (if the subtree contains that 
> many nodes).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4816) Property index: cost estimate with path restriction is too optimistic

[Thomas Mueller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller resolved OAK-4816.
-
Resolution: Fixed

> Property index: cost estimate with path restriction is too optimistic
> -
>
> Key: OAK-4816
> URL: https://issues.apache.org/jira/browse/OAK-4816
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Fix For: 1.5.12
>
>
> The property index cost estimation is too optimistic in case there is a 
> property restriction plus a path restriction. The current algorithm, as 
> documented in 
> http://jackrabbit.apache.org/oak/docs/query/property-index.html#Cost_Estimation
>  , assumes that matching entries are evenly distributed over the whole 
> repository. In many cases, this is not the case. In extreme cases, _all_ 
> entries that match the property restriction are in the subtree that matches 
> the path restriction. Example: 
> * 10'000 nodes with property color "red".
> * 1 million nodes in the repository
> * 10'000 nodes in the subtree /content
> * query {{/jcr:root/content//\*[@color = 'red']}}
> Currently, the cost estimate is about 100, there are about 10'000 entries for 
> "red", and "/content" contains 1% of all nodes. But in reality, there might 
> be 10'000 entries with color "red" in that subtree (that is, all of them).
> The cost estimation should take that into account, and assume that at least 
> 80% of the matching nodes are in that subtree (if the subtree contains that 
> many nodes).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4816) Property index: cost estimate with path restriction is too optimistic

[Thomas Mueller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller updated OAK-4816:

Fix Version/s: (was: 1.6)
   1.5.12

> Property index: cost estimate with path restriction is too optimistic
> -
>
> Key: OAK-4816
> URL: https://issues.apache.org/jira/browse/OAK-4816
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Fix For: 1.5.12
>
>
> The property index cost estimation is too optimistic in case there is a 
> property restriction plus a path restriction. The current algorithm, as 
> documented in 
> http://jackrabbit.apache.org/oak/docs/query/property-index.html#Cost_Estimation
>  , assumes that matching entries are evenly distributed over the whole 
> repository. In many cases, this is not the case. In extreme cases, _all_ 
> entries that match the property restriction are in the subtree that matches 
> the path restriction. Example: 
> * 10'000 nodes with property color "red".
> * 1 million nodes in the repository
> * 10'000 nodes in the subtree /content
> * query {{/jcr:root/content//\*[@color = 'red']}}
> Currently, the cost estimate is about 100, there are about 10'000 entries for 
> "red", and "/content" contains 1% of all nodes. But in reality, there might 
> be 10'000 entries with color "red" in that subtree (that is, all of them).
> The cost estimation should take that into account, and assume that at least 
> 80% of the matching nodes are in that subtree (if the subtree contains that 
> many nodes).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-4796) filter events before adding to ChangeProcessor's queue

[Stefan Egli (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548765#comment-15548765
 ] 

Stefan Egli edited comment on OAK-4796 at 10/5/16 2:11 PM:
---

bq. This is wrong. We need to either do prefiltering or postfiltering and can't 
mix the two. Therefore for prefiltering it's essential to pass around the 
applied filter in the ContentChange obj and use that later at delivery time.
Coming back to this point, there seems to be some issues with this based on the 
current design: Prior to prefiltering we had only postfiltering. And changing 
the FilterProvider was applied immediately - basically on all elements in the 
queue. With prefiltering this is, as pointed out, not correct: those elements 
in the queue already have gone through prefiltering, so postfiltering should be 
done with the same FilterProvider. Which means, the ChangeProcessor - which is 
in charge of postfiltering - should not use the FilterProvider set on its 
instance, but use the same that was used for prefiltering. Therefore the 
ChangeProcessor needs to be given the FilterProvider for each change that it 
processes. The way it receives changes though is via the 
Observer.contentChanged. Therefore about the only feasible place to pass the 
FilterProvider from BackgroundObserver to ChangeProcessor is via the CommitInfo.

Thing now is that for external and overflow entries the CommitInfo is null. So 
I'd say, as long as that's the case it's very hard to implement correctly 
switching the filter.

Unless this switch is done correctly, the only thing that can be said is that: 
when a filter is changed it is undefined for which changes both filters are 
applied (if the queue is not empty when switching).


was (Author: egli):
bq. This is wrong. We need to either do prefiltering or postfiltering and can't 
mix the two. Therefore for prefiltering it's essential to pass around the 
applied filter in the ContentChange obj and use that later at delivery time.
Coming back to this point, there seems to be some issues with this based on the 
current design: Prior to prefiltering we had only postfiltering. And changing 
the FilterProvider was applied immediately - basically on all elements in the 
queue. With prefiltering this is, as pointed out, not correct: those elements 
in the queue already have gone through prefiltering, so postfiltering should be 
done with the same FilterProvider. Which means, the ChangeProcessor - which is 
in charge of postfiltering - should not use the FilterProvider set on its 
instance, but use the same that was used for prefiltering. Therefore the 
ChangeProcessor needs to be given the FilterProvider for each change that it 
processes. The way it receives changes though is via the 
Observer.contentChanged. Therefore about the only feasible place to pass the 
FilterProvider from BackgroundObserver to ChangeProcessor is via the CommitInfo.

Thing now is that for external and overflow entries the CommitInfo is null. So 
I'd say, as long as that's the case it's very hard to implement correctly 
switching the filter.

Unless this switch is done correctly, the only thing that can be said is that: 
when a filter is changed it is undefined if the old, the new or both filters 
are applied to entries in the queue.

> filter events before adding to ChangeProcessor's queue
> --
>
> Key: OAK-4796
> URL: https://issues.apache.org/jira/browse/OAK-4796
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: jcr
>Affects Versions: 1.5.9
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>  Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4796.changeSet.patch, OAK-4796.patch
>
>
> Currently the 
> [ChangeProcessor.contentChanged|https://github.com/apache/jackrabbit-oak/blob/f4f4e01dd8f708801883260481d37fdcd5868deb/oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ChangeProcessor.java#L335]
>  is in charge of doing the event diffing and filtering and does so in a 
> pooled Thread, ie asynchronously, at a later stage independent from the 
> commit. This has the advantage that the commit is fast, but has the following 
> potentially negative effects:
> # events (in the form of ContentChange Objects) occupy a slot of the queue 
> even if the listener is not interested in it - any commit lands on any 
> listener's queue. This reduces the capacity of the queue for 'actual' events 
> to be delivered. It therefore increases the risk that the queue fills - and 
> when full has various consequences such as loosing the CommitInfo etc.
> # each event==ContentChange later on must be evaluated, and for that a diff 
> must be calculated. Depending on runtime behavior that diff might be 
> expensive if no longer in the cache (documentMk 

[jira] [Commented] (OAK-4796) filter events before adding to ChangeProcessor's queue

[Stefan Egli (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548765#comment-15548765
 ] 

Stefan Egli commented on OAK-4796:
--

bq. This is wrong. We need to either do prefiltering or postfiltering and can't 
mix the two. Therefore for prefiltering it's essential to pass around the 
applied filter in the ContentChange obj and use that later at delivery time.
Coming back to this point, there seems to be some issues with this based on the 
current design: Prior to prefiltering we had only postfiltering. And changing 
the FilterProvider was applied immediately - basically on all elements in the 
queue. With prefiltering this is, as pointed out, not correct: those elements 
in the queue already have gone through prefiltering, so postfiltering should be 
done with the same FilterProvider. Which means, the ChangeProcessor - which is 
in charge of postfiltering - should not use the FilterProvider set on its 
instance, but use the same that was used for prefiltering. Therefore the 
ChangeProcessor needs to be given the FilterProvider for each change that it 
processes. The way it receives changes though is via the 
Observer.contentChanged. Therefore about the only feasible place to pass the 
FilterProvider from BackgroundObserver to ChangeProcessor is via the CommitInfo.

Thing now is that for external and overflow entries the CommitInfo is null. So 
I'd say, as long as that's the case it's very hard to implement correctly 
switching the filter.

Unless this switch is done correctly, the only thing that can be said is that: 
when a filter is changed it is undefined if the old, the new or both filters 
are applied to entries in the queue.

> filter events before adding to ChangeProcessor's queue
> --
>
> Key: OAK-4796
> URL: https://issues.apache.org/jira/browse/OAK-4796
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: jcr
>Affects Versions: 1.5.9
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>  Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4796.changeSet.patch, OAK-4796.patch
>
>
> Currently the 
> [ChangeProcessor.contentChanged|https://github.com/apache/jackrabbit-oak/blob/f4f4e01dd8f708801883260481d37fdcd5868deb/oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ChangeProcessor.java#L335]
>  is in charge of doing the event diffing and filtering and does so in a 
> pooled Thread, ie asynchronously, at a later stage independent from the 
> commit. This has the advantage that the commit is fast, but has the following 
> potentially negative effects:
> # events (in the form of ContentChange Objects) occupy a slot of the queue 
> even if the listener is not interested in it - any commit lands on any 
> listener's queue. This reduces the capacity of the queue for 'actual' events 
> to be delivered. It therefore increases the risk that the queue fills - and 
> when full has various consequences such as loosing the CommitInfo etc.
> # each event==ContentChange later on must be evaluated, and for that a diff 
> must be calculated. Depending on runtime behavior that diff might be 
> expensive if no longer in the cache (documentMk specifically).
> As an improvement, this diffing+filtering could be done at an earlier stage 
> already, nearer to the commit, and in case the filter would ignore the event, 
> it would not have to be put into the queue at all, thus avoiding occupying a 
> slot and later potentially slower diffing.
> The suggestion is to implement this via the following algorithm:
> * During the commit, in a {{Validator}} the listener's filters are evaluated 
> - in an as-efficient-as-possible manner (Reason for doing it in a Validator 
> is that this doesn't add overhead as oak already goes through all changes for 
> other Validators). As a result a _list of potentially affected observers_ is 
> added to the {{CommitInfo}} (false positives are fine).
> ** Note that the above adds cost to the commit and must therefore be 
> carefully done and measured
> ** One potential measure could be to only do filtering when listener's queues 
> are larger than a certain threshold (eg 10)
> * The ChangeProcessor in {{contentChanged}} (in the one created in 
> [createObserver|https://github.com/apache/jackrabbit-oak/blob/f4f4e01dd8f708801883260481d37fdcd5868deb/oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ChangeProcessor.java#L224])
>  then checks the new commitInfo's _potentially affected observers_ list and 
> if it's not in the list, adds a {{NOOP}} token at the end of the queue. If 
> there's already a NOOP there, the two are collapsed (this way when a filter 
> is not affected it would have a NOOP at the end of the queue). If later on a 
> no-NOOP item is added, the NOOP's {{root}} is used as 

[jira] [Updated] (OAK-4796) filter events before adding to ChangeProcessor's queue

[Stefan Egli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Egli updated OAK-4796:
-
Attachment: OAK-4796.changeSet.patch

Attaching a second variant of the patch ([^OAK-4796.changeSet.patch]) which is 
based on Chetan's suggestion to compose a set of changes (parent-paths, 
propertyNames, nodeTypes, nodeNames) in an Editor (actually, I've used a 
ValidatorProvider/Validator pair), stores it as a property in the CommitContext 
of the CommitInfo, and evaluates it in oak-jcr's ChangeProcessor. This patch 
also includes some minimal statistics in the consolidated listener stats that 
shows how many commits were either skipped (because the feature was disabled or 
the CommitInfo null etc), included or excluded. The ObservationTest passes with 
the prefiltering enabled, however I plan to add some more specific testing 
still. 
Would welcome a review of this second approach (compared to the first which was 
EventFilter-based). /cc [~mreutegg], [~chetanm], [~mduerig], [~catholicon]

> filter events before adding to ChangeProcessor's queue
> --
>
> Key: OAK-4796
> URL: https://issues.apache.org/jira/browse/OAK-4796
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: jcr
>Affects Versions: 1.5.9
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>  Labels: observation
> Fix For: 1.6
>
> Attachments: OAK-4796.changeSet.patch, OAK-4796.patch
>
>
> Currently the 
> [ChangeProcessor.contentChanged|https://github.com/apache/jackrabbit-oak/blob/f4f4e01dd8f708801883260481d37fdcd5868deb/oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ChangeProcessor.java#L335]
>  is in charge of doing the event diffing and filtering and does so in a 
> pooled Thread, ie asynchronously, at a later stage independent from the 
> commit. This has the advantage that the commit is fast, but has the following 
> potentially negative effects:
> # events (in the form of ContentChange Objects) occupy a slot of the queue 
> even if the listener is not interested in it - any commit lands on any 
> listener's queue. This reduces the capacity of the queue for 'actual' events 
> to be delivered. It therefore increases the risk that the queue fills - and 
> when full has various consequences such as loosing the CommitInfo etc.
> # each event==ContentChange later on must be evaluated, and for that a diff 
> must be calculated. Depending on runtime behavior that diff might be 
> expensive if no longer in the cache (documentMk specifically).
> As an improvement, this diffing+filtering could be done at an earlier stage 
> already, nearer to the commit, and in case the filter would ignore the event, 
> it would not have to be put into the queue at all, thus avoiding occupying a 
> slot and later potentially slower diffing.
> The suggestion is to implement this via the following algorithm:
> * During the commit, in a {{Validator}} the listener's filters are evaluated 
> - in an as-efficient-as-possible manner (Reason for doing it in a Validator 
> is that this doesn't add overhead as oak already goes through all changes for 
> other Validators). As a result a _list of potentially affected observers_ is 
> added to the {{CommitInfo}} (false positives are fine).
> ** Note that the above adds cost to the commit and must therefore be 
> carefully done and measured
> ** One potential measure could be to only do filtering when listener's queues 
> are larger than a certain threshold (eg 10)
> * The ChangeProcessor in {{contentChanged}} (in the one created in 
> [createObserver|https://github.com/apache/jackrabbit-oak/blob/f4f4e01dd8f708801883260481d37fdcd5868deb/oak-jcr/src/main/java/org/apache/jackrabbit/oak/jcr/observation/ChangeProcessor.java#L224])
>  then checks the new commitInfo's _potentially affected observers_ list and 
> if it's not in the list, adds a {{NOOP}} token at the end of the queue. If 
> there's already a NOOP there, the two are collapsed (this way when a filter 
> is not affected it would have a NOOP at the end of the queue). If later on a 
> no-NOOP item is added, the NOOP's {{root}} is used as the {{previousRoot}} 
> for the newly added {{ContentChange}} obj.
> ** To achieve that, the ContentChange obj is extended to not only have the 
> "to" {{root}} pointer, but also the "from" {{previousRoot}} pointer which 
> currently is implicitly maintained.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4885:

Labels:   (was: candidate_oak_1_0)

> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
> Fix For: 1.2.20, 1.5.12, 1.0.35, 1.4.9
>
>
> See  
> (and ).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548257#comment-15548257
 ] 

Julian Reschke edited comment on OAK-4885 at 10/5/16 12:54 PM:
---

trunk: [r1763378|http://svn.apache.org/r1763378]
1.4: [r1763402|http://svn.apache.org/r1763402]
1.2: [r1763440|http://svn.apache.org/r1763440]
1.0: [r1763445|http://svn.apache.org/r1763445]


was (Author: reschke):
trunk: [r1763378|http://svn.apache.org/r1763378]
1.4: [r1763402|http://svn.apache.org/r1763402]
1.2: [r1763440|http://svn.apache.org/r1763440]


> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
> Fix For: 1.2.20, 1.5.12, 1.0.35, 1.4.9
>
>
> See  
> (and ).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4885:

Fix Version/s: 1.0.35

> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
> Fix For: 1.2.20, 1.5.12, 1.0.35, 1.4.9
>
>
> See  
> (and ).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548257#comment-15548257
 ] 

Julian Reschke edited comment on OAK-4885 at 10/5/16 12:05 PM:
---

trunk: [r1763378|http://svn.apache.org/r1763378]
1.4: [r1763402|http://svn.apache.org/r1763402]
1.2: [r1763440|http://svn.apache.org/r1763440]



was (Author: reschke):
trunk: [r1763378|http://svn.apache.org/r1763378]
1.4: [r1763402|http://svn.apache.org/r1763402]


> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
>  Labels: candidate_oak_1_0
> Fix For: 1.2.20, 1.5.12, 1.4.9
>
>
> See  
> (and ).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4885:

Fix Version/s: 1.2.20

> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
>  Labels: candidate_oak_1_0
> Fix For: 1.2.20, 1.5.12, 1.4.9
>
>
> See  
> (and ).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4885:

Labels: candidate_oak_1_0  (was: candidate_oak_1_0 candidate_oak_1_2)

> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
>  Labels: candidate_oak_1_0
> Fix For: 1.2.20, 1.5.12, 1.4.9
>
>
> See  
> (and ).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-2808) Active deletion of 'deleted' Lucene index files from DataStore without relying on full scale Blob GC

[Chetan Mehrotra (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548418#comment-15548418
 ] 

Chetan Mehrotra commented on OAK-2808:
--

[~tmueller] If we are not going to complete this feature soon would it possible 
to revert the changes done for this feature so far. This would simplify 
understanding the flow for some new features like OAK-3547. We can introduce 
them back once we complete feature.

> Active deletion of 'deleted' Lucene index files from DataStore without 
> relying on full scale Blob GC
> 
>
> Key: OAK-2808
> URL: https://issues.apache.org/jira/browse/OAK-2808
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Thomas Mueller
>  Labels: datastore, performance
> Fix For: 1.6
>
> Attachments: OAK-2808-1.patch, copyonread-stats.png
>
>
> With storing of Lucene index files within DataStore our usage pattern
> of DataStore has changed between JR2 and Oak.
> With JR2 the writes were mostly application based i.e. if application
> stores a pdf/image file then that would be stored in DataStore. JR2 by
> default would not write stuff to DataStore. Further in deployment
> where large number of binary content is present then systems tend to
> share the DataStore to avoid duplication of storage. In such cases
> running Blob GC is a non trivial task as it involves a manual step and
> coordination across multiple deployments. Due to this systems tend to
> delay frequency of GC
> Now with Oak apart from application the Oak system itself *actively*
> uses the DataStore to store the index files for Lucene and there the
> churn might be much higher i.e. frequency of creation and deletion of
> index file is lot higher. This would accelerate the rate of garbage
> generation and thus put lot more pressure on the DataStore storage
> requirements.
> Discussion thread http://markmail.org/thread/iybd3eq2bh372zrl



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4885:

Fix Version/s: 1.4.9

> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
>  Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.5.12, 1.4.9
>
>
> See  
> (and ).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548257#comment-15548257
 ] 

Julian Reschke edited comment on OAK-4885 at 10/5/16 10:46 AM:
---

trunk: [r1763378|http://svn.apache.org/r1763378]
1.4: [r1763402|http://svn.apache.org/r1763402]



was (Author: reschke):
trunk: [r1763378|http://svn.apache.org/r1763378]

> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
>  Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.5.12, 1.4.9
>
>
> See  
> (and ).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4885:

Labels: candidate_oak_1_0 candidate_oak_1_2  (was: candidate_oak_1_0 
candidate_oak_1_2 candidate_oak_1_4)

> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
>  Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.5.12, 1.4.9
>
>
> See  
> (and ).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4883) Missing BlobGarbageCollection MBean on standby instance

[Timothee Maret (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548300#comment-15548300
 ] 

Timothee Maret commented on OAK-4883:
-

Thanks [~alexparvulescu] for clarifying.

With the current code (segment-tar) a standby instance will not instantiate a 
{{SegmentNodeStore}} due to the check at [0]. AFAIU, this was introduced in 
OAK-4089 (before the code that issue, the code was registering a 
{{SegmentNodeStore}}).
A {{SegmentNodeStore}} seems to be required in order to register a 
{{BlobGCMBean}}. 
Indeed  the {{BlobGCMBean}} needs a {{MarkSweepGarbageCollector}} which needs a 
{{repoId}} which is obtained via a {{SegmentNodeStore}}.

AFAIU, we'd need to register a {{SegmentNodeStore}} while keeping the fix for 
OAK-4089.
[~frm] in OAK-4089 you wrote:

bq. It would be more appropriate not to instantiate the observation subsystem 
at all in the context of the standby store, thus preventing spurious references 
to SegmentId instances to float in memory.

With "observation subsystem", were you referring to the {{observerTracker}} in 
[1] ? 
If so, how about building a {{SegmentNodeStore}} in any case, but only avoid 
registering {{observerTracker}} for standby instances ?

[0] 
https://github.com/apache/jackrabbit-oak/blob/2b6c2f5340f3b6485dda5c493f6343d232c883e9/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentNodeStoreService.java#L374
[1] 
https://github.com/apache/jackrabbit-oak/blob/2b6c2f5340f3b6485dda5c493f6343d232c883e9/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentNodeStoreService.java#L595

> Missing BlobGarbageCollection MBean on standby instance
> ---
>
> Key: OAK-4883
> URL: https://issues.apache.org/jira/browse/OAK-4883
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar, segmentmk
>Reporter: Alex Parvulescu
>Assignee: Timothee Maret
>  Labels: gc
>
> The {{BlobGarbageCollection}} MBean is no longer available on a standby 
> instance, this affects non-shared datastore setups (on a shared datastore 
> you'd only need to run blob gc on the primary).
> This change was introduced by OAK-4089 (and backported to 1.4 branch with 
> OAK-4093).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548257#comment-15548257
 ] 

Julian Reschke commented on OAK-4885:
-

trunk: [r1763378|http://svn.apache.org/r1763378]

> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
>  Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.5.12
>
>
> See  
> (and ).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4893) Document conflict handling

[JIRA]

Michael Dürig created OAK-4893:
--

 Summary: Document conflict handling
 Key: OAK-4893
 URL: https://issues.apache.org/jira/browse/OAK-4893
 Project: Jackrabbit Oak
  Issue Type: Task
  Components: doc
Reporter: Michael Dürig
 Fix For: 1.6


We should add documentation how Oak deals with conflicts. This was once 
documented in the Javadocs of {{MicroKernel.rebase()}} but got lost along with 
that class. Note that OAK-1553 refines conflict handling but this refinement 
has not been implemented in all backends yet. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke resolved OAK-4885.
-
   Resolution: Fixed
Fix Version/s: 1.5.12

> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
>  Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.5.12
>
>
> See  
> (and ).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4844) Analyse effects of simplified record ids

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig resolved OAK-4844.

Resolution: Fixed

Committed the patch at http://svn.apache.org/viewvc?rev=1763371=rev

> Analyse effects of simplified record ids
> 
>
> Key: OAK-4844
> URL: https://issues.apache.org/jira/browse/OAK-4844
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: performance
> Fix For: Segment Tar 0.0.14
>
> Attachments: OAK-4844-02.patch, OAK-4844-3.patch, OAK-4844.patch
>
>
> OAK-4631 introduced a simplified serialisation for record ids. This causes 
> their footprint on disk to increase from 3 bytes to 18 bytes. OAK-4631 has 
> some initial analysis on the effect this is having on repositories as a 
> whole. 
> I'm opening this issue as a dedicated task to further look into mitigation 
> strategies (if necessary). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4884) Test failure: org.apache.jackrabbit.oak.segment.standby.ExternalSharedStoreIT.testProxyFlippedIntermediateByteChange2

[Francesco Mari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548166#comment-15548166
 ] 

Francesco Mari commented on OAK-4884:
-

I made a minor fix at r1763368. I adjusted the waiting timeouts in a way that 
they do not conflict with the graceful shutdowns made by the client and the 
server channels.

> Test failure: 
> org.apache.jackrabbit.oak.segment.standby.ExternalSharedStoreIT.testProxyFlippedIntermediateByteChange2
> -
>
> Key: OAK-4884
> URL: https://issues.apache.org/jira/browse/OAK-4884
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar
>Reporter: Francesco Mari
>Assignee: Francesco Mari
> Fix For: Segment Tar 0.0.14
>
>
> The test 
> {{org.apache.jackrabbit.oak.segment.standby.ExternalSharedStoreIT.testProxyFlippedIntermediateByteChange2}}
>  fails intermittently with the following error:
> {noformat}
> expected: org.apache.jackrabbit.oak.segment.SegmentNodeState<{ root = { ... } 
> }> but was: org.apache.jackrabbit.oak.segment.SegmentNodeState<{ root = { ... 
> } }>
> {noformat}
> The cause of the issue seems to be the proxy holding to a port after it's 
> been asked to disconnect:
> {noformat}
> java.net.BindException: Address already in use
>   at sun.nio.ch.Net.bind0(Native Method) ~[na:1.7.0_80]
>   at sun.nio.ch.Net.bind(Net.java:463) ~[na:1.7.0_80]
>   at sun.nio.ch.Net.bind(Net.java:455) ~[na:1.7.0_80]
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 
> ~[na:1.7.0_80]
>   at 
> io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127)
>  ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501)
>  ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218)
>  ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:505)
>  ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:490)
>  ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:965) 
> ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:210) 
> ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:353) 
> ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
>  ~[netty-common-4.0.41.Final.jar:4.0.41.Final]
>   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) 
> ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
>  ~[netty-common-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
>  ~[netty-common-4.0.41.Final.jar:4.0.41.Final]
>   at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80]
> {noformat}
> After that, the client fails when trying to connect to the server through the 
> proxy:
> {noformat}
> 09:39:11.795 ERROR [main] StandbyClientSync.java:160Failed 
> synchronizing state.
> io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection 
> refused: /127.0.0.1:41866
>   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 
> ~[na:1.7.0_80]
>   at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) 
> ~[na:1.7.0_80]
>   at 
> io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257)
>  ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)
>  ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628) 
> ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:552)
>  ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:466) 
> ~[netty-transport-4.0.41.Final.jar:4.0.41.Final]
>   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) 
> 

[jira] [Resolved] (OAK-4892) Speed up DocumentNodeStoreStatsCollectorIT

[Marcel Reutegger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcel Reutegger resolved OAK-4892.
---
   Resolution: Fixed
Fix Version/s: 1.5.12

Done in trunk: http://svn.apache.org/r1763367

> Speed up DocumentNodeStoreStatsCollectorIT
> --
>
> Key: OAK-4892
> URL: https://issues.apache.org/jira/browse/OAK-4892
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: core, documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.6, 1.5.12
>
>
> The test takes roughly 10 seconds to execute because the failed merge is 
> retried.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4892) Speed up DocumentNodeStoreStatsCollectorIT

[Marcel Reutegger (JIRA)]

Marcel Reutegger created OAK-4892:
-

 Summary: Speed up DocumentNodeStoreStatsCollectorIT
 Key: OAK-4892
 URL: https://issues.apache.org/jira/browse/OAK-4892
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: core, documentmk
Reporter: Marcel Reutegger
Assignee: Marcel Reutegger
Priority: Minor
 Fix For: 1.6


The test takes roughly 10 seconds to execute because the failed merge is 
retried.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4891) Mount-time sanity checks for mounted NodeStore instances

[Robert Munteanu (JIRA)]

Robert Munteanu created OAK-4891:


 Summary: Mount-time sanity checks for mounted NodeStore instances
 Key: OAK-4891
 URL: https://issues.apache.org/jira/browse/OAK-4891
 Project: Jackrabbit Oak
  Issue Type: Sub-task
  Components: core
Reporter: Robert Munteanu


When mounting a NodeStore we should be able to run a quick sanity check to make 
sure that it does not use any features which are troublesome, notably those 
which would contribute to one of the 'global' states of Oak:

- versioned nodes
- referenceable nodes
- etc ( more TDB )



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4890) Invalidate cache on missing previous document

[Marcel Reutegger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcel Reutegger resolved OAK-4890.
---
   Resolution: Fixed
Fix Version/s: 1.5.12

Done in trunk: http://svn.apache.org/r1763366

> Invalidate cache on missing previous document
> -
>
> Key: OAK-4890
> URL: https://issues.apache.org/jira/browse/OAK-4890
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: core, documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.6, 1.5.12
>
>
> After a revision garbage collection it may happen that references to previous 
> documents are temporarily stale until the referencing document is 
> invalidated. The DocumentNodeStore should therefore force invalidate 
> referencing documents when it detects a previous document does not exist 
> anymore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4167) Collect and backport lucene based suggestion improvements done in 1.4 into 1.2 branch

[Vikas Saurabh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548106#comment-15548106
 ] 

Vikas Saurabh commented on OAK-4167:


Relating to OAK-3917 as that is also a useful one in this regard. So, while 
this backport works at 1.2.14 but OAK-3917 (temp cleanup) is fixed for 1.2.20.

> Collect and backport lucene based suggestion improvements done in 1.4 into 
> 1.2 branch
> -
>
> Key: OAK-4167
> URL: https://issues.apache.org/jira/browse/OAK-4167
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: lucene
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
> Fix For: 1.2.14
>
>
> Here's a list of improvements done for lucene based suggestions that were 
> done for 1.4:
> # OAK-2754: Use non unique PathCursor in LucenePropertyIndex (*b) (/) 
> ([trunk@1682488|https://svn.apache.org/r1682488], 
> [1.2@1738061|https://svn.apache.org/r1738061])
> # OAK-3230: Query engine should support virtual index rows (*y) (/) 
> ([trunk@1697896|https://svn.apache.org/r1697896], 
> [1.2@1738062|https://svn.apache.org/r1738062])
> # OAK-3156: Allows suggestions to work on non-nt:base (*y) (/) 
> ([trunk@1708307|https://svn.apache.org/r1708307], 
> [1.2@1738063|https://svn.apache.org/r1738063])
> # OAK-3509: Returns suggesion result per row (while this isn't really 
> required, but it'd make other backports rather simple) (*y) (/) 
> ([trunk@1715716|https://svn.apache.org/r1715716], 
> [1.2@1738066|https://svn.apache.org/r1738066])
> # OAK-3149: Uses one suggest dictionary per index definition. Useful when 
> there are multiple indices supporting suggestions (*y) (/) 
> ([trunk@1716100|https://svn.apache.org/r1716100], 
> [1.2@1738068|https://svn.apache.org/r1738068])
> # OAK-3407: Allows to have single work suggestions instead of phrases (*y) 
> (/) ([trunk@1716588|https://svn.apache.org/r1716588], 
> [trunk@1716589|https://svn.apache.org/r1716589], 
> [trunk@1717203|https://svn.apache.org/r1717203], 
> [1.2@1738069|https://svn.apache.org/r1738069])
> # OAK-3692: java.lang.NoClassDefFoundError: 
> org/apache/lucene/index/sorter/Sorter$DocComparator (*y) (/) 
> ([trunk@1717410|https://svn.apache.org/r1717410], 
> [1.2@1738070|https://svn.apache.org/r1738070])
> # OAK-3838: Fixes issues of picking up a competing-non-suggest-capable index 
> incorrectly to provide suggestions (*y) (/) 
> ([trunk@1723565|https://svn.apache.org/r1723565], 
> [1.2@1738073|https://svn.apache.org/r1738073])
> # OAK-3825: Including Resource name to suggestions (*y) (/) 
> ([trunk@1725941|https://svn.apache.org/r1725941], 
> [1.2@1738074|https://svn.apache.org/r1738074])
> # OAK-2477: Namespaced global configs for suggestions (*y) (/) 
> ([trunk@1726237|https://svn.apache.org/r1726237], 
> [1.2@1738075|https://svn.apache.org/r1738075], 
> [1.2@1738085|https://svn.apache.org/r1738085])
> # OAK-3994: Simple query on suggestion/spellcheck with unambiguous index def 
> and one descendant clause should work (*y) (/) 
> ([trunk@1729505|https://svn.apache.org/r1729505], 
> [trunk@1729599|https://svn.apache.org/r1729599], 
> [1.2@1738076|https://svn.apache.org/r1738076])
> # OAK-4009: Search done via Lucene index might return duplicate results (*b) 
> (/) ([trunk@1729957|https://svn.apache.org/r1729957], 
> [1.2@1738077|https://svn.apache.org/r1738077])
> # OAK-4070: Refactor LuceneIndexEditorContext to allow for switchable clock 
> for testing (*b) (/) ([trunk@1732647|https://svn.apache.org/r1732647], 
> [1.2@1738078|https://svn.apache.org/r1738078])
> # OAK-4059: Tests in oak-lucene do not shut down repository (*b) (/) 
> ([trunk@1732268|https://svn.apache.org/r1732268], 
> [1.2@1738078|https://svn.apache.org/r1738078])
> # OAK-4068: Align default value of suggestUpdateFrequencyMinutes to match 
> documented 10 minutes (*y) (/) 
> ([trunk@1732648|https://svn.apache.org/r1732648], 
> [trunk@1732864|https://svn.apache.org/r1732864], 
> [1.2@1738078|https://svn.apache.org/r1738078])
> # OAK-4126: Suggestion and spellcheck queries throw exception if result 
> contain lucene query character (*y) (/) 
> ([trunk@1735052|https://svn.apache.org/r1735052], 
> [trunk@1736176|https://svn.apache.org/r1736176], 
> [1.2@1738079|https://svn.apache.org/r1738079], 
> [1.2@1738080|https://svn.apache.org/r1738080])
> We'd also need to update docs (*y) (/) (updated in 
> [r1738087|https://svn.apache.org/r1738087])
> (*y) - Required issues
> (*b) - Backported for dependency commit
> (x) - Not backported yet
> (/) - Backported



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3917) SuggestionHelper creating unnecessary temporary directories

[Julian Reschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548107#comment-15548107
 ] 

Julian Reschke commented on OAK-3917:
-

I see - sorry I missed that the commit to trunk was pre-1.4.

> SuggestionHelper creating unnecessary temporary directories
> ---
>
> Key: OAK-3917
> URL: https://issues.apache.org/jira/browse/OAK-3917
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
> Fix For: 1.4, 1.3.15, 1.2.20
>
> Attachments: OAK-3917.patch
>
>
> SuggestHelper create a temporary directory [1] mostly likely as a workaround 
> to allow use of {{OakDirectory}}. With time this can grow unbounded (as dir 
> is not cleaned) and following exception is seen.
> {noformat}
> java.lang.IllegalStateException: Failed to create directory within 1 
> attempts (tried 145369739-0 to 145369739-)
> at com.google.common.io.Files.createTempDir(Files.java:608)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:107)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:222)
> {noformat}
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.3.14/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/util/SuggestHelper.java#L107



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3917) SuggestionHelper creating unnecessary temporary directories

[Vikas Saurabh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548100#comment-15548100
 ] 

Vikas Saurabh commented on OAK-3917:


[~reschke], sorry my mistake. The issue was already part of 1.4. I just 
avoiding adding a new comment.

> SuggestionHelper creating unnecessary temporary directories
> ---
>
> Key: OAK-3917
> URL: https://issues.apache.org/jira/browse/OAK-3917
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
> Fix For: 1.4, 1.3.15, 1.2.20
>
> Attachments: OAK-3917.patch
>
>
> SuggestHelper create a temporary directory [1] mostly likely as a workaround 
> to allow use of {{OakDirectory}}. With time this can grow unbounded (as dir 
> is not cleaned) and following exception is seen.
> {noformat}
> java.lang.IllegalStateException: Failed to create directory within 1 
> attempts (tried 145369739-0 to 145369739-)
> at com.google.common.io.Files.createTempDir(Files.java:608)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:107)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:222)
> {noformat}
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.3.14/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/util/SuggestHelper.java#L107



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3917) SuggestionHelper creating unnecessary temporary directories

[Vikas Saurabh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Saurabh updated OAK-3917:
---
Fix Version/s: 1.4
   1.3.15

> SuggestionHelper creating unnecessary temporary directories
> ---
>
> Key: OAK-3917
> URL: https://issues.apache.org/jira/browse/OAK-3917
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
> Fix For: 1.4, 1.3.15, 1.2.20
>
> Attachments: OAK-3917.patch
>
>
> SuggestHelper create a temporary directory [1] mostly likely as a workaround 
> to allow use of {{OakDirectory}}. With time this can grow unbounded (as dir 
> is not cleaned) and following exception is seen.
> {noformat}
> java.lang.IllegalStateException: Failed to create directory within 1 
> attempts (tried 145369739-0 to 145369739-)
> at com.google.common.io.Files.createTempDir(Files.java:608)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:107)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:222)
> {noformat}
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.3.14/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/util/SuggestHelper.java#L107



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3917) SuggestionHelper creating unnecessary temporary directories

[Vikas Saurabh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Saurabh updated OAK-3917:
---
Labels:   (was: candidate_oak_1_4)

> SuggestionHelper creating unnecessary temporary directories
> ---
>
> Key: OAK-3917
> URL: https://issues.apache.org/jira/browse/OAK-3917
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
> Fix For: 1.4, 1.3.15, 1.2.20
>
> Attachments: OAK-3917.patch
>
>
> SuggestHelper create a temporary directory [1] mostly likely as a workaround 
> to allow use of {{OakDirectory}}. With time this can grow unbounded (as dir 
> is not cleaned) and following exception is seen.
> {noformat}
> java.lang.IllegalStateException: Failed to create directory within 1 
> attempts (tried 145369739-0 to 145369739-)
> at com.google.common.io.Files.createTempDir(Files.java:608)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:107)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:222)
> {noformat}
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.3.14/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/util/SuggestHelper.java#L107



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4890) Invalidate cache on missing previous document

[Marcel Reutegger (JIRA)]

Marcel Reutegger created OAK-4890:
-

 Summary: Invalidate cache on missing previous document
 Key: OAK-4890
 URL: https://issues.apache.org/jira/browse/OAK-4890
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: core, documentmk
Reporter: Marcel Reutegger
Assignee: Marcel Reutegger
Priority: Minor
 Fix For: 1.6


After a revision garbage collection it may happen that references to previous 
documents are temporarily stale until the referencing document is invalidated. 
The DocumentNodeStore should therefore force invalidate referencing documents 
when it detects a previous document does not exist anymore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4138) Decouple revision cleanup from the flush thread

[JIRA]

[ 
https://issues.apache.org/jira/browse/OAK-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548062#comment-15548062
 ] 

Michael Dürig commented on OAK-4138:


Just look at the last couple of commits from today: 
https://github.com/mduerig/jackrabbit-oak/commits/OAK-4138

This commit [1] corresponds to the latest trunk from yesterday. 

[1] 
https://github.com/mduerig/jackrabbit-oak/commit/2c44516ab81e15f4bff3c514677cf25bd269d8ad

> Decouple revision cleanup from the flush thread
> ---
>
> Key: OAK-4138
> URL: https://issues.apache.org/jira/browse/OAK-4138
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: resilience
> Fix For: Segment Tar 0.0.14
>
>
> I suggest we decouple revision cleanup from the flush thread. With large 
> repositories where cleanup can take several minutes to complete it blocks the 
> flush thread from updating the journal and the persisted head thus resulting 
> in larger then necessary data loss in case of a crash. 
> /cc [~alex.parvulescu]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3917) SuggestionHelper creating unnecessary temporary directories

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-3917:

Fix Version/s: (was: 1.4)

> SuggestionHelper creating unnecessary temporary directories
> ---
>
> Key: OAK-3917
> URL: https://issues.apache.org/jira/browse/OAK-3917
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
>  Labels: candidate_oak_1_4
> Fix For: 1.2.20
>
> Attachments: OAK-3917.patch
>
>
> SuggestHelper create a temporary directory [1] mostly likely as a workaround 
> to allow use of {{OakDirectory}}. With time this can grow unbounded (as dir 
> is not cleaned) and following exception is seen.
> {noformat}
> java.lang.IllegalStateException: Failed to create directory within 1 
> attempts (tried 145369739-0 to 145369739-)
> at com.google.common.io.Files.createTempDir(Files.java:608)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:107)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:222)
> {noformat}
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.3.14/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/util/SuggestHelper.java#L107



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3917) SuggestionHelper creating unnecessary temporary directories

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-3917:

Fix Version/s: (was: 1.3.15)

> SuggestionHelper creating unnecessary temporary directories
> ---
>
> Key: OAK-3917
> URL: https://issues.apache.org/jira/browse/OAK-3917
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
>  Labels: candidate_oak_1_4
> Fix For: 1.2.20
>
> Attachments: OAK-3917.patch
>
>
> SuggestHelper create a temporary directory [1] mostly likely as a workaround 
> to allow use of {{OakDirectory}}. With time this can grow unbounded (as dir 
> is not cleaned) and following exception is seen.
> {noformat}
> java.lang.IllegalStateException: Failed to create directory within 1 
> attempts (tried 145369739-0 to 145369739-)
> at com.google.common.io.Files.createTempDir(Files.java:608)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:107)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:222)
> {noformat}
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.3.14/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/util/SuggestHelper.java#L107



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4860) Backport OAK-4301 and OAK-4825

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dominique Jäggi resolved OAK-4860.
--
   Resolution: Fixed
Fix Version/s: 1.4.8

backported OAK-4825 in r1763363.

> Backport OAK-4301 and OAK-4825
> --
>
> Key: OAK-4860
> URL: https://issues.apache.org/jira/browse/OAK-4860
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: auth-external, core
>Reporter: Dominique Jäggi
>Assignee: Dominique Jäggi
> Fix For: 1.4.9, 1.4.8
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3917) SuggestionHelper creating unnecessary temporary directories

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-3917:

Labels: candidate_oak_1_4  (was: )

> SuggestionHelper creating unnecessary temporary directories
> ---
>
> Key: OAK-3917
> URL: https://issues.apache.org/jira/browse/OAK-3917
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
>  Labels: candidate_oak_1_4
> Fix For: 1.2.20
>
> Attachments: OAK-3917.patch
>
>
> SuggestHelper create a temporary directory [1] mostly likely as a workaround 
> to allow use of {{OakDirectory}}. With time this can grow unbounded (as dir 
> is not cleaned) and following exception is seen.
> {noformat}
> java.lang.IllegalStateException: Failed to create directory within 1 
> attempts (tried 145369739-0 to 145369739-)
> at com.google.common.io.Files.createTempDir(Files.java:608)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:107)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:222)
> {noformat}
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.3.14/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/util/SuggestHelper.java#L107



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4825) Support disabling of users instead of removal in DefaultSyncHandler

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dominique Jäggi updated OAK-4825:
-
Fix Version/s: 1.4.9

> Support disabling of users instead of removal in DefaultSyncHandler
> ---
>
> Key: OAK-4825
> URL: https://issues.apache.org/jira/browse/OAK-4825
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: auth-external
>Reporter: Alexander Klimetschek
>Assignee: Dominique Jäggi
> Fix For: 1.6, 1.4.9
>
> Attachments: OAK-4825-b.patch, OAK-4825-c.patch, OAK-4825-doc.patch, 
> OAK-4825.patch
>
>
> The DefaultSyncHandler by default will remove (local) users when they are no 
> longer active in the external system aka no longer provided by the 
> ExternalIdentityProvider. It would be useful to have an option to _disable_ 
> users instead of removing them completely.
> This is good for use cases that need to keep the user's data around in the 
> JCR and can't just delete it. Also, we have seen cases where the user is only 
> temporarily removed from the external identity system (e.g. accidentally 
> removed from group that maps them to the JCR system and quickly added back), 
> where a full removal can do unnecessary operations.
> (Note: There is an [option in the SyncContext 
> interface|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/SyncContext.java#L38]
>  to suppress purging completely, aka they won't be removed nor disabled, and 
> the JMX sync commands such as 
> [purgeOrphanedUsers()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/impl/jmx/Delegatee.java#L256]
>  "use" it. However, the JCR users look like "valid" users then locally. Even 
> if the authentication is done completely through the IDP and will fail 
> properly for these missing users, it can be difficult for other uses like 
> administration, monitoring, other application code to detect that such a user 
> is not active anymore.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3917) SuggestionHelper creating unnecessary temporary directories

[Julian Reschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548055#comment-15548055
 ] 

Julian Reschke commented on OAK-3917:
-

What about 1.4?

> SuggestionHelper creating unnecessary temporary directories
> ---
>
> Key: OAK-3917
> URL: https://issues.apache.org/jira/browse/OAK-3917
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
> Fix For: 1.4, 1.3.15, 1.2.20
>
> Attachments: OAK-3917.patch
>
>
> SuggestHelper create a temporary directory [1] mostly likely as a workaround 
> to allow use of {{OakDirectory}}. With time this can grow unbounded (as dir 
> is not cleaned) and following exception is seen.
> {noformat}
> java.lang.IllegalStateException: Failed to create directory within 1 
> attempts (tried 145369739-0 to 145369739-)
> at com.google.common.io.Files.createTempDir(Files.java:608)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:107)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:222)
> {noformat}
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.3.14/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/util/SuggestHelper.java#L107



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4138) Decouple revision cleanup from the flush thread

[Alex Parvulescu (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548052#comment-15548052
 ] 

Alex Parvulescu commented on OAK-4138:
--

bq. This branch is 2228 commits ahead, 2229 commits behind apache:trunk.
how would I go about looking at this patch?

> Decouple revision cleanup from the flush thread
> ---
>
> Key: OAK-4138
> URL: https://issues.apache.org/jira/browse/OAK-4138
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: resilience
> Fix For: Segment Tar 0.0.14
>
>
> I suggest we decouple revision cleanup from the flush thread. With large 
> repositories where cleanup can take several minutes to complete it blocks the 
> flush thread from updating the journal and the persisted head thus resulting 
> in larger then necessary data loss in case of a crash. 
> /cc [~alex.parvulescu]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-3917) SuggestionHelper creating unnecessary temporary directories

[Vikas Saurabh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119938#comment-15119938
 ] 

Vikas Saurabh edited comment on OAK-3917 at 10/5/16 8:16 AM:
-

Talked to Chetan and modified patch to create/cleanup temp dir in 
{{SuggestHelper#updateSuggester}}. Fixed in trunk at 
[r1727149|http://svn.apache.org/r1727149]. Backported to 1.2 at 
[r1763360|https://svn.apache.org/r1763360].


was (Author: catholicon):
Talked to Chetan and modified patch to create/cleanup temp dir in 
{{SuggestHelper#updateSuggester}}. Fixed in trunk at 
[r1727149|http://svn.apache.org/r1727149].

> SuggestionHelper creating unnecessary temporary directories
> ---
>
> Key: OAK-3917
> URL: https://issues.apache.org/jira/browse/OAK-3917
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
> Fix For: 1.4, 1.3.15, 1.2.20
>
> Attachments: OAK-3917.patch
>
>
> SuggestHelper create a temporary directory [1] mostly likely as a workaround 
> to allow use of {{OakDirectory}}. With time this can grow unbounded (as dir 
> is not cleaned) and following exception is seen.
> {noformat}
> java.lang.IllegalStateException: Failed to create directory within 1 
> attempts (tried 145369739-0 to 145369739-)
> at com.google.common.io.Files.createTempDir(Files.java:608)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:107)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:222)
> {noformat}
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.3.14/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/util/SuggestHelper.java#L107



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3917) SuggestionHelper creating unnecessary temporary directories

[Vikas Saurabh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Saurabh updated OAK-3917:
---
Fix Version/s: 1.2.20

> SuggestionHelper creating unnecessary temporary directories
> ---
>
> Key: OAK-3917
> URL: https://issues.apache.org/jira/browse/OAK-3917
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
> Fix For: 1.4, 1.3.15, 1.2.20
>
> Attachments: OAK-3917.patch
>
>
> SuggestHelper create a temporary directory [1] mostly likely as a workaround 
> to allow use of {{OakDirectory}}. With time this can grow unbounded (as dir 
> is not cleaned) and following exception is seen.
> {noformat}
> java.lang.IllegalStateException: Failed to create directory within 1 
> attempts (tried 145369739-0 to 145369739-)
> at com.google.common.io.Files.createTempDir(Files.java:608)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:107)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:222)
> {noformat}
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.3.14/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/util/SuggestHelper.java#L107



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3917) SuggestionHelper creating unnecessary temporary directories

[Vikas Saurabh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikas Saurabh updated OAK-3917:
---
Labels:   (was: candidate_oak_1_2)

> SuggestionHelper creating unnecessary temporary directories
> ---
>
> Key: OAK-3917
> URL: https://issues.apache.org/jira/browse/OAK-3917
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
> Fix For: 1.4, 1.3.15, 1.2.20
>
> Attachments: OAK-3917.patch
>
>
> SuggestHelper create a temporary directory [1] mostly likely as a workaround 
> to allow use of {{OakDirectory}}. With time this can grow unbounded (as dir 
> is not cleaned) and following exception is seen.
> {noformat}
> java.lang.IllegalStateException: Failed to create directory within 1 
> attempts (tried 145369739-0 to 145369739-)
> at com.google.common.io.Files.createTempDir(Files.java:608)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:107)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
> at 
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:222)
> {noformat}
> [1] 
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.3.14/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/util/SuggestHelper.java#L107



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4138) Decouple revision cleanup from the flush thread

[JIRA]

[ 
https://issues.apache.org/jira/browse/OAK-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547997#comment-15547997
 ] 

Michael Dürig commented on OAK-4138:


Initial implementation of this together with fixes for OAK-4685 and OAK-4621 at 
https://github.com/mduerig/jackrabbit-oak/tree/OAK-4138



> Decouple revision cleanup from the flush thread
> ---
>
> Key: OAK-4138
> URL: https://issues.apache.org/jira/browse/OAK-4138
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: resilience
> Fix For: Segment Tar 0.0.14
>
>
> I suggest we decouple revision cleanup from the flush thread. With large 
> repositories where cleanup can take several minutes to complete it blocks the 
> flush thread from updating the journal and the persisted head thus resulting 
> in larger then necessary data loss in case of a crash. 
> /cc [~alex.parvulescu]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4883) Missing BlobGarbageCollection MBean on standby instance

[Alex Parvulescu (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547994#comment-15547994
 ] 

Alex Parvulescu commented on OAK-4883:
--

bq. Before digging further, could you help me understand whether it is expected 
that the store.getBlobStore() may return null in some setups
yes, it's expected to have {{null}} when there's no external datastore 
configured. When there is one though, the primary will provide a 
{{BlobGCMBean}}, but the standby won't, and this is what the current issue is 
about: having a {{BlobGCMBean}} on the standby.

> Missing BlobGarbageCollection MBean on standby instance
> ---
>
> Key: OAK-4883
> URL: https://issues.apache.org/jira/browse/OAK-4883
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segment-tar, segmentmk
>Reporter: Alex Parvulescu
>Assignee: Timothee Maret
>  Labels: gc
>
> The {{BlobGarbageCollection}} MBean is no longer available on a standby 
> instance, this affects non-shared datastore setups (on a shared datastore 
> you'd only need to run blob gc on the primary).
> This change was introduced by OAK-4089 (and backported to 1.4 branch with 
> OAK-4093).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (OAK-4621) External invocation of background operations

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig reassigned OAK-4621:
--

Assignee: Michael Dürig

> External invocation of background operations
> 
>
> Key: OAK-4621
> URL: https://issues.apache.org/jira/browse/OAK-4621
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segment-tar
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: technical_debt
>
> The background operations (flush, compact, cleanup, etc.) are historically 
> part of the implementation of the {{FileStore}}. They should better be 
> scheduled and invoked by an external agent. The code deploying the 
> {{FileStore}} might have better insights on when and how these background 
> operations should be invoked. See also OAK-3468.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4825) Support disabling of users instead of removal in DefaultSyncHandler

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dominique Jäggi resolved OAK-4825.
--
Resolution: Fixed
  Assignee: Dominique Jäggi

committed in trunk in r1763355, doc in r1763356.

> Support disabling of users instead of removal in DefaultSyncHandler
> ---
>
> Key: OAK-4825
> URL: https://issues.apache.org/jira/browse/OAK-4825
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: auth-external
>Reporter: Alexander Klimetschek
>Assignee: Dominique Jäggi
> Fix For: 1.6
>
> Attachments: OAK-4825-b.patch, OAK-4825-c.patch, OAK-4825-doc.patch, 
> OAK-4825.patch
>
>
> The DefaultSyncHandler by default will remove (local) users when they are no 
> longer active in the external system aka no longer provided by the 
> ExternalIdentityProvider. It would be useful to have an option to _disable_ 
> users instead of removing them completely.
> This is good for use cases that need to keep the user's data around in the 
> JCR and can't just delete it. Also, we have seen cases where the user is only 
> temporarily removed from the external identity system (e.g. accidentally 
> removed from group that maps them to the JCR system and quickly added back), 
> where a full removal can do unnecessary operations.
> (Note: There is an [option in the SyncContext 
> interface|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/SyncContext.java#L38]
>  to suppress purging completely, aka they won't be removed nor disabled, and 
> the JMX sync commands such as 
> [purgeOrphanedUsers()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/impl/jmx/Delegatee.java#L256]
>  "use" it. However, the JCR users look like "valid" users then locally. Even 
> if the authentication is done completely through the IDP and will fail 
> properly for these missing users, it can be difficult for other uses like 
> administration, monitoring, other application code to detect that such a user 
> is not active anymore.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4825) Support disabling of users instead of removal in DefaultSyncHandler

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dominique Jäggi updated OAK-4825:
-
Fix Version/s: 1.6

> Support disabling of users instead of removal in DefaultSyncHandler
> ---
>
> Key: OAK-4825
> URL: https://issues.apache.org/jira/browse/OAK-4825
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: auth-external
>Reporter: Alexander Klimetschek
> Fix For: 1.6
>
> Attachments: OAK-4825-b.patch, OAK-4825-c.patch, OAK-4825-doc.patch, 
> OAK-4825.patch
>
>
> The DefaultSyncHandler by default will remove (local) users when they are no 
> longer active in the external system aka no longer provided by the 
> ExternalIdentityProvider. It would be useful to have an option to _disable_ 
> users instead of removing them completely.
> This is good for use cases that need to keep the user's data around in the 
> JCR and can't just delete it. Also, we have seen cases where the user is only 
> temporarily removed from the external identity system (e.g. accidentally 
> removed from group that maps them to the JCR system and quickly added back), 
> where a full removal can do unnecessary operations.
> (Note: There is an [option in the SyncContext 
> interface|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/SyncContext.java#L38]
>  to suppress purging completely, aka they won't be removed nor disabled, and 
> the JMX sync commands such as 
> [purgeOrphanedUsers()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-auth-external/src/main/java/org/apache/jackrabbit/oak/spi/security/authentication/external/impl/jmx/Delegatee.java#L256]
>  "use" it. However, the JCR users look like "valid" users then locally. Even 
> if the authentication is done completely through the IDP and will fail 
> properly for these missing users, it can be difficult for other uses like 
> administration, monitoring, other application code to detect that such a user 
> is not active anymore.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3253) Support caching in FileDataStoreService

[Amit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547934#comment-15547934
 ] 

Amit Jain commented on OAK-3253:


Backported to 1.2 branch - http://svn.apache.org/viewvc?rev=1763343=rev

> Support caching in FileDataStoreService
> ---
>
> Key: OAK-3253
> URL: https://issues.apache.org/jira/browse/OAK-3253
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob
>Affects Versions: 1.3.3
>Reporter: Shashank Gupta
>Assignee: Amit Jain
>Priority: Blocker
>  Labels: candidate_oak_1_0, docs-impacting, features, performance
> Fix For: 1.4, 1.3.15, 1.2.20
>
> Attachments: OAK-3253.txt, 
> org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.config
>
>
> FDS on SAN/NAS storage is not efficient as it involves network call. In OAK. 
> indexes are stored SAN/NAS and even idle system does lot of read system 
> generated data. 
> Enable caching in FDS so the reads are done locally and async upload to 
> SAN/NAS
> See [previous 
> discussions|https://issues.apache.org/jira/browse/OAK-3005?focusedCommentId=14700801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700801]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3253) Support caching in FileDataStoreService

[Amit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Jain updated OAK-3253:
---
Labels: candidate_oak_1_0 docs-impacting features performance  (was: 
candidate_oak_1_0 candidate_oak_1_2 docs-impacting features performance)

> Support caching in FileDataStoreService
> ---
>
> Key: OAK-3253
> URL: https://issues.apache.org/jira/browse/OAK-3253
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob
>Affects Versions: 1.3.3
>Reporter: Shashank Gupta
>Assignee: Amit Jain
>Priority: Blocker
>  Labels: candidate_oak_1_0, docs-impacting, features, performance
> Fix For: 1.4, 1.3.15, 1.2.20
>
> Attachments: OAK-3253.txt, 
> org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.config
>
>
> FDS on SAN/NAS storage is not efficient as it involves network call. In OAK. 
> indexes are stored SAN/NAS and even idle system does lot of read system 
> generated data. 
> Enable caching in FDS so the reads are done locally and async upload to 
> SAN/NAS
> See [previous 
> discussions|https://issues.apache.org/jira/browse/OAK-3005?focusedCommentId=14700801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700801]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3253) Support caching in FileDataStoreService

[Amit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Jain updated OAK-3253:
---
Fix Version/s: 1.2.20

> Support caching in FileDataStoreService
> ---
>
> Key: OAK-3253
> URL: https://issues.apache.org/jira/browse/OAK-3253
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: blob
>Affects Versions: 1.3.3
>Reporter: Shashank Gupta
>Assignee: Amit Jain
>Priority: Blocker
>  Labels: candidate_oak_1_0, docs-impacting, features, performance
> Fix For: 1.4, 1.3.15, 1.2.20
>
> Attachments: OAK-3253.txt, 
> org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.config
>
>
> FDS on SAN/NAS storage is not efficient as it involves network call. In OAK. 
> indexes are stored SAN/NAS and even idle system does lot of read system 
> generated data. 
> Enable caching in FDS so the reads are done locally and async upload to 
> SAN/NAS
> See [previous 
> discussions|https://issues.apache.org/jira/browse/OAK-3005?focusedCommentId=14700801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700801]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-4879) Proper implementation of getOrCreateReferenceKey in CachingFDS

[Amit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Jain resolved OAK-4879.

Resolution: Fixed

Fixed with:
* 1.2 - http://svn.apache.org/viewvc?rev=1763349=rev
* 1.4 - http://svn.apache.org/viewvc?rev=1763350=rev
* trunk - http://svn.apache.org/viewvc?rev=1763347=rev

> Proper implementation of getOrCreateReferenceKey in CachingFDS
> --
>
> Key: OAK-4879
> URL: https://issues.apache.org/jira/browse/OAK-4879
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: blob
>Reporter: Amit Jain
>Assignee: Amit Jain
> Fix For: 1.6, 1.2.20, 1.4.9
>
> Attachments: OAK-4879.patch
>
>
> When cacheSize is configured to be > 0 then {{CachingFDS}} is registered but 
> the implementation of {{getOrCreateReferenceKey}} is different from 
> {{FileDataStore}} since, it inherits from {{CachingDataStore}} which 
> implements a different method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4888) Warn or fail queries above a configurable cost value

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547910#comment-15547910
 ] 

Thomas Mueller commented on OAK-4888:
-

This is a nice feature, even thought it might be a bit tricky to get right 
(avoid false positives, and catch all bad cases).

We could log warnings if no good index is available, so traversing has to be 
used. There could be false positives, for example queries that traverse a 
small, limited number of nodes (queries with a tight path constraint).

We could log warnings if an index is available, but only a very broad one (for 
example just the nodetype index, for nodetypes that are common). The question 
is, how can the query engine decide which nodetypes / values for a property are 
common (in an almost empty repository).

Configuration could be done similar to how we can configure the maximum number 
of read rows (QueryEngineSettings).



> Warn or fail queries above a configurable cost value
> 
>
> Key: OAK-4888
> URL: https://issues.apache.org/jira/browse/OAK-4888
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Reporter: Alexander Klimetschek
>Assignee: Thomas Mueller
> Fix For: 1.6
>
>
> *Problem:* It's easy to run into performance problems with queries that are 
> not backed by an index or miss the right one. Developers writing these 
> queries typically do not have the real large production data, and thus don't 
> see that a query would scale badly, and would not see any traversal warnings, 
> as these only happen with a large number of results.
> *Proposal:* Oak's query engine already calculates a cost estimate to make a 
> decision which index to use, or even if there is any at all. This cost 
> estimate could be used to find out if a query is not supported by an index or 
> with one that is not suitable enough (e.g. ordering by property that is not 
> indexed)
> If a query is above a certain cost value, a big warning could be put out or 
> even the query could be made to fail (maybe a per query option, that you 
> might want to have to "fail" by default to ensure people are not overlooking 
> the problem). The cost limit should be configurable, as it might depend on 
> the hardware power.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (OAK-4888) Warn or fail queries above a configurable cost value

[Thomas Mueller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller reassigned OAK-4888:
---

Assignee: Thomas Mueller

> Warn or fail queries above a configurable cost value
> 
>
> Key: OAK-4888
> URL: https://issues.apache.org/jira/browse/OAK-4888
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Reporter: Alexander Klimetschek
>Assignee: Thomas Mueller
> Fix For: 1.6
>
>
> *Problem:* It's easy to run into performance problems with queries that are 
> not backed by an index or miss the right one. Developers writing these 
> queries typically do not have the real large production data, and thus don't 
> see that a query would scale badly, and would not see any traversal warnings, 
> as these only happen with a large number of results.
> *Proposal:* Oak's query engine already calculates a cost estimate to make a 
> decision which index to use, or even if there is any at all. This cost 
> estimate could be used to find out if a query is not supported by an index or 
> with one that is not suitable enough (e.g. ordering by property that is not 
> indexed)
> If a query is above a certain cost value, a big warning could be put out or 
> even the query could be made to fail (maybe a per query option, that you 
> might want to have to "fail" by default to ensure people are not overlooking 
> the problem). The cost limit should be configurable, as it might depend on 
> the hardware power.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-4887) Query cost estimation: ordering by an unindexed property not reflected

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547892#comment-15547892
 ] 

Thomas Mueller edited comment on OAK-4887 at 10/5/16 7:02 AM:
--

Yes, the cost of ordering is not accounted for within the query engine 
(org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan). For 
small results that easily fit in memory, let's say 1000 rows, the cost of 
ordering itself is very low, almost negligible. But one problem is that with 
"order by propertyName", the whole result needs to be read in memory before the 
first row can be returned. Without "order by", the first row can be returned 
much faster.

If there are two indexes with similar cost, one that returns rows in sorted 
order, and one that returns rows unsorted, then it would be better to use the 
sorted index. How much better is hard to say, it depends a lot on the number of 
rows that we expected to be read, and that number is not known.

Maybe we should support a way to specify the number of rows that the query 
engine should optimize for; that is, the number of rows that are _expected_ to 
be read. This would be similar to the "fastfirstrow" option / "option (fast 
)" in MS SQL Server: 
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/09a6060a-1f72-4438-a3b2-209c240ee4d6/fastfirstrow?forum=transactsql
 . Currently, we support the "limit" option, but maybe a user needs to read 
more than  rows in some cases, so the limit is not known.

By the way, currently, if the chosen index supports ordering, then ordering is 
not done afterwards. So this part is working fine.





was (Author: tmueller):
Yes, the cost of ordering is not accounted for within the query engine 
(org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan). For 
small results that easily fit in memory, let's say 1000 rows, the cost of 
ordering itself is very low, almost negligible. But one problem is that with 
"order by propertyName", the whole result needs to be read in memory before the 
first row can be returned. Without "order by", the first row can be returned 
much faster.

If there are two indexes with similar cost, one that returns rows in sorted 
order, and one that returns rows unsorted, then it would be better to use that 
index. How much better is hard to say, it depends a lot on the number of rows 
that we expected to be read, and that number is not known.

Maybe we should support a way to specify the number of rows that the query 
engine should optimize for; that is, the number of rows that are _expected_ to 
be read. This would be similar to the "fastfirstrow" option / "option (fast 
)" in MS SQL Server: 
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/09a6060a-1f72-4438-a3b2-209c240ee4d6/fastfirstrow?forum=transactsql
 . Currently, we support the "limit" option, but maybe a user needs to read 
more than  rows in some cases, so the limit is not known.

By the way, currently, if the chosen index supports ordering, then ordering is 
not done afterwards. So this part is working fine.




> Query cost estimation: ordering by an unindexed property not reflected
> --
>
> Key: OAK-4887
> URL: https://issues.apache.org/jira/browse/OAK-4887
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Affects Versions: 1.4.2
>Reporter: Alexander Klimetschek
>Assignee: Thomas Mueller
> Fix For: 1.6
>
>
> A query that orders by an unindexed property seems to have no effect on the 
> cost estimation, compared to the same query without the order by, although it 
> has a big impact on the execution performance for larger results/indexes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-4887) Query cost estimation: ordering by an unindexed property not reflected

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547892#comment-15547892
 ] 

Thomas Mueller commented on OAK-4887:
-

Yes, the cost of ordering is not accounted for within the query engine 
(org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan). For 
small results that easily fit in memory, let's say 1000 rows, the cost of 
ordering itself is very low, almost negligible. But one problem is that with 
"order by propertyName", the whole result needs to be read in memory before the 
first row can be returned. Without "order by", the first row can be returned 
much faster.

If there are two indexes with similar cost, one that returns rows in sorted 
order, and one that returns rows unsorted, then it would be better to use that 
index. How much better is hard to say, it depends a lot on the number of rows 
that we expected to be read, and that number is not known.

Maybe we should support a way to specify the number of rows that the query 
engine should optimize for; that is, the number of rows that are _expected_ to 
be read. This would be similar to the "fastfirstrow" option / "option (fast 
)" in MS SQL Server: 
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/09a6060a-1f72-4438-a3b2-209c240ee4d6/fastfirstrow?forum=transactsql
 . Currently, we support the "limit" option, but maybe a user needs to read 
more than  rows in some cases, so the limit is not known.

By the way, currently, if the chosen index supports ordering, then ordering is 
not done afterwards. So this part is working fine.




> Query cost estimation: ordering by an unindexed property not reflected
> --
>
> Key: OAK-4887
> URL: https://issues.apache.org/jira/browse/OAK-4887
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Affects Versions: 1.4.2
>Reporter: Alexander Klimetschek
>Assignee: Thomas Mueller
> Fix For: 1.6
>
>
> A query that orders by an unindexed property seems to have no effect on the 
> cost estimation, compared to the same query without the order by, although it 
> has a big impact on the execution performance for larger results/indexes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4885:

Description: See 
 (and 
).  (was: See 
.)

> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
>  Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
>
> See  
> (and ).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4885:

Affects Version/s: 1.2.19
   1.0.34
   1.4.8
   1.5.11

> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
>  Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
>
> See .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4885) RDB*Store: update mysql JDBC driver reference to 5.1.40

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-4885:

Labels: candidate_oak_1_0 candidate_oak_1_2 candidate_oak_1_4  (was: )

> RDB*Store: update mysql JDBC driver reference to 5.1.40
> ---
>
> Key: OAK-4885
> URL: https://issues.apache.org/jira/browse/OAK-4885
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.19, 1.0.34, 1.4.8, 1.5.11
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>Priority: Trivial
>  Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4
>
> See .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (OAK-4887) Query cost estimation: ordering by an unindexed property not reflected

[Thomas Mueller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller reassigned OAK-4887:
---

Assignee: Thomas Mueller

> Query cost estimation: ordering by an unindexed property not reflected
> --
>
> Key: OAK-4887
> URL: https://issues.apache.org/jira/browse/OAK-4887
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Affects Versions: 1.4.2
>Reporter: Alexander Klimetschek
>Assignee: Thomas Mueller
> Fix For: 1.6
>
>
> A query that orders by an unindexed property seems to have no effect on the 
> cost estimation, compared to the same query without the order by, although it 
> has a big impact on the execution performance for larger results/indexes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-4887) Query cost estimation: ordering by an unindexed property not reflected

[Thomas Mueller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller updated OAK-4887:

Fix Version/s: 1.6

> Query cost estimation: ordering by an unindexed property not reflected
> --
>
> Key: OAK-4887
> URL: https://issues.apache.org/jira/browse/OAK-4887
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Affects Versions: 1.4.2
>Reporter: Alexander Klimetschek
>Assignee: Thomas Mueller
> Fix For: 1.6
>
>
> A query that orders by an unindexed property seems to have no effect on the 
> cost estimation, compared to the same query without the order by, although it 
> has a big impact on the execution performance for larger results/indexes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-4889) Update Oak trunk to Jackrabbit 2.13.4

[Julian Reschke (JIRA)]

Julian Reschke created OAK-4889:
---

 Summary: Update Oak trunk to Jackrabbit 2.13.4
 Key: OAK-4889
 URL: https://issues.apache.org/jira/browse/OAK-4889
 Project: Jackrabbit Oak
  Issue Type: Task
  Components: parent
Affects Versions: 1.5.11
Reporter: Julian Reschke
Assignee: Julian Reschke
Priority: Trivial






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)