[jira] [Resolved] (OAK-3391) RDBBlobStore: speed up testBigBlob(), also improve memory usage

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke resolved OAK-3391.
-
Resolution: Fixed

> RDBBlobStore: speed up testBigBlob(), also improve memory usage
> ---
>
> Key: OAK-3391
> URL: https://issues.apache.org/jira/browse/OAK-3391
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.4, 1.3.5, 1.0.19
>Reporter: Julian Reschke
>Assignee: Julian Reschke
> Fix For: 1.0.20, 1.3.6, 1.2.5
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3371) Wrong evaluation of NOT clause

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740249#comment-14740249
 ] 

Thomas Mueller commented on OAK-3371:
-

Generally, it looks good, I like the approach. There are a few minor things I 
would like to change: I don't like the "instanceof" check there. I think it's 
possible to avoid it, or move it to the "simplify" method. The method name 
"restrictPropertyOnFilter": it seems to be unnecessary, as the same can be done 
using "enforcePropertyExistence".

I will try to work on the patch and upload a new one.

> Wrong evaluation of NOT clause
> --
>
> Key: OAK-3371
> URL: https://issues.apache.org/jira/browse/OAK-3371
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: query
>Affects Versions: 1.2.4, 1.3.5
>Reporter: Davide Giannella
>Assignee: Davide Giannella
> Fix For: 1.3.6
>
> Attachments: OAK-3371-2.patch, OAK-3371-test.diff, OAK-3371.patch
>
>
> When executing a query like 
> {noformat}
> SELECT * FROM [nt:unstructured] WHERE ISDESCENDANTNODE([/test]) AND NOT 
> CONTAINS(foo, 'bar')
> {noformat}
> and the {{nodeType}} index plays the not clause is not applied properly. 
> Nodes **with** the property are returned as well.
> [test|^OAK-3371-test.diff] showing the bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3031) [Blob GC] Mbean for reporting shared repository GC stats

[Amit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740170#comment-14740170
 ] 

Amit Jain commented on OAK-3031:


Commited in 1.2 branch with http://svn.apache.org/r1702374

> [Blob GC] Mbean for reporting shared repository GC stats
> 
>
> Key: OAK-3031
> URL: https://issues.apache.org/jira/browse/OAK-3031
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob
>Reporter: Amit Jain
>Assignee: Amit Jain
>  Labels: resilience, tooling
> Fix For: 1.3.5, 1.2.5
>
> Attachments: OAK-3031.patch
>
>
> For GC on a shared repository (OAK-1849) it is beneficial to add a JMX Mbean 
> which can provide visibility on the state of GC. It could possibly show:
> * Various repositories registered in the DataStore
> * State of the blob reference collection for the registered repositories
> * Time of the reference files for each registered repository
> * Time interval for the earliest and the latest reference file of the 
> registered repositories. This could be used to possibly automate the sweep 
> phase if the time interval is less than a configured value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3031) [Blob GC] Mbean for reporting shared repository GC stats

[Amit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Jain updated OAK-3031:
---
Fix Version/s: 1.2.5

> [Blob GC] Mbean for reporting shared repository GC stats
> 
>
> Key: OAK-3031
> URL: https://issues.apache.org/jira/browse/OAK-3031
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob
>Reporter: Amit Jain
>Assignee: Amit Jain
>  Labels: resilience, tooling
> Fix For: 1.3.5, 1.2.5
>
> Attachments: OAK-3031.patch
>
>
> For GC on a shared repository (OAK-1849) it is beneficial to add a JMX Mbean 
> which can provide visibility on the state of GC. It could possibly show:
> * Various repositories registered in the DataStore
> * State of the blob reference collection for the registered repositories
> * Time of the reference files for each registered repository
> * Time interval for the earliest and the latest reference file of the 
> registered repositories. This could be used to possibly automate the sweep 
> phase if the time interval is less than a configured value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3184) Consistency checker for data/blob store

[Amit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Jain updated OAK-3184:
---
Labels: candidate_oak_1_2 resilience tooling  (was: resilience tooling)

> Consistency checker for data/blob store
> ---
>
> Key: OAK-3184
> URL: https://issues.apache.org/jira/browse/OAK-3184
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob
>Reporter: Amit Jain
>Assignee: Amit Jain
>  Labels: candidate_oak_1_2, resilience, tooling
> Fix For: 1.3.6
>
> Attachments: OAK-3184.patch
>
>
> Consistency checker for data/blob store to report any missing blobs which can 
> result as a consequence of erroneous gc or extraneous factors. 
> * Will report any missing blob ids
> * Path of nodes referring to those blobs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3184) Consistency checker for data/blob store

[Amit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740143#comment-14740143
 ] 

Amit Jain commented on OAK-3184:


Committed in trunk with http://svn.apache.org/r1702371

> Consistency checker for data/blob store
> ---
>
> Key: OAK-3184
> URL: https://issues.apache.org/jira/browse/OAK-3184
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob
>Reporter: Amit Jain
>Assignee: Amit Jain
>  Labels: candidate_oak_1_2, resilience, tooling
> Fix For: 1.3.6
>
> Attachments: OAK-3184.patch
>
>
> Consistency checker for data/blob store to report any missing blobs which can 
> result as a consequence of erroneous gc or extraneous factors. 
> * Will report any missing blob ids
> * Path of nodes referring to those blobs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3371) Wrong evaluation of NOT clause

[Davide Giannella (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davide Giannella updated OAK-3371:
--
Attachment: OAK-3371-2.patch

[~chetanm], [~tmueller]: [second patch|^OAK-3371-2.patch] for reviewing. thanks.

> Wrong evaluation of NOT clause
> --
>
> Key: OAK-3371
> URL: https://issues.apache.org/jira/browse/OAK-3371
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: query
>Affects Versions: 1.2.4, 1.3.5
>Reporter: Davide Giannella
>Assignee: Davide Giannella
> Fix For: 1.3.6
>
> Attachments: OAK-3371-2.patch, OAK-3371-test.diff, OAK-3371.patch
>
>
> When executing a query like 
> {noformat}
> SELECT * FROM [nt:unstructured] WHERE ISDESCENDANTNODE([/test]) AND NOT 
> CONTAINS(foo, 'bar')
> {noformat}
> and the {{nodeType}} index plays the not clause is not applied properly. 
> Nodes **with** the property are returned as well.
> [test|^OAK-3371-test.diff] showing the bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3392) Security configurations shouldn't be bound to a SecurityProvider

[Francesco Mari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Francesco Mari updated OAK-3392:

Summary: Security configurations shouldn't be bound to a SecurityProvider  
(was: Security shouldn't be bound to a SecurityProvider)

> Security configurations shouldn't be bound to a SecurityProvider
> 
>
> Key: OAK-3392
> URL: https://issues.apache.org/jira/browse/OAK-3392
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.3.5
>Reporter: Francesco Mari
>Assignee: Francesco Mari
>
> {{ConfigurationBase}}, the base class for security configurations, allows a  
> {{SecurityProvider}} to be injected in a security configuration at will. This 
> mechanism is used to implement a basic form dependency injection in 
> {{SecurityProviderImpl}}, where every configuration receives a the instance 
> of {{SecurityProviderImpl}} that is currently using it.
> This mechanism is problematic in a dynamic scenario like OSGi, where a 
> security configuration can be loaded independently from the 
> {{SecurityProvider}} that is using it. Moreover, if multiple implementations 
> of {{SecurityProvider}} are available, it is impossible to determine which 
> instance of {{SecurityProvider}} will be injected in the security 
> configuration.
> I suggest to remove the reference to a {{SecurityProvider}} in the security 
> configurations. If a {{SecurityProvider}} is needed, it should be instead 
> passed as parameter in the appropriate methods.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-3371) Wrong evaluation of NOT clause

[Davide Giannella (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739070#comment-14739070
 ] 

Davide Giannella edited comment on OAK-3371 at 9/10/15 5:16 PM:


[~chetanm], [~tmueller]: was carrying out some more tests and the
above patch while works with lucene property index, does not work with
lucene aggregate v1. If you want to give a preliminary overview it's
welcome but I'll submit a new one.



was (Author: edivad):
[~chetanm], [~tmueller]: was carrying out some more tests and the
above patch while works with lucene property index, does not work with
lucene aggregate. If you want to give a preliminary overview it's
welcome but I'll submit a new one.


> Wrong evaluation of NOT clause
> --
>
> Key: OAK-3371
> URL: https://issues.apache.org/jira/browse/OAK-3371
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: query
>Affects Versions: 1.2.4, 1.3.5
>Reporter: Davide Giannella
>Assignee: Davide Giannella
> Fix For: 1.3.6
>
> Attachments: OAK-3371-test.diff, OAK-3371.patch
>
>
> When executing a query like 
> {noformat}
> SELECT * FROM [nt:unstructured] WHERE ISDESCENDANTNODE([/test]) AND NOT 
> CONTAINS(foo, 'bar')
> {noformat}
> and the {{nodeType}} index plays the not clause is not applied properly. 
> Nodes **with** the property are returned as well.
> [test|^OAK-3371-test.diff] showing the bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2986) RDB: switch to tomcat datasource implementation

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-2986:

Fix Version/s: (was: 1.0.20)
   (was: 1.2.5)

> RDB: switch to tomcat datasource implementation 
> 
>
> Key: OAK-2986
> URL: https://issues.apache.org/jira/browse/OAK-2986
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: rdbmk
>Affects Versions: 1.2.4, 1.3.5, 1.0.19
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>  Labels: resilience
> Fix For: 1.3.6
>
> Attachments: OAK-2986.diff, OAK-2986.diff
>
>
> See .
> In addition, this is the datasource used in Sling's datasource service, so 
> it's closer to what people will use in practice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-2986) RDB: switch to tomcat datasource implementation

[Julian Reschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738987#comment-14738987
 ] 

Julian Reschke edited comment on OAK-2986 at 9/10/15 4:45 PM:
--

Re-opening because making this a test dependency breaks unit tests in oak-jcr 
-- sorry for the commit noise.

The new plan is to keep RDBDataSourceFactory in main, but make the tomcat pool 
dependency a test dependency, relying on reflection. We can later discuss 
whether the factory should be moved.


was (Author: reschke):
Re-opening because making this a test dependency breaks unit tests in oak-jcr

> RDB: switch to tomcat datasource implementation 
> 
>
> Key: OAK-2986
> URL: https://issues.apache.org/jira/browse/OAK-2986
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: rdbmk
>Affects Versions: 1.2.4, 1.3.5, 1.0.19
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>  Labels: resilience
> Fix For: 1.3.6
>
> Attachments: OAK-2986.diff, OAK-2986.diff
>
>
> See .
> In addition, this is the datasource used in Sling's datasource service, so 
> it's closer to what people will use in practice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3371) Wrong evaluation of NOT clause

[Davide Giannella (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739070#comment-14739070
 ] 

Davide Giannella commented on OAK-3371:
---

[~chetanm], [~tmueller]: was carrying out some more tests and the
above patch while works with lucene property index, does not work with
lucene aggregate. If you want to give a preliminary overview it's
welcome but I'll submit a new one.


> Wrong evaluation of NOT clause
> --
>
> Key: OAK-3371
> URL: https://issues.apache.org/jira/browse/OAK-3371
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: query
>Affects Versions: 1.2.4, 1.3.5
>Reporter: Davide Giannella
>Assignee: Davide Giannella
> Fix For: 1.3.6
>
> Attachments: OAK-3371-test.diff, OAK-3371.patch
>
>
> When executing a query like 
> {noformat}
> SELECT * FROM [nt:unstructured] WHERE ISDESCENDANTNODE([/test]) AND NOT 
> CONTAINS(foo, 'bar')
> {noformat}
> and the {{nodeType}} index plays the not clause is not applied properly. 
> Nodes **with** the property are returned as well.
> [test|^OAK-3371-test.diff] showing the bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (OAK-2986) RDB: switch to tomcat datasource implementation

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke reopened OAK-2986:
-

Re-opening because making this a test dependency breaks unit tests in oak-jcr

> RDB: switch to tomcat datasource implementation 
> 
>
> Key: OAK-2986
> URL: https://issues.apache.org/jira/browse/OAK-2986
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: rdbmk
>Affects Versions: 1.2.4, 1.3.5, 1.0.19
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>  Labels: resilience
> Fix For: 1.0.20, 1.3.6, 1.2.5
>
> Attachments: OAK-2986.diff, OAK-2986.diff
>
>
> See .
> In addition, this is the datasource used in Sling's datasource service, so 
> it's closer to what people will use in practice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3390) Avoid instanceof check in DocumentNodeStore

[Julian Reschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738974#comment-14738974
 ] 

Julian Reschke commented on OAK-3390:
-

Proposed interface:

{code}
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *  http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.jackrabbit.oak.commons;

/**
 * Identifies objects which can be adapted to other types or representations of 
the same object.
 * 
 * Inspired by Apache Sling's Adaptable and Java SQL's Wrapper 
interfaces.
 */
public interface Adaptable {

/**
 * Checks whether the adaptable can be converted to the specified type.
 * 
 * @param type
 *The generic type to check for
 * @return {@code true} if a call to {link {@link #adaptTo(Class)}} is
 * likely to succeed
 */
public boolean isAdaptable(Class type);

/**
 * Adapts the adaptable to another type.
 * 
 * Please note that it is explicitly left as an implementation detail
 * whether each call to this method with the same type yields the same
 * object or a new object on each call.
 * 
 * Implementations of this method should document their adapted types as
 * well as their behavior with respect to returning newly created or not
 * instance on each call.
 * 
 * @param 
 *The generic type to which this object is adapted to
 * @param type
 *The generic type to which this object is adapted to
 * @return The adapter target or {@code null} if the object cannot adapt to
 * the requested type
 */
public  AdapterType adaptTo(Class type);
}
{code}

> Avoid instanceof check in DocumentNodeStore
> ---
>
> Key: OAK-3390
> URL: https://issues.apache.org/jira/browse/OAK-3390
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, mongomk
>Affects Versions: 1.3.4
>Reporter: Marcel Reutegger
> Fix For: 1.3.7
>
>
> The instanceof MongoDocumentStore check does not work anymore when the store 
> is wrapped with e.g. the LeaseCheckDocumentStoreWrapper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3391) RDBBlobStore: speed up testBigBlob(), also improve memory usage

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-3391:

Fix Version/s: 1.2.5
   1.3.6
   1.0.20

> RDBBlobStore: speed up testBigBlob(), also improve memory usage
> ---
>
> Key: OAK-3391
> URL: https://issues.apache.org/jira/browse/OAK-3391
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.4, 1.3.5, 1.0.19
>Reporter: Julian Reschke
>Assignee: Julian Reschke
> Fix For: 1.0.20, 1.3.6, 1.2.5
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-2986) RDB: switch to tomcat datasource implementation

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke resolved OAK-2986.
-
Resolution: Fixed

> RDB: switch to tomcat datasource implementation 
> 
>
> Key: OAK-2986
> URL: https://issues.apache.org/jira/browse/OAK-2986
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: rdbmk
>Affects Versions: 1.2.4, 1.3.5, 1.0.19
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>  Labels: resilience
> Fix For: 1.0.20, 1.3.6, 1.2.5
>
> Attachments: OAK-2986.diff, OAK-2986.diff
>
>
> See .
> In addition, this is the datasource used in Sling's datasource service, so 
> it's closer to what people will use in practice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2986) RDB: switch to tomcat datasource implementation

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-2986:

Fix Version/s: 1.0.20

> RDB: switch to tomcat datasource implementation 
> 
>
> Key: OAK-2986
> URL: https://issues.apache.org/jira/browse/OAK-2986
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: rdbmk
>Affects Versions: 1.2.4, 1.3.5, 1.0.19
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>  Labels: resilience
> Fix For: 1.0.20, 1.3.6, 1.2.5
>
> Attachments: OAK-2986.diff, OAK-2986.diff
>
>
> See .
> In addition, this is the datasource used in Sling's datasource service, so 
> it's closer to what people will use in practice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3392) Security shouldn't be bound to a SecurityProvider

[Francesco Mari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1473#comment-1473
 ] 

Francesco Mari commented on OAK-3392:
-

This issue blocks OAK-3201. The new proposed approach for the 
{{SecurityProviderImpl}}, i.e. don't register the component unless every 
required dependency is loaded, implies that security configurations may exist 
without an implementation of {{SecurityProvider}} already registered in the 
framework.

> Security shouldn't be bound to a SecurityProvider
> -
>
> Key: OAK-3392
> URL: https://issues.apache.org/jira/browse/OAK-3392
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.3.5
>Reporter: Francesco Mari
>Assignee: Francesco Mari
>
> {{ConfigurationBase}}, the base class for security configurations, allows a  
> {{SecurityProvider}} to be injected in a security configuration at will. This 
> mechanism is used to implement a basic form dependency injection in 
> {{SecurityProviderImpl}}, where every configuration receives a the instance 
> of {{SecurityProviderImpl}} that is currently using it.
> This mechanism is problematic in a dynamic scenario like OSGi, where a 
> security configuration can be loaded independently from the 
> {{SecurityProvider}} that is using it. Moreover, if multiple implementations 
> of {{SecurityProvider}} are available, it is impossible to determine which 
> instance of {{SecurityProvider}} will be injected in the security 
> configuration.
> I suggest to remove the reference to a {{SecurityProvider}} in the security 
> configurations. If a {{SecurityProvider}} is needed, it should be instead 
> passed as parameter in the appropriate methods.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-3392) Security shouldn't be bound to a SecurityProvider

[Francesco Mari (JIRA)]

Francesco Mari created OAK-3392:
---

 Summary: Security shouldn't be bound to a SecurityProvider
 Key: OAK-3392
 URL: https://issues.apache.org/jira/browse/OAK-3392
 Project: Jackrabbit Oak
  Issue Type: Improvement
  Components: core
Affects Versions: 1.3.5
Reporter: Francesco Mari
Assignee: Francesco Mari


{{ConfigurationBase}}, the base class for security configurations, allows a  
{{SecurityProvider}} to be injected in a security configuration at will. This 
mechanism is used to implement a basic form dependency injection in 
{{SecurityProviderImpl}}, where every configuration receives a the instance of 
{{SecurityProviderImpl}} that is currently using it.

This mechanism is problematic in a dynamic scenario like OSGi, where a security 
configuration can be loaded independently from the {{SecurityProvider}} that is 
using it. Moreover, if multiple implementations of {{SecurityProvider}} are 
available, it is impossible to determine which instance of {{SecurityProvider}} 
will be injected in the security configuration.

I suggest to remove the reference to a {{SecurityProvider}} in the security 
configurations. If a {{SecurityProvider}} is needed, it should be instead 
passed as parameter in the appropriate methods.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3371) Wrong evaluation of NOT clause

[Davide Giannella (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davide Giannella updated OAK-3371:
--
Attachment: OAK-3371.patch

[~chetanm], [~tmueller] could you please review the [attached 
patch|^OAK-3371.patch] that solves the issue. Ideally I'd like to resolve it 
for 1.3.6.

> Wrong evaluation of NOT clause
> --
>
> Key: OAK-3371
> URL: https://issues.apache.org/jira/browse/OAK-3371
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: query
>Affects Versions: 1.2.4, 1.3.5
>Reporter: Davide Giannella
>Assignee: Davide Giannella
> Fix For: 1.3.6
>
> Attachments: OAK-3371-test.diff, OAK-3371.patch
>
>
> When executing a query like 
> {noformat}
> SELECT * FROM [nt:unstructured] WHERE ISDESCENDANTNODE([/test]) AND NOT 
> CONTAINS(foo, 'bar')
> {noformat}
> and the {{nodeType}} index plays the not clause is not applied properly. 
> Nodes **with** the property are returned as well.
> [test|^OAK-3371-test.diff] showing the bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3391) RDBBlobStore: speed up testBigBlob(), also improve memory usage

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-3391:

Summary: RDBBlobStore: speed up testBigBlob(), also improve memory usage  
(was: RDBBlobStore: speed up testBigBlob(), also irmprove memory useage)

> RDBBlobStore: speed up testBigBlob(), also improve memory usage
> ---
>
> Key: OAK-3391
> URL: https://issues.apache.org/jira/browse/OAK-3391
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: rdbmk
>Affects Versions: 1.2.4, 1.3.5, 1.0.19
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-3391) RDBBlobStore: speed up testBigBlob(), also irmprove memory useage

[Julian Reschke (JIRA)]

Julian Reschke created OAK-3391:
---

 Summary: RDBBlobStore: speed up testBigBlob(), also irmprove 
memory useage
 Key: OAK-3391
 URL: https://issues.apache.org/jira/browse/OAK-3391
 Project: Jackrabbit Oak
  Issue Type: Technical task
  Components: rdbmk
Affects Versions: 1.0.19, 1.3.5, 1.2.4
Reporter: Julian Reschke
Assignee: Julian Reschke






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2986) RDB: switch to tomcat datasource implementation

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-2986:

Fix Version/s: 1.2.5

> RDB: switch to tomcat datasource implementation 
> 
>
> Key: OAK-2986
> URL: https://issues.apache.org/jira/browse/OAK-2986
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: rdbmk
>Affects Versions: 1.2.4, 1.3.5, 1.0.19
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>  Labels: resilience
> Fix For: 1.3.6, 1.2.5
>
> Attachments: OAK-2986.diff, OAK-2986.diff
>
>
> See .
> In addition, this is the datasource used in Sling's datasource service, so 
> it's closer to what people will use in practice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3390) Avoid instanceof check in DocumentNodeStore

[Julian Reschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738814#comment-14738814
 ] 

Julian Reschke commented on OAK-3390:
-

I think we shouldn't conflate the two issues. 1) is making the DS APIs powerful 
enough so that callers do not special-case certain implementations. 2) is the 
problem of wrapper classes and extension interfaces. For 1) we have Thomas' 
proposal (https://issues.apache.org/jira/browse/OAK-3213), for 2) I'd prefer 
something like my proposal above.

> Avoid instanceof check in DocumentNodeStore
> ---
>
> Key: OAK-3390
> URL: https://issues.apache.org/jira/browse/OAK-3390
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, mongomk
>Affects Versions: 1.3.4
>Reporter: Marcel Reutegger
> Fix For: 1.3.7
>
>
> The instanceof MongoDocumentStore check does not work anymore when the store 
> is wrapped with e.g. the LeaseCheckDocumentStoreWrapper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-955) Query: Filter doesn't contain fulltext constraints from joins

[Thomas Mueller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller resolved OAK-955.

Resolution: Won't Fix

> Query: Filter doesn't contain fulltext constraints from joins 
> --
>
> Key: OAK-955
> URL: https://issues.apache.org/jira/browse/OAK-955
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, query
>Reporter: Alex Parvulescu
>Assignee: Thomas Mueller
>Priority: Minor
>  Labels: performance, resilience
> Fix For: 1.3.7
>
>
> Example query:
> {code}
> SELECT a.* 
> FROM [nt:unstructured] AS a 
> INNER JOIN [nt:unstructured] AS b 
> ON b.[jcr:uuid] = a.testref 
> WHERE a.type = 'child' 
> AND (CONTAINS(a.*, 'testJoinWithOR4') OR b.type = 'parent' AND CONTAINS(b.*, 
> 'testJoinWithOR4'))
> {code}
> I'm not sure why this happens, but I noticed stepping through the code that 
> the filter generated on the query doesn't contain any fulltext constraints. 
> It does however contain the 'type' info which will trick the query engine 
> into picking a property index, failing the test because is returns more 
> results than it should.
> See failing tests on the lucene module:
>  - org.apache.jackrabbit.core.query.JoinTest#testJoinWithOR4
>  - org.apache.jackrabbit.core.query.JoinTest#testJoinWithOR5



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-955) Query: Filter doesn't contain fulltext constraints from joins

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738822#comment-14738822
 ] 

Thomas Mueller commented on OAK-955:


I don't see a reasonably easy way to solve this in Oak. A possible workaround 
is to use relative path in constraints, for example: "contains(title, 'x') and 
contains(child/title, 'y')".

> Query: Filter doesn't contain fulltext constraints from joins 
> --
>
> Key: OAK-955
> URL: https://issues.apache.org/jira/browse/OAK-955
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, query
>Reporter: Alex Parvulescu
>Assignee: Thomas Mueller
>Priority: Minor
>  Labels: performance, resilience
> Fix For: 1.3.7
>
>
> Example query:
> {code}
> SELECT a.* 
> FROM [nt:unstructured] AS a 
> INNER JOIN [nt:unstructured] AS b 
> ON b.[jcr:uuid] = a.testref 
> WHERE a.type = 'child' 
> AND (CONTAINS(a.*, 'testJoinWithOR4') OR b.type = 'parent' AND CONTAINS(b.*, 
> 'testJoinWithOR4'))
> {code}
> I'm not sure why this happens, but I noticed stepping through the code that 
> the filter generated on the query doesn't contain any fulltext constraints. 
> It does however contain the 'type' info which will trick the query engine 
> into picking a property index, failing the test because is returns more 
> results than it should.
> See failing tests on the lucene module:
>  - org.apache.jackrabbit.core.query.JoinTest#testJoinWithOR4
>  - org.apache.jackrabbit.core.query.JoinTest#testJoinWithOR5



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-3343) LIRS Cache: investigate concurrency behavior

[Thomas Mueller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller resolved OAK-3343.
-
Resolution: Cannot Reproduce

> LIRS Cache: investigate concurrency behavior
> 
>
> Key: OAK-3343
> URL: https://issues.apache.org/jira/browse/OAK-3343
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: cache, core
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Fix For: 1.3.7
>
>
> I have a report that the LIRS cache (in combination with the persistent 
> cache) does not perform well when using 50 threads that concurrently access 
> it. The Guava cache might not be affected (as much).
> I would like to investigate, with a simple test case, what is the behavior of 
> the LIRS and Guava cache with highly concurrent use cases, in combination 
> with a cache loader.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2037) Define standards for plan output

[Thomas Mueller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-2037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller updated OAK-2037:

Fix Version/s: (was: 1.3.7)
   1.3.9

> Define standards for plan output
> 
>
> Key: OAK-2037
> URL: https://issues.apache.org/jira/browse/OAK-2037
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Reporter: Justin Edelson
>Assignee: Thomas Mueller
>Priority: Minor
>  Labels: tooling
> Fix For: 1.3.9
>
>
> Currently, the syntax for the plan output is chaotic as it varies 
> significantly from index to index. Whereas some of this is expected - each 
> index type will have different data to output, Oak should provide some 
> standards about how a plan will appear.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3343) LIRS Cache: investigate concurrency behavior

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738811#comment-14738811
 ] 

Thomas Mueller commented on OAK-3343:
-

The cache concurrency for the LIRS cache can already be changed using the 
property "cacheSegmentCount" (default 16).

> LIRS Cache: investigate concurrency behavior
> 
>
> Key: OAK-3343
> URL: https://issues.apache.org/jira/browse/OAK-3343
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: cache, core
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Fix For: 1.3.7
>
>
> I have a report that the LIRS cache (in combination with the persistent 
> cache) does not perform well when using 50 threads that concurrently access 
> it. The Guava cache might not be affected (as much).
> I would like to investigate, with a simple test case, what is the behavior of 
> the LIRS and Guava cache with highly concurrent use cases, in combination 
> with a cache loader.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3343) LIRS Cache: investigate concurrency behavior

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738804#comment-14738804
 ] 

Thomas Mueller commented on OAK-3343:
-

http://svn.apache.org/r1702259 (trunk; adding a disabled performance test)
I noticed the Guava cache we use also uses a segment mechanism, same as the 
LIRS cache. My tests show that the LIRS cache is almost always faster than the 
Guava cache (average 30%). The default settings for the LIRS cache is 16 
segments, and for the Guava cache also 16. So I consider this closed.

> LIRS Cache: investigate concurrency behavior
> 
>
> Key: OAK-3343
> URL: https://issues.apache.org/jira/browse/OAK-3343
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: cache, core
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Fix For: 1.3.7
>
>
> I have a report that the LIRS cache (in combination with the persistent 
> cache) does not perform well when using 50 threads that concurrently access 
> it. The Guava cache might not be affected (as much).
> I would like to investigate, with a simple test case, what is the behavior of 
> the LIRS and Guava cache with highly concurrent use cases, in combination 
> with a cache loader.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3390) Avoid instanceof check in DocumentNodeStore

[Marcel Reutegger (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738803#comment-14738803
 ] 

Marcel Reutegger commented on OAK-3390:
---

I would rather move this kind of factory method into the Builder. That way the 
builder can decide what BlobReferenceIterator needs to be created.

> Avoid instanceof check in DocumentNodeStore
> ---
>
> Key: OAK-3390
> URL: https://issues.apache.org/jira/browse/OAK-3390
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, mongomk
>Affects Versions: 1.3.4
>Reporter: Marcel Reutegger
> Fix For: 1.3.7
>
>
> The instanceof MongoDocumentStore check does not work anymore when the store 
> is wrapped with e.g. the LeaseCheckDocumentStoreWrapper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3390) Avoid instanceof check in DocumentNodeStore

[Stefan Egli (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738798#comment-14738798
 ] 

Stefan Egli commented on OAK-3390:
--

or .. we could also demand DocumentStore to provide an adapter for the three 
cases we currently have (MissingLastRevSeeker, BlobReferenceIterator and 
VersionGCSupport) via a {{DocumentStore.adaptTo()}} - we could of course 
generalize this in an interface but so far the use cases are very limited and 
known..

> Avoid instanceof check in DocumentNodeStore
> ---
>
> Key: OAK-3390
> URL: https://issues.apache.org/jira/browse/OAK-3390
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, mongomk
>Affects Versions: 1.3.4
>Reporter: Marcel Reutegger
> Fix For: 1.3.7
>
>
> The instanceof MongoDocumentStore check does not work anymore when the store 
> is wrapped with e.g. the LeaseCheckDocumentStoreWrapper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3390) Avoid instanceof check in DocumentNodeStore

[Julian Reschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738787#comment-14738787
 ] 

Julian Reschke commented on OAK-3390:
-

Recommendation: borrow from both 
http://docs.oracle.com/javase/6/docs/api/java/sql/Wrapper.html and Sling's 
Adaptable.

- checker function canAdaptTo
- adapter function adaptTo, potentially returning null

The reason for having the checker is that sometimes it can be expensive to 
adapt, and the caller just needs to know whether adapting would be possible.

Define interface in oak-commons, and make DocumentStore implement that.


> Avoid instanceof check in DocumentNodeStore
> ---
>
> Key: OAK-3390
> URL: https://issues.apache.org/jira/browse/OAK-3390
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, mongomk
>Affects Versions: 1.3.4
>Reporter: Marcel Reutegger
> Fix For: 1.3.7
>
>
> The instanceof MongoDocumentStore check does not work anymore when the store 
> is wrapped with e.g. the LeaseCheckDocumentStoreWrapper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OAK-3390) Avoid instanceof check in DocumentNodeStore

[Stefan Egli (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738782#comment-14738782
 ] 

Stefan Egli edited comment on OAK-3390 at 9/10/15 2:02 PM:
---

-What about introducing {{DocumentStore.unwrap()}} which would provide the 
underlying documentstore should it be wrapped, or this ?- probably better to 
make something generic as suggested offline by Julian


was (Author: egli):
What about introducing {{DocumentStore.unwrap()}} which would provide the 
underlying documentstore should it be wrapped, or this ?

> Avoid instanceof check in DocumentNodeStore
> ---
>
> Key: OAK-3390
> URL: https://issues.apache.org/jira/browse/OAK-3390
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, mongomk
>Affects Versions: 1.3.4
>Reporter: Marcel Reutegger
> Fix For: 1.3.7
>
>
> The instanceof MongoDocumentStore check does not work anymore when the store 
> is wrapped with e.g. the LeaseCheckDocumentStoreWrapper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3390) Avoid instanceof check in DocumentNodeStore

[Stefan Egli (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738782#comment-14738782
 ] 

Stefan Egli commented on OAK-3390:
--

What about introducing {{DocumentStore.unwrap()}} which would provide the 
underlying documentstore should it be wrapped, or this ?

> Avoid instanceof check in DocumentNodeStore
> ---
>
> Key: OAK-3390
> URL: https://issues.apache.org/jira/browse/OAK-3390
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, mongomk
>Affects Versions: 1.3.4
>Reporter: Marcel Reutegger
> Fix For: 1.3.7
>
>
> The instanceof MongoDocumentStore check does not work anymore when the store 
> is wrapped with e.g. the LeaseCheckDocumentStoreWrapper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-2739) take appropriate action when lease cannot be renewed (in time)

[Marcel Reutegger (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738769#comment-14738769
 ] 

Marcel Reutegger commented on OAK-2739:
---

Good point. Created a new issue: OAK-3390.

> take appropriate action when lease cannot be renewed (in time)
> --
>
> Key: OAK-2739
> URL: https://issues.apache.org/jira/browse/OAK-2739
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: core
>Affects Versions: 1.2
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>  Labels: resilience
> Fix For: 1.3.4
>
> Attachments: OAK-2739.patch
>
>
> Currently, in an oak-cluster when (e.g.) one oak-client stops renewing its 
> lease (ClusterNodeInfo.renewLease()), this will be eventually noticed by the 
> others in the same oak-cluster. Those then mark this client as {{inactive}} 
> and start recoverying and subsequently removing that node from any further 
> merge etc operation.
> Now, whatever the reason was why that client stopped renewing the lease 
> (could be an exception, deadlock, whatever) - that client itself still 
> considers itself as {{active}} and continues to take part in the cluster 
> action.
> This will result in a unbalanced situation where that one client 'sees' 
> everybody as {{active}} while the others see this one as {{inactive}}.
> If this ClusterNodeInfo state should be something that can be built upon, and 
> to avoid any inconsistency due to unbalanced handling, the inactive node 
> should probably retire gracefully - or any other appropriate action should be 
> taken, other than just continuing as today.
> This ticket is to keep track of ideas and actions taken wrt this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-3390) Avoid instanceof check in DocumentNodeStore

[Marcel Reutegger (JIRA)]

Marcel Reutegger created OAK-3390:
-

 Summary: Avoid instanceof check in DocumentNodeStore
 Key: OAK-3390
 URL: https://issues.apache.org/jira/browse/OAK-3390
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: core, mongomk
Affects Versions: 1.3.4
Reporter: Marcel Reutegger
 Fix For: 1.3.7


The instanceof MongoDocumentStore check does not work anymore when the store is 
wrapped with e.g. the LeaseCheckDocumentStoreWrapper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-2739) take appropriate action when lease cannot be renewed (in time)

[Julian Reschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738716#comment-14738716
 ] 

Julian Reschke commented on OAK-2739:
-

[~mreutegg] [~egli] [~chetanm] unless I'm missing something, the concept of 
using a DS wrapper will cause the "instancof MongoDocumentStore" in the 
DocumentNodeStore not to function properly anymore!

> take appropriate action when lease cannot be renewed (in time)
> --
>
> Key: OAK-2739
> URL: https://issues.apache.org/jira/browse/OAK-2739
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: core
>Affects Versions: 1.2
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>  Labels: resilience
> Fix For: 1.3.4
>
> Attachments: OAK-2739.patch
>
>
> Currently, in an oak-cluster when (e.g.) one oak-client stops renewing its 
> lease (ClusterNodeInfo.renewLease()), this will be eventually noticed by the 
> others in the same oak-cluster. Those then mark this client as {{inactive}} 
> and start recoverying and subsequently removing that node from any further 
> merge etc operation.
> Now, whatever the reason was why that client stopped renewing the lease 
> (could be an exception, deadlock, whatever) - that client itself still 
> considers itself as {{active}} and continues to take part in the cluster 
> action.
> This will result in a unbalanced situation where that one client 'sees' 
> everybody as {{active}} while the others see this one as {{inactive}}.
> If this ClusterNodeInfo state should be something that can be built upon, and 
> to avoid any inconsistency due to unbalanced handling, the inactive node 
> should probably retire gracefully - or any other appropriate action should be 
> taken, other than just continuing as today.
> This ticket is to keep track of ideas and actions taken wrt this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3388) Inconsistent read in cluster with clock differences

[Marcel Reutegger (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738709#comment-14738709
 ] 

Marcel Reutegger commented on OAK-3388:
---

Add a test (currently ignored): http://svn.apache.org/r1702241

> Inconsistent read in cluster with clock differences
> ---
>
> Key: OAK-3388
> URL: https://issues.apache.org/jira/browse/OAK-3388
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, mongomk
>Affects Versions: 1.0, 1.2
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
> Fix For: 1.3.7
>
>
> This issue is similar to OAK-2929 but related to how the DocumentNodeStore 
> reads a node state when there is a clock difference between multiple cluster 
> nodes. The node state read from a NodeDocument may not be correct when there 
> is a clock difference.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-3281) Test failures on trunk: SolrIndexQueryTestIT.sql2

[Thomas Mueller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller resolved OAK-3281.
-
Resolution: Fixed

> Test failures on trunk: SolrIndexQueryTestIT.sql2
> -
>
> Key: OAK-3281
> URL: https://issues.apache.org/jira/browse/OAK-3281
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: lucene
> Environment: 
> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/
>Reporter: Michael Dürig
>Assignee: Thomas Mueller
>  Labels: ci, jenkins
> Fix For: 1.0.20, 1.3.6, 1.2.5
>
>
> {{org.apache.jackrabbit.oak.plugins.index.solr.query.SolrIndexQueryTestIT.sql2}}
>  fails regularly on Jenkins: 
> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/350/jdk=jdk1.8.0_11,label=Ubuntu,nsfixtures=DOCUMENT_RDB,profile=integrationTesting/testReport/junit/org.apache.jackrabbit.oak.plugins.index.solr.query/SolrIndexQueryTestIT/sql2/
> {noformat}
> java.lang.Exception: Results in 
> target/oajopi.solr.query.SolrIndexQueryTestIT_sql2.txt don't match expected 
> results in 
> file:/x1/jenkins/jenkins-slave/workspace/Apache%20Jackrabbit%20Oak%20matrix/jdk/jdk1.8.0_11/label/Ubuntu/nsfixtures/DOCUMENT_RDB/profile/integrationTesting/oak-core/target/oak-core-1.4-SNAPSHOT-tests.jar!/org/apache/jackrabbit/oak/query/sql2.txt;
>  compare the files for details
>   at 
> org.apache.jackrabbit.oak.query.AbstractQueryTest.test(AbstractQueryTest.java:232)
>   at 
> org.apache.jackrabbit.oak.plugins.index.solr.query.SolrIndexQueryTestIT.sql2(SolrIndexQueryTestIT.java:91)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:18)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:483)
>   at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3158) IAE when specifiying 2G cache for FileStore

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738704#comment-14738704
 ] 

Thomas Mueller commented on OAK-3158:
-

http://svn.apache.org/r1702240 (trunk) - no longer limited to 2 GB. The main 
problems was that "maximumSize" was called instead of "maximumWeight".

> IAE when specifiying 2G cache for FileStore
> ---
>
> Key: OAK-3158
> URL: https://issues.apache.org/jira/browse/OAK-3158
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Thomas Mueller
>Priority: Minor
>  Labels: resilience
> Fix For: 1.3.6
>
>
> {{FileStore.newFileStore(dir).withCacheSize(2048)}} results in
> {noformat}Max memory must not be negative
> java.lang.IllegalArgumentException: Max memory must not be negative
>   at 
> org.apache.jackrabbit.oak.cache.CacheLIRS.setMaxMemory(CacheLIRS.java:464)
>   at org.apache.jackrabbit.oak.cache.CacheLIRS.(CacheLIRS.java:163)
>   at 
> org.apache.jackrabbit.oak.cache.CacheLIRS$Builder.build(CacheLIRS.java:1537)
>   at 
> org.apache.jackrabbit.oak.cache.CacheLIRS$Builder.build(CacheLIRS.java:1533)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.StringCache.(StringCache.java:52)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.(SegmentTracker.java:126)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.(FileStore.java:343)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.(FileStore.java:84)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore$Builder.create(FileStore.java:294)
> {noformat}
> There is an integer overflow cause by using ints instead of longs to specify 
> the cache size.
> [~tmueller], could you have a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-3158) IAE when specifiying 2G cache for FileStore

[Thomas Mueller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller resolved OAK-3158.
-
Resolution: Fixed

> IAE when specifiying 2G cache for FileStore
> ---
>
> Key: OAK-3158
> URL: https://issues.apache.org/jira/browse/OAK-3158
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Thomas Mueller
>Priority: Minor
>  Labels: resilience
> Fix For: 1.3.6
>
>
> {{FileStore.newFileStore(dir).withCacheSize(2048)}} results in
> {noformat}Max memory must not be negative
> java.lang.IllegalArgumentException: Max memory must not be negative
>   at 
> org.apache.jackrabbit.oak.cache.CacheLIRS.setMaxMemory(CacheLIRS.java:464)
>   at org.apache.jackrabbit.oak.cache.CacheLIRS.(CacheLIRS.java:163)
>   at 
> org.apache.jackrabbit.oak.cache.CacheLIRS$Builder.build(CacheLIRS.java:1537)
>   at 
> org.apache.jackrabbit.oak.cache.CacheLIRS$Builder.build(CacheLIRS.java:1533)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.StringCache.(StringCache.java:52)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.(SegmentTracker.java:126)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.(FileStore.java:343)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.(FileStore.java:84)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore$Builder.create(FileStore.java:294)
> {noformat}
> There is an integer overflow cause by using ints instead of longs to specify 
> the cache size.
> [~tmueller], could you have a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-3389) oak-run check for dropped tables broken

[Julian Reschke (JIRA)]

Julian Reschke created OAK-3389:
---

 Summary: oak-run check for dropped tables broken
 Key: OAK-3389
 URL: https://issues.apache.org/jira/browse/OAK-3389
 Project: Jackrabbit Oak
  Issue Type: Technical task
  Components: rdbmk, run
Affects Versions: 1.3.5
Reporter: Julian Reschke


@Override
public void tearDownCluster() {
String dropped = "";
for (DocumentMK kernel : kernels) {
kernel.dispose();
if (kernel.getDocumentStore() instanceof RDBDocumentStore) {
dropped += 
((RDBDocumentStore)kernel.getDocumentStore()).getDroppedTables();
}
}
if (dropDBAfterTest) {
if(blobStoreFixture != null){
blobStoreFixture.tearDown();
}

if (dropped.isEmpty()) {
throw new RuntimeException("dropdb was set, but tables 
have not been dropped");
}
}
}

has been broken because the DS is now wrapped by 
LeaseCheckDocumentStoreWrapper. Figure out a different way to obtain the 
information.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OAK-3388) Inconsistent read in cluster with clock differences

[Marcel Reutegger (JIRA)]

Marcel Reutegger created OAK-3388:
-

 Summary: Inconsistent read in cluster with clock differences
 Key: OAK-3388
 URL: https://issues.apache.org/jira/browse/OAK-3388
 Project: Jackrabbit Oak
  Issue Type: Bug
  Components: core, mongomk
Affects Versions: 1.2, 1.0
Reporter: Marcel Reutegger
Assignee: Marcel Reutegger
 Fix For: 1.3.7


This issue is similar to OAK-2929 but related to how the DocumentNodeStore 
reads a node state when there is a clock difference between multiple cluster 
nodes. The node state read from a NodeDocument may not be correct when there is 
a clock difference.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3348) Cross gc sessions might introduce references to pre-compacted segments

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-3348:
---
Attachment: OAK-3348-1.patch

[^OAK-3348-1.patch]: updated patch. This one passes all ITs. However there are 
a couple of FIXMEs left to address before it can be applied as this patch hack 
itself through some layers of encapsulations.

> Cross gc sessions might introduce references to pre-compacted segments
> --
>
> Key: OAK-3348
> URL: https://issues.apache.org/jira/browse/OAK-3348
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: cleanup, compaction, gc
> Fix For: 1.3.7
>
> Attachments: OAK-3348-1.patch, OAK-3348.patch, image.png
>
>
> I suspect that certain write operations during compaction can cause 
> references from compacted segments to pre-compacted ones. This would 
> effectively prevent the pre-compacted segments from getting evicted in 
> subsequent cleanup phases. 
> The scenario is as follows:
> * A session is opened and a lot of content is written to it such that the 
> update limit is exceeded. This causes the changes to be written to disk. 
> * Revision gc runs causing a new, compacted root node state to be written to 
> disk.
> * The session saves its changes. This causes rebasing of its changes onto the 
> current root (the compacted one). At this point any node that has been added 
> will be added again in the sub-tree rooted at the current root. Such nodes 
> however might have been written to disk *before* revision gc ran and might 
> thus be contained in pre-compacted segments. As I suspect the node-add 
> operation in the rebasing process *not* to create a deep copy of such nodes 
> but to rather create a *reference* to them, a reference to a pre-compacted 
> segment is introduced here. 
> Going forward we need to validate above hypothesis, assess its impact if 
> necessary come up with a solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3386) ConcurrentAddNodesClusterIT.addNodesConcurrent() blocks occasionally

[Marcel Reutegger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcel Reutegger updated OAK-3386:
--
Affects Version/s: (was: 1.3.6)

> ConcurrentAddNodesClusterIT.addNodesConcurrent() blocks occasionally
> 
>
> Key: OAK-3386
> URL: https://issues.apache.org/jira/browse/OAK-3386
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, mongomk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.3.6
>
>
> The test ConcurrentAddNodesClusterIT.addNodesConcurrent() blocks occasionally 
> and travis stops the build after 10 minutes.
> This is most likely caused by recent changes done in OAK-3042. In the test it 
> may happen that both workers mark the other commit with a collision marker 
> and then wait for each other.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-3386) ConcurrentAddNodesClusterIT.addNodesConcurrent() blocks occasionally

[Marcel Reutegger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcel Reutegger resolved OAK-3386.
---
   Resolution: Fixed
Fix Version/s: 1.3.6

Fixed in trunk: http://svn.apache.org/r1702226

> ConcurrentAddNodesClusterIT.addNodesConcurrent() blocks occasionally
> 
>
> Key: OAK-3386
> URL: https://issues.apache.org/jira/browse/OAK-3386
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core, mongomk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.3.6
>
>
> The test ConcurrentAddNodesClusterIT.addNodesConcurrent() blocks occasionally 
> and travis stops the build after 10 minutes.
> This is most likely caused by recent changes done in OAK-3042. In the test it 
> may happen that both workers mark the other commit with a collision marker 
> and then wait for each other.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3380) Property index pruning should happen asynchronously

[Vikas Saurabh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738603#comment-14738603
 ] 

Vikas Saurabh commented on OAK-3380:


{quote}
If there are multiple cluster nodes, then which one would do the pruning? How 
would the others inform the one that does the pruning?
We could do some kind of "garbage collection" process, but that would take some 
time as it would have to process the whole index.
{quote}
I thought that we can think along the lines of async indexers (well pruner in 
this case) which would hence run only on leader nodes. SInce, the diff gets 
calculated anyway for other async indexers, this one could just look for nodes 
being removed under property index trees and then inspect those paths for 
further pruning).
Having a marker node like {:trash} could also be an interesting idea - I'm 
assuming if the pruner finds a trash node but it has got some siblings by the 
time it runs it just removes the trash node.

{quote}
 instead of pruning while updating the index, we might be able to prune while 
reading from the index. I'm not sure if that's really a good idea (write while 
reading), but anyway: when traversing a empty node, remove it with a 
probability of let's say 5% 
{quote}
While I don't have very strong feelings towards not writing while reading 
(specially when the structure is oak's implementation detail and is not exposed 
at JCR) - but I feel it removes a bit of determinism which I feel uncomfortable 
with.

> Property index pruning should happen asynchronously
> ---
>
> Key: OAK-3380
> URL: https://issues.apache.org/jira/browse/OAK-3380
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Affects Versions: 1.3.5
>Reporter: Vikas Saurabh
>Priority: Minor
>  Labels: resilience
>
> Following up on this (a relatively old) thread \[1], we should do pruning of 
> property index structure asynchronously. The thread was never concluded.. 
> here are a couple of ideas picked from the thread:
> * Move pruning to an async thread
> * Throttle pruning i.e. prune only once in a while
> ** I'm not sure how that would work though -- an unpruned part would remain 
> as is until another index happens on that path.
> Once we can move pruning to some async thread (reducing concurrent updates), 
> OAK-2673 + OAK-2929 can take care of add-add conflicts.
> 
> h6. Why is this an issue despite merge retries taking care of it?
> A couple of cases which have concurrent updates hitting merge conflicts in 
> our product (Adobe AEM):
> * Some index are very volatile (in the sense that indexed property switches 
> its values very quickly) e.g. sling job status, AEM workflow status.
> * Multiple threads take care of jobs. Although sling maintains a bucketed 
> structure for job storage to reduce conflicts... but inside index tree the 
> bucket structure, at times, gets pruned and needs to be created in the next 
> job status change
> While retries do take care of these conflict a lot of times and even when 
> they don't, AEM workflows has it's own retry to work around. But, retrying, 
> IMHO, is just a waste of time -- more importantly in paths where application 
> doesn't really have a control.
> h6. Would this add to cost of traversing index structure?
> Yes, there'd be some left over paths in index structure between asynchronous 
> prunes. But, I think the cost of such wasted traversals would be covered up 
> with time saved in avoiding the concurrent update conflict.
> 
> (cc [~tmueller], [~mreutegg], [~alex.parvulescu], [~chetanm])
> \[1]: 
> http://mail-archives.apache.org/mod_mbox/jackrabbit-oak-dev/201506.mbox/%3ccadichf66u2vh-hlrjunansytxfidj2mt3vktr4ybkngpzy9...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2986) RDB: switch to tomcat datasource implementation

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-2986:

Affects Version/s: (was: 1.3.4)
   1.3.5

> RDB: switch to tomcat datasource implementation 
> 
>
> Key: OAK-2986
> URL: https://issues.apache.org/jira/browse/OAK-2986
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: rdbmk
>Affects Versions: 1.2.4, 1.3.5, 1.0.19
>Reporter: Julian Reschke
>Assignee: Julian Reschke
>  Labels: resilience
> Fix For: 1.3.6
>
> Attachments: OAK-2986.diff, OAK-2986.diff
>
>
> See .
> In addition, this is the datasource used in Sling's datasource service, so 
> it's closer to what people will use in practice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (OAK-3265) Test failures: NodeLocalNameTest, NodeNameTest

[Thomas Mueller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller resolved OAK-3265.
-
Resolution: Fixed

> Test failures: NodeLocalNameTest, NodeNameTest
> --
>
> Key: OAK-3265
> URL: https://issues.apache.org/jira/browse/OAK-3265
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: jcr
>Affects Versions: 1.3.5, 1.0.20, 1.2.5
>Reporter: Michael Dürig
>Assignee: Thomas Mueller
> Fix For: 1.0.20, 1.3.6, 1.2.5
>
>
> Trunk's it fail for me:
> {noformat}
> testStringLiteralInvalidName(org.apache.jackrabbit.test.api.query.qom.NodeLocalNameTest)
>   Time elapsed: 0.007 sec  <<< ERROR!
> javax.jcr.query.InvalidQueryException: java.lang.IllegalArgumentException: 
> Not a valid JCR path: [node1
>   at 
> org.apache.jackrabbit.oak.jcr.query.QueryManagerImpl.executeQuery(QueryManagerImpl.java:142)
>   at 
> org.apache.jackrabbit.oak.jcr.query.qom.QueryObjectModelImpl.execute(QueryObjectModelImpl.java:131)
>   at 
> org.apache.jackrabbit.test.api.query.qom.NodeLocalNameTest.testStringLiteralInvalidName(NodeLocalNameTest.java:68)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at junit.framework.TestCase.runTest(TestCase.java:168)
>   at junit.framework.TestCase.runBare(TestCase.java:134)
>   at junit.framework.TestResult$1.protect(TestResult.java:110)
>   at junit.framework.TestResult.runProtected(TestResult.java:128)
>   at junit.framework.TestResult.run(TestResult.java:113)
>   at junit.framework.TestCase.run(TestCase.java:124)
>   at 
> org.apache.jackrabbit.test.AbstractJCRTest.run(AbstractJCRTest.java:464)
>   at junit.framework.TestSuite.runTest(TestSuite.java:243)
>   at junit.framework.TestSuite.run(TestSuite.java:238)
>   at junit.framework.TestSuite.runTest(TestSuite.java:243)
>   at junit.framework.TestSuite.run(TestSuite.java:238)
>   at junit.framework.TestSuite.runTest(TestSuite.java:243)
>   at junit.framework.TestSuite.run(TestSuite.java:238)
>   at 
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Caused by: java.lang.IllegalArgumentException: Not a valid JCR path: [node1
>   at 
> org.apache.jackrabbit.oak.spi.query.PropertyValues.getOakPath(PropertyValues.java:405)
>   at 
> org.apache.jackrabbit.oak.query.ast.NodeNameImpl.getName(NodeNameImpl.java:131)
>   at 
> org.apache.jackrabbit.oak.query.ast.NodeLocalNameImpl.restrict(NodeLocalNameImpl.java:89)
>   at 
> org.apache.jackrabbit.oak.query.ast.ComparisonImpl.restrict(ComparisonImpl.java:184)
>   at 
> org.apache.jackrabbit.oak.query.ast.AndImpl.restrict(AndImpl.java:153)
>   at 
> org.apache.jackrabbit.oak.query.ast.SelectorImpl.createFilter(SelectorImpl.java:389)
>   at 
> org.apache.jackrabbit.oak.query.ast.SelectorImpl.prepare(SelectorImpl.java:284)
>   at org.apache.jackrabbit.oak.query.QueryImpl.prepare(QueryImpl.java:591)
>   at 
> org.apache.jackrabbit.oak.query.QueryEngineImpl.executeQuery(QueryEngineImpl.java:193)
>   at 
> org.apache.jackrabbit.oak.jcr.query.QueryManagerImpl.executeQuery(QueryManagerImpl.java:132)
>   ... 32 more
> testURILiteral(org.apache.jackrabbit.test.api.query.qom.NodeLocalNameTest)  
> Time elapsed: 0.005 sec  <<< ERROR!
> javax.jcr.query.InvalidQueryException: java.lang.IllegalArgumentException: 
> Not a valid JCR path: http://example.c

[jira] [Created] (OAK-3387) Enable NodeLocalNameTest tests

[Thomas Mueller (JIRA)]

Thomas Mueller created OAK-3387:
---

 Summary: Enable NodeLocalNameTest tests
 Key: OAK-3387
 URL: https://issues.apache.org/jira/browse/OAK-3387
 Project: Jackrabbit Oak
  Issue Type: Test
  Components: jcr
Reporter: Thomas Mueller
Assignee: Thomas Mueller
 Fix For: 1.4


Enable the tests that were disabled in OAK-3265, once Jackrabbit is released.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3265) Test failures: NodeLocalNameTest, NodeNameTest

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738525#comment-14738525
 ] 

Thomas Mueller commented on OAK-3265:
-

http://svn.apache.org/r1702201 (jackrabbit-jcr-tests and jackrabbit-core) fixed 
the test cases, and marked as "known issue" in jackrabbit-core

Will now wait for the next Jackrabbit release, so that the tests can be removed 
from the "known issue" list in oak-jcr, as follows:

{noformat}
Index: pom.xml
===
--- pom.xml (revision 1701913)
+++ pom.xml (working copy)
@@ -105,9 +105,6 @@
   org.apache.jackrabbit.test.api.query.SQLJoinTest#testJoinNtBase  
  
   
org.apache.jackrabbit.test.api.query.SQLJoinTest#testJoinFilterPrimaryType  
   
   org.apache.jackrabbit.test.api.query.SQLJoinTest#testJoinSNS 
  
-  
org.apache.jackrabbit.test.api.query.qom.NodeLocalNameTest#testStringLiteralInvalidName

-  
org.apache.jackrabbit.test.api.query.qom.NodeLocalNameTest#testPathLiteral  
   
-  
org.apache.jackrabbit.test.api.query.qom.NodeLocalNameTest#testURILiteral   
   
 
   org.apache.jackrabbit.core.query.ExcerptTest#testMoreTextDotsAtEnd   
  
   org.apache.jackrabbit.core.query.ExcerptTest#testMoreTextDotsAtStart 
  
{noformat}

> Test failures: NodeLocalNameTest, NodeNameTest
> --
>
> Key: OAK-3265
> URL: https://issues.apache.org/jira/browse/OAK-3265
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: jcr
>Affects Versions: 1.3.5, 1.0.20, 1.2.5
>Reporter: Michael Dürig
>Assignee: Thomas Mueller
> Fix For: 1.0.20, 1.3.6, 1.2.5
>
>
> Trunk's it fail for me:
> {noformat}
> testStringLiteralInvalidName(org.apache.jackrabbit.test.api.query.qom.NodeLocalNameTest)
>   Time elapsed: 0.007 sec  <<< ERROR!
> javax.jcr.query.InvalidQueryException: java.lang.IllegalArgumentException: 
> Not a valid JCR path: [node1
>   at 
> org.apache.jackrabbit.oak.jcr.query.QueryManagerImpl.executeQuery(QueryManagerImpl.java:142)
>   at 
> org.apache.jackrabbit.oak.jcr.query.qom.QueryObjectModelImpl.execute(QueryObjectModelImpl.java:131)
>   at 
> org.apache.jackrabbit.test.api.query.qom.NodeLocalNameTest.testStringLiteralInvalidName(NodeLocalNameTest.java:68)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at junit.framework.TestCase.runTest(TestCase.java:168)
>   at junit.framework.TestCase.runBare(TestCase.java:134)
>   at junit.framework.TestResult$1.protect(TestResult.java:110)
>   at junit.framework.TestResult.runProtected(TestResult.java:128)
>   at junit.framework.TestResult.run(TestResult.java:113)
>   at junit.framework.TestCase.run(TestCase.java:124)
>   at 
> org.apache.jackrabbit.test.AbstractJCRTest.run(AbstractJCRTest.java:464)
>   at junit.framework.TestSuite.runTest(TestSuite.java:243)
>   at junit.framework.TestSuite.run(TestSuite.java:238)
>   at junit.framework.TestSuite.runTest(TestSuite.java:243)
>   at junit.framework.TestSuite.run(TestSuite.java:238)
>   at junit.framework.TestSuite.runTest(TestSuite.java:243)
>   at junit.framework.TestSuite.run(TestSuite.java:238)
>   at 
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
>   at 
> org.apache.maven.surefire.boo

[jira] [Resolved] (OAK-3377) Two spaces in SQL2 fulltext search -> error

[Thomas Mueller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Mueller resolved OAK-3377.
-
Resolution: Fixed

> Two spaces in SQL2 fulltext search -> error
> ---
>
> Key: OAK-3377
> URL: https://issues.apache.org/jira/browse/OAK-3377
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: query
>Affects Versions: 1.0.19
>Reporter: Laurie byrum
>Assignee: Thomas Mueller
> Fix For: 1.3.6
>
>
> Execute the following SQL2 query (eg, in crx/de's query tool)
> SELECT * FROM [nt:unstructured] AS c WHERE (CONTAINS(c.[jcr:title], 'a  b') 
> AND ISDESCENDANTNODE(c, '/content'))
> (note there are 2 spaces between "a" and "b")
> Result: java.lang.IllegalArgumentException: Invalid expression: 'a b'
> If there is only 1 space between a and b, there is no error. 
> Per jsr-283, fulltext expressions should be able to have strings of 
> whitespace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2920) RDBDocumentStore: fail init when database config seems to be inadequate

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-2920:

Fix Version/s: (was: 1.3.6)
   1.3.7

> RDBDocumentStore: fail init when database config seems to be inadequate
> ---
>
> Key: OAK-2920
> URL: https://issues.apache.org/jira/browse/OAK-2920
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: rdbmk
>Reporter: Julian Reschke
>Priority: Minor
>  Labels: resilience
> Fix For: 1.3.7
>
>
> It has been suggested that the implementation should fail to start (rather 
> than warn) when it detects a DB configuration that is likely to cause 
> problems (such as wrt character encoding or collation sequences)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2126) retry strategy for failed JDBC requests

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-2126:

Fix Version/s: (was: 1.3.6)
   1.3.7

> retry strategy for failed JDBC requests
> ---
>
> Key: OAK-2126
> URL: https://issues.apache.org/jira/browse/OAK-2126
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: rdbmk
>Reporter: Julian Reschke
>  Labels: resilience
> Fix For: 1.3.7
>
>
> Discussion: should we have a retry strategy for failed commits?
> Things to consider:
> - does this potentially interfere with other retry strategies (either on a 
> lower layer or in the DocumentMK)?
> - what failure scenarios would it address?
> - how to test those?
> - how to configure it?
> - what would be good defaults? (number of retries, interval)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2665) RDB: Tool/scripts for repair, recovery and sub-tree deletion

[Julian Reschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julian Reschke updated OAK-2665:

Fix Version/s: (was: 1.3.6)
   1.3.7

> RDB: Tool/scripts for repair, recovery and sub-tree deletion
> 
>
> Key: OAK-2665
> URL: https://issues.apache.org/jira/browse/OAK-2665
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: rdbmk
>Reporter: Amit Jain
>Assignee: Amit Jain
>  Labels: production, tools
> Fix For: 1.3.7
>
>
> Scripts and/or support should be added in oak-run console to support repair 
> and recovery options as supported for Mongo.
> Also, needed are options that are supported in the oak-mongo.js file 
> specially sub-tree deletion which has been found useful to delete corrupted 
> indexes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3184) Consistency checker for data/blob store

[Amit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738480#comment-14738480
 ] 

Amit Jain commented on OAK-3184:


Thanks [~tmueller] for the reivew. Yes, currently only 1 is added because if we 
use this list to restore from a backup the other should be fine.
Also, I was thinking rather than the node id the path should be recorded in the 
logs. wdyt?

> Consistency checker for data/blob store
> ---
>
> Key: OAK-3184
> URL: https://issues.apache.org/jira/browse/OAK-3184
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob
>Reporter: Amit Jain
>Assignee: Amit Jain
>  Labels: resilience, tooling
> Fix For: 1.3.6
>
> Attachments: OAK-3184.patch
>
>
> Consistency checker for data/blob store to report any missing blobs which can 
> result as a consequence of erroneous gc or extraneous factors. 
> * Will report any missing blob ids
> * Path of nodes referring to those blobs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3184) Consistency checker for data/blob store

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738466#comment-14738466
 ] 

Thomas Mueller commented on OAK-3184:
-

The patch looks good to me. You only remember one reference (path) per binary, 
right? Which one is picked seems to be random. I think that's fine, just wanted 
to know if I got the logic right.

> Consistency checker for data/blob store
> ---
>
> Key: OAK-3184
> URL: https://issues.apache.org/jira/browse/OAK-3184
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: blob
>Reporter: Amit Jain
>Assignee: Amit Jain
>  Labels: resilience, tooling
> Fix For: 1.3.6
>
> Attachments: OAK-3184.patch
>
>
> Consistency checker for data/blob store to report any missing blobs which can 
> result as a consequence of erroneous gc or extraneous factors. 
> * Will report any missing blob ids
> * Path of nodes referring to those blobs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3348) Cross gc sessions might introduce references to pre-compacted segments

[JIRA]

[ 
https://issues.apache.org/jira/browse/OAK-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738461#comment-14738461
 ] 

Michael Dürig commented on OAK-3348:


ITs fail as multi valued properties are not properly handled. Also binaries are 
currently not being duplicated thus they still create back links. 

> Cross gc sessions might introduce references to pre-compacted segments
> --
>
> Key: OAK-3348
> URL: https://issues.apache.org/jira/browse/OAK-3348
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: cleanup, compaction, gc
> Fix For: 1.3.7
>
> Attachments: OAK-3348.patch, image.png
>
>
> I suspect that certain write operations during compaction can cause 
> references from compacted segments to pre-compacted ones. This would 
> effectively prevent the pre-compacted segments from getting evicted in 
> subsequent cleanup phases. 
> The scenario is as follows:
> * A session is opened and a lot of content is written to it such that the 
> update limit is exceeded. This causes the changes to be written to disk. 
> * Revision gc runs causing a new, compacted root node state to be written to 
> disk.
> * The session saves its changes. This causes rebasing of its changes onto the 
> current root (the compacted one). At this point any node that has been added 
> will be added again in the sub-tree rooted at the current root. Such nodes 
> however might have been written to disk *before* revision gc ran and might 
> thus be contained in pre-compacted segments. As I suspect the node-add 
> operation in the rebasing process *not* to create a deep copy of such nodes 
> but to rather create a *reference* to them, a reference to a pre-compacted 
> segment is introduced here. 
> Going forward we need to validate above hypothesis, assess its impact if 
> necessary come up with a solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3380) Property index pruning should happen asynchronously

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738447#comment-14738447
 ] 

Thomas Mueller commented on OAK-3380:
-

As an alternative: instead of pruning while updating the index, we might be 
able to prune while reading from the index. I'm not sure if that's really a 
good idea (write while reading), but anyway: when traversing a empty node, 
remove it with a probability of let's say 5% (using a pseudo random number 
generator). That way, empty nodes are removed eventually, and the effect of 
query performance would be limited.

> Property index pruning should happen asynchronously
> ---
>
> Key: OAK-3380
> URL: https://issues.apache.org/jira/browse/OAK-3380
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Affects Versions: 1.3.5
>Reporter: Vikas Saurabh
>Priority: Minor
>  Labels: resilience
>
> Following up on this (a relatively old) thread \[1], we should do pruning of 
> property index structure asynchronously. The thread was never concluded.. 
> here are a couple of ideas picked from the thread:
> * Move pruning to an async thread
> * Throttle pruning i.e. prune only once in a while
> ** I'm not sure how that would work though -- an unpruned part would remain 
> as is until another index happens on that path.
> Once we can move pruning to some async thread (reducing concurrent updates), 
> OAK-2673 + OAK-2929 can take care of add-add conflicts.
> 
> h6. Why is this an issue despite merge retries taking care of it?
> A couple of cases which have concurrent updates hitting merge conflicts in 
> our product (Adobe AEM):
> * Some index are very volatile (in the sense that indexed property switches 
> its values very quickly) e.g. sling job status, AEM workflow status.
> * Multiple threads take care of jobs. Although sling maintains a bucketed 
> structure for job storage to reduce conflicts... but inside index tree the 
> bucket structure, at times, gets pruned and needs to be created in the next 
> job status change
> While retries do take care of these conflict a lot of times and even when 
> they don't, AEM workflows has it's own retry to work around. But, retrying, 
> IMHO, is just a waste of time -- more importantly in paths where application 
> doesn't really have a control.
> h6. Would this add to cost of traversing index structure?
> Yes, there'd be some left over paths in index structure between asynchronous 
> prunes. But, I think the cost of such wasted traversals would be covered up 
> with time saved in avoiding the concurrent update conflict.
> 
> (cc [~tmueller], [~mreutegg], [~alex.parvulescu], [~chetanm])
> \[1]: 
> http://mail-archives.apache.org/mod_mbox/jackrabbit-oak-dev/201506.mbox/%3ccadichf66u2vh-hlrjunansytxfidj2mt3vktr4ybkngpzy9...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-3380) Property index pruning should happen asynchronously

[Thomas Mueller (JIRA)]

[ 
https://issues.apache.org/jira/browse/OAK-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738428#comment-14738428
 ] 

Thomas Mueller commented on OAK-3380:
-

> Move pruning to an async thread

If there are multiple cluster nodes, then which one would do the pruning? How 
would the others inform the one that does the pruning?

We could do some kind of "garbage collection" process, but that would take some 
time as it would have to process the whole index. To speed it up: Merge 
conflicts don't happen for concurrent "add new node", right? If that's the 
case, then we could do this: Instead of pruning (deleting empty nodes), we 
could add a respective node below /oak:index/indexName/:trash. To reduce write 
access, not the full path, but maybe just the parent, or the parent of the 
parent (but at least 1 child node of :trash). That way, the garbage collection 
process would only have to process those nodes.

> Property index pruning should happen asynchronously
> ---
>
> Key: OAK-3380
> URL: https://issues.apache.org/jira/browse/OAK-3380
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: query
>Affects Versions: 1.3.5
>Reporter: Vikas Saurabh
>Priority: Minor
>  Labels: resilience
>
> Following up on this (a relatively old) thread \[1], we should do pruning of 
> property index structure asynchronously. The thread was never concluded.. 
> here are a couple of ideas picked from the thread:
> * Move pruning to an async thread
> * Throttle pruning i.e. prune only once in a while
> ** I'm not sure how that would work though -- an unpruned part would remain 
> as is until another index happens on that path.
> Once we can move pruning to some async thread (reducing concurrent updates), 
> OAK-2673 + OAK-2929 can take care of add-add conflicts.
> 
> h6. Why is this an issue despite merge retries taking care of it?
> A couple of cases which have concurrent updates hitting merge conflicts in 
> our product (Adobe AEM):
> * Some index are very volatile (in the sense that indexed property switches 
> its values very quickly) e.g. sling job status, AEM workflow status.
> * Multiple threads take care of jobs. Although sling maintains a bucketed 
> structure for job storage to reduce conflicts... but inside index tree the 
> bucket structure, at times, gets pruned and needs to be created in the next 
> job status change
> While retries do take care of these conflict a lot of times and even when 
> they don't, AEM workflows has it's own retry to work around. But, retrying, 
> IMHO, is just a waste of time -- more importantly in paths where application 
> doesn't really have a control.
> h6. Would this add to cost of traversing index structure?
> Yes, there'd be some left over paths in index structure between asynchronous 
> prunes. But, I think the cost of such wasted traversals would be covered up 
> with time saved in avoiding the concurrent update conflict.
> 
> (cc [~tmueller], [~mreutegg], [~alex.parvulescu], [~chetanm])
> \[1]: 
> http://mail-archives.apache.org/mod_mbox/jackrabbit-oak-dev/201506.mbox/%3ccadichf66u2vh-hlrjunansytxfidj2mt3vktr4ybkngpzy9...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3384) Revisit PartialCompactionMapTest

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-3384:
---
Fix Version/s: (was: 1.3.6)
   1.3.7

> Revisit PartialCompactionMapTest
> 
>
> Key: OAK-3384
> URL: https://issues.apache.org/jira/browse/OAK-3384
> Project: Jackrabbit Oak
>  Issue Type: Technical task
>  Components: segmentmk
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
>Priority: Minor
> Fix For: 1.3.7
>
> Attachments: PartialCompactionMapTest.java.patch
>
>
> Things I wanted to tweak
> * log the benchmark results to a logger instead of system.out so the output 
> doesn't get lost
> * split the _bench_ variable to specific tests, so you can choose which one 
> you want to run
> * cleanup the imports
> * cleanup of test directories simply doesn't happen
> * revisit the default values (ex. benchLargeMap has the requirement of 1G but 
> goes all the way up to 1000 iterations, so 1g is no where near enough heap).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-1828) Improved SegmentWriter

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-1828:
---
Fix Version/s: (was: 1.3.6)
   1.3.7

> Improved SegmentWriter
> --
>
> Key: OAK-1828
> URL: https://issues.apache.org/jira/browse/OAK-1828
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: segmentmk
>Reporter: Jukka Zitting
>Assignee: Alex Parvulescu
>Priority: Minor
>  Labels: technical_debt
> Fix For: 1.3.7
>
>
> At about 1kLOC and dozens of methods, the SegmentWriter class currently a bit 
> too complex for one of the key components of the TarMK. It also uses a 
> somewhat non-obvious mix of synchronized and unsynchronized code to 
> coordinate multiple concurrent threads that may be writing content at the 
> same time. The synchronization blocks are also broader than what really would 
> be needed, which in some cases causes unnecessary lock contention in 
> concurrent write loads.
> To improve the readability and maintainability of the code, and to increase 
> performance of concurrent writes, it would be useful to split part of the 
> SegmentWriter functionality to a separate RecordWriter class that would be 
> responsible for writing individual records into a segment. The 
> SegmentWriter.prepare() method would return a new RecordWriter instance, and 
> the higher-level SegmentWriter methods would use the returned instance for 
> all the work that's currently guarded in synchronization blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3133) Make compaction map more efficient for offline compaction

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-3133:
---
Fix Version/s: (was: 1.0.20)
   (was: 1.2.5)
   (was: 1.3.6)
   1.0.21
   1.2.6
   1.3.7

> Make compaction map more efficient for offline compaction
> -
>
> Key: OAK-3133
> URL: https://issues.apache.org/jira/browse/OAK-3133
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: run
>Reporter: Michael Dürig
>Assignee: Alex Parvulescu
>  Labels: compaction, gc
> Fix For: 1.3.7, 1.2.6, 1.0.21
>
> Attachments: OAK-3133-01.patch, OAK-3133-partial.patch, OAK-3133.patch
>
>
> The compaction map might be expensive. See OAK-2862 for the analysis. We 
> should find ways to lower the impact of this on offline compaction. One 
> option would be to make the compress cycle configurable, which would require 
> more heap. Another option would be to leverage the persisted compaction map 
> here also.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3325) MissingIndexProviderStrategy should warn when setting the reindex flag

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-3325:
---
Fix Version/s: (was: 1.3.6)
   1.3.7

> MissingIndexProviderStrategy should warn when setting the reindex flag
> --
>
> Key: OAK-3325
> URL: https://issues.apache.org/jira/browse/OAK-3325
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
> Fix For: 1.3.7
>
> Attachments: OAK-3325.patch
>
>
> Currently the _MissingIndexProviderStrategy_ would silently flip the reindex 
> flag on missing provider types. We need to add some warning logs to know when 
> this happens for the synchronous indexes too.
> [0] 
> https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/index/IndexUpdate.java#L323



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2835) TARMK Cold Standby inefficient cleanup

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-2835:
---
Fix Version/s: (was: 1.3.6)
   1.3.7

> TARMK Cold Standby inefficient cleanup
> --
>
> Key: OAK-2835
> URL: https://issues.apache.org/jira/browse/OAK-2835
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segmentmk, tarmk-standby
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
>Priority: Critical
>  Labels: compaction, gc, production, resilience
> Fix For: 1.3.7
>
> Attachments: OAK-2835.patch
>
>
> Following OAK-2817, it turns out that patching the data corruption issue 
> revealed an inefficiency of the cleanup method. similar to the online 
> compaction situation, the standby has issues clearing some of the in-memory 
> references to old revisions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3303) FileStore flush thread can get stuck

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-3303:
---
Fix Version/s: (was: 1.3.6)
   1.3.7

> FileStore flush thread can get stuck
> 
>
> Key: OAK-3303
> URL: https://issues.apache.org/jira/browse/OAK-3303
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segmentmk
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
> Fix For: 1.3.7
>
>
> In some very rare circumstances the flush thread was seen as possibly stuck 
> for a while following a restart of the system. This results in data loss on 
> restart (the system will roll back to the latest persisted revision on 
> restart), and worse off there's no way of extracting the latest head revision 
> using the tar files, so recovery is not (yet) possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3270) Improve DocumentMK resilience

[Marcel Reutegger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcel Reutegger updated OAK-3270:
--
Fix Version/s: (was: 1.3.6)
   1.3.7

> Improve DocumentMK resilience
> -
>
> Key: OAK-3270
> URL: https://issues.apache.org/jira/browse/OAK-3270
> Project: Jackrabbit Oak
>  Issue Type: Epic
>  Components: mongomk, rdbmk
>Reporter: Michael Marth
>  Labels: resilience
> Fix For: 1.3.7
>
>
> Collection of DocMK resilience improvements



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3148) Online migration process for the binaries

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomek Rękawek updated OAK-3148:
---
Fix Version/s: (was: 1.4)
   1.3.6

> Online migration process for the binaries
> -
>
> Key: OAK-3148
> URL: https://issues.apache.org/jira/browse/OAK-3148
> Project: Jackrabbit Oak
>  Issue Type: New Feature
>  Components: blob, upgrade
>Reporter: Tomek Rękawek
>Priority: Minor
> Fix For: 1.3.6
>
>
> For clients that want to migrate their blob stores, let's add a new feature 
> that allows copy them in the background.
> AC:
> # SplitBlobStore
> ## Administrator can configure Oak to use the {{SplitBlobStore}} that 
> references the source (old) and the destination (new) blob store.
> ## Data stores can be used as well via the {{DataStoreBlobStore}}.
> ## On the read operation, if the requested blob exists on the new store, 
> SplitBlobStore will return it.
> ## Otherwise, SplitBlobStore will try to read the blob from the old store.
> ## All write requests will be directed to the new blob store.
> # Copy process
> ## Administrator can start, stop and resume the copy process using JMX 
> command.
> ## Administrator can see the progress in JMX and logs
> ## The process will read the {{SplitBlobStore}} configuration and copy the 
> binaries from source to destination
> ## Once a binary is moved, its reference in the {{NodeStore}} is updated and 
> commited.
> ## Only the head revision has to be updated.
> The idea is that after all binaries are copied, the old revisions will be 
> gradually removed by the compaction mechanisms and then binaries will be 
> removed from the source store by the blob garbage collector. Future 
> improvements are possible, eg. to invoke the compaction and GC manually.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3287) DocumentMK revision GC

[Marcel Reutegger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcel Reutegger updated OAK-3287:
--
Fix Version/s: (was: 1.3.7)
   1.3.8

> DocumentMK revision GC
> --
>
> Key: OAK-3287
> URL: https://issues.apache.org/jira/browse/OAK-3287
> Project: Jackrabbit Oak
>  Issue Type: Epic
>  Components: mongomk, rdbmk
>Reporter: Michael Marth
> Fix For: 1.3.8
>
>
> Collector for various tasks on implementing DocMK revision GC



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3271) Improve DocumentMK performance

[Marcel Reutegger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcel Reutegger updated OAK-3271:
--
Fix Version/s: (was: 1.3.8)
   1.3.9

> Improve DocumentMK performance
> --
>
> Key: OAK-3271
> URL: https://issues.apache.org/jira/browse/OAK-3271
> Project: Jackrabbit Oak
>  Issue Type: Epic
>  Components: mongomk, rdbmk
>Reporter: Michael Marth
>  Labels: performance
> Fix For: 1.3.9
>
>
> Collector issue for DocMK performance improvements



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2635) TimeSeriesMax's frequent 'drops to 0'

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-2635:
---
Fix Version/s: (was: 1.3.7)
   1.3.9

> TimeSeriesMax's frequent 'drops to 0'
> -
>
> Key: OAK-2635
> URL: https://issues.apache.org/jira/browse/OAK-2635
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.12
>Reporter: Stefan Egli
>Assignee: Michael Dürig
>  Labels: observation, tooling
> Fix For: 1.3.9
>
>
> The current implementation of TimeSeriesMax - which is what is backing eg the 
> very important 'ObservationQueueMaxLength' statistics - has a very infamous 
> behavior: it does very frequent, intermittent 'jumps back to 0'. This even 
> though the queue-lengths are still at the previous highs, as can often be 
> seen with subsequent measurements (which eg are still showing there are 1000 
> events in the observation queue).
> The reason seems to be that
> * the value is increased via {{TimeSeriesMax.recordValue()}} during a 1 
> second interval
> * reset to 0 via {{TimeSeriesMax.run()}} every second
> So basically, every second the counter is reset, then during 1 second if any 
> call to {{recordValue()}} happens, it is increased.
> This in my view is rather unfortunate - as it can result in mentioned 
> 'jumpy-0' behavior, but it can also jump to values in between if the largest 
> queue does not reports its length during 1 second.
> It sounds a bit like this was done this way intentionally? (perhaps to make 
> it as inexpensive as possible) or could this be fixed?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3272) DocumentMK scalability improvements

[Marcel Reutegger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcel Reutegger updated OAK-3272:
--
Fix Version/s: (was: 1.3.8)
   1.3.9

> DocumentMK scalability improvements
> ---
>
> Key: OAK-3272
> URL: https://issues.apache.org/jira/browse/OAK-3272
> Project: Jackrabbit Oak
>  Issue Type: Epic
>  Components: mongomk, rdbmk
>Reporter: Michael Marth
>  Labels: scalability
> Fix For: 1.3.9
>
>
> Collector issue for tracking DocMK issues concerning scalability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-1576) SegmentMK: Implement refined conflict resolution for addExistingNode conflicts

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-1576:
---
Fix Version/s: (was: 1.3.7)
   1.3.9

> SegmentMK: Implement refined conflict resolution for addExistingNode conflicts
> --
>
> Key: OAK-1576
> URL: https://issues.apache.org/jira/browse/OAK-1576
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: concurrency, resilience, scalability
> Fix For: 1.3.9
>
>
> Implement refined conflict resolution for addExistingNode conflicts as 
> defined in the parent issue for the SegementMK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3290) Revision gc blocks repository shutdown

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-3290:
---
Fix Version/s: (was: 1.3.6)
   1.3.8

> Revision gc blocks repository shutdown
> --
>
> Key: OAK-3290
> URL: https://issues.apache.org/jira/browse/OAK-3290
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: cleanup, compaction, gc
> Fix For: 1.0.20, 1.2.5, 1.3.8
>
>
> Shutting down the repository while revision gc is running might block for a 
> long time until either compaction estimation/compaction or clean up has 
> finished. We should provide a way to interrupt those operations for a timely 
> shut down. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2405) Monitoring to track old NodeStates

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-2405:
---
Fix Version/s: (was: 1.3.6)
   1.3.8

> Monitoring to track old NodeStates
> --
>
> Key: OAK-2405
> URL: https://issues.apache.org/jira/browse/OAK-2405
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: gc, monitoring, tooling
> Fix For: 1.3.8
>
>
> We should add some monitoring that allows us to track "old" node states, 
> which potentially block revision gc. 
> Possible approaches:
> * Add monitoring too old revisions (root node states) along with the stack 
> traces from where they have been acquired.
> * Include RecordId of root node state in the {{SessionMBean}}.
> * Add additional tooling on top of the {{SessionMBean}} to make it easier to 
> make sense of the wealth of information provided. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3290) Revision gc blocks repository shutdown

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-3290:
---
Fix Version/s: (was: 1.0.20)
   (was: 1.2.5)
   1.2.6
   1.0.21

> Revision gc blocks repository shutdown
> --
>
> Key: OAK-3290
> URL: https://issues.apache.org/jira/browse/OAK-3290
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: cleanup, compaction, gc
> Fix For: 1.3.8, 1.2.6, 1.0.21
>
>
> Shutting down the repository while revision gc is running might block for a 
> long time until either compaction estimation/compaction or clean up has 
> finished. We should provide a way to interrupt those operations for a timely 
> shut down. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2881) ConsistencyChecker#checkConsistency can't cope with inconsistent journal

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-2881:
---
Fix Version/s: (was: 1.3.6)
   1.3.8

> ConsistencyChecker#checkConsistency can't cope with inconsistent journal
> 
>
> Key: OAK-2881
> URL: https://issues.apache.org/jira/browse/OAK-2881
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: run
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: resilience, tooling
> Fix For: 1.3.8
>
>
> When running the consistency checker against a repository with a corrupt 
> journal, it fails with an {{ISA}} instead of trying to skip over invalid 
> revision identifiers:
> {noformat}
> Exception in thread "main" java.lang.IllegalArgumentException: Bad record 
> identifier: foobar
> at 
> org.apache.jackrabbit.oak.plugins.segment.RecordId.fromString(RecordId.java:57)
> at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.(FileStore.java:227)
> at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.(FileStore.java:178)
> at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.(FileStore.java:156)
> at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.(FileStore.java:166)
> at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore$ReadOnlyStore.(FileStore.java:805)
> at 
> org.apache.jackrabbit.oak.plugins.segment.file.tooling.ConsistencyChecker.(ConsistencyChecker.java:108)
> at 
> org.apache.jackrabbit.oak.plugins.segment.file.tooling.ConsistencyChecker.checkConsistency(ConsistencyChecker.java:70)
> at org.apache.jackrabbit.oak.run.Main.check(Main.java:701)
> at org.apache.jackrabbit.oak.run.Main.main(Main.java:158)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2507) Truncate journal.log after off line compaction

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-2507:
---
Fix Version/s: (was: 1.3.6)
   1.3.8

> Truncate journal.log after off line compaction
> --
>
> Key: OAK-2507
> URL: https://issues.apache.org/jira/browse/OAK-2507
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Minor
>  Labels: compaction, gc
> Fix For: 1.3.8
>
>
> After off line compaction the repository contains a single revision. However 
> the journal.log file will still contain the trail of all revisions that have 
> been removed during the compaction process. I suggest we truncate the 
> journal.log to only contain the latest revision created during compaction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3340) DocumentMK technical debt

[Marcel Reutegger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcel Reutegger updated OAK-3340:
--
Fix Version/s: (was: 1.3.9)
   1.3.10

> DocumentMK technical debt 
> --
>
> Key: OAK-3340
> URL: https://issues.apache.org/jira/browse/OAK-3340
> Project: Jackrabbit Oak
>  Issue Type: Epic
>  Components: mongomk
>Reporter: Daniel Hasler
> Fix For: 1.3.10
>
>
> As discussed bilaterally grouping the technical debt for MongoMK in this 
> issue for easier tracking



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3348) Cross gc sessions might introduce references to pre-compacted segments

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-3348:
---
Fix Version/s: (was: 1.3.6)
   1.3.7

> Cross gc sessions might introduce references to pre-compacted segments
> --
>
> Key: OAK-3348
> URL: https://issues.apache.org/jira/browse/OAK-3348
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: cleanup, compaction, gc
> Fix For: 1.3.7
>
> Attachments: OAK-3348.patch, image.png
>
>
> I suspect that certain write operations during compaction can cause 
> references from compacted segments to pre-compacted ones. This would 
> effectively prevent the pre-compacted segments from getting evicted in 
> subsequent cleanup phases. 
> The scenario is as follows:
> * A session is opened and a lot of content is written to it such that the 
> update limit is exceeded. This causes the changes to be written to disk. 
> * Revision gc runs causing a new, compacted root node state to be written to 
> disk.
> * The session saves its changes. This causes rebasing of its changes onto the 
> current root (the compacted one). At this point any node that has been added 
> will be added again in the sub-tree rooted at the current root. Such nodes 
> however might have been written to disk *before* revision gc ran and might 
> thus be contained in pre-compacted segments. As I suspect the node-add 
> operation in the rebasing process *not* to create a deep copy of such nodes 
> but to rather create a *reference* to them, a reference to a pre-compacted 
> segment is introduced here. 
> Going forward we need to validate above hypothesis, assess its impact if 
> necessary come up with a solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3329) TarMK cleanup blocks writers

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-3329:
---
Fix Version/s: 1.3.7

> TarMK cleanup blocks writers
> 
>
> Key: OAK-3329
> URL: https://issues.apache.org/jira/browse/OAK-3329
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segmentmk
>Reporter: Michael Dürig
>  Labels: cleanup, gc
> Fix For: 1.3.7
>
>
> TarMK cleanup exclusively locks the {{FileStore}}, which causes concurrent 
> writers to block until cleanup finished. Initially cleanup was expected to be 
> reasonably fast, however I have seen it taking dozens of minutes under 
> certain circumstances (most likely many tar files with many small segments, 
> aka OAK-2896).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (OAK-3329) TarMK cleanup blocks writers

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig reassigned OAK-3329:
--

Assignee: Michael Dürig

> TarMK cleanup blocks writers
> 
>
> Key: OAK-3329
> URL: https://issues.apache.org/jira/browse/OAK-3329
> Project: Jackrabbit Oak
>  Issue Type: Improvement
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: cleanup, gc
> Fix For: 1.3.7
>
>
> TarMK cleanup exclusively locks the {{FileStore}}, which causes concurrent 
> writers to block until cleanup finished. Initially cleanup was expected to be 
> reasonably fast, however I have seen it taking dozens of minutes under 
> certain circumstances (most likely many tar files with many small segments, 
> aka OAK-2896).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2833) Refactor TarMK

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-2833:
---
Fix Version/s: (was: 1.3.7)
   1.4

> Refactor TarMK
> --
>
> Key: OAK-2833
> URL: https://issues.apache.org/jira/browse/OAK-2833
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: technical_debt
> Fix For: 1.4
>
>
> Container issue for refactoring the TarMK to make it more testable, 
> maintainable, extensible and less entangled. 
> For example the segment format should be readable, writeable through 
> standalone means so tests, tools and production code can share this code. 
> Currently there is a lot of code duplication involved here. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2403) Improve monitoring capabilities for TarMk revision gc

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-2403:
---
Fix Version/s: (was: 1.3.6)
   1.4

> Improve monitoring capabilities for TarMk revision gc
> -
>
> Key: OAK-2403
> URL: https://issues.apache.org/jira/browse/OAK-2403
> Project: Jackrabbit Oak
>  Issue Type: Task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: gc, monitoring, tooling
> Fix For: 1.4
>
>
> Container devoted to improving monitoring of the TarMk revision garbage 
> collection process. The overall goal is to make it more transparent what 
> revision gc does, how it performs, why it failed etc. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2879) Compaction should check for required disk space before running

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-2879:
---
Fix Version/s: (was: 1.3.6)
   1.3.7

> Compaction should check for required disk space before running
> --
>
> Key: OAK-2879
> URL: https://issues.apache.org/jira/browse/OAK-2879
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: compaction, doc-impacting, gc, resilience
> Fix For: 1.3.7
>
>
> In the worst case compaction doubles the repository size while running. As 
> this is somewhat unexpected we should check whether there is enough free disk 
> space before running compaction and log a warning otherwise. This is to avoid 
> a common source of running out of disk space and ending up with a corrupted 
> repository. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OAK-2879) Compaction should check for required disk space before running

[JIRA]

[ 
https://issues.apache.org/jira/browse/OAK-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738346#comment-14738346
 ] 

Michael Dürig commented on OAK-2879:


Instead of checking disk space *before* running compaction a better solution 
would be to monitor disk space *during* compaction. Once it drops below a 
certain threshold, bail out and log. The extra segments introduced by the 
aborted compaction run will then get collected by the subsequent cleanup run. 

[~frm] assigning to you as discussed. 

> Compaction should check for required disk space before running
> --
>
> Key: OAK-2879
> URL: https://issues.apache.org/jira/browse/OAK-2879
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: compaction, doc-impacting, gc, resilience
> Fix For: 1.3.7
>
>
> In the worst case compaction doubles the repository size while running. As 
> this is somewhat unexpected we should check whether there is enough free disk 
> space before running compaction and log a warning otherwise. This is to avoid 
> a common source of running out of disk space and ending up with a corrupted 
> repository. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2879) Compaction should check for required disk space before running

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-2879:
---
Assignee: Francesco Mari  (was: Michael Dürig)

> Compaction should check for required disk space before running
> --
>
> Key: OAK-2879
> URL: https://issues.apache.org/jira/browse/OAK-2879
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Francesco Mari
>  Labels: compaction, doc-impacting, gc, resilience
> Fix For: 1.3.7
>
>
> In the worst case compaction doubles the repository size while running. As 
> this is somewhat unexpected we should check whether there is enough free disk 
> space before running compaction and log a warning otherwise. This is to avoid 
> a common source of running out of disk space and ending up with a corrupted 
> repository. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-2849) Improve revision gc on SegmentMK

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-2849:
---
Fix Version/s: (was: 1.3.6)
   1.4

> Improve revision gc on SegmentMK
> 
>
> Key: OAK-2849
> URL: https://issues.apache.org/jira/browse/OAK-2849
> Project: Jackrabbit Oak
>  Issue Type: Epic
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: compaction, gc
> Fix For: 1.4
>
> Attachments: SegmentCompactionIT-conflicts.png
>
>
> This is a container issue for the ongoing effort to improve revision gc of 
> the SegmentMK. 
> I'm exploring 
> * ways to make the reference graph as exact as possible and necessary: it 
> should not contain segments that are not referenceable any more and but must 
> contain all segments that are referenceable. 
> * ways to segregate the reference graph reducing dependencies between certain 
> set of segments as much as possible. 
> * Reducing the number of in memory references and their impact on gc as much 
> as possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3350) Incremental compaction

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-3350:
---
Fix Version/s: (was: 1.3.6)
   1.4

> Incremental compaction
> --
>
> Key: OAK-3350
> URL: https://issues.apache.org/jira/browse/OAK-3350
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: compaction, gc
> Fix For: 1.4
>
>
> This is OAK-3349 taken to the extreme: given a segment that is almost not 
> referenced any more we could just rewrite the still referenced content. That 
> is, say a segment contains two properties reachable from the current root 
> node state and all its remaining content is not reachable from the root node 
> state. In that case we could rewrite these two properties and create a new 
> root node state referencing the rewritten properties. This would effectively 
> make the segment eligible for being gc-ed. 
> Such an approach would start from segments that are sparse and compact these 
> instead of compacting everything as we currently do, which might cause a lot 
> of copying around stuff that already is compact. The challenging part here is 
> probably finding the segments that are sparse as this involves inverting the 
> reference graph. 
> Todo: Asses feasibility and impact, implement prototype.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3236) integration test that simulates influence of clock drift

[Stefan Egli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OAK-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Egli updated OAK-3236:
-
Fix Version/s: (was: 1.3.6)
   1.3.7

> integration test that simulates influence of clock drift
> 
>
> Key: OAK-3236
> URL: https://issues.apache.org/jira/browse/OAK-3236
> Project: Jackrabbit Oak
>  Issue Type: Test
>  Components: core
>Affects Versions: 1.3.4
>Reporter: Stefan Egli
>Assignee: Stefan Egli
> Fix For: 1.3.7
>
>
> Spin-off of OAK-2739 [of this 
> comment|https://issues.apache.org/jira/browse/OAK-2739?focusedCommentId=14693398&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14693398]
>  - ie there should be an integration test that show cases the issues with 
> clock drift and why it is a good idea to have a lease-check (that refuses to 
> let the document store be used any further once the lease times out locally)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3349) Partial compaction

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-3349:
---
Fix Version/s: (was: 1.3.6)
   1.4

> Partial compaction
> --
>
> Key: OAK-3349
> URL: https://issues.apache.org/jira/browse/OAK-3349
> Project: Jackrabbit Oak
>  Issue Type: Sub-task
>  Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>  Labels: compaction, gc
> Fix For: 1.4
>
>
> On big repositories compaction can take quite a while to run as it needs to 
> create a full deep copy of the current root node state. For such cases it 
> could be beneficial if we could partially compact the repository thus 
> splitting full compaction over multiple cycles. 
> Partial compaction would run compaction on a sub-tree just like we now run it 
> on the full tree. Afterwards it would create a new root node state by 
> referencing the previous root node state replacing said sub-tree with the 
> compacted one. 
> Todo: Asses feasibility and impact, implement prototype.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OAK-3235) Deadlock when closing a concurrently used FileStore

[JIRA]

 [ 
https://issues.apache.org/jira/browse/OAK-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Dürig updated OAK-3235:
---
Fix Version/s: (was: 1.3.6)
   1.3.7

> Deadlock when closing a concurrently used FileStore
> ---
>
> Key: OAK-3235
> URL: https://issues.apache.org/jira/browse/OAK-3235
> Project: Jackrabbit Oak
>  Issue Type: Bug
>  Components: segmentmk
>Affects Versions: 1.3.3
>Reporter: Francesco Mari
>Assignee: Michael Dürig
>Priority: Critical
>  Labels: resilience
> Fix For: 1.3.7
>
> Attachments: OAK-3235-01.patch
>
>
> A deadlock was detected while stopping the {{SegmentCompactionIT}} using the 
> exposed MBean.
> {noformat}
> Found one Java-level deadlock:
> =
> "pool-1-thread-23":
>   waiting to lock monitor 0x7fa8cf1f0488 (object 0x0007a0081e48, a 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore),
>   which is held by "main"
> "main":
>   waiting to lock monitor 0x7fa8cc015ff8 (object 0x0007a011f750, a 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter),
>   which is held by "pool-1-thread-23"
> Java stack information for the threads listed above:
> ===
> "pool-1-thread-23":
>   at 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.writeSegment(FileStore.java:948)
>   - waiting to lock <0x0007a0081e48> (a 
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.flush(SegmentWriter.java:228)
>   - locked <0x0007a011f750> (a 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.prepare(SegmentWriter.java:329)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeListBucket(SegmentWriter.java:447)
>   - locked <0x0007a011f750> (a 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeList(SegmentWriter.java:698)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1190)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:400)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:400)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:400)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:400)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:400)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:400)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
>   at 
> org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:400)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1154)
>   at 
> org.apache.jackrabbit.oak.plugins.segment.S