[jira] [Commented] (OAK-5433) System Pacing Service
[
https://issues.apache.org/jira/browse/OAK-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816214#comment-15816214
]
Michael Marth commented on OAK-5433:
ack - thx!
> System Pacing Service
> -
>
> Key: OAK-5433
> URL: https://issues.apache.org/jira/browse/OAK-5433
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Stefan Eissing
> Attachments: obs-pacing.diff
>
>
> h3. tl;dr
> By adding Pacing, suitable to the application {{oak}} is running in, a system
> will dynamically adapt the load to its own capabilities. This effectively, in
> tests, keeps the system stable and responsive under stress.
> h3. The Situation
> During experimental Lab tests on large clusters, it became clear that the a
> web system based using oak is challenged by fluctuating load in relation to
> its own capabilities.
> When the load increases "too much" it shows the following symptoms:
> * event observation queues grow
> * maintenance tasks (on master) take too long
> * async tasks, triggered by requests, (e.g. workflows) accumulate
> and eventually
> * login sessions complain about freshness
> * revisions diffs are old and no longer in caches
> and sometimes
> * database lease times out and oak-core shuts down
> This problem can arise when outside requests increased, or when local
> maintenance tasks occupy resources, or when available CPU diminishes due to
> other processes or page faults or, or, or.
> Unfortunately, whenever the system becomes overburdened, the secondary
> effects make the system even slower and, thus, more overburdened. This can
> end in a vicious circle, making the system total unresponsive. Eventual
> recovery is an option, not a guarantee.
> h3. Pacing
> By _Pacing_ I mean a system behaviour that tries to balance load in relation
> to capabilities. If the latter one drops, the load must be reduced until the
> system recovers. This is related to what the {{CommitRateLimiter}} wanted to
> achieve
> by monitoring observation queues.
> The design of the {{CommitRateLimiter}} could be very efficient, if it only
> know _which_ commits to delay. But it does not know the application that oak
> runs in. I propose replacing the Limiter by a {{PacingService}} that can be
> provided by the application using oak. The service will get the data about
> the current commit, queue length and limits. Whatever else it does remains
> opaque. It may raise a proper exception to indicate that the commit shall
> fail. But mostly, it is expected to delay those commits that would negatively
> affect system stability.
> h3. An Example
> In a proof of concept, an AEM system was blasted with endless uploads on
> multiple connections in order to eventually overwhelm queues. The a pacing
> was patched into oak-core that delayed commits from servlet requests and from
> certain workflows for some milliseconds until the queue length shrank again.
> The pacing had a maximum wait time that would make the commit fail.
> The pacer was configured to trigger at 75% of maximum queue length and the
> system was blasted with uploads again. In the tests:
> # the max queue length stayed under 80%
> # no upload did reach the maximum time, all succeeded
> The system adapted the external load to its capabilities successfully.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-5433) System Pacing Service
[
https://issues.apache.org/jira/browse/OAK-5433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15815058#comment-15815058
]
Michael Marth commented on OAK-5433:
I like this idea better than the CommitRateLimiter (for the reason you state).
However, I wonder about the impact of this service on commit rate in the "happy
case" (system is stable). Would think it's not too much impact/overhead, but
did you check?
For the numbers you quote: did you measure this on Oak 1.5.x or on 1.4.x?
Asking out of curiosity as there were so many improvements on observation queue
lately.
Lastly, for the record: in my mind the ideal architecture would be that Oak
only *signals* overload situations to higher level, but does not let higher
level interfere with commits. Higher level would rather slow down itself and
*start* less commits. The reason why I prefer that is the danger of having
applications interfere with commits (and the associated danger to break sthg).
> System Pacing Service
> -
>
> Key: OAK-5433
> URL: https://issues.apache.org/jira/browse/OAK-5433
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: core
>Reporter: Stefan Eissing
> Attachments: obs-pacing.diff
>
>
> h3. tl;dr
> By adding Pacing, suitable to the application {{oak}} is running in, a system
> will dynamically adapt the load to its own capabilities. This effectively, in
> tests, keeps the system stable and responsive under stress.
> h3. The Situation
> During experimental Lab tests on large clusters, it became clear that the a
> web system based using oak is challenged by fluctuating load in relation to
> its own capabilities.
> When the load increases "too much" it shows the following symptoms:
> * event observation queues grow
> * maintenance tasks (on master) take too long
> * async tasks, triggered by requests, (e.g. workflows) accumulate
> and eventually
> * login sessions complain about freshness
> * revisions diffs are old and no longer in caches
> and sometimes
> * database lease times out and oak-core shuts down
> This problem can arise when outside requests increased, or when local
> maintenance tasks occupy resources, or when available CPU diminishes due to
> other processes or page faults or, or, or.
> Unfortunately, whenever the system becomes overburdened, the secondary
> effects make the system even slower and, thus, more overburdened. This can
> end in a vicious circle, making the system total unresponsive. Eventual
> recovery is an option, not a guarantee.
> h3. Pacing
> By _Pacing_ I mean a system behaviour that tries to balance load in relation
> to capabilities. If the latter one drops, the load must be reduced until the
> system recovers. This is related to what the {{CommitRateLimiter}} wanted to
> achieve
> by monitoring observation queues.
> The design of the {{CommitRateLimiter}} could be very efficient, if it only
> know _which_ commits to delay. But it does not know the application that oak
> runs in. I propose replacing the Limiter by a {{PacingService}} that can be
> provided by the application using oak. The service will get the data about
> the current commit, queue length and limits. Whatever else it does remains
> opaque. It may raise a proper exception to indicate that the commit shall
> fail. But mostly, it is expected to delay those commits that would negatively
> affect system stability.
> h3. An Example
> In a proof of concept, an AEM system was blasted with endless uploads on
> multiple connections in order to eventually overwhelm queues. The a pacing
> was patched into oak-core that delayed commits from servlet requests and from
> certain workflows for some milliseconds until the queue length shrank again.
> The pacing had a maximum wait time that would make the commit fail.
> The pacer was configured to trigger at 75% of maximum queue length and the
> system was blasted with uploads again. In the tests:
> # the max queue length stayed under 80%
> # no upload did reach the maximum time, all succeeded
> The system adapted the external load to its capabilities successfully.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2821) PersistentCache not used for RDBBlobStore
[ https://issues.apache.org/jira/browse/OAK-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2821: --- Issue Type: Improvement (was: Bug) > PersistentCache not used for RDBBlobStore > - > > Key: OAK-2821 > URL: https://issues.apache.org/jira/browse/OAK-2821 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: rdbmk >Affects Versions: 1.2.12, 1.0.28, 1.4.0 >Reporter: Julian Reschke >Assignee: Thomas Mueller >Priority: Minor > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4 > Fix For: 1.8 > > Attachments: OAK-2821.diff > > > DocumentMK is currently inconsistent wrt to the use of the PersistentCache > for BlobStore. It is used for Mongo, but not for RDB. We should be consistent > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-1819) oak-solr-core test failures on Java 8
[
https://issues.apache.org/jira/browse/OAK-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644119#comment-15644119
]
Michael Marth commented on OAK-1819:
[~teofili], saw you changed the fixVersion from 1.6 to 1.8.
I think we should either fix test failure or choose to ignore the tests for
whatever reason or set this issue to WONTFIX, but pushing out test failures to
the next stable release does not make a lot of sense to me.
The same comment applies to OAK-3809, OAK-3355.
In this specific case: the issue is more than 2 years old. Is it still relevant
at all?
WDYT?
> oak-solr-core test failures on Java 8
> -
>
> Key: OAK-1819
> URL: https://issues.apache.org/jira/browse/OAK-1819
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: solr
>Affects Versions: 1.0
> Environment: {noformat}
> Apache Maven 3.1.0 (893ca28a1da9d5f51ac03827af98bb730128f9f2; 2013-06-27
> 22:15:32-0400)
> Maven home: c:\Program Files\apache-maven-3.1.0
> Java version: 1.8.0, vendor: Oracle Corporation
> Java home: c:\Program Files\Java\jdk1.8.0\jre
> Default locale: en_US, platform encoding: Cp1252
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"
> {noformat}
>Reporter: Jukka Zitting
>Assignee: Tommaso Teofili
>Priority: Minor
> Labels: java8, jenkins, test
> Fix For: 1.8
>
>
> The following {{oak-solr-core}} test failures occur when building Oak with
> Java 8:
> {noformat}
> Failed tests:
>
> testNativeMLTQuery(org.apache.jackrabbit.oak.plugins.index.solr.query.SolrIndexQueryTest):
> expected: but was:
>
> testNativeMLTQueryWithStream(org.apache.jackrabbit.oak.plugins.index.solr.query.SolrIndexQueryTest):
> expected: but was:
> {noformat}
> The cause of this might well be something as simple as the test case
> incorrectly expecting a specific ordering of search results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-5022) add includeSubtreeOnDelete flag to OakEventFilter
[
https://issues.apache.org/jira/browse/OAK-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15620015#comment-15620015
]
Michael Marth commented on OAK-5022:
2 comments:
* Hooray for being able (in the application) to access property values of
deleted nodes. This is a long standing feature request - which should be rather
doable in MVCC.
* Re "includeSubtreeOnRemove": need to careful to not make this a very
expensive operation for large subtrees. Could this be handled lazily?
> add includeSubtreeOnDelete flag to OakEventFilter
> -
>
> Key: OAK-5022
> URL: https://issues.apache.org/jira/browse/OAK-5022
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: jcr
>Affects Versions: 1.5.12
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>
> (Originally reported as JCR-4037, but moved to Oak as a result of introducing
> the OakEventFilter in OAK-5013. From the original description: )
> In some cases of observation it would be useful to receive events of child
> node or properties of a parent/grandparent that was deleted. Currently (in
> Oak at least) one only receives a deleted event for the node that was deleted
> and none of the children.
> Suggesting to (re)introduce a flag, eg as follows to the
> JackrabbitEventFilter:
> {code}
> boolean includeSubtreeOnRemove;
> {code}
> (Open for any better name of course)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4882) Bottleneck in the asynchronous persistent cache
[ https://issues.apache.org/jira/browse/OAK-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15605437#comment-15605437 ] Michael Marth commented on OAK-4882: I see - thanks for explaining. In that case dropping put() seems reasonable (your option 2). But the way you describe it, it seems that the file maintenance is actually the problem (or rather the fact that maintenance blocks put operations). I thought that the pers-cache is a) append-only and b) uses rotating files. If so, it seems possible to keep adding entries even while files get rotated or old files removed... > Bottleneck in the asynchronous persistent cache > --- > > Key: OAK-4882 > URL: https://issues.apache.org/jira/browse/OAK-4882 > Project: Jackrabbit Oak > Issue Type: Bug > Components: cache, documentmk >Affects Versions: 1.5.10, 1.4.8 >Reporter: Tomek Rękawek >Assignee: Tomek Rękawek > Fix For: 1.6 > > > The class responsible for accepting new cache operations which will be > handled asynchronously is > [CacheActionDispatcher|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/persistentCache/async/CacheActionDispatcher.java]. > In case of a high load, when the queue is full (=1024 entries), the > [add()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/persistentCache/async/CacheActionDispatcher.java#L86] > method removes the oldest 256 entries. However, we can't afford losing the > updates (as it may result in having stale entries in the cache), so all the > removed entries are compacted into one big invalidate action. > The compaction action > ([CacheActionDispatcher#cleanTheQueue|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/persistentCache/async/CacheActionDispatcher.java#L97]) > still holds the lock taken in add() method, so threads which tries to add > something to the queue have to wait until cleanTheQueue() ends. > Maybe we can optimise the CacheActionDispatcher#add->cleanTheQueue part, so > it won't hold the lock for the whole time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4882) Bottleneck in the asynchronous persistent cache
[ https://issues.apache.org/jira/browse/OAK-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15605142#comment-15605142 ] Michael Marth commented on OAK-4882: I am not sure if dropping update/invalidate operations occasionally is a good idea (it makes it hard to understand the caching behaviour). IMO it would be clearer to make it explicit that the persistent cache is meant for immutable entries only - and actually get the benefits from that constraint, i.e. no worries about updates etc. If there are other use cases that do require mutable entries then those could maybe be treated in a separate pers-cache instance that explicitly allows mutable entries. > Bottleneck in the asynchronous persistent cache > --- > > Key: OAK-4882 > URL: https://issues.apache.org/jira/browse/OAK-4882 > Project: Jackrabbit Oak > Issue Type: Bug > Components: cache, documentmk >Affects Versions: 1.5.10, 1.4.8 >Reporter: Tomek Rękawek >Assignee: Tomek Rękawek > Fix For: 1.6 > > > The class responsible for accepting new cache operations which will be > handled asynchronously is > [CacheActionDispatcher|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/persistentCache/async/CacheActionDispatcher.java]. > In case of a high load, when the queue is full (=1024 entries), the > [add()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/persistentCache/async/CacheActionDispatcher.java#L86] > method removes the oldest 256 entries. However, we can't afford losing the > updates (as it may result in having stale entries in the cache), so all the > removed entries are compacted into one big invalidate action. > The compaction action > ([CacheActionDispatcher#cleanTheQueue|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/persistentCache/async/CacheActionDispatcher.java#L97]) > still holds the lock taken in add() method, so threads which tries to add > something to the queue have to wait until cleanTheQueue() ends. > Maybe we can optimise the CacheActionDispatcher#add->cleanTheQueue part, so > it won't hold the lock for the whole time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4882) Bottleneck in the asynchronous persistent cache
[ https://issues.apache.org/jira/browse/OAK-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604861#comment-15604861 ] Michael Marth commented on OAK-4882: fwiw: IMHO the persistent cache should be fully tuned to take advantage of the immutability of revisions (i.e. remove update operations) > Bottleneck in the asynchronous persistent cache > --- > > Key: OAK-4882 > URL: https://issues.apache.org/jira/browse/OAK-4882 > Project: Jackrabbit Oak > Issue Type: Bug > Components: cache, documentmk >Affects Versions: 1.5.10, 1.4.8 >Reporter: Tomek Rękawek >Assignee: Tomek Rękawek > Fix For: 1.6 > > > The class responsible for accepting new cache operations which will be > handled asynchronously is > [CacheActionDispatcher|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/persistentCache/async/CacheActionDispatcher.java]. > In case of a high load, when the queue is full (=1024 entries), the > [add()|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/persistentCache/async/CacheActionDispatcher.java#L86] > method removes the oldest 256 entries. However, we can't afford losing the > updates (as it may result in having stale entries in the cache), so all the > removed entries are compacted into one big invalidate action. > The compaction action > ([CacheActionDispatcher#cleanTheQueue|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/persistentCache/async/CacheActionDispatcher.java#L97]) > still holds the lock taken in add() method, so threads which tries to add > something to the queue have to wait until cleanTheQueue() ends. > Maybe we can optimise the CacheActionDispatcher#add->cleanTheQueue part, so > it won't hold the lock for the whole time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4903) Async uploads in S3 causes issues in a cluster
[ https://issues.apache.org/jira/browse/OAK-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604643#comment-15604643 ] Michael Marth commented on OAK-4903: IIUC the missing binary problem occurs in deployments where both, the DS (S3) *and* the node store (e.g. Mongo or RDB), are shared. If so: how about using Mongo (or RDB) as a shared cache while the upload is in progress? > Async uploads in S3 causes issues in a cluster > -- > > Key: OAK-4903 > URL: https://issues.apache.org/jira/browse/OAK-4903 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Critical > > S3DataStore and CachingFDS through the CachingDataStore enable async uploads. > This causes problems in clustered setups where uploads can sometimes be > visible after a delay. During this time any request for the corresponding > asset/file would return errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4043) Oak run checkpoints needs to account for multiple index lanes
[
https://issues.apache.org/jira/browse/OAK-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-4043:
---
Description:
Oak run {{checkpoints rm-unreferenced}} [0] currently is hardcoded on a single
checkpoint reference (the default one). Now is it possible to add multiple
lanes, which we already did in AEM, but the checkpoint tool is blissfully
unaware of this and it might trigger a full reindex following offline
compaction.
fyi [~edivad], [~chetanm]
[0] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run#checkpoints
was:
Oak run {{checkpoints rm-unreferenced}} [0] currently is hardcoded on a single
checkpoint reference (the default one). Now is it possible to add multiple
lanes, which we already did in AEM, but the checkpoint tool is blissfully
unaware of this and it might trigger a full reindex following offline
compaction.
This needs fixing before the big 1.4 release, so I'm marking it as a blocker.
fyi [~edivad], [~chetanm]
[0] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run#checkpoints
> Oak run checkpoints needs to account for multiple index lanes
> -
>
> Key: OAK-4043
> URL: https://issues.apache.org/jira/browse/OAK-4043
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, run
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
>Priority: Critical
> Labels: candidate_oak_1_4
> Fix For: 1.6, 1.5.11
>
> Attachments: OAK-4043.patch
>
>
> Oak run {{checkpoints rm-unreferenced}} [0] currently is hardcoded on a
> single checkpoint reference (the default one). Now is it possible to add
> multiple lanes, which we already did in AEM, but the checkpoint tool is
> blissfully unaware of this and it might trigger a full reindex following
> offline compaction.
> fyi [~edivad], [~chetanm]
> [0] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run#checkpoints
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4043) Oak run checkpoints needs to account for multiple index lanes
[
https://issues.apache.org/jira/browse/OAK-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-4043:
---
Priority: Critical (was: Blocker)
> Oak run checkpoints needs to account for multiple index lanes
> -
>
> Key: OAK-4043
> URL: https://issues.apache.org/jira/browse/OAK-4043
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, run
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
>Priority: Critical
> Labels: candidate_oak_1_4
> Fix For: 1.6, 1.5.11
>
> Attachments: OAK-4043.patch
>
>
> Oak run {{checkpoints rm-unreferenced}} [0] currently is hardcoded on a
> single checkpoint reference (the default one). Now is it possible to add
> multiple lanes, which we already did in AEM, but the checkpoint tool is
> blissfully unaware of this and it might trigger a full reindex following
> offline compaction.
> This needs fixing before the big 1.4 release, so I'm marking it as a blocker.
> fyi [~edivad], [~chetanm]
> [0] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run#checkpoints
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4043) Oak run checkpoints needs to account for multiple index lanes
[
https://issues.apache.org/jira/browse/OAK-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-4043:
---
Issue Type: Improvement (was: Bug)
> Oak run checkpoints needs to account for multiple index lanes
> -
>
> Key: OAK-4043
> URL: https://issues.apache.org/jira/browse/OAK-4043
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, run
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
>Priority: Blocker
> Labels: candidate_oak_1_4
> Fix For: 1.6, 1.5.11
>
> Attachments: OAK-4043.patch
>
>
> Oak run {{checkpoints rm-unreferenced}} [0] currently is hardcoded on a
> single checkpoint reference (the default one). Now is it possible to add
> multiple lanes, which we already did in AEM, but the checkpoint tool is
> blissfully unaware of this and it might trigger a full reindex following
> offline compaction.
> This needs fixing before the big 1.4 release, so I'm marking it as a blocker.
> fyi [~edivad], [~chetanm]
> [0] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run#checkpoints
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4810) FileDataStore: support SHA-2
[ https://issues.apache.org/jira/browse/OAK-4810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15496551#comment-15496551 ] Michael Marth commented on OAK-4810: Re implementation: could this be a special case of OAK-3140? In OAK-3140 several DSs are defined. In this case, the SHA-256 DS would be read-write and the SHA-1 DS would be read-only. > FileDataStore: support SHA-2 > > > Key: OAK-4810 > URL: https://issues.apache.org/jira/browse/OAK-4810 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: blob >Reporter: Thomas Mueller > > The FileDataStore currently uses SHA-1, but that algorithm is deprecated. We > should support other algorithms as well (mainly SHA-256). > Migration should be painless (no long downtime). I think default for writing > (if not configured explicitly) could still be SHA-1. But when reading, > SHA-256 should also be supported (depending on the identifier). That way, the > new Oak version for all repositories (in a cluster + shared datastore) can be > installed "slowly". > After all repositories are running with the new Oak version, the > configuration for SHA-256 can be enabled. That way, SHA-256 is used for new > binaries. Both SHA-1 and SHA-256 are supported for reading. > One potential downside is deduplication would suffer a bit if a new Blob with > same content is added again as digest based match would fail. That can be > mitigated by computing 2 types of digest if need arises. The downsides are > some additional file operations and CPU, and slower migration to SHA-256. > Some other open questions: > * While we are at it, it might makes senses to additionally support SHA-3 and > other algorithms (make it configurable). But the length of the identifier > alone might then not be enough information to know what algorithm is used, so > maybe add a prefix. > * The number of subdirectory levels: should we keep it as is, or should we > reduce it (for example one level less). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-1312) Bundle nodes into a document
[ https://issues.apache.org/jira/browse/OAK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431529#comment-15431529 ] Michael Marth commented on OAK-1312: It is a bit hard to understand the quoted part, but: afaiu this proposal is about bundling nodes on storage layer. Access control happens on oak-core - so I wonder if access control checks would in any way affected by this change? > Bundle nodes into a document > > > Key: OAK-1312 > URL: https://issues.apache.org/jira/browse/OAK-1312 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Marcel Reutegger >Assignee: Chetan Mehrotra > Labels: performance > Fix For: 1.6 > > Attachments: OAK-1312-review-v1.diff > > > For very fine grained content with many nodes and only few properties per > node it would be more efficient to bundle multiple nodes into a single > MongoDB document. Mostly reading would benefit because there are less > roundtrips to the backend. At the same time storage footprint would be lower > because metadata overhead is per document. > Feature branch - > https://github.com/chetanmeh/jackrabbit-oak/compare/trunk...chetanmeh:OAK-1312 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-1312) Bundle nodes into a document
[ https://issues.apache.org/jira/browse/OAK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431521#comment-15431521 ] Michael Marth commented on OAK-1312: [~chetanm] - don't want to be a pain, but since this (significant) improvement is motivated by performance improvements IMO we should not call it done without some benchmarks. WDYT? > Bundle nodes into a document > > > Key: OAK-1312 > URL: https://issues.apache.org/jira/browse/OAK-1312 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Marcel Reutegger >Assignee: Chetan Mehrotra > Labels: performance > Fix For: 1.6 > > Attachments: OAK-1312-review-v1.diff > > > For very fine grained content with many nodes and only few properties per > node it would be more efficient to bundle multiple nodes into a single > MongoDB document. Mostly reading would benefit because there are less > roundtrips to the backend. At the same time storage footprint would be lower > because metadata overhead is per document. > Feature branch - > https://github.com/chetanmeh/jackrabbit-oak/compare/trunk...chetanmeh:OAK-1312 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4412) Lucene hybrid index
[
https://issues.apache.org/jira/browse/OAK-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405790#comment-15405790
]
Michael Marth commented on OAK-4412:
LGTM. A comment on step 4 (pruning): could we use the approach taken in the
persisted cache, i.e. create rolling (local) index files and deleting the old
ones on roll-over?
> Lucene hybrid index
> ---
>
> Key: OAK-4412
> URL: https://issues.apache.org/jira/browse/OAK-4412
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
> Fix For: 1.6
>
> Attachments: OAK-4412.patch
>
>
> When running Oak in a cluster, each write operation is expensive. After
> performing some stress-tests with a geo-distributed Mongo cluster, we've
> found out that updating property indexes is a large part of the overall
> traffic.
> The asynchronous index would be an answer here (as the index update won't be
> made in the client request thread), but the AEM requires the updates to be
> visible immediately in order to work properly.
> The idea here is to enhance the existing asynchronous Lucene index with a
> synchronous, locally-stored counterpart that will persist only the data since
> the last Lucene background reindexing job.
> The new index can be stored in memory or (if necessary) in MMAPed local
> files. Once the "main" Lucene index is being updated, the local index will be
> purged.
> Queries will use an union of results from the {{lucene}} and
> {{lucene-memory}} indexes.
> The {{lucene-memory}} index, as a local stored entity, will be updated using
> an observer, so it'll get both local and remote changes.
> The original idea has been suggested by [~chetanm] in the discussion for the
> OAK-4233.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4596) The segment info record allows empty segments to be leaked
[ https://issues.apache.org/jira/browse/OAK-4596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-4596: --- Issue Type: Bug (was: Improvement) > The segment info record allows empty segments to be leaked > -- > > Key: OAK-4596 > URL: https://issues.apache.org/jira/browse/OAK-4596 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-tar >Reporter: Francesco Mari >Assignee: Francesco Mari > Fix For: Segment Tar 0.0.8 > > > The flush thread writes to disk dirty segment buffers every 5 seconds. A > segment buffer is defined to be dirty if the amount of record bytes it > contains is greater than zero. Since we always write an info record in every > segment at the moment they are created, no segment buffer is ever empty. This > means that the flush thread writes to disk almost empty segments every 5 > seconds, that waste an additional amount of space because of the padding > required by the TAR format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4596) The segment info record allows empty segments to be leaked
[ https://issues.apache.org/jira/browse/OAK-4596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-4596: --- Issue Type: Improvement (was: Bug) > The segment info record allows empty segments to be leaked > -- > > Key: OAK-4596 > URL: https://issues.apache.org/jira/browse/OAK-4596 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-tar >Reporter: Francesco Mari >Assignee: Francesco Mari > Fix For: Segment Tar 0.0.8 > > > The flush thread writes to disk dirty segment buffers every 5 seconds. A > segment buffer is defined to be dirty if the amount of record bytes it > contains is greater than zero. Since we always write an info record in every > segment at the moment they are created, no segment buffer is ever empty. This > means that the flush thread writes to disk almost empty segments every 5 > seconds, that waste an additional amount of space because of the padding > required by the TAR format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-1312) Bundle nodes into a document
[ https://issues.apache.org/jira/browse/OAK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387479#comment-15387479 ] Michael Marth commented on OAK-1312: thanks - makes sense! > Bundle nodes into a document > > > Key: OAK-1312 > URL: https://issues.apache.org/jira/browse/OAK-1312 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Marcel Reutegger >Assignee: Chetan Mehrotra > Labels: performance > Fix For: 1.6 > > > For very fine grained content with many nodes and only few properties per > node it would be more efficient to bundle multiple nodes into a single > MongoDB document. Mostly reading would benefit because there are less > roundtrips to the backend. At the same time storage footprint would be lower > because metadata overhead is per document. > Feature branch - > https://github.com/chetanmeh/jackrabbit-oak/compare/trunk...chetanmeh:OAK-1312 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-1312) Bundle nodes into a document
[ https://issues.apache.org/jira/browse/OAK-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387390#comment-15387390 ] Michael Marth commented on OAK-1312: [~chetanm], description says "Mostly reading would benefit because there are less roundtrips to the backend". I think theoretically reads and writes could be affected positively or negatively by this change. That might depend on read patterns and also on the deployment topology (e.g. latency between MongoDB and Oak). So, what I wonder: do you have specific test cases in mind to evaluate the effect of this change? > Bundle nodes into a document > > > Key: OAK-1312 > URL: https://issues.apache.org/jira/browse/OAK-1312 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Marcel Reutegger >Assignee: Chetan Mehrotra > Labels: performance > Fix For: 1.6 > > > For very fine grained content with many nodes and only few properties per > node it would be more efficient to bundle multiple nodes into a single > MongoDB document. Mostly reading would benefit because there are less > roundtrips to the backend. At the same time storage footprint would be lower > because metadata overhead is per document. > Feature branch - > https://github.com/chetanmeh/jackrabbit-oak/compare/trunk...chetanmeh:OAK-1312 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4428) Optimize RevisionVector methods
[
https://issues.apache.org/jira/browse/OAK-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-4428:
---
Summary: Optimize RevisionVector methods (was: Optmize RevisionVector
methods)
> Optimize RevisionVector methods
> ---
>
> Key: OAK-4428
> URL: https://issues.apache.org/jira/browse/OAK-4428
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Labels: performance
> Fix For: 1.6
>
> Attachments: OAK-4428-v1.patch, hot-methods-1.png
>
>
> {{RevisionVector}} is used in very critical paths and we should look into
> optimzing some of its critical method
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4422) support cluster for FileBlobStore
[ https://issues.apache.org/jira/browse/OAK-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314362#comment-15314362 ] Michael Marth commented on OAK-4422: The term "clustering" primarily applies to the node store. Do you use TarMK? If yes, what is the deployment architecture? Different independent TarMK-based instances sharing one File-system based Datastore (where the binaries live)? > support cluster for FileBlobStore > - > > Key: OAK-4422 > URL: https://issues.apache.org/jira/browse/OAK-4422 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob >Affects Versions: 1.4.3 >Reporter: Marco Piovesana > > I'm using Oak in a system where the user can store arbitrary large binary > files and because of that I thought the best option was to use the > FileBlobStore as storage subsystem. > Now I need to port this solution on a cluster environment, but i saw that > clustering is supported only for Mongo and RDBMS storage systems. Is there > any plan to suppor it also for the Blob storage? There's a better option? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4313) QueryImpl should avoid traversal with queries containing native constraints
[
https://issues.apache.org/jira/browse/OAK-4313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-4313:
---
Priority: Blocker (was: Major)
> QueryImpl should avoid traversal with queries containing native constraints
> ---
>
> Key: OAK-4313
> URL: https://issues.apache.org/jira/browse/OAK-4313
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: query
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Blocker
> Labels: candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.6, 1.4.2
>
> Attachments: OAK-4313.patch
>
>
> If no index supports suggestion (or spellcheck or similar) query, then a
> query like
> {noformat}
> SELECT * from [nt:base] where SUGGEST('test')
> {noformat}
> shouldn't get traversing index
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4313) QueryImpl should avoid traversal with queries containing native constraints
[
https://issues.apache.org/jira/browse/OAK-4313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-4313:
---
Fix Version/s: 1.4.2
> QueryImpl should avoid traversal with queries containing native constraints
> ---
>
> Key: OAK-4313
> URL: https://issues.apache.org/jira/browse/OAK-4313
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: query
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
> Labels: candidate_oak_1_2, candidate_oak_1_4
> Fix For: 1.6, 1.4.2
>
> Attachments: OAK-4313.patch
>
>
> If no index supports suggestion (or spellcheck or similar) query, then a
> query like
> {noformat}
> SELECT * from [nt:base] where SUGGEST('test')
> {noformat}
> shouldn't get traversing index
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4245) Decide on a final name for oak-segment-next
[
https://issues.apache.org/jira/browse/OAK-4245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253762#comment-15253762
]
Michael Marth commented on OAK-4245:
how about "oak-segment-file" (to denote that the implementation stores the
segments in files, as opposed to some other possible storage technologies)?
> Decide on a final name for oak-segment-next
> ---
>
> Key: OAK-4245
> URL: https://issues.apache.org/jira/browse/OAK-4245
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Blocker
> Fix For: 1.5.2
>
>
> We should come up with a definite and final name for {{oak-segment-name}}.
> Making this a blocker to avoid releasing with the current WIP name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4124) Persistent cache: consider using callstack blocking
[
https://issues.apache.org/jira/browse/OAK-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195103#comment-15195103
]
Michael Marth commented on OAK-4124:
For this case:
{quote}
2. high load -> consumer slower than producers -> queue is full -> put()s done
synchronously, to avoid flushing the queue and losing cache values
{quote}
why don't we just drop the entries? Persistent cache is "best effort" anyway.
> Persistent cache: consider using callstack blocking
> ---
>
> Key: OAK-4124
> URL: https://issues.apache.org/jira/browse/OAK-4124
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: cache
>Reporter: Tomek Rękawek
> Labels: performance
> Fix For: 1.6
>
> Attachments: OAK-4124.patch
>
>
> In the current implementation of the asynchronous add operation in
> persistence cache, data are added to a queue and if the queue is full, the
> oldest entries are removed (see OAK-2761).
> It may deteriorate the system performance in some cases (OAK-4123). Let's
> consider a different approach, in which we block the caller thread in case
> the queue is full.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3933) potential improvements to membership management
[ https://issues.apache.org/jira/browse/OAK-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3933: --- Epic Name: potential improvements to membership management > potential improvements to membership management > --- > > Key: OAK-3933 > URL: https://issues.apache.org/jira/browse/OAK-3933 > Project: Jackrabbit Oak > Issue Type: Epic > Components: core >Reporter: Julian Reschke >Assignee: angela > > Affected methods are > - Group.isMember(Authorizable) > - Group.isDeclaredMember(Authorizable) > - Group.addMember(Authorizable) > - Group.addMembers(String...) > - Group.removeMember(Authorizable) > - Group.removeMembers(String...) > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4123) Persistent cache: allow to configure the add data concurrency
[ https://issues.apache.org/jira/browse/OAK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-4123: --- Fix Version/s: 1.6 > Persistent cache: allow to configure the add data concurrency > - > > Key: OAK-4123 > URL: https://issues.apache.org/jira/browse/OAK-4123 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: cache >Reporter: Tomek Rękawek > Fix For: 1.6, 1.4.1 > > Attachments: OAK-4123.patch > > > During operations that creates and reads a large number of nodes (eg. > indexing content) it may happen that there's more items in the asynchronous > queue (introduced in OAK-2761) than the queue consumer can handle. As a > result, the queue is purged and items are not saved in the cache, which makes > the overall performance worse. > An easy fix is to add a property that allows to switch between async/sync > mode. By default, it should be synchronous. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4112) Replace the query exclusive lock with a cache tracker
[
https://issues.apache.org/jira/browse/OAK-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-4112:
---
Labels: performance (was: )
> Replace the query exclusive lock with a cache tracker
> -
>
> Key: OAK-4112
> URL: https://issues.apache.org/jira/browse/OAK-4112
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk, mongomk
>Reporter: Tomek Rękawek
> Labels: performance
> Fix For: 1.6
>
> Attachments: OAK-4112.patch
>
>
> The {{MongoDocumentStore#query()}} method uses an expensive
> {{TreeLock#acquireExclusive}} method, introduced in OAK-1897 to avoid caching
> outdated documents.
> It should be possible to avoid acquiring the exclusive lock, by tracking the
> cache changes that occurs during the Mongo find() operation. When the find()
> is done, we can update the cache with the received documents if they haven't
> been invalidated in the meantime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3850) Collect and expose Persistent Cache stats
[ https://issues.apache.org/jira/browse/OAK-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167348#comment-15167348 ] Michael Marth commented on OAK-3850: IMHO 10% degredation is far too much. I opt for setting StatististicsProvider.NOOP by default > Collect and expose Persistent Cache stats > - > > Key: OAK-3850 > URL: https://issues.apache.org/jira/browse/OAK-3850 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Teodor Rosu >Assignee: Thomas Mueller > Fix For: 1.4 > > Attachments: OAK-3850-v0.patch, OAK-3850-v1.patch, OAK-3850-v2.patch, > test-performance-v0.patch > > > Expose persistent cache statistics ( see: [Guava > CacheStats|http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/cache/CacheStats.html] > ) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4030) DocumentNodeStore: required server time accuracy
[ https://issues.apache.org/jira/browse/OAK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-4030: --- Labels: resilience (was: ) > DocumentNodeStore: required server time accuracy > > > Key: OAK-4030 > URL: https://issues.apache.org/jira/browse/OAK-4030 > Project: Jackrabbit Oak > Issue Type: Bug > Components: documentmk >Reporter: Julian Reschke > Labels: resilience > Fix For: 1.4 > > > The DocumentNodeStore currently requires that the local time and the > persistence time differ at most 2 seconds. > I recently tried to run a cluster with two Windows machines, and despite them > being configured to use the same NTP service, they were still 4..5 s off. > https://blogs.technet.microsoft.com/askds/2007/10/23/high-accuracy-w32time-requirements/ > seems to confirm that by default, Windows can't provide the required > accuracy. > One workaround seems to be to install custom ntp clients; but do we really > want to require this? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4030) DocumentNodeStore: required server time accuracy
[ https://issues.apache.org/jira/browse/OAK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-4030: --- Fix Version/s: 1.4 > DocumentNodeStore: required server time accuracy > > > Key: OAK-4030 > URL: https://issues.apache.org/jira/browse/OAK-4030 > Project: Jackrabbit Oak > Issue Type: Bug > Components: documentmk >Reporter: Julian Reschke > Labels: resilience > Fix For: 1.4 > > > The DocumentNodeStore currently requires that the local time and the > persistence time differ at most 2 seconds. > I recently tried to run a cluster with two Windows machines, and despite them > being configured to use the same NTP service, they were still 4..5 s off. > https://blogs.technet.microsoft.com/askds/2007/10/23/high-accuracy-w32time-requirements/ > seems to confirm that by default, Windows can't provide the required > accuracy. > One workaround seems to be to install custom ntp clients; but do we really > want to require this? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4043) Oak run checkpoints needs to account for multiple index lanes
[
https://issues.apache.org/jira/browse/OAK-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-4043:
---
Priority: Critical (was: Blocker)
> Oak run checkpoints needs to account for multiple index lanes
> -
>
> Key: OAK-4043
> URL: https://issues.apache.org/jira/browse/OAK-4043
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, run
>Reporter: Alex Parvulescu
>Priority: Critical
> Fix For: 1.4, 1.3.17
>
>
> Oak run {{checkpoints rm-unreferenced}} [0] currently is hardcoded on a
> single checkpoint reference (the default one). Now is it possible to add
> multiple lanes, which we already did in AEM, but the checkpoint tool is
> blissfully unaware of this and it might trigger a full reindex following
> offline compaction.
> This needs fixing before the big 1.4 release, so I'm marking it as a blocker.
> fyi [~edivad], [~chetanm]
> [0] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run#checkpoints
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3234) LIRS cache: possible deadlock while loading an entry
[ https://issues.apache.org/jira/browse/OAK-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3234: --- Labels: candidate_oak_1_0 candidate_oak_1_2 (was: ) > LIRS cache: possible deadlock while loading an entry > > > Key: OAK-3234 > URL: https://issues.apache.org/jira/browse/OAK-3234 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core >Reporter: Thomas Mueller >Assignee: Thomas Mueller > Labels: candidate_oak_1_0, candidate_oak_1_2 > Fix For: 1.3.4 > > Attachments: OAK-3234-backported.patch > > > If multiple threads concurrently load entries while they are already loading > entries (using a cache loader or callable), then it is possible to get into a > deadlock. For example: > * Thread 1 loads entry A, so it is calling the loader... > * Thread 2 loads entry B, so it is calling the loader... > * Thread 1 (within the loader) tries to load entry B, so it waits for thread > 2... > * Thread 2 (within the loader) tries to load entry A, so it waits for thread > 1... > One solution is to detect the case that the current thread is already loading > an entry, and then instead of waiting for the other thread, it can also load > the entry. A small optimization for that is to only load the entry (not > waiting for the other thread) if the hash code of the key is smaller or equal > the entry that this thread is loading. So that there is a clear order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3980) SharedDataStore - Set the unique repository ID using sling if configured
[ https://issues.apache.org/jira/browse/OAK-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3980: --- Fix Version/s: (was: 1.4) 1.6 > SharedDataStore - Set the unique repository ID using sling if configured > > > Key: OAK-3980 > URL: https://issues.apache.org/jira/browse/OAK-3980 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob >Reporter: Amit Jain > Labels: resilience > Fix For: 1.6 > > > As per discussion on the issue OAK-3935 [1] & [2] we could use sling to get a > repository ID injected obviating the need for a Osgi config param override in > case of a cloned instance. > [1] - > https://issues.apache.org/jira/browse/OAK-3935?focusedCommentId=15122957&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15122957 > [2] - > https://issues.apache.org/jira/browse/OAK-3935?focusedCommentId=15126096&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15126096 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-1962) Fix and Improve Locking
[ https://issues.apache.org/jira/browse/OAK-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-1962: --- Fix Version/s: (was: 1.4) 1.6 > Fix and Improve Locking > --- > > Key: OAK-1962 > URL: https://issues.apache.org/jira/browse/OAK-1962 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core, jcr >Affects Versions: 1.0 >Reporter: angela > Labels: technical_debt > Fix For: 1.6 > > > as discussed during our biweekly planning session, we are having the > following locking related use case in our product: > a page (which is an aggregation of nodes and properties) must be freezed > (prevented from being edited/modified/deleted) during the period of a > dedicated review (which in our case is triggered by a workflow) before the > page then is either published or reopened for further changes. > for this feature locking would be a perfect match but with the current > implementation in oak it's not possible to remove the lock token from a > given session and make sure it's no longer recognizes as lock owner; in > particular with the simple-locking feature which we introduced in order to > cope with the situation that in our product sessions are living just for the > time of a single request. > is was therefore thinking about ways to address this, keeping the > simple-locking requirement in mind while at the same time improving > compliance with the JCR specification. my suggestion as follows: > - use a dedicated hidden and write protected location the repository to store > the lock information independent of the protected properties defined by > mix:lockable which are just for information purpose (that would be the > replacement for the lock-related file we had in jackrabbit 2.x) > - format: simple lookup consisting of userId + associated lock tokens > - in case of simple locking LockManager#getLockTokens would make use of that > map even if the lock tokens have NOT been explicitly added to the Session > either upon LockManager#lock or LockManager#addLockToken > - in the default (non-simple case) LockManager#getLockTokens would work as > specified and the lookup would only be used for maintaining and enforcing the > lock related information) > - LockManager#removeLockToken removes the token from the lookup thus > effectively revoking the lock ownership from the given Session and thus > preventing it from performing further editing... even in the simple-locking > case > - LockManager#addLockToken results in a modification of the lookup table as > well depending on the API contract (i.e. verifying if the lock token would be > shared... i don't remember exactly if that's accepted by the specification) > - LockManager#isLockOwner no longer needs to perform a cheap hack comparing > the jcr:lockOwner property with the sessions userId but could really enforce > ownership based on the internal lock information stored in the repository in > which case getLockTokens would really reflect the ownership for open-scoped > locks; furthermore the 'jcr:lockOwner' information could be to what is > specified by the API consumer upon LockManager#lock as it is no longer used > for enforcing the locks. > in addition: > - we could generated proper lock tokens instead of just using the node id > and again keep those in a hidden (or otherwise protected) location in the > system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-495) Run TCK with local namespace remapping
[ https://issues.apache.org/jira/browse/OAK-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-495: -- Fix Version/s: (was: 1.4) 1.6 > Run TCK with local namespace remapping > -- > > Key: OAK-495 > URL: https://issues.apache.org/jira/browse/OAK-495 > Project: Jackrabbit Oak > Issue Type: Task > Components: it >Reporter: angela > Labels: test > Fix For: 1.6 > > > jukka suggested that we run the TCK with session local namespace > remappings in order to be able to detect more issues we might have > with usage of oak vs jcr names. > +1 from my side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3980) SharedDataStore - Set the unique repository ID using sling if configured
[ https://issues.apache.org/jira/browse/OAK-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3980: --- Labels: resilience (was: ) > SharedDataStore - Set the unique repository ID using sling if configured > > > Key: OAK-3980 > URL: https://issues.apache.org/jira/browse/OAK-3980 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob >Reporter: Amit Jain > Labels: resilience > Fix For: 1.4 > > > As per discussion on the issue OAK-3935 [1] & [2] we could use sling to get a > repository ID injected obviating the need for a Osgi config param override in > case of a cloned instance. > [1] - > https://issues.apache.org/jira/browse/OAK-3935?focusedCommentId=15122957&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15122957 > [2] - > https://issues.apache.org/jira/browse/OAK-3935?focusedCommentId=15126096&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15126096 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3717) Make it possible to declare SynonymFilter within Analyzer with WN dictionary
[
https://issues.apache.org/jira/browse/OAK-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3717:
---
Fix Version/s: (was: 1.4)
1.6
> Make it possible to declare SynonymFilter within Analyzer with WN dictionary
>
>
> Key: OAK-3717
> URL: https://issues.apache.org/jira/browse/OAK-3717
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.6
>
>
> Currently one can compose Lucene Analyzers via
> [composition|http://jackrabbit.apache.org/oak/docs/query/lucene.html#Create_analyzer_via_composition]
> within an index definition. It'd be good to be able to also use
> {{SynonymFIlter}} in there, eventually decorated with
> {{WordNetSynonymParser}} to leverage WordNet synonym files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3488) LastRevRecovery for self async?
[
https://issues.apache.org/jira/browse/OAK-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3488:
---
Fix Version/s: (was: 1.4)
1.6
> LastRevRecovery for self async?
> ---
>
> Key: OAK-3488
> URL: https://issues.apache.org/jira/browse/OAK-3488
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: documentmk
>Reporter: Julian Reschke
>Assignee: Marcel Reutegger
> Labels: resilience
> Fix For: 1.6
>
> Attachments: OAK-3488.patch
>
>
> Currently, when a cluster node starts and discovers that it wasn't properly
> shutdown, it first runs the complete LastRevRecovery and only continues
> startup when done.
> However, when it fails to acquire the recovery lock, which implies that a
> different cluster node is already running the recovery on its behalf, it
> simply skips recovery and continues startup?
> So what is it? Is running the recovery before proceeding critical or not? If
> it is, this code in {{LastRevRecoveryAgent}} needs to change:
> {code}
> //TODO What if recovery is being performed for current clusterNode by
> some other node
> //should we halt the startup
> if(!lockAcquired){
> log.info("Last revision recovery already being performed by some
> other node. " +
> "Would not attempt recovery");
> return 0;
> }
> {code}
> If it's not critical, we may want to run the recovery always asynchronously.
> cc [~mreutegg] and [~chetanm]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3268) Improve datastore resilience
[ https://issues.apache.org/jira/browse/OAK-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131903#comment-15131903 ] Michael Marth commented on OAK-3268: [~amitjain], all issues in this epic are done: should we close the epic as done? > Improve datastore resilience > > > Key: OAK-3268 > URL: https://issues.apache.org/jira/browse/OAK-3268 > Project: Jackrabbit Oak > Issue Type: Epic > Components: blob >Reporter: Michael Marth >Assignee: Amit Jain >Priority: Critical > Labels: resilience > > As discussed bilaterally grouping the improvements for datastore resilience > in this issue for easier tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3952) BackgroundObserver close should wait for stop to be processed
[
https://issues.apache.org/jira/browse/OAK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123197#comment-15123197
]
Michael Marth commented on OAK-3952:
[~chetanm], if this issue "only" leads to exceptions in the log, but no
corruption on shutdown then I think it is nice-to-have in 1.4, but not a must.
[~mduerig], thoughts?
> BackgroundObserver close should wait for stop to be processed
> -
>
> Key: OAK-3952
> URL: https://issues.apache.org/jira/browse/OAK-3952
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.4
>
>
> {{BackgroundObserver}} close method currently says that it does not wait for
> thread to finish. This can lead to situation when during shutdown the backing
> {{NodeStore}} goes away while the background thread is still busy with diff
> and that would lead to exceptions in the log.
> As being done in OAK-3923 the close call should wait for {{STOP}} signal to
> be processed before returning from close call.
> On side note we should also provide an option where the close call should
> wait for the queue to be processed instead of just silently clearing the
> queue which can lead to potential loss of important local events. Note that
> can happen for proper ordered shutdown!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3071) Add a compound index for _modified + _id
[
https://issues.apache.org/jira/browse/OAK-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3071:
---
Priority: Blocker (was: Major)
> Add a compound index for _modified + _id
>
>
> Key: OAK-3071
> URL: https://issues.apache.org/jira/browse/OAK-3071
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: mongomk
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Blocker
> Labels: performance, resilience
> Fix For: 1.4, 1.3.15
>
>
> As explained in OAK-1966 diff logic makes a call like
> bq. db.nodes.find({ _id: { $gt: "3:/content/foo/01/", $lt:
> "3:/content/foo010" }, _modified: { $gte: 1405085300 } }).sort({_id:1})
> For better and deterministic query performance we would need to create a
> compound index like \{_modified:1, _id:1\}. This index would ensure that
> Mongo does not have to perform object scan while evaluating such a query.
> Care must be taken that index is only created by default for fresh setup. For
> existing setup we should expose a JMX operation which can be invoked by
> system admin to create the required index as per maintenance window
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3071) Add a compound index for _modified + _id
[
https://issues.apache.org/jira/browse/OAK-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3071:
---
Fix Version/s: 1.3.15
> Add a compound index for _modified + _id
>
>
> Key: OAK-3071
> URL: https://issues.apache.org/jira/browse/OAK-3071
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: mongomk
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Labels: performance, resilience
> Fix For: 1.4, 1.3.15
>
>
> As explained in OAK-1966 diff logic makes a call like
> bq. db.nodes.find({ _id: { $gt: "3:/content/foo/01/", $lt:
> "3:/content/foo010" }, _modified: { $gte: 1405085300 } }).sort({_id:1})
> For better and deterministic query performance we would need to create a
> compound index like \{_modified:1, _id:1\}. This index would ensure that
> Mongo does not have to perform object scan while evaluating such a query.
> Care must be taken that index is only created by default for fresh setup. For
> existing setup we should expose a JMX operation which can be invoked by
> system admin to create the required index as per maintenance window
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3629) Index corruption seen with CopyOnRead when index defnition is recreated
[
https://issues.apache.org/jira/browse/OAK-3629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3629:
---
Fix Version/s: (was: 1.4)
1.3.15
> Index corruption seen with CopyOnRead when index defnition is recreated
> ---
>
> Key: OAK-3629
> URL: https://issues.apache.org/jira/browse/OAK-3629
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.3.15
>
>
> CopyOnRead logic relies on {{reindexCount}} to determine the name of
> directory in which index files would be copied. In normal flow if the index
> is reindexed then this count would get increased and newer index files would
> get copied to a new directory.
> However if the index definition node gets recreated due to some deployment
> process then this count gets reset to 0. Due to which newly created index
> files from reindexing would start getting copied to already existing
> directory and that can lead to corruption.
> So what happened here was
> # System started with index definition I1 and indexing got complete with
> index files saved under index/hash(indexpath)/1 (where 1 is current reindex
> count)
> # A new index definition package was deployed which reset the index count.
> Now reindex happened again and the CopyOnRead logic per current design reused
> the existing index directory. And it so happens that Lucene create file with
> same name and same size but different content. This trips the CopyOnRead
> defense of length based index corruption check and thus cause new lucene
> index to corrupt
> *Note that here corruption is transient i.e. persisted index is not
> corrupted*. Just that locally copied index gets corrupted. Cleaning up the
> index directory would fix the issue and that can be used as a workaround.
> *Fix*
> After discussing with [~tmueller] following approach can be used.
> Instead of relying on reindex count we can maintain a hidden randomly
> generated uuid and store it in the index config. This would be used to derive
> the name of directory on filesystem. If the index definition gets reset then
> the uuid can be regenerated.
> *Workaround*
> Clean the directory used by CopyOnRead which is /index before
> restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3629) Index corruption seen with CopyOnRead when index defnition is recreated
[
https://issues.apache.org/jira/browse/OAK-3629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3629:
---
Priority: Blocker (was: Major)
> Index corruption seen with CopyOnRead when index defnition is recreated
> ---
>
> Key: OAK-3629
> URL: https://issues.apache.org/jira/browse/OAK-3629
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Blocker
> Fix For: 1.3.15
>
>
> CopyOnRead logic relies on {{reindexCount}} to determine the name of
> directory in which index files would be copied. In normal flow if the index
> is reindexed then this count would get increased and newer index files would
> get copied to a new directory.
> However if the index definition node gets recreated due to some deployment
> process then this count gets reset to 0. Due to which newly created index
> files from reindexing would start getting copied to already existing
> directory and that can lead to corruption.
> So what happened here was
> # System started with index definition I1 and indexing got complete with
> index files saved under index/hash(indexpath)/1 (where 1 is current reindex
> count)
> # A new index definition package was deployed which reset the index count.
> Now reindex happened again and the CopyOnRead logic per current design reused
> the existing index directory. And it so happens that Lucene create file with
> same name and same size but different content. This trips the CopyOnRead
> defense of length based index corruption check and thus cause new lucene
> index to corrupt
> *Note that here corruption is transient i.e. persisted index is not
> corrupted*. Just that locally copied index gets corrupted. Cleaning up the
> index directory would fix the issue and that can be used as a workaround.
> *Fix*
> After discussing with [~tmueller] following approach can be used.
> Instead of relying on reindex count we can maintain a hidden randomly
> generated uuid and store it in the index config. This would be used to derive
> the name of directory on filesystem. If the index definition gets reset then
> the uuid can be regenerated.
> *Workaround*
> Clean the directory used by CopyOnRead which is /index before
> restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2744) Change default cache distribution ratio if persistent cache is enabled
[ https://issues.apache.org/jira/browse/OAK-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2744: --- Fix Version/s: (was: 1.4) 1.6 > Change default cache distribution ratio if persistent cache is enabled > -- > > Key: OAK-2744 > URL: https://issues.apache.org/jira/browse/OAK-2744 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk, rdbmk >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Labels: performance > Fix For: 1.6 > > > By default the cache memory in DocumentNodeStore is distributed in following > ratio > * nodeCache - 25% > * childrenCache - 10% > * docChildrenCache - 3% > * diffCache - 5% > * documentCache - Is given the rest i.e. 57% > However off late we have found that with persistent cache enabled we can > lower the cache allocated to Document cache. That would reduce the time spent > in invalidating cache entries in periodic reads. So far we are using > following ration in few setup and that is turning out well > * nodeCachePercentage=35 > * childrenCachePercentage=20 > * diffCachePercentage=30 > * docChildrenCachePercentage=10 > * documentCache - Is given the rest i.e. 5% > We should use the above distribution by default if the persistent cache is > found to be enabled -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2460) Resolve the base directory path of persistent cache against repository home
[
https://issues.apache.org/jira/browse/OAK-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-2460:
---
Fix Version/s: (was: 1.4)
1.6
> Resolve the base directory path of persistent cache against repository home
> ---
>
> Key: OAK-2460
> URL: https://issues.apache.org/jira/browse/OAK-2460
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: mongomk
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Labels: technical_debt
> Fix For: 1.6
>
>
> Currently PersistentCache uses the directory path directly. Various other
> parts in Oak which need access to the filesystem currently make use of
> {{repository.home}} framework property in OSGi env [1]
> Same should also be used in PersistentCache
> [1] http://jackrabbit.apache.org/oak/docs/osgi_config.html#SegmentNodeStore
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3642) Enable failing of commit upon Missing IndexEditor implementation by default
[ https://issues.apache.org/jira/browse/OAK-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3642: --- Fix Version/s: (was: 1.4) 1.6 > Enable failing of commit upon Missing IndexEditor implementation by default > --- > > Key: OAK-3642 > URL: https://issues.apache.org/jira/browse/OAK-3642 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.6 > > Attachments: OAK-3642.patch > > > With OAK-3505 we exposed config option to enable failing of commit if any > IndexEditor is missing. The default should be changed to always fail the > commit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3865) New strategy to optimize secondary reads
[ https://issues.apache.org/jira/browse/OAK-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3865: --- Fix Version/s: (was: 1.4) 1.6 > New strategy to optimize secondary reads > > > Key: OAK-3865 > URL: https://issues.apache.org/jira/browse/OAK-3865 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Tomek Rękawek > Labels: performance > Fix For: 1.6 > > Attachments: diagram.png > > > *Introduction* > In the current trunk we'll only read document _D_ from the secondary instance > if: > (1) we have the parent _P_ of document _D_ cached and > (2) the parent hasn't been modified in 6 hours. > The OAK-2106 tried to optimise (2) by estimating lag using MongoDB replica > stats. It was unreliable, so the second approach was to read the last > revisions directly from each Mongo instance. If the modification date of _P_ > is before last revisions on all secondary Mongos, then secondary can be used. > The main problem with this approach is that we still need to have the _P_ to > be in cache. I think we need another way to optimise the secondary reading, > as right now only about 3% of requests connects to the secondary, which is > bad especially for the global-clustering case (Mongo and Oak instances across > the globe). The optimisation provided in OAK-2106 doesn't make the things > much better and may introduce some consistency issues. > *Proposal* > I had following constraints in mind preparing this: > 1. Let's assume we have a sequence of commits with revisions _R1_, _R2_ and > _R3_ modifying nodes _N1_, _N2_ and _N3_. If we already read the _N1_ from > revision _R2_ then reading from a secondary shouldn't result in getting older > revision (eg. _R1_). > 2. If an Oak instance modifies a document, then reading from a secondary > shouldn't result in getting the old version (before modification). > So, let's have two maps: > * _M1_ the most recent document revision read from the Mongo for each cluster > id, > * _M2_ the oldest last rev value for root document for each cluster id read > from all the secondary instances. > Maintaining _M1_: > For every read from the Mongo we'll check if the lastRev for some cluster id > is newer than _M1_ entry. If so, we'll update _M1_. For all writes we'll add > the saved revision id with the current cluster id in _M1_. > Maintaining _M2_: > It should be periodically updated. Such mechanism is already prepared in the > OAK-2106 patch. > The method deciding whether we can read from the secondary instance should > compare two maps. If all entries in _M2_ are newer than _M1_ it means that > the secondary instances contains at least as new repository state as we > already accessed and therefore it's safe to read from secondary. > Regarding the documents modified by the local Oak instance, we should > remember all the locally-modified paths and their revisions and use primary > Mongo to access them as long as the changes are not replicated to all the > secondaries. When the secondaries are up to date with the modification, we > can remove it from the local-changes collections. > Attached image diagram.png presents the idea. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2066) DocumentStore API: batch create, but no batch update
[ https://issues.apache.org/jira/browse/OAK-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2066: --- Priority: Blocker (was: Major) > DocumentStore API: batch create, but no batch update > > > Key: OAK-2066 > URL: https://issues.apache.org/jira/browse/OAK-2066 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Julian Reschke >Priority: Blocker > Labels: performance > Fix For: 1.4, 1.3.15 > > > The DocumentStore API currently has a call for creating many nodes at once. > However, this will sometimes fail for large save operations in JCR, because > in the DS persistence, JCR-deleted nodes are still present (with a deleted > flag). This causes two subsequent sequences of > 1) create test container > 2) create many child nodes > 3) remove test container > to behave very differently, depending on whether the test container is > created for the first time or not. > (see CreateManyChildNodesTest) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2066) DocumentStore API: batch create, but no batch update
[ https://issues.apache.org/jira/browse/OAK-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2066: --- Fix Version/s: 1.3.15 > DocumentStore API: batch create, but no batch update > > > Key: OAK-2066 > URL: https://issues.apache.org/jira/browse/OAK-2066 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Julian Reschke > Labels: performance > Fix For: 1.4, 1.3.15 > > > The DocumentStore API currently has a call for creating many nodes at once. > However, this will sometimes fail for large save operations in JCR, because > in the DS persistence, JCR-deleted nodes are still present (with a deleted > flag). This causes two subsequent sequences of > 1) create test container > 2) create many child nodes > 3) remove test container > to behave very differently, depending on whether the test container is > created for the first time or not. > (see CreateManyChildNodesTest) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3724) Use the bulk createOrUpdate in Commit
[
https://issues.apache.org/jira/browse/OAK-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3724:
---
Priority: Blocker (was: Major)
> Use the bulk createOrUpdate in Commit
> -
>
> Key: OAK-3724
> URL: https://issues.apache.org/jira/browse/OAK-3724
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: documentmk
>Reporter: Tomek Rękawek
>Priority: Blocker
> Fix For: 1.4, 1.3.15
>
> Attachments: OAK-3724.patch
>
>
> The {{DocumentStore#createOrUpdate(Collection, UpdateOp)}} method is invoked
> in a loop in the {{Commit#applyToDocumentStore()}}, once for each changed
> node. Replace it with the bulk version created in OAK-3662.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3724) Use the bulk createOrUpdate in Commit
[
https://issues.apache.org/jira/browse/OAK-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3724:
---
Fix Version/s: 1.3.15
> Use the bulk createOrUpdate in Commit
> -
>
> Key: OAK-3724
> URL: https://issues.apache.org/jira/browse/OAK-3724
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: documentmk
>Reporter: Tomek Rękawek
> Fix For: 1.4, 1.3.15
>
> Attachments: OAK-3724.patch
>
>
> The {{DocumentStore#createOrUpdate(Collection, UpdateOp)}} method is invoked
> in a loop in the {{Commit#applyToDocumentStore()}}, once for each changed
> node. Replace it with the bulk version created in OAK-3662.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3866) Sorting on relative properties doesn't work in Solr
[
https://issues.apache.org/jira/browse/OAK-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3866:
---
Fix Version/s: (was: 1.4)
1.6
> Sorting on relative properties doesn't work in Solr
> ---
>
> Key: OAK-3866
> URL: https://issues.apache.org/jira/browse/OAK-3866
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: solr
>Affects Versions: 1.0.22, 1.3.13
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.6
>
>
> Executing a query like
> {noformat}
> /jcr:root/content/foo//*[(@sling:resourceType = 'x' or @sling:resourceType =
> 'y') and jcr:contains(., 'bar*~')] order by jcr:content/@jcr:primaryType
> descending
> {noformat}
> would assume sorting on the _jcr:primaryType_ property of resulting nodes'
> _jcr:content_ children.
> That is currently not supported in Solr, while it is in Lucene as the latter
> supports index time aggregation.
> We should inspect if it's possible to extend support for Solr too, most
> probably via index time aggregation.
> The query should not fail but at least log a warning about that limitation
> for the time being.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2727) NodeStateSolrServersObserver should be filtering path selectively
[ https://issues.apache.org/jira/browse/OAK-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2727: --- Fix Version/s: (was: 1.4) 1.6 > NodeStateSolrServersObserver should be filtering path selectively > - > > Key: OAK-2727 > URL: https://issues.apache.org/jira/browse/OAK-2727 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: solr >Affects Versions: 1.1.8 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili > Labels: performance > Fix For: 1.6 > > > As discussed in OAK-2718 it'd be good to be able to selectively find Solr > indexes by path, as done in Lucene index, see also OAK-2570. > This would avoid having to do full diffs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3253) Support caching in FileDataStoreService
[ https://issues.apache.org/jira/browse/OAK-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119324#comment-15119324 ] Michael Marth commented on OAK-3253: [~shgupta], do you think we can still get this in 1.4 (see release schedule on oak-dev)? > Support caching in FileDataStoreService > --- > > Key: OAK-3253 > URL: https://issues.apache.org/jira/browse/OAK-3253 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob >Affects Versions: 1.3.3 >Reporter: Shashank Gupta >Assignee: Shashank Gupta > Labels: candidate_oak_1_0, candidate_oak_1_2, docs-impacting, > features, performance > Fix For: 1.4 > > > FDS on SAN/NAS storage is not efficient as it involves network call. In OAK. > indexes are stored SAN/NAS and even idle system does lot of read system > generated data. > Enable caching in FDS so the reads are done locally and async upload to > SAN/NAS > See [previous > discussions|https://issues.apache.org/jira/browse/OAK-3005?focusedCommentId=14700801&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700801] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3559) Bulk document updates in MongoDocumentStore
[ https://issues.apache.org/jira/browse/OAK-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3559: --- Fix Version/s: 1.3.15 > Bulk document updates in MongoDocumentStore > --- > > Key: OAK-3559 > URL: https://issues.apache.org/jira/browse/OAK-3559 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: mongomk >Reporter: Tomek Rękawek >Assignee: Marcel Reutegger > Fix For: 1.4, 1.3.15 > > Attachments: OAK-3559.patch > > > Using the MongoDB [Bulk > API|https://docs.mongodb.org/manual/reference/method/Bulk/#Bulk] implement > the [batch version of createOrUpdate method|OAK-3662]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3559) Bulk document updates in MongoDocumentStore
[ https://issues.apache.org/jira/browse/OAK-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3559: --- Priority: Blocker (was: Major) > Bulk document updates in MongoDocumentStore > --- > > Key: OAK-3559 > URL: https://issues.apache.org/jira/browse/OAK-3559 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: mongomk >Reporter: Tomek Rękawek >Assignee: Marcel Reutegger >Priority: Blocker > Fix For: 1.4, 1.3.15 > > Attachments: OAK-3559.patch > > > Using the MongoDB [Bulk > API|https://docs.mongodb.org/manual/reference/method/Bulk/#Bulk] implement > the [batch version of createOrUpdate method|OAK-3662]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3761) Oak crypto API and implementation
[ https://issues.apache.org/jira/browse/OAK-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116989#comment-15116989 ] Michael Marth commented on OAK-3761: Right, but IIUC your point then that would affect where a crypto API is consumed. Agree with you that this needs to be on lowest level possible. My point, however, was where that API is developed. > Oak crypto API and implementation > - > > Key: OAK-3761 > URL: https://issues.apache.org/jira/browse/OAK-3761 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: security >Affects Versions: 1.3.12 >Reporter: Timothee Maret >Assignee: angela > Attachments: OAK-3761.patch, OAK-3761.patch > > > As discussed in [0], this issue tracks adding a simple API and implementation > for encryption/decryption in Oak. > [0] > http://oak.markmail.org/search/?q=crypto#query:crypto+page:1+mid:iwsfd66lku2dzs2n+state:results -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3761) Oak crypto API and implementation
[
https://issues.apache.org/jira/browse/OAK-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116911#comment-15116911
]
Michael Marth commented on OAK-3761:
{quote}
It can live independently and more suitable place for that would be Sling
{quote}
+1
IMHO a Crypto API is "too orthogonal" to the concerns of a content repository
and would better be kept elsewhere (even if we have use cases that need to
consume that API).
> Oak crypto API and implementation
> -
>
> Key: OAK-3761
> URL: https://issues.apache.org/jira/browse/OAK-3761
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: security
>Affects Versions: 1.3.12
>Reporter: Timothee Maret
>Assignee: angela
> Attachments: OAK-3761.patch, OAK-3761.patch
>
>
> As discussed in [0], this issue tracks adding a simple API and implementation
> for encryption/decryption in Oak.
> [0]
> http://oak.markmail.org/search/?q=crypto#query:crypto+page:1+mid:iwsfd66lku2dzs2n+state:results
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3914) Copy on read mbean is not properly documented anymore
[
https://issues.apache.org/jira/browse/OAK-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3914:
---
Component/s: doc
> Copy on read mbean is not properly documented anymore
> -
>
> Key: OAK-3914
> URL: https://issues.apache.org/jira/browse/OAK-3914
> Project: Jackrabbit Oak
> Issue Type: Documentation
> Components: doc
>Reporter: Stephan Becker
>Priority: Minor
>
> On http://jackrabbit.apache.org/oak/docs/query/lucene.html#copy-on-read the
> description for the mbean is false and the screenshot outdated it seems.
> It should contain a screenshot of org.apache.jackrabbit.oak: "IndexCopier
> support statistics" ("IndexCopierStats") mbean instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3871) ability to override ClusterNodeInfo#DEFAULT_LEASE_DURATION_MILLIS
[ https://issues.apache.org/jira/browse/OAK-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112097#comment-15112097 ] Michael Marth commented on OAK-3871: +1 > ability to override ClusterNodeInfo#DEFAULT_LEASE_DURATION_MILLIS > - > > Key: OAK-3871 > URL: https://issues.apache.org/jira/browse/OAK-3871 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Julian Reschke > Fix For: 1.4 > > > This constant currently is set to 120s. > In edge cases, it might be good to override the default (when essentially the > only other "fix" would be to turn off the lease check). > Proposal: introduce a system property allowing to tune the lease time (but > restrict allowable values to a "sane" range, such as up to 5min). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3033) Upgrade to Tika 1.7
[ https://issues.apache.org/jira/browse/OAK-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3033: --- Fix Version/s: (was: 1.4) 1.6 > Upgrade to Tika 1.7 > > > Key: OAK-3033 > URL: https://issues.apache.org/jira/browse/OAK-3033 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene, pojosr, solr >Affects Versions: 1.3.0 >Reporter: Ashok Kumar > Labels: technical_debt > Fix For: 1.6 > > > Upgrade Tika version in oak > Below modules need tika-version updated to 1.7 in concerned pom.xml > oak-solr-core > oak-pojosr > oak-lucene -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3536) Indexing with Lucene and copy-on-read generate too much garbage in the BlobStore
[ https://issues.apache.org/jira/browse/OAK-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3536: --- Issue Type: Improvement (was: Bug) > Indexing with Lucene and copy-on-read generate too much garbage in the > BlobStore > > > Key: OAK-3536 > URL: https://issues.apache.org/jira/browse/OAK-3536 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.3.9 >Reporter: Francesco Mari >Priority: Critical > Fix For: 1.4 > > > The copy-on-read strategy when using Lucene indexing performs too many copies > of the index files from the filesystem to the repository. Every copy discards > the previously stored binary, that sits there as garbage until the binary > garbage collection kicks in. When the load on the system is particularly > intense, this behaviour makes the repository grow at an unreasonable high > pace. > I spotted this on a system where some content is generated every day at a > specific time. The content generation process creates approx. 6 millions new > nodes, where each node contains 5 properties with small string, random > values. Nodes were saved in batches of 1000 nodes each. At the end of the > content generation process, the nodes are deleted to deliberately generate > garbage in the Segment Store. This is part of a testing effort to assess the > efficiency of the online compaction. > I was never able to complete the tests because the system run out of disk > space due to a lot of unused binary values. When debugging the system, on a > 400 GB (full) disk, the segments containing nodes and property values > occupied approx. 3 GB. The rest of the space was occupied by binary values in > form of bulk segments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2072) Lucene: inconsistent usage of the config option "persistence"
[ https://issues.apache.org/jira/browse/OAK-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2072: --- Priority: Minor (was: Major) > Lucene: inconsistent usage of the config option "persistence" > - > > Key: OAK-2072 > URL: https://issues.apache.org/jira/browse/OAK-2072 > Project: Jackrabbit Oak > Issue Type: Bug > Components: query >Reporter: Thomas Mueller >Assignee: Alex Parvulescu >Priority: Minor > Fix For: 1.4 > > > The Lucene index reader uses the configuration property "persistence", but > the editor (the component updating the index) does not. That leads to very > strange behavior if the property is missing, but the property "file" is set: > the reader would try to read from the file system, but those files are not > updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3629) Index corruption seen with CopyOnRead when index defnition is recreated
[
https://issues.apache.org/jira/browse/OAK-3629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3629:
---
Priority: Major (was: Minor)
> Index corruption seen with CopyOnRead when index defnition is recreated
> ---
>
> Key: OAK-3629
> URL: https://issues.apache.org/jira/browse/OAK-3629
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.4
>
>
> CopyOnRead logic relies on {{reindexCount}} to determine the name of
> directory in which index files would be copied. In normal flow if the index
> is reindexed then this count would get increased and newer index files would
> get copied to a new directory.
> However if the index definition node gets recreated due to some deployment
> process then this count gets reset to 0. Due to which newly created index
> files from reindexing would start getting copied to already existing
> directory and that can lead to corruption.
> So what happened here was
> # System started with index definition I1 and indexing got complete with
> index files saved under index/hash(indexpath)/1 (where 1 is current reindex
> count)
> # A new index definition package was deployed which reset the index count.
> Now reindex happened again and the CopyOnRead logic per current design reused
> the existing index directory. And it so happens that Lucene create file with
> same name and same size but different content. This trips the CopyOnRead
> defense of length based index corruption check and thus cause new lucene
> index to corrupt
> *Note that here corruption is transient i.e. persisted index is not
> corrupted*. Just that locally copied index gets corrupted. Cleaning up the
> index directory would fix the issue and that can be used as a workaround.
> *Fix*
> After discussing with [~tmueller] following approach can be used.
> Instead of relying on reindex count we can maintain a hidden randomly
> generated uuid and store it in the index config. This would be used to derive
> the name of directory on filesystem. If the index definition gets reset then
> the uuid can be regenerated.
> *Workaround*
> Clean the directory used by CopyOnRead which is /index before
> restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2477) Move suggester specific config to own configuration node
[ https://issues.apache.org/jira/browse/OAK-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2477: --- Labels: docs-impacting technical_debt (was: technical_debt) > Move suggester specific config to own configuration node > > > Key: OAK-2477 > URL: https://issues.apache.org/jira/browse/OAK-2477 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili > Labels: docs-impacting, technical_debt > Fix For: 1.4 > > > Currently suggester configuration is controlled via properties defined on > main config / props node but it'd be good if we would have its own place to > configure the whole suggest feature to not mix up configuration of other > features / parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2478) Move spellcheck config to own configuration node
[ https://issues.apache.org/jira/browse/OAK-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2478: --- Labels: docs-impacting technical_debt (was: technical_debt) > Move spellcheck config to own configuration node > > > Key: OAK-2478 > URL: https://issues.apache.org/jira/browse/OAK-2478 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili > Labels: docs-impacting, technical_debt > Fix For: 1.4 > > > Currently spellcheck configuration is controlled via properties defined on > main config / props node but it'd be good if we would have its own place to > configure the whole spellcheck feature to not mix up configuration of other > features / parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-1819) oak-solr-core test failures on Java 8
[
https://issues.apache.org/jira/browse/OAK-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-1819:
---
Labels: java8 jenkins test (was: java8 test)
> oak-solr-core test failures on Java 8
> -
>
> Key: OAK-1819
> URL: https://issues.apache.org/jira/browse/OAK-1819
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: solr
>Affects Versions: 1.0
> Environment: {noformat}
> Apache Maven 3.1.0 (893ca28a1da9d5f51ac03827af98bb730128f9f2; 2013-06-27
> 22:15:32-0400)
> Maven home: c:\Program Files\apache-maven-3.1.0
> Java version: 1.8.0, vendor: Oracle Corporation
> Java home: c:\Program Files\Java\jdk1.8.0\jre
> Default locale: en_US, platform encoding: Cp1252
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"
> {noformat}
>Reporter: Jukka Zitting
>Assignee: Tommaso Teofili
>Priority: Minor
> Labels: java8, jenkins, test
> Fix For: 1.4
>
>
> The following {{oak-solr-core}} test failures occur when building Oak with
> Java 8:
> {noformat}
> Failed tests:
>
> testNativeMLTQuery(org.apache.jackrabbit.oak.plugins.index.solr.query.SolrIndexQueryTest):
> expected: but was:
>
> testNativeMLTQueryWithStream(org.apache.jackrabbit.oak.plugins.index.solr.query.SolrIndexQueryTest):
> expected: but was:
> {noformat}
> The cause of this might well be something as simple as the test case
> incorrectly expecting a specific ordering of search results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3033) Upgrade to Tika 1.7
[ https://issues.apache.org/jira/browse/OAK-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15108273#comment-15108273 ] Michael Marth commented on OAK-3033: [~teofili], [~chetanm], [~baedke], thoughts on still doing this for 1.4 ? > Upgrade to Tika 1.7 > > > Key: OAK-3033 > URL: https://issues.apache.org/jira/browse/OAK-3033 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene, pojosr, solr >Affects Versions: 1.3.0 >Reporter: Ashok Kumar > Labels: technical_debt > Fix For: 1.4 > > > Upgrade Tika version in oak > Below modules need tika-version updated to 1.7 in concerned pom.xml > oak-solr-core > oak-pojosr > oak-lucene -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-1453) MongoMK failover support for replica sets (esp. shards)
[ https://issues.apache.org/jira/browse/OAK-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-1453: --- Fix Version/s: (was: 1.4) 1.6 > MongoMK failover support for replica sets (esp. shards) > --- > > Key: OAK-1453 > URL: https://issues.apache.org/jira/browse/OAK-1453 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Michael Marth > Labels: production, resilience > Fix For: 1.6 > > > With OAK-759 we have introduced replica support in MongoMK. I think we still > need to address the resilience for failover from primary to secoandary: > Consider a case where Oak writes to the primary. Replication to secondary is > ongoing. During that period the primary goes down and the secondary becomes > primary. There could be some "half-replicated" MVCC revisions, which need to > be either discarded or be ignored after the failover. > This might not be an issue if there is only one shard, as the commit root is > written last (and replicated last) > But with 2 shards the the replication state of these 2 shards could be > inconsistent. Oak needs to handle such a situation without falling over. > If we can detect a Mongo failover we could query Mongo which revisions are > fully replicated to the new primary and discard the potentially > half-replicated revisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3286) Persistent Cache improvements
[ https://issues.apache.org/jira/browse/OAK-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3286: --- Fix Version/s: (was: 1.4) 1.6 > Persistent Cache improvements > - > > Key: OAK-3286 > URL: https://issues.apache.org/jira/browse/OAK-3286 > Project: Jackrabbit Oak > Issue Type: Epic > Components: cache >Reporter: Michael Marth >Priority: Minor > Fix For: 1.6 > > > Issue for collecting various improvements to the persistent cache -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2761) Persistent cache: add data in a different thread
[
https://issues.apache.org/jira/browse/OAK-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-2761:
---
Fix Version/s: 1.4
> Persistent cache: add data in a different thread
>
>
> Key: OAK-2761
> URL: https://issues.apache.org/jira/browse/OAK-2761
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: cache, core, mongomk
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Labels: resilience
> Fix For: 1.4
>
>
> The persistent cache usually stores data in a background thread, but
> sometimes (if a lot of data is added quickly) the foreground thread is
> blocked.
> Even worse, switching the cache file can happen in a foreground thread, with
> the following stack trace.
> {noformat}
> "127.0.0.1 [1428931262206] POST /bin/replicate.json HTTP/1.1" prio=5
> tid=0x7fe5df819800 nid=0x9907 runnable [0x000113fc4000]
>java.lang.Thread.State: RUNNABLE
> ...
> at org.h2.mvstore.MVStoreTool.compact(MVStoreTool.java:404)
> at
> org.apache.jackrabbit.oak.plugins.document.persistentCache.PersistentCache$1.closeStore(PersistentCache.java:213)
> - locked <0x000782483050> (a
> org.apache.jackrabbit.oak.plugins.document.persistentCache.PersistentCache$1)
> at
> org.apache.jackrabbit.oak.plugins.document.persistentCache.PersistentCache.switchGenerationIfNeeded(PersistentCache.java:350)
> - locked <0x000782455710> (a
> org.apache.jackrabbit.oak.plugins.document.persistentCache.PersistentCache)
> at
> org.apache.jackrabbit.oak.plugins.document.persistentCache.NodeCache.write(NodeCache.java:85)
> at
> org.apache.jackrabbit.oak.plugins.document.persistentCache.NodeCache.put(NodeCache.java:130)
> at
> org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.applyChanges(DocumentNodeStore.java:1060)
> at
> org.apache.jackrabbit.oak.plugins.document.Commit.applyToCache(Commit.java:599)
> at
> org.apache.jackrabbit.oak.plugins.document.CommitQueue.afterTrunkCommit(CommitQueue.java:127)
> - locked <0x000781890788> (a
> org.apache.jackrabbit.oak.plugins.document.CommitQueue)
> at
> org.apache.jackrabbit.oak.plugins.document.CommitQueue.done(CommitQueue.java:83)
> at
> org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.done(DocumentNodeStore.java:637)
> {noformat}
> To avoid blocking the foreground thread, one solution is to store all data in
> a separate thread. If there is too much data added, then some of the data is
> not stored. If possible, the data that was not referenced a lot, and / or old
> revisions of documents (if new revisions are available).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-3839) Suggest missing indexes
Michael Marth created OAK-3839: -- Summary: Suggest missing indexes Key: OAK-3839 URL: https://issues.apache.org/jira/browse/OAK-3839 Project: Jackrabbit Oak Issue Type: Improvement Components: query Affects Versions: 1.3.13, 1.0.25, 1.2.9 Reporter: Michael Marth For queries that lack the appropriate indexes (i.e. falling back on traversal) it would be helpful if the query engine could suggest which index to create. The missing index should not be created automatically and the suggestion should only be displayed in DEBUG mode. To begin with a message in the log would be great (we could extend with an MBean later). Also, it would be helpful to cover the simple cases first (I imagine the implementation of a complete suggestion mechanism could be quite complex). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3827) TarMK JCR data -> Solr == Exception
[ https://issues.apache.org/jira/browse/OAK-3827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3827: --- Labels: resilience (was: ) > TarMK JCR data -> Solr == Exception > --- > > Key: OAK-3827 > URL: https://issues.apache.org/jira/browse/OAK-3827 > Project: Jackrabbit Oak > Issue Type: Bug > Components: solr >Reporter: PuzanovsP > Labels: resilience > > One node was not in Solr after searching for it... > After looking into logs have found following: > ERROR - 2015-12-21 17:05:29.598; org.apache.solr.common.SolrException; > null:org.apache.solr.common.SolrException: Exception writing document id > /content/dam/my/example.pdf/jcr:content/metadata/wn_previews:previews/wn_previews:spreads/1 > to the index; possible analysis error. > at > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:168) > at > org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69) > at > org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:870) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1024) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:693) > at > org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100) > at > org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:96) > at > org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166) > at > org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136) > at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225) > at > org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121) > at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190) > at > org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116) > at > org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173) > at > org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106) > at > org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58) > at > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) > at > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1962) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:368) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) > at > org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) > at > org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpC
[jira] [Updated] (OAK-3717) Make it possible to declare SynonymFilter within Analyzer with WN dictionary
[
https://issues.apache.org/jira/browse/OAK-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3717:
---
Summary: Make it possible to declare SynonymFilter within Analyzer with WN
dictionary (was: Make it possible to declare SynonyFilter within Analyzer with
WN dictionary)
> Make it possible to declare SynonymFilter within Analyzer with WN dictionary
>
>
> Key: OAK-3717
> URL: https://issues.apache.org/jira/browse/OAK-3717
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.3.13
>
>
> Currently one can compose Lucene Analyzers via
> [composition|http://jackrabbit.apache.org/oak/docs/query/lucene.html#Create_analyzer_via_composition]
> within an index definition. It'd be good to be able to also use
> {{SynonymFIlter}} in there, eventually decorated with
> {{WordNetSynonymParser}} to leverage WordNet synonym files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-1695) Document Solr index
[ https://issues.apache.org/jira/browse/OAK-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15073721#comment-15073721 ] Michael Marth commented on OAK-1695: [~teofili], anything left to do here? If yes, we might want to track this in dedicated issues. WDYT? > Document Solr index > --- > > Key: OAK-1695 > URL: https://issues.apache.org/jira/browse/OAK-1695 > Project: Jackrabbit Oak > Issue Type: Task > Components: doc, solr >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili > Labels: documentation > Fix For: 1.3.13 > > > Provide documentation about the Oak Solr index. That should contain > information about the design and how to configure it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-1819) oak-solr-core test failures on Java 8
[
https://issues.apache.org/jira/browse/OAK-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-1819:
---
Fix Version/s: (was: 1.3.13)
1.4
> oak-solr-core test failures on Java 8
> -
>
> Key: OAK-1819
> URL: https://issues.apache.org/jira/browse/OAK-1819
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: solr
>Affects Versions: 1.0
> Environment: {noformat}
> Apache Maven 3.1.0 (893ca28a1da9d5f51ac03827af98bb730128f9f2; 2013-06-27
> 22:15:32-0400)
> Maven home: c:\Program Files\apache-maven-3.1.0
> Java version: 1.8.0, vendor: Oracle Corporation
> Java home: c:\Program Files\Java\jdk1.8.0\jre
> Default locale: en_US, platform encoding: Cp1252
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"
> {noformat}
>Reporter: Jukka Zitting
>Assignee: Tommaso Teofili
>Priority: Minor
> Labels: java8, test
> Fix For: 1.4
>
>
> The following {{oak-solr-core}} test failures occur when building Oak with
> Java 8:
> {noformat}
> Failed tests:
>
> testNativeMLTQuery(org.apache.jackrabbit.oak.plugins.index.solr.query.SolrIndexQueryTest):
> expected: but was:
>
> testNativeMLTQueryWithStream(org.apache.jackrabbit.oak.plugins.index.solr.query.SolrIndexQueryTest):
> expected: but was:
> {noformat}
> The cause of this might well be something as simple as the test case
> incorrectly expecting a specific ordering of search results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3286) Persistent Cache improvements
[ https://issues.apache.org/jira/browse/OAK-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3286: --- Fix Version/s: (was: 1.3.13) 1.4 > Persistent Cache improvements > - > > Key: OAK-3286 > URL: https://issues.apache.org/jira/browse/OAK-3286 > Project: Jackrabbit Oak > Issue Type: Epic > Components: cache >Reporter: Michael Marth >Priority: Minor > Fix For: 1.4 > > > Issue for collecting various improvements to the persistent cache -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3825) Including Resource name to suggestions
[ https://issues.apache.org/jira/browse/OAK-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15070856#comment-15070856 ] Michael Marth commented on OAK-3825: [~anagarwa], why is this needed (or even desirable)? I think many or most node names are of a technical nature (and at least can contain underscores, hyphen, etc, but might also be names like "subpage1", "XYZ234567.jpg" etc). Do you want to expose these as suggestions to end users? > Including Resource name to suggestions > -- > > Key: OAK-3825 > URL: https://issues.apache.org/jira/browse/OAK-3825 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Reporter: Ankit Agarwal >Assignee: Vikas Saurabh > Fix For: 1.4 > > > Currently it is possible to include properties of a resource into > suggestions. > There should be a way so that its possible to include resource name itself > into suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3814) Collect and expose important runtime statistics across various part of Oak
[ https://issues.apache.org/jira/browse/OAK-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066588#comment-15066588 ] Michael Marth commented on OAK-3814: related to OAK-3791: I think what would be very useful is to measure the read request flow/cache hits across the whole Oak stack, i.e. 1. cache hit/miss in persistent cache 2. cache hit/miss in DocStore cache (would be covered by OAK-3791) 3. at this point we would hit persistence (Mongo, RDB): timing on this would be useful Other than that: do we have stats on query execution time indexing? > Collect and expose important runtime statistics across various part of Oak > -- > > Key: OAK-3814 > URL: https://issues.apache.org/jira/browse/OAK-3814 > Project: Jackrabbit Oak > Issue Type: Epic >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.4 > > > Aim of this epic is to improve the collection of runtime statistics across > various sub systems in Oak. This would enable following benefits > # Better insight into operational aspects of Oak > # Better integration with external monitoring system based on JMX > # Based on stats we can decide what needs to be done to make Oak perform > better i.e. based on actual usage data -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-144) Implement observation
[ https://issues.apache.org/jira/browse/OAK-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048727#comment-15048727 ] Michael Marth commented on OAK-144: --- Last update on OAK-1145 was 2 years ago - IMO close LATER seems reasonable > Implement observation > - > > Key: OAK-144 > URL: https://issues.apache.org/jira/browse/OAK-144 > Project: Jackrabbit Oak > Issue Type: Epic > Components: core, jcr >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: observation > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3134) Identify functionality offered by oak-run
[ https://issues.apache.org/jira/browse/OAK-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048725#comment-15048725 ] Michael Marth commented on OAK-3134: [~edivad], do you think we have collected enough info to propose modules? > Identify functionality offered by oak-run > - > > Key: OAK-3134 > URL: https://issues.apache.org/jira/browse/OAK-3134 > Project: Jackrabbit Oak > Issue Type: Epic > Components: run >Reporter: Davide Giannella >Assignee: Davide Giannella > > oak-run reached the size of 50MB+ and offers indeed various functionalities > that could be moved to their own module. > This ticket is about to identify what oak-run currently offers and what/if > could be split. > ML thread: http://markmail.org/thread/w34bphrk57l7pkaz > || Functionality || Package || Module || > | micro-benchmark | org.apache.jackrabbit.oak.benchmark |oak-development | > | scalability benchmark | org.apache.jackrabbit.oak.scalability | > oak-development| > | Groovy console | org.apache.jackrabbit.oak.console, > /oak-run/src/main/groovy/org/apache/jackrabbit/oak/console | oak-operations| > | Backup | | oak-operations| > | Restore | | oak-operations| > | Debug | | oak-operations| > | Check | | oak-operations| > | Compact | | oak-operations| > | Server | | oak-development| > | Upgrade | | oak-upgrade| > | Explore | |oak-development | > | Primary | | oak-development| > | Standby | | oak-development| > | Help | | | > | Checkpoints | | oak-operations| > | Recovery | | oak-operations| > | Repair | | oak-operations| > | Tika | | | > Modules left after the actions: > **oak-development** > Used to group and execute all the tools used during development. > _deployed to maven:_ No. > **oak-operations** > Used to group and execute all the tools used normally in production > environment for tasks like maintenance & C. > _deployed to maven:_ Yes. > **oak-run** > Will group what's left after the split. > _deployed to maven:_ No. > **oak-upgrade** > Will group tools for upgrading/migrating from one repo/version to another > _deployed to maven:_ yes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-144) Implement observation
[ https://issues.apache.org/jira/browse/OAK-144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048721#comment-15048721 ] Michael Marth commented on OAK-144: --- [~mduerig], should we close this epic done? > Implement observation > - > > Key: OAK-144 > URL: https://issues.apache.org/jira/browse/OAK-144 > Project: Jackrabbit Oak > Issue Type: Epic > Components: core, jcr >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: observation > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3272) DocumentMK scalability improvements
[ https://issues.apache.org/jira/browse/OAK-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3272: --- Component/s: documentmk > DocumentMK scalability improvements > --- > > Key: OAK-3272 > URL: https://issues.apache.org/jira/browse/OAK-3272 > Project: Jackrabbit Oak > Issue Type: Epic > Components: documentmk, mongomk, rdbmk >Reporter: Michael Marth > Labels: scalability > > Collector issue for tracking DocMK issues concerning scalability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3698) Make it easier to monitor state of observation queue
Michael Marth created OAK-3698: -- Summary: Make it easier to monitor state of observation queue Key: OAK-3698 URL: https://issues.apache.org/jira/browse/OAK-3698 Project: Jackrabbit Oak Issue Type: Improvement Reporter: Michael Marth Priority: Minor As suggest offline by [~ianeboston]: expose some stats (not time series!) to make it easier to monitor the state of the observation queue, in particular: * number of items in queue * number of items entering queue * number of items leaving queue * percentage of queue size compared to the upper limit by when events get collapsed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3693) Expose the internal state of the repository through indicators and checks
[ https://issues.apache.org/jira/browse/OAK-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3693: --- Fix Version/s: 1.4 > Expose the internal state of the repository through indicators and checks > - > > Key: OAK-3693 > URL: https://issues.apache.org/jira/browse/OAK-3693 > Project: Jackrabbit Oak > Issue Type: Epic >Reporter: Valentin Olteanu > Labels: production > Fix For: 1.4 > > > This container groups all the issues related to areas where we could improve > the monitoring of an OAK repository. This can be achieved by exposing > different indicators of the internal state and adding checks for certain > properties. > Areas to improve: > * Async Indexing > * Revision GC > * Topology changes > * Workflows > * Replication -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3693) Expose the internal state of the repository through indicators and checks
[ https://issues.apache.org/jira/browse/OAK-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3693: --- Labels: production (was: ) > Expose the internal state of the repository through indicators and checks > - > > Key: OAK-3693 > URL: https://issues.apache.org/jira/browse/OAK-3693 > Project: Jackrabbit Oak > Issue Type: Epic >Reporter: Valentin Olteanu > Labels: production > Fix For: 1.4 > > > This container groups all the issues related to areas where we could improve > the monitoring of an OAK repository. This can be achieved by exposing > different indicators of the internal state and adding checks for certain > properties. > Areas to improve: > * Async Indexing > * Revision GC > * Topology changes > * Workflows > * Replication -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-1553) More sophisticated conflict resolution when concurrently adding nodes
[
https://issues.apache.org/jira/browse/OAK-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-1553:
---
added this to DocMK resilience epic (because the corresponding sub-tasks cannot
be linked to epics and the DocMK subtask seemed more pressing)
> More sophisticated conflict resolution when concurrently adding nodes
> -
>
> Key: OAK-1553
> URL: https://issues.apache.org/jira/browse/OAK-1553
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk, segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: concurrency, resilience, scalability
> Attachments: OAK-1553.patch
>
>
> {{MicroKernel.rebase}} currently specifies: "addExistingNode: A node has been
> added that is different from a node of them same name that has been added to
> the trunk."
> This is somewhat troublesome in the case where the same node with different
> but non conflicting child items is added concurrently:
> {code}
> f.add("fo").add("u1"); commit();
> f.add("fo").add("u2"); commit();
> {code}
> currently fails with a conflict because {{fo}} is not the same node for the
> both cases. See discussion http://markmail.org/message/flst4eiqvbp4gi3z
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3696) Improve SegmentMK resilience
[ https://issues.apache.org/jira/browse/OAK-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3696: --- Labels: resilience (was: ) > Improve SegmentMK resilience > > > Key: OAK-3696 > URL: https://issues.apache.org/jira/browse/OAK-3696 > Project: Jackrabbit Oak > Issue Type: Epic > Components: segmentmk >Reporter: Michael Marth > Labels: resilience > > Epic for collection SegmentMK resilience improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3696) Improve SegmentMK resilience
[ https://issues.apache.org/jira/browse/OAK-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3696: --- Component/s: segmentmk > Improve SegmentMK resilience > > > Key: OAK-3696 > URL: https://issues.apache.org/jira/browse/OAK-3696 > Project: Jackrabbit Oak > Issue Type: Epic > Components: segmentmk >Reporter: Michael Marth > Labels: resilience > > Epic for collection SegmentMK resilience improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3696) Improve SegmentMK resilience
Michael Marth created OAK-3696: -- Summary: Improve SegmentMK resilience Key: OAK-3696 URL: https://issues.apache.org/jira/browse/OAK-3696 Project: Jackrabbit Oak Issue Type: Epic Reporter: Michael Marth Epic for collection SegmentMK resilience improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3269) Improve indexing resilience
[ https://issues.apache.org/jira/browse/OAK-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3269: --- Epic Name: indexer resilience (was: Lucene indexer resilience) > Improve indexing resilience > --- > > Key: OAK-3269 > URL: https://issues.apache.org/jira/browse/OAK-3269 > Project: Jackrabbit Oak > Issue Type: Epic > Components: lucene >Reporter: Michael Marth >Priority: Critical > Labels: resilience > > As discussed bilaterally grouping the improvements for Lucene indexer > resilience in this issue for easier tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3269) Improve indexing resilience
[ https://issues.apache.org/jira/browse/OAK-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3269: --- Description: As discussed bilaterally grouping the improvements for indexer resilience in this issue for easier tracking (was: As discussed bilaterally grouping the improvements for Lucene indexer resilience in this issue for easier tracking) > Improve indexing resilience > --- > > Key: OAK-3269 > URL: https://issues.apache.org/jira/browse/OAK-3269 > Project: Jackrabbit Oak > Issue Type: Epic > Components: lucene >Reporter: Michael Marth >Priority: Critical > Labels: resilience > > As discussed bilaterally grouping the improvements for indexer resilience in > this issue for easier tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3269) Improve indexing resilience
[ https://issues.apache.org/jira/browse/OAK-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3269: --- Summary: Improve indexing resilience (was: Improve Lucene indexer resilience) > Improve indexing resilience > --- > > Key: OAK-3269 > URL: https://issues.apache.org/jira/browse/OAK-3269 > Project: Jackrabbit Oak > Issue Type: Epic > Components: lucene >Reporter: Michael Marth >Priority: Critical > Labels: resilience > > As discussed bilaterally grouping the improvements for Lucene indexer > resilience in this issue for easier tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3436) Prevent missing checkpoint due to unstable topology from causing complete reindexing
[
https://issues.apache.org/jira/browse/OAK-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3436:
---
Labels: resilience (was: )
> Prevent missing checkpoint due to unstable topology from causing complete
> reindexing
>
>
> Key: OAK-3436
> URL: https://issues.apache.org/jira/browse/OAK-3436
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: query
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Labels: resilience
> Fix For: 1.2.9, 1.0.25, 1.3.12
>
> Attachments: AsyncIndexUpdateClusterTest.java, OAK-3436-0.patch
>
>
> Async indexing logic relies on embedding application to ensure that async
> indexing job is run as a singleton in a cluster. For Sling based apps it
> depends on Sling Discovery support. At times it is being seen that if
> topology is not stable then different cluster nodes can consider them as
> leader and execute the async indexing job concurrently.
> This can cause problem as both cluster node might not see same repository
> state (due to write skew and eventual consistency) and might remove the
> checkpoint which other cluster node is still relying upon. For e.g. consider
> a 2 node cluster N1 and N2 where both are performing async indexing.
> # Base state - CP1 is the checkpoint for "async" job
> # N2 starts indexing and removes changes CP1 to CP2. For Mongo the
> checkpoints are saved in {{settings}} collection
> # N1 also decides to execute indexing but has yet not seen the latest
> repository state so still thinks that CP1 is the base checkpoint and tries to
> read it. However CP1 is already removed from {{settings}} and this makes N1
> think that checkpoint is missing and it decides to reindex everything!
> To avoid this topology must be stable but at Oak level we should still handle
> such a case and avoid doing a full reindexing. So we would need to have a
> {{MissingCheckpointStrategy}} similar to {{MissingIndexEditorStrategy}} as
> done in OAK-2203
> Possible approaches
> # A1 - Fail the indexing run if checkpoint is missing - Checkpoint being
> missing can have valid reason and invalid reason. Need to see what are valid
> scenarios where a checkpoint can go missing
> # A2 - When a checkpoint is created also store the creation time. When a
> checkpoint is found to be missing and its a *recent* checkpoint then fail the
> run. For e.g. we would fail the run till checkpoint found to be missing is
> less than an hour old (for just started take startup time into account)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-3680) Partial re-index from last known good state
Michael Marth created OAK-3680: -- Summary: Partial re-index from last known good state Key: OAK-3680 URL: https://issues.apache.org/jira/browse/OAK-3680 Project: Jackrabbit Oak Issue Type: New Feature Components: lucene, query Affects Versions: 1.2, 1.0 Reporter: Michael Marth ATM indexes break (by whatever circumstances) users need to perform a full re-index. Depending on the size off the repository this can take a long time. If the user knows that the indexes were in a good state at a certain revision in the past then it would be very useful, if the user could trigger a "partial" re-index where only the content added after a certain revision was updated in the index. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3529) NodeStore API should expose an Instance ID
[ https://issues.apache.org/jira/browse/OAK-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993278#comment-14993278 ] Michael Marth commented on OAK-3529: I think we should be very clear about the expected characteristics of this ID, e.g. if it is supposed to be stable on restarts or even re-installs - or if the opposite is true (i.e. a new ID is expected in these cases). > NodeStore API should expose an Instance ID > -- > > Key: OAK-3529 > URL: https://issues.apache.org/jira/browse/OAK-3529 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Reporter: Davide Giannella >Assignee: Davide Giannella > Fix For: 1.4 > > Attachments: OAK-3529-1.patch, OAK-3529-2.patch > > > For better leveraging cluster oriented algorithm: discovery, atomic > operations; it would be very helpful if the NodeStore could expose a > unique instance id. > This can be the same as a cluster ID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3341) lucene technical debt
[ https://issues.apache.org/jira/browse/OAK-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3341: --- Issue Type: Epic (was: Improvement) > lucene technical debt > - > > Key: OAK-3341 > URL: https://issues.apache.org/jira/browse/OAK-3341 > Project: Jackrabbit Oak > Issue Type: Epic > Components: lucene >Reporter: Daniel Hasler > > As discussed bilaterally grouping the technical debt for Lucene in this issue > for easier tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
