[jira] [Commented] (OAK-3606) Improvements for IndexStatsMBean usage
[ https://issues.apache.org/jira/browse/OAK-3606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671377#comment-15671377 ] Manfred Baedke commented on OAK-3606: - Thanks [~chetanm], that actually seems to work! > Improvements for IndexStatsMBean usage > -- > > Key: OAK-3606 > URL: https://issues.apache.org/jira/browse/OAK-3606 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, lucene >Affects Versions: 1.3.9 >Reporter: Thierry Ygé >Assignee: Manfred Baedke > Fix For: 1.6, 1.5.14 > > Attachments: adding_new_MBean.patch, > new_mbean_interface_and_implementation.patch > > > When running integration tests, it is common to have the need to wait for the > async indexes to have been executed. So that the test can successfully > validate operations that depend on the search result. > With the current IndexStatsMBean implementation it cannot return the start > time of the last successful indexing. It provide a "LastIndexedTime" which is > not sufficient to know if changes made recently are now indexed. > The idea is to set the start time as value of a new attribute (i.e > "StartLastSuccessIndexedTime") to the IndexStatsMBean. > Then create a new Mbean that calculate from all existing IndexStatsMBean (as > multiple are possible now) the oldest "StartLastSuccessIndexedTime". > That will allow integration tests to be able to wait until that oldest > "StartLastSuccessIndexedTime" is greater than the time it started to wait. > Attached is a sample patch containing the necessary changes (for a Oak core > 1.4.0-SNAPSHOT). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4943) Keep Lucene Commits so that if the Segments file gets corrupted recovery can be attempted.
[ https://issues.apache.org/jira/browse/OAK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671017#comment-15671017 ] Chetan Mehrotra commented on OAK-4943: -- [~ianeboston] Are there any more changes in your github branch? I see only changes around enabling Deletion Policy but no change around how attempt opening the directory to previous commits. > Keep Lucene Commits so that if the Segments file gets corrupted recovery can > be attempted. > -- > > Key: OAK-4943 > URL: https://issues.apache.org/jira/browse/OAK-4943 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene >Affects Versions: 1.6 >Reporter: Ian Boston >Assignee: Chetan Mehrotra > Fix For: 1.6 > > > Currently, Lucene deletes all previous generations of the segments files as > it has it uses the default IndexDeletionPolicy. Changing this to an > IndexDeletionPolicy that keeps a number of previous segments files will allow > an operator to find the most recent valid commit when the current segments > file reports corruption. The patch found at > https://github.com/apache/jackrabbit-oak/compare/trunk...ieb:KeepLuceneCommits > keeps 10 previous commits. > A more sophisticated policy could be used to save commits non-linearly over > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4898) Allow for external changes to have a CommitInfo attached
[ https://issues.apache.org/jira/browse/OAK-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670983#comment-15670983 ] Stefan Egli commented on OAK-4898: -- +1, perfect, looks good to me. > Allow for external changes to have a CommitInfo attached > > > Key: OAK-4898 > URL: https://issues.apache.org/jira/browse/OAK-4898 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.6 > > Attachments: OAK-4898-v1.patch, OAK-4898-v2.patch, OAK-4898-v3.patch, > OAK-4898-v4.patch > > > Currently the observation logic relies on fact that CommitInfo being null > means that changes are from other cluster node i.e. external changes. > We should change this semantic and provide a different way to indicate that > changes are external. This would allow a NodeStore implementation to still > pass in a CommitInfo which captures useful information about commit like > brief summary on what got changed which can be used for pre filtering > (OAK-4796) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4898) Allow for external changes to have a CommitInfo attached
[
https://issues.apache.org/jira/browse/OAK-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-4898:
-
Attachment: OAK-4898-v4.patch
[take 4|^OAK-4898-v4.patch] for including above change
bq. AsyncIndexUpdate introduces a typo in the change
That was erroneously included
bq. ChangeProcessor.contentChanged (still) has a check for info != null, but
that should probably now be 'CommitContext' != null
Made some change. Check now
bq. NodeEventHandler (still) has a check for commitInfo == null, which it
translates into EMPTY. In theory that check is no longer necessary now? Or
should it be changed to EMPTY_EXTERNAL..
Removed that as CommitInfo is now NonNull
bq. And a question: I'm wondering if we should make the info map of EMPTY and
EMPTY_EXTERNAL immutable to avoid erroneously updating that (and then updating
the singleton). As by having non-empty CommitInfo all the time now we might
make use of this and add a CommitContext even to EMPTY and EMPTY_EXTERNAL
CommitInfos at some point - but to do that we first have to clone it to avoid
this mistake. Adding a safety check might help in any case. Wdyt?
This is already the case as the constants are backed by
{{Collections#emptyMap}} which are immutable
> Allow for external changes to have a CommitInfo attached
>
>
> Key: OAK-4898
> URL: https://issues.apache.org/jira/browse/OAK-4898
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6
>
> Attachments: OAK-4898-v1.patch, OAK-4898-v2.patch, OAK-4898-v3.patch,
> OAK-4898-v4.patch
>
>
> Currently the observation logic relies on fact that CommitInfo being null
> means that changes are from other cluster node i.e. external changes.
> We should change this semantic and provide a different way to indicate that
> changes are external. This would allow a NodeStore implementation to still
> pass in a CommitInfo which captures useful information about commit like
> brief summary on what got changed which can be used for pre filtering
> (OAK-4796)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4898) Allow for external changes to have a CommitInfo attached
[
https://issues.apache.org/jira/browse/OAK-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670884#comment-15670884
]
Stefan Egli commented on OAK-4898:
--
[~chetanm], patch looks good. Some comments:
* AsyncIndexUpdate introduces a typo in the change
* ChangeProcessor.contentChanged (still) has a check for info != null, but that
should probably now be 'CommitContext' != null
* NodeEventHandler (still) has a check for commitInfo == null, which it
translates into EMPTY. In theory that check is no longer necessary now? Or
should it be changed to EMPTY_EXTERNAL..
* And a question: I'm wondering if we should make the {{info}} map of EMPTY and
EMPTY_EXTERNAL immutable to avoid erroneously updating that (and then updating
the singleton). As by having non-empty CommitInfo all the time now we might
make use of this and add a CommitContext even to EMPTY and EMPTY_EXTERNAL
CommitInfos at some point - but to do that we first have to clone it to avoid
this mistake. Adding a safety check might help in any case. Wdyt?
bq. Can you look into changes required in BackgroundObserver
Created OAK-5121 to follow up on that.
> Allow for external changes to have a CommitInfo attached
>
>
> Key: OAK-4898
> URL: https://issues.apache.org/jira/browse/OAK-4898
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6
>
> Attachments: OAK-4898-v1.patch, OAK-4898-v2.patch, OAK-4898-v3.patch
>
>
> Currently the observation logic relies on fact that CommitInfo being null
> means that changes are from other cluster node i.e. external changes.
> We should change this semantic and provide a different way to indicate that
> changes are external. This would allow a NodeStore implementation to still
> pass in a CommitInfo which captures useful information about commit like
> brief summary on what got changed which can be used for pre filtering
> (OAK-4796)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-5121) review CommitInfo==null in BackgroundObserver with isExternal change
[ https://issues.apache.org/jira/browse/OAK-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Egli updated OAK-5121: - Fix Version/s: 1.5.14 > review CommitInfo==null in BackgroundObserver with isExternal change > > > Key: OAK-5121 > URL: https://issues.apache.org/jira/browse/OAK-5121 > Project: Jackrabbit Oak > Issue Type: Task > Components: core >Affects Versions: 1.5.13 >Reporter: Stefan Egli >Assignee: Stefan Egli > Fix For: 1.6, 1.5.14 > > > OAK-4898 changes CommitInfo to be never null. This is the case outside of the > BackgroundObserver - but in the BackgroundObserver itself it is explicitly > set to null when compacting. > Once OAK-4898 is committed this task is about reviewing the implications in > BackgroundObserver wrt compaction and CommitInfo==null -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-5117) Improve checkMemory for compaction
[
https://issues.apache.org/jira/browse/OAK-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Parvulescu resolved OAK-5117.
--
Resolution: Fixed
thanks for the patch! fixed with
http://svn.apache.org/viewvc?rev=1770010&view=rev
> Improve checkMemory for compaction
> --
>
> Key: OAK-5117
> URL: https://issues.apache.org/jira/browse/OAK-5117
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Valentin Olteanu
>Assignee: Alex Parvulescu
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-5117-v2.patch, OAK-5117.patch
>
>
> The check introduced in OAK-4966 is relying on the first {{MemoryPoolMXBean}}
> found that matches the criteria. This means that, depending on the JVM, it
> will be checking any of the: PS Eden Space, PS Survivor Space or PS Old Gen
> Space - whichever is found first.
> I am proposing a more reliable way to check this:
> 1. check the biggest pool - this usually corresponds to PS Old Gen (by
> default 2/3 of the heap), which is more stable and reflects better the usage
> of the heap.
> 2. use {{getCollectionUsage()}} instead of {{getUsage()}} to check the state
> of the heap after a GC. This way we avoid checking during temporary peaks of
> the used heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-5121) review CommitInfo==null in BackgroundObserver with isExternal change
Stefan Egli created OAK-5121: Summary: review CommitInfo==null in BackgroundObserver with isExternal change Key: OAK-5121 URL: https://issues.apache.org/jira/browse/OAK-5121 Project: Jackrabbit Oak Issue Type: Task Components: core Affects Versions: 1.5.13 Reporter: Stefan Egli Assignee: Stefan Egli Fix For: 1.6 OAK-4898 changes CommitInfo to be never null. This is the case outside of the BackgroundObserver - but in the BackgroundObserver itself it is explicitly set to null when compacting. Once OAK-4898 is committed this task is about reviewing the implications in BackgroundObserver wrt compaction and CommitInfo==null -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-5117) Improve checkMemory for compaction
[
https://issues.apache.org/jira/browse/OAK-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670839#comment-15670839
]
Valentin Olteanu commented on OAK-5117:
---
I agree, it's simpler and cleaner.
> Improve checkMemory for compaction
> --
>
> Key: OAK-5117
> URL: https://issues.apache.org/jira/browse/OAK-5117
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Valentin Olteanu
>Assignee: Alex Parvulescu
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-5117-v2.patch, OAK-5117.patch
>
>
> The check introduced in OAK-4966 is relying on the first {{MemoryPoolMXBean}}
> found that matches the criteria. This means that, depending on the JVM, it
> will be checking any of the: PS Eden Space, PS Survivor Space or PS Old Gen
> Space - whichever is found first.
> I am proposing a more reliable way to check this:
> 1. check the biggest pool - this usually corresponds to PS Old Gen (by
> default 2/3 of the heap), which is more stable and reflects better the usage
> of the heap.
> 2. use {{getCollectionUsage()}} instead of {{getUsage()}} to check the state
> of the heap after a GC. This way we avoid checking during temporary peaks of
> the used heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3985) MongoDocumentStore: implement new conditional remove method
[ https://issues.apache.org/jira/browse/OAK-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Saurabh resolved OAK-3985. Resolution: Fixed > MongoDocumentStore: implement new conditional remove method > --- > > Key: OAK-3985 > URL: https://issues.apache.org/jira/browse/OAK-3985 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: mongomk >Reporter: Julian Reschke >Assignee: Vikas Saurabh > Fix For: 1.6, 1.5.14 > > > As introduced in OAK-3982. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3985) MongoDocumentStore: implement new conditional remove method
[ https://issues.apache.org/jira/browse/OAK-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670772#comment-15670772 ] Vikas Saurabh commented on OAK-3985: Thanks [~mreutegg]. Fixed this in [r1770003|https://svn.apache.org/r1770003]. > MongoDocumentStore: implement new conditional remove method > --- > > Key: OAK-3985 > URL: https://issues.apache.org/jira/browse/OAK-3985 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: mongomk >Reporter: Julian Reschke >Assignee: Vikas Saurabh > Fix For: 1.6, 1.5.14 > > > As introduced in OAK-3982. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3001) Simplify JournalGarbageCollector using a dedicated timestamp property
[
https://issues.apache.org/jira/browse/OAK-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670778#comment-15670778
]
Vikas Saurabh commented on OAK-3001:
Done (OAK-5119).
> Simplify JournalGarbageCollector using a dedicated timestamp property
> -
>
> Key: OAK-3001
> URL: https://issues.apache.org/jira/browse/OAK-3001
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
>Priority: Critical
> Labels: scalability
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-3001.take1.patch
>
>
> This subtask is about spawning out a
> [comment|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585733&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585733]
> from [~chetanm] re JournalGC:
> {quote}
> Further looking at JournalGarbageCollector ... it would be simpler if you
> record the journal entry timestamp as an attribute in JournalEntry document
> and then you can delete all the entries which are older than some time by a
> simple query. This would avoid fetching all the entries to be deleted on the
> Oak side
> {quote}
> and a corresponding
> [reply|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585870&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585870]
> from myself:
> {quote}
> Re querying by timestamp: that would indeed be simpler. With the current set
> of DocumentStore API however, I believe this is not possible. But:
> [DocumentStore.query|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java#L127]
> comes quite close: it would probably just require the opposite of that
> method too:
> {code}
> public List query(Collection collection,
> String fromKey,
> String toKey,
> String indexedProperty,
> long endValue,
> int limit) {
> {code}
> .. or what about generalizing this method to have both a {{startValue}} and
> an {{endValue}} - with {{-1}} indicating when one of them is not used?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3975) Change DELETE_BATCH_SIZE to 100 (from 450), and make it configurable
[
https://issues.apache.org/jira/browse/OAK-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670766#comment-15670766
]
Vikas Saurabh commented on OAK-3975:
Btw, OAK-5119 has undone this as OAK-3001 is committed now.
> Change DELETE_BATCH_SIZE to 100 (from 450), and make it configurable
>
>
> Key: OAK-3975
> URL: https://issues.apache.org/jira/browse/OAK-3975
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Affects Versions: 1.3.14
>Reporter: Stefan Egli
>Assignee: Stefan Egli
>Priority: Critical
> Fix For: 1.4, 1.3.16
>
>
> When large journal entries (with large {{_c}} values) exist, the journal gc
> can get into trouble as it currently loads 450 of them
> ({{DELETE_BATCH_SIZE}}). In order to improve this situation without having to
> fix OAK-3001 we can reduce the default of 450 to 100.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-5119) JournalGarbageCollector: journalGcBatchSize configuration is irrelevant post implementation tasks of bulk remove support in DocumentStore
[
https://issues.apache.org/jira/browse/OAK-5119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh resolved OAK-5119.
Resolution: Fixed
Fix Version/s: 1.6
Done on trunk at [r1770004|https://svn.apache.org/r1770004].
> JournalGarbageCollector: journalGcBatchSize configuration is irrelevant post
> implementation tasks of bulk remove support in DocumentStore
> -
>
> Key: OAK-5119
> URL: https://issues.apache.org/jira/browse/OAK-5119
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: documentmk
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
> Fix For: 1.6, 1.5.14
>
>
> Following up with this
> [comment|https://issues.apache.org/jira/browse/OAK-3001?focusedCommentId=15670100&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15670100]:
> {quote}
> ... noticed OAK-3975 in related issues. I think we should revert that
> configuration and code for house-keeping/maintenance!?
> {quote}
> OAK-3875 added a configuration to {{DocumentNodeStoreService}} to have
> configurable batch sizes while removing journal entries. With implementation
> tasks of OAK-3001, we do bulk remove at persistence layer and hence batched
> query isn't required. Thus, the configuration should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4898) Allow for external changes to have a CommitInfo attached
[ https://issues.apache.org/jira/browse/OAK-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-4898: - Attachment: OAK-4898-v3.patch [take 3|^OAK-4898-v3.patch] which makes CommitInfo non null. Due to this changes are bit widespread [~mreutegg] [~egli] [~mduerig] Please review (specially change in respective stores) Stefan - I have left BackgroundObserver not rely on non null as I am not sure about impact of collapsing. Can you look into changes required in BackgroundObserver > Allow for external changes to have a CommitInfo attached > > > Key: OAK-4898 > URL: https://issues.apache.org/jira/browse/OAK-4898 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.6 > > Attachments: OAK-4898-v1.patch, OAK-4898-v2.patch, OAK-4898-v3.patch > > > Currently the observation logic relies on fact that CommitInfo being null > means that changes are from other cluster node i.e. external changes. > We should change this semantic and provide a different way to indicate that > changes are external. This would allow a NodeStore implementation to still > pass in a CommitInfo which captures useful information about commit like > brief summary on what got changed which can be used for pre filtering > (OAK-4796) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-5117) Improve checkMemory for compaction
[
https://issues.apache.org/jira/browse/OAK-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Parvulescu updated OAK-5117:
-
Attachment: OAK-5117-v2.patch
[~volteanu] attached a simpler version, wdyt?
> Improve checkMemory for compaction
> --
>
> Key: OAK-5117
> URL: https://issues.apache.org/jira/browse/OAK-5117
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Valentin Olteanu
>Assignee: Alex Parvulescu
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-5117-v2.patch, OAK-5117.patch
>
>
> The check introduced in OAK-4966 is relying on the first {{MemoryPoolMXBean}}
> found that matches the criteria. This means that, depending on the JVM, it
> will be checking any of the: PS Eden Space, PS Survivor Space or PS Old Gen
> Space - whichever is found first.
> I am proposing a more reliable way to check this:
> 1. check the biggest pool - this usually corresponds to PS Old Gen (by
> default 2/3 of the heap), which is more stable and reflects better the usage
> of the heap.
> 2. use {{getCollectionUsage()}} instead of {{getUsage()}} to check the state
> of the heap after a GC. This way we avoid checking during temporary peaks of
> the used heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-4938) clarify contract for UpdateOp with missing operation on _id
[ https://issues.apache.org/jira/browse/OAK-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger resolved OAK-4938. --- Resolution: Fixed Fix Version/s: (was: 1.5.15) 1.5.14 All sub-tasks are resolved. > clarify contract for UpdateOp with missing operation on _id > --- > > Key: OAK-4938 > URL: https://issues.apache.org/jira/browse/OAK-4938 > Project: Jackrabbit Oak > Issue Type: Task > Components: documentmk >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Labels: technical_debt > Fix For: 1.6, 1.5.14 > > > See OAK-4937 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-4952) reject UpdateOp with set operation for _id, clarify API, add tests, fix implementations
[ https://issues.apache.org/jira/browse/OAK-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger resolved OAK-4952. --- Resolution: Fixed Fix Version/s: 1.5.14 Applied changes to trunk: http://svn.apache.org/r1769997 > reject UpdateOp with set operation for _id, clarify API, add tests, fix > implementations > --- > > Key: OAK-4952 > URL: https://issues.apache.org/jira/browse/OAK-4952 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: documentmk >Affects Versions: 1.5.12 >Reporter: Julian Reschke >Assignee: Marcel Reutegger >Priority: Minor > Fix For: 1.6, 1.5.14 > > Attachments: OAK-4952.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (OAK-4952) reject UpdateOp with set operation for _id, clarify API, add tests, fix implementations
[ https://issues.apache.org/jira/browse/OAK-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger reassigned OAK-4952: - Assignee: Marcel Reutegger (was: Julian Reschke) > reject UpdateOp with set operation for _id, clarify API, add tests, fix > implementations > --- > > Key: OAK-4952 > URL: https://issues.apache.org/jira/browse/OAK-4952 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: documentmk >Affects Versions: 1.5.12 >Reporter: Julian Reschke >Assignee: Marcel Reutegger >Priority: Minor > Fix For: 1.6 > > Attachments: OAK-4952.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-5120) Automatically convert *all* "or" queries to "union" for SQL-2, take 2
Thomas Mueller created OAK-5120:
---
Summary: Automatically convert *all* "or" queries to "union" for
SQL-2, take 2
Key: OAK-5120
URL: https://issues.apache.org/jira/browse/OAK-5120
Project: Jackrabbit Oak
Issue Type: Bug
Components: query
Reporter: Thomas Mueller
Assignee: Thomas Mueller
Fix For: 1.5.14
OAK-4681 is trying to convert "or" SQL-2 queries to union (if that's faster).
However, the following nested case is not yet converted fully to a union (one
"or" is converted, but the other one is not):
{noformat}
select * from [nt:unstructured] as [c]
where isdescendantnode('/tmp')
and ([a]=1 or [b]=2) and ([c]=3 or [d]=4)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3626) Provide bind credentials callback
[ https://issues.apache.org/jira/browse/OAK-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670619#comment-15670619 ] Alex COLLIGNON commented on OAK-3626: - While I agree with the outcome, I don't think resolved is the correct resolution for this issue. I would rather see it as won't fix. Not sure you can change it now, though. As far as I can see, RFC 227 (ConfigAdminUpdates) is actively edited and still in draft. I was wondering if we should already test some implementation in order to evaluate the work required to incorporate this in Oak. I guess that would require the oak team to maintain an internal release until approval of the spec, correct ? > Provide bind credentials callback > - > > Key: OAK-3626 > URL: https://issues.apache.org/jira/browse/OAK-3626 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: auth-ldap >Reporter: Tobias Bocanegra > > The ldap identity provider reads the admin bind credentials from the given > config which might originate from a un-encrypted source (eg. osgi config). > in order to facilitate secure provisioning of the bind credentials, the ldap > idp should offer some sort of credentials provider callback. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-5119) JournalGarbageCollector: journalGcBatchSize configuration is irrelevant post implementation tasks of bulk remove support in DocumentStore
[
https://issues.apache.org/jira/browse/OAK-5119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh updated OAK-5119:
---
Summary: JournalGarbageCollector: journalGcBatchSize configuration is
irrelevant post implementation tasks of bulk remove support in DocumentStore
(was: JourncalGarbageCollector: journalGcBatchSize configuration is irrelevant
post implementation tasks of bulk remove support in DocumentStore)
> JournalGarbageCollector: journalGcBatchSize configuration is irrelevant post
> implementation tasks of bulk remove support in DocumentStore
> -
>
> Key: OAK-5119
> URL: https://issues.apache.org/jira/browse/OAK-5119
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: documentmk
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
> Fix For: 1.5.14
>
>
> Following up with this
> [comment|https://issues.apache.org/jira/browse/OAK-3001?focusedCommentId=15670100&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15670100]:
> {quote}
> ... noticed OAK-3975 in related issues. I think we should revert that
> configuration and code for house-keeping/maintenance!?
> {quote}
> OAK-3875 added a configuration to {{DocumentNodeStoreService}} to have
> configurable batch sizes while removing journal entries. With implementation
> tasks of OAK-3001, we do bulk remove at persistence layer and hence batched
> query isn't required. Thus, the configuration should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-5119) JourncalGarbageCollector: journalGcBatchSize configuration is irrelevant post implementation tasks of bulk remove support in DocumentStore
Vikas Saurabh created OAK-5119:
--
Summary: JourncalGarbageCollector: journalGcBatchSize
configuration is irrelevant post implementation tasks of bulk remove support in
DocumentStore
Key: OAK-5119
URL: https://issues.apache.org/jira/browse/OAK-5119
Project: Jackrabbit Oak
Issue Type: Technical task
Components: documentmk
Reporter: Vikas Saurabh
Assignee: Vikas Saurabh
Priority: Minor
Fix For: 1.5.14
Following up with this
[comment|https://issues.apache.org/jira/browse/OAK-3001?focusedCommentId=15670100&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15670100]:
{quote}
... noticed OAK-3975 in related issues. I think we should revert that
configuration and code for house-keeping/maintenance!?
{quote}
OAK-3875 added a configuration to {{DocumentNodeStoreService}} to have
configurable batch sizes while removing journal entries. With implementation
tasks of OAK-3001, we do bulk remove at persistence layer and hence batched
query isn't required. Thus, the configuration should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-5117) Improve checkMemory for compaction
[
https://issues.apache.org/jira/browse/OAK-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670594#comment-15670594
]
Alex Parvulescu commented on OAK-5117:
--
clarified it with [~volteanu]. it seems all pools return {{true}} to
{{getCollectionUsage()}} (not the case for {{getUsage()}}) so the method is
indeed flawed.
> Improve checkMemory for compaction
> --
>
> Key: OAK-5117
> URL: https://issues.apache.org/jira/browse/OAK-5117
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Valentin Olteanu
>Assignee: Alex Parvulescu
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-5117.patch
>
>
> The check introduced in OAK-4966 is relying on the first {{MemoryPoolMXBean}}
> found that matches the criteria. This means that, depending on the JVM, it
> will be checking any of the: PS Eden Space, PS Survivor Space or PS Old Gen
> Space - whichever is found first.
> I am proposing a more reliable way to check this:
> 1. check the biggest pool - this usually corresponds to PS Old Gen (by
> default 2/3 of the heap), which is more stable and reflects better the usage
> of the heap.
> 2. use {{getCollectionUsage()}} instead of {{getUsage()}} to check the state
> of the heap after a GC. This way we avoid checking during temporary peaks of
> the used heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-5117) Improve checkMemory for compaction
[
https://issues.apache.org/jira/browse/OAK-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670549#comment-15670549
]
Alex Parvulescu commented on OAK-5117:
--
bq. This means that, depending on the JVM, it will be checking any of the: PS
Eden Space, PS Survivor Space or PS Old Gen Space - whichever is found first.
are you _absolutely sure_ this is the case? the mentioned criteria also
includes checking {{isCollectionUsageThresholdSupported()}}
this is what I see when loading the jmx stats for {{PS Survivor Space}} and
{{PS Eden Space}}
{noformat}
javax.management.RuntimeMBeanException:
java.lang.UnsupportedOperationException: Usage threshold is not supported
{noformat}
bq. use getCollectionUsage() instead of getUsage() to check the state of the
heap after a GC. This way we avoid checking during temporary peaks of the used
heap.
this is a good find! I've initially written the code to set a notification on
{{getUsage()}}, and then we switched to {{getCollectionUsage()}}, this probably
got left behind.
> Improve checkMemory for compaction
> --
>
> Key: OAK-5117
> URL: https://issues.apache.org/jira/browse/OAK-5117
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Valentin Olteanu
> Attachments: OAK-5117.patch
>
>
> The check introduced in OAK-4966 is relying on the first {{MemoryPoolMXBean}}
> found that matches the criteria. This means that, depending on the JVM, it
> will be checking any of the: PS Eden Space, PS Survivor Space or PS Old Gen
> Space - whichever is found first.
> I am proposing a more reliable way to check this:
> 1. check the biggest pool - this usually corresponds to PS Old Gen (by
> default 2/3 of the heap), which is more stable and reflects better the usage
> of the heap.
> 2. use {{getCollectionUsage()}} instead of {{getUsage()}} to check the state
> of the heap after a GC. This way we avoid checking during temporary peaks of
> the used heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OAK-5117) Improve checkMemory for compaction
[
https://issues.apache.org/jira/browse/OAK-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Parvulescu reassigned OAK-5117:
Assignee: Alex Parvulescu
> Improve checkMemory for compaction
> --
>
> Key: OAK-5117
> URL: https://issues.apache.org/jira/browse/OAK-5117
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Valentin Olteanu
>Assignee: Alex Parvulescu
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-5117.patch
>
>
> The check introduced in OAK-4966 is relying on the first {{MemoryPoolMXBean}}
> found that matches the criteria. This means that, depending on the JVM, it
> will be checking any of the: PS Eden Space, PS Survivor Space or PS Old Gen
> Space - whichever is found first.
> I am proposing a more reliable way to check this:
> 1. check the biggest pool - this usually corresponds to PS Old Gen (by
> default 2/3 of the heap), which is more stable and reflects better the usage
> of the heap.
> 2. use {{getCollectionUsage()}} instead of {{getUsage()}} to check the state
> of the heap after a GC. This way we avoid checking during temporary peaks of
> the used heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-5117) Improve checkMemory for compaction
[
https://issues.apache.org/jira/browse/OAK-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Parvulescu updated OAK-5117:
-
Fix Version/s: 1.5.14
1.6
> Improve checkMemory for compaction
> --
>
> Key: OAK-5117
> URL: https://issues.apache.org/jira/browse/OAK-5117
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Valentin Olteanu
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-5117.patch
>
>
> The check introduced in OAK-4966 is relying on the first {{MemoryPoolMXBean}}
> found that matches the criteria. This means that, depending on the JVM, it
> will be checking any of the: PS Eden Space, PS Survivor Space or PS Old Gen
> Space - whichever is found first.
> I am proposing a more reliable way to check this:
> 1. check the biggest pool - this usually corresponds to PS Old Gen (by
> default 2/3 of the heap), which is more stable and reflects better the usage
> of the heap.
> 2. use {{getCollectionUsage()}} instead of {{getUsage()}} to check the state
> of the heap after a GC. This way we avoid checking during temporary peaks of
> the used heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-5118) SynchronizationMBean: Deprecate syncAllUsers and syncAllExternalUsers
angela created OAK-5118:
---
Summary: SynchronizationMBean: Deprecate syncAllUsers and
syncAllExternalUsers
Key: OAK-5118
URL: https://issues.apache.org/jira/browse/OAK-5118
Project: Jackrabbit Oak
Issue Type: Improvement
Components: auth-external
Reporter: angela
The following 2 methods defined by {{SyncronizationMBean}} have been introduced
with a warning about being potentially expensive [0]:
- {{syncAllUsers}}
- {{syncAllExternalUsers}}
To me defining API that is designed with scalability issues in mind, looks like
a troublesome pattern. As far as I understood from the history associated with
this, I assume that it was introduced to allow for upfront sync to workaround
other issues faced by sync-upon-login.
With the latest improvements and features introduced wrt to the user-sync, I
think we should move forward deprecating the troublesome methods for the
upcoming 1.6 release.
[0] Quote from Javadoc: "Note that this can be an expensive operation since all
potential users need to be examined."
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-5117) Improve checkMemory for compaction
[
https://issues.apache.org/jira/browse/OAK-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670476#comment-15670476
]
Valentin Olteanu commented on OAK-5117:
---
[~alex.parvulescu], could you please take a look at the proposed patch?
> Improve checkMemory for compaction
> --
>
> Key: OAK-5117
> URL: https://issues.apache.org/jira/browse/OAK-5117
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Valentin Olteanu
> Attachments: OAK-5117.patch
>
>
> The check introduced in OAK-4966 is relying on the first {{MemoryPoolMXBean}}
> found that matches the criteria. This means that, depending on the JVM, it
> will be checking any of the: PS Eden Space, PS Survivor Space or PS Old Gen
> Space - whichever is found first.
> I am proposing a more reliable way to check this:
> 1. check the biggest pool - this usually corresponds to PS Old Gen (by
> default 2/3 of the heap), which is more stable and reflects better the usage
> of the heap.
> 2. use {{getCollectionUsage()}} instead of {{getUsage()}} to check the state
> of the heap after a GC. This way we avoid checking during temporary peaks of
> the used heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-5117) Improve checkMemory for compaction
[
https://issues.apache.org/jira/browse/OAK-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentin Olteanu updated OAK-5117:
--
Attachment: OAK-5117.patch
> Improve checkMemory for compaction
> --
>
> Key: OAK-5117
> URL: https://issues.apache.org/jira/browse/OAK-5117
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Valentin Olteanu
> Attachments: OAK-5117.patch
>
>
> The check introduced in OAK-4966 is relying on the first {{MemoryPoolMXBean}}
> found that matches the criteria. This means that, depending on the JVM, it
> will be checking any of the: PS Eden Space, PS Survivor Space or PS Old Gen
> Space - whichever is found first.
> I am proposing a more reliable way to check this:
> 1. check the biggest pool - this usually corresponds to PS Old Gen (by
> default 2/3 of the heap), which is more stable and reflects better the usage
> of the heap.
> 2. use {{getCollectionUsage()}} instead of {{getUsage()}} to check the state
> of the heap after a GC. This way we avoid checking during temporary peaks of
> the used heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-5117) Improve checkMemory for compaction
Valentin Olteanu created OAK-5117:
-
Summary: Improve checkMemory for compaction
Key: OAK-5117
URL: https://issues.apache.org/jira/browse/OAK-5117
Project: Jackrabbit Oak
Issue Type: Improvement
Components: segment-tar
Reporter: Valentin Olteanu
The check introduced in OAK-4966 is relying on the first {{MemoryPoolMXBean}}
found that matches the criteria. This means that, depending on the JVM, it will
be checking any of the: PS Eden Space, PS Survivor Space or PS Old Gen Space -
whichever is found first.
I am proposing a more reliable way to check this:
1. check the biggest pool - this usually corresponds to PS Old Gen (by default
2/3 of the heap), which is more stable and reflects better the usage of the
heap.
2. use {{getCollectionUsage()}} instead of {{getUsage()}} to check the state of
the heap after a GC. This way we avoid checking during temporary peaks of the
used heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4179) Difference in splitdn and User DN paths options in Oak
[ https://issues.apache.org/jira/browse/OAK-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angela updated OAK-4179: Issue Type: Improvement (was: Bug) > Difference in splitdn and User DN paths options in Oak > -- > > Key: OAK-4179 > URL: https://issues.apache.org/jira/browse/OAK-4179 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: auth-ldap >Affects Versions: 1.2.11 >Reporter: Kunwar Preet Singh >Priority: Minor > Labels: ldap > > In Jackrabbit, We had an option of "splitdn" in order to maintain AD > structure while registering AD users but now in Oak(1.2.x) we didn’t find > anything similar to configure. > Assuming “User DN paths”(user.makeDnPath) as similar configuration in Oak, it > didn’t work as expected. > Below are the differences for the paths synced in case of CRX and Oak :- > After the sync happens, users created are under the path is as follows : > For Jackrabbit(using splitdn option), > Path= */home/users/MIS/TSD/329421* > For Oak 1.2.x (Using 'User DN paths' option ) > Path= */home/users/CN=Mosca\, > Patrick/OU=TSD/OU=MIS/OU=CITYOFCHICAGO/DC=cityofchicago/DC=local/HxOYFwEec_hmNHfoW68d* > It seems like the algorithm with Oak is different than the previous version, > Hence the difference in path structure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-3626) Provide bind credentials callback
[ https://issues.apache.org/jira/browse/OAK-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angela resolved OAK-3626. - Resolution: Resolved > Provide bind credentials callback > - > > Key: OAK-3626 > URL: https://issues.apache.org/jira/browse/OAK-3626 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: auth-ldap >Reporter: Tobias Bocanegra > > The ldap identity provider reads the admin bind credentials from the given > config which might originate from a un-encrypted source (eg. osgi config). > in order to facilitate secure provisioning of the bind credentials, the ldap > idp should offer some sort of credentials provider callback. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3626) Provide bind credentials callback
[
https://issues.apache.org/jira/browse/OAK-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670455#comment-15670455
]
angela commented on OAK-3626:
-
based on discussions with [~asanso] and [~acollign] I got convinced that we
should leverage RFC 227 [0] to get this solved in a generic way. This would
beneficial for every type of {{ExternalIdentityProvider}} implementation and
not just limited to the LDAP case we kept discussing here.
[0]
https://github.com/osgi/design/blob/master/rfcs/rfc0227/rfc-0227-ConfigAdminUpdates.pdf
> Provide bind credentials callback
> -
>
> Key: OAK-3626
> URL: https://issues.apache.org/jira/browse/OAK-3626
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: auth-ldap
>Reporter: Tobias Bocanegra
>
> The ldap identity provider reads the admin bind credentials from the given
> config which might originate from a un-encrypted source (eg. osgi config).
> in order to facilitate secure provisioning of the bind credentials, the ldap
> idp should offer some sort of credentials provider callback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3606) Improvements for IndexStatsMBean usage
[ https://issues.apache.org/jira/browse/OAK-3606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670451#comment-15670451 ] Chetan Mehrotra commented on OAK-3606: -- This would need some change. The IndexUpdate constructor used in AsyncIndexUpdate would need to be changed which passes in a CommitInfo instance. So far the CommitInfo accessed in LucenIndexEditorProvider was working for sync commit hence worked fine. > Improvements for IndexStatsMBean usage > -- > > Key: OAK-3606 > URL: https://issues.apache.org/jira/browse/OAK-3606 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, lucene >Affects Versions: 1.3.9 >Reporter: Thierry Ygé >Assignee: Manfred Baedke > Fix For: 1.6, 1.5.14 > > Attachments: adding_new_MBean.patch, > new_mbean_interface_and_implementation.patch > > > When running integration tests, it is common to have the need to wait for the > async indexes to have been executed. So that the test can successfully > validate operations that depend on the search result. > With the current IndexStatsMBean implementation it cannot return the start > time of the last successful indexing. It provide a "LastIndexedTime" which is > not sufficient to know if changes made recently are now indexed. > The idea is to set the start time as value of a new attribute (i.e > "StartLastSuccessIndexedTime") to the IndexStatsMBean. > Then create a new Mbean that calculate from all existing IndexStatsMBean (as > multiple are possible now) the oldest "StartLastSuccessIndexedTime". > That will allow integration tests to be able to wait until that oldest > "StartLastSuccessIndexedTime" is greater than the time it started to wait. > Attached is a sample patch containing the necessary changes (for a Oak core > 1.4.0-SNAPSHOT). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-5116) GCJournal should persist size only when compaction is successful
[ https://issues.apache.org/jira/browse/OAK-5116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Parvulescu updated OAK-5116: - Fix Version/s: 1.5.14 1.6 > GCJournal should persist size only when compaction is successful > > > Key: OAK-5116 > URL: https://issues.apache.org/jira/browse/OAK-5116 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-tar >Reporter: Alex Parvulescu >Assignee: Alex Parvulescu > Fix For: 1.6, 1.5.14 > > > In the case where compaction fails, the gc journal persists the current size, > delaying the next compaction cycle and potentially introducing even more work > for the gc process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-5116) GCJournal should persist size only when compaction is successful
Alex Parvulescu created OAK-5116: Summary: GCJournal should persist size only when compaction is successful Key: OAK-5116 URL: https://issues.apache.org/jira/browse/OAK-5116 Project: Jackrabbit Oak Issue Type: Improvement Components: segment-tar Reporter: Alex Parvulescu Assignee: Alex Parvulescu In the case where compaction fails, the gc journal persists the current size, delaying the next compaction cycle and potentially introducing even more work for the gc process. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3001) Simplify JournalGarbageCollector using a dedicated timestamp property
[
https://issues.apache.org/jira/browse/OAK-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670200#comment-15670200
]
Marcel Reutegger commented on OAK-3001:
---
That's better, yes.
> Simplify JournalGarbageCollector using a dedicated timestamp property
> -
>
> Key: OAK-3001
> URL: https://issues.apache.org/jira/browse/OAK-3001
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
>Priority: Critical
> Labels: scalability
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-3001.take1.patch
>
>
> This subtask is about spawning out a
> [comment|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585733&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585733]
> from [~chetanm] re JournalGC:
> {quote}
> Further looking at JournalGarbageCollector ... it would be simpler if you
> record the journal entry timestamp as an attribute in JournalEntry document
> and then you can delete all the entries which are older than some time by a
> simple query. This would avoid fetching all the entries to be deleted on the
> Oak side
> {quote}
> and a corresponding
> [reply|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585870&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585870]
> from myself:
> {quote}
> Re querying by timestamp: that would indeed be simpler. With the current set
> of DocumentStore API however, I believe this is not possible. But:
> [DocumentStore.query|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java#L127]
> comes quite close: it would probably just require the opposite of that
> method too:
> {code}
> public List query(Collection collection,
> String fromKey,
> String toKey,
> String indexedProperty,
> long endValue,
> int limit) {
> {code}
> .. or what about generalizing this method to have both a {{startValue}} and
> an {{endValue}} - with {{-1}} indicating when one of them is not used?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3001) Simplify JournalGarbageCollector using a dedicated timestamp property
[
https://issues.apache.org/jira/browse/OAK-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670194#comment-15670194
]
Vikas Saurabh commented on OAK-3001:
Thanks for the review guys. Would incorporate feedback in a bit.
> Simplify JournalGarbageCollector using a dedicated timestamp property
> -
>
> Key: OAK-3001
> URL: https://issues.apache.org/jira/browse/OAK-3001
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
>Priority: Critical
> Labels: scalability
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-3001.take1.patch
>
>
> This subtask is about spawning out a
> [comment|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585733&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585733]
> from [~chetanm] re JournalGC:
> {quote}
> Further looking at JournalGarbageCollector ... it would be simpler if you
> record the journal entry timestamp as an attribute in JournalEntry document
> and then you can delete all the entries which are older than some time by a
> simple query. This would avoid fetching all the entries to be deleted on the
> Oak side
> {quote}
> and a corresponding
> [reply|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585870&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585870]
> from myself:
> {quote}
> Re querying by timestamp: that would indeed be simpler. With the current set
> of DocumentStore API however, I believe this is not possible. But:
> [DocumentStore.query|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java#L127]
> comes quite close: it would probably just require the opposite of that
> method too:
> {code}
> public List query(Collection collection,
> String fromKey,
> String toKey,
> String indexedProperty,
> long endValue,
> int limit) {
> {code}
> .. or what about generalizing this method to have both a {{startValue}} and
> an {{endValue}} - with {{-1}} indicating when one of them is not used?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3001) Simplify JournalGarbageCollector using a dedicated timestamp property
[
https://issues.apache.org/jira/browse/OAK-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670192#comment-15670192
]
Vikas Saurabh commented on OAK-3001:
[~mreutegg], oh!, i meant if I need to update this code to utilize (extended)
condition class. Your comment clarifies what I was seeing.
> Simplify JournalGarbageCollector using a dedicated timestamp property
> -
>
> Key: OAK-3001
> URL: https://issues.apache.org/jira/browse/OAK-3001
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
>Priority: Critical
> Labels: scalability
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-3001.take1.patch
>
>
> This subtask is about spawning out a
> [comment|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585733&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585733]
> from [~chetanm] re JournalGC:
> {quote}
> Further looking at JournalGarbageCollector ... it would be simpler if you
> record the journal entry timestamp as an attribute in JournalEntry document
> and then you can delete all the entries which are older than some time by a
> simple query. This would avoid fetching all the entries to be deleted on the
> Oak side
> {quote}
> and a corresponding
> [reply|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585870&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585870]
> from myself:
> {quote}
> Re querying by timestamp: that would indeed be simpler. With the current set
> of DocumentStore API however, I believe this is not possible. But:
> [DocumentStore.query|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java#L127]
> comes quite close: it would probably just require the opposite of that
> method too:
> {code}
> public List query(Collection collection,
> String fromKey,
> String toKey,
> String indexedProperty,
> long endValue,
> int limit) {
> {code}
> .. or what about generalizing this method to have both a {{startValue}} and
> an {{endValue}} - with {{-1}} indicating when one of them is not used?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3001) Simplify JournalGarbageCollector using a dedicated timestamp property
[
https://issues.apache.org/jira/browse/OAK-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670187#comment-15670187
]
Vikas Saurabh commented on OAK-3001:
[~mreutegg], I should've clarified my statement better. What I meant was that
the test was leaving a scope to skip the edge ceck (and I "think" that travis
somehow had hit that... although I can't formulate any logic for "why" with
VirtualClock in place).
bq. you added more
Yes, I actually want to just do a hard check for cp-head
bq. I'm also not sure we should add it. It assumes a journal entry is written
on DocumentNodeStore init.
We can do a fake commit, storeHead, runBkOps before creating a check-point -
that should create a journal entry and give us head at checkpoint. That should
be ok, right?
> Simplify JournalGarbageCollector using a dedicated timestamp property
> -
>
> Key: OAK-3001
> URL: https://issues.apache.org/jira/browse/OAK-3001
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
>Priority: Critical
> Labels: scalability
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-3001.take1.patch
>
>
> This subtask is about spawning out a
> [comment|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585733&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585733]
> from [~chetanm] re JournalGC:
> {quote}
> Further looking at JournalGarbageCollector ... it would be simpler if you
> record the journal entry timestamp as an attribute in JournalEntry document
> and then you can delete all the entries which are older than some time by a
> simple query. This would avoid fetching all the entries to be deleted on the
> Oak side
> {quote}
> and a corresponding
> [reply|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585870&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585870]
> from myself:
> {quote}
> Re querying by timestamp: that would indeed be simpler. With the current set
> of DocumentStore API however, I believe this is not possible. But:
> [DocumentStore.query|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java#L127]
> comes quite close: it would probably just require the opposite of that
> method too:
> {code}
> public List query(Collection collection,
> String fromKey,
> String toKey,
> String indexedProperty,
> long endValue,
> int limit) {
> {code}
> .. or what about generalizing this method to have both a {{startValue}} and
> an {{endValue}} - with {{-1}} indicating when one of them is not used?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2108) Killing a cluster node may stop async index update to to 30 minutes
[ https://issues.apache.org/jira/browse/OAK-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra updated OAK-2108: - Attachment: OAK-2108-v1.patch [patch|^OAK-2108-v1.patch] for the same. [~alexparvulescu] Please review. Remember it touches lease stuff ;) > Killing a cluster node may stop async index update to to 30 minutes > --- > > Key: OAK-2108 > URL: https://issues.apache.org/jira/browse/OAK-2108 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: query >Reporter: Thomas Mueller >Assignee: Chetan Mehrotra > Labels: resilience > Fix For: 1.6 > > Attachments: OAK-2108-v1.patch > > > When killing a cluster node that is running the sync index update, then this > async index update will not run for up to 15 minutes, because the lease time > is set to 15 minutes. > I think the lease time should be much smaller, for example 1 minute, or maybe > even 10 seconds. > Also, we might need to better document this issue (in addition to the warning > in the log file). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3001) Simplify JournalGarbageCollector using a dedicated timestamp property
[
https://issues.apache.org/jira/browse/OAK-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670134#comment-15670134
]
Stefan Egli commented on OAK-3001:
--
bq. can you please review if I wavered off from what we planned to do?
lgtm (reviewed API and usage)
> Simplify JournalGarbageCollector using a dedicated timestamp property
> -
>
> Key: OAK-3001
> URL: https://issues.apache.org/jira/browse/OAK-3001
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
>Priority: Critical
> Labels: scalability
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-3001.take1.patch
>
>
> This subtask is about spawning out a
> [comment|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585733&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585733]
> from [~chetanm] re JournalGC:
> {quote}
> Further looking at JournalGarbageCollector ... it would be simpler if you
> record the journal entry timestamp as an attribute in JournalEntry document
> and then you can delete all the entries which are older than some time by a
> simple query. This would avoid fetching all the entries to be deleted on the
> Oak side
> {quote}
> and a corresponding
> [reply|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585870&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585870]
> from myself:
> {quote}
> Re querying by timestamp: that would indeed be simpler. With the current set
> of DocumentStore API however, I believe this is not possible. But:
> [DocumentStore.query|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java#L127]
> comes quite close: it would probably just require the opposite of that
> method too:
> {code}
> public List query(Collection collection,
> String fromKey,
> String toKey,
> String indexedProperty,
> long endValue,
> int limit) {
> {code}
> .. or what about generalizing this method to have both a {{startValue}} and
> an {{endValue}} - with {{-1}} indicating when one of them is not used?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OAK-4292) Document Oak segment-tar
[ https://issues.apache.org/jira/browse/OAK-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francesco Mari reassigned OAK-4292: --- Assignee: Francesco Mari (was: Michael Dürig) > Document Oak segment-tar > > > Key: OAK-4292 > URL: https://issues.apache.org/jira/browse/OAK-4292 > Project: Jackrabbit Oak > Issue Type: Task > Components: doc, segment-tar >Reporter: Michael Dürig >Assignee: Francesco Mari > Labels: documentation, gc > Fix For: 1.6, 1.5.17 > > > Document Oak Segment Tar. Specifically: > * New and changed configuration and monitoring options > * Changes in gc (OAK-3348 et. all) > * Changes in segment / tar format (OAK-3348) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3001) Simplify JournalGarbageCollector using a dedicated timestamp property
[
https://issues.apache.org/jira/browse/OAK-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670100#comment-15670100
]
Stefan Egli commented on OAK-3001:
--
bq. noticed OAK-3975 in related issues. I think we should revert that
configuration and code for house-keeping/maintenance!?
+1
> Simplify JournalGarbageCollector using a dedicated timestamp property
> -
>
> Key: OAK-3001
> URL: https://issues.apache.org/jira/browse/OAK-3001
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
>Priority: Critical
> Labels: scalability
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-3001.take1.patch
>
>
> This subtask is about spawning out a
> [comment|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585733&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585733]
> from [~chetanm] re JournalGC:
> {quote}
> Further looking at JournalGarbageCollector ... it would be simpler if you
> record the journal entry timestamp as an attribute in JournalEntry document
> and then you can delete all the entries which are older than some time by a
> simple query. This would avoid fetching all the entries to be deleted on the
> Oak side
> {quote}
> and a corresponding
> [reply|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585870&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585870]
> from myself:
> {quote}
> Re querying by timestamp: that would indeed be simpler. With the current set
> of DocumentStore API however, I believe this is not possible. But:
> [DocumentStore.query|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java#L127]
> comes quite close: it would probably just require the opposite of that
> method too:
> {code}
> public List query(Collection collection,
> String fromKey,
> String toKey,
> String indexedProperty,
> long endValue,
> int limit) {
> {code}
> .. or what about generalizing this method to have both a {{startValue}} and
> an {{endValue}} - with {{-1}} indicating when one of them is not used?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3985) MongoDocumentStore: implement new conditional remove method
[
https://issues.apache.org/jira/browse/OAK-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669976#comment-15669976
]
Marcel Reutegger commented on OAK-3985:
---
The finally block surrounding {{dbCollection.remove()}} should invalidate the
cache even when {{num}} is zero. When the call fails with an exception some
documents may still have been removed. MongoDB does not guarantee atomicity
over multiple documents.
> MongoDocumentStore: implement new conditional remove method
> ---
>
> Key: OAK-3985
> URL: https://issues.apache.org/jira/browse/OAK-3985
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: mongomk
>Reporter: Julian Reschke
>Assignee: Vikas Saurabh
> Fix For: 1.6, 1.5.14
>
>
> As introduced in OAK-3982.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4898) Allow for external changes to have a CommitInfo attached
[
https://issues.apache.org/jira/browse/OAK-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669965#comment-15669965
]
Chetan Mehrotra commented on OAK-4898:
--
bq. what about existing code outside of Oak that currently implement Observer
(e.g. Apache Sling) and work according to the current contract?
At least for Sling code was changed to move to EventListener (See SLING-6138).
Also the api is currently not versioned
{noformat}
org.apache.jackrabbit.oak.security,version=0.0.0
org.apache.jackrabbit.oak.spi.commit,version=0.0.0
{noformat}
So for making CommitInfo NonNull should I come up with a static
CommitInfo.EMPTY_EXTERNAL which would be used by NodeStore where they are
passing null?
Would come up with a new patch then
> Allow for external changes to have a CommitInfo attached
>
>
> Key: OAK-4898
> URL: https://issues.apache.org/jira/browse/OAK-4898
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6
>
> Attachments: OAK-4898-v1.patch, OAK-4898-v2.patch
>
>
> Currently the observation logic relies on fact that CommitInfo being null
> means that changes are from other cluster node i.e. external changes.
> We should change this semantic and provide a different way to indicate that
> changes are external. This would allow a NodeStore implementation to still
> pass in a CommitInfo which captures useful information about commit like
> brief summary on what got changed which can be used for pre filtering
> (OAK-4796)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4898) Allow for external changes to have a CommitInfo attached
[
https://issues.apache.org/jira/browse/OAK-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669952#comment-15669952
]
Marcel Reutegger commented on OAK-4898:
---
bq. Making CommitInfo non null would require change in NodeStore
implementations also. That would also make code simpler. What do others think?
I'd be in favour of such a move, since your patch will lead to an incompatible
change of the Observer.contentChanged() contract anyway. But then, what about
existing code outside of Oak that currently implement Observer (e.g. Apache
Sling) and work according to the current contract?
>From Observer JavaDoc:
{quote}
A repository may capture the optional CommitInfo instance passed to a commit
and make it available to observers along with the committed content changes. In
such cases, i.e. when the commit info argument is non-nul}, the reported
content change is guaranteed to contain only changes from that specific commit
(and the applied commit hooks).
{quote}
> Allow for external changes to have a CommitInfo attached
>
>
> Key: OAK-4898
> URL: https://issues.apache.org/jira/browse/OAK-4898
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6
>
> Attachments: OAK-4898-v1.patch, OAK-4898-v2.patch
>
>
> Currently the observation logic relies on fact that CommitInfo being null
> means that changes are from other cluster node i.e. external changes.
> We should change this semantic and provide a different way to indicate that
> changes are external. This would allow a NodeStore implementation to still
> pass in a CommitInfo which captures useful information about commit like
> brief summary on what got changed which can be used for pre filtering
> (OAK-4796)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4898) Allow for external changes to have a CommitInfo attached
[
https://issues.apache.org/jira/browse/OAK-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-4898:
-
Attachment: OAK-4898-v2.patch
[updated patch|^OAK-4898-v2.patch] which adds a static method {{isExternal}} to
{{CommitInfo}}
> Allow for external changes to have a CommitInfo attached
>
>
> Key: OAK-4898
> URL: https://issues.apache.org/jira/browse/OAK-4898
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6
>
> Attachments: OAK-4898-v1.patch, OAK-4898-v2.patch
>
>
> Currently the observation logic relies on fact that CommitInfo being null
> means that changes are from other cluster node i.e. external changes.
> We should change this semantic and provide a different way to indicate that
> changes are external. This would allow a NodeStore implementation to still
> pass in a CommitInfo which captures useful information about commit like
> brief summary on what got changed which can be used for pre filtering
> (OAK-4796)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-5113) Avoid dispatching changes for checkpoint related changes
[
https://issues.apache.org/jira/browse/OAK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-5113:
-
Component/s: segmentmk
> Avoid dispatching changes for checkpoint related changes
>
>
> Key: OAK-5113
> URL: https://issues.apache.org/jira/browse/OAK-5113
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar, segmentmk
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-5113-v1.patch
>
>
> As discussed
> [here|https://issues.apache.org/jira/browse/OAK-4935?focusedCommentId=15666420&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15666420]
> {{SegmentNodeStore}} currently dispatches changes even for checkpoint
> related operations. Given these changes are under super root and not affect
> the tree under normal root we can avoid dispatching such changes at all
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-5113) Avoid dispatching changes for checkpoint related changes
[
https://issues.apache.org/jira/browse/OAK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-5113.
--
Resolution: Fixed
Fix Version/s: 1.5.14
Done changes with 1769947,1769948, 1769949
> Avoid dispatching changes for checkpoint related changes
>
>
> Key: OAK-5113
> URL: https://issues.apache.org/jira/browse/OAK-5113
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar, segmentmk
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-5113-v1.patch
>
>
> As discussed
> [here|https://issues.apache.org/jira/browse/OAK-4935?focusedCommentId=15666420&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15666420]
> {{SegmentNodeStore}} currently dispatches changes even for checkpoint
> related operations. Given these changes are under super root and not affect
> the tree under normal root we can avoid dispatching such changes at all
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3001) Simplify JournalGarbageCollector using a dedicated timestamp property
[
https://issues.apache.org/jira/browse/OAK-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669900#comment-15669900
]
Marcel Reutegger commented on OAK-3001:
---
bq. I didn't extend Condition class for this as afaics usage of condition
seemed to be somehow linked with id (I'm not completely sure of the intended
contract). May be that bit needs correction/refactor.
I think we can keep {{Condition}} as is but add clarifications to JavaDoc if
needed. Conditions are tied to an UpdateOp, which in turn in linked with an ID.
You can construct an independent Condition, but its usage in the DocumentStore
interface is always related to an UpdateOp (or id/key).
> Simplify JournalGarbageCollector using a dedicated timestamp property
> -
>
> Key: OAK-3001
> URL: https://issues.apache.org/jira/browse/OAK-3001
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
>Priority: Critical
> Labels: scalability
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-3001.take1.patch
>
>
> This subtask is about spawning out a
> [comment|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585733&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585733]
> from [~chetanm] re JournalGC:
> {quote}
> Further looking at JournalGarbageCollector ... it would be simpler if you
> record the journal entry timestamp as an attribute in JournalEntry document
> and then you can delete all the entries which are older than some time by a
> simple query. This would avoid fetching all the entries to be deleted on the
> Oak side
> {quote}
> and a corresponding
> [reply|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585870&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585870]
> from myself:
> {quote}
> Re querying by timestamp: that would indeed be simpler. With the current set
> of DocumentStore API however, I believe this is not possible. But:
> [DocumentStore.query|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java#L127]
> comes quite close: it would probably just require the opposite of that
> method too:
> {code}
> public List query(Collection collection,
> String fromKey,
> String toKey,
> String indexedProperty,
> long endValue,
> int limit) {
> {code}
> .. or what about generalizing this method to have both a {{startValue}} and
> an {{endValue}} - with {{-1}} indicating when one of them is not used?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3001) Simplify JournalGarbageCollector using a dedicated timestamp property
[
https://issues.apache.org/jira/browse/OAK-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669885#comment-15669885
]
Marcel Reutegger commented on OAK-3001:
---
AFAICS you didn't change any checks, you added more. The additional checks are
OK, but go beyond the core issue this test is checking. I'm also not sure we
should add it. It assumes a journal entry is written on DocumentNodeStore init.
While that may be the case right now, it is probably not strictly necessary and
could change in the future.
> Simplify JournalGarbageCollector using a dedicated timestamp property
> -
>
> Key: OAK-3001
> URL: https://issues.apache.org/jira/browse/OAK-3001
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk
>Reporter: Stefan Egli
>Assignee: Vikas Saurabh
>Priority: Critical
> Labels: scalability
> Fix For: 1.6, 1.5.14
>
> Attachments: OAK-3001.take1.patch
>
>
> This subtask is about spawning out a
> [comment|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585733&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585733]
> from [~chetanm] re JournalGC:
> {quote}
> Further looking at JournalGarbageCollector ... it would be simpler if you
> record the journal entry timestamp as an attribute in JournalEntry document
> and then you can delete all the entries which are older than some time by a
> simple query. This would avoid fetching all the entries to be deleted on the
> Oak side
> {quote}
> and a corresponding
> [reply|https://issues.apache.org/jira/browse/OAK-2829?focusedCommentId=14585870&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585870]
> from myself:
> {quote}
> Re querying by timestamp: that would indeed be simpler. With the current set
> of DocumentStore API however, I believe this is not possible. But:
> [DocumentStore.query|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/document/DocumentStore.java#L127]
> comes quite close: it would probably just require the opposite of that
> method too:
> {code}
> public List query(Collection collection,
> String fromKey,
> String toKey,
> String indexedProperty,
> long endValue,
> int limit) {
> {code}
> .. or what about generalizing this method to have both a {{startValue}} and
> an {{endValue}} - with {{-1}} indicating when one of them is not used?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4898) Allow for external changes to have a CommitInfo attached
[ https://issues.apache.org/jira/browse/OAK-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669880#comment-15669880 ] Stefan Egli commented on OAK-4898: -- bq. But would we want to merge the ChangeSet also when collapsing we could do that indeed - given the ChangeSet limits the number of items it holds there shouldn't be a memory problem. Something for OAK-5101 to pick up perhaps (adding a note there) > Allow for external changes to have a CommitInfo attached > > > Key: OAK-4898 > URL: https://issues.apache.org/jira/browse/OAK-4898 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.6 > > Attachments: OAK-4898-v1.patch > > > Currently the observation logic relies on fact that CommitInfo being null > means that changes are from other cluster node i.e. external changes. > We should change this semantic and provide a different way to indicate that > changes are external. This would allow a NodeStore implementation to still > pass in a CommitInfo which captures useful information about commit like > brief summary on what got changed which can be used for pre filtering > (OAK-4796) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-5101) Enable prefiltering for external changes
[ https://issues.apache.org/jira/browse/OAK-5101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669884#comment-15669884 ] Stefan Egli commented on OAK-5101: -- Note that when collapsing we could merge the ChangeSets, see [this thread|https://issues.apache.org/jira/browse/OAK-4898?focusedCommentId=15669880&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15669880] > Enable prefiltering for external changes > > > Key: OAK-5101 > URL: https://issues.apache.org/jira/browse/OAK-5101 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Chetan Mehrotra > Fix For: 1.6 > > > With OAK-4796 its possible to prefilter an event change and avoid performing > the diff if it can be determined that there is nothing of interest for the > given listener. Currently this works for local changes. > We should extend this support for external changes as those are more > expensive to compute -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-4114) Cached lucene index gets corrupted in case of unclean shutdown and journal rollback in SegmentNodeStore
[
https://issues.apache.org/jira/browse/OAK-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669866#comment-15669866
]
Chetan Mehrotra edited comment on OAK-4114 at 11/16/16 8:53 AM:
Did a fix with 1769940, 1769939
Now before any Directory is opened for any index a simple sanity check is done
where local index files are compared against remote in terms of file size. In
case of any mismatch whole of the local index dir is purged and the files would
be fetched again later
This check is weak as it compares file size only. But should catch most of such
cases.
Later we can look into some checksum based approach
was (Author: chetanm):
Did a fix with 1769940, 1769939
Now before and Directory is opened for any index a simple sanity check is done
where local index files are compared against remote in terms of file size. In
case of any mismatch whole of the local index dir is purged and the files would
be fetched again later
This check is weak as it compares file size only. But should catch most of such
cases.
Later we can look into some checksum based approach
> Cached lucene index gets corrupted in case of unclean shutdown and journal
> rollback in SegmentNodeStore
> ---
>
> Key: OAK-4114
> URL: https://issues.apache.org/jira/browse/OAK-4114
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Critical
> Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4,
> resilience
> Fix For: 1.6, 1.5.14
>
>
> Currently Oak Lucene support would copy index files to local file system as
> part of CopyOnRead feature. In one of the setup it has been observed that
> index logic was failing with following error
> {noformat}
> 04.02.2016 17:47:52.391 *WARN* [oak-lucene-3]
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier
> [/oak:index/lucene] Found local copy for _2ala.cfs in
> MMapDirectory@/mnt/crx/author/crx-quickstart/repository/index/e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293/1
>
> lockFactory=NativeFSLockFactory@/mnt/crx/author/crx-quickstart/repository/index/e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293/1
> but size of local 9320 differs from remote 3714150. Content would be read
> from remote file only
> 04.02.2016 17:47:52.399 *WARN* [oak-lucene-3]
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier
> [/oak:index/lucene] Found local copy for segments_28je in
> MMapDirectory@/mnt/crx/author/crx-quickstart/repository/index/e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293/1
>
> lockFactory=NativeFSLockFactory@/mnt/crx/author/crx-quickstart/repository/index/e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293/1
> but size of local 1214 differs from remote 1175. Content would be read from
> remote file only
> 04.02.2016 17:47:52.491 *ERROR* [oak-lucene-3]
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker Failed to open
> Lucene index at /oak:index/lucene
> org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual
> header=1953790076 vs expected header=1071082519 (resource:
> SlicedIndexInput(SlicedIndexInput(_2ala.fnm in _2ala.cfs) in _2ala.cfs
> slice=8810:9320))
> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:128)
> at
> org.apache.lucene.codecs.lucene46.Lucene46FieldInfosReader.read(Lucene46FieldInfosReader.java:56)
> at
> org.apache.lucene.index.SegmentReader.readFieldInfos(SegmentReader.java:215)
> {noformat}
> Here size of __2ala.cfs_ differed from remote copy and possible other index
> file may have same size but different content. Comparing the modified time of
> the files with those in Oak it can be seen that one of file system was older
> than one in Oak
> {noformat}
> _2alr.cfs={name=_2alr.cfs, size=1152402, sizeStr=1.2 MB, modified=Thu Feb 04
> 17:52:31 GMT 2016, osModified=Feb 4 17:52, osSize=1152402, mismatch=false}
> _2ala.cfe={name=_2ala.cfe, size=224, sizeStr=224 B, modified=Thu Feb 04
> 17:47:28 GMT 2016, osModified=Feb 4 17:17, osSize=224, mismatch=false}
> _2ala.si={name=_2ala.si, size=252, sizeStr=252 B, modified=Thu Feb 04
> 17:47:28 GMT 2016, osModified=Feb 4 17:17, osSize=252, mismatch=false}
> _2ala.cfs={name=_2ala.cfs, size=3714150, sizeStr=3.7 MB, modified=Thu Feb 04
> 17:47:28 GMT 2016, osModified=Feb 4 17:17, osSize=9320, mismatch=true}
> _14u3_29.del={name=_14u3_29.del, size=1244036, sizeStr=1.2 MB, modified=Thu
> Feb 04 16:37:35 GMT 2016, osModified=Feb 4 16:37, osSize=1244036,
> mismatch=false}
> _2akw.si={name=_2akw.si, size=252, sizeStr=252 B, modified=Thu Feb 04
> 16
[jira] [Resolved] (OAK-4114) Cached lucene index gets corrupted in case of unclean shutdown and journal rollback in SegmentNodeStore
[
https://issues.apache.org/jira/browse/OAK-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-4114.
--
Resolution: Fixed
Fix Version/s: 1.5.14
Did a fix with 1769940, 1769939
Now before and Directory is opened for any index a simple sanity check is done
where local index files are compared against remote in terms of file size. In
case of any mismatch whole of the local index dir is purged and the files would
be fetched again later
This check is weak as it compares file size only. But should catch most of such
cases.
Later we can look into some checksum based approach
> Cached lucene index gets corrupted in case of unclean shutdown and journal
> rollback in SegmentNodeStore
> ---
>
> Key: OAK-4114
> URL: https://issues.apache.org/jira/browse/OAK-4114
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Critical
> Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4,
> resilience
> Fix For: 1.6, 1.5.14
>
>
> Currently Oak Lucene support would copy index files to local file system as
> part of CopyOnRead feature. In one of the setup it has been observed that
> index logic was failing with following error
> {noformat}
> 04.02.2016 17:47:52.391 *WARN* [oak-lucene-3]
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier
> [/oak:index/lucene] Found local copy for _2ala.cfs in
> MMapDirectory@/mnt/crx/author/crx-quickstart/repository/index/e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293/1
>
> lockFactory=NativeFSLockFactory@/mnt/crx/author/crx-quickstart/repository/index/e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293/1
> but size of local 9320 differs from remote 3714150. Content would be read
> from remote file only
> 04.02.2016 17:47:52.399 *WARN* [oak-lucene-3]
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier
> [/oak:index/lucene] Found local copy for segments_28je in
> MMapDirectory@/mnt/crx/author/crx-quickstart/repository/index/e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293/1
>
> lockFactory=NativeFSLockFactory@/mnt/crx/author/crx-quickstart/repository/index/e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293/1
> but size of local 1214 differs from remote 1175. Content would be read from
> remote file only
> 04.02.2016 17:47:52.491 *ERROR* [oak-lucene-3]
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker Failed to open
> Lucene index at /oak:index/lucene
> org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual
> header=1953790076 vs expected header=1071082519 (resource:
> SlicedIndexInput(SlicedIndexInput(_2ala.fnm in _2ala.cfs) in _2ala.cfs
> slice=8810:9320))
> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:128)
> at
> org.apache.lucene.codecs.lucene46.Lucene46FieldInfosReader.read(Lucene46FieldInfosReader.java:56)
> at
> org.apache.lucene.index.SegmentReader.readFieldInfos(SegmentReader.java:215)
> {noformat}
> Here size of __2ala.cfs_ differed from remote copy and possible other index
> file may have same size but different content. Comparing the modified time of
> the files with those in Oak it can be seen that one of file system was older
> than one in Oak
> {noformat}
> _2alr.cfs={name=_2alr.cfs, size=1152402, sizeStr=1.2 MB, modified=Thu Feb 04
> 17:52:31 GMT 2016, osModified=Feb 4 17:52, osSize=1152402, mismatch=false}
> _2ala.cfe={name=_2ala.cfe, size=224, sizeStr=224 B, modified=Thu Feb 04
> 17:47:28 GMT 2016, osModified=Feb 4 17:17, osSize=224, mismatch=false}
> _2ala.si={name=_2ala.si, size=252, sizeStr=252 B, modified=Thu Feb 04
> 17:47:28 GMT 2016, osModified=Feb 4 17:17, osSize=252, mismatch=false}
> _2ala.cfs={name=_2ala.cfs, size=3714150, sizeStr=3.7 MB, modified=Thu Feb 04
> 17:47:28 GMT 2016, osModified=Feb 4 17:17, osSize=9320, mismatch=true}
> _14u3_29.del={name=_14u3_29.del, size=1244036, sizeStr=1.2 MB, modified=Thu
> Feb 04 16:37:35 GMT 2016, osModified=Feb 4 16:37, osSize=1244036,
> mismatch=false}
> _2akw.si={name=_2akw.si, size=252, sizeStr=252 B, modified=Thu Feb 04
> 16:37:07 GMT 2016, osModified=Feb 4 16:37, osSize=252, mismatch=false}
> _2akw.cfe={name=_2akw.cfe, size=224, sizeStr=224 B, modified=Thu Feb 04
> 16:37:07 GMT 2016, osModified=Feb 4 16:37, osSize=224, mismatch=false}
> _2akw.cfs={name=_2akw.cfs, size=4952761, sizeStr=5.0 MB, modified=Thu Feb 04
> 16:37:07 GMT 2016, osModified=Feb 4 16:37, osSize=4952761, mismatch=false}
> {noformat}
> And on same setup the system did saw a rollback in segment node store
> {noformat}
> -rw-rw-r--. 1 crx crx 25961984 Feb 4 16:4
[jira] [Updated] (OAK-4114) Cached lucene index gets corrupted in case of unclean shutdown and journal rollback in SegmentNodeStore
[
https://issues.apache.org/jira/browse/OAK-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-4114:
-
Labels: candidate_oak_1_0 candidate_oak_1_2 candidate_oak_1_4 resilience
(was: resilience)
> Cached lucene index gets corrupted in case of unclean shutdown and journal
> rollback in SegmentNodeStore
> ---
>
> Key: OAK-4114
> URL: https://issues.apache.org/jira/browse/OAK-4114
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Critical
> Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4,
> resilience
> Fix For: 1.6, 1.5.14
>
>
> Currently Oak Lucene support would copy index files to local file system as
> part of CopyOnRead feature. In one of the setup it has been observed that
> index logic was failing with following error
> {noformat}
> 04.02.2016 17:47:52.391 *WARN* [oak-lucene-3]
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier
> [/oak:index/lucene] Found local copy for _2ala.cfs in
> MMapDirectory@/mnt/crx/author/crx-quickstart/repository/index/e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293/1
>
> lockFactory=NativeFSLockFactory@/mnt/crx/author/crx-quickstart/repository/index/e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293/1
> but size of local 9320 differs from remote 3714150. Content would be read
> from remote file only
> 04.02.2016 17:47:52.399 *WARN* [oak-lucene-3]
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier
> [/oak:index/lucene] Found local copy for segments_28je in
> MMapDirectory@/mnt/crx/author/crx-quickstart/repository/index/e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293/1
>
> lockFactory=NativeFSLockFactory@/mnt/crx/author/crx-quickstart/repository/index/e5a943cdec3000bd8ce54924fd2070ab5d1d35b9ecf530963a3583d43bf28293/1
> but size of local 1214 differs from remote 1175. Content would be read from
> remote file only
> 04.02.2016 17:47:52.491 *ERROR* [oak-lucene-3]
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker Failed to open
> Lucene index at /oak:index/lucene
> org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual
> header=1953790076 vs expected header=1071082519 (resource:
> SlicedIndexInput(SlicedIndexInput(_2ala.fnm in _2ala.cfs) in _2ala.cfs
> slice=8810:9320))
> at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:128)
> at
> org.apache.lucene.codecs.lucene46.Lucene46FieldInfosReader.read(Lucene46FieldInfosReader.java:56)
> at
> org.apache.lucene.index.SegmentReader.readFieldInfos(SegmentReader.java:215)
> {noformat}
> Here size of __2ala.cfs_ differed from remote copy and possible other index
> file may have same size but different content. Comparing the modified time of
> the files with those in Oak it can be seen that one of file system was older
> than one in Oak
> {noformat}
> _2alr.cfs={name=_2alr.cfs, size=1152402, sizeStr=1.2 MB, modified=Thu Feb 04
> 17:52:31 GMT 2016, osModified=Feb 4 17:52, osSize=1152402, mismatch=false}
> _2ala.cfe={name=_2ala.cfe, size=224, sizeStr=224 B, modified=Thu Feb 04
> 17:47:28 GMT 2016, osModified=Feb 4 17:17, osSize=224, mismatch=false}
> _2ala.si={name=_2ala.si, size=252, sizeStr=252 B, modified=Thu Feb 04
> 17:47:28 GMT 2016, osModified=Feb 4 17:17, osSize=252, mismatch=false}
> _2ala.cfs={name=_2ala.cfs, size=3714150, sizeStr=3.7 MB, modified=Thu Feb 04
> 17:47:28 GMT 2016, osModified=Feb 4 17:17, osSize=9320, mismatch=true}
> _14u3_29.del={name=_14u3_29.del, size=1244036, sizeStr=1.2 MB, modified=Thu
> Feb 04 16:37:35 GMT 2016, osModified=Feb 4 16:37, osSize=1244036,
> mismatch=false}
> _2akw.si={name=_2akw.si, size=252, sizeStr=252 B, modified=Thu Feb 04
> 16:37:07 GMT 2016, osModified=Feb 4 16:37, osSize=252, mismatch=false}
> _2akw.cfe={name=_2akw.cfe, size=224, sizeStr=224 B, modified=Thu Feb 04
> 16:37:07 GMT 2016, osModified=Feb 4 16:37, osSize=224, mismatch=false}
> _2akw.cfs={name=_2akw.cfs, size=4952761, sizeStr=5.0 MB, modified=Thu Feb 04
> 16:37:07 GMT 2016, osModified=Feb 4 16:37, osSize=4952761, mismatch=false}
> {noformat}
> And on same setup the system did saw a rollback in segment node store
> {noformat}
> -rw-rw-r--. 1 crx crx 25961984 Feb 4 16:47 data01357a.tar
> -rw-rw-r--. 1 crx crx 24385536 Feb 4 16:41 data01357a.tar.bak
> -rw-rw-r--. 1 crx crx359936 Feb 4 17:18 data01358a.tar
> -rw-rw-r--. 1 crx crx345088 Feb 4 17:17 data01358a.tar.bak
> -rw-rw-r--. 1 crx crx 70582272 Feb 4 18:35 data01359a.tar
> -rw-rw-r--. 1 crx crx 66359296 Feb 4 18:33 data01359a.tar.bak
> -rw-rw-r--. 1 crx crx282112 Feb 4 18:46 data01360
[jira] [Commented] (OAK-4898) Allow for external changes to have a CommitInfo attached
[
https://issues.apache.org/jira/browse/OAK-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669857#comment-15669857
]
Michael Dürig commented on OAK-4898:
One minor comment re. the patch: the {{info == null || info.isExternal()}}
logic is duplicated in various places. I would rather put this into a static
method of {{CommitInfo}}. This would also help for documenting this peculiarity
in the accompanying Javadoc.
> Allow for external changes to have a CommitInfo attached
>
>
> Key: OAK-4898
> URL: https://issues.apache.org/jira/browse/OAK-4898
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.6
>
> Attachments: OAK-4898-v1.patch
>
>
> Currently the observation logic relies on fact that CommitInfo being null
> means that changes are from other cluster node i.e. external changes.
> We should change this semantic and provide a different way to indicate that
> changes are external. This would allow a NodeStore implementation to still
> pass in a CommitInfo which captures useful information about commit like
> brief summary on what got changed which can be used for pre filtering
> (OAK-4796)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-5115) IndexCopier#cowCopyDoneOnCloseExceptionHandling test hangs intermittently
[
https://issues.apache.org/jira/browse/OAK-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-5115.
--
Resolution: Fixed
Fix Version/s: 1.5.14
Fixed with r1769941
> IndexCopier#cowCopyDoneOnCloseExceptionHandling test hangs intermittently
> -
>
> Key: OAK-5115
> URL: https://issues.apache.org/jira/browse/OAK-5115
> Project: Jackrabbit Oak
> Issue Type: Test
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.6, 1.5.14
>
>
> Seen the IndexCopier#cowCopyDoneOnCloseExceptionHandling test hang
> {noformat}
> "main" #1 prio=5 os_prio=0 tid=0x7f4f9000a000 nid=0x5504 waiting on
> condition [0x7f4f988bf000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xff1f24a0> (a
> java.util.concurrent.FutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopierTest.cowCopyDoneOnCloseExceptionHandling(IndexCopierTest.java:871)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> "pool-91-thread-2" #256 prio=5 os_prio=0 tid=0x7f4f90eb2800 nid=0x5637
> waiting on condition [0x7f4f782d9000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xff1881a0> (a
> java.util.concurrent.CountDownLatch$Sync)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnWriteDirectory.close(CopyOnWriteDirectory.java:221)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopierTest$11.call(IndexCopierTest.java:859)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$W
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2108) Killing a cluster node may stop async index update to to 30 minutes
[ https://issues.apache.org/jira/browse/OAK-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669841#comment-15669841 ] Alex Parvulescu commented on OAK-2108: -- bq. For setups involving SegmentNodeStore we can optimize this a bit by checking is repository is clustered or not. If not then we can reduce the time out. very interesting idea, if the setup is not clustered the entire lease problem goes away. bq. I can check that if NodeStore implements org.apache.jackrabbit.oak.spi.state.Clusterable interface or not +1. it seems safer than parsing an opaque descriptor, and it looks like this is also avail on 1.2 branch so that's a plus. > Killing a cluster node may stop async index update to to 30 minutes > --- > > Key: OAK-2108 > URL: https://issues.apache.org/jira/browse/OAK-2108 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: query >Reporter: Thomas Mueller >Assignee: Chetan Mehrotra > Labels: resilience > Fix For: 1.6 > > > When killing a cluster node that is running the sync index update, then this > async index update will not run for up to 15 minutes, because the lease time > is set to 15 minutes. > I think the lease time should be much smaller, for example 1 minute, or maybe > even 10 seconds. > Also, we might need to better document this issue (in addition to the warning > in the log file). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-5115) IndexCopier#cowCopyDoneOnCloseExceptionHandling test hangs intermittently
Chetan Mehrotra created OAK-5115:
Summary: IndexCopier#cowCopyDoneOnCloseExceptionHandling test
hangs intermittently
Key: OAK-5115
URL: https://issues.apache.org/jira/browse/OAK-5115
Project: Jackrabbit Oak
Issue Type: Test
Components: lucene
Reporter: Chetan Mehrotra
Assignee: Chetan Mehrotra
Priority: Minor
Fix For: 1.6
Seen the IndexCopier#cowCopyDoneOnCloseExceptionHandling test hang
{noformat}
"main" #1 prio=5 os_prio=0 tid=0x7f4f9000a000 nid=0x5504 waiting on
condition [0x7f4f988bf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xff1f24a0> (a
java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopierTest.cowCopyDoneOnCloseExceptionHandling(IndexCopierTest.java:871)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
"pool-91-thread-2" #256 prio=5 os_prio=0 tid=0x7f4f90eb2800 nid=0x5637
waiting on condition [0x7f4f782d9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xff1881a0> (a
java.util.concurrent.CountDownLatch$Sync)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnWriteDirectory.close(CopyOnWriteDirectory.java:221)
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopierTest$11.call(IndexCopierTest.java:859)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$W
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-5113) Avoid dispatching changes for checkpoint related changes
[
https://issues.apache.org/jira/browse/OAK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669827#comment-15669827
]
Alex Parvulescu commented on OAK-5113:
--
looks good!
> Avoid dispatching changes for checkpoint related changes
>
>
> Key: OAK-5113
> URL: https://issues.apache.org/jira/browse/OAK-5113
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-tar
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.6
>
> Attachments: OAK-5113-v1.patch
>
>
> As discussed
> [here|https://issues.apache.org/jira/browse/OAK-4935?focusedCommentId=15666420&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15666420]
> {{SegmentNodeStore}} currently dispatches changes even for checkpoint
> related operations. Given these changes are under super root and not affect
> the tree under normal root we can avoid dispatching such changes at all
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
