[jira] [Created] (OAK-7108) TraverseWithSortStrategy fails to register memory pool listener with heap less than 2GB
Vikas Saurabh created OAK-7108:
--
Summary: TraverseWithSortStrategy fails to register memory pool
listener with heap less than 2GB
Key: OAK-7108
URL: https://issues.apache.org/jira/browse/OAK-7108
Project: Jackrabbit Oak
Issue Type: Bug
Components: run
Reporter: Vikas Saurabh
Assignee: Vikas Saurabh
Priority: Minor
Fix For: 1.8
{{TraverseWithSortStrategy}} sets up its listener on memory pool using a
configurable minMemory. But, as noticed by [~rombert] \[0] with failure of
{{FlatFileStoreTest#basicTest}} with less that 2G heap - the class should
account for available memory too.
\[0]:
https://lists.apache.org/thread.html/881e49cca9df85bf506218d6323466b6ad265bd06815194f4ac08e4d@%3Coak-dev.jackrabbit.apache.org%3E
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-7107) Ability to run AbstractJCRTest derived tests with different fixtures
Marcel Reutegger created OAK-7107: - Summary: Ability to run AbstractJCRTest derived tests with different fixtures Key: OAK-7107 URL: https://issues.apache.org/jira/browse/OAK-7107 Project: Jackrabbit Oak Issue Type: Test Components: jcr Reporter: Marcel Reutegger Assignee: Marcel Reutegger Priority: Minor Fix For: 1.10 Tests in oak-jcr derived from AbstractJCRTest currently always run on segment-tar. It would be good if those tests can also be used with other backends like RDB and MongoDB. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-6460) Index related tooling
[ https://issues.apache.org/jira/browse/OAK-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-6460. -- Resolution: Fixed > Index related tooling > - > > Key: OAK-6460 > URL: https://issues.apache.org/jira/browse/OAK-6460 > Project: Jackrabbit Oak > Issue Type: Epic > Components: indexing, run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8 > > > To enable better management for indexing related operation specially around > reindexing indexes on large repository setup we should implement some tooling > as part of oak-run. This epic is meant to track all work done in this area -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-7106) Index Tooling for Oak 1.10
Chetan Mehrotra created OAK-7106: Summary: Index Tooling for Oak 1.10 Key: OAK-7106 URL: https://issues.apache.org/jira/browse/OAK-7106 Project: Jackrabbit Oak Issue Type: Epic Components: indexing, run Reporter: Chetan Mehrotra Fix For: 1.10 Epic to track tooling work for Oak 1.10 release -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-5317) MongoBlobStore creates _id index unnecessarily
[ https://issues.apache.org/jira/browse/OAK-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger updated OAK-5317: -- Labels: candidate_oak_1_0 candidate_oak_1_2 candidate_oak_1_4 (was: ) > MongoBlobStore creates _id index unnecessarily > -- > > Key: OAK-5317 > URL: https://issues.apache.org/jira/browse/OAK-5317 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, mongomk >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4 > Fix For: 1.5.16, 1.6.0 > > > In MongoDB each collection automatically has an index on the _id field. > MongoBlobStore therefore does not need to create that index. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-7101) Stale documents in RDBDocumentStore cache
[ https://issues.apache.org/jira/browse/OAK-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger updated OAK-7101: -- Affects Version/s: 1.0 1.4.0 1.2.0 Merged the ignored test to branches (they are all affected): - 1.6: http://svn.apache.org/r1818904, http://svn.apache.org/r1818907 - 1.4: http://svn.apache.org/r1818905, http://svn.apache.org/r1818908 - 1.2: http://svn.apache.org/r1818913 - 1.0: http://svn.apache.org/r1818914 > Stale documents in RDBDocumentStore cache > - > > Key: OAK-7101 > URL: https://issues.apache.org/jira/browse/OAK-7101 > Project: Jackrabbit Oak > Issue Type: Bug > Components: rdbmk >Affects Versions: 1.0, 1.4.0, 1.6.0, 1.2.0 >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger > Fix For: 1.8 > > Attachments: OAK-7101.patch > > > Concurrent query and update operations on RDBDocumentStore may result in > stale entries in the document cache. > Potentially related issues are OAK-5387 and OAK-6062. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-7105) Implement a traverse with sort strategy for DocumentStoreIndexer
[ https://issues.apache.org/jira/browse/OAK-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299858#comment-16299858 ] Chetan Mehrotra commented on OAK-7105: -- Switched the default with 1818900 > Implement a traverse with sort strategy for DocumentStoreIndexer > > > Key: OAK-7105 > URL: https://issues.apache.org/jira/browse/OAK-7105 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.15 > > > Currently the DocumentStoreIndexer logic uses a StoreAndSortStrategy in which > it first dumps all nodestates to a json file -> sort them in batches -> merge > the sorted file. In whole indexing the sorting phase is taking decent amount > of time (40 mins out of 3 hr run). > Further this approach suffers with potential OOM while ExternalSort creates > in memory batches where actual size of batch exceeds the estimated size > considerably. So we need to constant tweak the > "oak.indexer.maxSortMemoryInGB" (currently set to 2 GB) > As an improvement we can do following changes > # Implement a traverse with sort strategy - Here instead of first dumping all > nodestate in a single big json we instead add them to an in memory buffer and > then at some stage sort the batch and save it to file > # Use better memory checks - Use the approach as implemented in GCBarrier > i.e. monitor the current memory usage and if it goes below certain threshold > trigger the batch sort -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-7105) Implement a traverse with sort strategy for DocumentStoreIndexer
[ https://issues.apache.org/jira/browse/OAK-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-7105. -- Resolution: Fixed > Implement a traverse with sort strategy for DocumentStoreIndexer > > > Key: OAK-7105 > URL: https://issues.apache.org/jira/browse/OAK-7105 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.15 > > > Currently the DocumentStoreIndexer logic uses a StoreAndSortStrategy in which > it first dumps all nodestates to a json file -> sort them in batches -> merge > the sorted file. In whole indexing the sorting phase is taking decent amount > of time (40 mins out of 3 hr run). > Further this approach suffers with potential OOM while ExternalSort creates > in memory batches where actual size of batch exceeds the estimated size > considerably. So we need to constant tweak the > "oak.indexer.maxSortMemoryInGB" (currently set to 2 GB) > As an improvement we can do following changes > # Implement a traverse with sort strategy - Here instead of first dumping all > nodestate in a single big json we instead add them to an in memory buffer and > then at some stage sort the batch and save it to file > # Use better memory checks - Use the approach as implemented in GCBarrier > i.e. monitor the current memory usage and if it goes below certain threshold > trigger the batch sort -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-7101) Stale documents in RDBDocumentStore cache
[ https://issues.apache.org/jira/browse/OAK-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299831#comment-16299831 ] Marcel Reutegger commented on OAK-7101: --- Removed RDBCacheConsistencyIT and added as RDBCacheConsistency2Test again: http://svn.apache.org/r1818902 & http://svn.apache.org/r1818903 > Stale documents in RDBDocumentStore cache > - > > Key: OAK-7101 > URL: https://issues.apache.org/jira/browse/OAK-7101 > Project: Jackrabbit Oak > Issue Type: Bug > Components: rdbmk >Affects Versions: 1.6.0 >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger > Fix For: 1.8 > > Attachments: OAK-7101.patch > > > Concurrent query and update operations on RDBDocumentStore may result in > stale entries in the document cache. > Potentially related issues are OAK-5387 and OAK-6062. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-7101) Stale documents in RDBDocumentStore cache
[ https://issues.apache.org/jira/browse/OAK-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299825#comment-16299825 ] Marcel Reutegger commented on OAK-7101: --- I noticed there is already a RDBCacheConsistencyIT class in branches 1.4 and earlier. I will therefore rename the ignored test. > Stale documents in RDBDocumentStore cache > - > > Key: OAK-7101 > URL: https://issues.apache.org/jira/browse/OAK-7101 > Project: Jackrabbit Oak > Issue Type: Bug > Components: rdbmk >Affects Versions: 1.6.0 >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger > Fix For: 1.8 > > Attachments: OAK-7101.patch > > > Concurrent query and update operations on RDBDocumentStore may result in > stale entries in the document cache. > Potentially related issues are OAK-5387 and OAK-6062. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-7105) Implement a traverse with sort strategy for DocumentStoreIndexer
[ https://issues.apache.org/jira/browse/OAK-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299800#comment-16299800 ] Chetan Mehrotra commented on OAK-7105: -- Implemented the above flow with 1818896 > Implement a traverse with sort strategy for DocumentStoreIndexer > > > Key: OAK-7105 > URL: https://issues.apache.org/jira/browse/OAK-7105 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.15 > > > Currently the DocumentStoreIndexer logic uses a StoreAndSortStrategy in which > it first dumps all nodestates to a json file -> sort them in batches -> merge > the sorted file. In whole indexing the sorting phase is taking decent amount > of time (40 mins out of 3 hr run). > Further this approach suffers with potential OOM while ExternalSort creates > in memory batches where actual size of batch exceeds the estimated size > considerably. So we need to constant tweak the > "oak.indexer.maxSortMemoryInGB" (currently set to 2 GB) > As an improvement we can do following changes > # Implement a traverse with sort strategy - Here instead of first dumping all > nodestate in a single big json we instead add them to an in memory buffer and > then at some stage sort the batch and save it to file > # Use better memory checks - Use the approach as implemented in GCBarrier > i.e. monitor the current memory usage and if it goes below certain threshold > trigger the batch sort -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-7102) Refactor DocumentIndexer logic to enable different sort approaches
[ https://issues.apache.org/jira/browse/OAK-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Mehrotra resolved OAK-7102. -- Resolution: Fixed Done with various commits in trunk > Refactor DocumentIndexer logic to enable different sort approaches > -- > > Key: OAK-7102 > URL: https://issues.apache.org/jira/browse/OAK-7102 > Project: Jackrabbit Oak > Issue Type: Task > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.7.14, 1.8 > > > DocumentStoreIndexer logic needs to be refactored to support plugging in > different sort approaches -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-7101) Stale documents in RDBDocumentStore cache
[
https://issues.apache.org/jira/browse/OAK-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299727#comment-16299727
]
Marcel Reutegger commented on OAK-7101:
---
Some notes on the patch. The implementation previously put the missing
documents in the cache at the beginning of the method. Instead the updated
documents are now put into the cache after the update in the DB via
{{NodeDocumentCache.putNonConflictingDocs()}}.
> Stale documents in RDBDocumentStore cache
> -
>
> Key: OAK-7101
> URL: https://issues.apache.org/jira/browse/OAK-7101
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: rdbmk
>Affects Versions: 1.6.0
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
> Fix For: 1.8
>
> Attachments: OAK-7101.patch
>
>
> Concurrent query and update operations on RDBDocumentStore may result in
> stale entries in the document cache.
> Potentially related issues are OAK-5387 and OAK-6062.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-7101) Stale documents in RDBDocumentStore cache
[
https://issues.apache.org/jira/browse/OAK-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger updated OAK-7101:
--
Attachment: OAK-7101.patch
Attached patch uses a logic in {{RDBDocumentStore.bulkUpdate()}} method similar
to the one in {{MongoDocumentStore}}.
> Stale documents in RDBDocumentStore cache
> -
>
> Key: OAK-7101
> URL: https://issues.apache.org/jira/browse/OAK-7101
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: rdbmk
>Affects Versions: 1.6.0
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
> Fix For: 1.8
>
> Attachments: OAK-7101.patch
>
>
> Concurrent query and update operations on RDBDocumentStore may result in
> stale entries in the document cache.
> Potentially related issues are OAK-5387 and OAK-6062.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6956) RepositoryUpgrade hardcodes SecurityProvider
[
https://issues.apache.org/jira/browse/OAK-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6956:
---
Fix Version/s: 1.9.0
> RepositoryUpgrade hardcodes SecurityProvider
>
>
> Key: OAK-6956
> URL: https://issues.apache.org/jira/browse/OAK-6956
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: upgrade
>Reporter: angela
>Assignee: angela
>Priority: Critical
> Fix For: 1.9.0
>
>
> [~tomek.rekawek] Looking at non-test usage of the (to be deprecated)
> {{SecurityProviderImpl}} I noticed one usage in the {{RepositoryUpgrade}}
> that looks troublesome to me.
> I would strongly recommend to fix that given the fact that we no longer use
> {{SecurityProviderImpl}} in production ready setup scenarios.
> cc: [~stillalex]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6956) RepositoryUpgrade hardcodes SecurityProvider
[
https://issues.apache.org/jira/browse/OAK-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299713#comment-16299713
]
Tomek Rękawek commented on OAK-6956:
Since I wasn't able to find an alternative to SecurityProviderImpl, I'll
reschedule it and re-assign back to Angela.
[~anchela] - I'll be happy to replace the SecurityProviderImpl if we find a way
to instantiate another SecurityProvider implementation within the oak-upgrade
runtime.
> RepositoryUpgrade hardcodes SecurityProvider
>
>
> Key: OAK-6956
> URL: https://issues.apache.org/jira/browse/OAK-6956
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: upgrade
>Reporter: angela
>Assignee: Tomek Rękawek
>Priority: Critical
>
> [~tomek.rekawek] Looking at non-test usage of the (to be deprecated)
> {{SecurityProviderImpl}} I noticed one usage in the {{RepositoryUpgrade}}
> that looks troublesome to me.
> I would strongly recommend to fix that given the fact that we no longer use
> {{SecurityProviderImpl}} in production ready setup scenarios.
> cc: [~stillalex]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (OAK-6956) RepositoryUpgrade hardcodes SecurityProvider
[
https://issues.apache.org/jira/browse/OAK-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek reassigned OAK-6956:
--
Assignee: angela (was: Tomek Rękawek)
> RepositoryUpgrade hardcodes SecurityProvider
>
>
> Key: OAK-6956
> URL: https://issues.apache.org/jira/browse/OAK-6956
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: upgrade
>Reporter: angela
>Assignee: angela
>Priority: Critical
>
> [~tomek.rekawek] Looking at non-test usage of the (to be deprecated)
> {{SecurityProviderImpl}} I noticed one usage in the {{RepositoryUpgrade}}
> that looks troublesome to me.
> I would strongly recommend to fix that given the fact that we no longer use
> {{SecurityProviderImpl}} in production ready setup scenarios.
> cc: [~stillalex]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6956) RepositoryUpgrade hardcodes SecurityProvider
[
https://issues.apache.org/jira/browse/OAK-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-6956:
---
Fix Version/s: (was: 1.8)
> RepositoryUpgrade hardcodes SecurityProvider
>
>
> Key: OAK-6956
> URL: https://issues.apache.org/jira/browse/OAK-6956
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: upgrade
>Reporter: angela
>Assignee: Tomek Rękawek
>Priority: Critical
>
> [~tomek.rekawek] Looking at non-test usage of the (to be deprecated)
> {{SecurityProviderImpl}} I noticed one usage in the {{RepositoryUpgrade}}
> that looks troublesome to me.
> I would strongly recommend to fix that given the fact that we no longer use
> {{SecurityProviderImpl}} in production ready setup scenarios.
> cc: [~stillalex]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
