[jira] [Commented] (OAK-3923) Async indexing delayed by 30 minutes because stop order is incorrect
[
https://issues.apache.org/jira/browse/OAK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120937#comment-15120937
]
Chetan Mehrotra commented on OAK-3923:
--
On a side note (adding more logs below). It appears that repository is being
closed multiple time. The logs start with service unregistration calls for
various services registered in Oak class. And in the end it appears that close
is called again. Which leads to those warn logs . For this I think we should
have logic in Oak#close to avoid/ignore multiple close calls
{noformat}
17.12.2015 16:53:24.439 *INFO* [FelixStartLevel]
com.adobe.granite.repository.impl.SlingRepositoryManager stop: Repository still
running, forcing shutdown
17.12.2015 16:53:24.453 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [com.adobe.granite.repository.impl.SlingRepositoryManager,522,
[org.apache.sling.jcr.api.SlingRepository, javax.jcr.Repository,
org.apache.jackrabbit.api.management.RepositoryManager,
org.apache.jackrabbit.api.JackrabbitRepository, com.day.crx.CRXRepository]]
ServiceEvent UNREGISTERING
17.12.2015 16:53:24.464 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [com.day.crx.sling.server.impl.jmx.ManagedRepository,551,
[com.day.crx.sling.server.jmx.ManagedRepositoryMBean]] ServiceEvent
UNREGISTERING
17.12.2015 16:53:24.467 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [com.day.crx.core.backup.FileStoreBackupServiceImpl,550,
[com.day.crx.core.backup.FileStoreBackupService]] ServiceEvent UNREGISTERING
17.12.2015 16:53:24.471 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [518, [org.apache.jackrabbit.api.jmx.QueryStatManagerMBean]]
ServiceEvent UNREGISTERING
17.12.2015 16:53:24.473 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [519, [org.apache.jackrabbit.oak.api.jmx.RepositoryStatsMBean]]
ServiceEvent UNREGISTERING
17.12.2015 16:53:24.475 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [520, [java.lang.Runnable]] ServiceEvent UNREGISTERING
17.12.2015 16:53:24.476 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [521, [org.apache.jackrabbit.oak.spi.gc.GCMonitor]] ServiceEvent
UNREGISTERING
17.12.2015 16:53:24.479 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [510, [java.util.concurrent.Executor]] ServiceEvent UNREGISTERING
17.12.2015 16:53:24.481 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [511, [java.lang.Runnable]] ServiceEvent UNREGISTERING
17.12.2015 16:53:24.483 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [512, [org.apache.jackrabbit.oak.api.jmx.IndexStatsMBean]] ServiceEvent
UNREGISTERING
17.12.2015 16:53:24.485 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [513, [java.lang.RunnableAdapterble]] ServiceEvent UNREGISTERING
17.12.2015 16:53:24.486 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [514,
[org.apache.jackrabbit.oak.plugins.index.property.jmx.PropertyIndexAsyncReindexMBean]]
ServiceEvent UNREGISTERING
17.12.2015 16:53:24.488 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [515,
[org.apache.jackrabbit.oak.plugins.index.counter.jmx.NodeCounterMBean]]
ServiceEvent UNREGISTERING
17.12.2015 16:53:24.490 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [516, [org.apache.jackrabbit.oak.api.jmx.QueryEngineSettingsMBean]]
ServiceEvent UNREGISTERING
17.12.2015 16:53:24.491 *INFO* [FelixStartLevel] com.adobe.granite.repository
Service [517, [org.apache.jackrabbit.oak.api.jmx.RepositoryManagementMBean]]
ServiceEvent UNREGISTERING
17.12.2015 16:53:24.494 *WARN* [FelixStartLevel]
org.apache.jackrabbit.oak.osgi.OsgiWhiteboard Error unregistering service:
com.adobe.granite.repository.impl.SlingRepositoryManager$1@20eec2a6 of type
java.util.concurrent.Executor
java.lang.IllegalStateException: Service already unregistered.
at
org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:136)
at
org.apache.jackrabbit.oak.osgi.OsgiWhiteboard$1.unregister(OsgiWhiteboard.java:81)
at
org.apache.jackrabbit.oak.spi.whiteboard.CompositeRegistration.unregister(CompositeRegistration.java:43)
at org.apache.jackrabbit.oak.Oak$6.close(Oak.java:592)
at
com.adobe.granite.repository.impl.CRX3RepositoryImpl.shutdown(CRX3RepositoryImpl.java:124)
at
com.adobe.granite.repository.impl.SlingRepositoryManager.disposeRepository(SlingRepositoryManager.java:308)
at
org.apache.sling.jcr.base.AbstractSlingRepositoryManager.stop(AbstractSlingRepositoryManager.java:361)
at
com.adobe.granite.repository.impl.SlingRepositoryManager.deactivate(SlingRepositoryManager.java:282)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.Delega
[jira] [Commented] (OAK-3936) [oak-run] Option to dump blob references
[ https://issues.apache.org/jira/browse/OAK-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120931#comment-15120931 ] Chetan Mehrotra commented on OAK-3936: -- [~amitj_76] Can you update the readme and provide the usage instructions also? > [oak-run] Option to dump blob references > - > > Key: OAK-3936 > URL: https://issues.apache.org/jira/browse/OAK-3936 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob, run >Reporter: Amit Jain >Assignee: Amit Jain > Labels: resilience, tooling > Fix For: 1.2.11, 1.3.15, 1.0.27 > > > Add an option in oak-run to dump blob references currently used. This should > help in analyzing GC issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-3936) [oak-run] Option to dump blob references
[ https://issues.apache.org/jira/browse/OAK-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Jain resolved OAK-3936. Resolution: Fixed Committed changes with: * 1.2 - http://svn.apache.org/viewvc?rev=1727255&view=rev * 1.0 - http://svn.apache.org/viewvc?rev=1727257&view=rev * trunk - http://svn.apache.org/viewvc?rev=1727254&view=rev > [oak-run] Option to dump blob references > - > > Key: OAK-3936 > URL: https://issues.apache.org/jira/browse/OAK-3936 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob, run >Reporter: Amit Jain >Assignee: Amit Jain > Labels: resilience, tooling > Fix For: 1.2.11, 1.3.15, 1.0.27 > > > Add an option in oak-run to dump blob references currently used. This should > help in analyzing GC issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3936) [oak-run] Option to dump blob references
[ https://issues.apache.org/jira/browse/OAK-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Jain updated OAK-3936: --- Fix Version/s: 1.0.27 1.2.11 > [oak-run] Option to dump blob references > - > > Key: OAK-3936 > URL: https://issues.apache.org/jira/browse/OAK-3936 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob, run >Reporter: Amit Jain >Assignee: Amit Jain > Labels: resilience, tooling > Fix For: 1.2.11, 1.3.15, 1.0.27 > > > Add an option in oak-run to dump blob references currently used. This should > help in analyzing GC issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3936) [oak-run] Option to dump blob references
Amit Jain created OAK-3936: -- Summary: [oak-run] Option to dump blob references Key: OAK-3936 URL: https://issues.apache.org/jira/browse/OAK-3936 Project: Jackrabbit Oak Issue Type: Improvement Components: blob, run Reporter: Amit Jain Assignee: Amit Jain Fix For: 1.3.15 Add an option in oak-run to dump blob references currently used. This should help in analyzing GC issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3935) SharedDataStore - Allow unique repository ID to be specified by config
Amit Jain created OAK-3935: -- Summary: SharedDataStore - Allow unique repository ID to be specified by config Key: OAK-3935 URL: https://issues.apache.org/jira/browse/OAK-3935 Project: Jackrabbit Oak Issue Type: Improvement Components: blob Reporter: Amit Jain Assignee: Amit Jain Fix For: 1.2.11, 1.3.15 For GC in a shared DataStore, a unique repository Id is currently saved in a hidden path in the node store on startup and registered in the DataStore. This will cause problems where the publish environments are cloned and share a datastore. There should be an option to specify a unique id in the config when setting up the node store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3879) Lucene index / compatVersion 2: search for 'abc!' does not work
[
https://issues.apache.org/jira/browse/OAK-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-3879:
-
Attachment: OAK-3879-v1.patch
[proposed patch|^OAK-3879-v1.patch] for the same.
(Copying some comment from OAK-3769)
Lucene Query has [following special
chars|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/queryparser/src/java/org/apache/lucene/queryparser/classic/QueryParserBase.java#L983-L995]
{code}
char c = s.charAt(i);
// These characters are part of the query syntax and must be escaped
if (c == '\\' || c == '+' || c == '-' || c == '!' || c == '(' || c == ')'
|| c == ':'
|| c == '^' || c == '[' || c == ']' || c == '\"' || c == '{' || c ==
'}' || c == '~'
|| c == '*' || c == '?' || c == '|' || c == '&' || c == '/') {
sb.append('\\');
}
{code}
Refer to Solr doc for details on these operators
https://cwiki.apache.org/confluence/display/solr/The+Standard+Query+Parser
For now I have kept the chars which need to be escaped
{code}
private static final char[] LUCENE_QUERY_OPERATORS = {':' , '/', '!', '&', '|'};
{code}
Note JR2 only escaped ':'. So we are escaping few more (which might have also
added after 3.x released used by JR2). From Lucene list I think following are
still useful
*A - Supported operators* Should NOT be included in escape list
# \? - {{te?t}}
# \* - {{te*t}}
# \~ - {{roam~1}} - Fuzzy Search and Proximity Search
# \^ - Boost
# \- - Excluding a term from
# \( \) - This *might* be useful to group terms
*B - Unsupported operators* Should be included in escape list
Now that leaves following
# \+
# \[ \] - Used for range queries. {{mod_date:[20020101 TO 20030101]}}
# \{ \} - {{\{Aida TO Carmen\}}}
[~tmueller] [~teofili] [~catholicon] Can you have a look now and we take a
final call on what should be escaped and what should be not. Also review the
patch. I would like to commit it by tomorrow
> Lucene index / compatVersion 2: search for 'abc!' does not work
> ---
>
> Key: OAK-3879
> URL: https://issues.apache.org/jira/browse/OAK-3879
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Thomas Mueller
>Assignee: Chetan Mehrotra
> Fix For: 1.4
>
> Attachments: OAK-3879-v1.patch
>
>
> When using a Lucene fulltext index with compatVersion 2, then the following
> query does not return any results. When using compatVersion 1, the correct
> result is returned.
> {noformat}
> SELECT * FROM [nt:unstructured] AS c
> WHERE CONTAINS(c.[jcr:description], 'abc!')
> AND ISDESCENDANTNODE(c, '/content')
> {noformat}
> With compatVersion 1 and 2, searching for just 'abc' works. Also, searching
> with '=' instead of 'contains' works.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3907) Sync the files to directory upon copy from remote
[
https://issues.apache.org/jira/browse/OAK-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-3907:
-
Labels: candidate_oak_1_0 candidate_oak_1_2 (was: )
> Sync the files to directory upon copy from remote
> -
>
> Key: OAK-3907
> URL: https://issues.apache.org/jira/browse/OAK-3907
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.3.15
>
>
> {{IndexCopier}} does not {{sync}} the file to file system once files are
> copied from remote. This might cause problem on setup where local index file
> directory is on some SAN.
> For safety we should also sync the files upon copy from remote
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3907) Sync the files to directory upon copy from remote
[
https://issues.apache.org/jira/browse/OAK-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-3907.
--
Resolution: Fixed
Fix Version/s: (was: 1.0.27)
(was: 1.2.11)
(was: 1.4)
1.3.15
Done in trunk with 1727233
> Sync the files to directory upon copy from remote
> -
>
> Key: OAK-3907
> URL: https://issues.apache.org/jira/browse/OAK-3907
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.3.15
>
>
> {{IndexCopier}} does not {{sync}} the file to file system once files are
> copied from remote. This might cause problem on setup where local index file
> directory is on some SAN.
> For safety we should also sync the files upon copy from remote
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OAK-3923) Async indexing delayed by 30 minutes because stop order is incorrect
[
https://issues.apache.org/jira/browse/OAK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra reassigned OAK-3923:
Assignee: Chetan Mehrotra
> Async indexing delayed by 30 minutes because stop order is incorrect
>
>
> Key: OAK-3923
> URL: https://issues.apache.org/jira/browse/OAK-3923
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
>Reporter: Thomas Mueller
>Assignee: Chetan Mehrotra
> Fix For: 1.4
>
>
> The stop order of Oak components is incorrect, and this can lead to an async
> indexing delay of 30 minutes, because the indexing lease is not removed. The
> problem is that the node store is stopped before the async index is stopped,
> so that async indexing can still be in progress, and then when async indexing
> is done, the lease can not be removed because the node store is not available.
> From the log file:
> {noformat}
> error.log:
> 21.01.2016 11:53:56.898 *INFO* [FelixStartLevel]
> org.apache.jackrabbit.oak-tarmk-standby BundleEvent STOPPED
> 21.01.2016 11:53:56.900 *INFO* [FelixStartLevel]
> org.apache.jackrabbit.oak-solr-osgi Service
> [org.apache.jackrabbit.oak.plugins.index.solr.osgi.SolrIndexEditorProviderService,571,
> [org.apache.jackrabbit.oak.plugins.index.IndexEditorProvider]] ServiceEvent
> UNREGISTERING
> ...
> 21.01.2016 11:53:56.930 *INFO* [FelixStartLevel]
> org.apache.jackrabbit.oak-lucene BundleEvent STOPPING
> 21.01.2016 11:53:56.930 *INFO* [FelixStartLevel]
> org.apache.jackrabbit.oak-lucene BundleEvent STOPPED
> 21.01.2016 11:53:56.931 *INFO* [FelixStartLevel]
> org.apache.jackrabbit.oak-core Service
> [org.apache.jackrabbit.oak.plugins.index.property.PropertyIndexProvider,405,
> [org.apache.jackrabbit.oak.spi.query.QueryIndexProvider]] ServiceEvent
> UNREGISTERING
> ...
> 21.01.2016 11:53:56.936 *INFO* [FelixStartLevel]
> com.adobe.granite.repository.impl.SlingRepositoryManager stop: Repository
> still running, forcing shutdown
> ...
> 21.01.2016 11:53:56.960 *WARN* [FelixStartLevel]
> org.apache.jackrabbit.oak.osgi.OsgiWhiteboard Error unregistering service:
> com.adobe.granite.repository.impl.SlingRepositoryManager$1@7c052458 of type
> java.util.concurrent.Executor
> java.lang.IllegalStateException: Service already unregistered.
> at
> org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:136)
> at
> org.apache.jackrabbit.oak.osgi.OsgiWhiteboard$1.unregister(OsgiWhiteboard.java:81)
> at
> org.apache.jackrabbit.oak.spi.whiteboard.CompositeRegistration.unregister(CompositeRegistration.java:43)
> at org.apache.jackrabbit.oak.Oak$6.close(Oak.java:592)
> ...
> 21.01.2016 11:56:50.985 *INFO* [FelixStartLevel]
> org.apache.jackrabbit.oak-core Service [763,
> [org.apache.jackrabbit.oak.plugins.segment.SegmentStoreProvider]]
> ServiceEvent UNREGISTERING
>
> debug.log:
> 21.01.2016 11:56:51.964 *WARN* [sling-default-4]
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate [async] The index
> update failed
> java.lang.IllegalStateException: service must be activated when used
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:150)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.getNodeStore(SegmentNodeStoreService.java:233)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.getNodeStore(SegmentNodeStoreService.java:92)
> at
> org.apache.jackrabbit.oak.spi.state.ProxyNodeStore.getRoot(ProxyNodeStore.java:36)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate$AsyncUpdateCallback.close(AsyncIndexUpdate.java:266)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:451)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:351)
>
> error.log:
> 21.01.2016 11:56:51.965 *ERROR* [sling-default-4]
> org.apache.sling.commons.scheduler.impl.QuartzScheduler Exception during job
> execution of
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate@1706b18c : service
> must be activated when used
> java.lang.IllegalStateException: service must be activated when used
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:150)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.getNodeStore(SegmentNodeStoreService.java:233)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.getNodeStore(SegmentNodeStoreService.java:92)
> at
> org.apache.jackrabbit.oak.spi.state.ProxyNodeStore.release(ProxyNodeStore.java:90)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:372)
> at
> org.apache
[jira] [Commented] (OAK-3916) Cor/Cow should copy in suggest dir as well if available
[
https://issues.apache.org/jira/browse/OAK-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120660#comment-15120660
]
Vikas Saurabh commented on OAK-3916:
[~chetanm], here's a snip of stats from my dev setup with an ootb aem instance
having suggestions enabled for {{jcr:title}}, {{jcr:description}} for
{{cq:Page}} (I think most of the size in cq:Page would be due to extracted text
from resource node). I had setup suggestions for regex prop ({{prop0}}) for
aggregate lucene (that, I think, would be the worst case scenario in terms of
index-size to suggestions-size ratio)
||indexSize||indexSizeStr||maxDoc||numDeletedDocs||numDocs||path||suggesterSize||suggesterSizeStr||
|7276420|7.3 MB|1871|0|1871|/oak:index/cqPageLucene|64841|64.8 kB|
|66179527|66.2 MB|202136|0|202136|/oak:index/lucene|6438853|6.4 MB|
May be, we can use this as a hint to prioritize this issue.
> Cor/Cow should copy in suggest dir as well if available
> ---
>
> Key: OAK-3916
> URL: https://issues.apache.org/jira/browse/OAK-3916
> Project: Jackrabbit Oak
> Issue Type: Improvement
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
> Fix For: 1.4
>
>
> Suggest dictionary is now stored as another OakDir under lucene indices. That
> means that looking up for suggestions peeks into a lucene index which can get
> expensive if it requires going to NFS/Blobstore. It'd be nice to utilize
> Cor/Cow for this as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3066) Persistent cache for previous documents
[
https://issues.apache.org/jira/browse/OAK-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh updated OAK-3066:
---
Attachment: OAK-3066.take1.patch
Attaching a working implementation (no tests cases yet) -
[^OAK-3066.take1.patch]. Things it does:
* Sets up persistent cache for prev docs
* Assigns a nominal 4% mem cache for prev docs
* NodeDocumentCache updated to divide its load between already existing
nodesCache and newly added persistent cache
** quite a few method renames
** decision of putting into prev cache or not is done using id of doc being
added/fetched (added Utils#isLeafPreviousDocId)
[~mreutegg], can you please review these changes. Please see a few *TODO* I've
added in {{NodeDocumentCache}}. Should we worry about immutability of leaf prev
docs at this layer (may be WARN traces are enough??)?
I've verified that configuring pers. cache for JournalIT#cacheInvalidationTest
add 1 split doc into pers cache. I dumped out the value of that and read in
NodeDocument.fromString() to get the same doc back.
I'd like to get this into 1.3.15 but with my other issues pending, I was
wondering if it's ok to open a separate issue for 1.3.16 for adding tests.
> Persistent cache for previous documents
> ---
>
> Key: OAK-3066
> URL: https://issues.apache.org/jira/browse/OAK-3066
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk
>Reporter: Marcel Reutegger
>Assignee: Vikas Saurabh
> Labels: performance
> Fix For: 1.4
>
> Attachments: OAK-3066.take1.patch, OAK-3066.wip.patch
>
>
> Previous (aka split) documents contain old revisions and are immutable
> documents. Those documents should go into the persistent cache to reduce
> calls to the underlying DocumentStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3917) SuggestionHelper creating unnecessary temporary directories
[
https://issues.apache.org/jira/browse/OAK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh resolved OAK-3917.
Resolution: Fixed
Fix Version/s: (was: 1.4)
1.3.15
Talked to Chetan and modified patch to create/cleanup temp dir in
{{SuggestHelper#updateSuggester}}. Fixed in trunk at
[r1727149|http://svn.apache.org/r1727149].
> SuggestionHelper creating unnecessary temporary directories
> ---
>
> Key: OAK-3917
> URL: https://issues.apache.org/jira/browse/OAK-3917
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
> Fix For: 1.3.15
>
> Attachments: OAK-3917.patch
>
>
> SuggestHelper create a temporary directory [1] mostly likely as a workaround
> to allow use of {{OakDirectory}}. With time this can grow unbounded (as dir
> is not cleaned) and following exception is seen.
> {noformat}
> java.lang.IllegalStateException: Failed to create directory within 1
> attempts (tried 145369739-0 to 145369739-)
> at com.google.common.io.Files.createTempDir(Files.java:608)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.util.SuggestHelper.getLookup(SuggestHelper.java:107)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.(IndexNode.java:106)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:69)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:162)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:137)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:222)
> {noformat}
> [1]
> https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.3.14/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/util/SuggestHelper.java#L107
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OAK-3901) SecurityProviderRegistration must respect service ranking of aggregated configurations
[
https://issues.apache.org/jira/browse/OAK-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angela reassigned OAK-3901:
---
Assignee: angela
> SecurityProviderRegistration must respect service ranking of aggregated
> configurations
> --
>
> Key: OAK-3901
> URL: https://issues.apache.org/jira/browse/OAK-3901
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
>Reporter: angela
>Assignee: angela
>Priority: Critical
> Fix For: 1.4
>
>
> while working on aggregating multiple authorization configurations i spotted
> that in contrast to {{SecurityProviderImpl}} the
> {{SecurityProviderRegistration}} doesn't respect the service ranking (or the
> corresponding config option) to define the order of the individual
> configurations in the aggregation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OAK-3902) SecurityProviderRegistration doesn't fill the composite context
[
https://issues.apache.org/jira/browse/OAK-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angela reassigned OAK-3902:
---
Assignee: angela
> SecurityProviderRegistration doesn't fill the composite context
> ---
>
> Key: OAK-3902
> URL: https://issues.apache.org/jira/browse/OAK-3902
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
>Reporter: angela
>Assignee: angela
>Priority: Critical
> Fix For: 1.4
>
>
> when aggregating security configurations of the same type in
> {{SecurityProviderRegistration}} the context hold by
> {{CompositeConfiguration}} remains empty (in contrast to the corresponding
> call in {{SecurityProviderImpl}}). This particularly affects authorization
> where it is crucial to identify that a given item is defined by the
> authorization model and thus is a different type of node/property.
> To fix that we might consider changing the way
> {{SecurityProviderRegistration}} populates the aggregates or (if that is not
> possible) fix the {{CompositeConfiguration}}. In general the issue was
> introduced when removing the {{final}} flag from the various
> {{Composite*Configuration}}s together with a optimization in
> {{CompositeConfiguration}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3927) oak-run primary/standby should check segment version
[ https://issues.apache.org/jira/browse/OAK-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119777#comment-15119777 ] Francesco Mari commented on OAK-3927: - bq. For primary/standby I'm not so sure whether the flag makes sense. Me neither. Maybe [~alex.parvulescu] can shed some light? > oak-run primary/standby should check segment version > > > Key: OAK-3927 > URL: https://issues.apache.org/jira/browse/OAK-3927 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: run >Reporter: Michael Dürig >Assignee: Francesco Mari >Priority: Critical > Labels: resilience, tooling > Fix For: 1.4 > > Attachments: OAK-3854-01.patch > > > The primary and standby run modes should exit with an error if run on a store > with non matching segment version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3927) oak-run primary/standby should check segment version
[ https://issues.apache.org/jira/browse/OAK-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119760#comment-15119760 ] Michael Dürig commented on OAK-3927: bq. migration path the compact run mode would take care of this. For primary/standby I'm not so sure whether the flag makes sense. bq. adapting the fixes for the issues you mentioned. That'd be great! > oak-run primary/standby should check segment version > > > Key: OAK-3927 > URL: https://issues.apache.org/jira/browse/OAK-3927 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: run >Reporter: Michael Dürig >Assignee: Francesco Mari >Priority: Critical > Labels: resilience, tooling > Fix For: 1.4 > > Attachments: OAK-3854-01.patch > > > The primary and standby run modes should exit with an error if run on a store > with non matching segment version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3927) oak-run primary/standby should check segment version
[
https://issues.apache.org/jira/browse/OAK-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119748#comment-15119748
]
Francesco Mari commented on OAK-3927:
-
If we omit the {{--force}} flag, would we also need a clear migration path from
one version of the {{FileStore}} to another? I can take care of committing
{{checkFileStoreVersionOrFail}} and adapting the fixes for the issues you
mentioned.
> oak-run primary/standby should check segment version
>
>
> Key: OAK-3927
> URL: https://issues.apache.org/jira/browse/OAK-3927
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: run
>Reporter: Michael Dürig
>Assignee: Francesco Mari
>Priority: Critical
> Labels: resilience, tooling
> Fix For: 1.4
>
> Attachments: OAK-3854-01.patch
>
>
> The primary and standby run modes should exit with an error if run on a store
> with non matching segment version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3927) oak-run primary/standby should check segment version
[
https://issues.apache.org/jira/browse/OAK-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119723#comment-15119723
]
Michael Dürig commented on OAK-3927:
LGTM. Do we need the {{--force}} flag here? Could this make sense or are we
just providing additional rope for users to hang themselves?
I like {{checkFileStoreVersionOrFail}}. We should also incorporate it into my
fixes of OAK-3855, OAK-3925 and OAK-3926.
> oak-run primary/standby should check segment version
>
>
> Key: OAK-3927
> URL: https://issues.apache.org/jira/browse/OAK-3927
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: run
>Reporter: Michael Dürig
>Assignee: Francesco Mari
>Priority: Critical
> Labels: resilience, tooling
> Fix For: 1.4
>
> Attachments: OAK-3854-01.patch
>
>
> The primary and standby run modes should exit with an error if run on a store
> with non matching segment version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3672) SegmentDiscoveryLiteService does not persist clusterView.id
[
https://issues.apache.org/jira/browse/OAK-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119708#comment-15119708
]
Stefan Egli commented on OAK-3672:
--
SLING-5458 is in the Sling release process - expected to be publicly released
not-before JAN-30 5pm UTC
> SegmentDiscoveryLiteService does not persist clusterView.id
> ---
>
> Key: OAK-3672
> URL: https://issues.apache.org/jira/browse/OAK-3672
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
>Affects Versions: 1.3.10
>Reporter: Stefan Egli
>Assignee: Stefan Egli
> Fix For: 1.4
>
>
> The discovery-lite-descriptor introduced with OAK-2844 has a property {{id}}
> that uniquely and persistently identifies a cluster. However, the
> {{SegmentDiscoveryLiteService}} creates this id upon each instance restart
> (by setting {{runtimeClusterId}}).
> This should be fixed to have this {{id}} persisted somehow.
> Note that the consequences of this id changing upon each restart is that the
> corresponding presumed-to-be-persistent {{ClusterView.id}} of the
> discovery.oak will also change upon restart. Which is a violation of the
> discovery API and upper level applications might thus misbehave in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3934) Log ids of segments being released for gc because of their age.
[
https://issues.apache.org/jira/browse/OAK-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-3934.
Resolution: Fixed
Fix Version/s: (was: 1.6)
1.3.15
Fixed at http://svn.apache.org/viewvc?rev=1727118&view=rev
> Log ids of segments being released for gc because of their age.
>
>
> Key: OAK-3934
> URL: https://issues.apache.org/jira/browse/OAK-3934
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: cleanup, gc
> Fix For: 1.3.15
>
>
> When {{CompactionStrategy.CleanupType#CLEAN_OLD}} releases a segment for gc
> because of its age it should log a message. This helps to determine the root
> cause of a {{SNFE}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-3934) Log ids of segments being released for gc because of their age.
Michael Dürig created OAK-3934:
--
Summary: Log ids of segments being released for gc because of
their age.
Key: OAK-3934
URL: https://issues.apache.org/jira/browse/OAK-3934
Project: Jackrabbit Oak
Issue Type: Technical task
Components: segmentmk
Reporter: Michael Dürig
Assignee: Michael Dürig
When {{CompactionStrategy.CleanupType#CLEAN_OLD}} releases a segment for gc
because of its age it should log a message. This helps to determine the root
cause of a {{SNFE}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3842) Adjust package export declarations
[
https://issues.apache.org/jira/browse/OAK-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-3842.
Resolution: Fixed
Fix Version/s: (was: 1.4)
Applied the updated patch at http://svn.apache.org/viewvc?rev=1727106&view=rev.
For those packages that have been exported at a non zero version before I had
to add an exclusion filter to the baseline goal. For now I did so in
{{oak-parent}} for all modules. We can change this to per module exclusions if
there is a consensus that this would be better.
> Adjust package export declarations
> ---
>
> Key: OAK-3842
> URL: https://issues.apache.org/jira/browse/OAK-3842
> Project: Jackrabbit Oak
> Issue Type: Task
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Blocker
> Labels: api, modularization, technical_debt
> Fix For: 1.3.15
>
> Attachments: OAK-3842-2.patch, OAK-3842.patch
>
>
> We need to adjust the package export declarations such that they become
> manageable with our branch / release model.
> See http://markmail.org/thread/5g3viq5pwtdryapr for discussion.
> I propose to remove package export declarations from all packages that we
> don't consider public API / SPI beyond Oak itself. This would allow us to
> evolve Oak internal stuff (e.g. things used across Oak modules) freely
> without having to worry about merges to branches messing up semantic
> versioning. OTOH it would force us to keep externally facing public API / SPI
> reasonably stable also across the branches. Furthermore such an approach
> would send the right signal to Oak API / SPI consumers regarding the
> stability assumptions they can make.
> An external API / SPI having a (transitive) dependency on internals might be
> troublesome. In doubt I would remove the export version here until we can
> make reasonable guarantees (either through decoupling the code or stabilising
> the dependencies).
> I would start digging through the export version and prepare an initial
> proposal for further discussion.
> /cc [~frm], [~chetanm], [~mmarth]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3871) ability to override ClusterNodeInfo#DEFAULT_LEASE_DURATION_MILLIS
[ https://issues.apache.org/jira/browse/OAK-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke resolved OAK-3871. - Resolution: Fixed trunk: http://svn.apache.org/r1727105 > ability to override ClusterNodeInfo#DEFAULT_LEASE_DURATION_MILLIS > - > > Key: OAK-3871 > URL: https://issues.apache.org/jira/browse/OAK-3871 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Julian Reschke >Assignee: Julian Reschke > Fix For: 1.3.15 > > Attachments: OAK-3871.diff > > > This constant currently is set to 120s. > In edge cases, it might be good to override the default (when essentially the > only other "fix" would be to turn off the lease check). > Proposal: introduce a system property allowing to tune the lease time (but > restrict allowable values to a "sane" range, such as up to 5min). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3871) ability to override ClusterNodeInfo#DEFAULT_LEASE_DURATION_MILLIS
[ https://issues.apache.org/jira/browse/OAK-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-3871: Fix Version/s: (was: 1.4) 1.3.15 > ability to override ClusterNodeInfo#DEFAULT_LEASE_DURATION_MILLIS > - > > Key: OAK-3871 > URL: https://issues.apache.org/jira/browse/OAK-3871 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Julian Reschke >Assignee: Julian Reschke > Fix For: 1.3.15 > > Attachments: OAK-3871.diff > > > This constant currently is set to 120s. > In edge cases, it might be good to override the default (when essentially the > only other "fix" would be to turn off the lease check). > Proposal: introduce a system property allowing to tune the lease time (but > restrict allowable values to a "sane" range, such as up to 5min). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-3915) Include suggest directory size into lucene stats jmx
[
https://issues.apache.org/jira/browse/OAK-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh resolved OAK-3915.
Resolution: Fixed
Fix Version/s: (was: 1.4)
1.3.15
Committed [r1727095|http://svn.apache.org/r1727095] on trunk. It adds 2 columns
in lucene index stats to show suggester size -- bytes and human readable string
(-1 and 0 respectively in case suggester is not enabled for the index)
> Include suggest directory size into lucene stats jmx
>
>
> Key: OAK-3915
> URL: https://issues.apache.org/jira/browse/OAK-3915
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
> Fix For: 1.3.15
>
> Attachments: OAK-3915.patch
>
>
> We now store suggest dictionary under {{:suggest-data}} under index which
> supports suggestions. It'd nice to have some size statistics (similar to
> usual index) for this sub-dir
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2847) Dependency cleanup
[ https://issues.apache.org/jira/browse/OAK-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Saurabh updated OAK-2847: --- Fix Version/s: (was: 1.4) 1.6 > Dependency cleanup > --- > > Key: OAK-2847 > URL: https://issues.apache.org/jira/browse/OAK-2847 > Project: Jackrabbit Oak > Issue Type: Epic >Reporter: Michael Dürig >Assignee: Vikas Saurabh > Labels: technical_debt > Fix For: 1.6 > > > Early in the next release cycle we should go through the list of Oak's > dependencies and decide whether we have candidates we want to upgrade and > remove orphaned dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-2675) Include change type information in perf logs for diff logic
[
https://issues.apache.org/jira/browse/OAK-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh resolved OAK-2675.
Resolution: Fixed
Fix Version/s: (was: 1.4)
1.3.15
Resolving this issue. If we need any more data/modification, we can open up a
new isuse.
> Include change type information in perf logs for diff logic
> ---
>
> Key: OAK-2675
> URL: https://issues.apache.org/jira/browse/OAK-2675
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Vikas Saurabh
>Priority: Minor
> Labels: observation, performance, resilience, tooling
> Fix For: 1.3.15
>
>
> Currently the diff perf logs in {{NodeObserver}} does not indicate what type
> of change was processed i.e. was the change an internal one or an external
> one.
> Having this information would allow us to determine how the cache is being
> used. For e.g. if we see slower number even for local changes then that would
> indicate that there is some issue with the diff cache and its not be being
> utilized effectively
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2847) Dependency cleanup
[ https://issues.apache.org/jira/browse/OAK-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119629#comment-15119629 ] Vikas Saurabh commented on OAK-2847: Moving to 1.6. > Dependency cleanup > --- > > Key: OAK-2847 > URL: https://issues.apache.org/jira/browse/OAK-2847 > Project: Jackrabbit Oak > Issue Type: Epic >Reporter: Michael Dürig >Assignee: Vikas Saurabh > Labels: technical_debt > Fix For: 1.6 > > > Early in the next release cycle we should go through the list of Oak's > dependencies and decide whether we have candidates we want to upgrade and > remove orphaned dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3694) As a user, I want to know when a node that I've created has been indexed.
[
https://issues.apache.org/jira/browse/OAK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119403#comment-15119403
]
Andrei Dulvac commented on OAK-3694:
In my opinion, this logic should be delegated to oak and provided to users, as
this logic could change at any time. Also, I think it's ok to rely on querying
to see if a node was indexed, but does not seem sensible to me to ask the
client to do that.
> As a user, I want to know when a node that I've created has been indexed.
> -
>
> Key: OAK-3694
> URL: https://issues.apache.org/jira/browse/OAK-3694
> Project: Jackrabbit Oak
> Issue Type: Story
>Reporter: Valentin Olteanu
>Assignee: Chetan Mehrotra
> Fix For: 1.4
>
>
> As a user, I want to know when a node that I've created has been indexed so
> that I can start using it.
> Generalization: As a user, I want to know when everything created before
> timestamp T0 has been indexed.
> Ideal solution: MBean operation {{bool isIndexedBefore(long timestamp)}} that
> returns {{true}} if everything created before {{timestamp}} has been indexed
> (by all indexers).
> Current options:
> # check IndexStatsMBean for Start and Done to determine if a cycle has
> started after adding the node and has finished. Issue: there can be multiple
> async indexers. OAK-3606 proposes a solution.
> # add a node and search for it until it is retrieved. Issue: must add
> different nodes for different indexers and wait for all.
> These options are not precise (give false positives) and are not resilient to
> changes in indexing strategy (e.g. adding one more indexer breaks the
> checks).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-3694) As a user, I want to know when a node that I've created has been indexed.
[
https://issues.apache.org/jira/browse/OAK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119403#comment-15119403
]
Andrei Dulvac edited comment on OAK-3694 at 1/27/16 3:10 PM:
-
In my opinion, this logic should be delegated to oak and provided to users, as
this logic could change at any time. Also, I think it's ok to rely on querying
to see if a node was indexed in the internal implementation, but does not seem
sensible to me to ask the client to do that.
was (Author: andrei.dulvac):
In my opinion, this logic should be delegated to oak and provided to users, as
this logic could change at any time. Also, I think it's ok to rely on querying
to see if a node was indexed, but does not seem sensible to me to ask the
client to do that.
> As a user, I want to know when a node that I've created has been indexed.
> -
>
> Key: OAK-3694
> URL: https://issues.apache.org/jira/browse/OAK-3694
> Project: Jackrabbit Oak
> Issue Type: Story
>Reporter: Valentin Olteanu
>Assignee: Chetan Mehrotra
> Fix For: 1.4
>
>
> As a user, I want to know when a node that I've created has been indexed so
> that I can start using it.
> Generalization: As a user, I want to know when everything created before
> timestamp T0 has been indexed.
> Ideal solution: MBean operation {{bool isIndexedBefore(long timestamp)}} that
> returns {{true}} if everything created before {{timestamp}} has been indexed
> (by all indexers).
> Current options:
> # check IndexStatsMBean for Start and Done to determine if a cycle has
> started after adding the node and has finished. Issue: there can be multiple
> async indexers. OAK-3606 proposes a solution.
> # add a node and search for it until it is retrieved. Issue: must add
> different nodes for different indexers and wait for all.
> These options are not precise (give false positives) and are not resilient to
> changes in indexing strategy (e.g. adding one more indexer breaks the
> checks).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3071) Add a compound index for _modified + _id
[
https://issues.apache.org/jira/browse/OAK-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3071:
---
Priority: Blocker (was: Major)
> Add a compound index for _modified + _id
>
>
> Key: OAK-3071
> URL: https://issues.apache.org/jira/browse/OAK-3071
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: mongomk
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Blocker
> Labels: performance, resilience
> Fix For: 1.4, 1.3.15
>
>
> As explained in OAK-1966 diff logic makes a call like
> bq. db.nodes.find({ _id: { $gt: "3:/content/foo/01/", $lt:
> "3:/content/foo010" }, _modified: { $gte: 1405085300 } }).sort({_id:1})
> For better and deterministic query performance we would need to create a
> compound index like \{_modified:1, _id:1\}. This index would ensure that
> Mongo does not have to perform object scan while evaluating such a query.
> Care must be taken that index is only created by default for fresh setup. For
> existing setup we should expose a JMX operation which can be invoked by
> system admin to create the required index as per maintenance window
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OAK-3071) Add a compound index for _modified + _id
[
https://issues.apache.org/jira/browse/OAK-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Mueller reassigned OAK-3071:
---
Assignee: Thomas Mueller (was: Chetan Mehrotra)
> Add a compound index for _modified + _id
>
>
> Key: OAK-3071
> URL: https://issues.apache.org/jira/browse/OAK-3071
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: mongomk
>Reporter: Chetan Mehrotra
>Assignee: Thomas Mueller
>Priority: Blocker
> Labels: performance, resilience
> Fix For: 1.4, 1.3.15
>
>
> As explained in OAK-1966 diff logic makes a call like
> bq. db.nodes.find({ _id: { $gt: "3:/content/foo/01/", $lt:
> "3:/content/foo010" }, _modified: { $gte: 1405085300 } }).sort({_id:1})
> For better and deterministic query performance we would need to create a
> compound index like \{_modified:1, _id:1\}. This index would ensure that
> Mongo does not have to perform object scan while evaluating such a query.
> Care must be taken that index is only created by default for fresh setup. For
> existing setup we should expose a JMX operation which can be invoked by
> system admin to create the required index as per maintenance window
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3924) Fix database-level row deadlock during bulk updates in RDB
[
https://issues.apache.org/jira/browse/OAK-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119383#comment-15119383
]
Julian Reschke commented on OAK-3924:
-
1) Yes.
2) Before we invest time on that we should see whether that actually helps. I
just tried the tests without "shuffle" and that didn't seem to make a
difference...
3) Likely.
> Fix database-level row deadlock during bulk updates in RDB
> --
>
> Key: OAK-3924
> URL: https://issues.apache.org/jira/browse/OAK-3924
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Components: rdbmk
>Reporter: Tomek Rękawek
>Priority: Critical
> Fix For: 1.4
>
> Attachments: OAK-3924.patch
>
>
> It seems that the new bulk createOrUpdate() implementation in RDB is prone
> for the deadlocks. It isn't a bug in the Oak code, but rather something
> related to the database implementations. The bug can be observed if we have
> multiple simultaneous bulk updates and some of the rows repeat among them.
> The attached patch contains an unit test {{testConcurrentWithConflict}}
> presenting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3071) Add a compound index for _modified + _id
[
https://issues.apache.org/jira/browse/OAK-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3071:
---
Fix Version/s: 1.3.15
> Add a compound index for _modified + _id
>
>
> Key: OAK-3071
> URL: https://issues.apache.org/jira/browse/OAK-3071
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: mongomk
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Labels: performance, resilience
> Fix For: 1.4, 1.3.15
>
>
> As explained in OAK-1966 diff logic makes a call like
> bq. db.nodes.find({ _id: { $gt: "3:/content/foo/01/", $lt:
> "3:/content/foo010" }, _modified: { $gte: 1405085300 } }).sort({_id:1})
> For better and deterministic query performance we would need to create a
> compound index like \{_modified:1, _id:1\}. This index would ensure that
> Mongo does not have to perform object scan while evaluating such a query.
> Care must be taken that index is only created by default for fresh setup. For
> existing setup we should expose a JMX operation which can be invoked by
> system admin to create the required index as per maintenance window
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OAK-3850) Collect and expose Persistent Cache stats
[ https://issues.apache.org/jira/browse/OAK-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller reassigned OAK-3850: --- Assignee: Thomas Mueller (was: Chetan Mehrotra) > Collect and expose Persistent Cache stats > - > > Key: OAK-3850 > URL: https://issues.apache.org/jira/browse/OAK-3850 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Teodor Rosu >Assignee: Thomas Mueller > Fix For: 1.4 > > Attachments: OAK-3850-v0.patch, OAK-3850-v1.patch, > test-performance-v0.patch > > > Expose persistent cache statistics ( see: [Guava > CacheStats|http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/cache/CacheStats.html] > ) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3629) Index corruption seen with CopyOnRead when index defnition is recreated
[
https://issues.apache.org/jira/browse/OAK-3629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3629:
---
Fix Version/s: (was: 1.4)
1.3.15
> Index corruption seen with CopyOnRead when index defnition is recreated
> ---
>
> Key: OAK-3629
> URL: https://issues.apache.org/jira/browse/OAK-3629
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
> Fix For: 1.3.15
>
>
> CopyOnRead logic relies on {{reindexCount}} to determine the name of
> directory in which index files would be copied. In normal flow if the index
> is reindexed then this count would get increased and newer index files would
> get copied to a new directory.
> However if the index definition node gets recreated due to some deployment
> process then this count gets reset to 0. Due to which newly created index
> files from reindexing would start getting copied to already existing
> directory and that can lead to corruption.
> So what happened here was
> # System started with index definition I1 and indexing got complete with
> index files saved under index/hash(indexpath)/1 (where 1 is current reindex
> count)
> # A new index definition package was deployed which reset the index count.
> Now reindex happened again and the CopyOnRead logic per current design reused
> the existing index directory. And it so happens that Lucene create file with
> same name and same size but different content. This trips the CopyOnRead
> defense of length based index corruption check and thus cause new lucene
> index to corrupt
> *Note that here corruption is transient i.e. persisted index is not
> corrupted*. Just that locally copied index gets corrupted. Cleaning up the
> index directory would fix the issue and that can be used as a workaround.
> *Fix*
> After discussing with [~tmueller] following approach can be used.
> Instead of relying on reindex count we can maintain a hidden randomly
> generated uuid and store it in the index config. This would be used to derive
> the name of directory on filesystem. If the index definition gets reset then
> the uuid can be regenerated.
> *Workaround*
> Clean the directory used by CopyOnRead which is /index before
> restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3629) Index corruption seen with CopyOnRead when index defnition is recreated
[
https://issues.apache.org/jira/browse/OAK-3629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3629:
---
Priority: Blocker (was: Major)
> Index corruption seen with CopyOnRead when index defnition is recreated
> ---
>
> Key: OAK-3629
> URL: https://issues.apache.org/jira/browse/OAK-3629
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Blocker
> Fix For: 1.3.15
>
>
> CopyOnRead logic relies on {{reindexCount}} to determine the name of
> directory in which index files would be copied. In normal flow if the index
> is reindexed then this count would get increased and newer index files would
> get copied to a new directory.
> However if the index definition node gets recreated due to some deployment
> process then this count gets reset to 0. Due to which newly created index
> files from reindexing would start getting copied to already existing
> directory and that can lead to corruption.
> So what happened here was
> # System started with index definition I1 and indexing got complete with
> index files saved under index/hash(indexpath)/1 (where 1 is current reindex
> count)
> # A new index definition package was deployed which reset the index count.
> Now reindex happened again and the CopyOnRead logic per current design reused
> the existing index directory. And it so happens that Lucene create file with
> same name and same size but different content. This trips the CopyOnRead
> defense of length based index corruption check and thus cause new lucene
> index to corrupt
> *Note that here corruption is transient i.e. persisted index is not
> corrupted*. Just that locally copied index gets corrupted. Cleaning up the
> index directory would fix the issue and that can be used as a workaround.
> *Fix*
> After discussing with [~tmueller] following approach can be used.
> Instead of relying on reindex count we can maintain a hidden randomly
> generated uuid and store it in the index config. This would be used to derive
> the name of directory on filesystem. If the index definition gets reset then
> the uuid can be regenerated.
> *Workaround*
> Clean the directory used by CopyOnRead which is /index before
> restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2744) Change default cache distribution ratio if persistent cache is enabled
[ https://issues.apache.org/jira/browse/OAK-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2744: --- Fix Version/s: (was: 1.4) 1.6 > Change default cache distribution ratio if persistent cache is enabled > -- > > Key: OAK-2744 > URL: https://issues.apache.org/jira/browse/OAK-2744 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk, rdbmk >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Labels: performance > Fix For: 1.6 > > > By default the cache memory in DocumentNodeStore is distributed in following > ratio > * nodeCache - 25% > * childrenCache - 10% > * docChildrenCache - 3% > * diffCache - 5% > * documentCache - Is given the rest i.e. 57% > However off late we have found that with persistent cache enabled we can > lower the cache allocated to Document cache. That would reduce the time spent > in invalidating cache entries in periodic reads. So far we are using > following ration in few setup and that is turning out well > * nodeCachePercentage=35 > * childrenCachePercentage=20 > * diffCachePercentage=30 > * docChildrenCachePercentage=10 > * documentCache - Is given the rest i.e. 5% > We should use the above distribution by default if the persistent cache is > found to be enabled -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2460) Resolve the base directory path of persistent cache against repository home
[
https://issues.apache.org/jira/browse/OAK-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-2460:
---
Fix Version/s: (was: 1.4)
1.6
> Resolve the base directory path of persistent cache against repository home
> ---
>
> Key: OAK-2460
> URL: https://issues.apache.org/jira/browse/OAK-2460
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: mongomk
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Labels: technical_debt
> Fix For: 1.6
>
>
> Currently PersistentCache uses the directory path directly. Various other
> parts in Oak which need access to the filesystem currently make use of
> {{repository.home}} framework property in OSGi env [1]
> Same should also be used in PersistentCache
> [1] http://jackrabbit.apache.org/oak/docs/osgi_config.html#SegmentNodeStore
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3642) Enable failing of commit upon Missing IndexEditor implementation by default
[ https://issues.apache.org/jira/browse/OAK-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3642: --- Fix Version/s: (was: 1.4) 1.6 > Enable failing of commit upon Missing IndexEditor implementation by default > --- > > Key: OAK-3642 > URL: https://issues.apache.org/jira/browse/OAK-3642 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.6 > > Attachments: OAK-3642.patch > > > With OAK-3505 we exposed config option to enable failing of commit if any > IndexEditor is missing. The default should be changed to always fail the > commit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-3694) As a user, I want to know when a node that I've created has been indexed.
[
https://issues.apache.org/jira/browse/OAK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Mueller resolved OAK-3694.
-
Resolution: Won't Fix
> As a user, I want to know when a node that I've created has been indexed.
> -
>
> Key: OAK-3694
> URL: https://issues.apache.org/jira/browse/OAK-3694
> Project: Jackrabbit Oak
> Issue Type: Story
>Reporter: Valentin Olteanu
>Assignee: Chetan Mehrotra
> Fix For: 1.4
>
>
> As a user, I want to know when a node that I've created has been indexed so
> that I can start using it.
> Generalization: As a user, I want to know when everything created before
> timestamp T0 has been indexed.
> Ideal solution: MBean operation {{bool isIndexedBefore(long timestamp)}} that
> returns {{true}} if everything created before {{timestamp}} has been indexed
> (by all indexers).
> Current options:
> # check IndexStatsMBean for Start and Done to determine if a cycle has
> started after adding the node and has finished. Issue: there can be multiple
> async indexers. OAK-3606 proposes a solution.
> # add a node and search for it until it is retrieved. Issue: must add
> different nodes for different indexers and wait for all.
> These options are not precise (give false positives) and are not resilient to
> changes in indexing strategy (e.g. adding one more indexer breaks the
> checks).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3694) As a user, I want to know when a node that I've created has been indexed.
[
https://issues.apache.org/jira/browse/OAK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119361#comment-15119361
]
Thomas Mueller commented on OAK-3694:
-
The alternative approach works well, so I think we no longer need this feature.
> As a user, I want to know when a node that I've created has been indexed.
> -
>
> Key: OAK-3694
> URL: https://issues.apache.org/jira/browse/OAK-3694
> Project: Jackrabbit Oak
> Issue Type: Story
>Reporter: Valentin Olteanu
>Assignee: Chetan Mehrotra
> Fix For: 1.4
>
>
> As a user, I want to know when a node that I've created has been indexed so
> that I can start using it.
> Generalization: As a user, I want to know when everything created before
> timestamp T0 has been indexed.
> Ideal solution: MBean operation {{bool isIndexedBefore(long timestamp)}} that
> returns {{true}} if everything created before {{timestamp}} has been indexed
> (by all indexers).
> Current options:
> # check IndexStatsMBean for Start and Done to determine if a cycle has
> started after adding the node and has finished. Issue: there can be multiple
> async indexers. OAK-3606 proposes a solution.
> # add a node and search for it until it is retrieved. Issue: must add
> different nodes for different indexers and wait for all.
> These options are not precise (give false positives) and are not resilient to
> changes in indexing strategy (e.g. adding one more indexer breaks the
> checks).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3871) ability to override ClusterNodeInfo#DEFAULT_LEASE_DURATION_MILLIS
[ https://issues.apache.org/jira/browse/OAK-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119341#comment-15119341 ] Stefan Egli commented on OAK-3871: -- bq. not restricting the value (should it?). in my view we don't have to restrict the values, just document reasonable value ranges > ability to override ClusterNodeInfo#DEFAULT_LEASE_DURATION_MILLIS > - > > Key: OAK-3871 > URL: https://issues.apache.org/jira/browse/OAK-3871 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Julian Reschke >Assignee: Julian Reschke > Fix For: 1.4 > > Attachments: OAK-3871.diff > > > This constant currently is set to 120s. > In edge cases, it might be good to override the default (when essentially the > only other "fix" would be to turn off the lease check). > Proposal: introduce a system property allowing to tune the lease time (but > restrict allowable values to a "sane" range, such as up to 5min). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3865) New strategy to optimize secondary reads
[ https://issues.apache.org/jira/browse/OAK-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3865: --- Fix Version/s: (was: 1.4) 1.6 > New strategy to optimize secondary reads > > > Key: OAK-3865 > URL: https://issues.apache.org/jira/browse/OAK-3865 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Tomek Rękawek > Labels: performance > Fix For: 1.6 > > Attachments: diagram.png > > > *Introduction* > In the current trunk we'll only read document _D_ from the secondary instance > if: > (1) we have the parent _P_ of document _D_ cached and > (2) the parent hasn't been modified in 6 hours. > The OAK-2106 tried to optimise (2) by estimating lag using MongoDB replica > stats. It was unreliable, so the second approach was to read the last > revisions directly from each Mongo instance. If the modification date of _P_ > is before last revisions on all secondary Mongos, then secondary can be used. > The main problem with this approach is that we still need to have the _P_ to > be in cache. I think we need another way to optimise the secondary reading, > as right now only about 3% of requests connects to the secondary, which is > bad especially for the global-clustering case (Mongo and Oak instances across > the globe). The optimisation provided in OAK-2106 doesn't make the things > much better and may introduce some consistency issues. > *Proposal* > I had following constraints in mind preparing this: > 1. Let's assume we have a sequence of commits with revisions _R1_, _R2_ and > _R3_ modifying nodes _N1_, _N2_ and _N3_. If we already read the _N1_ from > revision _R2_ then reading from a secondary shouldn't result in getting older > revision (eg. _R1_). > 2. If an Oak instance modifies a document, then reading from a secondary > shouldn't result in getting the old version (before modification). > So, let's have two maps: > * _M1_ the most recent document revision read from the Mongo for each cluster > id, > * _M2_ the oldest last rev value for root document for each cluster id read > from all the secondary instances. > Maintaining _M1_: > For every read from the Mongo we'll check if the lastRev for some cluster id > is newer than _M1_ entry. If so, we'll update _M1_. For all writes we'll add > the saved revision id with the current cluster id in _M1_. > Maintaining _M2_: > It should be periodically updated. Such mechanism is already prepared in the > OAK-2106 patch. > The method deciding whether we can read from the secondary instance should > compare two maps. If all entries in _M2_ are newer than _M1_ it means that > the secondary instances contains at least as new repository state as we > already accessed and therefore it's safe to read from secondary. > Regarding the documents modified by the local Oak instance, we should > remember all the locally-modified paths and their revisions and use primary > Mongo to access them as long as the changes are not replicated to all the > secondaries. When the secondaries are up to date with the modification, we > can remove it from the local-changes collections. > Attached image diagram.png presents the idea. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (OAK-2066) DocumentStore API: batch create, but no batch update
[ https://issues.apache.org/jira/browse/OAK-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger reassigned OAK-2066: - Assignee: Marcel Reutegger > DocumentStore API: batch create, but no batch update > > > Key: OAK-2066 > URL: https://issues.apache.org/jira/browse/OAK-2066 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Julian Reschke >Assignee: Marcel Reutegger >Priority: Blocker > Labels: performance > Fix For: 1.4, 1.3.15 > > > The DocumentStore API currently has a call for creating many nodes at once. > However, this will sometimes fail for large save operations in JCR, because > in the DS persistence, JCR-deleted nodes are still present (with a deleted > flag). This causes two subsequent sequences of > 1) create test container > 2) create many child nodes > 3) remove test container > to behave very differently, depending on whether the test container is > created for the first time or not. > (see CreateManyChildNodesTest) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3842) Adjust package export declarations
[ https://issues.apache.org/jira/browse/OAK-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3842: --- Fix Version/s: 1.4 > Adjust package export declarations > --- > > Key: OAK-3842 > URL: https://issues.apache.org/jira/browse/OAK-3842 > Project: Jackrabbit Oak > Issue Type: Task >Reporter: Michael Dürig >Assignee: Michael Dürig >Priority: Blocker > Labels: api, modularization, technical_debt > Fix For: 1.4, 1.3.15 > > Attachments: OAK-3842-2.patch, OAK-3842.patch > > > We need to adjust the package export declarations such that they become > manageable with our branch / release model. > See http://markmail.org/thread/5g3viq5pwtdryapr for discussion. > I propose to remove package export declarations from all packages that we > don't consider public API / SPI beyond Oak itself. This would allow us to > evolve Oak internal stuff (e.g. things used across Oak modules) freely > without having to worry about merges to branches messing up semantic > versioning. OTOH it would force us to keep externally facing public API / SPI > reasonably stable also across the branches. Furthermore such an approach > would send the right signal to Oak API / SPI consumers regarding the > stability assumptions they can make. > An external API / SPI having a (transitive) dependency on internals might be > troublesome. In doubt I would remove the export version here until we can > make reasonable guarantees (either through decoupling the code or stabilising > the dependencies). > I would start digging through the export version and prepare an initial > proposal for further discussion. > /cc [~frm], [~chetanm], [~mmarth] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2066) DocumentStore API: batch create, but no batch update
[ https://issues.apache.org/jira/browse/OAK-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2066: --- Priority: Blocker (was: Major) > DocumentStore API: batch create, but no batch update > > > Key: OAK-2066 > URL: https://issues.apache.org/jira/browse/OAK-2066 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Julian Reschke >Priority: Blocker > Labels: performance > Fix For: 1.4, 1.3.15 > > > The DocumentStore API currently has a call for creating many nodes at once. > However, this will sometimes fail for large save operations in JCR, because > in the DS persistence, JCR-deleted nodes are still present (with a deleted > flag). This causes two subsequent sequences of > 1) create test container > 2) create many child nodes > 3) remove test container > to behave very differently, depending on whether the test container is > created for the first time or not. > (see CreateManyChildNodesTest) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2066) DocumentStore API: batch create, but no batch update
[ https://issues.apache.org/jira/browse/OAK-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2066: --- Fix Version/s: 1.3.15 > DocumentStore API: batch create, but no batch update > > > Key: OAK-2066 > URL: https://issues.apache.org/jira/browse/OAK-2066 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk >Reporter: Julian Reschke > Labels: performance > Fix For: 1.4, 1.3.15 > > > The DocumentStore API currently has a call for creating many nodes at once. > However, this will sometimes fail for large save operations in JCR, because > in the DS persistence, JCR-deleted nodes are still present (with a deleted > flag). This causes two subsequent sequences of > 1) create test container > 2) create many child nodes > 3) remove test container > to behave very differently, depending on whether the test container is > created for the first time or not. > (see CreateManyChildNodesTest) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3871) ability to override ClusterNodeInfo#DEFAULT_LEASE_DURATION_MILLIS
[ https://issues.apache.org/jira/browse/OAK-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Parvulescu updated OAK-3871: - Assignee: Julian Reschke > ability to override ClusterNodeInfo#DEFAULT_LEASE_DURATION_MILLIS > - > > Key: OAK-3871 > URL: https://issues.apache.org/jira/browse/OAK-3871 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Julian Reschke >Assignee: Julian Reschke > Fix For: 1.4 > > Attachments: OAK-3871.diff > > > This constant currently is set to 120s. > In edge cases, it might be good to override the default (when essentially the > only other "fix" would be to turn off the lease check). > Proposal: introduce a system property allowing to tune the lease time (but > restrict allowable values to a "sane" range, such as up to 5min). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3724) Use the bulk createOrUpdate in Commit
[
https://issues.apache.org/jira/browse/OAK-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3724:
---
Priority: Blocker (was: Major)
> Use the bulk createOrUpdate in Commit
> -
>
> Key: OAK-3724
> URL: https://issues.apache.org/jira/browse/OAK-3724
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: documentmk
>Reporter: Tomek Rękawek
>Priority: Blocker
> Fix For: 1.4, 1.3.15
>
> Attachments: OAK-3724.patch
>
>
> The {{DocumentStore#createOrUpdate(Collection, UpdateOp)}} method is invoked
> in a loop in the {{Commit#applyToDocumentStore()}}, once for each changed
> node. Replace it with the bulk version created in OAK-3662.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3724) Use the bulk createOrUpdate in Commit
[
https://issues.apache.org/jira/browse/OAK-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3724:
---
Fix Version/s: 1.3.15
> Use the bulk createOrUpdate in Commit
> -
>
> Key: OAK-3724
> URL: https://issues.apache.org/jira/browse/OAK-3724
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: documentmk
>Reporter: Tomek Rękawek
> Fix For: 1.4, 1.3.15
>
> Attachments: OAK-3724.patch
>
>
> The {{DocumentStore#createOrUpdate(Collection, UpdateOp)}} method is invoked
> in a loop in the {{Commit#applyToDocumentStore()}}, once for each changed
> node. Replace it with the bulk version created in OAK-3662.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3866) Sorting on relative properties doesn't work in Solr
[
https://issues.apache.org/jira/browse/OAK-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Marth updated OAK-3866:
---
Fix Version/s: (was: 1.4)
1.6
> Sorting on relative properties doesn't work in Solr
> ---
>
> Key: OAK-3866
> URL: https://issues.apache.org/jira/browse/OAK-3866
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: solr
>Affects Versions: 1.0.22, 1.3.13
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.6
>
>
> Executing a query like
> {noformat}
> /jcr:root/content/foo//*[(@sling:resourceType = 'x' or @sling:resourceType =
> 'y') and jcr:contains(., 'bar*~')] order by jcr:content/@jcr:primaryType
> descending
> {noformat}
> would assume sorting on the _jcr:primaryType_ property of resulting nodes'
> _jcr:content_ children.
> That is currently not supported in Solr, while it is in Lucene as the latter
> supports index time aggregation.
> We should inspect if it's possible to extend support for Solr too, most
> probably via index time aggregation.
> The query should not fail but at least log a warning about that limitation
> for the time being.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2727) NodeStateSolrServersObserver should be filtering path selectively
[ https://issues.apache.org/jira/browse/OAK-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-2727: --- Fix Version/s: (was: 1.4) 1.6 > NodeStateSolrServersObserver should be filtering path selectively > - > > Key: OAK-2727 > URL: https://issues.apache.org/jira/browse/OAK-2727 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: solr >Affects Versions: 1.1.8 >Reporter: Tommaso Teofili >Assignee: Tommaso Teofili > Labels: performance > Fix For: 1.6 > > > As discussed in OAK-2718 it'd be good to be able to selectively find Solr > indexes by path, as done in Lucene index, see also OAK-2570. > This would avoid having to do full diffs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-3924) Fix database-level row deadlock during bulk updates in RDB
[
https://issues.apache.org/jira/browse/OAK-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119323#comment-15119323
]
Tomek Rękawek edited comment on OAK-3924 at 1/27/16 2:42 PM:
-
OK, so we need to do following things here:
1. remove the Oak-level locking from the patch, as it doesn't fix the issue
anyway,
2. sort updates before sending them to DB,
3. disable the bulk updates by default and enable them with a system property.
[~reschke], does it make sense to you?
was (Author: tomek.rekawek):
OK, so we need to do following things here:
1. remove the Oak-level locking from the patch, as it doesn't fix the issue
anyway,
2. sort updates before sending them to DB,
3. disable the bulk updates by default and enable them with a system property.
Makes sense?
> Fix database-level row deadlock during bulk updates in RDB
> --
>
> Key: OAK-3924
> URL: https://issues.apache.org/jira/browse/OAK-3924
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Components: rdbmk
>Reporter: Tomek Rękawek
>Priority: Critical
> Fix For: 1.4
>
> Attachments: OAK-3924.patch
>
>
> It seems that the new bulk createOrUpdate() implementation in RDB is prone
> for the deadlocks. It isn't a bug in the Oak code, but rather something
> related to the database implementations. The bug can be observed if we have
> multiple simultaneous bulk updates and some of the rows repeat among them.
> The attached patch contains an unit test {{testConcurrentWithConflict}}
> presenting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3253) Support caching in FileDataStoreService
[ https://issues.apache.org/jira/browse/OAK-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119324#comment-15119324 ] Michael Marth commented on OAK-3253: [~shgupta], do you think we can still get this in 1.4 (see release schedule on oak-dev)? > Support caching in FileDataStoreService > --- > > Key: OAK-3253 > URL: https://issues.apache.org/jira/browse/OAK-3253 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob >Affects Versions: 1.3.3 >Reporter: Shashank Gupta >Assignee: Shashank Gupta > Labels: candidate_oak_1_0, candidate_oak_1_2, docs-impacting, > features, performance > Fix For: 1.4 > > > FDS on SAN/NAS storage is not efficient as it involves network call. In OAK. > indexes are stored SAN/NAS and even idle system does lot of read system > generated data. > Enable caching in FDS so the reads are done locally and async upload to > SAN/NAS > See [previous > discussions|https://issues.apache.org/jira/browse/OAK-3005?focusedCommentId=14700801&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700801] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3559) Bulk document updates in MongoDocumentStore
[ https://issues.apache.org/jira/browse/OAK-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3559: --- Fix Version/s: 1.3.15 > Bulk document updates in MongoDocumentStore > --- > > Key: OAK-3559 > URL: https://issues.apache.org/jira/browse/OAK-3559 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: mongomk >Reporter: Tomek Rękawek >Assignee: Marcel Reutegger > Fix For: 1.4, 1.3.15 > > Attachments: OAK-3559.patch > > > Using the MongoDB [Bulk > API|https://docs.mongodb.org/manual/reference/method/Bulk/#Bulk] implement > the [batch version of createOrUpdate method|OAK-3662]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3559) Bulk document updates in MongoDocumentStore
[ https://issues.apache.org/jira/browse/OAK-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Marth updated OAK-3559: --- Priority: Blocker (was: Major) > Bulk document updates in MongoDocumentStore > --- > > Key: OAK-3559 > URL: https://issues.apache.org/jira/browse/OAK-3559 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: mongomk >Reporter: Tomek Rękawek >Assignee: Marcel Reutegger >Priority: Blocker > Fix For: 1.4, 1.3.15 > > Attachments: OAK-3559.patch > > > Using the MongoDB [Bulk > API|https://docs.mongodb.org/manual/reference/method/Bulk/#Bulk] implement > the [batch version of createOrUpdate method|OAK-3662]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3924) Fix database-level row deadlock during bulk updates in RDB
[
https://issues.apache.org/jira/browse/OAK-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119323#comment-15119323
]
Tomek Rękawek commented on OAK-3924:
OK, so we need to do following things here:
1. remove the Oak-level locking from the patch, as it doesn't fix the issue
anyway,
2. sort updates before sending them to DB,
3. disable the bulk updates by default and enable them with a system property.
Makes sense?
> Fix database-level row deadlock during bulk updates in RDB
> --
>
> Key: OAK-3924
> URL: https://issues.apache.org/jira/browse/OAK-3924
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Components: rdbmk
>Reporter: Tomek Rękawek
>Priority: Critical
> Fix For: 1.4
>
> Attachments: OAK-3924.patch
>
>
> It seems that the new bulk createOrUpdate() implementation in RDB is prone
> for the deadlocks. It isn't a bug in the Oak code, but rather something
> related to the database implementations. The bug can be observed if we have
> multiple simultaneous bulk updates and some of the rows repeat among them.
> The attached patch contains an unit test {{testConcurrentWithConflict}}
> presenting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3458) Add support for the LIMIT and OFFSET SQL clauses
[
https://issues.apache.org/jira/browse/OAK-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Mueller updated OAK-3458:
Fix Version/s: (was: 1.4)
> Add support for the LIMIT and OFFSET SQL clauses
>
>
> Key: OAK-3458
> URL: https://issues.apache.org/jira/browse/OAK-3458
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Radu Cotescu
> Attachments: OAK-3458.1.patch
>
>
> The {{org.apache.jackrabbit.oak.query.SQL2Parser}} should support the
> {{LIMIT}} and {{OFFSET}} clauses, since the
> {{org.apache.jackrabbit.oak.query.Query}} object provides support for setting
> a limit and an offset for returning results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3924) Fix database-level row deadlock during bulk updates in RDB
[
https://issues.apache.org/jira/browse/OAK-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119302#comment-15119302
]
Julian Reschke commented on OAK-3924:
-
We also should think about sorting the document update operations internally...
> Fix database-level row deadlock during bulk updates in RDB
> --
>
> Key: OAK-3924
> URL: https://issues.apache.org/jira/browse/OAK-3924
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Components: rdbmk
>Reporter: Tomek Rękawek
>Priority: Critical
> Fix For: 1.4
>
> Attachments: OAK-3924.patch
>
>
> It seems that the new bulk createOrUpdate() implementation in RDB is prone
> for the deadlocks. It isn't a bug in the Oak code, but rather something
> related to the database implementations. The bug can be observed if we have
> multiple simultaneous bulk updates and some of the rows repeat among them.
> The attached patch contains an unit test {{testConcurrentWithConflict}}
> presenting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3923) Async indexing delayed by 30 minutes because stop order is incorrect
[
https://issues.apache.org/jira/browse/OAK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Mueller updated OAK-3923:
Fix Version/s: (was: 1.3.15)
1.4
> Async indexing delayed by 30 minutes because stop order is incorrect
>
>
> Key: OAK-3923
> URL: https://issues.apache.org/jira/browse/OAK-3923
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
>Reporter: Thomas Mueller
> Fix For: 1.4
>
>
> The stop order of Oak components is incorrect, and this can lead to an async
> indexing delay of 30 minutes, because the indexing lease is not removed. The
> problem is that the node store is stopped before the async index is stopped,
> so that async indexing can still be in progress, and then when async indexing
> is done, the lease can not be removed because the node store is not available.
> From the log file:
> {noformat}
> error.log:
> 21.01.2016 11:53:56.898 *INFO* [FelixStartLevel]
> org.apache.jackrabbit.oak-tarmk-standby BundleEvent STOPPED
> 21.01.2016 11:53:56.900 *INFO* [FelixStartLevel]
> org.apache.jackrabbit.oak-solr-osgi Service
> [org.apache.jackrabbit.oak.plugins.index.solr.osgi.SolrIndexEditorProviderService,571,
> [org.apache.jackrabbit.oak.plugins.index.IndexEditorProvider]] ServiceEvent
> UNREGISTERING
> ...
> 21.01.2016 11:53:56.930 *INFO* [FelixStartLevel]
> org.apache.jackrabbit.oak-lucene BundleEvent STOPPING
> 21.01.2016 11:53:56.930 *INFO* [FelixStartLevel]
> org.apache.jackrabbit.oak-lucene BundleEvent STOPPED
> 21.01.2016 11:53:56.931 *INFO* [FelixStartLevel]
> org.apache.jackrabbit.oak-core Service
> [org.apache.jackrabbit.oak.plugins.index.property.PropertyIndexProvider,405,
> [org.apache.jackrabbit.oak.spi.query.QueryIndexProvider]] ServiceEvent
> UNREGISTERING
> ...
> 21.01.2016 11:53:56.936 *INFO* [FelixStartLevel]
> com.adobe.granite.repository.impl.SlingRepositoryManager stop: Repository
> still running, forcing shutdown
> ...
> 21.01.2016 11:53:56.960 *WARN* [FelixStartLevel]
> org.apache.jackrabbit.oak.osgi.OsgiWhiteboard Error unregistering service:
> com.adobe.granite.repository.impl.SlingRepositoryManager$1@7c052458 of type
> java.util.concurrent.Executor
> java.lang.IllegalStateException: Service already unregistered.
> at
> org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:136)
> at
> org.apache.jackrabbit.oak.osgi.OsgiWhiteboard$1.unregister(OsgiWhiteboard.java:81)
> at
> org.apache.jackrabbit.oak.spi.whiteboard.CompositeRegistration.unregister(CompositeRegistration.java:43)
> at org.apache.jackrabbit.oak.Oak$6.close(Oak.java:592)
> ...
> 21.01.2016 11:56:50.985 *INFO* [FelixStartLevel]
> org.apache.jackrabbit.oak-core Service [763,
> [org.apache.jackrabbit.oak.plugins.segment.SegmentStoreProvider]]
> ServiceEvent UNREGISTERING
>
> debug.log:
> 21.01.2016 11:56:51.964 *WARN* [sling-default-4]
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate [async] The index
> update failed
> java.lang.IllegalStateException: service must be activated when used
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:150)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.getNodeStore(SegmentNodeStoreService.java:233)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.getNodeStore(SegmentNodeStoreService.java:92)
> at
> org.apache.jackrabbit.oak.spi.state.ProxyNodeStore.getRoot(ProxyNodeStore.java:36)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate$AsyncUpdateCallback.close(AsyncIndexUpdate.java:266)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:451)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:351)
>
> error.log:
> 21.01.2016 11:56:51.965 *ERROR* [sling-default-4]
> org.apache.sling.commons.scheduler.impl.QuartzScheduler Exception during job
> execution of
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate@1706b18c : service
> must be activated when used
> java.lang.IllegalStateException: service must be activated when used
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:150)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.getNodeStore(SegmentNodeStoreService.java:233)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.getNodeStore(SegmentNodeStoreService.java:92)
> at
> org.apache.jackrabbit.oak.spi.state.ProxyNodeStore.release(ProxyNodeStore.java:90)
> at
> org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:372)
> at
> org.apache.sling.commons.sch
[jira] [Updated] (OAK-2761) Persistent cache: add data in a different thread
[
https://issues.apache.org/jira/browse/OAK-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Mueller updated OAK-2761:
Fix Version/s: (was: 1.4)
> Persistent cache: add data in a different thread
>
>
> Key: OAK-2761
> URL: https://issues.apache.org/jira/browse/OAK-2761
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: cache, core, mongomk
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Labels: resilience
>
> The persistent cache usually stores data in a background thread, but
> sometimes (if a lot of data is added quickly) the foreground thread is
> blocked.
> Even worse, switching the cache file can happen in a foreground thread, with
> the following stack trace.
> {noformat}
> "127.0.0.1 [1428931262206] POST /bin/replicate.json HTTP/1.1" prio=5
> tid=0x7fe5df819800 nid=0x9907 runnable [0x000113fc4000]
>java.lang.Thread.State: RUNNABLE
> ...
> at org.h2.mvstore.MVStoreTool.compact(MVStoreTool.java:404)
> at
> org.apache.jackrabbit.oak.plugins.document.persistentCache.PersistentCache$1.closeStore(PersistentCache.java:213)
> - locked <0x000782483050> (a
> org.apache.jackrabbit.oak.plugins.document.persistentCache.PersistentCache$1)
> at
> org.apache.jackrabbit.oak.plugins.document.persistentCache.PersistentCache.switchGenerationIfNeeded(PersistentCache.java:350)
> - locked <0x000782455710> (a
> org.apache.jackrabbit.oak.plugins.document.persistentCache.PersistentCache)
> at
> org.apache.jackrabbit.oak.plugins.document.persistentCache.NodeCache.write(NodeCache.java:85)
> at
> org.apache.jackrabbit.oak.plugins.document.persistentCache.NodeCache.put(NodeCache.java:130)
> at
> org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.applyChanges(DocumentNodeStore.java:1060)
> at
> org.apache.jackrabbit.oak.plugins.document.Commit.applyToCache(Commit.java:599)
> at
> org.apache.jackrabbit.oak.plugins.document.CommitQueue.afterTrunkCommit(CommitQueue.java:127)
> - locked <0x000781890788> (a
> org.apache.jackrabbit.oak.plugins.document.CommitQueue)
> at
> org.apache.jackrabbit.oak.plugins.document.CommitQueue.done(CommitQueue.java:83)
> at
> org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.done(DocumentNodeStore.java:637)
> {noformat}
> To avoid blocking the foreground thread, one solution is to store all data in
> a separate thread. If there is too much data added, then some of the data is
> not stored. If possible, the data that was not referenced a lot, and / or old
> revisions of documents (if new revisions are available).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3393) NodeStore wrapper implementation
[ https://issues.apache.org/jira/browse/OAK-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller updated OAK-3393: Fix Version/s: (was: 1.4) > NodeStore wrapper implementation > > > Key: OAK-3393 > URL: https://issues.apache.org/jira/browse/OAK-3393 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Reporter: Thomas Mueller > Attachments: OAK-3393.patch > > > I would like to have a node store wrapper implementation. Initial use cases: > * logging (log all NodeStore API calls, possibly with a filter, to be > analyzed later) > * statistics (counting the number of calls, possibly by path) > * profiling (measuring how long calls take) > Later use cases: > * SegmentStore compaction > * on-the-fly migration from one nodestore to another > * maybe: virtual repository (mounting node stores) > I made a first prototype, and found some problems with the NodeStore API and > the way we have used it in Oak. Repository initialization fails with an > IllegalArgumentException if the NodeBuilder does not extend > MemoryNodeBuilder. Also, I have trouble understanding some of the methods > (for example rebase, merge). I think the NodeStore API is much much harder to > wrap than (for example) the DataStore API. I think we should fix that, to > make Oak more modular. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3924) Fix database-level row deadlock during bulk updates in RDB
[
https://issues.apache.org/jira/browse/OAK-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-3924:
Priority: Critical (was: Major)
> Fix database-level row deadlock during bulk updates in RDB
> --
>
> Key: OAK-3924
> URL: https://issues.apache.org/jira/browse/OAK-3924
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Components: rdbmk
>Reporter: Tomek Rękawek
>Priority: Critical
> Fix For: 1.4
>
> Attachments: OAK-3924.patch
>
>
> It seems that the new bulk createOrUpdate() implementation in RDB is prone
> for the deadlocks. It isn't a bug in the Oak code, but rather something
> related to the database implementations. The bug can be observed if we have
> multiple simultaneous bulk updates and some of the rows repeat among them.
> The attached patch contains an unit test {{testConcurrentWithConflict}}
> presenting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3924) Fix database-level row deadlock during bulk updates in RDB
[
https://issues.apache.org/jira/browse/OAK-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119289#comment-15119289
]
Julian Reschke commented on OAK-3924:
-
The main problem is that once we make transactions more complex/big, the risk
of both deadlocks (and "regular" locks) goes up. Previously, almost all DB
interactions we did were rather simply (and designed to be that way).
Right now I don't see how we can achieve the performance improvements related
to more bulk operations without running into the risk of locking problems.
For OAK 1.4, we either need to switch off this code path, or at least add a
system variable that allows to do so.
Other things we can look into:
- break down JDBC transactions into smaller *logical* pieces
- break down JDBC transaction size (partition big update operations)
> Fix database-level row deadlock during bulk updates in RDB
> --
>
> Key: OAK-3924
> URL: https://issues.apache.org/jira/browse/OAK-3924
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Components: rdbmk
>Reporter: Tomek Rękawek
> Fix For: 1.4
>
> Attachments: OAK-3924.patch
>
>
> It seems that the new bulk createOrUpdate() implementation in RDB is prone
> for the deadlocks. It isn't a bug in the Oak code, but rather something
> related to the database implementations. The bug can be observed if we have
> multiple simultaneous bulk updates and some of the rows repeat among them.
> The attached patch contains an unit test {{testConcurrentWithConflict}}
> presenting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3846) Add parameter to skip SNS nodes
[
https://issues.apache.org/jira/browse/OAK-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119290#comment-15119290
]
Tomek Rękawek commented on OAK-3846:
Patch attached. It contains a new editor and the unit tests. [~jsedding], could
you take a look on this?
> Add parameter to skip SNS nodes
> ---
>
> Key: OAK-3846
> URL: https://issues.apache.org/jira/browse/OAK-3846
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: upgrade
>Affects Versions: 1.3.13
>Reporter: Ilyas Türkben
> Attachments: OAK-3846.patch
>
>
> Add a parameter to skip migration of SNS nodes.
> Currently SNS nodes break the upgrade process with an error similar to the
> following:
> {code:java}
> 31.08.2015 13:18:56.121 *ERROR* [FelixFrameworkWiring]
> org.apache.sling.jcr.base.internal.loader.Loader Error loading node types
> SLING-INF/nodetypes/folder.cnd from bundle org.apache.sling.jcr.resource: {}
> javax.jcr.nodetype.ConstraintViolationException: Failed to register node
> types.
> at
> org.apache.jackrabbit.oak.api.CommitFailedException.asRepositoryException(CommitFailedException.java:225)
> at
> org.apache.jackrabbit.oak.plugins.nodetype.write.ReadWriteNodeTypeManager.registerNodeTypes(ReadWriteNodeTypeManager.java:156)
> at
> org.apache.jackrabbit.commons.cnd.CndImporter.registerNodeTypes(CndImporter.java:162)
> at
> org.apache.sling.jcr.base.NodeTypeLoader.registerNodeType(NodeTypeLoader.java:124)
> at
> org.apache.sling.jcr.base.internal.loader.Loader.registerNodeTypes(Loader.java:296)
> at
> org.apache.sling.jcr.base.internal.loader.Loader.registerBundleInternal(Loader.java:237)
> at
> org.apache.sling.jcr.base.internal.loader.Loader.registerBundle(Loader.java:166)
> at
> org.apache.sling.jcr.base.internal.loader.Loader.(Loader.java:78)
> at
> org.apache.sling.jcr.base.NamespaceMappingSupport.setup(NamespaceMappingSupport.java:81)
> at
> org.apache.sling.jcr.base.AbstractSlingRepositoryManager.start(AbstractSlingRepositoryManager.java:313)
> at
> com.adobe.granite.repository.impl.SlingRepositoryManager.activate(SlingRepositoryManager.java:267)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:222)
> at
> org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:37)
> at
> org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:615)
> at
> org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:499)
> at
> org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:295)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:302)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:113)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:832)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:799)
> at
> org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:724)
> at
> org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:927)
> at
> org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:891)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1492)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1413)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:1222)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:1158)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1444)
> at
> org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:987)
> at
> org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
> at
> org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
> at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4547)
>
[jira] [Comment Edited] (OAK-3846) Add parameter to skip SNS nodes
[
https://issues.apache.org/jira/browse/OAK-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097938#comment-15097938
]
Tomek Rękawek edited comment on OAK-3846 at 1/27/16 2:22 PM:
-
[~tuerkben], I was thinking about the the {{old_name_2_}}, {{old_name_3_}},
etc. schema and a warning message for each renamed node.
was (Author: tomek.rekawek):
[~tuerkben], I was thinking about the the {{old_name[2]}}, {{old_name[3]}},
etc. schema and a warning message for each renamed node.
> Add parameter to skip SNS nodes
> ---
>
> Key: OAK-3846
> URL: https://issues.apache.org/jira/browse/OAK-3846
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: upgrade
>Affects Versions: 1.3.13
>Reporter: Ilyas Türkben
> Attachments: OAK-3846.patch
>
>
> Add a parameter to skip migration of SNS nodes.
> Currently SNS nodes break the upgrade process with an error similar to the
> following:
> {code:java}
> 31.08.2015 13:18:56.121 *ERROR* [FelixFrameworkWiring]
> org.apache.sling.jcr.base.internal.loader.Loader Error loading node types
> SLING-INF/nodetypes/folder.cnd from bundle org.apache.sling.jcr.resource: {}
> javax.jcr.nodetype.ConstraintViolationException: Failed to register node
> types.
> at
> org.apache.jackrabbit.oak.api.CommitFailedException.asRepositoryException(CommitFailedException.java:225)
> at
> org.apache.jackrabbit.oak.plugins.nodetype.write.ReadWriteNodeTypeManager.registerNodeTypes(ReadWriteNodeTypeManager.java:156)
> at
> org.apache.jackrabbit.commons.cnd.CndImporter.registerNodeTypes(CndImporter.java:162)
> at
> org.apache.sling.jcr.base.NodeTypeLoader.registerNodeType(NodeTypeLoader.java:124)
> at
> org.apache.sling.jcr.base.internal.loader.Loader.registerNodeTypes(Loader.java:296)
> at
> org.apache.sling.jcr.base.internal.loader.Loader.registerBundleInternal(Loader.java:237)
> at
> org.apache.sling.jcr.base.internal.loader.Loader.registerBundle(Loader.java:166)
> at
> org.apache.sling.jcr.base.internal.loader.Loader.(Loader.java:78)
> at
> org.apache.sling.jcr.base.NamespaceMappingSupport.setup(NamespaceMappingSupport.java:81)
> at
> org.apache.sling.jcr.base.AbstractSlingRepositoryManager.start(AbstractSlingRepositoryManager.java:313)
> at
> com.adobe.granite.repository.impl.SlingRepositoryManager.activate(SlingRepositoryManager.java:267)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:222)
> at
> org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:37)
> at
> org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:615)
> at
> org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:499)
> at
> org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:295)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:302)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:113)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:832)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:799)
> at
> org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:724)
> at
> org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:927)
> at
> org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:891)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1492)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1413)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:1222)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:1158)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1444)
> at
> org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:987)
> at
> org.apache.felix.framework.util.EventDispatc
[jira] [Updated] (OAK-3846) Add parameter to skip SNS nodes
[
https://issues.apache.org/jira/browse/OAK-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-3846:
---
Attachment: OAK-3846.patch
> Add parameter to skip SNS nodes
> ---
>
> Key: OAK-3846
> URL: https://issues.apache.org/jira/browse/OAK-3846
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: upgrade
>Affects Versions: 1.3.13
>Reporter: Ilyas Türkben
> Attachments: OAK-3846.patch
>
>
> Add a parameter to skip migration of SNS nodes.
> Currently SNS nodes break the upgrade process with an error similar to the
> following:
> {code:java}
> 31.08.2015 13:18:56.121 *ERROR* [FelixFrameworkWiring]
> org.apache.sling.jcr.base.internal.loader.Loader Error loading node types
> SLING-INF/nodetypes/folder.cnd from bundle org.apache.sling.jcr.resource: {}
> javax.jcr.nodetype.ConstraintViolationException: Failed to register node
> types.
> at
> org.apache.jackrabbit.oak.api.CommitFailedException.asRepositoryException(CommitFailedException.java:225)
> at
> org.apache.jackrabbit.oak.plugins.nodetype.write.ReadWriteNodeTypeManager.registerNodeTypes(ReadWriteNodeTypeManager.java:156)
> at
> org.apache.jackrabbit.commons.cnd.CndImporter.registerNodeTypes(CndImporter.java:162)
> at
> org.apache.sling.jcr.base.NodeTypeLoader.registerNodeType(NodeTypeLoader.java:124)
> at
> org.apache.sling.jcr.base.internal.loader.Loader.registerNodeTypes(Loader.java:296)
> at
> org.apache.sling.jcr.base.internal.loader.Loader.registerBundleInternal(Loader.java:237)
> at
> org.apache.sling.jcr.base.internal.loader.Loader.registerBundle(Loader.java:166)
> at
> org.apache.sling.jcr.base.internal.loader.Loader.(Loader.java:78)
> at
> org.apache.sling.jcr.base.NamespaceMappingSupport.setup(NamespaceMappingSupport.java:81)
> at
> org.apache.sling.jcr.base.AbstractSlingRepositoryManager.start(AbstractSlingRepositoryManager.java:313)
> at
> com.adobe.granite.repository.impl.SlingRepositoryManager.activate(SlingRepositoryManager.java:267)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:222)
> at
> org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:37)
> at
> org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:615)
> at
> org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:499)
> at
> org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:295)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:302)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:113)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:832)
> at
> org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:799)
> at
> org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:724)
> at
> org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:927)
> at
> org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:891)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1492)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1413)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:1222)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:1158)
> at
> org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1444)
> at
> org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:987)
> at
> org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
> at
> org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
> at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4547)
> at org.apache.felix.framework.Felix.registerService(Felix.java:3521)
> at
> org.apache.felix.framework.BundleContextImpl.regis
[jira] [Commented] (OAK-3924) Fix database-level row deadlock during bulk updates in RDB
[
https://issues.apache.org/jira/browse/OAK-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119279#comment-15119279
]
Julian Reschke commented on OAK-3924:
-
Note that the test behavior seems to depend a lot on the DB's default deadlock
detection strategies. For instance, the tests seem to hand indefinitely on
PostgresQL and SQLServer,
> Fix database-level row deadlock during bulk updates in RDB
> --
>
> Key: OAK-3924
> URL: https://issues.apache.org/jira/browse/OAK-3924
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Components: rdbmk
>Reporter: Tomek Rękawek
> Fix For: 1.4
>
> Attachments: OAK-3924.patch
>
>
> It seems that the new bulk createOrUpdate() implementation in RDB is prone
> for the deadlocks. It isn't a bug in the Oak code, but rather something
> related to the database implementations. The bug can be observed if we have
> multiple simultaneous bulk updates and some of the rows repeat among them.
> The attached patch contains an unit test {{testConcurrentWithConflict}}
> presenting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3932) DocumentStore.getIfCached() must not return NodeDocument.NULL
[
https://issues.apache.org/jira/browse/OAK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119224#comment-15119224
]
Julian Reschke commented on OAK-3932:
-
trunk: http://svn.apache.org/r1727026 http://svn.apache.org/r1726993
1.2: http://svn.apache.org/r1727033
1.0: http://svn.apache.org/r1727041 http://svn.apache.org/r1727039
> DocumentStore.getIfCached() must not return NodeDocument.NULL
> -
>
> Key: OAK-3932
> URL: https://issues.apache.org/jira/browse/OAK-3932
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, mongomk, rdbmk
>Affects Versions: 1.0, 1.2
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.2.11, 1.3.15, 1.0.27
>
>
> In OAK-3559 [~tomek.rekawek] identified an issue with
> {{MongoDocumentStore.getIfCached()}}. The method may return
> NodeDocument.NULL, when it should actually return {{null}}. The contract of
> the method does not mention it explicitly, but returning {{null}} is
> consistent with other find methods and makes sense IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-2896) Putting many elements into a map results in many small segments.
[
https://issues.apache.org/jira/browse/OAK-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119218#comment-15119218
]
Michael Dürig edited comment on OAK-2896 at 1/27/16 1:26 PM:
-
Some empirical data from a production system backing this:
\# data segments: 1454935
min. size: 64.0
max. size: 262144.0
mean size: 3692.428376476769
std dev: 15269.807288015352
median size: 1120.0
!size-dist.png|width=500!
Distribution of the number of segments vs. segment size. Mind the log scale on
the y axis!!
was (Author: mduerig):
Some empirical data from a production system backing this:
\# data segments: 1454935
min. size: 64.0
max. size: 262144.0
mean size: 3692.428376476769
std dev: 15269.807288015352
median size: 1120.0
!size-dist.png|width=500!
Distribution of the number of segments vs. segment size.
> Putting many elements into a map results in many small segments.
> -
>
> Key: OAK-2896
> URL: https://issues.apache.org/jira/browse/OAK-2896
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: performance
> Attachments: OAK-2896.png, OAK-2896.xlsx, size-dist.png
>
>
> There is an issue with how the HAMT implementation
> ({{SegmentWriter.writeMap()}} interacts with the 256 segment references limit
> when putting many entries into the map: This limit gets regularly reached
> once the maps contains about 200k entries. At that points segments get
> prematurely flushed resulting in more segments, thus more references and thus
> even smaller segments. It is common for segments to be as small as 7k with a
> tar file containing up to 35k segments. This is problematic as at this point
> handling of the segment graph becomes expensive, both memory and CPU wise. I
> have seen persisted segment graphs as big as 35M where the usual size is a
> couple of ks.
> As the HAMT map is used for storing children of a node this might have an
> advert effect on nodes with many child nodes.
> The following code can be used to reproduce the issue:
> {code}
> SegmentWriter writer = new SegmentWriter(segmentStore, getTracker(), V_11);
> MapRecord baseMap = null;
> for (;;) {
> Map map = newHashMap();
> for (int k = 0; k < 1000; k++) {
> RecordId stringId =
> writer.writeString(String.valueOf(rnd.nextLong()));
> map.put(String.valueOf(rnd.nextLong()), stringId);
> }
> Stopwatch w = Stopwatch.createStarted();
> baseMap = writer.writeMap(baseMap, map);
> System.out.println(baseMap.size() + " " + w.elapsed());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-2896) Putting many elements into a map results in many small segments.
[
https://issues.apache.org/jira/browse/OAK-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119218#comment-15119218
]
Michael Dürig edited comment on OAK-2896 at 1/27/16 1:25 PM:
-
Some empirical data from a production system backing this:
\# data segments: 1454935
min. size: 64.0
max. size: 262144.0
mean size: 3692.428376476769
std dev: 15269.807288015352
median size: 1120.0
!size-dist.png|width=500!
Distribution of the number of segments vs. segment size.
was (Author: mduerig):
Some empirical data from a production system backing this:
\# data segments: 1454935
min. size: 64.0
max. size: 262144.0
mean size: 3692.428376476769
std dev: 15269.807288015352
median size: 1120.0
!size-dist.png|widht=500!
Distribution of the number of segments vs. segment size.
> Putting many elements into a map results in many small segments.
> -
>
> Key: OAK-2896
> URL: https://issues.apache.org/jira/browse/OAK-2896
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: performance
> Attachments: OAK-2896.png, OAK-2896.xlsx, size-dist.png
>
>
> There is an issue with how the HAMT implementation
> ({{SegmentWriter.writeMap()}} interacts with the 256 segment references limit
> when putting many entries into the map: This limit gets regularly reached
> once the maps contains about 200k entries. At that points segments get
> prematurely flushed resulting in more segments, thus more references and thus
> even smaller segments. It is common for segments to be as small as 7k with a
> tar file containing up to 35k segments. This is problematic as at this point
> handling of the segment graph becomes expensive, both memory and CPU wise. I
> have seen persisted segment graphs as big as 35M where the usual size is a
> couple of ks.
> As the HAMT map is used for storing children of a node this might have an
> advert effect on nodes with many child nodes.
> The following code can be used to reproduce the issue:
> {code}
> SegmentWriter writer = new SegmentWriter(segmentStore, getTracker(), V_11);
> MapRecord baseMap = null;
> for (;;) {
> Map map = newHashMap();
> for (int k = 0; k < 1000; k++) {
> RecordId stringId =
> writer.writeString(String.valueOf(rnd.nextLong()));
> map.put(String.valueOf(rnd.nextLong()), stringId);
> }
> Stopwatch w = Stopwatch.createStarted();
> baseMap = writer.writeMap(baseMap, map);
> System.out.println(baseMap.size() + " " + w.elapsed());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2896) Putting many elements into a map results in many small segments.
[
https://issues.apache.org/jira/browse/OAK-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-2896:
---
Attachment: size-dist.png
Some empirical data from a production system backing this:
\# data segments: 1454935
min. size: 64.0
max. size: 262144.0
mean size: 3692.428376476769
std dev: 15269.807288015352
median size: 1120.0
!size-dist.png|widht=500!
Distribution of the number of segments vs. segment size.
> Putting many elements into a map results in many small segments.
> -
>
> Key: OAK-2896
> URL: https://issues.apache.org/jira/browse/OAK-2896
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: performance
> Attachments: OAK-2896.png, OAK-2896.xlsx, size-dist.png
>
>
> There is an issue with how the HAMT implementation
> ({{SegmentWriter.writeMap()}} interacts with the 256 segment references limit
> when putting many entries into the map: This limit gets regularly reached
> once the maps contains about 200k entries. At that points segments get
> prematurely flushed resulting in more segments, thus more references and thus
> even smaller segments. It is common for segments to be as small as 7k with a
> tar file containing up to 35k segments. This is problematic as at this point
> handling of the segment graph becomes expensive, both memory and CPU wise. I
> have seen persisted segment graphs as big as 35M where the usual size is a
> couple of ks.
> As the HAMT map is used for storing children of a node this might have an
> advert effect on nodes with many child nodes.
> The following code can be used to reproduce the issue:
> {code}
> SegmentWriter writer = new SegmentWriter(segmentStore, getTracker(), V_11);
> MapRecord baseMap = null;
> for (;;) {
> Map map = newHashMap();
> for (int k = 0; k < 1000; k++) {
> RecordId stringId =
> writer.writeString(String.valueOf(rnd.nextLong()));
> map.put(String.valueOf(rnd.nextLong()), stringId);
> }
> Stopwatch w = Stopwatch.createStarted();
> baseMap = writer.writeMap(baseMap, map);
> System.out.println(baseMap.size() + " " + w.elapsed());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3932) DocumentStore.getIfCached() must not return NodeDocument.NULL
[
https://issues.apache.org/jira/browse/OAK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-3932:
Labels: (was: candidate_oak_1_0)
> DocumentStore.getIfCached() must not return NodeDocument.NULL
> -
>
> Key: OAK-3932
> URL: https://issues.apache.org/jira/browse/OAK-3932
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, mongomk, rdbmk
>Affects Versions: 1.0, 1.2
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.2.11, 1.3.15, 1.0.27
>
>
> In OAK-3559 [~tomek.rekawek] identified an issue with
> {{MongoDocumentStore.getIfCached()}}. The method may return
> NodeDocument.NULL, when it should actually return {{null}}. The contract of
> the method does not mention it explicitly, but returning {{null}} is
> consistent with other find methods and makes sense IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3932) DocumentStore.getIfCached() must not return NodeDocument.NULL
[
https://issues.apache.org/jira/browse/OAK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-3932:
Fix Version/s: 1.0.27
> DocumentStore.getIfCached() must not return NodeDocument.NULL
> -
>
> Key: OAK-3932
> URL: https://issues.apache.org/jira/browse/OAK-3932
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, mongomk, rdbmk
>Affects Versions: 1.0, 1.2
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.2.11, 1.3.15, 1.0.27
>
>
> In OAK-3559 [~tomek.rekawek] identified an issue with
> {{MongoDocumentStore.getIfCached()}}. The method may return
> NodeDocument.NULL, when it should actually return {{null}}. The contract of
> the method does not mention it explicitly, but returning {{null}} is
> consistent with other find methods and makes sense IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3932) DocumentStore.getIfCached() must not return NodeDocument.NULL
[
https://issues.apache.org/jira/browse/OAK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-3932:
Fix Version/s: 1.2.11
> DocumentStore.getIfCached() must not return NodeDocument.NULL
> -
>
> Key: OAK-3932
> URL: https://issues.apache.org/jira/browse/OAK-3932
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, mongomk, rdbmk
>Affects Versions: 1.0, 1.2
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Labels: candidate_oak_1_0
> Fix For: 1.2.11, 1.3.15
>
>
> In OAK-3559 [~tomek.rekawek] identified an issue with
> {{MongoDocumentStore.getIfCached()}}. The method may return
> NodeDocument.NULL, when it should actually return {{null}}. The contract of
> the method does not mention it explicitly, but returning {{null}} is
> consistent with other find methods and makes sense IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3932) DocumentStore.getIfCached() must not return NodeDocument.NULL
[
https://issues.apache.org/jira/browse/OAK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-3932:
Labels: candidate_oak_1_0 (was: candidate_oak_1_0 candidate_oak_1_2)
> DocumentStore.getIfCached() must not return NodeDocument.NULL
> -
>
> Key: OAK-3932
> URL: https://issues.apache.org/jira/browse/OAK-3932
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, mongomk, rdbmk
>Affects Versions: 1.0, 1.2
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Labels: candidate_oak_1_0
> Fix For: 1.2.11, 1.3.15
>
>
> In OAK-3559 [~tomek.rekawek] identified an issue with
> {{MongoDocumentStore.getIfCached()}}. The method may return
> NodeDocument.NULL, when it should actually return {{null}}. The contract of
> the method does not mention it explicitly, but returning {{null}} is
> consistent with other find methods and makes sense IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3932) DocumentStore.getIfCached() must not return NodeDocument.NULL
[
https://issues.apache.org/jira/browse/OAK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119192#comment-15119192
]
Marcel Reutegger commented on OAK-3932:
---
I see, thanks for fixing.
> DocumentStore.getIfCached() must not return NodeDocument.NULL
> -
>
> Key: OAK-3932
> URL: https://issues.apache.org/jira/browse/OAK-3932
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, mongomk, rdbmk
>Affects Versions: 1.0, 1.2
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.3.15
>
>
> In OAK-3559 [~tomek.rekawek] identified an issue with
> {{MongoDocumentStore.getIfCached()}}. The method may return
> NodeDocument.NULL, when it should actually return {{null}}. The contract of
> the method does not mention it explicitly, but returning {{null}} is
> consistent with other find methods and makes sense IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-3932) DocumentStore.getIfCached() must not return NodeDocument.NULL
[
https://issues.apache.org/jira/browse/OAK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119068#comment-15119068
]
Julian Reschke edited comment on OAK-3932 at 1/27/16 12:57 PM:
---
In RDBDocumentStore, this should be done using "unwrap" (fixed in
http://svn.apache.org/r1727026).
was (Author: reschke):
In RDBDocumentStore, this should be done using "unwrap" (will fix myself later).
> DocumentStore.getIfCached() must not return NodeDocument.NULL
> -
>
> Key: OAK-3932
> URL: https://issues.apache.org/jira/browse/OAK-3932
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, mongomk, rdbmk
>Affects Versions: 1.0, 1.2
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.3.15
>
>
> In OAK-3559 [~tomek.rekawek] identified an issue with
> {{MongoDocumentStore.getIfCached()}}. The method may return
> NodeDocument.NULL, when it should actually return {{null}}. The contract of
> the method does not mention it explicitly, but returning {{null}} is
> consistent with other find methods and makes sense IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-3932) DocumentStore.getIfCached() must not return NodeDocument.NULL
[
https://issues.apache.org/jira/browse/OAK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119068#comment-15119068
]
Julian Reschke edited comment on OAK-3932 at 1/27/16 12:37 PM:
---
In RDBDocumentStore, this should be done using "unwrap" (will fix myself later).
was (Author: reschke):
In RDBDocumentStore, this should be done using "unwrap" (fill fix myself later).
> DocumentStore.getIfCached() must not return NodeDocument.NULL
> -
>
> Key: OAK-3932
> URL: https://issues.apache.org/jira/browse/OAK-3932
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, mongomk, rdbmk
>Affects Versions: 1.0, 1.2
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.3.15
>
>
> In OAK-3559 [~tomek.rekawek] identified an issue with
> {{MongoDocumentStore.getIfCached()}}. The method may return
> NodeDocument.NULL, when it should actually return {{null}}. The contract of
> the method does not mention it explicitly, but returning {{null}} is
> consistent with other find methods and makes sense IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3672) SegmentDiscoveryLiteService does not persist clusterView.id
[
https://issues.apache.org/jira/browse/OAK-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119160#comment-15119160
]
Stefan Egli commented on OAK-3672:
--
As suggested on the list the way forward is to not set the id in the
SegmentDiscoveryLiteService case as all and leave the responsibility of
managing the clusterId to upper layers.
There's currently only one known upper layer and that's discovery.oak - and
returning null for the id in the discovery-lite descriptor breaks discovery.oak.
Which means, a fix of OAK-3672 requires SLING-5458 first.
> SegmentDiscoveryLiteService does not persist clusterView.id
> ---
>
> Key: OAK-3672
> URL: https://issues.apache.org/jira/browse/OAK-3672
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
>Affects Versions: 1.3.10
>Reporter: Stefan Egli
>Assignee: Stefan Egli
> Fix For: 1.4
>
>
> The discovery-lite-descriptor introduced with OAK-2844 has a property {{id}}
> that uniquely and persistently identifies a cluster. However, the
> {{SegmentDiscoveryLiteService}} creates this id upon each instance restart
> (by setting {{runtimeClusterId}}).
> This should be fixed to have this {{id}} persisted somehow.
> Note that the consequences of this id changing upon each restart is that the
> corresponding presumed-to-be-persistent {{ClusterView.id}} of the
> discovery.oak will also change upon restart. Which is a violation of the
> discovery API and upper level applications might thus misbehave in this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3924) Fix database-level row deadlock during bulk updates in RDB
[
https://issues.apache.org/jira/browse/OAK-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119096#comment-15119096
]
Tomek Rękawek commented on OAK-3924:
That's right. I prepared a test covering concurrent bulk updates coming from
different DSes. It fails on the RDB, even with the locks, so we need some other
solution.
> Fix database-level row deadlock during bulk updates in RDB
> --
>
> Key: OAK-3924
> URL: https://issues.apache.org/jira/browse/OAK-3924
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Components: rdbmk
>Reporter: Tomek Rękawek
> Fix For: 1.4
>
> Attachments: OAK-3924.patch
>
>
> It seems that the new bulk createOrUpdate() implementation in RDB is prone
> for the deadlocks. It isn't a bug in the Oak code, but rather something
> related to the database implementations. The bug can be observed if we have
> multiple simultaneous bulk updates and some of the rows repeat among them.
> The attached patch contains an unit test {{testConcurrentWithConflict}}
> presenting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3924) Fix database-level row deadlock during bulk updates in RDB
[
https://issues.apache.org/jira/browse/OAK-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-3924:
---
Attachment: OAK-3924.patch
> Fix database-level row deadlock during bulk updates in RDB
> --
>
> Key: OAK-3924
> URL: https://issues.apache.org/jira/browse/OAK-3924
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Components: rdbmk
>Reporter: Tomek Rękawek
> Fix For: 1.4
>
> Attachments: OAK-3924.patch
>
>
> It seems that the new bulk createOrUpdate() implementation in RDB is prone
> for the deadlocks. It isn't a bug in the Oak code, but rather something
> related to the database implementations. The bug can be observed if we have
> multiple simultaneous bulk updates and some of the rows repeat among them.
> The attached patch contains an unit test {{testConcurrentWithConflict}}
> presenting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3924) Fix database-level row deadlock during bulk updates in RDB
[
https://issues.apache.org/jira/browse/OAK-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomek Rękawek updated OAK-3924:
---
Attachment: (was: OAK-3924.patch)
> Fix database-level row deadlock during bulk updates in RDB
> --
>
> Key: OAK-3924
> URL: https://issues.apache.org/jira/browse/OAK-3924
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Components: rdbmk
>Reporter: Tomek Rękawek
> Fix For: 1.4
>
> Attachments: OAK-3924.patch
>
>
> It seems that the new bulk createOrUpdate() implementation in RDB is prone
> for the deadlocks. It isn't a bug in the Oak code, but rather something
> related to the database implementations. The bug can be observed if we have
> multiple simultaneous bulk updates and some of the rows repeat among them.
> The attached patch contains an unit test {{testConcurrentWithConflict}}
> presenting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3932) DocumentStore.getIfCached() must not return NodeDocument.NULL
[
https://issues.apache.org/jira/browse/OAK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119068#comment-15119068
]
Julian Reschke commented on OAK-3932:
-
In RDBDocumentStore, this should be done using "unwrap" (fill fix myself later).
> DocumentStore.getIfCached() must not return NodeDocument.NULL
> -
>
> Key: OAK-3932
> URL: https://issues.apache.org/jira/browse/OAK-3932
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, mongomk, rdbmk
>Affects Versions: 1.0, 1.2
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.3.15
>
>
> In OAK-3559 [~tomek.rekawek] identified an issue with
> {{MongoDocumentStore.getIfCached()}}. The method may return
> NodeDocument.NULL, when it should actually return {{null}}. The contract of
> the method does not mention it explicitly, but returning {{null}} is
> consistent with other find methods and makes sense IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3932) DocumentStore.getIfCached() must not return NodeDocument.NULL
[
https://issues.apache.org/jira/browse/OAK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger resolved OAK-3932.
---
Resolution: Fixed
Fix Version/s: (was: 1.4)
1.3.15
Fixed in trunk: http://svn.apache.org/r1726993
> DocumentStore.getIfCached() must not return NodeDocument.NULL
> -
>
> Key: OAK-3932
> URL: https://issues.apache.org/jira/browse/OAK-3932
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, mongomk, rdbmk
>Affects Versions: 1.0, 1.2
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.3.15
>
>
> In OAK-3559 [~tomek.rekawek] identified an issue with
> {{MongoDocumentStore.getIfCached()}}. The method may return
> NodeDocument.NULL, when it should actually return {{null}}. The contract of
> the method does not mention it explicitly, but returning {{null}} is
> consistent with other find methods and makes sense IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3932) DocumentStore.getIfCached() must not return NodeDocument.NULL
[
https://issues.apache.org/jira/browse/OAK-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcel Reutegger updated OAK-3932:
--
Component/s: rdbmk
RDBDocumentStore needs to be fixed as well.
> DocumentStore.getIfCached() must not return NodeDocument.NULL
> -
>
> Key: OAK-3932
> URL: https://issues.apache.org/jira/browse/OAK-3932
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, mongomk, rdbmk
>Affects Versions: 1.0, 1.2
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Labels: candidate_oak_1_0, candidate_oak_1_2
> Fix For: 1.4
>
>
> In OAK-3559 [~tomek.rekawek] identified an issue with
> {{MongoDocumentStore.getIfCached()}}. The method may return
> NodeDocument.NULL, when it should actually return {{null}}. The contract of
> the method does not mention it explicitly, but returning {{null}} is
> consistent with other find methods and makes sense IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-3933) potential improvements to membership storage
Julian Reschke created OAK-3933: --- Summary: potential improvements to membership storage Key: OAK-3933 URL: https://issues.apache.org/jira/browse/OAK-3933 Project: Jackrabbit Oak Issue Type: Improvement Components: security Reporter: Julian Reschke Membership information of a group currently is stored as an unsorted set of IDs, spread over multiple child nodes, using multivalued properties (of 1000 each). This content structure makes certain operations relatively slow: 1) Checking for declared membership When the authorizable to be checked is not a member, all child nodes need to be read and examined (in the other case, checking stops when a match is found). 2) Checking for inherited membership The membership IDs do not reveal the type of authorizable. In order to check inherited membership as well, the authorizable with the given ID needs to be read from storage in order to check the type. Below are a few ideas how this might be improved (however, the change of structure would require a mgiration step). 1) Avoid having to read all child nodes to check declared membership Assuming an alphanumeric ID structure, this could be achieved my modifying the structure like that: - as before, start with a single node - when a new member needs to be inserted and the candidate node is already full (has 1000 entries), create a new child node named after the first character of the authorizable ID - when this "level 1" member is full, start using "level 2" members and so on (assuming the ID structure is suitable for that, otherwise a different hash could be used) To check for membership, we wouldn't need to read *all* child nodes, but only those where the node name is a prefix match of the ID. 2) Avoid having to instantiate authorizables for declared membership checks - put limited type information into the stored IDs, such as "u" and "g" prefixes; that way the code could identify authorizables that are users and avoid having to instantiate them (this assumes that an ID that refers to a user will never refer to a group in the future) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3932) DocumentStore.getIfCached() must not return NodeDocument.NULL
Marcel Reutegger created OAK-3932:
-
Summary: DocumentStore.getIfCached() must not return
NodeDocument.NULL
Key: OAK-3932
URL: https://issues.apache.org/jira/browse/OAK-3932
Project: Jackrabbit Oak
Issue Type: Bug
Components: core, mongomk
Affects Versions: 1.2, 1.0
Reporter: Marcel Reutegger
Assignee: Marcel Reutegger
Priority: Minor
Fix For: 1.4
In OAK-3559 [~tomek.rekawek] identified an issue with
{{MongoDocumentStore.getIfCached()}}. The method may return NodeDocument.NULL,
when it should actually return {{null}}. The contract of the method does not
mention it explicitly, but returning {{null}} is consistent with other find
methods and makes sense IMO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-3559) Bulk document updates in MongoDocumentStore
[
https://issues.apache.org/jira/browse/OAK-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118971#comment-15118971
]
Marcel Reutegger commented on OAK-3559:
---
The patch also contains a fix unrelated to this issue. With your patch
{{MongoDocumentStore.getIfCached()}} checks if the document is
{{NodeDocument.NULL}}. I created a separate issue to fix it: OAK-3932.
> Bulk document updates in MongoDocumentStore
> ---
>
> Key: OAK-3559
> URL: https://issues.apache.org/jira/browse/OAK-3559
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Components: mongomk
>Reporter: Tomek Rękawek
>Assignee: Marcel Reutegger
> Fix For: 1.4
>
> Attachments: OAK-3559.patch
>
>
> Using the MongoDB [Bulk
> API|https://docs.mongodb.org/manual/reference/method/Bulk/#Bulk] implement
> the [batch version of createOrUpdate method|OAK-3662].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3931) Identify own repository id in shared datastore gc stats
[ https://issues.apache.org/jira/browse/OAK-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Jain resolved OAK-3931. Resolution: Fixed Committed with: * 1.2 - http://svn.apache.org/viewvc?rev=1726990&view=rev * trunk - http://svn.apache.org/viewvc?rev=1726981&view=rev > Identify own repository id in shared datastore gc stats > --- > > Key: OAK-3931 > URL: https://issues.apache.org/jira/browse/OAK-3931 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Minor > Fix For: 1.2.11, 1.3.15 > > > Shared gc stats (BlobGCMbean#getGlobalMarkStats) lists the different ids > available for different repositories sharing the datastore. It should also > help in identifying what is the repository id for the system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3931) Identify own repository id in shared datastore gc stats
[ https://issues.apache.org/jira/browse/OAK-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Jain updated OAK-3931: --- Description: Shared gc stats (BlobGCMbean#getGlobalMarkStats) lists the different ids available for different repositories sharing the datastore. It should also help in identifying what is the repository id for the system. > Identify own repository id in shared datastore gc stats > --- > > Key: OAK-3931 > URL: https://issues.apache.org/jira/browse/OAK-3931 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: blob >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Minor > Fix For: 1.2.11, 1.3.15 > > > Shared gc stats (BlobGCMbean#getGlobalMarkStats) lists the different ids > available for different repositories sharing the datastore. It should also > help in identifying what is the repository id for the system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3927) oak-run primary/standby should check segment version
[ https://issues.apache.org/jira/browse/OAK-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francesco Mari updated OAK-3927: Attachment: OAK-3854-01.patch [~mduerig], can you take a quick look at the patch? > oak-run primary/standby should check segment version > > > Key: OAK-3927 > URL: https://issues.apache.org/jira/browse/OAK-3927 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: run >Reporter: Michael Dürig >Assignee: Francesco Mari >Priority: Critical > Labels: resilience, tooling > Fix For: 1.4 > > Attachments: OAK-3854-01.patch > > > The primary and standby run modes should exit with an error if run on a store > with non matching segment version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-3931) Identify own repository id in shared datastore gc stats
Amit Jain created OAK-3931: -- Summary: Identify own repository id in shared datastore gc stats Key: OAK-3931 URL: https://issues.apache.org/jira/browse/OAK-3931 Project: Jackrabbit Oak Issue Type: Improvement Components: blob Reporter: Amit Jain Assignee: Amit Jain Priority: Minor Fix For: 1.2.11, 1.3.15 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-3921) DataStoreBlobStore - Limit resolveChunks only to non inlined blobs
[ https://issues.apache.org/jira/browse/OAK-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Jain resolved OAK-3921. Resolution: Fixed Merged into 1.2 branch with http://svn.apache.org/viewvc?rev=1726959&view=rev > DataStoreBlobStore - Limit resolveChunks only to non inlined blobs > -- > > Key: OAK-3921 > URL: https://issues.apache.org/jira/browse/OAK-3921 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Minor > Labels: technical_debt > Fix For: 1.2.11, 1.3.15 > > > Currently, DataStoreBlobStore#resolveChunks resolves all blob ids including > in-lined blobs. This is different from the AbstractBlobStore#resolveChunks > which removes in-lined blobs. > Due to this the InMemoryDataRecord had to be made public for OAK-3184 which > it should not have. A proper solution would be to have the resolveChunks only > return blobs stored in blob/data store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (OAK-3927) oak-run primary/standby should check segment version
[ https://issues.apache.org/jira/browse/OAK-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francesco Mari reassigned OAK-3927: --- Assignee: Francesco Mari > oak-run primary/standby should check segment version > > > Key: OAK-3927 > URL: https://issues.apache.org/jira/browse/OAK-3927 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: run >Reporter: Michael Dürig >Assignee: Francesco Mari >Priority: Critical > Labels: resilience, tooling > Fix For: 1.4 > > > The primary and standby run modes should exit with an error if run on a store > with non matching segment version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3184) Consistency checker for data/blob store
[ https://issues.apache.org/jira/browse/OAK-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Jain updated OAK-3184: --- Fix Version/s: 1.2.11 > Consistency checker for data/blob store > --- > > Key: OAK-3184 > URL: https://issues.apache.org/jira/browse/OAK-3184 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: blob >Reporter: Amit Jain >Assignee: Amit Jain > Labels: resilience, tooling > Fix For: 1.3.6, 1.2.11 > > Attachments: OAK-3184.patch > > > Consistency checker for data/blob store to report any missing blobs which can > result as a consequence of erroneous gc or extraneous factors. > * Will report any missing blob ids > * Path of nodes referring to those blobs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3184) Consistency checker for data/blob store
[ https://issues.apache.org/jira/browse/OAK-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Jain updated OAK-3184: --- Labels: resilience tooling (was: candidate_oak_1_2 resilience tooling) > Consistency checker for data/blob store > --- > > Key: OAK-3184 > URL: https://issues.apache.org/jira/browse/OAK-3184 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: blob >Reporter: Amit Jain >Assignee: Amit Jain > Labels: resilience, tooling > Fix For: 1.3.6 > > Attachments: OAK-3184.patch > > > Consistency checker for data/blob store to report any missing blobs which can > result as a consequence of erroneous gc or extraneous factors. > * Will report any missing blob ids > * Path of nodes referring to those blobs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3184) Consistency checker for data/blob store
[ https://issues.apache.org/jira/browse/OAK-3184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118847#comment-15118847 ] Amit Jain commented on OAK-3184: Merged into 1.2 branch with http://svn.apache.org/viewvc?rev=1726952&view=rev > Consistency checker for data/blob store > --- > > Key: OAK-3184 > URL: https://issues.apache.org/jira/browse/OAK-3184 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: blob >Reporter: Amit Jain >Assignee: Amit Jain > Labels: resilience, tooling > Fix For: 1.3.6 > > Attachments: OAK-3184.patch > > > Consistency checker for data/blob store to report any missing blobs which can > result as a consequence of erroneous gc or extraneous factors. > * Will report any missing blob ids > * Path of nodes referring to those blobs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
