Upgrade to Oak and Mandatory CommitHooks

2014-04-09 Thread Angela Schreiber
Hi Jukka We had an issue during upgrade testing from jackrabbit to oak that seemed to be caused by a missing commit hook (PermissionHook) as the access control content was present after the upgrade but the entries in the permission store were missing altogether. When thinking about it I realised

jackrabbit-oak build #4074: Errored

2014-04-09 Thread Travis CI
Build Update for apache/jackrabbit-oak - Build: #4074 Status: Errored Duration: 3002 seconds Commit: 7779482d572c2b89bbbc6c858f96f695df27796a (trunk) Author: Chetan Mehrotra Message: OAK-1702 - Create a benchmark for Full text search Adding support for running

Re: jackrabbit-oak build #4073: Errored

2014-04-09 Thread Chetan Mehrotra
> I'm sorry but your test run exceeded 50.0 minutes. Build failure is due to TimeOut Chetan Mehrotra On Thu, Apr 10, 2014 at 11:49 AM, Travis CI wrote: > Build Update for apache/jackrabbit-oak > - > > Build: #4073 > Status: Errored > > Duration: 3002 seconds

Re: Oak Training: MikroKernels/NodeStores

2014-04-09 Thread Ben Zahler
Thanks, Thomas, the term MongoDataStore was a mistake on my side (meant to write MongoDocumentStore). I will discuss the other points with Michael, who is responsible for the training on the Adobe side. Regards, Ben Zahler Inside Solutions AG | Felsenstrasse 11 | 4450 Sissach | Schweiz Telefon:

jackrabbit-oak build #4073: Errored

2014-04-09 Thread Travis CI
Build Update for apache/jackrabbit-oak - Build: #4073 Status: Errored Duration: 3002 seconds Commit: a653c0f168842a5d9b1de8072fdcc5f6d216ad12 (trunk) Author: Chetan Mehrotra Message: OAK-1716 - Enable passing of a execution context to runTest in multi threaded

How to get ContentRepository ?

2014-04-09 Thread Tobias Bocanegra
Hi, for [0] I need to "login" to the content repository - but I don't know how to acquire it. it seems that it is neither registered as OSGi service, nor in the global whiteboard (if there would be one). so basically I need to do: ContentSession session = contentRepository.login(...); Root r

Re: Session concurrency warning

2014-04-09 Thread Alexander Klimetschek
On 09.04.2014, at 07:15, Michael Dürig wrote: > On 9.4.14 4:12 , Rob Ryan wrote: >> It would be appropriate to change the warning to warn about >> concurrent reads as well. That would help highlight performance >> risks. > > There is such a warning actually. The discussion here was about the inc

Re: Login with userid that contains windows domain

2014-04-09 Thread Felix Meschberger
Am 09.04.2014 um 19:31 schrieb Tobias Bocanegra : > oh another use case: login with case-insensitive user id. > > this is similar in that respect, that the 'id' in the credentials used > to login, is not (or must not be) identical to the userid of the > resolved authorizable. > but the question

Re: Login with userid that contains windows domain

2014-04-09 Thread Tobias Bocanegra
oh another use case: login with case-insensitive user id. this is similar in that respect, that the 'id' in the credentials used to login, is not (or must not be) identical to the userid of the resolved authorizable. but the question is, where would this be configured? on all login modules? regar

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Chetan Mehrotra
Current update 1. Tommaso provided a patch (OAK-1702) to disable compression and that also helps quite a bit 2. Currently we are storing the full tokenized text in Lucene Index [1]. This would cause fetching of doc fields to be slower. On disabling the storage the number improve quite a bit. This

jackrabbit-oak build #4072: Fixed

2014-04-09 Thread Travis CI
Build Update for apache/jackrabbit-oak - Build: #4072 Status: Fixed Duration: 2821 seconds Commit: 2371ef73a4cddafe56c6bf076927bb1661148944 (trunk) Author: Alexandru Parvulescu Message: OAK-1702 Create a benchmark for Full text search - temporarily removed exp

Re: Session concurrency warning

2014-04-09 Thread Michael Dürig
On 9.4.14 4:12 , Rob Ryan wrote: It would be appropriate to change the warning to warn about concurrent reads as well. That would help highlight performance risks. There is such a warning actually. The discussion here was about the incorrect warning that is issued when writing concurrently.

RE: Session concurrency warning

2014-04-09 Thread Rob Ryan
It would be appropriate to change the warning to warn about concurrent reads as well. That would help highlight performance risks. -Rob -Original Message- From: Michael Dürig [mailto:mdue...@apache.org] Sent: Wednesday, April 09, 2014 12:27 AM To: oak-dev@jackrabbit.apache.org Subject:

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Alex Parvulescu
Aside from the compression issue, there was another one related to the 'order by' clause. I saw Collections.sort taking up as far as 23% of the perf. I removed the order by temporarily so it doesn't get in the way of the Lucene stuff, but I think the QueryEngine should skip ordering results in thi

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Tommaso Teofili
I'm looking into the Lucene codecs right now. Tommaso 2014-04-09 15:20 GMT+02:00 Alex Parvulescu : > Profiling the result shows that quite a bit of time goes in > org.apache.lucene.codecs.compressing.LZ4.decompress() (40%). This I > think is part of Lucene 4.x and not present in 3.x. Any idea i

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Alex Parvulescu
Profiling the result shows that quite a bit of time goes in org.apache.lucene.codecs.compressing.LZ4.decompress() (40%). This I think is part of Lucene 4.x and not present in 3.x. Any idea if I can disable compression? +1 I noticed that too, we should try to disable compression and compare results

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Chetan Mehrotra
On Wed, Apr 9, 2014 at 5:14 PM, Jukka Zitting wrote: > Is that a common use case? To better simulate a normal usage scenario > I'd make the benchmark fetch up to N results (where N is configurable, > with default something like 20) and access the path and the title > property of the matching nodes

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Thomas Mueller
> >also, I wonder if we shouldn't also profile the stack of underlying calls >in the QueryEngine to measure how much time is spent there and how much >time is spent in the specific QueryIndex implementation. Analyzing full thread dumps will give you the statistical distribution, which is quite acc

jackrabbit-oak build #4067: Fixed

2014-04-09 Thread Travis CI
Build Update for apache/jackrabbit-oak - Build: #4067 Status: Fixed Duration: 2903 seconds Commit: 43312db2233de87780e052e0cbf495905ced9e4c (trunk) Author: Angela Schreiber Message: remove unused import git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/o

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Tommaso Teofili
2014-04-09 13:44 GMT+02:00 Jukka Zitting : > Hi, > > On Wed, Apr 9, 2014 at 7:24 AM, Chetan Mehrotra > wrote: > > ... the testcase only fetches the first result. > > Is that a common use case? To better simulate a normal usage scenario > I'd make the benchmark fetch up to N results (where N is co

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Thomas Mueller
Hi, We have results from a different test case with multiple threads (internal id GRANITE-5572). We have 50 full thread dumps, and there I count: * 259 cases of LuceneIndex.java line 365: IndexReader reader = DirectoryReader.open(directory); * 43 cases of LuceneIndex.java line 379: TopDocs d

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Jukka Zitting
Hi, On Wed, Apr 9, 2014 at 7:24 AM, Chetan Mehrotra wrote: > ... the testcase only fetches the first result. Is that a common use case? To better simulate a normal usage scenario I'd make the benchmark fetch up to N results (where N is configurable, with default something like 20) and access the

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Chetan Mehrotra
On Wed, Apr 9, 2014 at 3:00 PM, Alex Parvulescu wrote: > - the patch assumes that there is and will be a single lucene index > directly under the root node, which may not necessarily be the case. I > agree this assumption holds now, but I would not introduce any changes that > take away this flex

jackrabbit-oak build #4065: Broken

2014-04-09 Thread Travis CI
Build Update for apache/jackrabbit-oak - Build: #4065 Status: Broken Duration: 623 seconds Commit: b2eac9fa2a921b5e4cc8e59be7b6530aeb82588b (trunk) Author: Michael Duerig Message: OAK-1698: Improve LargeOperationIT accuracy for document nodes store fixture Exc

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Alex Parvulescu
Hi, I agree with the idea to find a way to share the readers across threads. Looking at the proposed patch I see a few problems: - the patch assumes that there is and will be a single lucene index directly under the root node, which may not necessarily be the case. I agree this assumption holds

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Chetan Mehrotra
On Wed, Apr 9, 2014 at 12:25 PM, Marcel Reutegger wrote: >> Since the Lucene index is in any case updated asynchronously, it >> should be fine for us to ignore the base NodeState of the current >> session and instead use an IndexSearcher based on the last state as >> updated by the async indexer.

Re: Session concurrency warning

2014-04-09 Thread Michael Dürig
On 9.4.14 4:55 , Rob Ryan wrote: I can't comment to the exact meaning of the warning, but I can say the impact of reading from multiple threads with one session is a significant concern in itself given the current implementation of Oak. Under high throughput conditions the lock contention quic

jackrabbit-oak build #4063: Fixed

2014-04-09 Thread Travis CI
Build Update for apache/jackrabbit-oak - Build: #4063 Status: Fixed Duration: 2567 seconds Commit: 85f2435e65c94e1c362b3e5a5787c2eef0030a63 (trunk) Author: Chetan Mehrotra Message: OAK-1702 - Create a benchmark for Full text search -- Added two new fixtures Oa

Re: Session concurrency warning

2014-04-09 Thread Michael Dürig
On 9.4.14 3:20 , Alexander Klimetschek wrote: if I get this warning below, does "another thread is concurrently writing" mean that the other threads is using session WRITE methods? The message is actually misleading if not wrong. This warning is issued if *this* thread does a write operation

Re: Slow full text query performance and Lucene Index handling in Oak

2014-04-09 Thread Thomas Mueller
Hi, Do we still have the option to store the Lucene files in the file system? If we have, maybe we could run the test with that option and see if it improves performance? I'm not suggesting this is a solution, it's just one step to better analyze things. And it might be easy to do. Regards, Thoma

Re: Login with userid that contains windows domain

2014-04-09 Thread Felix Meschberger
Hi Am 09.04.2014 um 09:09 schrieb Tobias Bocanegra : >>> we could solve this transparently for all login modules that extend >>> from AbstractLoginModule with a general processCredentials() method >>> that extracts the userid and/or domain specifier. but I would favor a >>> more general credentia

Re: Login with userid that contains windows domain

2014-04-09 Thread Tobias Bocanegra
>> we could solve this transparently for all login modules that extend >> from AbstractLoginModule with a general processCredentials() method >> that extracts the userid and/or domain specifier. but I would favor a >> more general credentials -> userid mapping. for example to support the >> use cas