Re: how do you enable lucene search without OSGi or repository.xml?

2014-10-21 Thread Adrien Lamoureux
Thanks Alex, I tried your steps, and did a simple query test but nothing comes back: (this is a query I used before in jackrabbit 2.x without issue) Node searchPathNode = session.getNode('/searchpath'); Node testNode = searchPathNode.add('testnode'); testNode.setProperty('someproperty','teststri

Re: Reducing traversing log to DEBUG

2014-10-21 Thread Davide Giannella
On 21/10/2014 09:34, Thomas Mueller wrote: > Hi, > > Keeping it at "WARN" is very important. Otherwise, we don't have a way to > see missing indexes / bad queries. > > > I think setting it to 10'000 (instead of 1000) is probably a good idea. > Making it configurable, if you really want then OK, if

Re: Reindex and external indexes - Possibility of stale index data

2014-10-21 Thread Thomas Mueller
Hi, Yes, that's my point. I wouldn't use MVCC for reindexing the Lucene index. Reindexing is very costly, and I wouldn't do it in one huge, and possibly hours long transaction. * You need to have access to the old and (for readers) the new data (to re-create the index) * Eventually, you want to r

Re: Reindex and external indexes - Possibility of stale index data

2014-10-21 Thread Chetan Mehrotra
Thanks for the details Thomas! But above model varies from current model which make use of MVCC. The reindex operation triggers removal of :data node in branch and IndexReader always looks for :data node to open the directory on trunk. So while reindex is in progress existing readers make use of t

Re: CopyInReadDirectory and local filesystem path

2014-10-21 Thread Chetan Mehrotra
On Tue, Oct 21, 2014 at 1:44 PM, Thomas Mueller wrote: > What about using the SHA-2 hash code of the JCR path That can also be used. Only thing is that it would not be possible to map back from index dir to corresponding index defintion. But that can be managed via exposing the mapping via JMX C

Re: Reducing traversing log to DEBUG

2014-10-21 Thread Thomas Mueller
Hi, Keeping it at "WARN" is very important. Otherwise, we don't have a way to see missing indexes / bad queries. I think setting it to 10'000 (instead of 1000) is probably a good idea. Making it configurable, if you really want then OK, if the default is 10'000. Regards, Thomas On 21/10/14 1

Reducing traversing log to DEBUG

2014-10-21 Thread Davide Giannella
Hello folks, the WARN[0] tracked by the PathIterator in the ContentMirrorStorageStrategy is causing IMO more "harm" than good as it tracks a WARN in the logs every 1k nodes traversed. On normal repository an index is going to have more than 1000 nodes in it and therefore even if we have always mor

Re: Reindex and external indexes - Possibility of stale index data

2014-10-21 Thread Thomas Mueller
Hi, The node doesn't need to be moved, even after multiple reindex operations. Please note index creation is no different from reindex. In both cases, a new index data node is created. So, if an index definition is created: /oak:index/lucene Then the index is being built: /oak:index/luc

Re: CopyInReadDirectory and local filesystem path

2014-10-21 Thread Thomas Mueller
Hi, What about using the SHA-2 hash code of the JCR path? Regards, Thomas On 21/10/14 09:53, "Chetan Mehrotra" wrote: >For implementing CopyOnReadDirectory (OAK-1724) I need to copy the >Lucene index content to a local directory. However as there can be >multiple Lucene indexes and they may b

Re: Reindex and external indexes - Possibility of stale index data

2014-10-21 Thread Chetan Mehrotra
On Tue, Oct 21, 2014 at 1:18 PM, Thomas Mueller wrote: > What we need is a distinction between the old and the new index *data*. Yes and that can be done by storing the index creation time. In the approach you suggested where two different nodes are used and later the nodes are renamed allows th

CopyInReadDirectory and local filesystem path

2014-10-21 Thread Chetan Mehrotra
For implementing CopyOnReadDirectory (OAK-1724) I need to copy the Lucene index content to a local directory. However as there can be multiple Lucene indexes and they may be at non root location (OAK-2211) I need a way to map the JCR path of index to path on local filesystem. For e.g. if we have i

Re: Reindex and external indexes - Possibility of stale index data

2014-10-21 Thread Thomas Mueller
Hi, >It might be simpler if we just record the index creation time in the >index definition node itself (or some predefined meta node under >definition node). This can be done in IndexUpdate itself where it >would set the time when it triggers a reindex or the first index. Sorry I don't understan

Re: Reindex and external indexes - Possibility of stale index data

2014-10-21 Thread Chetan Mehrotra
It might be simpler if we just record the index creation time in the index definition node itself (or some predefined meta node under definition node). This can be done in IndexUpdate itself where it would set the time when it triggers a reindex or the first index. Later Lucene would make use of t

Re: Does Lucene modifies existing file in an index

2014-10-21 Thread Chetan Mehrotra
> The index indeed gets rebuilt. In IndexUpdate.collectIndexEditors() the provider Did not realized that. So it can be safely assumed that file created in Lucene index 1. Never get updated 2. Names never get reused That would simplify the logic for CopyOnRead a lot. Now only thing to take care o