[jira] [Created] (OAK-6513) Journal based Async Indexer
Chetan Mehrotra created OAK-6513:
Summary: Journal based Async Indexer
Key: OAK-6513
URL: https://issues.apache.org/jira/browse/OAK-6513
Project: Jackrabbit Oak
Issue Type: New Feature
Components: indexing
Reporter: Chetan Mehrotra
Assignee: Chetan Mehrotra
Fix For: 1.8
Current async indexer design is based on NodeState diff. This has served us
fine so far however off late it is not able to perform well if rate of
repository writes is high. When changes happen faster than index-update can
process them, larger and larger diffs will happen. These make index-updates
slower, which again lead to the next diff being ever larger than the one before
(assuming a constant ingestion rate).
In current diff based flow the indexer performs complete diff for all changes
happening between 2 cycle. It may happen that lots of writes happens but not
much indexable content is written. So doing diff there is a wasted effort.
In 1.6 release for NRT Indexing we implemented a journal based indexing for
external changes(OAK-4808, OAK-5430). That approach can be generalized and used
for async indexing.
Before talking about the journal based approach lets see how IndexEditor work
currently
h4. IndexEditor
Currently any IndexEditor performs 2 tasks
# Identify which node is to be indexed based on some index definition. The
Editor gets invoked as part of content diff where it determines which NodeState
is to be indexed
# Update the index based on node to be indexed
For e.g. in oak-lucene we have LuceneIndexEditor which identifies the
NodeStates to be indexed and LuceneDocumentMaker which constructs the Lucene
Document from NodeState to be indexed. For journal based approach we can
decouple these 2 parts and thus have
* IndexEditor - Identifies which all paths need to be indexed for given index
definition
* IndexUpdater - Updates the index based on given NodeState and its path
h4. High Level Flow
# Session Commit Flow
## Each index type would provide a IndexEditor which would be invoked as part
of commit (like sync indexes). These IndexEditor would just determine which
paths needs to be indexed.
## As part of commit the paths to be indexed would be written to journal.
# AsyncIndexUpdate flow
## AsyncIndexUpdate would query this journal to fetch all such indexed paths
between the 2 checkpoints
## Based on the index path data it would invoke the {{IndexUpdater}} to update
the index for that path
## Merge the index updates
h4. Benefits
Such a design would have following impact
# More work done as part of write
# Marking of indexable content is distributed hence at indexing time lesser
work to be done
# Indexing can progress in batches
# The indexers can be called in parallel
h4. Journal Implementation
DocumentNodeStore currently has an in built journal which is being used for NRT
Indexing. That feature can be exposed as an api.
For scaling index this design is mostly required for cluster case. So we can
possibly have both indexing support implemented and use the journal based
support for DocumentNodeStore setups. Or we can look into implementing such a
journal for SegmentNodeStore setups also
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-5192) Reduce Lucene related growth of repository size
[ https://issues.apache.org/jira/browse/OAK-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112358#comment-16112358 ] Thomas Mueller commented on OAK-5192: - The plan is to make the merge policy configurable, [~teofili] should I create a new issue for this? Or re-use this issue? > Reduce Lucene related growth of repository size > --- > > Key: OAK-5192 > URL: https://issues.apache.org/jira/browse/OAK-5192 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene, segment-tar >Reporter: Michael Dürig >Assignee: Tommaso Teofili > Labels: perfomance, scalability > Fix For: 1.8, 1.7.8 > > Attachments: added-bytes-zoom.png, binSize100.txt, binSize16384.txt, > binSizeTotal.txt, diff.txt.zip, nonBinSizeTotal.txt, OAK-5192.0.patch, Screen > Shot 2017-07-03 at 16.50.00.png > > > I observed Lucene indexing contributing to up to 99% of repository growth. > While the size of the index itself is well inside reasonable bounds, the > overall turnover of data being written and removed again can be as much as > 99%. > In the case of the TarMK this negatively impacts overall system performance > due to fast growing number of tar files / segments, bad locality of > reference, cache misses/thrashing when looking up segments and vastly > prolonged garbage collection cycles. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-6514) Make Lucene merge policy configurable
Thomas Mueller created OAK-6514: --- Summary: Make Lucene merge policy configurable Key: OAK-6514 URL: https://issues.apache.org/jira/browse/OAK-6514 Project: Jackrabbit Oak Issue Type: New Feature Components: lucene, query Reporter: Thomas Mueller Assignee: Tommaso Teofili Fix For: 1.8 To test OAK-5192, it would be good if the merge policy (how often to merge) is configurable. Depending on test results, we might want to change the default settings. ([~teofili] feel free to update the issue description) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-6515) Decouple indexing and upload to datastore
Thomas Mueller created OAK-6515: --- Summary: Decouple indexing and upload to datastore Key: OAK-6515 URL: https://issues.apache.org/jira/browse/OAK-6515 Project: Jackrabbit Oak Issue Type: New Feature Components: indexing, lucene, query Reporter: Thomas Mueller Assignee: Thomas Mueller Priority: Minor Fix For: 1.9.0 Currently the default async index delay is 5 seconds. Using a larger delay (e.g. 15 seconds) reduces index related growth, however diffing is delayed 15 seconds, which can reduce indexing performance. One option (which might require bigger changes) is to index every 5 seconds, and store the index every 5 seconds in the local directory, but only write to the datastore / nodestore every 3rd time (that is, every 15 seconds). So that other cluster nodes will only see the index update every 15 seconds. The diffing is done every 5 seconds, and the local index could be used every 5 or every 15 seconds. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-6516) Consider reordering the index-update-hook to reside with other post-commit-hooks
angela created OAK-6516:
---
Summary: Consider reordering the index-update-hook to reside with
other post-commit-hooks
Key: OAK-6516
URL: https://issues.apache.org/jira/browse/OAK-6516
Project: Jackrabbit Oak
Issue Type: Task
Components: core, indexing
Reporter: angela
Currently the index-update hook is located together with other commit
validators and is followed by the security related validators that are plugged
to the final commit hook upon each call to {{Root.commit}} to allow for proper
pluggability of security modules.
Consequently a given commit may fail with some security related validation only
after the index updated has been proceeded already.
I would like to discuss the possibility to treat the index-update like a
post-commit-hook that is only executed after all validation has been performed.
In the security area this is achieved by collecting the validator separately
from the post-commit-hooks.
[~stillalex], [~tmueller], I would appreciate if we could discuss this topic as
well as possible implications/regressions that might be associated with it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6516) Consider reordering the index-update-hook to reside with other post-commit-hooks
[
https://issues.apache.org/jira/browse/OAK-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112407#comment-16112407
]
Chetan Mehrotra commented on OAK-6516:
--
Current list of hooks in a typical setup
{noformat}
- CompositeHook
- ResetCommitAttributeHook
- CompositeHook
- VersionHook
- ConflictHook
- EditorHook
- ItemSaveValidatorProvider
- NameValidatorProvider
- NamespaceEditorProvider
- TypeEditorProvider
- ConflictValidatorProvider
- ChangeCollectorProvider
- RepoStateCheckHook
- EditorHook
- IndexUpdateProvider
- ReferenceEditorProvider
- PropertyIndexEditorProvider
- NodeCounterEditorProvider
- OrderedPropertyIndexEditorProvider
- VersionablePathHook
- EditorHook
- PermissionStoreValidatorProvider
- PermissionValidatorProvider
- AccessControlValidatorProvider
- PrivilegeValidatorProvider
- UserValidatorProvider
- CacheValidatorProvider
- TokenValidatorProvider
- PermissionHook
- JcrAllCommitHook
{noformat}
> Consider reordering the index-update-hook to reside with other
> post-commit-hooks
>
>
> Key: OAK-6516
> URL: https://issues.apache.org/jira/browse/OAK-6516
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: core, indexing
>Reporter: angela
>
> Currently the index-update hook is located together with other commit
> validators and is followed by the security related validators that are
> plugged to the final commit hook upon each call to {{Root.commit}} to allow
> for proper pluggability of security modules.
> Consequently a given commit may fail with some security related validation
> only after the index updated has been proceeded already.
> I would like to discuss the possibility to treat the index-update like a
> post-commit-hook that is only executed after all validation has been
> performed. In the security area this is achieved by collecting the validator
> separately from the post-commit-hooks.
> [~stillalex], [~tmueller], I would appreciate if we could discuss this topic
> as well as possible implications/regressions that might be associated with it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-3269) Improve indexing resilience
[ https://issues.apache.org/jira/browse/OAK-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller updated OAK-3269: Epic Name: indexer resilience 1.6 (was: indexer resilience) > Improve indexing resilience > --- > > Key: OAK-3269 > URL: https://issues.apache.org/jira/browse/OAK-3269 > Project: Jackrabbit Oak > Issue Type: Epic > Components: lucene >Reporter: Michael Marth >Assignee: Chetan Mehrotra >Priority: Critical > Labels: resilience > Fix For: 1.6.0 > > > As discussed bilaterally grouping the improvements for indexer resilience in > this issue for easier tracking -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6516) Consider reordering the index-update-hook to reside with other post-commit-hooks
[
https://issues.apache.org/jira/browse/OAK-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112419#comment-16112419
]
angela commented on OAK-6516:
-
[~chetanm], please note that your list only applies for typical
Oak-out-of-the-box setup scenarios. For Adobe AEM the validators and commit
hooks provided by {{oak-authorization-cug}} and {{oak-auth-external}} also need
to be listed!
>From the list above following hooks are provided by the different security
>modules and may differ depending on the actual security setup of a given oak
>repository:
retrieved from {{SecurityConfiguration.getValidators}} are
{code}
- PermissionStoreValidatorProvider
- PermissionValidatorProvider
- AccessControlValidatorProvider
- PrivilegeValidatorProvider
- UserValidatorProvider
- CacheValidatorProvider
- TokenValidatorProvider
and additionally
- CugValidatorProvider
- ExternalIdentityValidatorProvider
{code}
retrieved from {{SecurityConfiguration.getCommitHooks}} are:
{code}
- VersionablePathHook
{code}
and the following {{PostValidationHook}}
{code}
- PermissionHook
- JcrAllCommitHook
and additionally
- NestedCugHook
{code}
> Consider reordering the index-update-hook to reside with other
> post-commit-hooks
>
>
> Key: OAK-6516
> URL: https://issues.apache.org/jira/browse/OAK-6516
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: core, indexing
>Reporter: angela
>
> Currently the index-update hook is located together with other commit
> validators and is followed by the security related validators that are
> plugged to the final commit hook upon each call to {{Root.commit}} to allow
> for proper pluggability of security modules.
> Consequently a given commit may fail with some security related validation
> only after the index updated has been proceeded already.
> I would like to discuss the possibility to treat the index-update like a
> post-commit-hook that is only executed after all validation has been
> performed. In the security area this is achieved by collecting the validator
> separately from the post-commit-hooks.
> [~stillalex], [~tmueller], I would appreciate if we could discuss this topic
> as well as possible implications/regressions that might be associated with it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (OAK-5192) Reduce Lucene related growth of repository size
[ https://issues.apache.org/jira/browse/OAK-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112358#comment-16112358 ] Thomas Mueller edited comment on OAK-5192 at 8/3/17 8:44 AM: - The plan is to make the merge policy configurable, [~teofili] should I create a new issue for this? Or re-use this issue? Update: never mind, I created OAK-6514. was (Author: tmueller): The plan is to make the merge policy configurable, [~teofili] should I create a new issue for this? Or re-use this issue? > Reduce Lucene related growth of repository size > --- > > Key: OAK-5192 > URL: https://issues.apache.org/jira/browse/OAK-5192 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene, segment-tar >Reporter: Michael Dürig >Assignee: Tommaso Teofili > Labels: perfomance, scalability > Fix For: 1.8, 1.7.8 > > Attachments: added-bytes-zoom.png, binSize100.txt, binSize16384.txt, > binSizeTotal.txt, diff.txt.zip, nonBinSizeTotal.txt, OAK-5192.0.patch, Screen > Shot 2017-07-03 at 16.50.00.png > > > I observed Lucene indexing contributing to up to 99% of repository growth. > While the size of the index itself is well inside reasonable bounds, the > overall turnover of data being written and removed again can be as much as > 99%. > In the case of the TarMK this negatively impacts overall system performance > due to fast growing number of tar files / segments, bad locality of > reference, cache misses/thrashing when looking up segments and vastly > prolonged garbage collection cycles. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-6514) Make Lucene merge policy configurable
[ https://issues.apache.org/jira/browse/OAK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller updated OAK-6514: Sprint: L8 > Make Lucene merge policy configurable > - > > Key: OAK-6514 > URL: https://issues.apache.org/jira/browse/OAK-6514 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: lucene, query >Reporter: Thomas Mueller >Assignee: Tommaso Teofili > Fix For: 1.8 > > > To test OAK-5192, it would be good if the merge policy (how often to merge) > is configurable. > Depending on test results, we might want to change the default settings. > ([~teofili] feel free to update the issue description) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (OAK-6516) Consider reordering the index-update-hook to reside with other post-commit-hooks
[
https://issues.apache.org/jira/browse/OAK-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angela reassigned OAK-6516:
---
Assignee: angela
> Consider reordering the index-update-hook to reside with other
> post-commit-hooks
>
>
> Key: OAK-6516
> URL: https://issues.apache.org/jira/browse/OAK-6516
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: core, indexing
>Reporter: angela
>Assignee: angela
>
> Currently the index-update hook is located together with other commit
> validators and is followed by the security related validators that are
> plugged to the final commit hook upon each call to {{Root.commit}} to allow
> for proper pluggability of security modules.
> Consequently a given commit may fail with some security related validation
> only after the index updated has been proceeded already.
> I would like to discuss the possibility to treat the index-update like a
> post-commit-hook that is only executed after all validation has been
> performed. In the security area this is achieved by collecting the validator
> separately from the post-commit-hooks.
> [~stillalex], [~tmueller], I would appreciate if we could discuss this topic
> as well as possible implications/regressions that might be associated with it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6517) ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection failing intermittently
Vikas Saurabh created OAK-6517:
--
Summary:
ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection failing
intermittently
Key: OAK-6517
URL: https://issues.apache.org/jira/browse/OAK-6517
Project: Jackrabbit Oak
Issue Type: Test
Components: lucene
Affects Versions: 1.7.1
Reporter: Vikas Saurabh
Assignee: Vikas Saurabh
Priority: Minor
[~chetanm] reported offline that
{{ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection}} is
failing for him intermittently.
{noformat}
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.746 sec <<<
FAILURE! - in
org.apache.jackrabbit.oak.plugins.index.lucene.directory.ActiveDeletedBlobCollectionIT
simpleAsyncIndexUpdateBasedBlobCollection[WITH_FDS](org.apache.jackrabbit.oak.plugins.index.lucene.directory.ActiveDeletedBlobCollectionIT)
Time elapsed: 2.301 sec <<< FAILURE!
java.lang.AssertionError: First GC should delete some chunks
at
org.apache.jackrabbit.oak.plugins.index.lucene.directory.ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection(ActiveDeletedBlobCollectionIT.java:227)
Results :
Failed tests:
ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection:227
First GC should delete some chunks
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6517) ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection failing intermittently
[
https://issues.apache.org/jira/browse/OAK-6517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112574#comment-16112574
]
Vikas Saurabh commented on OAK-6517:
[~chetanm], I couldn't reproduce the issue. Can you please get some more data:
* temporarily log values of {{firstCommitNumChunks}}, {{initialNumChunks}}, etc
in/before asserts of that test
* DEBUG for
{{org.apache.jackrabbit.oak.plugins.index.lucene.directory.ActiveDeletedBlobCollectorFactory}}
> ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection
> failing intermittently
> --
>
> Key: OAK-6517
> URL: https://issues.apache.org/jira/browse/OAK-6517
> Project: Jackrabbit Oak
> Issue Type: Test
> Components: lucene
>Affects Versions: 1.7.1
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
>
> [~chetanm] reported offline that
> {{ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection}}
> is failing for him intermittently.
> {noformat}
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.746 sec <<<
> FAILURE! - in
> org.apache.jackrabbit.oak.plugins.index.lucene.directory.ActiveDeletedBlobCollectionIT
> simpleAsyncIndexUpdateBasedBlobCollection[WITH_FDS](org.apache.jackrabbit.oak.plugins.index.lucene.directory.ActiveDeletedBlobCollectionIT)
> Time elapsed: 2.301 sec <<< FAILURE!
> java.lang.AssertionError: First GC should delete some chunks
>at
> org.apache.jackrabbit.oak.plugins.index.lucene.directory.ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection(ActiveDeletedBlobCollectionIT.java:227)
> Results :
> Failed tests:
> ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection:227
> First GC should delete some chunks
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6517) ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection failing intermittently
[
https://issues.apache.org/jira/browse/OAK-6517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh updated OAK-6517:
---
Fix Version/s: 1.8
> ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection
> failing intermittently
> --
>
> Key: OAK-6517
> URL: https://issues.apache.org/jira/browse/OAK-6517
> Project: Jackrabbit Oak
> Issue Type: Test
> Components: lucene
>Affects Versions: 1.7.1
>Reporter: Vikas Saurabh
>Assignee: Vikas Saurabh
>Priority: Minor
> Fix For: 1.8
>
>
> [~chetanm] reported offline that
> {{ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection}}
> is failing for him intermittently.
> {noformat}
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.746 sec <<<
> FAILURE! - in
> org.apache.jackrabbit.oak.plugins.index.lucene.directory.ActiveDeletedBlobCollectionIT
> simpleAsyncIndexUpdateBasedBlobCollection[WITH_FDS](org.apache.jackrabbit.oak.plugins.index.lucene.directory.ActiveDeletedBlobCollectionIT)
> Time elapsed: 2.301 sec <<< FAILURE!
> java.lang.AssertionError: First GC should delete some chunks
>at
> org.apache.jackrabbit.oak.plugins.index.lucene.directory.ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection(ActiveDeletedBlobCollectionIT.java:227)
> Results :
> Failed tests:
> ActiveDeletedBlobCollectionIT.simpleAsyncIndexUpdateBasedBlobCollection:227
> First GC should delete some chunks
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (OAK-6498) Update Oak trunk to Jackrabbit 2.15.5
[ https://issues.apache.org/jira/browse/OAK-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke resolved OAK-6498. - Resolution: Fixed Fix Version/s: 1.7.6 trunk: [r1803985|http://svn.apache.org/r1803985] > Update Oak trunk to Jackrabbit 2.15.5 > - > > Key: OAK-6498 > URL: https://issues.apache.org/jira/browse/OAK-6498 > Project: Jackrabbit Oak > Issue Type: Task > Components: parent >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Fix For: 1.7.6 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (OAK-6518) Implement transparent usage of different TAR index formats
Francesco Mari created OAK-6518: --- Summary: Implement transparent usage of different TAR index formats Key: OAK-6518 URL: https://issues.apache.org/jira/browse/OAK-6518 Project: Jackrabbit Oak Issue Type: Improvement Components: segment-tar Reporter: Francesco Mari Assignee: Francesco Mari Fix For: 1.8, 1.7.6 The segment store should be able to read and validate two different format of indexes, the one in use before the introduction of the tail generation number and the one after. The detection of the different index types should be fully transparent and automatic. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6518) Implement transparent usage of different TAR index formats
[ https://issues.apache.org/jira/browse/OAK-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112671#comment-16112671 ] Francesco Mari commented on OAK-6518: - The logic for index load and validation was extracted in a different class and package at r1803995. > Implement transparent usage of different TAR index formats > -- > > Key: OAK-6518 > URL: https://issues.apache.org/jira/browse/OAK-6518 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-tar >Reporter: Francesco Mari >Assignee: Francesco Mari > Fix For: 1.8, 1.7.6 > > > The segment store should be able to read and validate two different format of > indexes, the one in use before the introduction of the tail generation number > and the one after. The detection of the different index types should be fully > transparent and automatic. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OAK-6514) Make Lucene merge policy configurable
[ https://issues.apache.org/jira/browse/OAK-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Mueller updated OAK-6514: Sprint: L9 (was: L8) > Make Lucene merge policy configurable > - > > Key: OAK-6514 > URL: https://issues.apache.org/jira/browse/OAK-6514 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: lucene, query >Reporter: Thomas Mueller >Assignee: Tommaso Teofili > Fix For: 1.8 > > > To test OAK-5192, it would be good if the merge policy (how often to merge) > is configurable. > Depending on test results, we might want to change the default settings. > ([~teofili] feel free to update the issue description) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6511) Switch to official OSGi versioning annotations
[
https://issues.apache.org/jira/browse/OAK-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112713#comment-16112713
]
Robert Munteanu commented on OAK-6511:
--
I filed a bnd issue at https://github.com/bndtools/bnd/issues/2112 and also
posted on users@felix at
https://lists.apache.org/thread.html/f35aeeb2e6424e9a17312d9864ec568405f719dae035f46fa7e7b71b@%3Cusers.felix.apache.org%3E
.
I'll resume work once I have a definite answer - either by increasing the micro
version or by fixing the baselining in another manner.
> Switch to official OSGi versioning annotations
> --
>
> Key: OAK-6511
> URL: https://issues.apache.org/jira/browse/OAK-6511
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: parent
>Reporter: Robert Munteanu
>Assignee: Robert Munteanu
> Fix For: 1.8, 1.7.6
>
>
> When building Oak, several warnings are printed related to the usage of the
> bnd versioning annotations:
> {noformat}[WARNING] Manifest
> org.apache.jackrabbit:oak-auth-external:bundle:1.8-SNAPSHOT :
> aQute.bnd.annotation.Version annotation used in class
> org.apache.jackrabbit.oak.spi.security.authentication.external.package-info.
> Bnd versioning annotations are deprecated as of Bnd 3.2 and support will be
> removed in Bnd 4.0. Please change to use OSGi versioning
> annotations.{noformat}
> Since it's a mostly mechanical change with a limited number of scenarios I
> propose to just do it outright.
> {noformat}$ git grep aQute.bnd.annotation | wc -l
> 85{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6495) Fallback to classic diff when journal is broken
[ https://issues.apache.org/jira/browse/OAK-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6495. - Bulk close for 1.7.5 > Fallback to classic diff when journal is broken > --- > > Key: OAK-6495 > URL: https://issues.apache.org/jira/browse/OAK-6495 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Labels: candidate_oak_1_0, candidate_oak_1_2 > Fix For: 1.8, 1.7.5, 1.6.4, 1.4.18 > > > The DocumentNodeStore diff mechanism should fallback to the classic > implementation when the journal is broken. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-3349) Partial compaction
[ https://issues.apache.org/jira/browse/OAK-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-3349. - Bulk close for 1.7.5 > Partial compaction > -- > > Key: OAK-3349 > URL: https://issues.apache.org/jira/browse/OAK-3349 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Michael Dürig >Assignee: Francesco Mari > Labels: compaction, gc, scalability > Fix For: 1.8, 1.7.5 > > Attachments: compaction-time.png, cycle-count.png, post-gc-size.png > > > On big repositories compaction can take quite a while to run as it needs to > create a full deep copy of the current root node state. For such cases it > could be beneficial if we could partially compact the repository thus > splitting full compaction over multiple cycles. > Partial compaction would run compaction on a sub-tree just like we now run it > on the full tree. Afterwards it would create a new root node state by > referencing the previous root node state replacing said sub-tree with the > compacted one. > Todo: Asses feasibility and impact, implement prototype. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6472) Expose constructor in JsonSerializer which takes filer and JsopBuilder
[ https://issues.apache.org/jira/browse/OAK-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6472. - Bulk close for 1.7.5 > Expose constructor in JsonSerializer which takes filer and JsopBuilder > -- > > Key: OAK-6472 > URL: https://issues.apache.org/jira/browse/OAK-6472 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: store-spi >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.8, 1.7.5 > > > Need a constructor in JsonSerializer which takes both filter and JsopBuilder. > Currently no constructor exists which takes both -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6461) Merge all security related validators into a single hook
[
https://issues.apache.org/jira/browse/OAK-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6461.
-
Bulk close for 1.7.5
> Merge all security related validators into a single hook
>
>
> Key: OAK-6461
> URL: https://issues.apache.org/jira/browse/OAK-6461
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, security
>Reporter: Alex Deparvu
>Assignee: Alex Deparvu
> Fix For: 1.7.5
>
> Attachments: OAK-6461.patch
>
>
> I'd like to see if it's feasible to merge all security related validators
> into a single hook, instead of a hook per _SecurityConfiguration_.
> Pros
> * all validators will be merged into a single hook, meaning processing will
> happen via a single diff over the content
> Cons
> * order of hooks will change, there will be commit hooks first, all
> aggregated validators next and post validation hooks last. I don't think
> there's any issue with validation itself as all data added by the hooks will
> be visible to the composite validator.
> This is how the chaining looks like in the current setup:
> {noformat}
> EditorHook :
> (TokenValidatorProvider),
> VersionablePathHook,
> EditorHook :
> (CompositeEditorProvider : ([
> PermissionStoreValidatorProvider,
> PermissionValidatorProvider,
> AccessControlValidatorProvider])),
> EditorHook :
> (PrivilegeValidatorProvider),
> EditorHook :
> (CompositeEditorProvider : ([
> UserValidatorProvider,
> CacheValidatorProvider])),
> PermissionHook,
> JcrAllCommitHook
> {noformat}
> If we merged them, this is the result:
> {noformat}
> VersionablePathHook,
> EditorHook :
> (CompositeEditorProvider : ([
> PermissionStoreValidatorProvider,
> PermissionValidatorProvider,
> AccessControlValidatorProvider,
> UserValidatorProvider,
> CacheValidatorProvider,
> PrivilegeValidatorProvider,
> TokenValidatorProvider])),
> PermissionHook,
> JcrAllCommitHook
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6456) Include the tail generation in the TAR index
[ https://issues.apache.org/jira/browse/OAK-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6456. - Bulk close for 1.7.5 > Include the tail generation in the TAR index > > > Key: OAK-6456 > URL: https://issues.apache.org/jira/browse/OAK-6456 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Francesco Mari >Assignee: Francesco Mari > Fix For: 1.8, 1.7.5 > > > OAK-3349 introduces the tail generation, an additional generation number > associated to each segment that is consulted during the cleanup phase to > determine if a segment should be removed. The tail generation is currently > only preset in the segments. For efficiency purposes, it should be added to > the TAR index instead, like the full generation number. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6471) Support adding or updating index definitions via oak-run
[ https://issues.apache.org/jira/browse/OAK-6471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6471. - Bulk close for 1.7.5 > Support adding or updating index definitions via oak-run > > > Key: OAK-6471 > URL: https://issues.apache.org/jira/browse/OAK-6471 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: indexing, run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.5 > > > Currently oak-run can be used to reindex an existing index present in > NodeStore. I would like to extend this feature set to support adding or > updating the index definition and then reindexing them. The end user story > would be like below > Development team updates or introduces a new index definition and gives it to > system admin. Then admin can use oak-run and give it reference to the index > definition and then tooling takes care of indexing (with read only mode) and > then importing the updated index definition back into the repository. > To start with the index definitions would be provided as a json file with a > single json file containing index definitions which needs to be reindexed. > The file format would be similar to one we used for dumping index definition > (OAK-6224). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6470) Remove extra call to createGroupPrincipal in UserPrincipalProvider
[
https://issues.apache.org/jira/browse/OAK-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6470.
-
Bulk close for 1.7.5
> Remove extra call to createGroupPrincipal in UserPrincipalProvider
> --
>
> Key: OAK-6470
> URL: https://issues.apache.org/jira/browse/OAK-6470
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, security
>Reporter: Alex Deparvu
>Assignee: Alex Deparvu
>Priority: Trivial
> Fix For: 1.7.5
>
>
> There's an extra call to {{createGroupPrincipal}} that could be removed [0],
> the group was already created.
> [0]
> https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/security/user/UserPrincipalProvider.java#L255
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6499) MultiplexingPermissionProvider wrong privileges composition
[ https://issues.apache.org/jira/browse/OAK-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6499. - Bulk close for 1.7.5 > MultiplexingPermissionProvider wrong privileges composition > --- > > Key: OAK-6499 > URL: https://issues.apache.org/jira/browse/OAK-6499 > Project: Jackrabbit Oak > Issue Type: Bug > Components: composite, core >Reporter: Alex Deparvu >Assignee: Alex Deparvu > Fix For: 1.7.5 > > > It seems the CompositePermissionProvider eagerly applies deny permissions to > the composed permission calculation which is good for the 'AND' version. In > the case of the OR composition, this should not happen. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6473) Use JsonSerializer instead of custom json logic in IndexDefinitionPrinter
[ https://issues.apache.org/jira/browse/OAK-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6473. - Bulk close for 1.7.5 > Use JsonSerializer instead of custom json logic in IndexDefinitionPrinter > - > > Key: OAK-6473 > URL: https://issues.apache.org/jira/browse/OAK-6473 > Project: Jackrabbit Oak > Issue Type: Task > Components: indexing >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.8, 1.7.5 > > > Currently IndexDefinitionPrinter uses custom NodeStateJsonUtils. This should > be replaced with JsonSerializer -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6475) JsonSerializer should not emit :childNodeCount property
[
https://issues.apache.org/jira/browse/OAK-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6475.
-
Bulk close for 1.7.5
> JsonSerializer should not emit :childNodeCount property
> ---
>
> Key: OAK-6475
> URL: https://issues.apache.org/jira/browse/OAK-6475
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: store-spi
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.8, 1.7.5
>
>
> If a custom filter is used with JsonSerializer where we need to emit the
> hidden properties but not hidden node like below then it also emits
> ":childNodeCount".
> {noformat}
> "{\"nodes\":[\"*\", \"-:*\"]}"
> {noformat}
> By default ":childNodeCount" is excluded in default filter hence not being
> added currently. This looks a left over of initial KernalNodeStore support
> which supported such properties but which are not used any more. So this
> should be removed
> {noformat}
> "{\"properties\":[\"*\", \"-:childNodeCount\"]}"
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6477) RevisionGC metrics for DocumentNodeStore
[ https://issues.apache.org/jira/browse/OAK-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6477. - Bulk close for 1.7.5 > RevisionGC metrics for DocumentNodeStore > > > Key: OAK-6477 > URL: https://issues.apache.org/jira/browse/OAK-6477 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: core, documentmk >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Fix For: 1.8, 1.7.5 > > > Running RevisionGC on a DocumentNodeStore currently returns statistics and > logs them at info level, but there is no easily accessible fine granular > information available while the GC is running. There are periodic log > messages while a GC is running, but those are not really machine readable. > The system should expose metrics that allow to monitor the progress and track > statistics over time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6484) Include tail generation in the GC journal
[ https://issues.apache.org/jira/browse/OAK-6484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6484. - Bulk close for 1.7.5 > Include tail generation in the GC journal > - > > Key: OAK-6484 > URL: https://issues.apache.org/jira/browse/OAK-6484 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Francesco Mari >Assignee: Francesco Mari > Fix For: 1.8, 1.7.5 > > > The GC journal should be able to read and write both the full and the tail > generation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6265) Remove Mounts.defaultMount methods
[
https://issues.apache.org/jira/browse/OAK-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6265.
-
Bulk close for 1.7.5
> Remove Mounts.defaultMount methods
> --
>
> Key: OAK-6265
> URL: https://issues.apache.org/jira/browse/OAK-6265
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: composite
>Reporter: Robert Munteanu
>Assignee: Robert Munteanu
> Fix For: 1.8, 1.7.5
>
> Attachments: 0001-OAK-6265-Remove-Mounts.defaultMount-methods.patch,
> 0002-OAK-6265-Remove-Mounts.defaultMount-methods.patch
>
>
> These methods break the abstraction of constructing a composite setup
> and should not be used. Since the exported package version of
> {{org.apache.jackrabbit.oak.spi.mount}} is set to 3.0.0 for 1.8-SNAPSHOT and
> it was 2.2.0 in 1.6.1 it's safe to remove these methods if we do it before a
> release from the 1.8 stream is cut.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6359) Change behavior for very complex queries
[
https://issues.apache.org/jira/browse/OAK-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6359.
-
Bulk close for 1.7.5
> Change behavior for very complex queries
>
>
> Key: OAK-6359
> URL: https://issues.apache.org/jira/browse/OAK-6359
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: query
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
>Priority: Critical
> Labels: candidate_oak_1_4, candidate_oak_1_6
> Fix For: 1.8, 1.7.5, 1.6.4, 1.4.18
>
>
> Very complex queries can cause take a long time to parse, and can possibly
> cause OOME. It would be good if processing queries automatically stops after
> some number of loops, after some time, or some amount of memory was used. And
> then throw an exception "query too complex".
> Example query:
> {noformat}
> //element(*,rep:Authorizable)[ ( (
> jcr:like(fn:lower-case(fn:name()), 'audio%')
> or jcr:contains(@rep:principalName,'audio*')
> or jcr:contains(@cq:first-name,'audio*')
> or jcr:contains(@cq:last-name,'audio*')
> or jcr:contains(profile/@givenName,'audio*')
> or jcr:contains(profile/@familyName,'audio*') )
> and ( jcr:like(fn:lower-case(fn:name()), 'conference%')
> or jcr:contains(@rep:principalName,'conference*')
> or jcr:contains(@cq:first-name,'conference*')
> or jcr:contains(@cq:last-name,'conference*')
> or jcr:contains(profile/@givenName,'conference*')
> or jcr:contains(profile/@familyName,'conference*') )
> and ( jcr:like(fn:lower-case(fn:name()), 'details:%')
> or jcr:contains(@rep:principalName,'details:*')
> or jcr:contains(@cq:first-name,'details:*')
> or jcr:contains(@cq:last-name,'details:*')
> or jcr:contains(profile/@givenName,'details:*')
> or jcr:contains(profile/@familyName,'details:*') )
> and ( jcr:like(fn:lower-case(fn:name()), 'conference%')
> or jcr:contains(@rep:principalName,'conference*')
> or jcr:contains(@cq:first-name,'conference*')
> or jcr:contains(@cq:last-name,'conference*')
> or jcr:contains(profile/@givenName,'conference*')
> or jcr:contains(profile/@familyName,'conference*') )
> and ( jcr:like(fn:lower-case(fn:name()), 'code:%')
> or jcr:contains(@rep:principalName,'code:*')
> or jcr:contains(@cq:first-name,'code:*')
> or jcr:contains(@cq:last-name,'code:*')
> or jcr:contains(profile/@givenName,'code:*')
> or jcr:contains(profile/@familyName,'code:*') )
> and ( jcr:like(fn:lower-case(fn:name()), '123456%')
> or jcr:contains(@rep:principalName,'123456*')
> or jcr:contains(@cq:first-name,'123456*')
> or jcr:contains(@cq:last-name,'123456*')
> or jcr:contains(profile/@givenName,'123456*')
> or jcr:contains(profile/@familyName,'123456*') )
> and ( jcr:like(fn:lower-case(fn:name()), 'further%')
> or jcr:contains(@rep:principalName,'further*')
> or jcr:contains(@cq:first-name,'further*')
> or jcr:contains(@cq:last-name,'further*')
> or jcr:contains(profile/@givenName,'further*')
> or jcr:contains(profile/@familyName,'further*') )
> and ( jcr:like(fn:lower-case(fn:name()), 'dial%')
> or jcr:contains(@rep:principalName,'dial*')
> or jcr:contains(@cq:first-name,'dial*')
> or jcr:contains(@cq:last-name,'dial*')
> or jcr:contains(profile/@givenName,'dial*')
> or jcr:contains(profile/@familyName,'dial*') )
> and ( jcr:like(fn:lower-case(fn:name()), 'in%')
> or jcr:contains(@rep:principalName,'in*')
> or jcr:contains(@cq:first-name,'in*')
> or jcr:contains(@cq:last-name,'in*')
> or jcr:contains(profile/@givenName,'in*')
> or jcr:contains(profile/@familyName,'in*') )
> and ( jcr:like(fn:lower-case(fn:name()), 'numbers:%')
> or jcr:contains(@rep:principalName,'numbers:*')
> or jcr:contains(@cq:first-name,'numbers:*')
> or jcr:contains(@cq:last-name,'numbers:*')
> or jcr:contains(profile/@givenName,'numbers:*')
> or jcr:contains(profile/@familyName,'numbers:*') )
> and ( jcr:like(fn:lower-case(fn:name()), 'http://wikipedia.org%')
> or jcr:contains(@rep:principalName,'http://wikipedia.org*')
> or jcr:contains(@cq:first-name,'http://wikipedia.org*')
> or jcr:contains(@cq:last-name,'http://wikipedia.org*')
> or jcr:contains(profile/@givenName,'http://wikipedia.org*')
> or jcr:contains(profile/@familyName,'http://wikipedia.org*') )
> and ( jcr:like(fn:lower-case(fn:name()), 'conference%')
> or jcr:contains(@rep:principalName,'conference*')
> or jcr:contains(@cq:first-name,'conference*')
> or jcr:contains(@cq:last-name,'conference*')
> or jcr:contains(profile/@givenName,'conference*')
> or jcr:contains(profile/@familyName,'conference*') )
> and ( jcr:like(fn:lower-case(fn:name()), 'number(s):%')
> or jcr:contains(@rep:principalName,'number(s):*')
> or jcr:contains(@cq:first-name,'number(s):*')
> or jcr:contains(@cq:last-name,'number(s):*')
> or jcr:contains(profile/@given
[jira] [Closed] (OAK-6469) CompositePermissionProvider should implement AggregatedPermissionProvider
[
https://issues.apache.org/jira/browse/OAK-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6469.
-
Bulk close for 1.7.5
> CompositePermissionProvider should implement AggregatedPermissionProvider
> -
>
> Key: OAK-6469
> URL: https://issues.apache.org/jira/browse/OAK-6469
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, security
>Reporter: Alex Deparvu
>Assignee: Alex Deparvu
> Fix For: 1.7.5
>
> Attachments: OAK-6469.patch, OAK-6469-v2.patch
>
>
> A more in-depth followup of OAK-6451. It turns out you can't mix CUG setup
> with the Multiplexing setup because the {{CompositePermissionProvider}}
> cannot contain another composite.
> Making {{CompositePermissionProvider}} implement
> {{AggregatedPermissionProvider}} opens the door for composites of composites.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6370) Improve documentation for text pre-extraction
[ https://issues.apache.org/jira/browse/OAK-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6370. - Bulk close for 1.7.5 > Improve documentation for text pre-extraction > - > > Key: OAK-6370 > URL: https://issues.apache.org/jira/browse/OAK-6370 > Project: Jackrabbit Oak > Issue Type: Documentation > Components: lucene, run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.5 > > > The docs for pre-extraction does not make things very clear. This should be > improved -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6490) Pre-Extraction support fails for empty binaries
[
https://issues.apache.org/jira/browse/OAK-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6490.
-
Bulk close for 1.7.5
> Pre-Extraction support fails for empty binaries
> ---
>
> Key: OAK-6490
> URL: https://issues.apache.org/jira/browse/OAK-6490
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.8, 1.7.5
>
>
> If text pre-extraction is configured then for empty blobs following exception
> is seen
> {noformat}
> 2017-07-19 11:14:49,584 ERROR [main] o.a.j.oak.index.IndexCommand - Error
> occurred while performing index tasks
> java.lang.StringIndexOutOfBoundsException: String index out of range: 4
> at java.lang.String.substring(String.java:1963) ~[na:1.8.0_102]
> at
> org.apache.jackrabbit.oak.plugins.index.datastore.DataStoreTextWriter.getFile(DataStoreTextWriter.java:187)
> ~[oak-run-1.8-SNAPSHOT.jar:1.8-SNAPSHOT]
> at
> org.apache.jackrabbit.oak.plugins.index.datastore.DataStoreTextWriter.getText(DataStoreTextWriter.java:104)
> ~[oak-run-1.8-SNAPSHOT.jar:1.8-SNAPSHOT]
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.ExtractedTextCache.get(ExtractedTextCache.java:90)
> ~[oak-run-1.8-SNAPSHOT.jar:1.8-SNAPSHOT]
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.binary.BinaryTextExtractor.parseStringValue(BinaryTextExtractor.java:127)
> ~[oak-run-1.8-SNAPSHOT.jar:1.8-SNAPSHOT]
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.binary.BinaryTextExtractor.newBinary(BinaryTextExtractor.java:112)
> ~[oak-run-1.8-SNAPSHOT.jar:1.8-SNAPSHOT]
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneDocumentMaker.newBinary(LuceneDocumentMaker.java:398)
> ~[oak-run-1.8-SNAPSHOT.jar:1.8-SNAPSHOT]
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneDocumentMaker.indexProperty(LuceneDocumentMaker.java:257)
> ~[oak-run-1.8-SNAPSHOT.jar:1.8-SNAPSHOT]
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneDocumentMaker.makeDocument(LuceneDocumentMaker.java:128)
> ~[oak-run-1.8-SNAPSHOT.jar:1.8-SNAPSHOT]
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.makeDocument(LuceneIndexEditor.java:268)
> ~[oak-run-1.8-SNAPSHOT.jar:1.8-SNAPSHOT]
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.addOrUpdate(LuceneIndexEditor.java:244)
> ~[oak-run-1.8-SNAPSHOT.jar:1.8-SNAPSHOT]
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.leave(LuceneIndexEditor.java:140)
> ~[oak-run-1.8-SNAPSHOT.jar:1.8-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6482) Remove org.apache.sling.testing.osgi-mock dependency from oak-run
[
https://issues.apache.org/jira/browse/OAK-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6482.
-
Bulk close for 1.7.5
> Remove org.apache.sling.testing.osgi-mock dependency from oak-run
> -
>
> Key: OAK-6482
> URL: https://issues.apache.org/jira/browse/OAK-6482
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: run
>Reporter: Alex Deparvu
>Assignee: Alex Deparvu
>Priority: Trivial
> Fix For: 1.7.5
>
>
> Oak Run has a {{compile}} dependency on
> {{org.apache.sling.testing.osgi-mock}} which seems unneeded. I think we can
> remove it.
> [0] https://github.com/apache/jackrabbit-oak/blob/trunk/oak-run/pom.xml#L301
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6491) Add JCR_CREATED support to JcrLastModifiedConflictHandler
[
https://issues.apache.org/jira/browse/OAK-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6491.
-
Bulk close for 1.7.5
> Add JCR_CREATED support to JcrLastModifiedConflictHandler
> -
>
> Key: OAK-6491
> URL: https://issues.apache.org/jira/browse/OAK-6491
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: Alex Deparvu
>Assignee: Alex Deparvu
> Fix For: 1.7.5
>
>
> Currently the {{JcrLastModifiedConflictHandler}} only handles
> {{JCR_LASTMODIFIED}} updates by picking the newer of the 2 dates. I suggest
> to improve this by adding support for the {{JCR_CREATED}} concurrent updates,
> except the merge would pick the first of the 2 date properties.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-5602) Avoid missing journal entries
[ https://issues.apache.org/jira/browse/OAK-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-5602. - Bulk close for 1.7.5 > Avoid missing journal entries > - > > Key: OAK-5602 > URL: https://issues.apache.org/jira/browse/OAK-5602 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core >Affects Versions: 1.6.0 >Reporter: Stefan Egli >Assignee: Marcel Reutegger > Fix For: 1.8, 1.7.5 > > > As a follow up of OAK-5601: that one is about a situation where > backgroundRead falls way behind and journal GC cleans up journal entries > before it was able to read it. We should generally look into improving this > situation, eg by having journal GC not do the clean up as long as there are > instances that haven't read the entry. This could perhaps be achieved by > periodically having each instance inform the rest about what journal entries > it has processed (or simpler: how far back it would be -find- fine for > journal entries to be GC'ed) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6492) Allow DummyDataStore to read files written by it
[ https://issues.apache.org/jira/browse/OAK-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6492. - Bulk close for 1.7.5 > Allow DummyDataStore to read files written by it > > > Key: OAK-6492 > URL: https://issues.apache.org/jira/browse/OAK-6492 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.8, 1.7.5 > > > Currently DummyDataStore always returns an empty DataRecord. This is fine for > the case where binaries are not accessible. However for case where files are > written by this DummyDataStore itself it should be possible to return proper > DataRecord. > This would allow this DataStore to be used in simulating online indexing > where actual DataStore is not accessible but newly created index files need > to be written -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6502) Property index: include/exclude key pattern list (escaping)
[ https://issues.apache.org/jira/browse/OAK-6502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6502. - Bulk close for 1.7.5 > Property index: include/exclude key pattern list (escaping) > --- > > Key: OAK-6502 > URL: https://issues.apache.org/jira/browse/OAK-6502 > Project: Jackrabbit Oak > Issue Type: Bug > Components: property-index >Affects Versions: 1.7.2 >Reporter: Thomas Mueller >Assignee: Thomas Mueller > Fix For: 1.7.5 > > > There is a bug in OAK-4637: values with special characters (space, dash,...) > are escaped before testing for matches. So the index only works if the > include / exclude pattern is also escaped. This is a bug. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6478) Move LoggingGCMonitor from segment-tar to core-spi
[ https://issues.apache.org/jira/browse/OAK-6478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6478. - Bulk close for 1.7.5 > Move LoggingGCMonitor from segment-tar to core-spi > -- > > Key: OAK-6478 > URL: https://issues.apache.org/jira/browse/OAK-6478 > Project: Jackrabbit Oak > Issue Type: Task > Components: core-spi, segment-tar >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Fix For: 1.8, 1.7.5 > > Attachments: OAK-6478.patch > > > I need a LoggingGCMonitor for the DocumentNodeStore, which is already > available in oak-segment-tar. There is nothing segment specific to that class. > I suggest moving the class to oak-core-spi into the o.a.j.o.spi.gc package. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6485) Expose tail and full compaction via different methods
[
https://issues.apache.org/jira/browse/OAK-6485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6485.
-
Bulk close for 1.7.5
> Expose tail and full compaction via different methods
> -
>
> Key: OAK-6485
> URL: https://issues.apache.org/jira/browse/OAK-6485
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-tar
>Reporter: Francesco Mari
>Assignee: Francesco Mari
> Fix For: 1.8, 1.7.5
>
>
> There should be a way to invoke a full or a tail compaction programmatically.
> Moreover, it should be possible to invoke the two different types of
> compaction or, alternatively, to choose which compaction should be run by
> default, via the {{SegmentRevisionGC}} MBean.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6493) LuceneIndexProviderService.enableHybridIndexing=false results in NullPointerException
[
https://issues.apache.org/jira/browse/OAK-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6493.
-
Bulk close for 1.7.5
> LuceneIndexProviderService.enableHybridIndexing=false results in
> NullPointerException
> -
>
> Key: OAK-6493
> URL: https://issues.apache.org/jira/browse/OAK-6493
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.6.1
>Reporter: Stefan Egli
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.8, 1.7.5, 1.6.4
>
>
> Setting
> {{org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProviderService.enableHybridIndexing=false}}
> on oak 1.6.1 results in the following exception, preventing the repository
> to start up properly:
> {noformat}java.lang.NullPointerException: null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:192)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.LuceneDocumentHolder.(LuceneDocumentHolder.java:52)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorProvider.getDocumentHolder(LuceneIndexEditorProvider.java:190)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorProvider.getIndexEditor(LuceneIndexEditorProvider.java:139)
> at
> org.apache.jackrabbit.oak.plugins.index.CompositeIndexEditorProvider.getIndexEditor(CompositeIndexEditorProvider.java:74)
> at
> org.apache.jackrabbit.oak.spi.whiteboard.WhiteboardIndexEditorProvider.getIndexEditor(WhiteboardIndexEditorProvider.java:52)
> at
> org.apache.jackrabbit.oak.plugins.index.IndexUpdate.collectIndexEditors(IndexUpdate.java:240)
> at
> org.apache.jackrabbit.oak.plugins.index.IndexUpdate.enter(IndexUpdate.java:160)
> at
> org.apache.jackrabbit.oak.spi.commit.VisibleEditor.enter(VisibleEditor.java:57)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:49)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorHook.processCommit(EditorHook.java:54)
> at
> org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:603)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:634)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:690)
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeStore.merge(SegmentNodeStore.java:334)
> at
> org.apache.jackrabbit.oak.spi.lifecycle.OakInitializer.initialize(OakInitializer.java:52)
> at
> org.apache.jackrabbit.oak.Oak.createNewContentRepository(Oak.java:636)
> at org.apache.jackrabbit.oak.Oak.createContentRepository(Oak.java:624)
> at
> org.apache.jackrabbit.oak.jcr.Jcr.createContentRepository(Jcr.java:386){noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6481) Missing versionable path property for oak:mount-libs-crx.default
[
https://issues.apache.org/jira/browse/OAK-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6481.
-
Bulk close for 1.7.5
> Missing versionable path property for oak:mount-libs-crx.default
>
>
> Key: OAK-6481
> URL: https://issues.apache.org/jira/browse/OAK-6481
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: composite
>Affects Versions: 1.7.3
>Reporter: Tomek Rękawek
> Fix For: 1.8, 1.7.5
>
> Attachments: OAK-6481-2.patch, OAK-6481.patch
>
>
> After applying the OAK-3777, following warning can be found in the logs:
> {noformat}
> 21.07.2017 02:00:00.713 *WARN* [sling-default-999-HourlySnapshotJob]
> org.apache.jackrabbit.oak.plugins.version.ReadOnlyVersionManager Missing
> versionable path property for oak:mount-libs-crx.default at
> /jcr:system/jcr:versionStorage/5e/8d/89/5e8d8989-f86b-4e1b-b7f8-5089c44f5ed7
> 21.07.2017 02:00:00.713 *WARN* [sling-default-999-HourlySnapshotJob]
> org.apache.jackrabbit.oak.security.authorization.permission.CompiledPermissionImpl
> Cannot retrieve versionable node for
> /jcr:system/jcr:versionStorage/5e/8d/89/5e8d8989-f86b-4e1b-b7f8-5089c44f5ed7/1.0/jcr:frozenNode
> {noformat}
> It suggests that somehow the name of the new permission store
> (oak:mount-libs-crx.default) is being used as the workspace name for the
> version storage.
> If I understand correctly, it shouldn't happen. The workspace name is
> sometimes used as the name of the permission store node (eg.
> {{/jcr:system/rep:permissionStore/crx.default}}), but not the other way
> around - the name of the permission store node is not always a valid
> workspace name (eg.
> {{/jcr:system/rep:permissionStore/oak:mount-libs-crx.default}})
> //cc: [~stillalex]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6486) NPE in CompositeNodeStore
[
https://issues.apache.org/jira/browse/OAK-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6486.
-
Bulk close for 1.7.5
> NPE in CompositeNodeStore
> -
>
> Key: OAK-6486
> URL: https://issues.apache.org/jira/browse/OAK-6486
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: composite
>Affects Versions: 1.7.4
>Reporter: Tomek Rękawek
>Assignee: Tomek Rękawek
> Fix For: 1.8, 1.7.5
>
>
> Following NPE can be found in logs when running the Composite Node Store:
> {noformat}
> 24.07.2017 20:53:47.297 *ERROR* [oak-lucene-0]
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider Uncaught
> exception in
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider@6580fe4c
> java.lang.NullPointerException: null
> at
> org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:471)
> at
> org.apache.jackrabbit.oak.composite.CompositeNodeState.compareAgainstBaseState(CompositeNodeState.java:187)
> at
> org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.diffAndUpdate(IndexTracker.java:142)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.update(IndexTracker.java:113)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider.contentChanged(LuceneIndexProvider.java:75)
> at
> org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:128)
> at
> org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:122)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> Probably it's related to changes introduced in OAK-6425.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6494) Change journalGCMaxAge default
[ https://issues.apache.org/jira/browse/OAK-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6494. - Bulk close for 1.7.5 > Change journalGCMaxAge default > -- > > Key: OAK-6494 > URL: https://issues.apache.org/jira/browse/OAK-6494 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, documentmk >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Fix For: 1.8, 1.7.5 > > > The default journal retention time is currently set to 6 hours. With > OAK-5602, a long running transaction is aborted when it tries to merge branch > changes older than half the journal retention time. I suggest to increase the > default journal retention time to 24 hours. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6458) Increment the manifest version number
[ https://issues.apache.org/jira/browse/OAK-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6458. - Bulk close for 1.7.5 > Increment the manifest version number > - > > Key: OAK-6458 > URL: https://issues.apache.org/jira/browse/OAK-6458 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Francesco Mari >Assignee: Francesco Mari > Fix For: 1.8, 1.7.5 > > > OAK-3349 introduces a tail generation number, and OAK-6456 plans to adding > the tail generation number in the TAR index. To avoid old version of the code > to attempt reading data that would be considered malformed, the manifest > version number should be incremented. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6409) Oak-run indexing: improved (user friendly) output
[ https://issues.apache.org/jira/browse/OAK-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6409. - Bulk close for 1.7.5 > Oak-run indexing: improved (user friendly) output > - > > Key: OAK-6409 > URL: https://issues.apache.org/jira/browse/OAK-6409 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: indexing >Reporter: Thomas Mueller >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.5 > > > The oak-run indexing (OAK-6081) output should be human readable, and if > possible minimal. Detailed output should be written to a log file, but stdout > should be easy for a user to understand. For example some header info when > starting, where to find the detailed output, then one line every 3 seconds > about the progress (in %, number of nodes read, ETA), and when done some info > on what to do next. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6468) Include the tail generation in the binary references index
[ https://issues.apache.org/jira/browse/OAK-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6468. - Bulk close for 1.7.5 > Include the tail generation in the binary references index > -- > > Key: OAK-6468 > URL: https://issues.apache.org/jira/browse/OAK-6468 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Francesco Mari >Assignee: Francesco Mari > Fix For: 1.8, 1.7.5 > > > OAK-3349 introduces the tail generation, an additional generation number > associated to each segment that is consulted during the cleanup phase to > determine if a segment should be removed. The tail generation is currently > only preset in the segments. For efficiency purposes, it should be added to > the binary references index too, like the full generation number. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6488) Move journal maxRevisionAge to DocumentNodeStore
[
https://issues.apache.org/jira/browse/OAK-6488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide Giannella closed OAK-6488.
-
Bulk close for 1.7.5
> Move journal maxRevisionAge to DocumentNodeStore
>
>
> Key: OAK-6488
> URL: https://issues.apache.org/jira/browse/OAK-6488
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.8, 1.7.5
>
>
> The journal entry retention time {{maxRevisionAge}} is currently not known to
> the DocumentNodeStore. The DocumentNodeStoreService uses the value as a
> parameter when it calls the JournalGarbageCollector. For OAK-5602 the
> DocumentNodeStore must know the {{maxRevisionAge}} to prevent transaction
> from running longer than half this value.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (OAK-6487) IndexUpdate should not reindex newly imported index definitions
[ https://issues.apache.org/jira/browse/OAK-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6487. - Bulk close for 1.7.5 > IndexUpdate should not reindex newly imported index definitions > --- > > Key: OAK-6487 > URL: https://issues.apache.org/jira/browse/OAK-6487 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: indexing >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.8, 1.7.5 > > > IndexUpdate currently decides to reindex any index which is not found in > before state i.e. state corresponding to the last checkpointed state. This > was done to support cases like OAK-1874 where a new index may not have > reindex flag set to true. > For importing new indexed definitions i.e. where indexing is already done > out-of-band by oak-run this poses problem as this new index definition would > not be present in before state. > To account for that this logic should be changed to check if the new index > definition is already indexed or not. This can be checked by checking for > presence of any hidden node under index definitions. If present then it means > that index is already indexed so no need to reindex -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (OAK-6476) Support deserializing json as NodeState
[ https://issues.apache.org/jira/browse/OAK-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella closed OAK-6476. - Bulk close for 1.7.5 > Support deserializing json as NodeState > --- > > Key: OAK-6476 > URL: https://issues.apache.org/jira/browse/OAK-6476 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: store-spi >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra > Fix For: 1.8, 1.7.5 > > Attachments: OAK-6476-v1.patch > > > Oak has JsonSerializer in oak-store-spi which supports serializing the > NodeState as json (preserving the property types). For OAK-6471 I need a way > to deserialize this json as NodeState. > So I would like to implement a JsonDeserializer which does the reverse. This > would be based on logic already implemented in > [KernelNodeState|https://github.com/apache/jackrabbit-oak/blob/1.0/oak-core/src/main/java/org/apache/jackrabbit/oak/kernel/KernelNodeState.java] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6506) Ensure unique property indexes are consistent when mounting NodeStores
[
https://issues.apache.org/jira/browse/OAK-6506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112880#comment-16112880
]
Robert Munteanu commented on OAK-6506:
--
Currently, I think using the {{IndexStoreStrategy}} is the way to go. I would
use the {{Multiplexers}} class to get all the defined indexes, and then:
* generate the combinations needed to match all indexes
* iterate over each combination and check if there are matching values
The checks are not reflexive, so we need to test both index_n against index_n+1
and index_n+1 against index_n.
> Ensure unique property indexes are consistent when mounting NodeStores
> --
>
> Key: OAK-6506
> URL: https://issues.apache.org/jira/browse/OAK-6506
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: composite
>Reporter: Robert Munteanu
>Assignee: Robert Munteanu
> Fix For: 1.8, 1.7.6
>
>
> When mounting a NodeStore in a composite setup, we should validate that the
> unique property indexes do not contain duplicate entries across all the node
> stores.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6518) Implement transparent usage of different TAR index formats
[ https://issues.apache.org/jira/browse/OAK-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112903#comment-16112903 ] Francesco Mari commented on OAK-6518: - A new data type to represent the index has been introduced at r1804011. > Implement transparent usage of different TAR index formats > -- > > Key: OAK-6518 > URL: https://issues.apache.org/jira/browse/OAK-6518 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-tar >Reporter: Francesco Mari >Assignee: Francesco Mari > Fix For: 1.8, 1.7.6 > > > The segment store should be able to read and validate two different format of > indexes, the one in use before the introduction of the tail generation number > and the one after. The detection of the different index types should be fully > transparent and automatic. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-6500) NRTIndex leaks file handles due to unclosed IndexReader
[
https://issues.apache.org/jira/browse/OAK-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra resolved OAK-6500.
--
Resolution: Fixed
> NRTIndex leaks file handles due to unclosed IndexReader
> ---
>
> Key: OAK-6500
> URL: https://issues.apache.org/jira/browse/OAK-6500
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.6.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Critical
> Labels: candidate_oak_1_6
> Fix For: 1.8, 1.7.6
>
> Attachments: OAK-6500-approach-a-v1.patch,
> OAK-6500-ref-count-v1.patch, OAK-6500-v1.patch
>
>
> On some setups under stress it has been seen that NRTIndex leaks file handles
> over time.
> Checking with lsof indicates that more than 3 nrt folders per index are being
> used. However per design there can be max 3 and after system is not in use
> max 1 should be present.
> {noformat}
> $ lsof -p 9550 | grep '/nrt' | gawk 'match($0,
> /.*crx-quickstart\/repository\/index\/(.*?)\/\_.*$/, m) { print m[1]; }' |
> sort | uniq
> cqPageLucene-1501065263331/nrt1501065335930
> cqPageLucene-1501065263331/nrt1501065374667
> cqPageLucene-1501065263331/nrt1501065392492
> cqPageLucene-1501065263331/nrt1501065440955
> cqPageLucene-1501065263331/nrt1501065473286
> cqPageLucene-1501065263331/nrt1501065507345
> slingeventJob-1501065263330/nrt1501065356975
> slingeventJob-1501065263330/nrt1501065373229
> slingeventJob-1501065263330/nrt1501065394142
> slingeventJob-1501065263330/nrt1501065440953
> slingeventJob-1501065263330/nrt1501065473282
> slingeventJob-1501065263330/nrt1501065507342
> versionStoreIndex-1501065263332/nrt1501065335925
> versionStoreIndex-1501065263332/nrt1501065366781
> versionStoreIndex-1501065263332/nrt1501065392490
> versionStoreIndex-1501065263332/nrt1501065441232
> versionStoreIndex-1501065263332/nrt1501065473285
> {noformat}
> Further actually checking index folder indicates that those folder are
> actually deleted. So some where the file handle is still referring them.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6500) NRTIndex leaks file handles due to unclosed IndexReader
[
https://issues.apache.org/jira/browse/OAK-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-6500:
-
Fix Version/s: (was: 1.6.4)
> NRTIndex leaks file handles due to unclosed IndexReader
> ---
>
> Key: OAK-6500
> URL: https://issues.apache.org/jira/browse/OAK-6500
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.6.0
>Reporter: Chetan Mehrotra
>Assignee: Chetan Mehrotra
>Priority: Critical
> Labels: candidate_oak_1_6
> Fix For: 1.8, 1.7.6
>
> Attachments: OAK-6500-approach-a-v1.patch,
> OAK-6500-ref-count-v1.patch, OAK-6500-v1.patch
>
>
> On some setups under stress it has been seen that NRTIndex leaks file handles
> over time.
> Checking with lsof indicates that more than 3 nrt folders per index are being
> used. However per design there can be max 3 and after system is not in use
> max 1 should be present.
> {noformat}
> $ lsof -p 9550 | grep '/nrt' | gawk 'match($0,
> /.*crx-quickstart\/repository\/index\/(.*?)\/\_.*$/, m) { print m[1]; }' |
> sort | uniq
> cqPageLucene-1501065263331/nrt1501065335930
> cqPageLucene-1501065263331/nrt1501065374667
> cqPageLucene-1501065263331/nrt1501065392492
> cqPageLucene-1501065263331/nrt1501065440955
> cqPageLucene-1501065263331/nrt1501065473286
> cqPageLucene-1501065263331/nrt1501065507345
> slingeventJob-1501065263330/nrt1501065356975
> slingeventJob-1501065263330/nrt1501065373229
> slingeventJob-1501065263330/nrt1501065394142
> slingeventJob-1501065263330/nrt1501065440953
> slingeventJob-1501065263330/nrt1501065473282
> slingeventJob-1501065263330/nrt1501065507342
> versionStoreIndex-1501065263332/nrt1501065335925
> versionStoreIndex-1501065263332/nrt1501065366781
> versionStoreIndex-1501065263332/nrt1501065392490
> versionStoreIndex-1501065263332/nrt1501065441232
> versionStoreIndex-1501065263332/nrt1501065473285
> {noformat}
> Further actually checking index folder indicates that those folder are
> actually deleted. So some where the file handle is still referring them.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
