[jira] [Comment Edited] (OAK-10610) Adding OSGi wiring test for SplitPersistenceService is failing
[
https://issues.apache.org/jira/browse/OAK-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808552#comment-17808552
]
Julian Sedding edited comment on OAK-10610 at 1/19/24 1:06 PM:
---
As discussed privately, the issue is with mock-osgi.
At the end of the test it unregisters all services. While unregistereing the
component {{SplitPersistenceService}}, it calls its deactivate method, where
the {{SplitPersistence}} service is manually unregistered. Because
{{SplitPersistence}} is registered as a {{SegmentNodeStorePersistence}},
mock-osgi tries to find all references to this service, in order to deactivate
any referents. The {{SplitPersistenceService}} component has two references to
{{SegmentNodeStorePersistence}} and thus mock-osgi tries to deactivate it.
Multiple things are wrong with this:
# when determining referents, it should check that the service object matches
# when determining referents, it should consider a refernce's target filter
# it should probably keep track of components it is already deactivating in
order to prevent this sort of cycle, I suppose it could throw an exception in
that case.
cc [~sseifert]
was (Author: jsedding):
As discussed privately, the issue is with mock-osgi.
At the end of the test it unregisters all services. While unregistereing the
component {{SplitPersistenceService}}, it calls its deactivate method, where
the {{SplitPersistence}} service is manually unregistered. Because
{{SplitPersistence}} is registered as a {{SegmentNodeStorePersistence}},
mock-osgi tries to find all references to this service, in order to deactivate
any referents. The {{SplitPersistenceService}} component has two references to
{{SegmentNodeStorePersistence}} and thus mock-osgi tries to deactivate it.
# Multiple things are wrong with this:
# when determining referents, it should check that the service object matches
# when determining referents, it should consider a refernce's target filter
# it should probably keep track of components it is already deactivating in
order to prevent this sort of cycle, I suppose it could throw an exception in
that case.
cc [~sseifert]
> Adding OSGi wiring test for SplitPersistenceService is failing
> --
>
> Key: OAK-10610
> URL: https://issues.apache.org/jira/browse/OAK-10610
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: osgi, segment-tar
>Reporter: Andrei Dulceanu
>Priority: Minor
> Attachments: SplitPersistenceServiceTest.patch
>
>
> While running the test in the patch attached, I always get:
> {noformat}
> java.lang.RuntimeException: Unable to invoke method 'deactivate' for class
> org.apache.jackrabbit.oak.segment.osgi.SplitPersistenceService
> at
> org.apache.sling.testing.mock.osgi.OsgiServiceUtil.invokeMethod(OsgiServiceUtil.java:325)
> at
> org.apache.sling.testing.mock.osgi.OsgiServiceUtil.invokeLifecycleMethod(OsgiServiceUtil.java:225)
> at
> org.apache.sling.testing.mock.osgi.OsgiServiceUtil.activateDeactivate(OsgiServiceUtil.java:89)
> at
> org.apache.sling.testing.mock.osgi.MockOsgi.deactivate(MockOsgi.java:231)
> at
> org.apache.sling.testing.mock.osgi.MockBundleContext.shutdown(MockBundleContext.java:449)
> at
> org.apache.sling.testing.mock.osgi.MockOsgi.shutdown(MockOsgi.java:330)
> at
> org.apache.sling.testing.mock.osgi.context.OsgiContextImpl.tearDown(OsgiContextImpl.java:58)
> at
> org.apache.sling.testing.mock.osgi.junit.OsgiContext.access$200(OsgiContext.java:35)
> at
> org.apache.sling.testing.mock.osgi.junit.OsgiContext$1.after(OsgiContext.java:86)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:59)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at
> org.junit.runners.ParentRunner$<...>.invokeLifecycleMethod(OsgiServiceUtil.java:225)
> at
> org.apache.sling.testing.mock.osgi.OsgiServiceUtil.activateDeactivate(OsgiServiceUtil.java:89)
> at
> org.apache.sling.testing.mock.osgi.MockOsgi.deactivate(MockOsgi.java:231)
> at
> org.apache.sling.testing.mock.osgi.MockOsgi.deactivate(MockOsgi.java:218)
> at
> org.apache.sling.testing.mock.osgi.MockBundleContext.restartService(MockBundleContext.java:202)
> at
> org.apache.sling.testing.mock.osgi.MockBundleContext.handleRefsUpdateOnUnregister(MockBundleContext.java:259)
> at
> org.apache.sling.testing.mock.osgi.MockBundleContext.unregisterService(MockBundleContext.java:191)
> at
> org.apache.sling.testing.mock.osgi.MockServiceRegistration.unregister(MockServiceRegistration.java:100)
> at
> org.apache.jackrabbit.oak.osgi.OsgiWhiteboard$1.unregister(OsgiWhiteboard.java:87)
> at
>
[jira] [Commented] (OAK-10610) Adding OSGi wiring test for SplitPersistenceService is failing
[
https://issues.apache.org/jira/browse/OAK-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808552#comment-17808552
]
Julian Sedding commented on OAK-10610:
--
As discussed privately, the issue is with mock-osgi.
At the end of the test it unregisters all services. While unregistereing the
component {{SplitPersistenceService}}, it calls its deactivate method, where
the {{SplitPersistence}} service is manually unregistered. Because
{{SplitPersistence}} is registered as a {{SegmentNodeStorePersistence}},
mock-osgi tries to find all references to this service, in order to deactivate
any referents. The {{SplitPersistenceService}} component has two references to
{{SegmentNodeStorePersistence}} and thus mock-osgi tries to deactivate it.
# Multiple things are wrong with this:
# when determining referents, it should check that the service object matches
# when determining referents, it should consider a refernce's target filter
# it should probably keep track of components it is already deactivating in
order to prevent this sort of cycle, I suppose it could throw an exception in
that case.
cc [~sseifert]
> Adding OSGi wiring test for SplitPersistenceService is failing
> --
>
> Key: OAK-10610
> URL: https://issues.apache.org/jira/browse/OAK-10610
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: osgi, segment-tar
>Reporter: Andrei Dulceanu
>Priority: Minor
> Attachments: SplitPersistenceServiceTest.patch
>
>
> While running the test in the patch attached, I always get:
> {noformat}
> java.lang.RuntimeException: Unable to invoke method 'deactivate' for class
> org.apache.jackrabbit.oak.segment.osgi.SplitPersistenceService
> at
> org.apache.sling.testing.mock.osgi.OsgiServiceUtil.invokeMethod(OsgiServiceUtil.java:325)
> at
> org.apache.sling.testing.mock.osgi.OsgiServiceUtil.invokeLifecycleMethod(OsgiServiceUtil.java:225)
> at
> org.apache.sling.testing.mock.osgi.OsgiServiceUtil.activateDeactivate(OsgiServiceUtil.java:89)
> at
> org.apache.sling.testing.mock.osgi.MockOsgi.deactivate(MockOsgi.java:231)
> at
> org.apache.sling.testing.mock.osgi.MockBundleContext.shutdown(MockBundleContext.java:449)
> at
> org.apache.sling.testing.mock.osgi.MockOsgi.shutdown(MockOsgi.java:330)
> at
> org.apache.sling.testing.mock.osgi.context.OsgiContextImpl.tearDown(OsgiContextImpl.java:58)
> at
> org.apache.sling.testing.mock.osgi.junit.OsgiContext.access$200(OsgiContext.java:35)
> at
> org.apache.sling.testing.mock.osgi.junit.OsgiContext$1.after(OsgiContext.java:86)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:59)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at
> org.junit.runners.ParentRunner$<...>.invokeLifecycleMethod(OsgiServiceUtil.java:225)
> at
> org.apache.sling.testing.mock.osgi.OsgiServiceUtil.activateDeactivate(OsgiServiceUtil.java:89)
> at
> org.apache.sling.testing.mock.osgi.MockOsgi.deactivate(MockOsgi.java:231)
> at
> org.apache.sling.testing.mock.osgi.MockOsgi.deactivate(MockOsgi.java:218)
> at
> org.apache.sling.testing.mock.osgi.MockBundleContext.restartService(MockBundleContext.java:202)
> at
> org.apache.sling.testing.mock.osgi.MockBundleContext.handleRefsUpdateOnUnregister(MockBundleContext.java:259)
> at
> org.apache.sling.testing.mock.osgi.MockBundleContext.unregisterService(MockBundleContext.java:191)
> at
> org.apache.sling.testing.mock.osgi.MockServiceRegistration.unregister(MockServiceRegistration.java:100)
> at
> org.apache.jackrabbit.oak.osgi.OsgiWhiteboard$1.unregister(OsgiWhiteboard.java:87)
> at
> org.apache.jackrabbit.oak.segment.osgi.SplitPersistenceService.deactivate(SplitPersistenceService.java:70)
> at jdk.internal.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
> {noformat}
> The problem is that there is a {{StackOverflowException}} being thrown when
> calling {{deactivate()}}. This might stem from the fact that
> {{SplitPersistenceService}} has a reference to the same type it tries to
> register, but this is only an assumption.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-9897) SplitPersistence: FileReaper cannot finish cleanup
[
https://issues.apache.org/jira/browse/OAK-9897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17622176#comment-17622176
]
Julian Sedding commented on OAK-9897:
-
Proposed fix in [PR #665|https://github.com/apache/jackrabbit-oak/pull/665].
> SplitPersistence: FileReaper cannot finish cleanup
> --
>
> Key: OAK-9897
> URL: https://issues.apache.org/jira/browse/OAK-9897
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.44.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Minor
>
> When running revision compaction (aka GC) on a setup with split persistence,
> it is possible that the "cleanup" phase identifies files from the read-only
> part of the persistence as redundant and submits them to the {{FileReaper}}
> for deletion.
> However, the delete method of the {{SplitSegmentArchiveManager}} is
> implemented to return "false" when attempting the deletion of a file from the
> read-only persistence (which I would argue is correct). The {{FileReaper}}
> then re-adds the offending file to its queue in order to retry deleting it.
> Of course this fails again and so it can never finish and logs warnings for
> the files it cannot delete.
> It should be possible to prevent deletion of read-only files in the first
> place. In fact, I think there is no need to mark and sweep them at all.
> cc [~adulceanu]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-9785:
Fix Version/s: 1.22.14
> Tar SegmentStore can be corrupted during compaction
> ---
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.42.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
> Labels: candidate_oak_1_22, candidate_oak_1_8
> Fix For: 1.46.0, 1.22.14
>
> Attachments: error.log.2022-06-09
>
>
> There is a scenario where a segment store can become corrupted, leading to
> {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the
> 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect
> cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was
> a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a
> {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially
> written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision
> data
> - the {{SegmentNotFoundException}} is first observed either during the
> remainder of the compaction run or when the respective node is requested the
> next time, usually during a traversal
> The root cause is in
> [{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
> where only {{IOExceptions}} are caught.
> In order to improve the robustness of the code, I think we need to catch all
> {{Throwables}}. Otherwise we cannot guarantee that compaction is correctly
> aborted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17619405#comment-17619405
]
Julian Sedding edited comment on OAK-9785 at 10/21/22 8:02 AM:
---
Backported to the 1.22 branch with [PR
#733|https://github.com/apache/jackrabbit-oak/pull/733].
was (Author: jsedding):
I created a [PR to backport the change to the 1.22
branch|https://github.com/apache/jackrabbit-oak/pull/733].
> Tar SegmentStore can be corrupted during compaction
> ---
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.42.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
> Labels: candidate_oak_1_22, candidate_oak_1_8
> Fix For: 1.46.0
>
> Attachments: error.log.2022-06-09
>
>
> There is a scenario where a segment store can become corrupted, leading to
> {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the
> 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect
> cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was
> a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a
> {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially
> written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision
> data
> - the {{SegmentNotFoundException}} is first observed either during the
> remainder of the compaction run or when the respective node is requested the
> next time, usually during a traversal
> The root cause is in
> [{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
> where only {{IOExceptions}} are caught.
> In order to improve the robustness of the code, I think we need to catch all
> {{Throwables}}. Otherwise we cannot guarantee that compaction is correctly
> aborted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-9785:
Labels: candidate_oak_1_8 (was: candidate_oak_1_22 candidate_oak_1_8)
> Tar SegmentStore can be corrupted during compaction
> ---
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.42.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
> Labels: candidate_oak_1_8
> Fix For: 1.46.0, 1.22.14
>
> Attachments: error.log.2022-06-09
>
>
> There is a scenario where a segment store can become corrupted, leading to
> {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the
> 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect
> cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was
> a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a
> {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially
> written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision
> data
> - the {{SegmentNotFoundException}} is first observed either during the
> remainder of the compaction run or when the respective node is requested the
> next time, usually during a traversal
> The root cause is in
> [{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
> where only {{IOExceptions}} are caught.
> In order to improve the robustness of the code, I think we need to catch all
> {{Throwables}}. Otherwise we cannot guarantee that compaction is correctly
> aborted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17619405#comment-17619405
]
Julian Sedding commented on OAK-9785:
-
I created a [PR to backport the change to the 1.22
branch|https://github.com/apache/jackrabbit-oak/pull/733].
> Tar SegmentStore can be corrupted during compaction
> ---
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.42.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
> Labels: candidate_oak_1_22, candidate_oak_1_8
> Fix For: 1.46.0
>
> Attachments: error.log.2022-06-09
>
>
> There is a scenario where a segment store can become corrupted, leading to
> {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the
> 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect
> cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was
> a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a
> {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially
> written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision
> data
> - the {{SegmentNotFoundException}} is first observed either during the
> remainder of the compaction run or when the respective node is requested the
> next time, usually during a traversal
> The root cause is in
> [{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
> where only {{IOExceptions}} are caught.
> In order to improve the robustness of the code, I think we need to catch all
> {{Throwables}}. Otherwise we cannot guarantee that compaction is correctly
> aborted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-9785:
Labels: candidate_oak_1_22 candidate_oak_1_8 (was: candidate_oak_1_22)
> Tar SegmentStore can be corrupted during compaction
> ---
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.42.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
> Labels: candidate_oak_1_22, candidate_oak_1_8
> Fix For: 1.46.0
>
> Attachments: error.log.2022-06-09
>
>
> There is a scenario where a segment store can become corrupted, leading to
> {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the
> 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect
> cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was
> a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a
> {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially
> written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision
> data
> - the {{SegmentNotFoundException}} is first observed either during the
> remainder of the compaction run or when the respective node is requested the
> next time, usually during a traversal
> The root cause is in
> [{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
> where only {{IOExceptions}} are caught.
> In order to improve the robustness of the code, I think we need to catch all
> {{Throwables}}. Otherwise we cannot guarantee that compaction is correctly
> aborted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-9785:
Fix Version/s: 1.46.0
> Tar SegmentStore can be corrupted during compaction
> ---
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.42.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
> Fix For: 1.46.0
>
> Attachments: error.log.2022-06-09
>
>
> There is a scenario where a segment store can become corrupted, leading to
> {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the
> 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect
> cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was
> a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a
> {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially
> written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision
> data
> - the {{SegmentNotFoundException}} is first observed either during the
> remainder of the compaction run or when the respective node is requested the
> next time, usually during a traversal
> The root cause is in
> [{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
> where only {{IOExceptions}} are caught.
> In order to improve the robustness of the code, I think we need to catch all
> {{Throwables}}. Otherwise we cannot guarantee that compaction is correctly
> aborted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-9785:
Labels: candidate_oak_1_22 (was: )
> Tar SegmentStore can be corrupted during compaction
> ---
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.42.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
> Labels: candidate_oak_1_22
> Fix For: 1.46.0
>
> Attachments: error.log.2022-06-09
>
>
> There is a scenario where a segment store can become corrupted, leading to
> {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the
> 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect
> cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was
> a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a
> {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially
> written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision
> data
> - the {{SegmentNotFoundException}} is first observed either during the
> remainder of the compaction run or when the respective node is requested the
> next time, usually during a traversal
> The root cause is in
> [{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
> where only {{IOExceptions}} are caught.
> In order to improve the robustness of the code, I think we need to catch all
> {{Throwables}}. Otherwise we cannot guarantee that compaction is correctly
> aborted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding resolved OAK-9785.
-
Resolution: Fixed
I have merged the PR.
> Tar SegmentStore can be corrupted during compaction
> ---
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.42.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
> Attachments: error.log.2022-06-09
>
>
> There is a scenario where a segment store can become corrupted, leading to
> {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the
> 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect
> cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was
> a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a
> {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially
> written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision
> data
> - the {{SegmentNotFoundException}} is first observed either during the
> remainder of the compaction run or when the respective node is requested the
> next time, usually during a traversal
> The root cause is in
> [{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
> where only {{IOExceptions}} are caught.
> In order to improve the robustness of the code, I think we need to catch all
> {{Throwables}}. Otherwise we cannot guarantee that compaction is correctly
> aborted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17601761#comment-17601761
]
Julian Sedding commented on OAK-9785:
-
[~adulceanu] any thoughts? Can we move forward with this?
> Tar SegmentStore can be corrupted during compaction
> ---
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.42.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
> Attachments: error.log.2022-06-09
>
>
> There is a scenario where a segment store can become corrupted, leading to
> {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the
> 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect
> cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was
> a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a
> {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially
> written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision
> data
> - the {{SegmentNotFoundException}} is first observed either during the
> remainder of the compaction run or when the respective node is requested the
> next time, usually during a traversal
> The root cause is in
> [{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
> where only {{IOExceptions}} are caught.
> In order to improve the robustness of the code, I think we need to catch all
> {{Throwables}}. Otherwise we cannot guarantee that compaction is correctly
> aborted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding reassigned OAK-9785:
---
Assignee: Julian Sedding
> Tar SegmentStore can be corrupted during compaction
> ---
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.42.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
> Attachments: error.log.2022-06-09
>
>
> There is a scenario where a segment store can become corrupted, leading to
> {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the
> 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect
> cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was
> a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a
> {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially
> written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision
> data
> - the {{SegmentNotFoundException}} is first observed either during the
> remainder of the compaction run or when the respective node is requested the
> next time, usually during a traversal
> The root cause is in
> [{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
> where only {{IOExceptions}} are caught.
> In order to improve the robustness of the code, I think we need to catch all
> {{Throwables}}. Otherwise we cannot guarantee that compaction is correctly
> aborted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (OAK-9914) Starting Oak with Azure persistence in read-only mode while another Oak process is running will initiate repo recovery
[ https://issues.apache.org/jira/browse/OAK-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Sedding resolved OAK-9914. - Fix Version/s: 1.46.0 Assignee: Julian Sedding (was: Miroslav Smiljanic) Resolution: Fixed > Starting Oak with Azure persistence in read-only mode while another Oak > process is running will initiate repo recovery > -- > > Key: OAK-9914 > URL: https://issues.apache.org/jira/browse/OAK-9914 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-azure >Affects Versions: 1.44.0 >Reporter: Miroslav Smiljanic >Assignee: Julian Sedding >Priority: Major > Fix For: 1.46.0 > > Attachments: AzureArchiveManager.patch, > AzureArchiveManager_sys_prop.patch, OAK-9914_test.patch > > > The sequence of events: > # Oak process with read/write file store utilizing Azure persistence is > already running > # New Oak process is starting up, with read-only file store using Azure > persistence and the same storage account and container like the previous Oak > process > ## New Oak process starts recovery procedure for the last tar directory that > is not closed, and that has an impact on the startup time > Scenario presented in the attached test case: > [^OAK-9914_test.patch] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OAK-9914) Starting Oak with Azure persistence in read-only mode while another Oak process is running will initiate repo recovery
[ https://issues.apache.org/jira/browse/OAK-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17599570#comment-17599570 ] Julian Sedding commented on OAK-9914: - [~miroslav] bq. ... missed GitHub notification ... No worries! Next time I'll mention the PR in the JIRA ticket, it's easy to miss some of the automated emails. bq. ... recovery is impacting the startup time ... Ah, I hadn't realized. Good to know! I've merged [PR #685|https://github.com/apache/jackrabbit-oak/pull/685] now. Thanks for the review! > Starting Oak with Azure persistence in read-only mode while another Oak > process is running will initiate repo recovery > -- > > Key: OAK-9914 > URL: https://issues.apache.org/jira/browse/OAK-9914 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-azure >Affects Versions: 1.44.0 >Reporter: Miroslav Smiljanic >Assignee: Miroslav Smiljanic >Priority: Major > Attachments: AzureArchiveManager.patch, > AzureArchiveManager_sys_prop.patch, OAK-9914_test.patch > > > The sequence of events: > # Oak process with read/write file store utilizing Azure persistence is > already running > # New Oak process is starting up, with read-only file store using Azure > persistence and the same storage account and container like the previous Oak > process > ## New Oak process starts recovery procedure for the last tar directory that > is not closed, and that has an impact on the startup time > Scenario presented in the attached test case: > [^OAK-9914_test.patch] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OAK-9914) Starting Oak with Azure persistence in read-only mode while another Oak process is running will initiate repo recovery
[ https://issues.apache.org/jira/browse/OAK-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17599483#comment-17599483 ] Julian Sedding commented on OAK-9914: - [~miroslav] have you not seen https://github.com/apache/jackrabbit-oak/pull/685? It's very similar to your PR. > Starting Oak with Azure persistence in read-only mode while another Oak > process is running will initiate repo recovery > -- > > Key: OAK-9914 > URL: https://issues.apache.org/jira/browse/OAK-9914 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-azure >Affects Versions: 1.44.0 >Reporter: Miroslav Smiljanic >Assignee: Miroslav Smiljanic >Priority: Major > Attachments: AzureArchiveManager.patch, > AzureArchiveManager_sys_prop.patch, OAK-9914_test.patch > > > The sequence of events: > # Oak process with read/write file store utilizing Azure persistence is > already running > # New Oak process is starting up, with read-only file store using Azure > persistence and the same storage account and container like the previous Oak > process > ## New Oak process starts recovery procedure for the last tar directory that > is not closed > Scenario presented in the attached test case: > [^OAK-9914_test.patch] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OAK-9914) Starting Oak with Azure persistence in read-only mode while another Oak process is running will initiate repo recovery
[ https://issues.apache.org/jira/browse/OAK-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17599482#comment-17599482 ] Julian Sedding commented on OAK-9914: - [~joerghoh] Yes the change I propose would change the behaviour of oak-segment-azure slightly. It would align it with (my interpretation) of the javadocs and with the implementation of oak-segment-tar (which I consider the canonical implementation). The change doesn't introduce any test failures and I assume the only consumers of this internal API catch the exception and handle it as if it were null. On a practical level, returning null avoids the stack-trace being printed to the logs, which IIUC was part of the problem. > Starting Oak with Azure persistence in read-only mode while another Oak > process is running will initiate repo recovery > -- > > Key: OAK-9914 > URL: https://issues.apache.org/jira/browse/OAK-9914 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-azure >Affects Versions: 1.44.0 >Reporter: Miroslav Smiljanic >Assignee: Miroslav Smiljanic >Priority: Major > Attachments: AzureArchiveManager.patch, > AzureArchiveManager_sys_prop.patch, OAK-9914_test.patch > > > The sequence of events: > # Oak process with read/write file store utilizing Azure persistence is > already running > # New Oak process is starting up, with read-only file store using Azure > persistence and the same storage account and container like the previous Oak > process > ## New Oak process starts recovery procedure for the last tar directory that > is not closed > Scenario presented in the attached test case: > [^OAK-9914_test.patch] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (OAK-9914) Starting Oak with Azure persistence in read-only mode while another Oak process is running will initiate repo recovery
[
https://issues.apache.org/jira/browse/OAK-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598909#comment-17598909
]
Julian Sedding commented on OAK-9914:
-
Hi [~miroslav]
bq. Returning null wouldn't allow having the second Oak process in read-only
mode.
The TAR-file-based persistence returns null for files without an index (which
is equivalent to an azure folder without a "close" file). And it does allow for
a second read-only process. The test-case I shared yesterday shows that the
TAR-file-based RO-store also "recovers" and creates a {{*.ro.bak}} file.
Unfortunately, the javadocs for {{SegmentArchiveReader#open()}} do not specify
when an {{IOException}} should be thrown. However it _does_ specify that
{{null}} should be returned "if the archive doesn't exist or doesn't have a
valid index". I would interpret "doesn't have a valid index" to "has not been
closed" based on the precedent of the {{SegmentTarManager}} implementation.
Based on this rationale, I would align the behaviour of the
{{AzureArchiveManager}} with the one of the {{SegmentTarManager}}. That
certainly does not violate the javadocs, and it passes all existing unit-tests
in {{oak-segment-azure}}. Returning {{null}} for an "unclosed" archive gets rid
of the ugly stack-trace in the log file.
On the other hand, if I apply your patch [^AzureArchiveManager.patch], the test
{{AzureArchiveManagerTest#testUncleanStopSegmentMissing}} starts failing. I.e.
it breaks "real" recovery.
I agree that the read-only case could be implemented more nicely (that's in
[{{TarReader#openRO()}}|https://github.com/apache/jackrabbit-oak/blob/c6ddcc55bee3de915459af01e91edad32d538f3d/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/tar/TarReader.java#L115-L135]).
I see the following opportunities for improvements:
1. It does not attempt to read an archive using
{{SegmentArchiveManager#forceOpen()}} once {{SegmentArchiveManager#open()}}
returns {{null}}. {{#forceOpen}} is documented to "[open] an archive that
wasn't closed correctly". Implementing it is optional, and the
{{SegmentTarManager}} does not currently implement it (it delegates to #open()).
2. The archive is "recovered" persistently. I think for the read-only case it
would be nicer to reconstruct a missing index in-memory.
Number (1), i.e. using {{forceOpen}} would already work with the
{{AzureArchivemanager}}.
Using a system property sounds strange to me. There are no two correct
behaviours, so let's just fix it. And BTW, the system property patch would also
fail {{AzureArchiveManagerTest#testUncleanStopSegmentMissing}} when the system
property is set.
> Starting Oak with Azure persistence in read-only mode while another Oak
> process is running will initiate repo recovery
> --
>
> Key: OAK-9914
> URL: https://issues.apache.org/jira/browse/OAK-9914
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-azure
>Affects Versions: 1.44.0
>Reporter: Miroslav Smiljanic
>Assignee: Miroslav Smiljanic
>Priority: Major
> Attachments: AzureArchiveManager.patch,
> AzureArchiveManager_sys_prop.patch, OAK-9914_test.patch
>
>
> The sequence of events:
> # Oak process with read/write file store utilizing Azure persistence is
> already running
> # New Oak process is starting up, with read-only file store using Azure
> persistence and the same storage account and container like the previous Oak
> process
> ## New Oak process starts recovery procedure for the last tar directory that
> is not closed
> Scenario presented in the attached test case:
> [^OAK-9914_test.patch]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-9914) Starting Oak with Azure persistence in read-only mode while another Oak process is running will initiate repo recovery
[
https://issues.apache.org/jira/browse/OAK-9914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17598456#comment-17598456
]
Julian Sedding commented on OAK-9914:
-
I think the proposed fix is incorrect. A simpler fix changes
{{AzureArchiveManager#open()}} to return {{null}} instead of throwing an
exception, and reduces the log level of the read-only recovery messages from
WARN to INFO. I explain my rationale below.
Note that the file-based segment store (i.e. TarPersistence) also creates a
file ending in {{.ro.bak}} when a read-only store is opened and the last tar
file does not contain an index (yet).
I believe that the index for an open file in the rw store is maintained
in-memory and only written when the file is closed. It can of course happen
that an application crashes/is killed and a tar file is left without an index,
which warrants a WARN log message about recovery.
For the read-only case, I would argue that the log message should be at most at
INFO level, which would be a simple fix in {{TarReader}}. Also, the javadocs of
{{SegmentArchiveManager#open()}} say the following regarding the return value:
"the archive reader or null if the archive doesn't exist or doesn't have a
valid index". Therefore, I would argue that the {{AzureArchiveManager}} mustn't
throw an exception, but should just return {{null}}.
Test case using TarPersistence for comparison, also look at the log messages:
{code:java}
@Test
public void testReadOnlyRecoveryTar() throws
InvalidFileStoreVersionException, IOException, CommitFailedException {
final File directory = folder.newFolder();
FileStore rwFileStore =
FileStoreBuilder.fileStoreBuilder(directory).build();
SegmentNodeStore segmentNodeStore =
SegmentNodeStoreBuilders.builder(rwFileStore).build();
NodeBuilder builder = segmentNodeStore.getRoot().builder();
builder.setProperty("foo", "bar");
segmentNodeStore.merge(builder, EmptyHook.INSTANCE, CommitInfo.EMPTY);
rwFileStore.flush();
assertTrue(new File(directory, "data0a.tar").isFile());
assertFalse(new File(directory, "data0a.tar.ro.bak").exists());
// create read-only FS
ReadOnlyFileStore roFileStore =
FileStoreBuilder.fileStoreBuilder(directory).buildReadOnly();
roFileStore.close();
rwFileStore.close();
assertTrue(new File(directory, "data0a.tar").isFile());
assertFalse(new File(directory, "data0a.tar.ro.bak").exists()); //
fails and the log contains warn message regarding recovery
}
{code}
> Starting Oak with Azure persistence in read-only mode while another Oak
> process is running will initiate repo recovery
> --
>
> Key: OAK-9914
> URL: https://issues.apache.org/jira/browse/OAK-9914
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-azure
>Affects Versions: 1.44.0
>Reporter: Miroslav Smiljanic
>Assignee: Miroslav Smiljanic
>Priority: Major
> Attachments: AzureArchiveManager.patch, OAK-9914_test.patch
>
>
> The sequence of events:
> # Oak process with read/write file store utilizing Azure persistence is
> already running
> # New Oak process is starting up, with read-only file store using Azure
> persistence and the same storage account and container like the previous Oak
> process
> ## New Oak process starts recovery procedure for the last tar directory that
> is not closed
> Scenario presented in the attached test case:
> [^OAK-9914_test.patch]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (OAK-9911) Resource leak in ChunkedBlobStream
[
https://issues.apache.org/jira/browse/OAK-9911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding resolved OAK-9911.
-
Resolution: Fixed
> Resource leak in ChunkedBlobStream
> --
>
> Key: OAK-9911
> URL: https://issues.apache.org/jira/browse/OAK-9911
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.44.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
>
> When running the test case
> {{{color:#00}ExternalPrivateStoreIT#testSyncBigBlob{color}}} I see the
> following error logged about a resource leak:
> {noformat}
> 14:55:50.492 ERROR [standby-1] ResourceLeakDetector.java:319 LEAK:
> ByteBuf.release() was not called before it's garbage-collected. See
> https://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records:
> Created at:
>
> io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:402)
>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:174)
>
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:108)
>
> org.apache.jackrabbit.oak.segment.standby.codec.ChunkedBlobStream.readChunk(ChunkedBlobStream.java:120)
>
> org.apache.jackrabbit.oak.segment.standby.codec.ChunkedBlobStream.readChunk(ChunkedBlobStream.java:34)
>
> io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:229)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (OAK-9911) Resource leak in ChunkedBlobStream
[
https://issues.apache.org/jira/browse/OAK-9911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-9911:
Fix Version/s: 1.46.0
> Resource leak in ChunkedBlobStream
> --
>
> Key: OAK-9911
> URL: https://issues.apache.org/jira/browse/OAK-9911
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.44.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
> Fix For: 1.46.0
>
>
> When running the test case
> {{{color:#00}ExternalPrivateStoreIT#testSyncBigBlob{color}}} I see the
> following error logged about a resource leak:
> {noformat}
> 14:55:50.492 ERROR [standby-1] ResourceLeakDetector.java:319 LEAK:
> ByteBuf.release() was not called before it's garbage-collected. See
> https://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records:
> Created at:
>
> io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:402)
>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:174)
>
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:108)
>
> org.apache.jackrabbit.oak.segment.standby.codec.ChunkedBlobStream.readChunk(ChunkedBlobStream.java:120)
>
> org.apache.jackrabbit.oak.segment.standby.codec.ChunkedBlobStream.readChunk(ChunkedBlobStream.java:34)
>
> io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:229)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-9911) Resource leak in ChunkedBlobStream
[
https://issues.apache.org/jira/browse/OAK-9911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17584839#comment-17584839
]
Julian Sedding commented on OAK-9911:
-
Log disappears with the fix proposed in [PR
#677|https://github.com/apache/jackrabbit-oak/pull/677].
> Resource leak in ChunkedBlobStream
> --
>
> Key: OAK-9911
> URL: https://issues.apache.org/jira/browse/OAK-9911
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.44.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
>
> When running the test case
> {{{color:#00}ExternalPrivateStoreIT#testSyncBigBlob{color}}} I see the
> following error logged about a resource leak:
> {noformat}
> 14:55:50.492 ERROR [standby-1] ResourceLeakDetector.java:319 LEAK:
> ByteBuf.release() was not called before it's garbage-collected. See
> https://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records:
> Created at:
>
> io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:402)
>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:174)
>
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:108)
>
> org.apache.jackrabbit.oak.segment.standby.codec.ChunkedBlobStream.readChunk(ChunkedBlobStream.java:120)
>
> org.apache.jackrabbit.oak.segment.standby.codec.ChunkedBlobStream.readChunk(ChunkedBlobStream.java:34)
>
> io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:229)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (OAK-9911) Resource leak in ChunkedBlobStream
Julian Sedding created OAK-9911:
---
Summary: Resource leak in ChunkedBlobStream
Key: OAK-9911
URL: https://issues.apache.org/jira/browse/OAK-9911
Project: Jackrabbit Oak
Issue Type: Bug
Components: segment-tar
Affects Versions: 1.44.0
Reporter: Julian Sedding
Assignee: Julian Sedding
When running the test case
{{{color:#00}ExternalPrivateStoreIT#testSyncBigBlob{color}}} I see the
following error logged about a resource leak:
{noformat}
14:55:50.492 ERROR [standby-1] ResourceLeakDetector.java:319 LEAK:
ByteBuf.release() was not called before it's garbage-collected. See
https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:402)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:174)
io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:108)
org.apache.jackrabbit.oak.segment.standby.codec.ChunkedBlobStream.readChunk(ChunkedBlobStream.java:120)
org.apache.jackrabbit.oak.segment.standby.codec.ChunkedBlobStream.readChunk(ChunkedBlobStream.java:34)
io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:229)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-9905) File Store GC leaves the Store in an unconsistent state
[
https://issues.apache.org/jira/browse/OAK-9905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17584782#comment-17584782
]
Julian Sedding commented on OAK-9905:
-
[~piercarlo_s] Revision {{55db2d1f-ffb0-4c47-ab7f-eca779bb4e08}} references a
previous root segment from your journal file. As far as I know the journal file
is not truncated after a GC run. Instead, only the latest revision is appended.
This means that the journal file may well reference revisions that no longer
exist. Note: the journal is read backwards, thus the order of revisions in your
output should be the reverse of the lines in your journal file.
The difference between your code and the code in oak-run is that your code
performs "online-compaction" and oak-run performs "offline-compaction". During
online-compaction two generations of segments are kept in order not to disrupt
live code (otherwise you could see random SegmentNotFoundExceptions in your
code). Offline-compaction doesn't need to deal with this complication and thus
does not keep multiple generations. Also, it truncates the journal file.
In short: I think everything you observed works as designed.
If you disagree, please provide a failing test case that allows us to further
analyse the issue.
> File Store GC leaves the Store in an unconsistent state
> ---
>
> Key: OAK-9905
> URL: https://issues.apache.org/jira/browse/OAK-9905
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar, segmentmk
>Affects Versions: 1.42.0
>Reporter: Piercarlo Slavazza
>Priority: Blocker
> Attachments: full_gc.log
>
>
> I created a File Store using Jackrabbit Oak v. 1.42.0.
> I added some nodes, and then deleted some of them.
> I then inspect the "history" of the store, using the very same code used by
> {{oak-run}} - but executed in my application (no OSGi):
> {code:java}
> Iterator history = new
> RevisionHistory(fileStoreFolder).getHistory(
> new LocalJournalFile(journal),
> path);
> while (history.hasNext()) {
> printWriter.println(history.next().toString(depth));
> } {code}
> I got the following output:
> {code:java}
> 7b35a984-cf56-4a6d-a571-db38ac0f12aa:6={"root":{}}
> c5b06947-64ea-48b9-a0c9-dbb65fdefb82:147={"root":{}}
> 8c0036ef-0160-4f6c-a408-a956f38b421d:7653={"root":{}}
> 55db2d1f-ffb0-4c47-ab7f-eca779bb4e08:450={"root":{}}
> daadf7ca-0183-4a93-a94a-0c2347c89443:302={"root":{}}
> fd234c75-9329-41f1-a9be-f7cef1d2ae7b:371={"root":{}}
> 3e86d6a1-76a5-4983-a779-48b3ee0e98fd:6={"root":{}}
> 5d05a2d8-f239-4fa0-a643-27bd281a6573:7102={"root":{}} {code}
> that is fine - as far as I can tell.
> Then, I run {{{}fileSto{}}}{{{}re.fullGC();{}}} - log attached [^full_gc.log].
> Then, I run again the code that inspect the history, and I got the following
> output:
> {code:java}
> 10cedc98-86ed-4a48-a22b-18fe8c34ed3e:7424={"root":{}}
> 7b35a984-cf56-4a6d-a571-db38ac0f12aa:6={"root":{}}
> c5b06947-64ea-48b9-a0c9-dbb65fdefb82:147={"root":{}}
> 8c0036ef-0160-4f6c-a408-a956f38b421d:7653={"root":{}}
> Segment 55db2d1f-ffb0-4c47-ab7f-eca779bb4e08 not found{code}
> The expected behaviour is that:
> * either revision {{55db2d1f-ffb0-4c47-ab7f-eca779bb4e08}} is no more
> present in the history of the file store
> * or revision is present and can be found
> Despite this issue, it _seems_ that everything is running smoothly, but I
> cannot tell whether this issue will bug some thing at some point.
> Also note that, if I stop my application and run {{oak-run compact}} from the
> command line, the troubled revision is removed and I end up with just one
> revision:
> {code:java}
> Apache Jackrabbit Oak 1.42.0
> 07:53:00.534 [main] WARN o.a.j.oak.segment.file.JournalReader - Skipping
> invalid journal entry:
> 07:53:00.549 [main] WARN o.a.j.oak.segment.file.JournalReader - Skipping
> invalid journal entry:
> 635924e4-2e18-4bad-ac43-7347f4b1dfe0:7424={"root":{}}
> {code}
> (Please also note the two warnings printed by {{{}oak-run{}}}…).
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17584215#comment-17584215
]
Julian Sedding edited comment on OAK-9785 at 8/24/22 12:01 PM:
---
[~adulceanu] unfortunately I had a hardware failure a little while ago and I
lost the original logs. However, if you look at the last stack-trace in the
attached log file, you'll notice that [{{AbstractBlob.equals()}} wraps an
{{IOException}} in an
{{IllegalStateException}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/plugins/memory/AbstractBlob.java#L68]
before throwing it. The thread-name has then changed to
{{sling-oak-observation-7}}, but a look into
[{{SafeRunnable}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/SafeRunnable.java#L63-L74]
easily explains this. The {{SafeRunnable}}, [initialized in the
{{FileStore}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L278-L284],
would never catch an exception if compaction was properly aborted.
{noformat}
09.06.2022 02:00:18.755 *ERROR* [sling-oak-observation-7]
org.apache.jackrabbit.oak.segment.file.SafeRunnable Uncaught exception in TarMK
revision gc [/opt/aem/crx-quickstart/repository/segmentstore]
java.lang.IllegalStateException: Blob equality check failed
at
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:68)
[org.apache.jackrabbit.oak-store-spi:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentBlob.equals(SegmentBlob.java:205)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at com.google.common.base.Objects.equal(Objects.java:55)
at
org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:59)
[org.apache.jackrabbit.oak-store-spi:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentPropertyState.equals(SegmentPropertyState.java:249)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:664)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:558)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:598)
[jira] [Comment Edited] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17584215#comment-17584215
]
Julian Sedding edited comment on OAK-9785 at 8/24/22 11:59 AM:
---
[~adulceanu] unfortunately I had a hardware failure a little while ago and I
lost the original logs. However, if you look at the last stack-trace in the
attached log file, you'll notice that [{{AbstractBlob.equals()}} wraps an
{{IOException}} in an
{{IllegalStateException}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/plugins/memory/AbstractBlob.java#L68]
before throwing it. The thread-name has then changed to
{{sling-oak-observation-7}}, but a look into
[{{SafeRunnable}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/SafeRunnable.java#L63-L74]
easily explains this. The {{SafeRunnable}}, [initialized in the
{{FileStore}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L278-L284],
would never catch an exception if compaction was properly aborted.
{noformat}
09.06.2022 02:00:18.755 *ERROR* [sling-oak-observation-7]
org.apache.jackrabbit.oak.segment.file.SafeRunnable Uncaught exception in TarMK
revision gc [/opt/aem/crx-quickstart/repository/segmentstore]
java.lang.IllegalStateException: Blob equality check failed
at
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:68)
[org.apache.jackrabbit.oak-store-spi:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentBlob.equals(SegmentBlob.java:205)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at com.google.common.base.Objects.equal(Objects.java:55)
at
org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:59)
[org.apache.jackrabbit.oak-store-spi:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentPropertyState.equals(SegmentPropertyState.java:249)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:664)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:558)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:598)
[jira] [Commented] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17584215#comment-17584215
]
Julian Sedding commented on OAK-9785:
-
[~adulceanu] unfortunately I had a hardware failure a little while ago and I
lost the original logs. However, if you look at the last stack-trace in the
attached log file, you'll notice that [{{AbstractBlob.equals()}} wraps an
{{IOException}} in an
{{IllegalStateException}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9//oak-store-spi/src/main/java/org/apache/jackrabbit/oak/plugins/memory/AbstractBlob.java#L68]
before throwing it. The thread-name has then changed to
{{sling-oak-observation-7}}, but a look into
[{{SafeRunnable}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9//oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/SafeRunnable.java#L63-L74]
easily explains this. The {{SafeRunnable}}, [initialized in the
{{FileStore}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L278-L284],
would never catch an exception if compaction was properly aborted.
{noformat}
09.06.2022 02:00:18.755 *ERROR* [sling-oak-observation-7]
org.apache.jackrabbit.oak.segment.file.SafeRunnable Uncaught exception in TarMK
revision gc [/opt/aem/crx-quickstart/repository/segmentstore]
java.lang.IllegalStateException: Blob equality check failed
at
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:68)
[org.apache.jackrabbit.oak-store-spi:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentBlob.equals(SegmentBlob.java:205)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at com.google.common.base.Objects.equal(Objects.java:55)
at
org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:59)
[org.apache.jackrabbit.oak-store-spi:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentPropertyState.equals(SegmentPropertyState.java:249)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:664)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:558)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:598)
[org.apache.jackrabbit.oak-segment-tar:1.22.9]
[jira] [Resolved] (OAK-9888) Support more flexible SplitPersistence setups via OSGi
[
https://issues.apache.org/jira/browse/OAK-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding resolved OAK-9888.
-
Fix Version/s: 1.46.0
Resolution: Fixed
Fixed by merging [PR #663|https://github.com/apache/jackrabbit-oak/pull/663].
> Support more flexible SplitPersistence setups via OSGi
> --
>
> Key: OAK-9888
> URL: https://issues.apache.org/jira/browse/OAK-9888
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-azure, segment-tar
>Affects Versions: 1.44.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
> Fix For: 1.46.0
>
>
> I am working on implementing a use-case where we want to use a
> {{SplitPesistence}} that starts with a {{TarPersistence}} base "seed"
> repository and layers its mutable storage on top using an
> {{AzurePersistence}}.
> Currently, in OSGi, it is only possible to configure the reverse of this
> setup, i.e. an {{AzurePersistence}} as the base and a {{TarPersistence}} as
> the mutable layer on top.
> I propose to add a generic mechanism to configure and use a
> {{SplitPersistence}} instance in the following way:
> * support configuration of an optional "role" property for
> {{AzureSegmentStoreService}}
> * create a {{TarPersistenceService}} to allow configuring and registering a
> {{TarPersistence}} instance in OSGi, also with an optional "role" property
> * create a {{SplitPersistenceService}} to allow configuring and registering a
> {{SplitPersistence}} instance in OSGi, also with an optional "role" property
> By default the {{SplitPersistenceService}} references its read-only
> persistence with the target filter "(role=split-persistence-ro)" and its
> read-write persistence with the target filter "(role=split-persistence-rw)".
> It registers itself with "role=split-persistence".
> Finally, in order to use the {{SplitPersistence}} a
> {{SegmentNodeStoreService}} (or {{SegmentNodeStoreFactory}}) can be
> configured with "customSegmentStore=true" and
> "segmentStore.target=(role=split-persistence)" (leveraging the feature of
> OSGi Declarative Services to [configure target
> filters|https://docs.osgi.org/specification/osgi.cmpn/8.0.0/service.component.html#service.component-target.property]
> for references).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583665#comment-17583665
]
Julian Sedding commented on OAK-9785:
-
Proposed [PR #676|https://github.com/apache/jackrabbit-oak/pull/676].
> Tar SegmentStore can be corrupted during compaction
> ---
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.42.0
>Reporter: Julian Sedding
>Priority: Major
> Attachments: error.log.2022-06-09
>
>
> There is a scenario where a segment store can become corrupted, leading to
> {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the
> 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect
> cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was
> a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a
> {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially
> written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision
> data
> - the {{SegmentNotFoundException}} is first observed either during the
> remainder of the compaction run or when the respective node is requested the
> next time, usually during a traversal
> The root cause is in
> [{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
> where only {{IOExceptions}} are caught.
> In order to improve the robustness of the code, I think we need to catch all
> {{Throwables}}. Otherwise we cannot guarantee that compaction is correctly
> aborted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (OAK-9896) Running unit-tests in IntelliJ dos not work
[
https://issues.apache.org/jira/browse/OAK-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-9896:
Fix Version/s: 1.46.0
> Running unit-tests in IntelliJ dos not work
> ---
>
> Key: OAK-9896
> URL: https://issues.apache.org/jira/browse/OAK-9896
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: parent
>Affects Versions: 1.44.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Minor
> Fix For: 1.46.0
>
>
> When I try to run any unit-test in IntelliJ I get an exception that it cannot
> load the class {{{}SCRDescriptorBndPlugin{}}}.
> It is possible to work around the issue by removing this plugin from the
> configuration of the {{maven-bundle-plugin}} if, and only if, we are running
> within IntelliJ.
> {noformat}
> osgi: [oak-jackrabbit-api] Exception: java.lang.ClassNotFoundException:
> org.apache.felix.scrplugin.bnd.SCRDescriptorBndPlugin not found, parent:
> java.net.URLClassLoader@75b84c92 urls:[]
> exception:java.lang.ClassNotFoundException:
> org.apache.felix.scrplugin.bnd.SCRDescriptorBndPlugin not found in
> aQute.bnd.osgi.Processor$CL@516137d0
> at
> aQute.bnd.osgi.ActivelyClosingClassLoader.lambda$findClass$3(ActivelyClosingClassLoader.java:102)
> at java.util.Optional.orElseThrow(Optional.java:290)
> at
> aQute.bnd.osgi.ActivelyClosingClassLoader.findClass(ActivelyClosingClassLoader.java:102)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> aQute.bnd.osgi.ActivelyClosingClassLoader.loadClass(ActivelyClosingClassLoader.java:204)
> ...
> {noformat}
> A similar issue was reported with JetBrains as
> [IDEA-141732|https://youtrack.jetbrains.com/issue/IDEA-141732/OSMORC-unable-to-load-bnd-plugin].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (OAK-9896) Running unit-tests in IntelliJ dos not work
[
https://issues.apache.org/jira/browse/OAK-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding resolved OAK-9896.
-
Resolution: Fixed
Merged [PR #664|https://github.com/apache/jackrabbit-oak/pull/664].
> Running unit-tests in IntelliJ dos not work
> ---
>
> Key: OAK-9896
> URL: https://issues.apache.org/jira/browse/OAK-9896
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: parent
>Affects Versions: 1.44.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Minor
>
> When I try to run any unit-test in IntelliJ I get an exception that it cannot
> load the class {{{}SCRDescriptorBndPlugin{}}}.
> It is possible to work around the issue by removing this plugin from the
> configuration of the {{maven-bundle-plugin}} if, and only if, we are running
> within IntelliJ.
> {noformat}
> osgi: [oak-jackrabbit-api] Exception: java.lang.ClassNotFoundException:
> org.apache.felix.scrplugin.bnd.SCRDescriptorBndPlugin not found, parent:
> java.net.URLClassLoader@75b84c92 urls:[]
> exception:java.lang.ClassNotFoundException:
> org.apache.felix.scrplugin.bnd.SCRDescriptorBndPlugin not found in
> aQute.bnd.osgi.Processor$CL@516137d0
> at
> aQute.bnd.osgi.ActivelyClosingClassLoader.lambda$findClass$3(ActivelyClosingClassLoader.java:102)
> at java.util.Optional.orElseThrow(Optional.java:290)
> at
> aQute.bnd.osgi.ActivelyClosingClassLoader.findClass(ActivelyClosingClassLoader.java:102)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> aQute.bnd.osgi.ActivelyClosingClassLoader.loadClass(ActivelyClosingClassLoader.java:204)
> ...
> {noformat}
> A similar issue was reported with JetBrains as
> [IDEA-141732|https://youtrack.jetbrains.com/issue/IDEA-141732/OSMORC-unable-to-load-bnd-plugin].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-9841) Race condition in NodeStoreChecksService
[
https://issues.apache.org/jira/browse/OAK-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17581296#comment-17581296
]
Julian Sedding commented on OAK-9841:
-
[~kwin] Agreed that such a generic plugin would be better located at Apache
Felix.
It would need some thought. A very simple approach could be to mimic the naming
scheme of Declarative Services and name the property e.g.
{{.requiredObjectClasses}} and prefix it with the reference field's name. I.e.
in this case it would be {{checkers.requiredObjectClasses}}. I don't think it
can be avoided that the plugin is called for each modified configuration
dictionary. It would need to iterate over all keys to check for the suffix
{{.requiredObjectClasses}} and only add the other properties if such a property
is found. Of course the {{.target}} filter allows for very flexible checks, so
a plugin could either try to be clever and somehow capture a more complete
subset of its functionality. Or it targets only the most common use case, which
I think is what we are trying to achieve here.
The next challenge might be to get the services injected in the correct order
;) It's not always relevant, but it should never hurt. I have no idea if that's
possible without custom code inside the component.
> Race condition in NodeStoreChecksService
>
>
> Key: OAK-9841
> URL: https://issues.apache.org/jira/browse/OAK-9841
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: composite
>Reporter: Konrad Windszus
>Priority: Major
>
> The {{NodeStoreChecksService}}
> (https://github.com/apache/jackrabbit-oak/blob/bbc141fd1fb9ff0d9ce742279445df9eb698c3e3/oak-store-composite/src/main/java/org/apache/jackrabbit/oak/composite/checks/NodeStoreChecksService.java#L41)
> executes all bound {{MountedNodeStoreChecker}} s which have been there at
> the time of activation.
> The references are not greedily referenced and also there is no wait for
> specific {{MountedNodeStoreChecker}} services to be active.
> That leads to the fact that the usage of {{NodeStoreChecksService}} in
> {{CompositeNodeStoreService.registerCompositeNodeStore(...)}} is not
> deterministic as the starting order of OSGi services determine which
> {{MountedNodeStoreChecker}} are active during
> {{CompositeNodeStoreService.activate()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (OAK-9841) Race condition in NodeStoreChecksService
[
https://issues.apache.org/jira/browse/OAK-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17581277#comment-17581277
]
Julian Sedding edited comment on OAK-9841 at 8/18/22 10:33 AM:
---
[~kwin] the problem of configuring required services seems to be recurring. I
wonder if it would be possible to intercept the {{requiredCheckers}} (or indeed
{{requiredServices}}) in a generic way and use the contained information to
automatically set the reference's {{.target}} and {{.cardinality.minimum}}
properties. Just a thought, I haven't looked into the feasibility of this
approach.
_edit:_ it looks like this should be feasible with a
[ConfigurationPlugin|https://docs.osgi.org/specification/osgi.cmpn/8.0.0/service.cm.html#i1459884].
was (Author: jsedding):
[~kwin] the problem of configuring required services seems to be recurring. I
wonder if it would be possible to intercept the {{requiredCheckers}} (or indeed
{{requiredServices}}) in a generic way and use the contained information to
automatically set the reference's {{.target}} and {{.cardinality.minimum}}
properties. Just a thought, I haven't looked into the feasibility of this
approach.
> Race condition in NodeStoreChecksService
>
>
> Key: OAK-9841
> URL: https://issues.apache.org/jira/browse/OAK-9841
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: composite
>Reporter: Konrad Windszus
>Priority: Major
>
> The {{NodeStoreChecksService}}
> (https://github.com/apache/jackrabbit-oak/blob/bbc141fd1fb9ff0d9ce742279445df9eb698c3e3/oak-store-composite/src/main/java/org/apache/jackrabbit/oak/composite/checks/NodeStoreChecksService.java#L41)
> executes all bound {{MountedNodeStoreChecker}} s which have been there at
> the time of activation.
> The references are not greedily referenced and also there is no wait for
> specific {{MountedNodeStoreChecker}} services to be active.
> That leads to the fact that the usage of {{NodeStoreChecksService}} in
> {{CompositeNodeStoreService.registerCompositeNodeStore(...)}} is not
> deterministic as the starting order of OSGi services determine which
> {{MountedNodeStoreChecker}} are active during
> {{CompositeNodeStoreService.activate()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-9841) Race condition in NodeStoreChecksService
[
https://issues.apache.org/jira/browse/OAK-9841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17581277#comment-17581277
]
Julian Sedding commented on OAK-9841:
-
[~kwin] the problem of configuring required services seems to be recurring. I
wonder if it would be possible to intercept the {{requiredCheckers}} (or indeed
{{requiredServices}}) in a generic way and use the contained information to
automatically set the reference's {{.target}} and {{.cardinality.minimum}}
properties. Just a thought, I haven't looked into the feasibility of this
approach.
> Race condition in NodeStoreChecksService
>
>
> Key: OAK-9841
> URL: https://issues.apache.org/jira/browse/OAK-9841
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: composite
>Reporter: Konrad Windszus
>Priority: Major
>
> The {{NodeStoreChecksService}}
> (https://github.com/apache/jackrabbit-oak/blob/bbc141fd1fb9ff0d9ce742279445df9eb698c3e3/oak-store-composite/src/main/java/org/apache/jackrabbit/oak/composite/checks/NodeStoreChecksService.java#L41)
> executes all bound {{MountedNodeStoreChecker}} s which have been there at
> the time of activation.
> The references are not greedily referenced and also there is no wait for
> specific {{MountedNodeStoreChecker}} services to be active.
> That leads to the fact that the usage of {{NodeStoreChecksService}} in
> {{CompositeNodeStoreService.registerCompositeNodeStore(...)}} is not
> deterministic as the starting order of OSGi services determine which
> {{MountedNodeStoreChecker}} are active during
> {{CompositeNodeStoreService.activate()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (OAK-9897) SplitPersistence: FileReaper cannot finish cleanup
Julian Sedding created OAK-9897:
---
Summary: SplitPersistence: FileReaper cannot finish cleanup
Key: OAK-9897
URL: https://issues.apache.org/jira/browse/OAK-9897
Project: Jackrabbit Oak
Issue Type: Bug
Components: segment-tar
Affects Versions: 1.44.0
Reporter: Julian Sedding
Assignee: Julian Sedding
When running revision compaction (aka GC) on a setup with split persistence, it
is possible that the "cleanup" phase identifies files from the read-only part
of the persistence as redundant and submits them to the {{FileReaper}} for
deletion.
However, the delete method of the {{SplitSegmentArchiveManager}} is implemented
to return "false" when attempting the deletion of a file from the read-only
persistence (which I would argue is correct). The {{FileReaper}} then re-adds
the offending file to its queue in order to retry deleting it. Of course this
fails again and so it can never finish and logs warnings for the files it
cannot delete.
It should be possible to prevent deletion of read-only files in the first
place. In fact, I think there is no need to mark and sweep them at all.
cc [~adulceanu]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (OAK-9896) Running unit-tests in IntelliJ dos not work
[
https://issues.apache.org/jira/browse/OAK-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-9896:
Description:
When I try to run any unit-test in IntelliJ I get an exception that it cannot
load the class {{{}SCRDescriptorBndPlugin{}}}.
It is possible to work around the issue by removing this plugin from the
configuration of the {{maven-bundle-plugin}} if, and only if, we are running
within IntelliJ.
{noformat}
osgi: [oak-jackrabbit-api] Exception: java.lang.ClassNotFoundException:
org.apache.felix.scrplugin.bnd.SCRDescriptorBndPlugin not found, parent:
java.net.URLClassLoader@75b84c92 urls:[]
exception:java.lang.ClassNotFoundException:
org.apache.felix.scrplugin.bnd.SCRDescriptorBndPlugin not found in
aQute.bnd.osgi.Processor$CL@516137d0
at
aQute.bnd.osgi.ActivelyClosingClassLoader.lambda$findClass$3(ActivelyClosingClassLoader.java:102)
at java.util.Optional.orElseThrow(Optional.java:290)
at
aQute.bnd.osgi.ActivelyClosingClassLoader.findClass(ActivelyClosingClassLoader.java:102)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
aQute.bnd.osgi.ActivelyClosingClassLoader.loadClass(ActivelyClosingClassLoader.java:204)
...
{noformat}
A similar issue was reported with JetBrains as
[IDEA-141732|https://youtrack.jetbrains.com/issue/IDEA-141732/OSMORC-unable-to-load-bnd-plugin].
was:
When I try to run any unit-test in IntelliJ I get an exception that it cannot
load the class {{{}SCRDescriptorBndPlugin}}.
It is possible to work around the issue by removing this plugin from the
configuration of the {{maven-bundle-plugin}} if, and only if, we are running
within IntelliJ.
{noformat}
osgi: [oak-jackrabbit-api] Exception: java.lang.ClassNotFoundException:
org.apache.felix.scrplugin.bnd.SCRDescriptorBndPlugin not found, parent:
java.net.URLClassLoader@75b84c92 urls:[]
exception:java.lang.ClassNotFoundException:
org.apache.felix.scrplugin.bnd.SCRDescriptorBndPlugin not found in
aQute.bnd.osgi.Processor$CL@516137d0
at
aQute.bnd.osgi.ActivelyClosingClassLoader.lambda$findClass$3(ActivelyClosingClassLoader.java:102)
at java.util.Optional.orElseThrow(Optional.java:290)
at
aQute.bnd.osgi.ActivelyClosingClassLoader.findClass(ActivelyClosingClassLoader.java:102)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
aQute.bnd.osgi.ActivelyClosingClassLoader.loadClass(ActivelyClosingClassLoader.java:204)
...
{noformat}
> Running unit-tests in IntelliJ dos not work
> ---
>
> Key: OAK-9896
> URL: https://issues.apache.org/jira/browse/OAK-9896
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: parent
>Affects Versions: 1.44.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Minor
>
> When I try to run any unit-test in IntelliJ I get an exception that it cannot
> load the class {{{}SCRDescriptorBndPlugin{}}}.
> It is possible to work around the issue by removing this plugin from the
> configuration of the {{maven-bundle-plugin}} if, and only if, we are running
> within IntelliJ.
> {noformat}
> osgi: [oak-jackrabbit-api] Exception: java.lang.ClassNotFoundException:
> org.apache.felix.scrplugin.bnd.SCRDescriptorBndPlugin not found, parent:
> java.net.URLClassLoader@75b84c92 urls:[]
> exception:java.lang.ClassNotFoundException:
> org.apache.felix.scrplugin.bnd.SCRDescriptorBndPlugin not found in
> aQute.bnd.osgi.Processor$CL@516137d0
> at
> aQute.bnd.osgi.ActivelyClosingClassLoader.lambda$findClass$3(ActivelyClosingClassLoader.java:102)
> at java.util.Optional.orElseThrow(Optional.java:290)
> at
> aQute.bnd.osgi.ActivelyClosingClassLoader.findClass(ActivelyClosingClassLoader.java:102)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> aQute.bnd.osgi.ActivelyClosingClassLoader.loadClass(ActivelyClosingClassLoader.java:204)
> ...
> {noformat}
> A similar issue was reported with JetBrains as
> [IDEA-141732|https://youtrack.jetbrains.com/issue/IDEA-141732/OSMORC-unable-to-load-bnd-plugin].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (OAK-9896) Running unit-tests in IntelliJ dos not work
Julian Sedding created OAK-9896:
---
Summary: Running unit-tests in IntelliJ dos not work
Key: OAK-9896
URL: https://issues.apache.org/jira/browse/OAK-9896
Project: Jackrabbit Oak
Issue Type: Improvement
Components: parent
Affects Versions: 1.44.0
Reporter: Julian Sedding
Assignee: Julian Sedding
When I try to run any unit-test in IntelliJ I get an exception that it cannot
load the class {{{}SCRDescriptorBndPlugin}}.
It is possible to work around the issue by removing this plugin from the
configuration of the {{maven-bundle-plugin}} if, and only if, we are running
within IntelliJ.
{noformat}
osgi: [oak-jackrabbit-api] Exception: java.lang.ClassNotFoundException:
org.apache.felix.scrplugin.bnd.SCRDescriptorBndPlugin not found, parent:
java.net.URLClassLoader@75b84c92 urls:[]
exception:java.lang.ClassNotFoundException:
org.apache.felix.scrplugin.bnd.SCRDescriptorBndPlugin not found in
aQute.bnd.osgi.Processor$CL@516137d0
at
aQute.bnd.osgi.ActivelyClosingClassLoader.lambda$findClass$3(ActivelyClosingClassLoader.java:102)
at java.util.Optional.orElseThrow(Optional.java:290)
at
aQute.bnd.osgi.ActivelyClosingClassLoader.findClass(ActivelyClosingClassLoader.java:102)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
aQute.bnd.osgi.ActivelyClosingClassLoader.loadClass(ActivelyClosingClassLoader.java:204)
...
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (OAK-9888) Support more flexible SplitPersistence setups via OSGi
[
https://issues.apache.org/jira/browse/OAK-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-9888:
Issue Type: New Feature (was: Story)
> Support more flexible SplitPersistence setups via OSGi
> --
>
> Key: OAK-9888
> URL: https://issues.apache.org/jira/browse/OAK-9888
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-azure, segment-tar
>Affects Versions: 1.44.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
>
> I am working on implementing a use-case where we want to use a
> {{SplitPesistence}} that starts with a {{TarPersistence}} base "seed"
> repository and layers its mutable storage on top using an
> {{AzurePersistence}}.
> Currently, in OSGi, it is only possible to configure the reverse of this
> setup, i.e. an {{AzurePersistence}} as the base and a {{TarPersistence}} as
> the mutable layer on top.
> I propose to add a generic mechanism to configure and use a
> {{SplitPersistence}} instance in the following way:
> * support configuration of an optional "role" property for
> {{AzureSegmentStoreService}}
> * create a {{TarPersistenceService}} to allow configuring and registering a
> {{TarPersistence}} instance in OSGi, also with an optional "role" property
> * create a {{SplitPersistenceService}} to allow configuring and registering a
> {{SplitPersistence}} instance in OSGi, also with an optional "role" property
> By default the {{SplitPersistenceService}} references its read-only
> persistence with the target filter "(role=split-persistence-ro)" and its
> read-write persistence with the target filter "(role=split-persistence-rw)".
> It registers itself with "role=split-persistence".
> Finally, in order to use the {{SplitPersistence}} a
> {{SegmentNodeStoreService}} (or {{SegmentNodeStoreFactory}}) can be
> configured with "customSegmentStore=true" and
> "segmentStore.target=(role=split-persistence)" (leveraging the feature of
> OSGi Declarative Services to [configure target
> filters|https://docs.osgi.org/specification/osgi.cmpn/8.0.0/service.component.html#service.component-target.property]
> for references).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (OAK-9888) Support more flexible SplitPersistence setups via OSGi
[
https://issues.apache.org/jira/browse/OAK-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578021#comment-17578021
]
Julian Sedding commented on OAK-9888:
-
[PR #663|https://github.com/apache/jackrabbit-oak/pull/663] implements the
required changes.
> Support more flexible SplitPersistence setups via OSGi
> --
>
> Key: OAK-9888
> URL: https://issues.apache.org/jira/browse/OAK-9888
> Project: Jackrabbit Oak
> Issue Type: Story
> Components: segment-azure, segment-tar
>Affects Versions: 1.44.0
>Reporter: Julian Sedding
>Assignee: Julian Sedding
>Priority: Major
>
> I am working on implementing a use-case where we want to use a
> {{SplitPesistence}} that starts with a {{TarPersistence}} base "seed"
> repository and layers its mutable storage on top using an
> {{AzurePersistence}}.
> Currently, in OSGi, it is only possible to configure the reverse of this
> setup, i.e. an {{AzurePersistence}} as the base and a {{TarPersistence}} as
> the mutable layer on top.
> I propose to add a generic mechanism to configure and use a
> {{SplitPersistence}} instance in the following way:
> * support configuration of an optional "role" property for
> {{AzureSegmentStoreService}}
> * create a {{TarPersistenceService}} to allow configuring and registering a
> {{TarPersistence}} instance in OSGi, also with an optional "role" property
> * create a {{SplitPersistenceService}} to allow configuring and registering a
> {{SplitPersistence}} instance in OSGi, also with an optional "role" property
> By default the {{SplitPersistenceService}} references its read-only
> persistence with the target filter "(role=split-persistence-ro)" and its
> read-write persistence with the target filter "(role=split-persistence-rw)".
> It registers itself with "role=split-persistence".
> Finally, in order to use the {{SplitPersistence}} a
> {{SegmentNodeStoreService}} (or {{SegmentNodeStoreFactory}}) can be
> configured with "customSegmentStore=true" and
> "segmentStore.target=(role=split-persistence)" (leveraging the feature of
> OSGi Declarative Services to [configure target
> filters|https://docs.osgi.org/specification/osgi.cmpn/8.0.0/service.component.html#service.component-target.property]
> for references).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (OAK-9888) Support more flexible SplitPersistence setups via OSGi
Julian Sedding created OAK-9888:
---
Summary: Support more flexible SplitPersistence setups via OSGi
Key: OAK-9888
URL: https://issues.apache.org/jira/browse/OAK-9888
Project: Jackrabbit Oak
Issue Type: Story
Components: segment-azure, segment-tar
Affects Versions: 1.44.0
Reporter: Julian Sedding
Assignee: Julian Sedding
I am working on implementing a use-case where we want to use a
{{SplitPesistence}} that starts with a {{TarPersistence}} base "seed"
repository and layers its mutable storage on top using an {{AzurePersistence}}.
Currently, in OSGi, it is only possible to configure the reverse of this setup,
i.e. an {{AzurePersistence}} as the base and a {{TarPersistence}} as the
mutable layer on top.
I propose to add a generic mechanism to configure and use a
{{SplitPersistence}} instance in the following way:
* support configuration of an optional "role" property for
{{AzureSegmentStoreService}}
* create a {{TarPersistenceService}} to allow configuring and registering a
{{TarPersistence}} instance in OSGi, also with an optional "role" property
* create a {{SplitPersistenceService}} to allow configuring and registering a
{{SplitPersistence}} instance in OSGi, also with an optional "role" property
By default the {{SplitPersistenceService}} references its read-only persistence
with the target filter "(role=split-persistence-ro)" and its read-write
persistence with the target filter "(role=split-persistence-rw)". It registers
itself with "role=split-persistence".
Finally, in order to use the {{SplitPersistence}} a {{SegmentNodeStoreService}}
(or {{SegmentNodeStoreFactory}}) can be configured with
"customSegmentStore=true" and "segmentStore.target=(role=split-persistence)"
(leveraging the feature of OSGi Declarative Services to [configure target
filters|https://docs.osgi.org/specification/osgi.cmpn/8.0.0/service.component.html#service.component-target.property]
for references).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[
https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-9785:
Description:
There is a scenario where a segment store can become corrupted, leading to
{{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the 1-2
digit millisecond range. E.g. {{SegmentId age=2ms}}.
The scenario I observed looks as follows:
- a blob is "lost" from the external blob store (presumably due to incorrect
cloning of the instance, most likely only happens with unfortunate timing)
- a tail revision GC run is performed (not sure if it matters that this was a
tail compaction)
-- the missing blob is encountered during compaction
-- an exception other than an {{IOException}} (IIRC it was a
{{{}IllegalArgumentException{}}}) is thrown due to the missing blob
-- revision GC fails WITHOUT properly being aborted, and thus the partially
written revision of the compaction run is not removed
- more data is written on the instance
- a full revision GC run is performed
-- a referenced segment is removed due to the incorrect/confused revision data
- the {{SegmentNotFoundException}} is first observed either during the
remainder of the compaction run or when the respective node is requested the
next time, usually during a traversal
The root cause is in
[{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
where only {{IOExceptions}} are caught.
In order to improve the robustness of the code, I think we need to catch all
{{Throwables}}. Otherwise we cannot guarantee that compaction is correctly
aborted.
was:
There is a scenario where a segment store can become corrupted, leading to
{{{}SegmentNotFoundException}}s with very "young" \{{SegmentId}}s, i.e. in the
1-2 digit millisecond range. E.g. {{SegmentId age=2ms{}}}.
The scenario I observed looks as follows:
- a blob is "lost" from the external blob store (presumably due to incorrect
cloning of the instance, most likely only happens with unfortunate timing)
- a tail revision GC run is performed (not sure if it matters that this was a
tail compaction)
-- the missing blob is encountered during compaction
-- an exception other than an {{IOException}} (IIRC it was a
{{{}IllegalArgumentException{}}}) is thrown due to the missing blob
-- revision GC fails WITHOUT properly being aborted, and thus the partially
written revision of the compaction run is not removed
- more data is written on the instance
- a full revision GC run is performed
-- a referenced segment is removed due to the incorrect/confused revision data
- the {{SegmentNotFoundException}} is first observed either during the
remainder of the compaction run or when the respective node is requested the
next time, usually during a traversal
The root cause is in
[{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
where only \{{IOException}}s are caught.
In order to improve the robustness of the code, I think we need to catch all
\{{Throwable}}s. Otherwise we cannot guarantee that compaction is correctly
aborted.
> Tar SegmentStore can be corrupted during compaction
> ---
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: 1.42.0
>Reporter: Julian Sedding
>Priority: Major
>
> There is a scenario where a segment store can become corrupted, leading to
> {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the
> 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect
> cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was
> a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a
> {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially
> written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision
> data
> - the {{SegmentNotFoundException}} is first observed either during the
> remainder of the compaction run or when the respective node is requested the
> next time,
[jira] [Created] (OAK-9785) Tar SegmentStore can be corrupted during compaction
Julian Sedding created OAK-9785:
---
Summary: Tar SegmentStore can be corrupted during compaction
Key: OAK-9785
URL: https://issues.apache.org/jira/browse/OAK-9785
Project: Jackrabbit Oak
Issue Type: Bug
Components: segment-tar
Affects Versions: 1.42.0
Reporter: Julian Sedding
There is a scenario where a segment store can become corrupted, leading to
{{{}SegmentNotFoundException}}s with very "young" \{{SegmentId}}s, i.e. in the
1-2 digit millisecond range. E.g. {{SegmentId age=2ms{}}}.
The scenario I observed looks as follows:
- a blob is "lost" from the external blob store (presumably due to incorrect
cloning of the instance, most likely only happens with unfortunate timing)
- a tail revision GC run is performed (not sure if it matters that this was a
tail compaction)
-- the missing blob is encountered during compaction
-- an exception other than an {{IOException}} (IIRC it was a
{{{}IllegalArgumentException{}}}) is thrown due to the missing blob
-- revision GC fails WITHOUT properly being aborted, and thus the partially
written revision of the compaction run is not removed
- more data is written on the instance
- a full revision GC run is performed
-- a referenced segment is removed due to the incorrect/confused revision data
- the {{SegmentNotFoundException}} is first observed either during the
remainder of the compaction run or when the respective node is requested the
next time, usually during a traversal
The root cause is in
[{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233],
where only \{{IOException}}s are caught.
In order to improve the robustness of the code, I think we need to catch all
\{{Throwable}}s. Otherwise we cannot guarantee that compaction is correctly
aborted.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (OAK-8048) VersionHistory not removed when removing node and all its versions
[
https://issues.apache.org/jira/browse/OAK-8048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17390418#comment-17390418
]
Julian Sedding commented on OAK-8048:
-
Thanks [~mreutegg]. I wrote a little test case to try this workaround (see
[^OAK-8048-remove-version-history-workaround-test.patch]), because I was
wondering if setting the protected {{jcr:uuid}} property would cause any
issues. It turns out that it is important to set the {{jcr:uuid}} before adding
the {{mix:versionable}} mixin. If these two lines are reversed, an exception is
thrown due to {{jcr:uuid}} being protected. So I can confirm the workaround you
suggest works.
> VersionHistory not removed when removing node and all its versions
> --
>
> Key: OAK-8048
> URL: https://issues.apache.org/jira/browse/OAK-8048
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
>Affects Versions: 1.8.9
>Reporter: Marco Piovesana
>Assignee: Manfred Baedke
>Priority: Major
> Attachments: OAK-8048-remove-version-history-workaround-test.patch,
> OAK-8048-test.diff, OAK-8048.diff, fix-OAK-8048.patch
>
>
> Hi all,
> I'm trying to delete a node and all its versions, but the version history is
> not removed. I'm doing the following steps (as described in OAK-4370 and
> JCR-134):
> # retrieve the version history
> # delete the node and save the session
> # delete all versions except for the base version
> # save the session
> The versions are all gone but the versionHistory node, and the base version
> node, are still there. Am I doing something wrong?
> The only test related to this that I found is
> {{ReadOnlyVersionManagerTest.testRemoveEmptyHistoryAfterRemovingVersionable}}.
> It does work, but uses Oak related classes and not the JCR interface.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (OAK-8048) VersionHistory not removed when removing node and all its versions
[
https://issues.apache.org/jira/browse/OAK-8048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-8048:
Attachment: OAK-8048-remove-version-history-workaround-test.patch
> VersionHistory not removed when removing node and all its versions
> --
>
> Key: OAK-8048
> URL: https://issues.apache.org/jira/browse/OAK-8048
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
>Affects Versions: 1.8.9
>Reporter: Marco Piovesana
>Assignee: Manfred Baedke
>Priority: Major
> Attachments: OAK-8048-remove-version-history-workaround-test.patch,
> OAK-8048-test.diff, OAK-8048.diff, fix-OAK-8048.patch
>
>
> Hi all,
> I'm trying to delete a node and all its versions, but the version history is
> not removed. I'm doing the following steps (as described in OAK-4370 and
> JCR-134):
> # retrieve the version history
> # delete the node and save the session
> # delete all versions except for the base version
> # save the session
> The versions are all gone but the versionHistory node, and the base version
> node, are still there. Am I doing something wrong?
> The only test related to this that I found is
> {{ReadOnlyVersionManagerTest.testRemoveEmptyHistoryAfterRemovingVersionable}}.
> It does work, but uses Oak related classes and not the JCR interface.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OAK-8048) VersionHistory not removed when removing node and all its versions
[
https://issues.apache.org/jira/browse/OAK-8048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389790#comment-17389790
]
Julian Sedding commented on OAK-8048:
-
I have also encountered the lack of support for removing empty and orphaned
version-histories. While the {{OrphanedVersionCleaner}} handles automatic
removal of an empty version-history upon removal of the versionable node, the
other part of the equation is missing. Removing empty, orphaned
version-histories. This ticket attempts to address this part IIUC. +1
Additionally, it would be desirable to remove any empty intermediate nodes in
the version storage hierarchy when version-histories get deleted. There are 3
levels of hierarchy with names made up of 2 hex digits each. I.e. there can be
up to 256 * 256 * 256 (aka 16'777'216) nodes of "waste". IMHO that's worth
keeping tidy.
[~mreutegg] You wrote the following (and I realise it's been over a year ago):
bq. Creating a versionable node with a UUID as recorded in the empty version
history will link the new node to the empty version history.
I haven't tried this, but do you mean to create a new node, add the
{{mix:versionable}} mixin and set its {{jcr:versionHistory}} property to
reference the orphaned version history? Save that. Aand then delete the newly
created node again to trigger the {{OrphanedVersionCleaner}}?
> VersionHistory not removed when removing node and all its versions
> --
>
> Key: OAK-8048
> URL: https://issues.apache.org/jira/browse/OAK-8048
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
>Affects Versions: 1.8.9
>Reporter: Marco Piovesana
>Assignee: Manfred Baedke
>Priority: Major
> Attachments: OAK-8048-test.diff, OAK-8048.diff, fix-OAK-8048.patch
>
>
> Hi all,
> I'm trying to delete a node and all its versions, but the version history is
> not removed. I'm doing the following steps (as described in OAK-4370 and
> JCR-134):
> # retrieve the version history
> # delete the node and save the session
> # delete all versions except for the base version
> # save the session
> The versions are all gone but the versionHistory node, and the base version
> node, are still there. Am I doing something wrong?
> The only test related to this that I found is
> {{ReadOnlyVersionManagerTest.testRemoveEmptyHistoryAfterRemovingVersionable}}.
> It does work, but uses Oak related classes and not the JCR interface.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (OAK-9515) IndexCopier$DeleteOldDirOnClose#close() should handle missing file
Julian Sedding created OAK-9515:
---
Summary: IndexCopier$DeleteOldDirOnClose#close() should handle
missing file
Key: OAK-9515
URL: https://issues.apache.org/jira/browse/OAK-9515
Project: Jackrabbit Oak
Issue Type: Bug
Components: lucene
Affects Versions: 1.22.4
Reporter: Julian Sedding
{{IndexCopier$DeleteOldDirOnClose#close()}} catches {{IOException}}, however,
it uses {{FileUtils#sizeOf(File)}}, which throws an
{{IllegalArgumentException}} if the file does not exist.
{noformat}
22.07.2021 10:23:21.166 *WARN* [oak-lucene-5018]
org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProviderService Error
occurred in asynchronous processing
java.lang.IllegalArgumentException:
/path/to/repository/index/indexname-1626941798326/data does not exist
at org.apache.commons.io.FileUtils.sizeOf(FileUtils.java:2541)
[org.apache.commons.io:2.6.0]
at
org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$DeleteOldDirOnClose.close(IndexCopier.java:468)
[org.apache.jackrabbit.oak-lucene:1.22.4]
at
org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory$2.run(CopyOnReadDirectory.java:304)
[org.apache.jackrabbit.oak-lucene:1.22.4]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
The [relevant block of
code|https://github.com/apache/jackrabbit-oak/blob/c6ddcc55bee3de915459af01e91edad32d538f3d/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/IndexCopier.java#L465-L475]
attempts to delete and old index directory. So skipping this if the directory
does not exist seems appropriate.
Unfortunately the log files I analysed don't reach far back into the past, so I
cannot say under which circumstances it can happen that the directory is
already removed. What I know is that a few days later an out-of-disk situation
was noticed, which was likely caused by repeated re-indexing runs. Furthermore,
I have seen a {{SegmentNotFoundException}} for one specific segment. Once after
a "successful" reindexing run (i.e. {{IndexUpdate}} reports "Reindexing
completed"), during {{AsyncIndexUpdate#mergeWithConcurrencyCheck()}}. But most
occurrences look like queries running into the missing segment.
Surely the issue described here is not the root-cause of the problem I am
analysing. But at the very least it adds unnecessary noise.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OAK-9159) Add test support to NodeStoreTask
[
https://issues.apache.org/jira/browse/OAK-9159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226046#comment-17226046
]
Julian Sedding commented on OAK-9159:
-
[~stefanegli] fine with me!
> Add test support to NodeStoreTask
> -
>
> Key: OAK-9159
> URL: https://issues.apache.org/jira/browse/OAK-9159
> Project: Jackrabbit Oak
> Issue Type: Test
> Components: documentmk
>Affects Versions: 1.32.0
>Reporter: Stefan Egli
>Priority: Minor
>
> As suggested by [~jsedding] in [this PR review
> comment|https://github.com/apache/jackrabbit-oak/pull/243#discussion_r462359594]
> the newly introduced (test method)
> {{DocumentNodeStore.stopBackgroundUpdateThread}} could be more elegantly
> implemented via a thread-factory approach. We should look into refactoring
> that code along those lines, if feasible extending the approach to the other
> background tasks too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OAK-9159) Add test support to NodeStoreTask
[
https://issues.apache.org/jira/browse/OAK-9159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170020#comment-17170020
]
Julian Sedding commented on OAK-9159:
-
[~stefanegli], it is upon me to apologise. It had escaped my attention that the
method was package private. So there shouldn't be any API relevance.
> Add test support to NodeStoreTask
> -
>
> Key: OAK-9159
> URL: https://issues.apache.org/jira/browse/OAK-9159
> Project: Jackrabbit Oak
> Issue Type: Test
> Components: documentmk
>Affects Versions: 1.32.0
>Reporter: Stefan Egli
>Priority: Major
>
> As suggested by [~jsedding] in [this PR review
> comment|https://github.com/apache/jackrabbit-oak/pull/243#discussion_r462359594]
> the newly introduced (test method)
> {{DocumentNodeStore.stopBackgroundUpdateThread}} could be more elegantly
> implemented via a thread-factory approach. We should look into refactoring
> that code along those lines, if feasible extending the approach to the other
> background tasks too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OAK-9159) Add test support to NodeStoreTask
[
https://issues.apache.org/jira/browse/OAK-9159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168002#comment-17168002
]
Julian Sedding commented on OAK-9159:
-
Thanks for opening this ticket [~stefanegli]. First of all, please note that
I'm not attached to any particular outcome to this ticket.
IIUC, the addition of the method
{{DocumentNodeStore.stopBackgroundUpdateThread}} to the API implies that once
released, it cannot be simply removed. So for this ticket to be valuable, this
method would need to be removed again before the current committed state is
released. I cannot say if this (hopefully) small refactoring would provide any
other benefits beyond avoiding an API change.
> Add test support to NodeStoreTask
> -
>
> Key: OAK-9159
> URL: https://issues.apache.org/jira/browse/OAK-9159
> Project: Jackrabbit Oak
> Issue Type: Test
> Components: documentmk
>Affects Versions: 1.32.0
>Reporter: Stefan Egli
>Priority: Major
>
> As suggested by [~jsedding] in [this PR review
> comment|https://github.com/apache/jackrabbit-oak/pull/243#discussion_r462359594]
> the newly introduced (test method)
> {{DocumentNodeStore.stopBackgroundUpdateThread}} could be more elegantly
> implemented via a thread-factory approach. We should look into refactoring
> that code along those lines, if feasible extending the approach to the other
> background tasks too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (OAK-9024) oak-solr-osgi imports org.slf4j.impl
[
https://issues.apache.org/jira/browse/OAK-9024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103950#comment-17103950
]
Julian Sedding edited comment on OAK-9024 at 5/10/20, 9:17 PM:
---
My bottom line is: *{{oak-solr-osgi}} MUST import {{org.slf4j.impl}}* in order
for the logging to work correctly.
[~baedke], I get the impression (and I don't mean this personally, so my
apologies if I am wrong) that you don't fully separate the Maven class-path and
the bundle's class-path in OSGi. The bnd tool (here in the form of the
maven-bundle-plugin IIRC) is concerned with making sure that the bundle has a
consistent class-space. That consists of two major parts: (a) all packages (and
classes) inside the bundle and (b) all packages (and classes) that need to be
provided to the bundle from the outside, i.e. everything listed in
{{Import-Package}}.
The bnd tool computes a graph of dependencies between classes using static
byte-code analysis. That means it just uses what it can find on the class-path,
but does not care about dependencies defined in Maven otherwise. Notably, it
ignores available Java packages for the {{Import-Package}} computation, if they
are not statically reachable via the classes inside the bundle (i.e. they
aren't needed). This mechanism would obviously miss any classes that are loaded
manually using a String class-name.
A consequence of embedding the {{slf4j-log4j12}} artifact is that on the
package level you embed {{org.slf4j.impl}} (and others I believe). Since that
package is now inside the bundle, bnd can remove it from the {{Import-Package}}
header. The same can be achieved by embedding any other artifact that contains
{{org.slf4j.impl}}, of which there are quite a few.
Now the but. But slf4j's contract for bindings seems to be that each binding
defines {{org.slf4j.impl.StaticLoggerBinder}}, which essentially makes this an
SPI. In Sling-based setups there is an {{slf4j.api}} bundle and an
{{org.apache.sling.commons.log}} bundle (the binding used in Sling). Notably,
the {{slf4j.api}} bundle imports the package {{org.slf4j.impl}} from
{{org.apache.sling.commons.log}}. That is because essentially
{{org.slf4j.impl}} is the SPI that SLF4J uses to connect the logging API with a
binding.
Furthermore, I am under the impression that SLF4J only supports a single
binding in a deployment. This would make embedding {{slf4j-log4j12}} in
{{oak-solr-osgi}} problematic, because (a) it would introduce a second binding
in Sling-based setups and (b) it would eliminate the possibility to swap out
logging bindings. Regarding (a) my guess would be that at least log messages
might get lost (there's no log4j present that would actually write the logs) or
possibly more obvious errors.
I think it is a little bit odd for SLF4J to call the package that hooks in the
bindings {{impl}}. However, once we ignore this suggestive name, there is no
reason why {{oak-solr-osgi}} should not import {{org.slf4j.impl}}. (Of course
there would be good reasons for Solr to avoid directly calling into SLF4J's
bindings SPI, but that's out-of-scope for Oak).
So that's why my bottom line is: {{oak-solr-osgi}} MUST import
{{org.slf4j.impl}} in order for the logging to work correctly.
Still, I think it would be better to control the import version range for
{{org.slf4j.impl}}. At the moment, I believe that it just "happens" to be
whatever we get transitively from Zookeeper or Solr or elsewhere. Controlling
the version range could be achieved by declaring a provided dependency on e.g.
{{slf4j-simple}}. Maybe with a comment that it is only used to help bnd compute
the correct version range.
cc [~reschke] as you created the ticket, maybe there was an issue with this
import that is not stated in the description.
was (Author: jsedding):
My bottom line is: *{{oak-solr-osgi}} MUST import {{org.slf4j.impl}}* in order
for the logging to work correctly.
[~baedke], I get the impression (and I don't mean this personally, so my
apologies if I am wrong) that you don't fully separate the Maven class-path and
the bundle's class-path in OSGi. The bnd tool (here in the form of the
maven-bundle-plugin IIRC) is concerned with making sure that the bundle has a
consistent class-space. That consists of two major parts: (a) all packages (and
classes) inside the bundle and (b) all packages (and classes) that need to be
provided to the bundle from the outside, i.e. everything listed in
{{Import-Package}}.
The bnd tool computes a graph of dependencies between classes using static
byte-code analysis. That means it just uses what it can find on the class-path,
but does not care about dependencies defined in Maven otherwise. Notably, it
ignores available Java packages for the {{Import-Package}} computation, if they
are not statically reachable via the classes inside the bundle (i.e. they
aren't needed).
[jira] [Commented] (OAK-9024) oak-solr-osgi imports org.slf4j.impl
[
https://issues.apache.org/jira/browse/OAK-9024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103950#comment-17103950
]
Julian Sedding commented on OAK-9024:
-
My bottom line is: *{{oak-solr-osgi}} MUST import {{org.slf4j.impl}}* in order
for the logging to work correctly.
[~baedke], I get the impression (and I don't mean this personally, so my
apologies if I am wrong) that you don't fully separate the Maven class-path and
the bundle's class-path in OSGi. The bnd tool (here in the form of the
maven-bundle-plugin IIRC) is concerned with making sure that the bundle has a
consistent class-space. That consists of two major parts: (a) all packages (and
classes) inside the bundle and (b) all packages (and classes) that need to be
provided to the bundle from the outside, i.e. everything listed in
{{Import-Package}}.
The bnd tool computes a graph of dependencies between classes using static
byte-code analysis. That means it just uses what it can find on the class-path,
but does not care about dependencies defined in Maven otherwise. Notably, it
ignores available Java packages for the {{Import-Package}} computation, if they
are not statically reachable via the classes inside the bundle (i.e. they
aren't needed). This mechanism would obviously miss any classes that are loaded
manually using a String class-name.
A consequence of embedding the {{slf4j-log4j12}} artifact is that on the
package level you embed {{org.slf4j.impl}} (and others I believe). Since that
package is now inside the bundle, bnd can remove it from the {{Import-Package}}
header. The same can be achieved by embedding any other artifact that contains
{{org.slf4j.impl}}, of which there are quite a few.
Now the but. But slf4j's contract for bindings seems to be that each binding
defines {{org.slf4j.impl.StaticLoggerBinder}}, which essentially makes this an
SPI. In Sling-based setups there is an {{slf4j.api}} bundle and an
{{org.apache.sling.commons.log}} bundle (the binding used in Sling). Notably,
the {{slf4j.api}} bundle imports the package {{org.slf4j.impl}} from
{{org.apache.sling.commons.log}}. That is because essentially
{{org.slf4j.impl}} is the SPI that SLF4J uses to connect the logging API with a
binding.
Furthermore, I am under the impression that SLF4J only supports a single
binding in a deployment. This would make embedding {{slf4j-log4j12}} in
{{oak-solr-osgi}} problematic, because (a) it would introduce a second binding
in Sling-based setups and (b) it would eliminate the possibility to swap out
logging bindings. Regarding (a) my guess would be that at least log messages
might get lost (there's no log4j present that would actually write the logs) or
possibly more obvious errors.
I think it is a little bit odd for SLF4J to call the package that hooks in the
bindings {{impl}}. However, once we ignore this suggestive name, there is no
reason why {{oak-solr-osgi}} should not import {{org.slf4j.impl}}. (Of course
there would be good reasons for Solr to avoid directly calling into SLF4J's
bindings SPI, but that's out-of-scope for Oak).
So that's why my bottom line is: {{oak-solr-osgi}} MUST import
{{org.slf4j.impl}} in order for the logging to work correctly.
Still, I think it would be better to control the import version range for
{{org.slf4j.impl}}. At the moment, I believe that it just "happens" to be
whatever we get transitively from Zookeeper or Solr or elsewhere. Controlling
the version range could be achieved by declaring a provided dependency on e.g.
{{slf4j-simple}}. Maybe with a comment that it is only used to help bnd compute
the correct version range.
> oak-solr-osgi imports org.slf4j.impl
>
>
> Key: OAK-9024
> URL: https://issues.apache.org/jira/browse/OAK-9024
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: solr
>Reporter: Julian Reschke
>Assignee: Manfred Baedke
>Priority: Minor
> Fix For: 1.28.0
>
> Attachments: OAK-9024.patch
>
>
> From the manifest:
> {{org.slf4j.impl;version="[1.6,2)"}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (OAK-9024) oak-solr-osgi imports org.slf4j.impl
[
https://issues.apache.org/jira/browse/OAK-9024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100554#comment-17100554
]
Julian Sedding edited comment on OAK-9024 at 5/6/20, 7:55 AM:
--
[~baedke] I performed a simple test. In the pom of the oak-solr-osgi module I
prevent the zookeeper dependency from being embedded by changing the dependency
scope to "provided". Then I created a diff of the Import-Package statement of
the oak-solr-osgi bundle before and after this change.
{noformat}
14d13
< com.ibm.security.krb5.internal {resolution:=optional}
42,44d40
< javax.security.auth
< javax.security.auth.callback
< javax.security.auth.kerberos
46d41
< javax.security.auth.spi
48d42
< javax.security.sasl
135d128
< org.apache.log4j.jmx {resolution:=optional,
version=[1.2,2)}
156a150,154
> org.apache.zookeeper {version=[3.4,4)}
> org.apache.zookeeper.data {version=[3.4,4)}
> org.apache.zookeeper.server{version=[3.4,4)}
> org.apache.zookeeper.server.auth {version=[3.4,4)}
> org.apache.zookeeper.server.quorum {version=[3.4,4)}
166,170d163
< org.jboss.netty.bootstrap {resolution:=optional,
version=[3.7,4)}
< org.jboss.netty.buffer {resolution:=optional,
version=[3.7,4)}
< org.jboss.netty.channel{resolution:=optional,
version=[3.7,4)}
< org.jboss.netty.channel.group {resolution:=optional,
version=[3.7,4)}
< org.jboss.netty.channel.socket.nio {resolution:=optional,
version=[3.7,4)}
187d179
< sun.security.krb5 {resolution:=optional}
{noformat}
As you can see the zookeeper packages are now imported (because they are not
provided within the bundle), and some other packages that are only used by
zookeeper are no longer imported. If zookeeper with its org.slf4j.slf4j-log4j12
dependency was the cause for {{org.slf4j.impl}} to be imported, then we would
expect it to disappear when zookeeper is no longer embedded.
This is not the case, ergo zookeeper is not the root cause you're looking for.
As I outlined before, the root cause is Solr itself.
> Note that IIUC embedding dependencies of oak-solr-core is one of the points
> of oak-solr-osgi.
I don't think embedding a logging binding was ever the goal of oak-solr-osgi.
If you are interested in the original reasoning why this bundle was created,
the discussion [https://markmail.org/thread/5xsyx5l4c6euqtt2] may be
interesting.
EDIT: The list of imports for each of the two bundles was generated using {{bnd
print -i }} and the diff was generated by diffing the unedited
outputs.
was (Author: jsedding):
[~baedke] I performed a simple test. In the pom of the oak-solr-osgi module I
prevent the zookeeper dependency from being embedded by changing the dependency
scope to "provided". Then I created a diff of the Import-Package statement of
the oak-solr-osgi bundle before and after this change.
{noformat}
14d13
< com.ibm.security.krb5.internal {resolution:=optional}
42,44d40
< javax.security.auth
< javax.security.auth.callback
< javax.security.auth.kerberos
46d41
< javax.security.auth.spi
48d42
< javax.security.sasl
135d128
< org.apache.log4j.jmx {resolution:=optional,
version=[1.2,2)}
156a150,154
> org.apache.zookeeper {version=[3.4,4)}
> org.apache.zookeeper.data {version=[3.4,4)}
> org.apache.zookeeper.server{version=[3.4,4)}
> org.apache.zookeeper.server.auth {version=[3.4,4)}
> org.apache.zookeeper.server.quorum {version=[3.4,4)}
166,170d163
< org.jboss.netty.bootstrap {resolution:=optional,
version=[3.7,4)}
< org.jboss.netty.buffer {resolution:=optional,
version=[3.7,4)}
< org.jboss.netty.channel{resolution:=optional,
version=[3.7,4)}
< org.jboss.netty.channel.group {resolution:=optional,
version=[3.7,4)}
< org.jboss.netty.channel.socket.nio {resolution:=optional,
version=[3.7,4)}
187d179
< sun.security.krb5 {resolution:=optional}
{noformat}
As you can see the zookeeper packages are now imported (because they are not
provided within the bundle), and some other packages that are only used by
zookeeper are no longer imported. If zookeeper with its org.slf4j.slf4j-log4j12
dependency was the cause for {{org.slf4j.impl}} to be imported, then we would
expect it to disappear when zookeeper is no longer embedded.
This is not the case, ergo zookeeper is not the root cause you're looking for.
As I outlined before, the root cause is Solr itself.
> Note that IIUC embedding dependencies of oak-solr-core is one of the points
>
[jira] [Commented] (OAK-9024) oak-solr-osgi imports org.slf4j.impl
[
https://issues.apache.org/jira/browse/OAK-9024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100554#comment-17100554
]
Julian Sedding commented on OAK-9024:
-
[~baedke] I performed a simple test. In the pom of the oak-solr-osgi module I
prevent the zookeeper dependency from being embedded by changing the dependency
scope to "provided". Then I created a diff of the Import-Package statement of
the oak-solr-osgi bundle before and after this change.
{noformat}
14d13
< com.ibm.security.krb5.internal {resolution:=optional}
42,44d40
< javax.security.auth
< javax.security.auth.callback
< javax.security.auth.kerberos
46d41
< javax.security.auth.spi
48d42
< javax.security.sasl
135d128
< org.apache.log4j.jmx {resolution:=optional,
version=[1.2,2)}
156a150,154
> org.apache.zookeeper {version=[3.4,4)}
> org.apache.zookeeper.data {version=[3.4,4)}
> org.apache.zookeeper.server{version=[3.4,4)}
> org.apache.zookeeper.server.auth {version=[3.4,4)}
> org.apache.zookeeper.server.quorum {version=[3.4,4)}
166,170d163
< org.jboss.netty.bootstrap {resolution:=optional,
version=[3.7,4)}
< org.jboss.netty.buffer {resolution:=optional,
version=[3.7,4)}
< org.jboss.netty.channel{resolution:=optional,
version=[3.7,4)}
< org.jboss.netty.channel.group {resolution:=optional,
version=[3.7,4)}
< org.jboss.netty.channel.socket.nio {resolution:=optional,
version=[3.7,4)}
187d179
< sun.security.krb5 {resolution:=optional}
{noformat}
As you can see the zookeeper packages are now imported (because they are not
provided within the bundle), and some other packages that are only used by
zookeeper are no longer imported. If zookeeper with its org.slf4j.slf4j-log4j12
dependency was the cause for {{org.slf4j.impl}} to be imported, then we would
expect it to disappear when zookeeper is no longer embedded.
This is not the case, ergo zookeeper is not the root cause you're looking for.
As I outlined before, the root cause is Solr itself.
> Note that IIUC embedding dependencies of oak-solr-core is one of the points
> of oak-solr-osgi.
I don't think embedding a logging binding was ever the goal of oak-solr-osgi.
If you are interested in the original reasoning why this bundle was created,
the discussion https://markmail.org/thread/5xsyx5l4c6euqtt2 may be interesting.
> oak-solr-osgi imports org.slf4j.impl
>
>
> Key: OAK-9024
> URL: https://issues.apache.org/jira/browse/OAK-9024
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: solr
>Reporter: Julian Reschke
>Assignee: Manfred Baedke
>Priority: Minor
> Fix For: 1.28.0
>
> Attachments: OAK-9024.patch
>
>
> From the manifest:
> {{org.slf4j.impl;version="[1.6,2)"}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (OAK-9047) Make the DefaultAuthorizableActionProvider require a configuration
[
https://issues.apache.org/jira/browse/OAK-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099855#comment-17099855
]
Julian Sedding edited comment on OAK-9047 at 5/5/20, 1:06 PM:
--
{quote}[...] that's why Francesco Mari introduced the
SecurityProviderRegistration and the mandatory services in the first place.
{quote}
That seems to be a variation of the same problem discussed in this issue. The
constituent services for a functioning repository must not become available
before they are properly configured. This is inherently difficult to achieve in
OSGi.
I wonder if a better strategy to address this would be de-modularisation of the
repository startup, i.e. have a single required configuration and hook all
parts together in plain Java before registering the repository.
Clearly there is a trade-off between a stable repository startup sequence some
disadvantages, e.g. backwards compatibility with current configurations and
substituting a custom implementations for some components would not work/would
need to be done differently.
Maybe a possibility could be a repository factory service that is used by a
repository registration service. The repository registration service could be
configured to require a specific repository factory and, once available, would
call it to get a repository instance and register it. Repository factory
implementations could require a configuration to prevent eager startup or be
manually registered after checking a number of requirements, thus making sure
everything is properly available before a repository is created (similar to
SecurityProviderRegistration does today). The current status quo could become
the default implementation for the repository factory (without required
configuration). However, another factory could then use stricter checks.
{quote}[We can solve any problem by introducing an extra level of
indirection.|https://en.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering]
{quote}
On a somewhat related note: (at least in AEM) the NodeStore service is
currently explicitly hidden, which makes sense, because it is an implementation
detail. Registering the NodeStore as a service would not be required at all
with a suitable repository factory. Maybe the desire to hide this service is an
indication that the OSGi service registry is the wrong place to hook all parts
together?
was (Author: jsedding):
bq. [...] that's why Francesco Mari introduced the SecurityProviderRegistration
and the mandatory services in the first place.
That seems to be a variation of the same problem discussed in this issue. The
constituent services for a functioning repository must not become available
before they are properly configured. This is inherently difficult to achieve in
OSGi.
I wonder if a better strategy to address this would be de-modularisation of the
repository startup, i.e. have a single required configuration and hook all
parts together in plain Java before registering the repository.
Clearly there is a trade-off between a stable repository startup sequence some
disadvantages, e.g. backwards compatibility with current configurations and
substituting a custom implementations for some components would not work/would
need to be done differently.
Maybe a possibility could be a repository factory service that is used by a
repository registration service. The repository registration service could be
configured to require a specific repository factory and, once available, would
call it to get a repository instance and register it. Repository factory
implementations could require a configuration to prevent eager startup or be
manually registered after checking a number of requirements, thus making sure
everything is properly available before a repository is created (similar to
SecurityProviderRegistration does today). The current status quo could become
the default implementation for the repository factory (without required
configuration). However, another factory could then use stricter checks.
bq. [We can solve any problem by introducing an extra level of
indirection.|https://en.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering]
> Make the DefaultAuthorizableActionProvider require a configuration
> --
>
> Key: OAK-9047
> URL: https://issues.apache.org/jira/browse/OAK-9047
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: security-spi
>Reporter: Konrad Windszus
>Priority: Major
>
> Currently configuring the {{DefaultAuthorizableActionProvider}} leads to
> restart of the Oak repository in the context of
>
[jira] [Commented] (OAK-9047) Make the DefaultAuthorizableActionProvider require a configuration
[
https://issues.apache.org/jira/browse/OAK-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099855#comment-17099855
]
Julian Sedding commented on OAK-9047:
-
bq. [...] that's why Francesco Mari introduced the SecurityProviderRegistration
and the mandatory services in the first place.
That seems to be a variation of the same problem discussed in this issue. The
constituent services for a functioning repository must not become available
before they are properly configured. This is inherently difficult to achieve in
OSGi.
I wonder if a better strategy to address this would be de-modularisation of the
repository startup, i.e. have a single required configuration and hook all
parts together in plain Java before registering the repository.
Clearly there is a trade-off between a stable repository startup sequence some
disadvantages, e.g. backwards compatibility with current configurations and
substituting a custom implementations for some components would not work/would
need to be done differently.
Maybe a possibility could be a repository factory service that is used by a
repository registration service. The repository registration service could be
configured to require a specific repository factory and, once available, would
call it to get a repository instance and register it. Repository factory
implementations could require a configuration to prevent eager startup or be
manually registered after checking a number of requirements, thus making sure
everything is properly available before a repository is created (similar to
SecurityProviderRegistration does today). The current status quo could become
the default implementation for the repository factory (without required
configuration). However, another factory could then use stricter checks.
bq. [We can solve any problem by introducing an extra level of
indirection.|https://en.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering]
> Make the DefaultAuthorizableActionProvider require a configuration
> --
>
> Key: OAK-9047
> URL: https://issues.apache.org/jira/browse/OAK-9047
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: security-spi
>Reporter: Konrad Windszus
>Priority: Major
>
> Currently configuring the {{DefaultAuthorizableActionProvider}} leads to
> restart of the Oak repository in the context of
> https://issues.apache.org/jira/browse/SLING-7811?focusedCommentId=16573171=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16573171.
>
> It would make sense to only start the service
> https://github.com/apache/jackrabbit-oak/blob/1f90e8c632868e658cc60b95bc5ec49182c4e173/oak-security-spi/src/main/java/org/apache/jackrabbit/oak/spi/security/user/action/DefaultAuthorizableActionProvider.java#L45
> once a mandatory configuration is in place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OAK-9024) oak-solr-osgi imports org.slf4j.impl
[
https://issues.apache.org/jira/browse/OAK-9024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098861#comment-17098861
]
Julian Sedding commented on OAK-9024:
-
[~baedke] I posted my comment, because I prefer to understand an issue before
it is fixed. I didn't get the impression that it was clear where the slf4j.impl
impoprt came from, nor why (exactly) embedding slf4j-log4j12 seems to fix it.
Assuming that {{org.slf4j.impl.StaticLoggerBinder}} is never actually called,
embedding slf4j-log4j12 would be fine. However, I would then rather chose to
include slf4j-nop, which has no dependencies other than slf4j.api. In case the
class {{org.slf4j.impl.StaticLoggerBinder}} is never loaded, excluding
org.slf4j.impl from the imports would also be ok.
If {{org.slf4j.impl.StaticLoggerBinder}} is called, I think the situation is
more problematic. The Oak setups I know of (all Sling based) use an OSGi
wrapped logback-classic (org.apache.sling.commons.log), which exports
{{org.slf4j.impl}}. For this scenario, I think importing it would be the best
solution. Embedding any other binding may well lead to class loading issues in
OSGi.
So I think embedding slf4j-log4j12 may not solve the problem, or may only solve
it superficially. I would actually lean towards importing {{org.slf4j.impl}},
maybe adjusting the version range (if necessary) and linking back to this issue
from the POM to document this oddity.
> oak-solr-osgi imports org.slf4j.impl
>
>
> Key: OAK-9024
> URL: https://issues.apache.org/jira/browse/OAK-9024
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: solr
>Reporter: Julian Reschke
>Assignee: Manfred Baedke
>Priority: Minor
> Fix For: 1.28.0
>
> Attachments: OAK-9024.patch
>
>
> From the manifest:
> {{org.slf4j.impl;version="[1.6,2)"}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OAK-9024) oak-solr-osgi imports org.slf4j.impl
[
https://issues.apache.org/jira/browse/OAK-9024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096360#comment-17096360
]
Julian Sedding commented on OAK-9024:
-
bndtools may have the answer, in particular the [xref
command|https://bnd.bndtools.org/commands/xref.html]
The following is an excerpt from its output, indicating which classes use
(i.e. import on a java level) classes from {{org.slf4j.impl}}.
{noformat}
$ bnd xref -cf target/oak-solr-osgi-1.27-SNAPSHOT.jar
...
org.slf4j.impl.StaticLoggerBinder < org.apache.solr.logging.LogWatcher
org.apache.solr.servlet.StartupLoggingUtils
...
{noformat}
Removing these two classes from the {{Embed-Dependency}} instruction should
remove the import for {{org.slf4j.impl}}. However, it woudl still be required
to analyze where the Oak code (transitively) depends on these classes and
examine if it can be safely used without them. If removing these classes is not
an option, it may still be safe to exclude the import for {{org.slf4j.impl}}
explicitly, iff no code path is used, where the {{StaticLoggerBinder}} is used.
If it _is_ used, however, then it we really need {{StaticLoggerBinder}} on the
class path (or provide an upstream fix for Solr).
> oak-solr-osgi imports org.slf4j.impl
>
>
> Key: OAK-9024
> URL: https://issues.apache.org/jira/browse/OAK-9024
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: solr
>Reporter: Julian Reschke
>Assignee: Manfred Baedke
>Priority: Minor
> Fix For: 1.28.0
>
> Attachments: OAK-9024.patch
>
>
> From the manifest:
> {{org.slf4j.impl;version="[1.6,2)"}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OAK-7182) Make it possible to update Guava
[ https://issues.apache.org/jira/browse/OAK-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779194#comment-16779194 ] Julian Sedding commented on OAK-7182: - bq. That would be similar to having an oak bundle that just wraps & shades Guava, right? Yes, except you wouldn't shade Guava but deploy it instead. Thus Oak could stay slim. Furthermore, embedding (and not exporting) Guava may lead to class loader issues for cases where Guava classes are used in API signatures. I.e. class GuavaA inside Oak would be loaded from embedded/shaded classes, whereas client bundles may load it from another buzndle. Thus GuavaA != GuavaA as far as Java is concerned, because their class loaders mismatch. bq. ... biggest roadblock ... Agreed. That's were the most urgent action is needed and probably the biggest effort. > Make it possible to update Guava > > > Key: OAK-7182 > URL: https://issues.apache.org/jira/browse/OAK-7182 > Project: Jackrabbit Oak > Issue Type: Wish >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Attachments: GuavaTests.java, OAK-7182-guava-21-3.diff, > OAK-7182-guava-21-4.diff, OAK-7182-guava-21.diff, OAK-7182-guava-23.6.1.diff, > guava.diff > > > We currently rely on Guava 15, and this affects all users of Oak because they > essentially need to use the same version. > This is an overall issue to investigate what would need to be done in Oak in > order to make updates possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OAK-7182) Make it possible to update Guava
[ https://issues.apache.org/jira/browse/OAK-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779194#comment-16779194 ] Julian Sedding edited comment on OAK-7182 at 2/27/19 12:14 PM: --- bq. That would be similar to having an oak bundle that just wraps & shades Guava, right? Yes, except you wouldn't shade Guava but deploy it instead. Thus Oak could stay slim. Furthermore, embedding (and not exporting) Guava may lead to class loader issues for cases where Guava classes are used in API signatures. I.e. class GuavaA inside Oak would be loaded from embedded/shaded classes, whereas client bundles may load it from another buzndle. Thus GuavaA != GuavaA as far as Java is concerned, because their class loaders mismatch. If Guava classes keep static internal state that may also be problematic. I don't know if this is the case and whether it would be a problem in Oak. bq. ... biggest roadblock ... Agreed. That's were the most urgent action is needed and probably the biggest effort. was (Author: jsedding): bq. That would be similar to having an oak bundle that just wraps & shades Guava, right? Yes, except you wouldn't shade Guava but deploy it instead. Thus Oak could stay slim. Furthermore, embedding (and not exporting) Guava may lead to class loader issues for cases where Guava classes are used in API signatures. I.e. class GuavaA inside Oak would be loaded from embedded/shaded classes, whereas client bundles may load it from another buzndle. Thus GuavaA != GuavaA as far as Java is concerned, because their class loaders mismatch. bq. ... biggest roadblock ... Agreed. That's were the most urgent action is needed and probably the biggest effort. > Make it possible to update Guava > > > Key: OAK-7182 > URL: https://issues.apache.org/jira/browse/OAK-7182 > Project: Jackrabbit Oak > Issue Type: Wish >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Attachments: GuavaTests.java, OAK-7182-guava-21-3.diff, > OAK-7182-guava-21-4.diff, OAK-7182-guava-21.diff, OAK-7182-guava-23.6.1.diff, > guava.diff > > > We currently rely on Guava 15, and this affects all users of Oak because they > essentially need to use the same version. > This is an overall issue to investigate what would need to be done in Oak in > order to make updates possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7182) Make it possible to update Guava
[ https://issues.apache.org/jira/browse/OAK-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779127#comment-16779127 ] Julian Sedding commented on OAK-7182: - It might be an option to create a custom API (owned by Oak) in a separate bundle. This bundle would be the only one alloed to import Guava. Furthermore, there may be multiple implementations of the API. E.g. one using Guava 15, another using Guava 25 and possibly later an implementation owned entirely by Oak. In order to be able to update Guava in an OSGi container only the matching implementation of this Oak API would need to be deployed. This solution would allow for a gradual transition away from Guava while easing the pain immediately. WDYT? > Make it possible to update Guava > > > Key: OAK-7182 > URL: https://issues.apache.org/jira/browse/OAK-7182 > Project: Jackrabbit Oak > Issue Type: Wish >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Attachments: GuavaTests.java, OAK-7182-guava-21-3.diff, > OAK-7182-guava-21-4.diff, OAK-7182-guava-21.diff, OAK-7182-guava-23.6.1.diff, > guava.diff > > > We currently rely on Guava 15, and this affects all users of Oak because they > essentially need to use the same version. > This is an overall issue to investigate what would need to be done in Oak in > order to make updates possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-6756) Convert oak-auth-external to OSGi R6 annotations
[
https://issues.apache.org/jira/browse/OAK-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699113#comment-16699113
]
Julian Sedding commented on OAK-6756:
-
[~anchela], I haven't got any further documentation that I am holding back. I
suppose it is important to understand that the diff output relates to XML files
being compared. For one annotated class, there can be two XML files: one for
Declarative Services (aka SCR) and onw for the MetaType. These files can be
found in the two compared JAR files, and it may be helpful to look at the diff
report and these XML files in order to get accustomed to the diff output.
In the diff output,
- the indentation is meant to mimic the XML structure;
- lines without leading '+' or '-' just indicate the location in the XML
hierarchy;
- lines with leading '+' or '-' are additions and removals respectively.
Modifications are shown as addition and removal.
Is this information helpful? If not, can you please indicate if there is
anything in particular that I could explain?
In the attached report, I would e.g. consider it problematic that the IDs of
MetaType properties change in {{ExternalLoginModuleFactory}}. That would likely
break existing configurations. It might also be problematic that
{{ExternalPrincipalConfiguration}} before the changes is registered as both
{{PrincipalConfiguration}} and {{SecurityConfiguration}}, whereas after the
changes it would only be registered as {{PrincipalConfiguration}}. These are
just two very obvious issues. I would aim at reducing the diff to be empty or
to just contain removals of "bind"/"unbind", which get replaced (due to new
defaults) to "field" with optional "field-option" (this change only indicates
that the injection mechanism was altered).
> Convert oak-auth-external to OSGi R6 annotations
>
>
> Key: OAK-6756
> URL: https://issues.apache.org/jira/browse/OAK-6756
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: auth-external
>Reporter: Robert Munteanu
>Assignee: angela
>Priority: Major
> Fix For: 1.10
>
> Attachments: cschneider-OAK-6756.osgi-diff.txt
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OAK-7668) Optimization: SegmentNodeState#fastEquals should compare getStableIdBytes instead of getStableId
Julian Sedding created OAK-7668:
---
Summary: Optimization: SegmentNodeState#fastEquals should compare
getStableIdBytes instead of getStableId
Key: OAK-7668
URL: https://issues.apache.org/jira/browse/OAK-7668
Project: Jackrabbit Oak
Issue Type: Improvement
Components: segment-tar
Affects Versions: 1.9.6
Reporter: Julian Sedding
I have observed via stack trace sampling that during the installation of a
Vault package with many small nodes on an otherwise quiet system, about 1/4 of
stack traces had the following lines on the top (this was on Oak 1.6.11, but
reading the code it applies equally on trunk):
{code}
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.getStableId(SegmentNodeState.java:124)+
at
org.apache.jackrabbit.oak.segment.SegmentNodeState.fastEquals(SegmentNodeState.java:632)
...
{code}
I suggest comparing the result of {{SegmentNodeState#getStableIdBytes}} instead
of {{SegmentNodeState#getStableId}}, which bypasses building the String
representation.
So far I have no proof in the form of a benchmark, but it seems quite clear
that this change will result in less work being performed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OAK-6770) Convert oak-segment-tar to OSGi R6 annotations
[ https://issues.apache.org/jira/browse/OAK-6770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531121#comment-16531121 ] Julian Sedding edited comment on OAK-6770 at 7/3/18 9:47 AM: - [~frm] I was convinced I had included a link in my previous comment, but must have gotten distracted. Sorry about that! Here it is: https://github.com/jsedding/osgi-ds-metatype-diff I like your idea of having tests for the metadata though. Maybe that could be simplified with (generic) tooling as well. was (Author: jsedding): [~frm] I was convinced I had included a link in my previous comment, but must have gotten distracted. Sorry about that! Here it is: https://github.com/jsedding/osgi-ds-metatype-diff > Convert oak-segment-tar to OSGi R6 annotations > -- > > Key: OAK-6770 > URL: https://issues.apache.org/jira/browse/OAK-6770 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: segment-tar >Reporter: Robert Munteanu >Assignee: Francesco Mari >Priority: Minor > Labels: osgi, technical_debt > Fix For: 1.10 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-6770) Convert oak-segment-tar to OSGi R6 annotations
[ https://issues.apache.org/jira/browse/OAK-6770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531121#comment-16531121 ] Julian Sedding commented on OAK-6770: - [~frm] I was convinced I had included a link in my previous comment, but must have gotten distracted. Sorry about that! Here it is: https://github.com/jsedding/osgi-ds-metatype-diff > Convert oak-segment-tar to OSGi R6 annotations > -- > > Key: OAK-6770 > URL: https://issues.apache.org/jira/browse/OAK-6770 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: segment-tar >Reporter: Robert Munteanu >Assignee: Francesco Mari >Priority: Minor > Labels: osgi, technical_debt > Fix For: 1.10 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-6770) Convert oak-segment-tar to OSGi R6 annotations
[ https://issues.apache.org/jira/browse/OAK-6770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530246#comment-16530246 ] Julian Sedding commented on OAK-6770: - [~frm] I wrote a little tool to help me diff metatype and declarative service component descriptors from different bundles. Exactly for the use-case of migrating to the official OSGi annotations. It's not perfect, but I found it helpful for avoiding bugs and learning the finer details of how the official OSGi annotations are processed. > Convert oak-segment-tar to OSGi R6 annotations > -- > > Key: OAK-6770 > URL: https://issues.apache.org/jira/browse/OAK-6770 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: segment-tar >Reporter: Robert Munteanu >Assignee: Francesco Mari >Priority: Minor > Labels: osgi, technical_debt > Fix For: 1.10 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7356) CugConfiguration may not pick up CugExclude
[
https://issues.apache.org/jira/browse/OAK-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411289#comment-16411289
]
Julian Sedding commented on OAK-7356:
-
[~anchela] Sorry, I just saw the issue now. I'm just sharing my thoughts, no
action is required. Your solution should work just fine. However, I'm pretty
sure "immediate = true" is unnecessary, because {{CugExcludeImpl}} would be
started anyway, as soon as it is needed to start {{CugConfiguration}}.
An alternative (simpler) solution could have been to specify
{{CugConfiguration}}'s reference to {{CugExclude}} to use
{{ReferencePolicyOption.GREEDY}}. That would have caused {{CugConfiguration}}
to be restarted as soon as the (optional) {{CugExclude}} service becomes
available. Restarting {{CugConfiguration}} could of course be undesirable due
to ripple-on effects causing consumers of {{CugConfiguration}} to be restarted
- I don't know.
> CugConfiguration may not pick up CugExclude
> ---
>
> Key: OAK-7356
> URL: https://issues.apache.org/jira/browse/OAK-7356
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: authorization-cug
>Reporter: angela
>Assignee: angela
>Priority: Major
> Fix For: 1.9.0, 1.10
>
> Attachments: OAK-7356-2.patch, OAK-7356.patch
>
>
> It seems that the {{CugConfiguration}} may under some circumstances not
> properly pick up the {{CugExclude}}, which results in the
> {{CugConfiguration}} falling back to the default, that only excludes the
> {{AdminPrincipal}}, {{SystemPrincipal}} and {{SystemUserPrincipals}} from CUG
> evaluation.
> In order to address the issue without disrupting the default setup, I would
> like to propose the following changes:
> - {{CugConfiguration}}: Change reference cardinality of the the
> {{CugExclude}} from {{ReferenceCardinality.OPTIONAL_UNARY}} to
> {{ReferenceCardinality.MANDATORY_UNARY}}
> - {{CugExcludeImpl}}: Don't require an explicit configuration (i.e. drop
> {{ConfigurationPolicy.REQUIRE}}). Since it extends from
> {{CugExclude.Default}} the default behavior in absense of a configured set of
> principal names will be equivalent to the intended default in
> {{CugConfiguration}} with the optional reference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OAK-6756) Convert oak-auth-external to OSGi R6 annotations
[
https://issues.apache.org/jira/browse/OAK-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-6756:
Attachment: cschneider-OAK-6756.osgi-diff.txt
Attached is a [report with potentially problematic
changes|^cschneider-OAK-6756.osgi-diff.txt] to the metatype and declarative
services declarations in {{cschneider/OAK-6756}}.
> Convert oak-auth-external to OSGi R6 annotations
>
>
> Key: OAK-6756
> URL: https://issues.apache.org/jira/browse/OAK-6756
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: auth-external
>Reporter: Robert Munteanu
>Assignee: angela
> Fix For: 1.10
>
> Attachments: cschneider-OAK-6756.osgi-diff.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6946) org.apache.jackrabbit.oak.security.authentication.ldap must not be exported / filtered in parent
[
https://issues.apache.org/jira/browse/OAK-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255052#comment-16255052
]
Julian Sedding commented on OAK-6946:
-
FYI, the {{maven-bundle-plugin}} implicitly does not export any packages called
"impl" or "internal" and their descendant packages. That's why we're fine here.
> org.apache.jackrabbit.oak.security.authentication.ldap must not be exported /
> filtered in parent
>
>
> Key: OAK-6946
> URL: https://issues.apache.org/jira/browse/OAK-6946
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: auth-ldap
>Reporter: angela
>Assignee: angela
> Fix For: 1.8, 1.7.12
>
> Attachments: OAK-6946.patch
>
>
> The _org.apache.jackrabbit.oak.security.authentication.ldap_ package space is
> empty and there is nothing we plan to export there as the whole
> _oak-auth-ldap_ module just contains an implementation of the external-IDP
> API.
> Since the _maven-bundle-plugin_ doesn't have explicit {{}}
> instructions the plugin uses the default and thus (unintended by us) exports
> the _org.apache.jackrabbit.oak.security.authentication.ldap_ package.
> [~stillalex], fyi
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-5519) Skip problematic binaries instead of blocking indexing
[
https://issues.apache.org/jira/browse/OAK-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247189#comment-16247189
]
Julian Sedding commented on OAK-5519:
-
[~tmueller] could the processing thread be terminated by closing the stream
after the timeout? I suppose that should trip up the parser and cause an
{{IOException}} on the next read. Granted, I don't understand the full
background of this issue. Maybe the endless loop scenarios don't read from the
stream any more.
> Skip problematic binaries instead of blocking indexing
> --
>
> Key: OAK-5519
> URL: https://issues.apache.org/jira/browse/OAK-5519
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: indexing
>Reporter: Alexander Klimetschek
>Assignee: Thomas Mueller
> Labels: resilience
> Fix For: 1.8, 1.7.12
>
>
> If a text extraction is blocked (weird PDF) or a blob cannot be found in the
> datastore or any other error upon indexing one item from the repository that
> is outside the scope of the indexer, it currently halts the indexing (lane).
> Thus one item (that maybe isn't important to the users at all) can block the
> indexing of other, new content (that might be important to users), and it
> always requires manual intervention (which is also not easy and requires oak
> experts).
> Instead, the item could be remembered in a known issue list, proper warnings
> given, and indexing continue. Maintenance operations should be available to
> come back to reindex these, or the indexer could automatically retry after
> some time. This would allow normal user activity to go on without manual
> intervention, and solving the problem (if it's isolated to some binaries) can
> be deferred.
> I think the line should probably be drawn for binary properties. Not sure if
> other JCR property types could trigger a similar issue, and if a failure in
> them might actually warrant a halt, as it could lead to an "incorrect" index,
> if these properties are important. But maybe the line is simply a try & catch
> around "full text extraction".
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6917) Configuration presets for DocumentNodeStoreService
[
https://issues.apache.org/jira/browse/OAK-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245637#comment-16245637
]
Julian Sedding commented on OAK-6917:
-
bq. Nope, that doesn't work
A shame, but not entirely unexpected. Did you try changing the field's type to
{{Map}}? That should work (at least with DS 1.3).
>From the Declarative Services 1.3 Spec, References to Services section 112.3.3
>Field Strategy (OSGi R6 Compendium):
{quote}
For a reference with unary cardinality, a field must be of one of the following
types:
...
Map - An unmodifiable Map containing the service properties of the bound
service. This Map must additionally implement Comparable with the compareTo
method comparing service prop- erty maps using the same ordering as
ServiceReference.compareTo based upon service ranking and service id.
{quote}
> Configuration presets for DocumentNodeStoreService
> --
>
> Key: OAK-6917
> URL: https://issues.apache.org/jira/browse/OAK-6917
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.8
>
> Attachments: OAK-6917-alternative-approach.patch, OAK-6917.patch
>
>
> When Oak is deployed in an OSGi container, applications usually want to ship
> a default configuration which is different from the defaults present in Oak.
> E.g. an application may want to use a default cache size of 1G for the
> DocumentNodeStoreService instead of the default 256M. Now if a user of the
> application provides a custom configuration and does not specify the cache
> size, the value for this configuration will flip back to the Oak default of
> 256M.
> There should be a way to configure presets for the application that are
> different from the Oak defaults and then allow a user to customize the
> configuration while still respecting the presets.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6917) Configuration presets for DocumentNodeStoreService
[
https://issues.apache.org/jira/browse/OAK-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Sedding updated OAK-6917:
Attachment: OAK-6917-alternative-approach.patch
[~mreutegg] I attached a patch with a sketch for an alternative approach, YMMV.
It is limited to the OSGi aspect, but lacks merging the "preset" (or default)
configuration with the config for {{DocumentNodeStoreService}}. Check it out,
maybe you find it useful.
Basically, I register an additional service, that references the same
{{Configuration}} annotation. That way, it gets all the defaults set in the
annotation, but these can be overridden by config admin (config policy is
optional though). I created the (mostly empty) inner class {{Defaults}} to
achieve this. The {{DocumentNodeStoreService}} then references the
{{DNSS.Defaults}} service, but instead of having the service instance injected,
we ask to get the {{Configuration}} annotation injected (not 100% sure this
works, if not, try with {{Map}}, which I have done before).
I didn't test the patch, but quickly looked at the generated XML files, and
they look as intended (at first glance at least). Note that I deliberately
didn't get metatype information generated for the {{DNSS.Defaults}} service, as
I understood that it should be a "private" config that should only be set by
product vendors/distributors.
> Configuration presets for DocumentNodeStoreService
> --
>
> Key: OAK-6917
> URL: https://issues.apache.org/jira/browse/OAK-6917
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
>Priority: Minor
> Fix For: 1.8
>
> Attachments: OAK-6917-alternative-approach.patch, OAK-6917.patch
>
>
> When Oak is deployed in an OSGi container, applications usually want to ship
> a default configuration which is different from the defaults present in Oak.
> E.g. an application may want to use a default cache size of 1G for the
> DocumentNodeStoreService instead of the default 256M. Now if a user of the
> application provides a custom configuration and does not specify the cache
> size, the value for this configuration will flip back to the Oak default of
> 256M.
> There should be a way to configure presets for the application that are
> different from the Oak defaults and then allow a user to customize the
> configuration while still respecting the presets.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6889) Followup on OAK-6755: fix OSGi component descriptors
[ https://issues.apache.org/jira/browse/OAK-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Sedding updated OAK-6889: Description: The fix for OAK-6755 (see also OAK-6741) caused some OSGi metadata to be lost or otherwise incorrect. Issues were found using [osgi-ds-metatype-diff|https://github.com/jsedding/osgi-ds-metatype-diff]. was:The fix for OAK-6755 (see also OAK-6741) caused some OSGi metadata to be lost or otherwise incorrect. > Followup on OAK-6755: fix OSGi component descriptors > > > Key: OAK-6889 > URL: https://issues.apache.org/jira/browse/OAK-6889 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core, security >Affects Versions: 1.7.9 >Reporter: Julian Sedding >Assignee: Julian Sedding > Fix For: 1.7.11 > > > The fix for OAK-6755 (see also OAK-6741) caused some OSGi metadata to be lost > or otherwise incorrect. > Issues were found using > [osgi-ds-metatype-diff|https://github.com/jsedding/osgi-ds-metatype-diff]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (OAK-6889) Followup on OAK-6755: fix OSGi component descriptors
[ https://issues.apache.org/jira/browse/OAK-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Sedding resolved OAK-6889. - Resolution: Fixed > Followup on OAK-6755: fix OSGi component descriptors > > > Key: OAK-6889 > URL: https://issues.apache.org/jira/browse/OAK-6889 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core, security >Affects Versions: 1.7.9 >Reporter: Julian Sedding >Assignee: Julian Sedding > Fix For: 1.7.11 > > > The fix for OAK-6755 (see also OAK-6741) caused some OSGi metadata to be lost > or otherwise incorrect. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6889) Followup on OAK-6755: fix OSGi component descriptors
[
https://issues.apache.org/jira/browse/OAK-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226938#comment-16226938
]
Julian Sedding commented on OAK-6889:
-
Fixed in [r1813889|https://svn.apache.org/r1813889].
Most issues were caused by configuration annotations *not* being referenced in
the {{@Activate}} method signature, which is necessary for DS component
descriptors to include properties described by the annotation.
>From the OSGi Declarative Services Specification Version 1.3, §112.8.3
>(emphasis is mine):
{quote}
Properties defined through component property types _used in the signatures of
the life cycle methods_.
If any of the referenced component property types have _methods with defaults_,
then the generated component description must include a property element for
each such method with the property name mapped from the method name, the
property type mapped from the method type, and the property value set to the
method's default value.
{quote}
{{AuthorizationConfigurationImpl}} was missing the {{@Designate}} annotation
and had a typo in a property name (importBehaviour vs importBehavior).
{{DefaultAuthorizableActionProvider}} (by now moved to {{oak-security-spi}})
needed its option labels and values swapped around.
> Followup on OAK-6755: fix OSGi component descriptors
>
>
> Key: OAK-6889
> URL: https://issues.apache.org/jira/browse/OAK-6889
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core, security
>Affects Versions: 1.7.9
>Reporter: Julian Sedding
>Assignee: Julian Sedding
> Fix For: 1.7.11
>
>
> The fix for OAK-6755 (see also OAK-6741) caused some OSGi metadata to be lost
> or otherwise incorrect.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6889) Followup on OAK-6755: fix OSGi component descriptors
[ https://issues.apache.org/jira/browse/OAK-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Sedding updated OAK-6889: Affects Version/s: (was: 1.7.10) 1.7.9 > Followup on OAK-6755: fix OSGi component descriptors > > > Key: OAK-6889 > URL: https://issues.apache.org/jira/browse/OAK-6889 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core, security >Affects Versions: 1.7.9 >Reporter: Julian Sedding >Assignee: Julian Sedding > Fix For: 1.7.11 > > > The fix for OAK-6755 (see also OAK-6741) caused some OSGi metadata to be lost > or otherwise incorrect. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6741) Switch to official OSGi component and metatype annotations
[
https://issues.apache.org/jira/browse/OAK-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226904#comment-16226904
]
Julian Sedding commented on OAK-6741:
-
I created OAK-6889 for following up on the changes done for {{oak-core}}.
> Switch to official OSGi component and metatype annotations
> --
>
> Key: OAK-6741
> URL: https://issues.apache.org/jira/browse/OAK-6741
> Project: Jackrabbit Oak
> Issue Type: Improvement
>Reporter: Robert Munteanu
> Fix For: 1.8, 1.7.11
>
> Attachments: OAK-6741-proposed-changes-chetans-feedback.patch,
> osgi-metadata-1.7.8.json, osgi-metadata-trunk.json
>
>
> We should remove the 'old' Felix SCR annotations and move to the 'new' OSGi
> R6 annotations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (OAK-6889) Followup on OAK-6755: fix OSGi component descriptors
Julian Sedding created OAK-6889: --- Summary: Followup on OAK-6755: fix OSGi component descriptors Key: OAK-6889 URL: https://issues.apache.org/jira/browse/OAK-6889 Project: Jackrabbit Oak Issue Type: Bug Components: core, security Affects Versions: 1.7.10 Reporter: Julian Sedding Assignee: Julian Sedding Fix For: 1.7.11 The fix for OAK-6755 (see also OAK-6741) caused some OSGi metadata to be lost or otherwise incorrect. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6741) Switch to official OSGi component and metatype annotations
[ https://issues.apache.org/jira/browse/OAK-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198310#comment-16198310 ] Julian Sedding commented on OAK-6741: - [~anchela] I agree with you that we need to write tests and increase coverage. Also I am very glad that you _are_ improving coverage in the areas you work. Unfortunately, just like with the first Oak release you mention, reality sometimes gets in the way. As this is a cross-cutting concern (because it's essentially tooling), it literally touches all areas of the code base. I simply cannot make enough time available to read and understand the code in all these places and improve test coverage sufficiently for this task. In an ideal world, I would have that time and then I would be happy to write the tests. In reality it was a trade off between staying with the old annotations (and thus locking us into older tooling) or updating with some risks. I made a choice, based on experiences with projects that were literally stuck on 5 year old tooling. But I can understand if you disagree with the choice I made. [~chetanm] Last week I proposed a patch to address the issues you mentioned. It would be great if you could review the changes. I'd like to commit them soon. > Switch to official OSGi component and metatype annotations > -- > > Key: OAK-6741 > URL: https://issues.apache.org/jira/browse/OAK-6741 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Robert Munteanu > Fix For: 1.8, 1.7.10 > > Attachments: OAK-6741-proposed-changes-chetans-feedback.patch, > osgi-metadata-1.7.8.json, osgi-metadata-trunk.json > > > We should remove the 'old' Felix SCR annotations and move to the 'new' OSGi > R6 annotations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6741) Switch to official OSGi component and metatype annotations
[
https://issues.apache.org/jira/browse/OAK-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194320#comment-16194320
]
Julian Sedding commented on OAK-6741:
-
[~stillalex] I agree that the OSGi testing situation is currently still lacking
good support.
PaxExam seems too heavy, slow and is poorly documented IMHO. Felix Connect,
which is used in {{oak-pojosr}} seems like a promising building block, but
lacks a simple testing facade. At last week's Sling Hackathon, I suggested to
use Felix Connect under the hood for Sling's {{osgi-mock}}. If that can be done
in a backwards compatible way, that might help. However, I have a feeling that
it would still not be quite enough.
Without an investment in tooling, I unfortunately don't see how we can clean up
the OSGi test situation any time soon.
Point taken and agreed about the risks. Just for the record, I didn't take them
lightly. I tried to reduce them as much as I could and I am sorry that a few
issues seem to have slipped through.
> Switch to official OSGi component and metatype annotations
> --
>
> Key: OAK-6741
> URL: https://issues.apache.org/jira/browse/OAK-6741
> Project: Jackrabbit Oak
> Issue Type: Improvement
>Reporter: Robert Munteanu
> Fix For: 1.8, 1.7.9
>
> Attachments: OAK-6741-proposed-changes-chetans-feedback.patch,
> osgi-metadata-1.7.8.json, osgi-metadata-trunk.json
>
>
> We should remove the 'old' Felix SCR annotations and move to the 'new' OSGi
> R6 annotations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OAK-6741) Switch to official OSGi component and metatype annotations
[ https://issues.apache.org/jira/browse/OAK-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Sedding updated OAK-6741: Attachment: OAK-6741-proposed-changes-chetans-feedback.patch Attached is a tentative patch to address [~chetanm]'s feedback. > Switch to official OSGi component and metatype annotations > -- > > Key: OAK-6741 > URL: https://issues.apache.org/jira/browse/OAK-6741 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Robert Munteanu > Fix For: 1.8, 1.7.9 > > Attachments: OAK-6741-proposed-changes-chetans-feedback.patch, > osgi-metadata-1.7.8.json, osgi-metadata-trunk.json > > > We should remove the 'old' Felix SCR annotations and move to the 'new' OSGi > R6 annotations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6741) Switch to official OSGi component and metatype annotations
[
https://issues.apache.org/jira/browse/OAK-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194303#comment-16194303
]
Julian Sedding commented on OAK-6741:
-
Thank you [~chetanm] for the list of differences. I'll try to address them
below.
bq. No 'Designate' found in
OSGI-INF/metatype/org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl$Configuration.xml
- Looks like AuthorizationConfigurationImpl does not have designate specified
Agreed, I think the {{@Designate}} annotation is required.
bq. COWNodeStoreService is having the "role" property missing. Which is required
You may be right. This component is confusing, however. Before my changes, it
did not have {{metatype = true}} set, yet it declares a property with (metatype
specific) {{label}} and {{description}}. Furthermore, it requires a
configuration before it becomes active.
This leads to a behaviour, where no configuration form is displayed in the web
console. But if you supply a configuration (e.g. via Slings' OSGi installer)
you _should_ specify the {{role}} property. If this is done, the service is
registered with such a property and it can be used for filtering.
So I believe that the component after my changes behaves as before. However, we
should clarify the intent, and possibly add a configuration annotation object
that is referenced in the {{@Designate}} annotation.
WDYT?
{quote}
DocumentNodeStoreService - Missing props
updateLimit
repository.home
role
{quote}
I think that's just a difference in processing. Possibly semantically
equivalent. The properties are all defined in the configuration annotation
object, but have no default values (as before).
bq. SecondaryStoreCacheService - immediate missing
This is just a component (does not expose a service). In that case {{immediate
= true}} is the default. Thereofore, I think it's fine.
bq. MountInfoProviderService - supportFragment missing (may be it was changed
so valid)
I didn't touch the {{oak-store-composite}} bundle with these changes.
{quote}
AuthenticationConfigurationImpl
org.apache.jackrabbit.oak.authentication.appName missing
org.apache.jackrabbit.oak.authentication.configSpiName missing
{quote}
We _may_ have lost {{appName}} if no configuration is present.
{{configSpiName}} has no default, so it should be fine if it's missing (it
should be present in the metatype).
bq. DefaultAuthorizableActionProvider - label and value swapped
Yes, this is wrong. It seems that the old annotations expected it the wrong way
around.
> Switch to official OSGi component and metatype annotations
> --
>
> Key: OAK-6741
> URL: https://issues.apache.org/jira/browse/OAK-6741
> Project: Jackrabbit Oak
> Issue Type: Improvement
>Reporter: Robert Munteanu
> Fix For: 1.8, 1.7.9
>
> Attachments: osgi-metadata-1.7.8.json, osgi-metadata-trunk.json
>
>
> We should remove the 'old' Felix SCR annotations and move to the 'new' OSGi
> R6 annotations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (OAK-6741) Switch to official OSGi component and metatype annotations
[
https://issues.apache.org/jira/browse/OAK-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194264#comment-16194264
]
Julian Sedding edited comment on OAK-6741 at 10/6/17 8:00 AM:
--
bq. Julian Sedding you might want to check your facts before pointing the
finger. we already have a test for this [0]. why this test is not failing with
the changed annotation, is a different question (it's probably the test
framework that does not respect the wiring, but I'm no OSGi expert).
_edit_: removed less constructive comment, replaced with hopefully more
constructive one
The test uses Sling's osgi-mocks via {{OsgiContext}}. This is (currently) not
backed by a true OSGi implementation and re-implements certain features like
Declarative Services and MetaType. Sometimes this can lead to problems, which
usually manifest in lacking features. However, in this case, I think that the
test should have been adjusted to bind multiple {{CredentialsSupport}} services
when OAK-6753 was done.
was (Author: jsedding):
bq. Julian Sedding you might want to check your facts before pointing the
finger. we already have a test for this [0]. why this test is not failing with
the changed annotation, is a different question (it's probably the test
framework that does not respect the wiring, but I'm no OSGi expert).
With all due respect [~stillalex], I think your quoting me out of context. As
for the facts, you say yourself that the test does not fail. So we don't have a
test that covers the scenario imlemented in OAK-6753. This is evident by the
fact that the test was not adjusted when OAK-6753 was implemented.
Anyway, we're splitting hairs. If you would like me to revert the patch, please
tell me so.
> Switch to official OSGi component and metatype annotations
> --
>
> Key: OAK-6741
> URL: https://issues.apache.org/jira/browse/OAK-6741
> Project: Jackrabbit Oak
> Issue Type: Improvement
>Reporter: Robert Munteanu
> Fix For: 1.8, 1.7.9
>
> Attachments: osgi-metadata-1.7.8.json, osgi-metadata-trunk.json
>
>
> We should remove the 'old' Felix SCR annotations and move to the 'new' OSGi
> R6 annotations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6741) Switch to official OSGi component and metatype annotations
[ https://issues.apache.org/jira/browse/OAK-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194264#comment-16194264 ] Julian Sedding commented on OAK-6741: - bq. Julian Sedding you might want to check your facts before pointing the finger. we already have a test for this [0]. why this test is not failing with the changed annotation, is a different question (it's probably the test framework that does not respect the wiring, but I'm no OSGi expert). With all due respect [~stillalex], I think your quoting me out of context. As for the facts, you say yourself that the test does not fail. So we don't have a test that covers the scenario imlemented in OAK-6753. This is evident by the fact that the test was not adjusted when OAK-6753 was implemented. Anyway, we're splitting hairs. If you would like me to revert the patch, please tell me so. > Switch to official OSGi component and metatype annotations > -- > > Key: OAK-6741 > URL: https://issues.apache.org/jira/browse/OAK-6741 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Robert Munteanu > Fix For: 1.8, 1.7.9 > > Attachments: osgi-metadata-1.7.8.json, osgi-metadata-trunk.json > > > We should remove the 'old' Felix SCR annotations and move to the 'new' OSGi > R6 annotations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6741) Switch to official OSGi component and metatype annotations
[
https://issues.apache.org/jira/browse/OAK-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192825#comment-16192825
]
Julian Sedding commented on OAK-6741:
-
I agree that such tooling would be desirable. I created a (non-semantic) way to
compare two bundles, by diffing the generated XML. However, as Chetan also
realized, the diff would need to happen on a semantic level, because the
descriptors can look different despite identical semantics. This "noise" makes
manual diffing very error prone.
Talking with Robert, I mentioned, that I believe the semantic model may be
possible to generate using the Felix SCR and MetaType implementations. Maybe
this could be implemented in {{bnd}}'s
[{{DiffPluginImpl}}|https://github.com/bndtools/bnd/blob/master/biz.aQute.bndlib/src/aQute/bnd/differ/DiffPluginImpl.java].
> Switch to official OSGi component and metatype annotations
> --
>
> Key: OAK-6741
> URL: https://issues.apache.org/jira/browse/OAK-6741
> Project: Jackrabbit Oak
> Issue Type: Improvement
>Reporter: Robert Munteanu
> Fix For: 1.8, 1.7.9
>
>
> We should remove the 'old' Felix SCR annotations and move to the 'new' OSGi
> R6 annotations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6741) Switch to official OSGi component and metatype annotations
[
https://issues.apache.org/jira/browse/OAK-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192671#comment-16192671
]
Julian Sedding commented on OAK-6741:
-
[~anchela] I'm sorry that I made a mistake. However, I have to wonder, why is
there no test coverage for OAK-6753?! Seriously, by your logic, I would need to
have written a complete OSGi test suite for pretty much all of {{oak-core}}.
Not quite realistic IMHO.
> Switch to official OSGi component and metatype annotations
> --
>
> Key: OAK-6741
> URL: https://issues.apache.org/jira/browse/OAK-6741
> Project: Jackrabbit Oak
> Issue Type: Improvement
>Reporter: Robert Munteanu
> Fix For: 1.8, 1.7.9
>
>
> We should remove the 'old' Felix SCR annotations and move to the 'new' OSGi
> R6 annotations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6741) Switch to official OSGi component and metatype annotations
[
https://issues.apache.org/jira/browse/OAK-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184224#comment-16184224
]
Julian Sedding commented on OAK-6741:
-
Fixed for {{oak-core}} and {{oak-store-document}} in
[rr1810002|https://svn.apache.org/r1810002].
> Switch to official OSGi component and metatype annotations
> --
>
> Key: OAK-6741
> URL: https://issues.apache.org/jira/browse/OAK-6741
> Project: Jackrabbit Oak
> Issue Type: Improvement
>Reporter: Robert Munteanu
> Fix For: 1.8, 1.7.9
>
>
> We should remove the 'old' Felix SCR annotations and move to the 'new' OSGi
> R6 annotations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (OAK-6741) Switch to official OSGi component and metatype annotations
[
https://issues.apache.org/jira/browse/OAK-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184224#comment-16184224
]
Julian Sedding edited comment on OAK-6741 at 9/28/17 2:12 PM:
--
Fixed for {{oak-core}} and {{oak-store-document}} in
[r1810002|https://svn.apache.org/r1810002].
Thanks for your help [~rombert]!
was (Author: jsedding):
Fixed for {{oak-core}} and {{oak-store-document}} in
[r1810002|https://svn.apache.org/r1810002].
> Switch to official OSGi component and metatype annotations
> --
>
> Key: OAK-6741
> URL: https://issues.apache.org/jira/browse/OAK-6741
> Project: Jackrabbit Oak
> Issue Type: Improvement
>Reporter: Robert Munteanu
> Fix For: 1.8, 1.7.9
>
>
> We should remove the 'old' Felix SCR annotations and move to the 'new' OSGi
> R6 annotations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (OAK-6741) Switch to official OSGi component and metatype annotations
[
https://issues.apache.org/jira/browse/OAK-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184224#comment-16184224
]
Julian Sedding edited comment on OAK-6741 at 9/28/17 2:11 PM:
--
Fixed for {{oak-core}} and {{oak-store-document}} in
[r1810002|https://svn.apache.org/r1810002].
was (Author: jsedding):
Fixed for {{oak-core}} and {{oak-store-document}} in
[rr1810002|https://svn.apache.org/r1810002].
> Switch to official OSGi component and metatype annotations
> --
>
> Key: OAK-6741
> URL: https://issues.apache.org/jira/browse/OAK-6741
> Project: Jackrabbit Oak
> Issue Type: Improvement
>Reporter: Robert Munteanu
> Fix For: 1.8, 1.7.9
>
>
> We should remove the 'old' Felix SCR annotations and move to the 'new' OSGi
> R6 annotations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6665) Move DocumentNodeStore into its own bundle
[
https://issues.apache.org/jira/browse/OAK-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181482#comment-16181482
]
Julian Sedding commented on OAK-6665:
-
A late +1 for {{oak-store-document}} also from me.
> Move DocumentNodeStore into its own bundle
> --
>
> Key: OAK-6665
> URL: https://issues.apache.org/jira/browse/OAK-6665
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: core, documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
> Labels: modularization, technical_debt
> Fix For: 1.8, 1.7.9
>
>
> Move the DocumentNodeStore implementation and the two backends (MongoDB, RDB)
> into its own bundle. This will make it possible to release oak-core and the
> NodeStore implementation independently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6628) More precise indexRules support via filtering criteria on property
[
https://issues.apache.org/jira/browse/OAK-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167782#comment-16167782
]
Julian Sedding commented on OAK-6628:
-
[~chetanm] I'm probably missing something because I'm not familiar with the
query/indexing code. However, it seems fairly trivial to create an
{{SQL2Parser}}, parse a query string and extract the {{Filter}} instances from
the resulting {{Query}}. These {{Filter}} instances would then need to be taken
into account, e.g. in {{IndexRule}}. I don't understand what you mean when you
say "filter criteria are evaluated at a lower level". Nor do I understand why
their evaluation would be more complex or expensive than inventing and
implementing a new notation.
Starting with simple cases is ok and unsupported property restrictions could be
rejected. However, (dreaming a bit further, I understand this isn't the goal of
this ticket) configuring an index by providing the query it is meant to support
would be both simple and powerful (for the user). You clearly showed that this
is possible with the [index definition generator
utility|http://oakutils.appspot.com/generate/index].
> More precise indexRules support via filtering criteria on property
> --
>
> Key: OAK-6628
> URL: https://issues.apache.org/jira/browse/OAK-6628
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
> Fix For: 1.8
>
>
> For Lucene index we currently support indexRules based on nodetype. Here the
> recommendation is that users must use most precise nodeType/mixinType to
> target the indexing rule so that only relevant nodes are indexed.
> For many Sling based applications its being seen that lots of content is
> nt:unstructured and it uses {{sling:resourceType}} property to distinguish
> various such nt:unstructured nodes. Currently its not possible to target
> index definition to index only those nt:unstructured which have specific
> {{sling:resourceType}}. Which makes it harder to provide a more precise index
> definitions.
> To help such cases we can generalize the indexRule support via a filtering
> criteria
> {noformat}
> activityIndex
> - type = "lucene"
> + indexRules
> + nt:unstructured
> - filter-property = "sling:resourceType"
> - filter-value = "app/activitystreams/components/activity"
> + properties
> - jcr:primaryType = "nt:unstructured"
> + verb
> - propertyIndex = true
> - name = "verb"
> {noformat}
> So indexRule would have 2 more config properties
> * filter-property - Name of property to match
> * filter-value - The value to match
> *Indexing*
> At time of indexing currently LuceneIndexEditor does a
> {{indexDefinition.getApplicableIndexingRule}} passing it the NodeState.
> Currently this checks only for jcr:PrimaryType and jxr:mixins to find
> matching rule.
> This logic would need to be extended to also check if any filter-property is
> defined in definition. If yes then check if NodeState has that value
> *Querying*
> On query side we need to change the IndexPlanner where it currently use query
> nodetype for finding matching indexRule. In addition it would need to pass on
> the property restrictions and the rule only be matched if the property
> restriction matches the filter
> *Open Item*
> # How to handle change in filter-property value. I think we have similar
> problem currently if an index nodes nodeType gets changed. In such a case we
> do not remove it from index. So we need to solve that for both
> # Ensure that all places where rules are matched account for this filter
> concept
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6665) Move DocumentNodeStore into its own bundle
[
https://issues.apache.org/jira/browse/OAK-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166309#comment-16166309
]
Julian Sedding commented on OAK-6665:
-
Name suggestion: {{oak-document}} - it is in line with {{oak-segment-tar}},
leaving the possibility to split it later into e.g. {{oak-document-mongo}} and
{{oak-document-rdb}}.
The plan sounds good to me.
> Move DocumentNodeStore into its own bundle
> --
>
> Key: OAK-6665
> URL: https://issues.apache.org/jira/browse/OAK-6665
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: core, documentmk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
> Labels: modularization, technical_debt
> Fix For: 1.8
>
>
> Move the DocumentNodeStore implementation and the two backends (MongoDB, RDB)
> into its own bundle. This will make it possible to release oak-core and the
> NodeStore implementation independently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (OAK-6628) More precise indexRules support via filtering criteria on property
[
https://issues.apache.org/jira/browse/OAK-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166160#comment-16166160
]
Julian Sedding commented on OAK-6628:
-
Rather than invent a new representation for filtering, why not express the
filter as a query string (SQL2 or XPath)? That should cover any current or
future use-case and we could probably re-use existing implementation classes.
WDYT?
> More precise indexRules support via filtering criteria on property
> --
>
> Key: OAK-6628
> URL: https://issues.apache.org/jira/browse/OAK-6628
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: lucene
>Reporter: Chetan Mehrotra
> Fix For: 1.8
>
>
> For Lucene index we currently support indexRules based on nodetype. Here the
> recommendation is that users must use most precise nodeType/mixinType to
> target the indexing rule so that only relevant nodes are indexed.
> For many Sling based applications its being seen that lots of content is
> nt:unstructured and it uses {{sling:resourceType}} property to distinguish
> various such nt:unstructured nodes. Currently its not possible to target
> index definition to index only those nt:unstructured which have specific
> {{sling:resourceType}}. Which makes it harder to provide a more precise index
> definitions.
> To help such cases we can generalize the indexRule support via a filtering
> criteria
> {noformat}
> activityIndex
> - type = "lucene"
> + indexRules
> + nt:unstructured
> - filter-property = "sling:resourceType"
> - filter-value = "app/activitystreams/components/activity"
> + properties
> - jcr:primaryType = "nt:unstructured"
> + verb
> - propertyIndex = true
> - name = "verb"
> {noformat}
> So indexRule would have 2 more config properties
> * filter-property - Name of property to match
> * filter-value - The value to match
> *Indexing*
> At time of indexing currently LuceneIndexEditor does a
> {{indexDefinition.getApplicableIndexingRule}} passing it the NodeState.
> Currently this checks only for jcr:PrimaryType and jxr:mixins to find
> matching rule.
> This logic would need to be extended to also check if any filter-property is
> defined in definition. If yes then check if NodeState has that value
> *Querying*
> On query side we need to change the IndexPlanner where it currently use query
> nodetype for finding matching indexRule. In addition it would need to pass on
> the property restrictions and the rule only be matched if the property
> restriction matches the filter
> *Open Item*
> # How to handle change in filter-property value. I think we have similar
> problem currently if an index nodes nodeType gets changed. In such a case we
> do not remove it from index. So we need to solve that for both
> # Ensure that all places where rules are matched account for this filter
> concept
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (OAK-3685) Support S3DataSource as the migration source
[ https://issues.apache.org/jira/browse/OAK-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Sedding reassigned OAK-3685: --- Assignee: Julian Sedding > Support S3DataSource as the migration source > > > Key: OAK-3685 > URL: https://issues.apache.org/jira/browse/OAK-3685 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: upgrade >Reporter: Tomek Rękawek >Assignee: Julian Sedding >Priority: Minor > Fix For: 1.3.12, 1.4 > > > In the current version, oak-upgrade doesn't support migrations from a > SegmentNodeStore or DocumentNodeStore with S3DataStore. Because some of the > S3DataStore users requests the possiblity of doing sidegrade (eg. from > SegmentNodeStore to Mongo), appropriate option should be added. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (OAK-3685) Support S3DataSource as the migration source
[ https://issues.apache.org/jira/browse/OAK-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Sedding reassigned OAK-3685: --- Assignee: (was: Julian Sedding) > Support S3DataSource as the migration source > > > Key: OAK-3685 > URL: https://issues.apache.org/jira/browse/OAK-3685 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: upgrade >Reporter: Tomek Rękawek >Priority: Minor > Fix For: 1.3.12, 1.4 > > > In the current version, oak-upgrade doesn't support migrations from a > SegmentNodeStore or DocumentNodeStore with S3DataStore. Because some of the > S3DataStore users requests the possiblity of doing sidegrade (eg. from > SegmentNodeStore to Mongo), appropriate option should be added. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (OAK-6220) Copy on write node store implementation
[ https://issues.apache.org/jira/browse/OAK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010111#comment-16010111 ] Julian Sedding commented on OAK-6220: - [~tomek.rekawek] that makes sense. Thanks for clarifying! > Copy on write node store implementation > --- > > Key: OAK-6220 > URL: https://issues.apache.org/jira/browse/OAK-6220 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: core >Reporter: Tomek Rękawek > Fix For: 1.8 > > Attachments: OAK-6220.patch > > > The copy-on-write (COW) node store implementation would allow to temporarily > switch the repository into the "testing" mode, in which all the changes are > saved in a reversible way. After switching back to the "production" mode, the > test changes should be dropped. > This should include checkpoints as well. > Because the SegmentNodeStore supports COW natively, we may simply copy the / > tree to some user-provided path and use it as a new root. For the > DocumentNodeStore we'll need something more complex. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-6220) Copy on write node store implementation
[ https://issues.apache.org/jira/browse/OAK-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010100#comment-16010100 ] Julian Sedding commented on OAK-6220: - [~tomek.rekawek] could you please explain if and in what way your proposal is different from setting a checkpoint and later rolling back to that checkpoint (i.e. resetting the HEAD)? I understand that you want checkpoints to work in the copy-on-write store. However, I fail to see why they wouldn't work if you just had a checkpoint. AFAIK checkpoints storage is also subject to MVCC and checkpoints should automatically be "restored" when you roll back to another checkpoint. Granted, we may need to add some additional tooling and allow switching compaction/GC into a more conservative mode. But all that should be solvable. > Copy on write node store implementation > --- > > Key: OAK-6220 > URL: https://issues.apache.org/jira/browse/OAK-6220 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: core >Reporter: Tomek Rękawek > Fix For: 1.8 > > Attachments: OAK-6220.patch > > > The copy-on-write (COW) node store implementation would allow to temporarily > switch the repository into the "testing" mode, in which all the changes are > saved in a reversible way. After switching back to the "production" mode, the > test changes should be dropped. > This should include checkpoints as well. > Because the SegmentNodeStore supports COW natively, we may simply copy the / > tree to some user-provided path and use it as a new root. For the > DocumentNodeStore we'll need something more complex. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (OAK-6136) Extract the multiplexing implementation code into a separate bundle
[ https://issues.apache.org/jira/browse/OAK-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Sedding resolved OAK-6136. - Resolution: Fixed Assignee: Julian Sedding Fixed in [r1793047|https://svn.apache.org/r1793047]. > Extract the multiplexing implementation code into a separate bundle > --- > > Key: OAK-6136 > URL: https://issues.apache.org/jira/browse/OAK-6136 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Robert Munteanu >Assignee: Julian Sedding > Fix For: 1.8 > > Attachments: > 0001-OAK-6136-Extract-the-multiplexing-implementation-cod.patch, > OAK-6136.patch > > > Since we already kicked off the m12n effort for 1.8 it would be good to > separate the multiplexing implementation from oak-core into its own bundle as > well. > I will look into providing a patch for the change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-6136) Extract the multiplexing implementation code into a separate bundle
[ https://issues.apache.org/jira/browse/OAK-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988625#comment-15988625 ] Julian Sedding commented on OAK-6136: - [~rombert] sure, I'll do it. > Extract the multiplexing implementation code into a separate bundle > --- > > Key: OAK-6136 > URL: https://issues.apache.org/jira/browse/OAK-6136 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Robert Munteanu > Fix For: 1.8 > > Attachments: > 0001-OAK-6136-Extract-the-multiplexing-implementation-cod.patch, > OAK-6136.patch > > > Since we already kicked off the m12n effort for 1.8 it would be good to > separate the multiplexing implementation from oak-core into its own bundle as > well. > I will look into providing a patch for the change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OAK-6136) Extract the multiplexing implementation code into a separate bundle
[ https://issues.apache.org/jira/browse/OAK-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988574#comment-15988574 ] Julian Sedding commented on OAK-6136: - [~rombert] the merged patch looks fine to me. Thanks. > Extract the multiplexing implementation code into a separate bundle > --- > > Key: OAK-6136 > URL: https://issues.apache.org/jira/browse/OAK-6136 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Robert Munteanu > Fix For: 1.8 > > Attachments: > 0001-OAK-6136-Extract-the-multiplexing-implementation-cod.patch, > OAK-6136.patch > > > Since we already kicked off the m12n effort for 1.8 it would be good to > separate the multiplexing implementation from oak-core into its own bundle as > well. > I will look into providing a patch for the change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (OAK-6136) Extract the multiplexing implementation code into a separate bundle
[ https://issues.apache.org/jira/browse/OAK-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988526#comment-15988526 ] Julian Sedding edited comment on OAK-6136 at 4/28/17 9:34 AM: -- [~rombert] I have the changes on [a branch on github|https://github.com/code-distillery/jackrabbit-oak/commit/6556c60ddea65c47c34ff9a27d6efd6bc7d276ee]. Feel free to pull them in. was (Author: jsedding): [~rombert] I have the changes on [a branch on github|https://github.com/code-distillery/jackrabbit-oak/tree/feature/OAK-6136-move-mount-info-to-spi]. Feel free to pull them in. > Extract the multiplexing implementation code into a separate bundle > --- > > Key: OAK-6136 > URL: https://issues.apache.org/jira/browse/OAK-6136 > Project: Jackrabbit Oak > Issue Type: Improvement >Reporter: Robert Munteanu > Fix For: 1.8 > > Attachments: > 0001-OAK-6136-Extract-the-multiplexing-implementation-cod.patch > > > Since we already kicked off the m12n effort for 1.8 it would be good to > separate the multiplexing implementation from oak-core into its own bundle as > well. > I will look into providing a patch for the change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
