[jira] [Created] (OAK-7229) Build Jackrabbit Oak #1218 failed
Hudson created OAK-7229: --- Summary: Build Jackrabbit Oak #1218 failed Key: OAK-7229 URL: https://issues.apache.org/jira/browse/OAK-7229 Project: Jackrabbit Oak Issue Type: Bug Components: continuous integration Reporter: Hudson No description is provided The build Jackrabbit Oak #1218 has failed. First failed run: [Jackrabbit Oak #1218|https://builds.apache.org/job/Jackrabbit%20Oak/1218/] [console log|https://builds.apache.org/job/Jackrabbit%20Oak/1218/console] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7223) Files could be kept partially in case of disconnection from backends
[ https://issues.apache.org/jira/browse/OAK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348147#comment-16348147 ] Amit Jain commented on OAK-7223: On trunk with http://svn.apache.org/viewvc?rev=1822850&view=rev > Files could be kept partially in case of disconnection from backends > > > Key: OAK-7223 > URL: https://issues.apache.org/jira/browse/OAK-7223 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.1 >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.9.0, 1.10, 1.8.2 > > Attachments: OAK-7223-2.patch, OAK-7223-3.patch, OAK-7223.patch > > > The FileCache#load method needs to clean the files which may have been > downloaded partially in case of errors from backend. This partially > downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7223) Files could be kept partially in case of disconnection from backends
[ https://issues.apache.org/jira/browse/OAK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Jain updated OAK-7223: --- Fix Version/s: 1.9.0 > Files could be kept partially in case of disconnection from backends > > > Key: OAK-7223 > URL: https://issues.apache.org/jira/browse/OAK-7223 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.1 >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.9.0, 1.10, 1.8.2 > > Attachments: OAK-7223-2.patch, OAK-7223-3.patch, OAK-7223.patch > > > The FileCache#load method needs to clean the files which may have been > downloaded partially in case of errors from backend. This partially > downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OAK-7227) MountPermissionProvider getNumEntries prone to overflow
[ https://issues.apache.org/jira/browse/OAK-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Deparvu resolved OAK-7227. --- Resolution: Fixed Fix Version/s: 1.8.2 applied patch provided on OAK-7228 with http://svn.apache.org/viewvc?rev=1822821&view=rev > MountPermissionProvider getNumEntries prone to overflow > --- > > Key: OAK-7227 > URL: https://issues.apache.org/jira/browse/OAK-7227 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core, security >Reporter: Alex Deparvu >Assignee: Alex Deparvu >Priority: Trivial > Fix For: 1.9.0, 1.8.2 > > > It's about this method MountPermissionProvider#getNumEntries [0] > thanks [~anchela] for spotting! > [0] > https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/security/authorization/permission/MountPermissionProvider.java#L105 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7227) MountPermissionProvider getNumEntries prone to overflow
[ https://issues.apache.org/jira/browse/OAK-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Deparvu updated OAK-7227: -- Issue Type: Bug (was: Improvement) > MountPermissionProvider getNumEntries prone to overflow > --- > > Key: OAK-7227 > URL: https://issues.apache.org/jira/browse/OAK-7227 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core, security >Reporter: Alex Deparvu >Assignee: Alex Deparvu >Priority: Trivial > Fix For: 1.9.0 > > > It's about this method MountPermissionProvider#getNumEntries [0] > thanks [~anchela] for spotting! > [0] > https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/security/authorization/permission/MountPermissionProvider.java#L105 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7227) MountPermissionProvider getNumEntries prone to overflow
[ https://issues.apache.org/jira/browse/OAK-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347034#comment-16347034 ] angela commented on OAK-7227: - see OAK-7228 for a proposed (but untested) patch :) > MountPermissionProvider getNumEntries prone to overflow > --- > > Key: OAK-7227 > URL: https://issues.apache.org/jira/browse/OAK-7227 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, security >Reporter: Alex Deparvu >Assignee: Alex Deparvu >Priority: Trivial > Fix For: 1.9.0 > > > It's about this method MountPermissionProvider#getNumEntries [0] > thanks [~anchela] for spotting! > [0] > https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/security/authorization/permission/MountPermissionProvider.java#L105 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OAK-7228) Potential long overflow in MountPermissionProvider.getNumEntries
[ https://issues.apache.org/jira/browse/OAK-7228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angela resolved OAK-7228. - Resolution: Duplicate :-) > Potential long overflow in MountPermissionProvider.getNumEntries > - > > Key: OAK-7228 > URL: https://issues.apache.org/jira/browse/OAK-7228 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core, security >Reporter: angela >Priority: Major > > [~stillalex], just came across {{MountPermissionProvider.getNumEntries}}, > which looks as follows: > {code} > @Override > public long getNumEntries(String principalName, long max) { > long num = 0; > for (PermissionStoreImpl store : stores) { > num += store.getNumEntries(principalName, max); > if (num >= max) { > break; > } > } > return num; > } > {code} > If I am not mistaken this may lead to long overflow similar to the one we > spotted it in {{PermissionEntryProviderImpl.init}}. > Proposed (but untested fix) could look as follows: > {code} > @Override > public long getNumEntries(String principalName, long max) { > long num = 0; > for (PermissionStoreImpl store : stores) { > num = LongUtils.safeAdd(num, > store.getNumEntries(principalName, max)) > if (num >= max) { > break; > } > } > return num; > } > {code} > wdyt? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7228) Potential long overflow in MountPermissionProvider.getNumEntries
[ https://issues.apache.org/jira/browse/OAK-7228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angela updated OAK-7228: Description: [~stillalex], just came across {{MountPermissionProvider.getNumEntries}}, which looks as follows: {code} @Override public long getNumEntries(String principalName, long max) { long num = 0; for (PermissionStoreImpl store : stores) { num += store.getNumEntries(principalName, max); if (num >= max) { break; } } return num; } {code} If I am not mistaken this may lead to long overflow similar to the one we spotted it in {{PermissionEntryProviderImpl.init}}. Proposed (but untested fix) could look as follows: {code} @Override public long getNumEntries(String principalName, long max) { long num = 0; for (PermissionStoreImpl store : stores) { num = LongUtils.safeAdd(num, store.getNumEntries(principalName, max)) if (num >= max) { break; } } return num; } {code} wdyt? was: [~stillalex], just came across {{MountPermissionProvider.getNumEntries}}, which looks as follows: {code} @Override public long getNumEntries(String principalName, long max) { long num = 0; for (PermissionStoreImpl store : stores) { num += store.getNumEntries(principalName, max); if (num >= max) { break; } } return num; } {code} If I am not mistaken this may lead to long overflow similar to the one we spotted it in {{PermissionEntryProviderImpl.init}}. Proposed (but untested fix) could look as follows: {code} @Override public long getNumEntries(String principalName, long max) { long num = 0; for (PermissionStoreImpl store : stores) { num = LongUtils.safeAdd(num, store.getNumEntries(principalName, max)) if (num >= max) { break; } } return num; } {code} > Potential long overflow in MountPermissionProvider.getNumEntries > - > > Key: OAK-7228 > URL: https://issues.apache.org/jira/browse/OAK-7228 > Project: Jackrabbit Oak > Issue Type: Bug > Components: core, security >Reporter: angela >Priority: Major > > [~stillalex], just came across {{MountPermissionProvider.getNumEntries}}, > which looks as follows: > {code} > @Override > public long getNumEntries(String principalName, long max) { > long num = 0; > for (PermissionStoreImpl store : stores) { > num += store.getNumEntries(principalName, max); > if (num >= max) { > break; > } > } > return num; > } > {code} > If I am not mistaken this may lead to long overflow similar to the one we > spotted it in {{PermissionEntryProviderImpl.init}}. > Proposed (but untested fix) could look as follows: > {code} > @Override > public long getNumEntries(String principalName, long max) { > long num = 0; > for (PermissionStoreImpl store : stores) { > num = LongUtils.safeAdd(num, > store.getNumEntries(principalName, max)) > if (num >= max) { > break; > } > } > return num; > } > {code} > wdyt? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OAK-7228) Potential long overflow in MountPermissionProvider.getNumEntries
angela created OAK-7228: --- Summary: Potential long overflow in MountPermissionProvider.getNumEntries Key: OAK-7228 URL: https://issues.apache.org/jira/browse/OAK-7228 Project: Jackrabbit Oak Issue Type: Bug Components: core, security Reporter: angela [~stillalex], just came across {{MountPermissionProvider.getNumEntries}}, which looks as follows: {code} @Override public long getNumEntries(String principalName, long max) { long num = 0; for (PermissionStoreImpl store : stores) { num += store.getNumEntries(principalName, max); if (num >= max) { break; } } return num; } {code} If I am not mistaken this may lead to long overflow similar to the one we spotted it in {{PermissionEntryProviderImpl.init}}. Proposed (but untested fix) could look as follows: {code} @Override public long getNumEntries(String principalName, long max) { long num = 0; for (PermissionStoreImpl store : stores) { num = LongUtils.safeAdd(num, store.getNumEntries(principalName, max)) if (num >= max) { break; } } return num; } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OAK-7227) MountPermissionProvider getNumEntries prone to overflow
Alex Deparvu created OAK-7227: - Summary: MountPermissionProvider getNumEntries prone to overflow Key: OAK-7227 URL: https://issues.apache.org/jira/browse/OAK-7227 Project: Jackrabbit Oak Issue Type: Improvement Components: core, security Reporter: Alex Deparvu Assignee: Alex Deparvu Fix For: 1.9.0 It's about this method MountPermissionProvider#getNumEntries [0] thanks [~anchela] for spotting! [0] https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/security/authorization/permission/MountPermissionProvider.java#L105 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-5506) reject item names with unpaired surrogates early
[ https://issues.apache.org/jira/browse/OAK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347006#comment-16347006 ] Michael Dürig commented on OAK-5506: Agree wrt. performance. In other places we already switched to byte buffers. Wrt. compatibility, this sounds reasonable. But I think we need to come up with a test case writing the old way and reading the new way. This should be doable if there is a way to disable the new behaviour I guess. > reject item names with unpaired surrogates early > > > Key: OAK-5506 > URL: https://issues.apache.org/jira/browse/OAK-5506 > Project: Jackrabbit Oak > Issue Type: Wish > Components: core, jcr, segment-tar >Affects Versions: 1.5.18 >Reporter: Julian Reschke >Assignee: Francesco Mari >Priority: Minor > Fix For: 1.10 > > Attachments: OAK-5506-01.patch, OAK-5506-02.patch, OAK-5506-4.diff, > OAK-5506-bench.diff, OAK-5506-name-conversion.diff, OAK-5506-segment.diff, > OAK-5506-segment2.diff, OAK-5506.diff, ValidNamesTest.java > > > Apparently, the following node name is accepted: >{{"foo\ud800"}} > but a subsequent {{getPath()}} call fails: > {noformat} > javax.jcr.InvalidItemStateException: This item [/test_node/foo?] does not > exist anymore > at > org.apache.jackrabbit.oak.jcr.delegate.ItemDelegate.checkAlive(ItemDelegate.java:86) > at > org.apache.jackrabbit.oak.jcr.session.operation.ItemOperation.checkPreconditions(ItemOperation.java:34) > at > org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.prePerform(SessionDelegate.java:615) > at > org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.perform(SessionDelegate.java:205) > at > org.apache.jackrabbit.oak.jcr.session.ItemImpl.perform(ItemImpl.java:112) > at > org.apache.jackrabbit.oak.jcr.session.ItemImpl.getPath(ItemImpl.java:140) > at > org.apache.jackrabbit.oak.jcr.session.NodeImpl.getPath(NodeImpl.java:106) > at > org.apache.jackrabbit.oak.jcr.ValidNamesTest.nameTest(ValidNamesTest.java:271) > at > org.apache.jackrabbit.oak.jcr.ValidNamesTest.testUnpairedSurrogate(ValidNamesTest.java:259) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source){noformat} > (test case follows) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-5506) reject item names with unpaired surrogates early
[ https://issues.apache.org/jira/browse/OAK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346957#comment-16346957 ] Julian Reschke commented on OAK-5506: - With respect to compatibility with segment-tar storage written by previous versions: my understanding is that any string value (be it an item name or a string property) persisted to segment-tar will have gone through {{String.getBytes(UTF8)}}. In the case of a string that doesn't roundtrip through UTF-8, this means that those characters will have been replaced by a replacement character (here: "?"). So the persisted state *does* represent valid UTF-8, and when read back, will just use that replacement character. The proposed patch only affects writing of strings that are invalid, thus previously written garbage shouldn't be an issue. > reject item names with unpaired surrogates early > > > Key: OAK-5506 > URL: https://issues.apache.org/jira/browse/OAK-5506 > Project: Jackrabbit Oak > Issue Type: Wish > Components: core, jcr, segment-tar >Affects Versions: 1.5.18 >Reporter: Julian Reschke >Assignee: Francesco Mari >Priority: Minor > Fix For: 1.10 > > Attachments: OAK-5506-01.patch, OAK-5506-02.patch, OAK-5506-4.diff, > OAK-5506-bench.diff, OAK-5506-name-conversion.diff, OAK-5506-segment.diff, > OAK-5506-segment2.diff, OAK-5506.diff, ValidNamesTest.java > > > Apparently, the following node name is accepted: >{{"foo\ud800"}} > but a subsequent {{getPath()}} call fails: > {noformat} > javax.jcr.InvalidItemStateException: This item [/test_node/foo?] does not > exist anymore > at > org.apache.jackrabbit.oak.jcr.delegate.ItemDelegate.checkAlive(ItemDelegate.java:86) > at > org.apache.jackrabbit.oak.jcr.session.operation.ItemOperation.checkPreconditions(ItemOperation.java:34) > at > org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.prePerform(SessionDelegate.java:615) > at > org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.perform(SessionDelegate.java:205) > at > org.apache.jackrabbit.oak.jcr.session.ItemImpl.perform(ItemImpl.java:112) > at > org.apache.jackrabbit.oak.jcr.session.ItemImpl.getPath(ItemImpl.java:140) > at > org.apache.jackrabbit.oak.jcr.session.NodeImpl.getPath(NodeImpl.java:106) > at > org.apache.jackrabbit.oak.jcr.ValidNamesTest.nameTest(ValidNamesTest.java:271) > at > org.apache.jackrabbit.oak.jcr.ValidNamesTest.testUnpairedSurrogate(ValidNamesTest.java:259) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source){noformat} > (test case follows) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OAK-7162) Race condition on revisions head between compaction and scheduler could result in skipped commit
[ https://issues.apache.org/jira/browse/OAK-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345030#comment-16345030 ] Andrei Dulceanu edited comment on OAK-7162 at 1/31/18 3:01 PM: --- Fixed in trunk at r1822640. was (Author: dulceanu): Fixed at r1822640. > Race condition on revisions head between compaction and scheduler could > result in skipped commit > > > Key: OAK-7162 > URL: https://issues.apache.org/jira/browse/OAK-7162 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-tar >Affects Versions: 1.8.0 >Reporter: Andrei Dulceanu >Assignee: Andrei Dulceanu >Priority: Blocker > Labels: scalability > Fix For: 1.9.0, 1.10, 1.8.2 > > Attachments: OAK-7162-02.patch, OAK-7162-03.patch, OAK-7162.patch > > > There is a race condition on {{TarRevisions#head}} between a running > compaction trying to set the new head [0] and the scheduler doing the same > after executing a specific commit [1]. If the compaction thread is first, > then the head assignment in the scheduler will fail and not be re-attempted. > IMO, the simple if statement should be changed to a while loop in which the > head is refreshed and the commit is re-applied against the new head, before > attempting again to set a new head in {{TarRevisions}}. This is somehow > similar to what we previously had [2], but without the unneeded > optimistic/pessimistic strategies involving tokens. > [0] > https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L764 > [1] > https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java#L253 > [2] > https://github.com/apache/jackrabbit-oak/blob/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStore.java#L686 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-5506) reject item names with unpaired surrogates early
[ https://issues.apache.org/jira/browse/OAK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346962#comment-16346962 ] Julian Reschke commented on OAK-5506: - With respect to performance: this code change opens the door to serialize to {{ByteBuffer}}s instead of {{byte[]}}, potentially avoiding array copies. Not sure whether segment-tar could take advantage of that... > reject item names with unpaired surrogates early > > > Key: OAK-5506 > URL: https://issues.apache.org/jira/browse/OAK-5506 > Project: Jackrabbit Oak > Issue Type: Wish > Components: core, jcr, segment-tar >Affects Versions: 1.5.18 >Reporter: Julian Reschke >Assignee: Francesco Mari >Priority: Minor > Fix For: 1.10 > > Attachments: OAK-5506-01.patch, OAK-5506-02.patch, OAK-5506-4.diff, > OAK-5506-bench.diff, OAK-5506-name-conversion.diff, OAK-5506-segment.diff, > OAK-5506-segment2.diff, OAK-5506.diff, ValidNamesTest.java > > > Apparently, the following node name is accepted: >{{"foo\ud800"}} > but a subsequent {{getPath()}} call fails: > {noformat} > javax.jcr.InvalidItemStateException: This item [/test_node/foo?] does not > exist anymore > at > org.apache.jackrabbit.oak.jcr.delegate.ItemDelegate.checkAlive(ItemDelegate.java:86) > at > org.apache.jackrabbit.oak.jcr.session.operation.ItemOperation.checkPreconditions(ItemOperation.java:34) > at > org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.prePerform(SessionDelegate.java:615) > at > org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.perform(SessionDelegate.java:205) > at > org.apache.jackrabbit.oak.jcr.session.ItemImpl.perform(ItemImpl.java:112) > at > org.apache.jackrabbit.oak.jcr.session.ItemImpl.getPath(ItemImpl.java:140) > at > org.apache.jackrabbit.oak.jcr.session.NodeImpl.getPath(NodeImpl.java:106) > at > org.apache.jackrabbit.oak.jcr.ValidNamesTest.nameTest(ValidNamesTest.java:271) > at > org.apache.jackrabbit.oak.jcr.ValidNamesTest.testUnpairedSurrogate(ValidNamesTest.java:259) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source){noformat} > (test case follows) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7188) guava: ListenableFuture.transform() changes to transformAsync in version 20
[ https://issues.apache.org/jira/browse/OAK-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346949#comment-16346949 ] Julian Reschke commented on OAK-7188: - a) I don't see how, if the test is supposed to run with Oak versions using both old and new Guava - am I missing something? b) If you can change the test not to need this piece of Guava, that would of course be preferable. > guava: ListenableFuture.transform() changes to transformAsync in version 20 > --- > > Key: OAK-7188 > URL: https://issues.apache.org/jira/browse/OAK-7188 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: segment-tar >Reporter: Julian Reschke >Priority: Major > Attachments: OAK-7188.diff, OAK-7188.diff > > > See > https://google.github.io/guava/releases/19.0/api/docs/com/google/common/util/concurrent/Futures.html#transform(com.google.common.util.concurrent.ListenableFuture,%20com.google.common.util.concurrent.AsyncFunction) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7188) guava: ListenableFuture.transform() changes to transformAsync in version 20
[ https://issues.apache.org/jira/browse/OAK-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346932#comment-16346932 ] Michael Dürig commented on OAK-7188: Can't we have a stable test dependency to Guava? If not, I'd prefer to rewrite that functionality by other means instead of relying on reflection. > guava: ListenableFuture.transform() changes to transformAsync in version 20 > --- > > Key: OAK-7188 > URL: https://issues.apache.org/jira/browse/OAK-7188 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: segment-tar >Reporter: Julian Reschke >Priority: Major > Attachments: OAK-7188.diff, OAK-7188.diff > > > See > https://google.github.io/guava/releases/19.0/api/docs/com/google/common/util/concurrent/Futures.html#transform(com.google.common.util.concurrent.ListenableFuture,%20com.google.common.util.concurrent.AsyncFunction) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OAK-7198) Index rule with REGEX_ALL_PROPS includes relative node
[ https://issues.apache.org/jira/browse/OAK-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger resolved OAK-7198. --- Resolution: Fixed Fix Version/s: 1.9.0 Thanks for the review. Applied the patch to trunk: http://svn.apache.org/r1822808 > Index rule with REGEX_ALL_PROPS includes relative node > -- > > Key: OAK-7198 > URL: https://issues.apache.org/jira/browse/OAK-7198 > Project: Jackrabbit Oak > Issue Type: Bug > Components: lucene >Affects Versions: 1.8.0 >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Labels: candidate_oak_1_8 > Fix For: 1.9.0, 1.10 > > Attachments: OAK-7198.patch > > > A lucene index with an index rule that includes properties with > {{LuceneIndexConstants.REGEX_ALL_PROPS}} also includes some child node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7198) Index rule with REGEX_ALL_PROPS includes relative node
[ https://issues.apache.org/jira/browse/OAK-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger updated OAK-7198: -- Labels: candidate_oak_1_8 (was: ) > Index rule with REGEX_ALL_PROPS includes relative node > -- > > Key: OAK-7198 > URL: https://issues.apache.org/jira/browse/OAK-7198 > Project: Jackrabbit Oak > Issue Type: Bug > Components: lucene >Affects Versions: 1.8.0 >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Labels: candidate_oak_1_8 > Fix For: 1.9.0, 1.10 > > Attachments: OAK-7198.patch > > > A lucene index with an index rule that includes properties with > {{LuceneIndexConstants.REGEX_ALL_PROPS}} also includes some child node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7188) guava: ListenableFuture.transform() changes to transformAsync in version 20
[ https://issues.apache.org/jira/browse/OAK-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-7188: Attachment: OAK-7188.diff > guava: ListenableFuture.transform() changes to transformAsync in version 20 > --- > > Key: OAK-7188 > URL: https://issues.apache.org/jira/browse/OAK-7188 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: segment-tar >Reporter: Julian Reschke >Priority: Major > Attachments: OAK-7188.diff, OAK-7188.diff > > > See > https://google.github.io/guava/releases/19.0/api/docs/com/google/common/util/concurrent/Futures.html#transform(com.google.common.util.concurrent.ListenableFuture,%20com.google.common.util.concurrent.AsyncFunction) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7188) guava: ListenableFuture.transform() changes to transformAsync in version 20
[ https://issues.apache.org/jira/browse/OAK-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346920#comment-16346920 ] Julian Reschke commented on OAK-7188: - [^OAK-7188.diff] uses the proper method using reflection and should be compatible with Guava 15..20 (or more) > guava: ListenableFuture.transform() changes to transformAsync in version 20 > --- > > Key: OAK-7188 > URL: https://issues.apache.org/jira/browse/OAK-7188 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: segment-tar >Reporter: Julian Reschke >Priority: Major > Attachments: OAK-7188.diff, OAK-7188.diff > > > See > https://google.github.io/guava/releases/19.0/api/docs/com/google/common/util/concurrent/Futures.html#transform(com.google.common.util.concurrent.ListenableFuture,%20com.google.common.util.concurrent.AsyncFunction) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OAK-7198) Index rule with REGEX_ALL_PROPS includes relative node
[ https://issues.apache.org/jira/browse/OAK-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger reassigned OAK-7198: - Assignee: Marcel Reutegger > Index rule with REGEX_ALL_PROPS includes relative node > -- > > Key: OAK-7198 > URL: https://issues.apache.org/jira/browse/OAK-7198 > Project: Jackrabbit Oak > Issue Type: Bug > Components: lucene >Affects Versions: 1.8.0 >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Fix For: 1.10 > > Attachments: OAK-7198.patch > > > A lucene index with an index rule that includes properties with > {{LuceneIndexConstants.REGEX_ALL_PROPS}} also includes some child node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7213) Avoid call for child node when bundle contains all children
[ https://issues.apache.org/jira/browse/OAK-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346878#comment-16346878 ] Marcel Reutegger commented on OAK-7213: --- Thanks for the review. I added an additional assertion to the test to check for the {{_children}} flag. In trunk: http://svn.apache.org/r1822802 > Avoid call for child node when bundle contains all children > --- > > Key: OAK-7213 > URL: https://issues.apache.org/jira/browse/OAK-7213 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: documentmk >Reporter: Marcel Reutegger >Assignee: Marcel Reutegger >Priority: Minor > Labels: bundling > Fix For: 1.9.0, 1.10 > > Attachments: OAK-7213.patch > > > When nodes are bundled in a document, the DocumentNodeStore keeps track of > whether all children are included in a document. The presence of the hidden > {{:doc-has-child-non-bundled}} property indicates there are non bundled child > nodes. For the case when a document contains all children in the bundle, the > DocumentNodeStore still does a find call on the DocumentStore when asked for > an unknown child node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-5506) reject item names with unpaired surrogates early
[ https://issues.apache.org/jira/browse/OAK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346875#comment-16346875 ] Julian Reschke commented on OAK-5506: - ...was just looking at the last columns. Will rerun later with more iterations. > reject item names with unpaired surrogates early > > > Key: OAK-5506 > URL: https://issues.apache.org/jira/browse/OAK-5506 > Project: Jackrabbit Oak > Issue Type: Wish > Components: core, jcr, segment-tar >Affects Versions: 1.5.18 >Reporter: Julian Reschke >Assignee: Francesco Mari >Priority: Minor > Fix For: 1.10 > > Attachments: OAK-5506-01.patch, OAK-5506-02.patch, OAK-5506-4.diff, > OAK-5506-bench.diff, OAK-5506-name-conversion.diff, OAK-5506-segment.diff, > OAK-5506-segment2.diff, OAK-5506.diff, ValidNamesTest.java > > > Apparently, the following node name is accepted: >{{"foo\ud800"}} > but a subsequent {{getPath()}} call fails: > {noformat} > javax.jcr.InvalidItemStateException: This item [/test_node/foo?] does not > exist anymore > at > org.apache.jackrabbit.oak.jcr.delegate.ItemDelegate.checkAlive(ItemDelegate.java:86) > at > org.apache.jackrabbit.oak.jcr.session.operation.ItemOperation.checkPreconditions(ItemOperation.java:34) > at > org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.prePerform(SessionDelegate.java:615) > at > org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.perform(SessionDelegate.java:205) > at > org.apache.jackrabbit.oak.jcr.session.ItemImpl.perform(ItemImpl.java:112) > at > org.apache.jackrabbit.oak.jcr.session.ItemImpl.getPath(ItemImpl.java:140) > at > org.apache.jackrabbit.oak.jcr.session.NodeImpl.getPath(NodeImpl.java:106) > at > org.apache.jackrabbit.oak.jcr.ValidNamesTest.nameTest(ValidNamesTest.java:271) > at > org.apache.jackrabbit.oak.jcr.ValidNamesTest.testUnpairedSurrogate(ValidNamesTest.java:259) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source){noformat} > (test case follows) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7194) Threads are going in blocked state - OAK segment
[ https://issues.apache.org/jira/browse/OAK-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-7194: --- Issue Type: Improvement (was: Bug) > Threads are going in blocked state - OAK segment > > > Key: OAK-7194 > URL: https://issues.apache.org/jira/browse/OAK-7194 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-tar >Affects Versions: 1.6.2 >Reporter: Kshitiz Garg >Priority: Critical > > We are using AEM and it internally is using OAK 1.6.2. Our application is > going to a choked state many a times. Our thread dump analysis is always > pointing to this stack trace with many blocked threads. Is it a known issue? > Or is there a setting to avoid it? We are using tar files based repository > underneath and are using default settings: > > {noformat} > - nativeId:0x1208 - state:BLOCKED > stackTrace: > java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:121) > - waiting to lock <0x000414d8c250> (a > org.apache.jackrabbit.oak.segment.SegmentId) > at org.apache.jackrabbit.oak.segment.Record.getSegment(Record.java:70) > at org.apache.jackrabbit.oak.segment.MapRecord.getEntry(MapRecord.java:160) > at org.apache.jackrabbit.oak.segment.MapRecord.getEntry(MapRecord.java:192) > at org.apache.jackrabbit.oak.segment.MapRecord.getEntry(MapRecord.java:192) > at org.apache.jackrabbit.oak.segment.MapRecord.getEntry(MapRecord.java:192) > at > org.apache.jackrabbit.oak.segment.SegmentNodeState.getChildNode(SegmentNodeState.java:412) > at > org.apache.jackrabbit.oak.plugins.tree.impl.ImmutableTree.createChild(ImmutableTree.java:125) > at > org.apache.jackrabbit.oak.plugins.tree.impl.ImmutableTree.getChild(ImmutableTree.java:176) > at > org.apache.jackrabbit.oak.plugins.tree.impl.ImmutableTree.getChild(ImmutableTree.java:81) > at > org.apache.jackrabbit.oak.security.authorization.permission.PermissionUtil.getPrincipalRoot(PermissionUtil.java:83) > at > org.apache.jackrabbit.oak.security.authorization.permission.PermissionStoreImpl.getPrincipalRoot(PermissionStoreImpl.java:132) > at > org.apache.jackrabbit.oak.security.authorization.permission.PermissionStoreImpl.getNumEntries(PermissionStoreImpl.java:103) > at > org.apache.jackrabbit.oak.security.authorization.permission.PermissionEntryCache.getNumEntries(PermissionEntryCache.java:102) > at > org.apache.jackrabbit.oak.security.authorization.permission.PermissionEntryProviderImpl.init(PermissionEntryProviderImpl.java:79) > at > org.apache.jackrabbit.oak.security.authorization.permission.PermissionEntryProviderImpl.(PermissionEntryProviderImpl.java:72) > at > org.apache.jackrabbit.oak.security.authorization.permission.CompiledPermissionImpl.(CompiledPermissionImpl.java:112) > at > org.apache.jackrabbit.oak.security.authorization.permission.CompiledPermissionImpl.create(CompiledPermissionImpl.java:126) > at > org.apache.jackrabbit.oak.security.authorization.permission.PermissionProviderImpl.getCompiledPermissions(PermissionProviderImpl.java:162) > at > org.apache.jackrabbit.oak.security.authorization.permission.PermissionProviderImpl.getTreePermission(PermissionProviderImpl.java:151) > at > org.apache.jackrabbit.oak.security.authorization.composite.CompositeTreePermission.create(CompositeTreePermission.java:67) > at > org.apache.jackrabbit.oak.security.authorization.composite.CompositePermissionProvider.getTreePermission(CompositePermissionProvider.java:147) > at > org.apache.jackrabbit.oak.core.SecureNodeBuilder.getTreePermission(SecureNodeBuilder.java:357) > at > org.apache.jackrabbit.oak.core.SecureNodeBuilder.getTreePermission(SecureNodeBuilder.java:360) > at > org.apache.jackrabbit.oak.core.SecureNodeBuilder.getTreePermission(SecureNodeBuilder.java:360) > at > org.apache.jackrabbit.oak.core.SecureNodeBuilder.access$100(SecureNodeBuilder.java:49) > at > org.apache.jackrabbit.oak.core.SecureNodeBuilder$ReadablePropertyPredicate.apply(SecureNodeBuilder.java:377) > at > org.apache.jackrabbit.oak.core.SecureNodeBuilder.getProperty(SecureNodeBuilder.java:184) > at > org.apache.jackrabbit.oak.plugins.tree.impl.AbstractTree.getProperty(AbstractTree.java:249) > at > org.apache.jackrabbit.oak.core.MutableTree.getProperty(MutableTree.java:128) > at > org.apache.jackrabbit.oak.util.TreeUtil.getStringInternal(TreeUtil.java:108) > at org.apache.jackrabbit.oak.util.TreeUtil.getString(TreeUtil.java:101) > at > org.apache.jackrabbit.oak.namepath.GlobalNameMapper.getNamespacesProperty(GlobalNameMapper.java:191) > at > org.apache.jackrabbit.oak.namepath.GlobalNameMapper.getOakURIOrNull(GlobalNameMapper.java:187) > - eliminated <0x00070c692f50> (a > org.apache.jackrabbit.oak.jcr.session.SessionNamespaces) > at > org.apache.jackrab
[jira] [Resolved] (OAK-7226) Warn messages ignoring exception parameter
[ https://issues.apache.org/jira/browse/OAK-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger resolved OAK-7226. --- Resolution: Invalid Fix Version/s: (was: 1.10) I see, thanks for the pointer. Resolving as invalid then. > Warn messages ignoring exception parameter > -- > > Key: OAK-7226 > URL: https://issues.apache.org/jira/browse/OAK-7226 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.0 >Reporter: Marcel Reutegger >Priority: Minor > > There are a few usages of Logger.warn() with a pattern similar to this > example: > {noformat} > log.warn("Error occurred while fetching DataRecord for identifier {}", input, > exeption); > {noformat} > The intention probably is that the third parameter is treated as an exception > and e.g. logged with the stack trace. However, this method signature > interprets the exception as a second argument for the message format. This > means the exception is effectively ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7203) Make MountInfoProvider service in AuthorizationConfigurationImpl optional
[ https://issues.apache.org/jira/browse/OAK-7203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346838#comment-16346838 ] Oliver Lietz commented on OAK-7203: --- [~stillalex], it's ugly main class (those methods are not required, only used for testing) vs ugly test class and I prefer ugly test classes in general (you can make it nicer by adding exception to method signatures of overridden methods). > Make MountInfoProvider service in AuthorizationConfigurationImpl optional > - > > Key: OAK-7203 > URL: https://issues.apache.org/jira/browse/OAK-7203 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: authorization-cug, core >Affects Versions: 1.8.1 >Reporter: Oliver Lietz >Priority: Major > Attachments: OAK-7203.patch > > > While testing Sling with Oak 1.8 I've observed that > AuthorizationConfigurationImpl gets not activated due to missing > MountInfoProvider service: > {noformat} > @Reference > private MountInfoProvider mountInfoProvider = > Mounts.defaultMountInfoProvider(); > {noformat} > {noformat} > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > Bundleorg.apache.jackrabbit.oak-core (63) > Implementation Class > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > Default State enabled > Activationdelayed > Configuration Policy optional > Service Type singleton > Services > org.apache.jackrabbit.oak.spi.security.authorization.AuthorizationConfiguration > org.apache.jackrabbit.oak.spi.security.SecurityConfiguration > PID > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > Reference mountInfoProvider Unsatisfied > Service Name: org.apache.jackrabbit.oak.spi.mount.MountInfoProvider > Cardinality: 1..1 > Policy: static > Policy Option: reluctant > No Services bound > Propertiescomponent.id = 35 > component.name = > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > configurationRanking = 100 > importBehavior = abort > oak.security.name = > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > readPaths = [/jcr:system/rep:namespaces, /jcr:system/jcr:nodeTypes, > /jcr:system/rep:privileges] > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OAK-7226) Warn messages ignoring exception parameter
[ https://issues.apache.org/jira/browse/OAK-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346825#comment-16346825 ] Chetan Mehrotra edited comment on OAK-7226 at 1/31/18 1:33 PM: --- Logback also supports it [1]. Also this is stated in [Logger|https://www.slf4j.org/api/org/slf4j/Logger.html] docs bq. Note that logging statements can be parameterized in presence of an exception/throwable. [1] https://github.com/qos-ch/logback/blob/master/logback-classic/src/main/java/ch/qos/logback/classic/spi/LoggingEvent.java#L114 was (Author: chetanm): Logback also supports it [1] [1] https://github.com/qos-ch/logback/blob/master/logback-classic/src/main/java/ch/qos/logback/classic/spi/LoggingEvent.java#L114 > Warn messages ignoring exception parameter > -- > > Key: OAK-7226 > URL: https://issues.apache.org/jira/browse/OAK-7226 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.0 >Reporter: Marcel Reutegger >Priority: Minor > Fix For: 1.10 > > > There are a few usages of Logger.warn() with a pattern similar to this > example: > {noformat} > log.warn("Error occurred while fetching DataRecord for identifier {}", input, > exeption); > {noformat} > The intention probably is that the third parameter is treated as an exception > and e.g. logged with the stack trace. However, this method signature > interprets the exception as a second argument for the message format. This > means the exception is effectively ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7203) Make MountInfoProvider service in AuthorizationConfigurationImpl optional
[ https://issues.apache.org/jira/browse/OAK-7203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346826#comment-16346826 ] Oliver Lietz commented on OAK-7203: --- +1 for moving {{MountInfoProviderService}} to {{oak-core}} > Make MountInfoProvider service in AuthorizationConfigurationImpl optional > - > > Key: OAK-7203 > URL: https://issues.apache.org/jira/browse/OAK-7203 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: authorization-cug, core >Affects Versions: 1.8.1 >Reporter: Oliver Lietz >Priority: Major > Attachments: OAK-7203.patch > > > While testing Sling with Oak 1.8 I've observed that > AuthorizationConfigurationImpl gets not activated due to missing > MountInfoProvider service: > {noformat} > @Reference > private MountInfoProvider mountInfoProvider = > Mounts.defaultMountInfoProvider(); > {noformat} > {noformat} > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > Bundleorg.apache.jackrabbit.oak-core (63) > Implementation Class > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > Default State enabled > Activationdelayed > Configuration Policy optional > Service Type singleton > Services > org.apache.jackrabbit.oak.spi.security.authorization.AuthorizationConfiguration > org.apache.jackrabbit.oak.spi.security.SecurityConfiguration > PID > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > Reference mountInfoProvider Unsatisfied > Service Name: org.apache.jackrabbit.oak.spi.mount.MountInfoProvider > Cardinality: 1..1 > Policy: static > Policy Option: reluctant > No Services bound > Propertiescomponent.id = 35 > component.name = > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > configurationRanking = 100 > importBehavior = abort > oak.security.name = > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > readPaths = [/jcr:system/rep:namespaces, /jcr:system/jcr:nodeTypes, > /jcr:system/rep:privileges] > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7226) Warn messages ignoring exception parameter
[ https://issues.apache.org/jira/browse/OAK-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346825#comment-16346825 ] Chetan Mehrotra commented on OAK-7226: -- Logback also supports it [1] [1] https://github.com/qos-ch/logback/blob/master/logback-classic/src/main/java/ch/qos/logback/classic/spi/LoggingEvent.java#L114 > Warn messages ignoring exception parameter > -- > > Key: OAK-7226 > URL: https://issues.apache.org/jira/browse/OAK-7226 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.0 >Reporter: Marcel Reutegger >Priority: Minor > Fix For: 1.10 > > > There are a few usages of Logger.warn() with a pattern similar to this > example: > {noformat} > log.warn("Error occurred while fetching DataRecord for identifier {}", input, > exeption); > {noformat} > The intention probably is that the third parameter is treated as an exception > and e.g. logged with the stack trace. However, this method signature > interprets the exception as a second argument for the message format. This > means the exception is effectively ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7226) Warn messages ignoring exception parameter
[ https://issues.apache.org/jira/browse/OAK-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346806#comment-16346806 ] Marcel Reutegger commented on OAK-7226: --- Hmm, now I'm confused. How can slf4j do that? Isn't this up to the actual logging framework how to implement the interface defined by slf4j? E.g. we use logback in Oak, which doesn't seem to have this 'feature'. > Warn messages ignoring exception parameter > -- > > Key: OAK-7226 > URL: https://issues.apache.org/jira/browse/OAK-7226 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.0 >Reporter: Marcel Reutegger >Priority: Minor > Fix For: 1.10 > > > There are a few usages of Logger.warn() with a pattern similar to this > example: > {noformat} > log.warn("Error occurred while fetching DataRecord for identifier {}", input, > exeption); > {noformat} > The intention probably is that the third parameter is treated as an exception > and e.g. logged with the stack trace. However, this method signature > interprets the exception as a second argument for the message format. This > means the exception is effectively ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-5506) reject item names with unpaired surrogates early
[ https://issues.apache.org/jira/browse/OAK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346786#comment-16346786 ] Marcel Reutegger commented on OAK-5506: --- The result doesn't look that useful. The measured time is zero up to the 50 percentile. > reject item names with unpaired surrogates early > > > Key: OAK-5506 > URL: https://issues.apache.org/jira/browse/OAK-5506 > Project: Jackrabbit Oak > Issue Type: Wish > Components: core, jcr, segment-tar >Affects Versions: 1.5.18 >Reporter: Julian Reschke >Assignee: Francesco Mari >Priority: Minor > Fix For: 1.10 > > Attachments: OAK-5506-01.patch, OAK-5506-02.patch, OAK-5506-4.diff, > OAK-5506-bench.diff, OAK-5506-name-conversion.diff, OAK-5506-segment.diff, > OAK-5506-segment2.diff, OAK-5506.diff, ValidNamesTest.java > > > Apparently, the following node name is accepted: >{{"foo\ud800"}} > but a subsequent {{getPath()}} call fails: > {noformat} > javax.jcr.InvalidItemStateException: This item [/test_node/foo?] does not > exist anymore > at > org.apache.jackrabbit.oak.jcr.delegate.ItemDelegate.checkAlive(ItemDelegate.java:86) > at > org.apache.jackrabbit.oak.jcr.session.operation.ItemOperation.checkPreconditions(ItemOperation.java:34) > at > org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.prePerform(SessionDelegate.java:615) > at > org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.perform(SessionDelegate.java:205) > at > org.apache.jackrabbit.oak.jcr.session.ItemImpl.perform(ItemImpl.java:112) > at > org.apache.jackrabbit.oak.jcr.session.ItemImpl.getPath(ItemImpl.java:140) > at > org.apache.jackrabbit.oak.jcr.session.NodeImpl.getPath(NodeImpl.java:106) > at > org.apache.jackrabbit.oak.jcr.ValidNamesTest.nameTest(ValidNamesTest.java:271) > at > org.apache.jackrabbit.oak.jcr.ValidNamesTest.testUnpairedSurrogate(ValidNamesTest.java:259) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source){noformat} > (test case follows) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7226) Warn messages ignoring exception parameter
[ https://issues.apache.org/jira/browse/OAK-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346776#comment-16346776 ] Chetan Mehrotra commented on OAK-7226: -- [~mreutegg] This is a "feature" of slf4j https://www.slf4j.org/faq.html#paramException > Warn messages ignoring exception parameter > -- > > Key: OAK-7226 > URL: https://issues.apache.org/jira/browse/OAK-7226 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.0 >Reporter: Marcel Reutegger >Priority: Minor > Fix For: 1.10 > > > There are a few usages of Logger.warn() with a pattern similar to this > example: > {noformat} > log.warn("Error occurred while fetching DataRecord for identifier {}", input, > exeption); > {noformat} > The intention probably is that the third parameter is treated as an exception > and e.g. logged with the stack trace. However, this method signature > interprets the exception as a second argument for the message format. This > means the exception is effectively ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (OAK-7221) Build Jackrabbit Oak #1212 failed
[ https://issues.apache.org/jira/browse/OAK-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger closed OAK-7221. - > Build Jackrabbit Oak #1212 failed > - > > Key: OAK-7221 > URL: https://issues.apache.org/jira/browse/OAK-7221 > Project: Jackrabbit Oak > Issue Type: Bug > Components: continuous integration >Reporter: Hudson >Priority: Major > > No description is provided > The build Jackrabbit Oak #1212 has failed. > First failed run: [Jackrabbit Oak > #1212|https://builds.apache.org/job/Jackrabbit%20Oak/1212/] [console > log|https://builds.apache.org/job/Jackrabbit%20Oak/1212/console] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OAK-7221) Build Jackrabbit Oak #1212 failed
[ https://issues.apache.org/jira/browse/OAK-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger resolved OAK-7221. --- Resolution: Duplicate The log doesn't contain any failed tests, but judging from the build time, the job was likely reaching the 90 minutes limit again. > Build Jackrabbit Oak #1212 failed > - > > Key: OAK-7221 > URL: https://issues.apache.org/jira/browse/OAK-7221 > Project: Jackrabbit Oak > Issue Type: Bug > Components: continuous integration >Reporter: Hudson >Priority: Major > > No description is provided > The build Jackrabbit Oak #1212 has failed. > First failed run: [Jackrabbit Oak > #1212|https://builds.apache.org/job/Jackrabbit%20Oak/1212/] [console > log|https://builds.apache.org/job/Jackrabbit%20Oak/1212/console] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7223) Files could be kept partially in case of disconnection from backends
[ https://issues.apache.org/jira/browse/OAK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346754#comment-16346754 ] Marcel Reutegger commented on OAK-7223: --- See OAK-7226 for the warn messages. > Files could be kept partially in case of disconnection from backends > > > Key: OAK-7223 > URL: https://issues.apache.org/jira/browse/OAK-7223 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.1 >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.10, 1.8.2 > > Attachments: OAK-7223-2.patch, OAK-7223-3.patch, OAK-7223.patch > > > The FileCache#load method needs to clean the files which may have been > downloaded partially in case of errors from backend. This partially > downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7226) Warn messages ignoring exception parameter
[ https://issues.apache.org/jira/browse/OAK-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger updated OAK-7226: -- Affects Version/s: 1.8.0 > Warn messages ignoring exception parameter > -- > > Key: OAK-7226 > URL: https://issues.apache.org/jira/browse/OAK-7226 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.0 >Reporter: Marcel Reutegger >Priority: Minor > Fix For: 1.10 > > > There are a few usages of Logger.warn() with a pattern similar to this > example: > {noformat} > log.warn("Error occurred while fetching DataRecord for identifier {}", input, > exeption); > {noformat} > The intention probably is that the third parameter is treated as an exception > and e.g. logged with the stack trace. However, this method signature > interprets the exception as a second argument for the message format. This > means the exception is effectively ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OAK-7226) Warn messages ignoring exception parameter
Marcel Reutegger created OAK-7226: - Summary: Warn messages ignoring exception parameter Key: OAK-7226 URL: https://issues.apache.org/jira/browse/OAK-7226 Project: Jackrabbit Oak Issue Type: Bug Components: blob-plugins Reporter: Marcel Reutegger Fix For: 1.10 There are a few usages of Logger.warn() with a pattern similar to this example: {noformat} log.warn("Error occurred while fetching DataRecord for identifier {}", input, exeption); {noformat} The intention probably is that the third parameter is treated as an exception and e.g. logged with the stack trace. However, this method signature interprets the exception as a second argument for the message format. This means the exception is effectively ignored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7221) Build Jackrabbit Oak #1212 failed
[ https://issues.apache.org/jira/browse/OAK-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346749#comment-16346749 ] Hudson commented on OAK-7221: - Previously failing build now is OK. Passed run: [Jackrabbit Oak #1214|https://builds.apache.org/job/Jackrabbit%20Oak/1214/] [console log|https://builds.apache.org/job/Jackrabbit%20Oak/1214/console] > Build Jackrabbit Oak #1212 failed > - > > Key: OAK-7221 > URL: https://issues.apache.org/jira/browse/OAK-7221 > Project: Jackrabbit Oak > Issue Type: Bug > Components: continuous integration >Reporter: Hudson >Priority: Major > > No description is provided > The build Jackrabbit Oak #1212 has failed. > First failed run: [Jackrabbit Oak > #1212|https://builds.apache.org/job/Jackrabbit%20Oak/1212/] [console > log|https://builds.apache.org/job/Jackrabbit%20Oak/1212/console] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7223) Files could be kept partially in case of disconnection from backends
[ https://issues.apache.org/jira/browse/OAK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346743#comment-16346743 ] Marcel Reutegger commented on OAK-7223: --- The warn message you added in the cache loader will not contain the exception or stack trace. The third parameter is interpreted as the second argument for the message format. There are a few more places in oak-blob-plugin with the same usage pattern. I'll create a separate issue for those. > Files could be kept partially in case of disconnection from backends > > > Key: OAK-7223 > URL: https://issues.apache.org/jira/browse/OAK-7223 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.1 >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.10, 1.8.2 > > Attachments: OAK-7223-2.patch, OAK-7223-3.patch, OAK-7223.patch > > > The FileCache#load method needs to clean the files which may have been > downloaded partially in case of errors from backend. This partially > downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7223) Files could be kept partially in case of disconnection from backends
[ https://issues.apache.org/jira/browse/OAK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346741#comment-16346741 ] Amit Jain commented on OAK-7223: Thanks [~mreutegg] for the review. Yes makes sense and I am in the process of adding test for the utility method. > Files could be kept partially in case of disconnection from backends > > > Key: OAK-7223 > URL: https://issues.apache.org/jira/browse/OAK-7223 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.1 >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.10, 1.8.2 > > Attachments: OAK-7223-2.patch, OAK-7223-3.patch, OAK-7223.patch > > > The FileCache#load method needs to clean the files which may have been > downloaded partially in case of errors from backend. This partially > downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OAK-7223) Files could be kept partially in case of disconnection from backends
[ https://issues.apache.org/jira/browse/OAK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346726#comment-16346726 ] Marcel Reutegger edited comment on OAK-7223 at 1/31/18 12:24 PM: - I would implement the new utility method slightly different. This avoids the issue with catching all types of exceptions but then wrapping it with an IOException (even if the original one was, as in most cases, an IOException). See attached [^OAK-7223-3.patch]. It would also be good to have a test case in oak-commons that verifies the expected behaviour. was (Author: mreutegg): I would implement the new utility method slightly different. This avoids the issue with catching all types of exceptions but then wrapping it with an IOException (even if the original one was, as in most cases, an IOException). See attached [^OAK-7223-3.patch]. It would also be good to have a test case that verifies the expected behaviour. > Files could be kept partially in case of disconnection from backends > > > Key: OAK-7223 > URL: https://issues.apache.org/jira/browse/OAK-7223 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.1 >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.10, 1.8.2 > > Attachments: OAK-7223-2.patch, OAK-7223-3.patch, OAK-7223.patch > > > The FileCache#load method needs to clean the files which may have been > downloaded partially in case of errors from backend. This partially > downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7223) Files could be kept partially in case of disconnection from backends
[ https://issues.apache.org/jira/browse/OAK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346726#comment-16346726 ] Marcel Reutegger commented on OAK-7223: --- I would implement the new utility method slightly different. This avoids the issue with catching all types of exceptions but then wrapping it with an IOException (even if the original one was, as in most cases, an IOException). See attached [^OAK-7223-3.patch]. It would also be good to have a test case that verifies the expected behaviour. > Files could be kept partially in case of disconnection from backends > > > Key: OAK-7223 > URL: https://issues.apache.org/jira/browse/OAK-7223 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.1 >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.10, 1.8.2 > > Attachments: OAK-7223-2.patch, OAK-7223-3.patch, OAK-7223.patch > > > The FileCache#load method needs to clean the files which may have been > downloaded partially in case of errors from backend. This partially > downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7223) Files could be kept partially in case of disconnection from backends
[ https://issues.apache.org/jira/browse/OAK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcel Reutegger updated OAK-7223: -- Attachment: OAK-7223-3.patch > Files could be kept partially in case of disconnection from backends > > > Key: OAK-7223 > URL: https://issues.apache.org/jira/browse/OAK-7223 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.1 >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.10, 1.8.2 > > Attachments: OAK-7223-2.patch, OAK-7223-3.patch, OAK-7223.patch > > > The FileCache#load method needs to clean the files which may have been > downloaded partially in case of errors from backend. This partially > downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-6922) Azure support for the segment-tar
[ https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomek Rękawek updated OAK-6922: --- Description: An Azure Blob Storage implementation of the segment storage, based on the OAK-6921 work. h3. Segment files layout Thew new implementation doesn't use tar files. They are replaced with directories, storing segments, named after their UUIDs. This approach has following advantages: * no need to call seek(), which may be expensive on a remote file system. Rather than that we can read the whole file (=segment) at once. * it's possible to send multiple segments at once, asynchronously, which reduces the performance overhead (see below). The file structure is as follows: {noformat} [~]$ az storage blob list -c oak --output table Name Blob TypeBlob TierLengthContent Type Last Modified --- --- - oak/data0a.tar/.ca1326d1-edf4-4d53-aef0-0f14a6d05b63 BlockBlob 192 application/octet-stream 2018-01-31T10:59:14+00:00 oak/data0a.tar/0001.c6e03426-db9d-4315-a20a-12559e6aee54 BlockBlob 262144application/octet-stream 2018-01-31T10:59:14+00:00 oak/data0a.tar/0002.b3784e27-6d16-4f80-afc1-6f3703f6bdb9 BlockBlob 262144application/octet-stream 2018-01-31T10:59:14+00:00 oak/data0a.tar/0003.5d2f9588-0c92-4547-abf7-0263ee7c37bb BlockBlob 259216application/octet-stream 2018-01-31T10:59:14+00:00 ... oak/data0a.tar/006e.7b8cf63d-849a-4120-aa7c-47c3dde25e48 BlockBlob 4368 application/octet-stream 2018-01-31T12:01:09+00:00 oak/data0a.tar/006f.93799ae9-288e-4b32-afc2-bbc676fad7e5 BlockBlob 3792 application/octet-stream 2018-01-31T12:01:14+00:00 oak/data0a.tar/0070.8b2d5ff2-6a74-4ac3-a3cc-cc439367c2aa BlockBlob 3680 application/octet-stream 2018-01-31T12:01:14+00:00 oak/data0a.tar/0071.2a1c49f0-ce33-4777-a042-8aa8a704d202 BlockBlob 7760 application/octet-stream 2018-01-31T12:10:54+00:00 oak/journal.log.001 AppendBlob 1010 application/octet-stream 2018-01-31T12:10:54+00:00 oak/manifest BlockBlob 46application/octet-stream 2018-01-31T10:59:14+00:00 oak/repo.lock BlockBlob application/octet-stream 2018-01-31T10:59:14+00:00 {noformat} For the segment files, each name is prefixed with the index number. This allows to maintain an order, as in the tar archive. This order is normally stored in the index files as well, but if it's missing, the recovery process needs it. Each file contains the raw segment data, with no padding/headers. Apart from the segment files, there are 3 special files: binary references (.brf), segment graph (.gph) and segment index (.idx). h3. Asynchronous writes Normally, all the TarWriter writes are synchronous, appending the segments to the tar file. In case of Azure Blob Storage each write involves a network latency. That's why the SegmentWriteQueue was introduced. The segments are added to the blocking dequeue, which is served by a number of the consumer threads, writing the segments to the cloud. There's also a map UUID->Segment, which allows to return the segments in case they are requested by the readSegment() method before they are actually persisted. Segments are removed from the map only after a successful write operation. The flush() method blocks accepting the new segments and returns after all waiting segments are written. The close() method waits until the current operations are finished and stops all threads. The asynchronous mode can be disabled by setting the number of threads to 0. h5. Queue recovery mode If the Azure Blob Storage write() operation fails, the segment will be re-added and the queue is switched to an "recovery mode". In this mode, all the threads are suspended and new segments are not accepted (active waiting). There's a single thread which retries adding the segment with some delay. If the segment is successfully written, the queue will back to the normal operation. This way the unavailable remote service is not flooded by the requests and we're not accepting the segments when we can't persist them. The close() method finishes the recovery mode - in this case, some of the awaiting segments won't be persisted. h5. Consistency The asynchronous mode isn't as reliable as the standard, synchronous case. Following cases are possible: * TarWriter#writeEntry() returns successfully, but the segments ar
[jira] [Commented] (OAK-7182) Make it possible to update Guava
[ https://issues.apache.org/jira/browse/OAK-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346685#comment-16346685 ] Robert Munteanu commented on OAK-7182: -- Here's the list of APIs I (semi-manually) extracted from oak as of r1822291 {noformat} org.apache.jackrabbit.oak.plugins.blob;uses:="com.google.common.base,com.google.common.cache,com.google.common.util.concrrent org.apache.jackrabbit.oak.commons.io;version="1.0.0";uses:="com.google.common.io", org.apache.jackrabbit.oak.commons;version="1.0.0";uses:="com.google.common.base,com.google.common.collect, org.apache.jackrabbit.oak.spi.whiteboard;version="1.0.0";uses:="com.google.common.base, org.apache.jackrabbit.oak.cache;version="1.0.0";uses:="com.google.common.cache,com.google.common.collect, org.apache.jackrabbit.oak.plugins.index.property.strategy;uses:="com.google.common.base org.apache.jackrabbit.oak.plugins.nodetype;uses:="com.google.common.base, org.apache.jackrabbit.oak.plugins.observation.filter;uses:="com.google.common.base org.apache.jackrabbit.oak.spi.security.authentication.credentials;version="2.1.0";uses:="com.google.common.collect org.apache.jackrabbit.oak.spi.state;uses:="com.google.common.base, org.apache.jackrabbit.oak.plugins.memory;uses:="com.google.common.hash, {noformat} > Make it possible to update Guava > > > Key: OAK-7182 > URL: https://issues.apache.org/jira/browse/OAK-7182 > Project: Jackrabbit Oak > Issue Type: Wish >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Attachments: guava.diff > > > We currently rely on Guava 15, and this affects all users of Oak because they > essentially need to use the same version. > This is an overall issue to investigate what would need to be done in Oak in > order to make updates possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7182) Make it possible to update Guava
[ https://issues.apache.org/jira/browse/OAK-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346672#comment-16346672 ] Robert Munteanu commented on OAK-7182: -- I am looking into the Guava packaging and release policy, and it seems that they have started following semantic versioning - https://github.com/google/guava/wiki/ReleasePolicy . In particular, they will only upgrade the major version " If there are API removals or other incompatible API changes", and that includes APIs annotated with {{\@Beta}} . Which in turn for us this means that once we incorporate a new version of Guava we should be safe for a longer time. They do have breaking changes, for instance this year they had 3 major releases ( and 6 minor ones ) but version 23 is now 23.6, which may mark a turn in how they approach backwards compatibility. What we can do is either "vet" certain versions, e.g. test with Guava 15 and latest ( 23.6 ) and adjust the import versions to be [15,24). Or alternatively hope for the test and import [15,). > Make it possible to update Guava > > > Key: OAK-7182 > URL: https://issues.apache.org/jira/browse/OAK-7182 > Project: Jackrabbit Oak > Issue Type: Wish >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Attachments: guava.diff > > > We currently rely on Guava 15, and this affects all users of Oak because they > essentially need to use the same version. > This is an overall issue to investigate what would need to be done in Oak in > order to make updates possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7223) Files could be kept partially in case of disconnection from backends
[ https://issues.apache.org/jira/browse/OAK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346656#comment-16346656 ] Amit Jain commented on OAK-7223: Added a 2nd version as needed to bump the package version also. > Files could be kept partially in case of disconnection from backends > > > Key: OAK-7223 > URL: https://issues.apache.org/jira/browse/OAK-7223 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.1 >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.10, 1.8.2 > > Attachments: OAK-7223-2.patch, OAK-7223.patch > > > The FileCache#load method needs to clean the files which may have been > downloaded partially in case of errors from backend. This partially > downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7223) Files could be kept partially in case of disconnection from backends
[ https://issues.apache.org/jira/browse/OAK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Jain updated OAK-7223: --- Attachment: OAK-7223-2.patch > Files could be kept partially in case of disconnection from backends > > > Key: OAK-7223 > URL: https://issues.apache.org/jira/browse/OAK-7223 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.1 >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.10, 1.8.2 > > Attachments: OAK-7223-2.patch, OAK-7223.patch > > > The FileCache#load method needs to clean the files which may have been > downloaded partially in case of errors from backend. This partially > downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-6373) oak-run check should also check checkpoints
[ https://issues.apache.org/jira/browse/OAK-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346637#comment-16346637 ] Michael Dürig commented on OAK-6373: Looking at the usage output of {{check}}: {noformat} --checkpoints [String] checks only specified checkpoints (comma separated); leave empty to check all (default: /checkpoints){noformat} I think we need to clarify this a bit: 1. not specifying anything to {{--checkpoints}} does not work. 2. The meaning of the default {{/checkpoints}} is not clear. I would prefer to change {{/checkpoint}} to {{all}} and change the wording to something that clarifies what {{all}} means. > oak-run check should also check checkpoints > > > Key: OAK-6373 > URL: https://issues.apache.org/jira/browse/OAK-6373 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run, segment-tar >Reporter: Michael Dürig >Assignee: Andrei Dulceanu >Priority: Major > Labels: candidate_oak_1_8, tooling > Fix For: 1.9.0, 1.10 > > > {{oak-run check}} does currently *not* traverse and check the items in the > checkpoint. I think we should change this and add an option to traverse all, > some or none of the checkpoints. When doing this we need to keep in mind the > interaction of this new feature with the {{filter}} option: the paths passed > through this option need then be prefixed with {{/root}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-5506) reject item names with unpaired surrogates early
[ https://issues.apache.org/jira/browse/OAK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346638#comment-16346638 ] Julian Reschke commented on OAK-5506: - results from {{java -jar target/oak-benchmarks-1.10-SNAPSHOT.jar benchmark Oak-Segment-Tar StringWriteTest}}: new: {noformat} Apache Jackrabbit Oak 1.10-SNAPSHOT # StringWriteTest C min 10% 50% 90% max N Oak-Segment-Tar1 0 0 0 16 33 16726 Oak-Segment-Tar1 0 0 0 16 47 17000 Oak-Segment-Tar1 0 0 0 16 47 17739 Oak-Segment-Tar1 0 0 0 15 46 17635 {noformat} old: {noformat} Apache Jackrabbit Oak 1.10-SNAPSHOT # StringWriteTest C min 10% 50% 90% max N Oak-Segment-Tar1 0 0 0 16 35 16790 Oak-Segment-Tar1 0 0 0 16 47 17092 Oak-Segment-Tar1 0 0 0 16 46 16937 Oak-Segment-Tar1 0 0 0 16 47 16851 {noformat} > reject item names with unpaired surrogates early > > > Key: OAK-5506 > URL: https://issues.apache.org/jira/browse/OAK-5506 > Project: Jackrabbit Oak > Issue Type: Wish > Components: core, jcr, segment-tar >Affects Versions: 1.5.18 >Reporter: Julian Reschke >Assignee: Francesco Mari >Priority: Minor > Fix For: 1.10 > > Attachments: OAK-5506-01.patch, OAK-5506-02.patch, OAK-5506-4.diff, > OAK-5506-bench.diff, OAK-5506-name-conversion.diff, OAK-5506-segment.diff, > OAK-5506-segment2.diff, OAK-5506.diff, ValidNamesTest.java > > > Apparently, the following node name is accepted: >{{"foo\ud800"}} > but a subsequent {{getPath()}} call fails: > {noformat} > javax.jcr.InvalidItemStateException: This item [/test_node/foo?] does not > exist anymore > at > org.apache.jackrabbit.oak.jcr.delegate.ItemDelegate.checkAlive(ItemDelegate.java:86) > at > org.apache.jackrabbit.oak.jcr.session.operation.ItemOperation.checkPreconditions(ItemOperation.java:34) > at > org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.prePerform(SessionDelegate.java:615) > at > org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.perform(SessionDelegate.java:205) > at > org.apache.jackrabbit.oak.jcr.session.ItemImpl.perform(ItemImpl.java:112) > at > org.apache.jackrabbit.oak.jcr.session.ItemImpl.getPath(ItemImpl.java:140) > at > org.apache.jackrabbit.oak.jcr.session.NodeImpl.getPath(NodeImpl.java:106) > at > org.apache.jackrabbit.oak.jcr.ValidNamesTest.nameTest(ValidNamesTest.java:271) > at > org.apache.jackrabbit.oak.jcr.ValidNamesTest.testUnpairedSurrogate(ValidNamesTest.java:259) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source){noformat} > (test case follows) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7203) Make MountInfoProvider service in AuthorizationConfigurationImpl optional
[ https://issues.apache.org/jira/browse/OAK-7203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346629#comment-16346629 ] Alex Deparvu commented on OAK-7203: --- bq. bind/unbind removed additionally could you explain why this is needed? To me the test code now looks pretty ugly: catch an IllegalAccessException to wrap it into a RuntimeException? The current design: * field is set to default so it has a value outside OSGi (tests & other setups), if needed one can change it by calling *bind * field is set via OSGi in which case unbind will also be called so the reference needs to be set to _null_ for consistency, _not the default value_, as we are not in the default case anymore To me this seems to cover both OSGi and non-OSGi cases but the current code has one issue: the reference is mandatory. We can just add the required tag to the reference and be done. Why all the extra changes? > Make MountInfoProvider service in AuthorizationConfigurationImpl optional > - > > Key: OAK-7203 > URL: https://issues.apache.org/jira/browse/OAK-7203 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: authorization-cug, core >Affects Versions: 1.8.1 >Reporter: Oliver Lietz >Priority: Major > Attachments: OAK-7203.patch > > > While testing Sling with Oak 1.8 I've observed that > AuthorizationConfigurationImpl gets not activated due to missing > MountInfoProvider service: > {noformat} > @Reference > private MountInfoProvider mountInfoProvider = > Mounts.defaultMountInfoProvider(); > {noformat} > {noformat} > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > Bundleorg.apache.jackrabbit.oak-core (63) > Implementation Class > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > Default State enabled > Activationdelayed > Configuration Policy optional > Service Type singleton > Services > org.apache.jackrabbit.oak.spi.security.authorization.AuthorizationConfiguration > org.apache.jackrabbit.oak.spi.security.SecurityConfiguration > PID > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > Reference mountInfoProvider Unsatisfied > Service Name: org.apache.jackrabbit.oak.spi.mount.MountInfoProvider > Cardinality: 1..1 > Policy: static > Policy Option: reluctant > No Services bound > Propertiescomponent.id = 35 > component.name = > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > configurationRanking = 100 > importBehavior = abort > oak.security.name = > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > readPaths = [/jcr:system/rep:namespaces, /jcr:system/jcr:nodeTypes, > /jcr:system/rep:privileges] > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7203) Make MountInfoProvider service in AuthorizationConfigurationImpl optional
[ https://issues.apache.org/jira/browse/OAK-7203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346625#comment-16346625 ] Robert Munteanu commented on OAK-7203: -- Thinking about this a bit more, I think the current setup is correct. We should not allow a 'default' service to be registered or default values to be used when it's possible that the value will change. The components must only use the 'right' MountInfoProvider from the beginning, otherwise we open the door to inconsistent behaviour. The only thing that is sub-optimal here is that the {{oak-store-composite}} bundle is now required. I don't think that's a huge issue, but we can work around it moving the {{MountInfoProviderService}} class from {{oak-store-composite}} to {{oak-core}}, so that the {{MountInfoProvider}} service can be registered with an empty config without adding an extra bundle. > Make MountInfoProvider service in AuthorizationConfigurationImpl optional > - > > Key: OAK-7203 > URL: https://issues.apache.org/jira/browse/OAK-7203 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: authorization-cug, core >Affects Versions: 1.8.1 >Reporter: Oliver Lietz >Priority: Major > Attachments: OAK-7203.patch > > > While testing Sling with Oak 1.8 I've observed that > AuthorizationConfigurationImpl gets not activated due to missing > MountInfoProvider service: > {noformat} > @Reference > private MountInfoProvider mountInfoProvider = > Mounts.defaultMountInfoProvider(); > {noformat} > {noformat} > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > Bundleorg.apache.jackrabbit.oak-core (63) > Implementation Class > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > Default State enabled > Activationdelayed > Configuration Policy optional > Service Type singleton > Services > org.apache.jackrabbit.oak.spi.security.authorization.AuthorizationConfiguration > org.apache.jackrabbit.oak.spi.security.SecurityConfiguration > PID > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > Reference mountInfoProvider Unsatisfied > Service Name: org.apache.jackrabbit.oak.spi.mount.MountInfoProvider > Cardinality: 1..1 > Policy: static > Policy Option: reluctant > No Services bound > Propertiescomponent.id = 35 > component.name = > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > configurationRanking = 100 > importBehavior = abort > oak.security.name = > org.apache.jackrabbit.oak.security.authorization.AuthorizationConfigurationImpl > readPaths = [/jcr:system/rep:namespaces, /jcr:system/jcr:nodeTypes, > /jcr:system/rep:privileges] > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7182) Make it possible to update Guava
[ https://issues.apache.org/jira/browse/OAK-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346628#comment-16346628 ] Davide Giannella commented on OAK-7182: --- I've added a subtask about the AtomicCounter aspect and guava. While it's not strictly an API it's used during repository construction and I'm in favour to try to remove the guava dependency completely in such case. That does not exclude that Guava is not used in other places of the functionality. > Make it possible to update Guava > > > Key: OAK-7182 > URL: https://issues.apache.org/jira/browse/OAK-7182 > Project: Jackrabbit Oak > Issue Type: Wish >Reporter: Julian Reschke >Assignee: Julian Reschke >Priority: Minor > Attachments: guava.diff > > > We currently rely on Guava 15, and this affects all users of Oak because they > essentially need to use the same version. > This is an overall issue to investigate what would need to be done in Oak in > order to make updates possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OAK-7225) Replace AtomicCounter Supplier
Davide Giannella created OAK-7225: - Summary: Replace AtomicCounter Supplier Key: OAK-7225 URL: https://issues.apache.org/jira/browse/OAK-7225 Project: Jackrabbit Oak Issue Type: Sub-task Components: core Affects Versions: 1.6.0, 1.4.0 Reporter: Davide Giannella Assignee: Davide Giannella In the [AtomicCounter|https://github.com/apache/jackrabbit-oak/blob/7a7aa1e5d4f53f5bfb410f58264c237b288f5c74/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/atomic/AtomicCounterEditorProvider.java#L121] we use guava's Supplier which should be trivially replaced by the JDK8 [java.util.function.Supplier|https://docs.oracle.com/javase/8/docs/api/java/util/function/Supplier.html]. In case of backports to Oak 1.4, and therefore Java7 it should be possible to workaround the FunctionalInterface with a utility class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-6373) oak-run check should also check checkpoints
[ https://issues.apache.org/jira/browse/OAK-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346620#comment-16346620 ] Andrei Dulceanu commented on OAK-6373: -- [~frm], thanks for reviewing! Fixed in trunk at r1822791. > oak-run check should also check checkpoints > > > Key: OAK-6373 > URL: https://issues.apache.org/jira/browse/OAK-6373 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run, segment-tar >Reporter: Michael Dürig >Assignee: Andrei Dulceanu >Priority: Major > Labels: candidate_oak_1_8, tooling > Fix For: 1.9.0, 1.10 > > > {{oak-run check}} does currently *not* traverse and check the items in the > checkpoint. I think we should change this and add an option to traverse all, > some or none of the checkpoints. When doing this we need to keep in mind the > interaction of this new feature with the {{filter}} option: the paths passed > through this option need then be prefixed with {{/root}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7223) Files could be kept partially in case of disconnection from backends
[ https://issues.apache.org/jira/browse/OAK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346619#comment-16346619 ] Amit Jain commented on OAK-7223: [~mreutegg] Could you please review the patch. > Files could be kept partially in case of disconnection from backends > > > Key: OAK-7223 > URL: https://issues.apache.org/jira/browse/OAK-7223 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.1 >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.10, 1.8.2 > > Attachments: OAK-7223.patch > > > The FileCache#load method needs to clean the files which may have been > downloaded partially in case of errors from backend. This partially > downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7223) Files could be kept partially in case of disconnection from backends
[ https://issues.apache.org/jira/browse/OAK-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Jain updated OAK-7223: --- Attachment: OAK-7223.patch > Files could be kept partially in case of disconnection from backends > > > Key: OAK-7223 > URL: https://issues.apache.org/jira/browse/OAK-7223 > Project: Jackrabbit Oak > Issue Type: Bug > Components: blob-plugins >Affects Versions: 1.8.1 >Reporter: Amit Jain >Assignee: Amit Jain >Priority: Blocker > Fix For: 1.10, 1.8.2 > > Attachments: OAK-7223.patch > > > The FileCache#load method needs to clean the files which may have been > downloaded partially in case of errors from backend. This partially > downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-6373) oak-run check should also check checkpoints
[ https://issues.apache.org/jira/browse/OAK-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346604#comment-16346604 ] Francesco Mari commented on OAK-6373: - [~dulceanu], the latest commit in your branch looks good to me. > oak-run check should also check checkpoints > > > Key: OAK-6373 > URL: https://issues.apache.org/jira/browse/OAK-6373 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run, segment-tar >Reporter: Michael Dürig >Assignee: Andrei Dulceanu >Priority: Major > Labels: candidate_oak_1_8, tooling > Fix For: 1.9.0, 1.10 > > > {{oak-run check}} does currently *not* traverse and check the items in the > checkpoint. I think we should change this and add an option to traverse all, > some or none of the checkpoints. When doing this we need to keep in mind the > interaction of this new feature with the {{filter}} option: the paths passed > through this option need then be prefixed with {{/root}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7224) oak-run check should have an option to check the segments checksums
[ https://issues.apache.org/jira/browse/OAK-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346566#comment-16346566 ] Valentin Olteanu commented on OAK-7224: --- If this is a completely different type of check, e.g. only checksum of segments, I would create another command: {{check-segments}}. But if it's doing both the classical check and the segments checksum, then it's ok to have a parameter... > oak-run check should have an option to check the segments checksums > --- > > Key: OAK-7224 > URL: https://issues.apache.org/jira/browse/OAK-7224 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run, segment-tar >Reporter: Andrei Dulceanu >Assignee: Andrei Dulceanu >Priority: Minor > Labels: tooling > Fix For: 1.9.0, 1.10 > > > {{oak-run check}} does currently *not* check the checksums of the segments. > As a consequence, there is no quick way of determining the state of the > repository (corrupt/valid), after corrupting some random node record, as we > currently do in {{CheckRepositoryTestBase#corruptRecord}}. To determine that, > there needs to be an attempt to read the corrupt record as part of a > traversal. > An easier way would be to have a new dedicated option for this (i.e., > {{--segments}}) which checks by default the content of segments against the > checksums from all the tar files in the specified location. Additionally, it > could accept as an argument a list of tar files, the segments of which to be > checked. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-7224) oak-run check should have an option to check the segments checksums
[ https://issues.apache.org/jira/browse/OAK-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Dulceanu updated OAK-7224: - Description: {{oak-run check}} does currently *not* check the checksums of the segments. As a consequence, there is no quick way of determining the state of the repository (corrupt/valid), after corrupting some random node record, as we currently do in {{CheckRepositoryTestBase#corruptRecord}}. To determine that, there needs to be an attempt to read the corrupt record as part of a traversal. An easier way would be to have a new dedicated option for this (i.e., {{--segments}}) which checks by default the content of segments against the checksums from all the tar files in the specified location. Additionally, it could accept as an argument a list of tar files, the segments of which to be checked. was: {{oak-run check}} does currently *not* check the checksums of the segments. As a consequence, there is no quick way of determining the state of the repository (corrupt/valid), after corrupting some random node record, as we currently do in {{CheckRepositoryTestBase#corruptRecord}}. To determine that, there needs to be an attempt to read the corrupt record as part of a traversal. An easier way would be to have a new dedicated option for this (i.e., {{--segments}}) which checks by default all segments from all the tar files in the specified location. Additionally, it could accept as an argument a list of tar files, the segments of which to be checked. > oak-run check should have an option to check the segments checksums > --- > > Key: OAK-7224 > URL: https://issues.apache.org/jira/browse/OAK-7224 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: run, segment-tar >Reporter: Andrei Dulceanu >Assignee: Andrei Dulceanu >Priority: Minor > Labels: tooling > Fix For: 1.9.0, 1.10 > > > {{oak-run check}} does currently *not* check the checksums of the segments. > As a consequence, there is no quick way of determining the state of the > repository (corrupt/valid), after corrupting some random node record, as we > currently do in {{CheckRepositoryTestBase#corruptRecord}}. To determine that, > there needs to be an attempt to read the corrupt record as part of a > traversal. > An easier way would be to have a new dedicated option for this (i.e., > {{--segments}}) which checks by default the content of segments against the > checksums from all the tar files in the specified location. Additionally, it > could accept as an argument a list of tar files, the segments of which to be > checked. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OAK-7224) oak-run check should have an option to check the segments checksums
Andrei Dulceanu created OAK-7224: Summary: oak-run check should have an option to check the segments checksums Key: OAK-7224 URL: https://issues.apache.org/jira/browse/OAK-7224 Project: Jackrabbit Oak Issue Type: Improvement Components: run, segment-tar Reporter: Andrei Dulceanu Assignee: Andrei Dulceanu Fix For: 1.9.0, 1.10 {{oak-run check}} does currently *not* check the checksums of the segments. As a consequence, there is no quick way of determining the state of the repository (corrupt/valid), after corrupting some random node record, as we currently do in {{CheckRepositoryTestBase#corruptRecord}}. To determine that, there needs to be an attempt to read the corrupt record as part of a traversal. An easier way would be to have a new dedicated option for this (i.e., {{--segments}}) which checks by default all segments from all the tar files in the specified location. Additionally, it could accept as an argument a list of tar files, the segments of which to be checked. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OAK-7162) Race condition on revisions head between compaction and scheduler could result in skipped commit
[ https://issues.apache.org/jira/browse/OAK-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346521#comment-16346521 ] Michael Dürig commented on OAK-7162: Reopened since backport to 1.8 is still pending. > Race condition on revisions head between compaction and scheduler could > result in skipped commit > > > Key: OAK-7162 > URL: https://issues.apache.org/jira/browse/OAK-7162 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-tar >Affects Versions: 1.8.0 >Reporter: Andrei Dulceanu >Assignee: Andrei Dulceanu >Priority: Blocker > Labels: scalability > Fix For: 1.9.0, 1.10, 1.8.2 > > Attachments: OAK-7162-02.patch, OAK-7162-03.patch, OAK-7162.patch > > > There is a race condition on {{TarRevisions#head}} between a running > compaction trying to set the new head [0] and the scheduler doing the same > after executing a specific commit [1]. If the compaction thread is first, > then the head assignment in the scheduler will fail and not be re-attempted. > IMO, the simple if statement should be changed to a while loop in which the > head is refreshed and the commit is re-applied against the new head, before > attempting again to set a new head in {{TarRevisions}}. This is somehow > similar to what we previously had [2], but without the unneeded > optimistic/pessimistic strategies involving tokens. > [0] > https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L764 > [1] > https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java#L253 > [2] > https://github.com/apache/jackrabbit-oak/blob/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStore.java#L686 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (OAK-7162) Race condition on revisions head between compaction and scheduler could result in skipped commit
[ https://issues.apache.org/jira/browse/OAK-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davide Giannella reopened OAK-7162: --- > Race condition on revisions head between compaction and scheduler could > result in skipped commit > > > Key: OAK-7162 > URL: https://issues.apache.org/jira/browse/OAK-7162 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-tar >Affects Versions: 1.8.0 >Reporter: Andrei Dulceanu >Assignee: Andrei Dulceanu >Priority: Blocker > Labels: scalability > Fix For: 1.9.0, 1.10, 1.8.2 > > Attachments: OAK-7162-02.patch, OAK-7162-03.patch, OAK-7162.patch > > > There is a race condition on {{TarRevisions#head}} between a running > compaction trying to set the new head [0] and the scheduler doing the same > after executing a specific commit [1]. If the compaction thread is first, > then the head assignment in the scheduler will fail and not be re-attempted. > IMO, the simple if statement should be changed to a while loop in which the > head is refreshed and the commit is re-applied against the new head, before > attempting again to set a new head in {{TarRevisions}}. This is somehow > similar to what we previously had [2], but without the unneeded > optimistic/pessimistic strategies involving tokens. > [0] > https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L764 > [1] > https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/scheduler/LockBasedScheduler.java#L253 > [2] > https://github.com/apache/jackrabbit-oak/blob/1.6/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentNodeStore.java#L686 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OAK-6922) Azure support for the segment-tar
[ https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomek Rękawek updated OAK-6922: --- Description: An Azure Blob Storage implementation of the segment storage, based on the OAK-6921 work. h3. Segment files layout Thew new implementation doesn't use tar files. They are replaced with directories, storing segments, named after their UUIDs. This approach has following advantages: * no need to call seek(), which may be expensive on a remote file system. Rather than that we can read the whole file (=segment) at once. * it's possible to send multiple segments at once, asynchronously, which reduces the performance overhead (see below). The file structure is as follows: {noformat} $ hdfs dfs -ls /oak/data0a.tar Found 517 items -rw-r--r-- 1 rekawek supergroup192 2017-11-08 13:24 /oak/data0a.tar/.b1d032cd-266d-4fd6-acf4-9828f54e2b40 -rw-r--r-- 1 rekawek supergroup 262112 2017-11-08 13:24 /oak/data0a.tar/0001.445ca696-d5d1-4843-a04b-044f84d93663 -rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24 /oak/data0a.tar/0002.91ce6f93-d7ed-4d34-a383-3c3d2eea2acb -rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24 /oak/data0a.tar/0003.43c09e6f-3d62-4747-ac75-de41b850862a (...) -rw-r--r-- 1 rekawek supergroup 191888 2017-11-08 13:32 /oak/data0a.tar/data0a.tar.brf -rw-r--r-- 1 rekawek supergroup 823436 2017-11-08 13:32 /oak/data0a.tar/data0a.tar.gph -rw-r--r-- 1 rekawek supergroup 17408 2017-11-08 13:32 /oak/data0a.tar/data0a.tar.idx {noformat} For the segment files, each name is prefixed with the index number. This allows to maintain an order, as in the tar archive. This order is normally stored in the index files as well, but if it's missing, the recovery process needs it. Each file contains the raw segment data, with no padding/headers. Apart from the segment files, there are 3 special files: binary references (.brf), segment graph (.gph) and segment index (.idx). h3. Asynchronous writes Normally, all the TarWriter writes are synchronous, appending the segments to the tar file. In case of Azure Blob Storage each write involves a network latency. That's why the SegmentWriteQueue was introduced. The segments are added to the blocking dequeue, which is served by a number of the consumer threads, writing the segments to the cloud. There's also a map UUID->Segment, which allows to return the segments in case they are requested by the readSegment() method before they are actually persisted. Segments are removed from the map only after a successful write operation. The flush() method blocks accepting the new segments and returns after all waiting segments are written. The close() method waits until the current operations are finished and stops all threads. The asynchronous mode can be disabled by setting the number of threads to 0. h5. Queue recovery mode If the Azure Blob Storage write() operation fails, the segment will be re-added and the queue is switched to an "recovery mode". In this mode, all the threads are suspended and new segments are not accepted (active waiting). There's a single thread which retries adding the segment with some delay. If the segment is successfully written, the queue will back to the normal operation. This way the unavailable remote service is not flooded by the requests and we're not accepting the segments when we can't persist them. The close() method finishes the recovery mode - in this case, some of the awaiting segments won't be persisted. h5. Consistency The asynchronous mode isn't as reliable as the standard, synchronous case. Following cases are possible: * TarWriter#writeEntry() returns successfully, but the segments are not persisted. * TarWriter#writeEntry() accepts a number of segments: S1, S2, S3. The S2 and S3 are persisted, but the S1 is not. On the other hand: * If the TarWriter#flush() returns successfully, it means that all the accepted segments has been persisted. h5. Recovery During the segment recovery (eg. if the index file is missing), the HDFS implementation checks if there's no missing segment in the middle. If so, only the consecutive segments are recovered. For instance, if we have S1, S2, S3, S5, S6, S7, then the recovery process will return only the first three. was: A HDFS implementation of the segment storage, based on the OAK-6921 work. h3. HDFS HDFS is a distributed, network file system. The most popular implementation is Apache Hadoop, but the HDFS is also available in the Amazon AWS and Microsoft Azure clouds. Despite being a file system, it requires a custom API to be used - it's not similar to NFS or CIFS which can be simply mounted locally. h3. Segment files layout Thew new implementation doesn't use tar files. They are replaced with directories, storing segments, named after their UUIDs.
[jira] [Updated] (OAK-6921) Support pluggable segment storage
[ https://issues.apache.org/jira/browse/OAK-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomek Rękawek updated OAK-6921: --- Description: h3. Rationale segment-tar, as names suggest, stores the segments in a bunch of tar archives, inside the {{segmentstore}} directory on the local file system. For some cases, especially in the cloud deployments, it may be interesting to store the segments outside the local FS - the remote storage such as Amazon S3, Azure Blob Storage or HDFS may be cheaper than a mounted disk, more scalable, easier for the provisioning, etc. h3. Storing segment in tar files !current-state.png! There are 3 classes responsible for handling tar files in the segment-tar: TarFiles, TarWriter and TarReader. The TarFiles manages the {{segmentstore}} directory, scans it for the .tars and for each one creates a TarReader. It also creates a single TarWriter object, used to write (and also read) the most recent tar file. The TarWriter appends segments to the latest tar file and also serializes the auxiliary indexes: segment index, binary references index and the segment graph. It also takes of synchronization, as we're dealing with a mutable data structure - tar file opened in the append mode. The TarReader not only reads the segments from the tar file, but is also responsible for the revision GC (mark & sweep methods) and recovering data from files which hasn't been closed cleanly (eg. have no index). h3. New abstraction layer - SegmentArchiveManager !new-interfaces.png! In order to store segments not in the tar files, but somewhere else, it'd be possible to create own implementation of the TarFiles, TarWriter and TarReader. However, such implementation would duplicate a lot of code, not strictly related to the persistence - mark(), sweep(), synchronization, etc. Rather than that, the attached patch presents a different approach: a new layer of abstraction is injected into TarFiles, TarWriter and TarReader - it only takes care of the segments persistence and knows nothing about the synchronization, GC, etc. - leaving it to the upper layer. The new abstraction layer is modelled using 3 new classes: SegmentArchiveManager, SegmentArchiveReader and SegmentArchiveWriter. They are strictly related to the existing Tar* classes and used by them. SegmentArchiveManager provides a bunch of file system-style methods, like open(), create(), delete(), exists(), etc. The open() and create() returns instances of the SAReader and SAWriter. SegmentArchiveReader, despite from reading segments, can also load and parse the index, graph and binary references. The logic responsible for parsing these structures has been already extracted, so it doesn't need to be duplicated in the SAReader implementations. Also, SAReader needs to be aware about the index, since it contains the segment offsets. The SAWriter class allows to write and read the segments and also store the indexes. It isn't thread safe - it assumes that the synchronization is already done on the higher layers. In the patch, I've moved the tar implementation to the new classes: SegmentTarManager, SegmentTarReader and SegmentTarWriter. h3. Other files Apart from the segments, the {{segmentstore}} directory also contains following files: * repo.lock * journal.log * gc.log * manifest All these files are supported by the new SegmentNodeStorePersistence. Usually there's a simple interface (RepositoryLock, JournalLogFile, etc.) for handling the files. h3. TODO * The names and package locations for all the affected classes are subjects to change - after applying the patch the TarFiles doesn't deal with the .tar files anymore, similarly the TarReader and TarWriter delegates the low-level file access duties to the SegmentArchiveReader and Writer. I didn't want to change the names yet, to make it easier to understand and rebase the patch with the trunk changes. was: h3. Rationale segment-tar, as names suggest, stores the segments in a bunch of tar archives, inside the {{segmentstore}} directory on the local file system. For some cases, especially in the cloud deployments, it may be interesting to store the segments outside the local FS - the remote storage such as Amazon S3, Azure Blob Storage or HDFS may be cheaper than a mounted disk, more scalable, easier for the provisioning, etc. h3. Storing segment in tar files !current-state.png! There are 3 classes responsible for handling tar files in the segment-tar: TarFiles, TarWriter and TarReader. The TarFiles manages the {{segmentstore}} directory, scans it for the .tars and for each one creates a TarReader. It also creates a single TarWriter object, used to write (and also read) the most recent tar file. The TarWriter appends segments to the latest tar file and also serializes the auxiliary indexes: segment index, binary references index and the segment graph. It also takes of
[jira] [Issue Comment Deleted] (OAK-6922) Azure support for the segment-tar
[ https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomek Rękawek updated OAK-6922: --- Comment: was deleted (was: https://github.com/trekawek/jackrabbit-oak/tree/segment-tar/hdfs) > Azure support for the segment-tar > - > > Key: OAK-6922 > URL: https://issues.apache.org/jira/browse/OAK-6922 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Tomek Rękawek >Priority: Major > Fix For: 1.9.0, 1.10 > > Attachments: OAK-6922.patch > > > An Azure Blob Storage implementation of the segment storage, based on the > OAK-6921 work. > h3. Segment files layout > Thew new implementation doesn't use tar files. They are replaced with > directories, storing segments, named after their UUIDs. This approach has > following advantages: > * no need to call seek(), which may be expensive on a remote file system. > Rather than that we can read the whole file (=segment) at once. > * it's possible to send multiple segments at once, asynchronously, which > reduces the performance overhead (see below). > The file structure is as follows: > {noformat} > $ hdfs dfs -ls /oak/data0a.tar > Found 517 items > -rw-r--r-- 1 rekawek supergroup192 2017-11-08 13:24 > /oak/data0a.tar/.b1d032cd-266d-4fd6-acf4-9828f54e2b40 > -rw-r--r-- 1 rekawek supergroup 262112 2017-11-08 13:24 > /oak/data0a.tar/0001.445ca696-d5d1-4843-a04b-044f84d93663 > -rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24 > /oak/data0a.tar/0002.91ce6f93-d7ed-4d34-a383-3c3d2eea2acb > -rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24 > /oak/data0a.tar/0003.43c09e6f-3d62-4747-ac75-de41b850862a > (...) > -rw-r--r-- 1 rekawek supergroup 191888 2017-11-08 13:32 > /oak/data0a.tar/data0a.tar.brf > -rw-r--r-- 1 rekawek supergroup 823436 2017-11-08 13:32 > /oak/data0a.tar/data0a.tar.gph > -rw-r--r-- 1 rekawek supergroup 17408 2017-11-08 13:32 > /oak/data0a.tar/data0a.tar.idx > {noformat} > For the segment files, each name is prefixed with the index number. This > allows to maintain an order, as in the tar archive. This order is normally > stored in the index files as well, but if it's missing, the recovery process > needs it. > Each file contains the raw segment data, with no padding/headers. Apart from > the segment files, there are 3 special files: binary references (.brf), > segment graph (.gph) and segment index (.idx). > h3. Asynchronous writes > Normally, all the TarWriter writes are synchronous, appending the segments to > the tar file. In case of Azure Blob Storage each write involves a network > latency. That's why the SegmentWriteQueue was introduced. The segments are > added to the blocking dequeue, which is served by a number of the consumer > threads, writing the segments to the cloud. There's also a map UUID->Segment, > which allows to return the segments in case they are requested by the > readSegment() method before they are actually persisted. Segments are removed > from the map only after a successful write operation. > The flush() method blocks accepting the new segments and returns after all > waiting segments are written. The close() method waits until the current > operations are finished and stops all threads. > The asynchronous mode can be disabled by setting the number of threads to 0. > h5. Queue recovery mode > If the Azure Blob Storage write() operation fails, the segment will be > re-added and the queue is switched to an "recovery mode". In this mode, all > the threads are suspended and new segments are not accepted (active waiting). > There's a single thread which retries adding the segment with some delay. If > the segment is successfully written, the queue will back to the normal > operation. > This way the unavailable remote service is not flooded by the requests and > we're not accepting the segments when we can't persist them. > The close() method finishes the recovery mode - in this case, some of the > awaiting segments won't be persisted. > h5. Consistency > The asynchronous mode isn't as reliable as the standard, synchronous case. > Following cases are possible: > * TarWriter#writeEntry() returns successfully, but the segments are not > persisted. > * TarWriter#writeEntry() accepts a number of segments: S1, S2, S3. The S2 and > S3 are persisted, but the S1 is not. > On the other hand: > * If the TarWriter#flush() returns successfully, it means that all the > accepted segments has been persisted. > h5. Recovery > During the segment recovery (eg. if the index file is missing), the HDFS > implementation checks if there's no missing segment in the middle. If so, > only the consecutive segments are recovered. For instance, if we have S
[jira] [Updated] (OAK-6922) Azure support for the segment-tar
[ https://issues.apache.org/jira/browse/OAK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomek Rękawek updated OAK-6922: --- Summary: Azure support for the segment-tar (was: HDFS support for the segment-tar) > Azure support for the segment-tar > - > > Key: OAK-6922 > URL: https://issues.apache.org/jira/browse/OAK-6922 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-tar >Reporter: Tomek Rękawek >Priority: Major > Fix For: 1.9.0, 1.10 > > Attachments: OAK-6922.patch > > > A HDFS implementation of the segment storage, based on the OAK-6921 work. > h3. HDFS > HDFS is a distributed, network file system. The most popular implementation > is Apache Hadoop, but the HDFS is also available in the Amazon AWS and > Microsoft Azure clouds. Despite being a file system, it requires a custom API > to be used - it's not similar to NFS or CIFS which can be simply mounted > locally. > h3. Segment files layout > Thew new implementation doesn't use tar files. They are replaced with > directories, storing segments, named after their UUIDs. This approach has > following advantages: > * no need to call seek(), which may be expensive on a remote file system. > Rather than that we can read the whole file (=segment) at once. > * it's possible to send multiple segments at once, asynchronously, which > reduces the performance overhead (see below). > The file structure is as follows: > {noformat} > $ hdfs dfs -ls /oak/data0a.tar > Found 517 items > -rw-r--r-- 1 rekawek supergroup192 2017-11-08 13:24 > /oak/data0a.tar/.b1d032cd-266d-4fd6-acf4-9828f54e2b40 > -rw-r--r-- 1 rekawek supergroup 262112 2017-11-08 13:24 > /oak/data0a.tar/0001.445ca696-d5d1-4843-a04b-044f84d93663 > -rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24 > /oak/data0a.tar/0002.91ce6f93-d7ed-4d34-a383-3c3d2eea2acb > -rw-r--r-- 1 rekawek supergroup 262144 2017-11-08 13:24 > /oak/data0a.tar/0003.43c09e6f-3d62-4747-ac75-de41b850862a > (...) > -rw-r--r-- 1 rekawek supergroup 191888 2017-11-08 13:32 > /oak/data0a.tar/data0a.tar.brf > -rw-r--r-- 1 rekawek supergroup 823436 2017-11-08 13:32 > /oak/data0a.tar/data0a.tar.gph > -rw-r--r-- 1 rekawek supergroup 17408 2017-11-08 13:32 > /oak/data0a.tar/data0a.tar.idx > {noformat} > For the segment files, each name is prefixed with the index number. This > allows to maintain an order, as in the tar archive. This order is normally > stored in the index files as well, but if it's missing, the recovery process > needs it. > Each file contains the raw segment data, with no padding/headers. Apart from > the segment files, there are 3 special files: binary references (.brf), > segment graph (.gph) and segment index (.idx). > h3. Asynchronous writes > Normally, all the TarWriter writes are synchronous, appending the segments to > the tar file. In case of HDFS each write involves a network latency. That's > why the SegmentWriteQueue was introduced. The segments are added to the > blocking deque, which is served by a number of the consumer threads, writing > the segments to HDFS. There's also a map UUID->Segment, which allows to > return the segments in case they are requested by the readSegment() method > before they are actually persisted. Segments are removed from the map only > after a successful write operation. > The flush() method blocks accepting the new segments and returns after all > waiting segments are written. The close() method waits until the current > operations are finished and stops all threads. > The asynchronous mode can be disabled by setting the number of threads to 0. > h5. Queue recovery mode > If the HDFS write() operation fails, the segment will be re-added and the > queue is switched to an "recovery mode". In this mode, all the threads are > suspended and new segments are not accepted (active waiting). There's a > single thread which retries adding the segment with some delay. If the > segment is successfully written, the queue will back to the normal operation. > This way the unavailable HDFS service is not flooded by the requests and > we're not accepting the segments when we can't persist them. > The close() method finishes the recovery mode - in this case, some of the > awaiting segments won't be persisted. > h5. Consistency > The asynchronous mode isn't as reliable as the standard, synchronous case. > Following cases are possible: > * TarWriter#writeEntry() returns successfully, but the segments are not > persisted. > * TarWriter#writeEntry() accepts a number of segments: S1, S2, S3. The S2 and > S3 are persisted, but the S1 is not. > On the other hand: > * If the TarWriter#flush() returns successfully, it means that all the > accepted segme
[jira] [Created] (OAK-7223) Files could be kept partially in case of disconnection from backends
Amit Jain created OAK-7223: -- Summary: Files could be kept partially in case of disconnection from backends Key: OAK-7223 URL: https://issues.apache.org/jira/browse/OAK-7223 Project: Jackrabbit Oak Issue Type: Bug Components: blob-plugins Affects Versions: 1.8.1 Reporter: Amit Jain Assignee: Amit Jain Fix For: 1.10, 1.8.2 The FileCache#load method needs to clean the files which may have been downloaded partially in case of errors from backend. This partially downloaded file should be removed before returning from the method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)