[jira] [Commented] (OAK-4211) FileAccess.Mapped leaks file channels
[
https://issues.apache.org/jira/browse/OAK-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251264#comment-15251264
]
Chetan Mehrotra commented on OAK-4211:
--
bq. IMO there shouldn't be anyone accessing the mapped segments any more once
closed
Yes that should be the case. So far (before compaction is implemented) once a
TarReader is opened and any SegmentNodeState is returned which is backed by a
bytebuffer from that Tar then it was not possible to safely say when the reader
can be closed. Now with current compaction logic I think all read should get
routed to new segments and things should be fine
One place recently where we faced this issue was in OAK-4236 where per testcase
a SegemtnNodeState remained alive and even after closing the FileStore and was
preventing the cleanup of tar files as it held a file handle
> FileAccess.Mapped leaks file channels
> --
>
> Key: OAK-4211
> URL: https://issues.apache.org/jira/browse/OAK-4211
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Michael Dürig
> Fix For: 1.6
>
> Attachments: OAK-4211.patch
>
>
> {{FileAccess.Mapped}} seems to leak {{FileChannnel}} instances. The file
> channels are acquired in the constructor but never release.
> FYI [~alex.parvulescu], [~frm]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4236) SegmentNodeStoreConfigTest#testDeadlock in oak-pojosr leaves out tmp file
[
https://issues.apache.org/jira/browse/OAK-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251248#comment-15251248
]
Chetan Mehrotra commented on OAK-4236:
--
Thanks Vikas for investigating this!
> SegmentNodeStoreConfigTest#testDeadlock in oak-pojosr leaves out tmp file
> -
>
> Key: OAK-4236
> URL: https://issues.apache.org/jira/browse/OAK-4236
> Project: Jackrabbit Oak
> Issue Type: Technical task
>Reporter: Vikas Saurabh
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.5.2
>
> Attachments: unit-tests.log
>
>
> Leftover temp folder contains {{repository/segmentstore/data0a.tar}}...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-4261) Add PropInfo.asPropertyState
[
https://issues.apache.org/jira/browse/OAK-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angela resolved OAK-4261.
-
Resolution: Fixed
Assignee: angela
Fix Version/s: 1.5.2
Committed revision 1740195.
> Add PropInfo.asPropertyState
>
>
> Key: OAK-4261
> URL: https://issues.apache.org/jira/browse/OAK-4261
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Fix For: 1.5.2
>
>
> building a new propertystate could be delegated to the dedicated {{PropInfo}}
> simplifying the importer code base and avoid code duplication in
> {{ProtectedItemImporter}} implementations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-4261) Add PropInfo.asPropertyState
angela created OAK-4261:
---
Summary: Add PropInfo.asPropertyState
Key: OAK-4261
URL: https://issues.apache.org/jira/browse/OAK-4261
Project: Jackrabbit Oak
Issue Type: Improvement
Components: core
Reporter: angela
Priority: Minor
building a new propertystate could be delegated to the dedicated {{PropInfo}}
simplifying the importer code base and avoid code duplication in
{{ProtectedItemImporter}} implementations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OAK-3208) Exercise for External Authentication, IDP and User Synchronization
[ https://issues.apache.org/jira/browse/OAK-3208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angela reassigned OAK-3208: --- Assignee: angela > Exercise for External Authentication, IDP and User Synchronization > -- > > Key: OAK-3208 > URL: https://issues.apache.org/jira/browse/OAK-3208 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: exercise >Reporter: angela >Assignee: angela >Priority: Minor > > the exercises section for the 'authentication/external' topic is still > completely empty and just contains an initial introduction stub. > possible areas for exercises are: > - understand the basic concepts (e.g. using LDAP integration as example) > - understand the external login module, how it interacts with external IDP > - learn how the the external IDP works and how custom implementations can be > used > - train the user synchronisation: configuration of existing default > implementation, plugging custom implementations, how it is used during > authentication, the jmx integration -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4222) Cleanup ExternalLoginModuleTest
[
https://issues.apache.org/jira/browse/OAK-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250389#comment-15250389
]
angela commented on OAK-4222:
-
hmm... sounds odd. wouldn't it be better then to clear the options in the
after-method?
but now that you mention it:
{{ExternalLoginModuleTest}} also hides the protected options provided from the
base, while I didn't see this in the other derived tests nor is the
finally-options.clear used consistently throughout the various derived tests. I
think I will look at the blame tomorrow to see if I understand that :)
> Cleanup ExternalLoginModuleTest
> ---
>
> Key: OAK-4222
> URL: https://issues.apache.org/jira/browse/OAK-4222
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Attachments: OAK-4222.patch
>
>
> While looking at the tests present in oak-auth-external I found the
> {{ExternalLoginModuleTest}} keeping it's own protected {{options}} map hiding
> the {{options}} defined and populated in the setup of the base-class.
> Not sure what this was intended to be used for but the fact that the options
> get cleared in the finally-section of some of the test-cases looks quite
> strange to me.
> [~tripod], I would appreciate if you review the attached patch and let me
> know if you think the proposed cleanup breaks the very intention of your
> test-cases. Thanks a lot.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4222) Cleanup ExternalLoginModuleTest
[
https://issues.apache.org/jira/browse/OAK-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250362#comment-15250362
]
Tobias Bocanegra commented on OAK-4222:
---
...but I would keep the {{options.clear()}} in the final blocks in order to be
consistent with the other tests that extend from
{{ExternalLoginModuleTestBase}}. For some reason I needed this, maybe a raise
condition during init.
> Cleanup ExternalLoginModuleTest
> ---
>
> Key: OAK-4222
> URL: https://issues.apache.org/jira/browse/OAK-4222
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Attachments: OAK-4222.patch
>
>
> While looking at the tests present in oak-auth-external I found the
> {{ExternalLoginModuleTest}} keeping it's own protected {{options}} map hiding
> the {{options}} defined and populated in the setup of the base-class.
> Not sure what this was intended to be used for but the fact that the options
> get cleared in the finally-section of some of the test-cases looks quite
> strange to me.
> [~tripod], I would appreciate if you review the attached patch and let me
> know if you think the proposed cleanup breaks the very intention of your
> test-cases. Thanks a lot.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4222) Cleanup ExternalLoginModuleTest
[
https://issues.apache.org/jira/browse/OAK-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250346#comment-15250346
]
Tobias Bocanegra commented on OAK-4222:
---
+1 to remove the {{options}}. probably copy-past leftover. thanks!
> Cleanup ExternalLoginModuleTest
> ---
>
> Key: OAK-4222
> URL: https://issues.apache.org/jira/browse/OAK-4222
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Attachments: OAK-4222.patch
>
>
> While looking at the tests present in oak-auth-external I found the
> {{ExternalLoginModuleTest}} keeping it's own protected {{options}} map hiding
> the {{options}} defined and populated in the setup of the base-class.
> Not sure what this was intended to be used for but the fact that the options
> get cleared in the finally-section of some of the test-cases looks quite
> strange to me.
> [~tripod], I would appreciate if you review the attached patch and let me
> know if you think the proposed cleanup breaks the very intention of your
> test-cases. Thanks a lot.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OAK-4215) Improve test-coverage for External Authentication
[
https://issues.apache.org/jira/browse/OAK-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angela reassigned OAK-4215:
---
Assignee: angela
> Improve test-coverage for External Authentication
> -
>
> Key: OAK-4215
> URL: https://issues.apache.org/jira/browse/OAK-4215
> Project: Jackrabbit Oak
> Issue Type: Epic
> Components: auth-external
>Reporter: angela
>Assignee: angela
> Fix For: 1.6
>
>
> IMO it would be really beneficial to have more tests with the
> {{oak-auth-external}} module in order to provide us is more stability when
> making fixes or introducing enhancements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (OAK-4218) Base SyncMBeanImpl on Oak API
[
https://issues.apache.org/jira/browse/OAK-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angela reassigned OAK-4218:
---
Assignee: angela
> Base SyncMBeanImpl on Oak API
> -
>
> Key: OAK-4218
> URL: https://issues.apache.org/jira/browse/OAK-4218
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: auth-external
>Reporter: angela
>Assignee: angela
>
> While looking at the oak-auth-external code base I found that
> {{SyncMBeanImpl}} is based on JCR API while sync called during authentication
> will solely rely on API defined by oak-core.
> This not only limits the implementations to operations that can executed on
> the JCR API but also introduces the risk of inconsistencies.
> As a matter of fact {{ExternalLoginModuleTestBase.createMBean}} also lists
> this as limitation and TODO:
> {quote}
> // todo: how to retrieve JCR repository here? maybe we should base
> the sync mbean on oak directly.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4224) DefaultSyncContext.sync(ExternalIdentity) should verify IDP
[
https://issues.apache.org/jira/browse/OAK-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250293#comment-15250293
]
Tobias Bocanegra commented on OAK-4224:
---
I think we should keep the identity in the sync result the same as was put into
the sync call. but set the status to FOREIGN.
I think that the {{SyncResult.getIdentity()}} should never be null. this
simplifies result processing, in cases where the user deals with a list of
results.
> DefaultSyncContext.sync(ExternalIdentity) should verify IDP
> ---
>
> Key: OAK-4224
> URL: https://issues.apache.org/jira/browse/OAK-4224
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: auth-external
>Reporter: angela
>Assignee: angela
>Priority: Minor
> Attachments: OAK-4224.patch, OAK-4224_2.patch
>
>
> while writing more test for {{DefaultSyncContext}} i realized that the
> implementation of {{sync(ExternalIdentity)}} doesn't verify that the given
> external identity belongs to the same IDP than the one associated with the
> context instance.
> IMHO this would be needed and useful particularly when multiple IDPs are
> combined. also, the {{DefaultSyncContext}} is a public exposed class, I
> would prefer if it would guard against mixing up sync of external identities
> from different sources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-4260) Define and implement migration from oak-segment to oak-segment-next
Michael Dürig created OAK-4260:
--
Summary: Define and implement migration from oak-segment to
oak-segment-next
Key: OAK-4260
URL: https://issues.apache.org/jira/browse/OAK-4260
Project: Jackrabbit Oak
Issue Type: Task
Components: segment-next, segmentmk
Reporter: Michael Dürig
Fix For: 1.6
We need to come up with a plan, implementation and documentation for how we
deal with migrating from {{oak-segment}} to {{oak-segment-next}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4211) FileAccess.Mapped leaks file channels
[
https://issues.apache.org/jira/browse/OAK-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250036#comment-15250036
]
Michael Dürig commented on OAK-4211:
Patch looks good to me. Just remember to apply it to {{oak-segment-next}}
instead of {{oak-segment}}.
> FileAccess.Mapped leaks file channels
> --
>
> Key: OAK-4211
> URL: https://issues.apache.org/jira/browse/OAK-4211
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Michael Dürig
> Fix For: 1.6
>
> Attachments: OAK-4211.patch
>
>
> {{FileAccess.Mapped}} seems to leak {{FileChannnel}} instances. The file
> channels are acquired in the constructor but never release.
> FYI [~alex.parvulescu], [~frm]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4260) Define and implement migration from oak-segment to oak-segment-next
[
https://issues.apache.org/jira/browse/OAK-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250051#comment-15250051
]
Michael Dürig commented on OAK-4260:
A side-grade seem to be the most promising approach to me.
> Define and implement migration from oak-segment to oak-segment-next
> ---
>
> Key: OAK-4260
> URL: https://issues.apache.org/jira/browse/OAK-4260
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-next, segmentmk
>Reporter: Michael Dürig
> Labels: migration
> Fix For: 1.6
>
>
> We need to come up with a plan, implementation and documentation for how we
> deal with migrating from {{oak-segment}} to {{oak-segment-next}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4211) FileAccess.Mapped leaks file channels
[
https://issues.apache.org/jira/browse/OAK-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250029#comment-15250029
]
Michael Dürig commented on OAK-4211:
Could you clarify a bit? IMO there shouldn't be anyone accessing the mapped
segments any more once closed. This would be a bug in its own right.
> FileAccess.Mapped leaks file channels
> --
>
> Key: OAK-4211
> URL: https://issues.apache.org/jira/browse/OAK-4211
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Michael Dürig
> Fix For: 1.6
>
> Attachments: OAK-4211.patch
>
>
> {{FileAccess.Mapped}} seems to leak {{FileChannnel}} instances. The file
> channels are acquired in the constructor but never release.
> FYI [~alex.parvulescu], [~frm]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4258) Don't release oak-segment-next when the reactor project is released
[
https://issues.apache.org/jira/browse/OAK-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francesco Mari updated OAK-4258:
Fix Version/s: 1.6
> Don't release oak-segment-next when the reactor project is released
> ---
>
> Key: OAK-4258
> URL: https://issues.apache.org/jira/browse/OAK-4258
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Francesco Mari
>Assignee: Francesco Mari
>Priority: Blocker
> Fix For: 1.6, 1.5.2
>
>
> {{oak-segment-next}} is currently included in the top-level reactor project.
> This implies that a release performed on the reactor project will be
> performed on {{oak-segment-next}} as well. {{oak-segment-next}} shouldn't
> share the same versioning and release policy as the rest of the project, at
> least temporarily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4258) Don't release oak-segment-next when the reactor project is released
[
https://issues.apache.org/jira/browse/OAK-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250013#comment-15250013
]
Michael Dürig commented on OAK-4258:
Made this a blocker for 1.5.2 so we don't inadvertently release
{{oak-segment-next}}.
> Don't release oak-segment-next when the reactor project is released
> ---
>
> Key: OAK-4258
> URL: https://issues.apache.org/jira/browse/OAK-4258
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Francesco Mari
>Assignee: Francesco Mari
>Priority: Blocker
> Fix For: 1.5.2
>
>
> {{oak-segment-next}} is currently included in the top-level reactor project.
> This implies that a release performed on the reactor project will be
> performed on {{oak-segment-next}} as well. {{oak-segment-next}} shouldn't
> share the same versioning and release policy as the rest of the project, at
> least temporarily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4258) Don't release oak-segment-next when the reactor project is released
[
https://issues.apache.org/jira/browse/OAK-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4258:
---
Fix Version/s: (was: 1.6)
1.5.2
> Don't release oak-segment-next when the reactor project is released
> ---
>
> Key: OAK-4258
> URL: https://issues.apache.org/jira/browse/OAK-4258
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Francesco Mari
>Assignee: Francesco Mari
>Priority: Blocker
> Fix For: 1.5.2
>
>
> {{oak-segment-next}} is currently included in the top-level reactor project.
> This implies that a release performed on the reactor project will be
> performed on {{oak-segment-next}} as well. {{oak-segment-next}} shouldn't
> share the same versioning and release policy as the rest of the project, at
> least temporarily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4258) Don't release oak-segment-next when the reactor project is released
[
https://issues.apache.org/jira/browse/OAK-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4258:
---
Priority: Blocker (was: Major)
> Don't release oak-segment-next when the reactor project is released
> ---
>
> Key: OAK-4258
> URL: https://issues.apache.org/jira/browse/OAK-4258
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Francesco Mari
>Assignee: Francesco Mari
>Priority: Blocker
> Fix For: 1.5.2
>
>
> {{oak-segment-next}} is currently included in the top-level reactor project.
> This implies that a release performed on the reactor project will be
> performed on {{oak-segment-next}} as well. {{oak-segment-next}} shouldn't
> share the same versioning and release policy as the rest of the project, at
> least temporarily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-4259) Implement fixtures for running again oak-segment and/or oak-segment-next
Michael Dürig created OAK-4259:
--
Summary: Implement fixtures for running again oak-segment and/or
oak-segment-next
Key: OAK-4259
URL: https://issues.apache.org/jira/browse/OAK-4259
Project: Jackrabbit Oak
Issue Type: Task
Components: segment-next
Reporter: Michael Dürig
We need fixtures to run UTs / ITs against either or both segment
implementations {{oak-segment}} and {{oak-segment-next}}.
Ideally we can enable them individually through e.g. environment variables. A
standard build would run against {{oak-segment}} so not to affect others.
{{oak-segment-next}} could be enabled on request locally or for the CI.
Once we deprecate {{oak-segment}} we would switch the default fixture to
{{oak-segment-next}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4202) leakage of temp files
[ https://issues.apache.org/jira/browse/OAK-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Reschke updated OAK-4202: Attachment: leftovers-1740146.txt now including integration tests, results are below: oak-segment-next/target/tmp/junit6236397128016695768/ oak-segment-next/target/tmp/junit7851543373414007619/ oak-segment/target/tmp/junit3794148232157765322/ oak-segment/target/tmp/junit8202170354036228078/ > leakage of temp files > - > > Key: OAK-4202 > URL: https://issues.apache.org/jira/browse/OAK-4202 > Project: Jackrabbit Oak > Issue Type: Task >Reporter: Julian Reschke >Priority: Minor > Attachments: leftovers-1739224.txt, leftovers-1739267.txt, > leftovers-1739428.txt, leftovers-1739696.txt, leftovers-1739725.txt, > leftovers-1739889.txt, leftovers-1739902.txt, leftovers-1739931.txt, > leftovers-1739958.txt, leftovers-1740146.txt, leftovers.txt > > > Running the tests currently leaves a ton of temporary files. I assume in > general these are caused by bugs in test cases, but they could also indicate > problems in the actual code. > Will attach a detailed list separately and open sub-tasks accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-4258) Don't release oak-segment-next when the reactor project is released
Francesco Mari created OAK-4258:
---
Summary: Don't release oak-segment-next when the reactor project
is released
Key: OAK-4258
URL: https://issues.apache.org/jira/browse/OAK-4258
Project: Jackrabbit Oak
Issue Type: Bug
Components: segment-next
Reporter: Francesco Mari
Assignee: Francesco Mari
Fix For: 1.6
{{oak-segment-next}} is currently included in the top-level reactor project.
This implies that a release performed on the reactor project will be performed
on {{oak-segment-next}} as well. {{oak-segment-next}} shouldn't share the same
versioning and release policy as the rest of the project, at least temporarily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4233) Property index stored locally
[
https://issues.apache.org/jira/browse/OAK-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249959#comment-15249959
]
Vikas Saurabh commented on OAK-4233:
Oh, btw, I think irrespective of how we go about async-prop-index, Marcel's
idea to do last level diff with async-prop-index-checkpoint v/s session-rev
should still apply.
> Property index stored locally
> -
>
> Key: OAK-4233
> URL: https://issues.apache.org/jira/browse/OAK-4233
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk, query
>Reporter: Tomek Rękawek
>Priority: Minor
> Fix For: 1.6
>
>
> When running Oak in a cluster, each write operation is expensive. After
> performing some stress-tests with a geo-distributed Mongo cluster, we've
> found out that updating property indexes is a large part of the overall
> traffic. Let's try to create a new property-local index, that will save the
> indexed data locally, without sharing it.
> Assumptions:
> -there's a new {{property-local}} index type for which the data are saved in
> the local SegmentNodeStore instance created specifically for this purpose,
> -local changes are indexed using a new editor, based on the
> {{PropertyIndexEditor}},
> -remote changes are extracted from the JournalEntries fetched in the
> background read operation and indexed as well,
> -the new index type won't support uniqueness restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4257) Findbug issues (mostly NPEs)
[ https://issues.apache.org/jira/browse/OAK-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249946#comment-15249946 ] Alfusainey Jallow commented on OAK-4257: i executed 'mvn clean install -PintegrationTesting' and the build passes on my side > Findbug issues (mostly NPEs) > > > Key: OAK-4257 > URL: https://issues.apache.org/jira/browse/OAK-4257 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: auth-external, authorization-cug, remoting, security, > upgrade >Reporter: Alfusainey Jallow >Priority: Trivial > Attachments: findbugs-trivial-issues.patch > > > Sub task addressing trivial issues reported by the findbugs static analyzer. > the issues addressed are mostly NPEs plus a few in oak-security -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4233) Property index stored locally
[
https://issues.apache.org/jira/browse/OAK-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249938#comment-15249938
]
Tomek Rękawek commented on OAK-4233:
[~catholicon], thanks for the suggestion. If I understand correctly, your idea
is to keep the separate, local SegmentMK (as in the original idea) but use
Lucene index rather than property index. Lucene would write data to the
SegmentMK using the OakDirectory wrapper, as it does right now.
bq. Pure file system would not get the MVCC truth-ness...
Could you elaborate on this? Replacing the SegmentMK+OakDirectory+Lucene with
FS+Lucene seems like a simpler solution here.
> Property index stored locally
> -
>
> Key: OAK-4233
> URL: https://issues.apache.org/jira/browse/OAK-4233
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk, query
>Reporter: Tomek Rękawek
>Priority: Minor
> Fix For: 1.6
>
>
> When running Oak in a cluster, each write operation is expensive. After
> performing some stress-tests with a geo-distributed Mongo cluster, we've
> found out that updating property indexes is a large part of the overall
> traffic. Let's try to create a new property-local index, that will save the
> indexed data locally, without sharing it.
> Assumptions:
> -there's a new {{property-local}} index type for which the data are saved in
> the local SegmentNodeStore instance created specifically for this purpose,
> -local changes are indexed using a new editor, based on the
> {{PropertyIndexEditor}},
> -remote changes are extracted from the JournalEntries fetched in the
> background read operation and indexed as well,
> -the new index type won't support uniqueness restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4233) Property index stored locally
[
https://issues.apache.org/jira/browse/OAK-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249918#comment-15249918
]
Marcel Reutegger commented on OAK-4233:
---
- The index would still be store in the repository, yes.
- The index editor would only have to update the index selected by the query
engine. Though, this would probably require more changes. Another optimization
is the path filter passed to the index. An index update may not be necessary
even if there are are pending changes, as long as those changes are not in the
subtree covered by the filter.
- Persisted index updates would still happen with the async index update as we
do right now.
- That's an implementation detail. In the most naive implementation each query
has it's own branch on top of the persisted base index. It may also be possible
to keep those in-memory branches for a short while (until the persisted index
is updated?) and share them when there are other concurrent queries.
> Property index stored locally
> -
>
> Key: OAK-4233
> URL: https://issues.apache.org/jira/browse/OAK-4233
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: documentmk, query
>Reporter: Tomek Rękawek
>Priority: Minor
> Fix For: 1.6
>
>
> When running Oak in a cluster, each write operation is expensive. After
> performing some stress-tests with a geo-distributed Mongo cluster, we've
> found out that updating property indexes is a large part of the overall
> traffic. Let's try to create a new property-local index, that will save the
> indexed data locally, without sharing it.
> Assumptions:
> -there's a new {{property-local}} index type for which the data are saved in
> the local SegmentNodeStore instance created specifically for this purpose,
> -local changes are indexed using a new editor, based on the
> {{PropertyIndexEditor}},
> -remote changes are extracted from the JournalEntries fetched in the
> background read operation and indexed as well,
> -the new index type won't support uniqueness restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4202) leakage of temp files
[ https://issues.apache.org/jira/browse/OAK-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249925#comment-15249925 ] Julian Reschke commented on OAK-4202: - Indeed, thanks! (now redoing this with integration profile) > leakage of temp files > - > > Key: OAK-4202 > URL: https://issues.apache.org/jira/browse/OAK-4202 > Project: Jackrabbit Oak > Issue Type: Task >Reporter: Julian Reschke >Priority: Minor > Attachments: leftovers-1739224.txt, leftovers-1739267.txt, > leftovers-1739428.txt, leftovers-1739696.txt, leftovers-1739725.txt, > leftovers-1739889.txt, leftovers-1739902.txt, leftovers-1739931.txt, > leftovers-1739958.txt, leftovers.txt > > > Running the tests currently leaves a ton of temporary files. I assume in > general these are caused by bugs in test cases, but they could also indicate > problems in the actual code. > Will attach a detailed list separately and open sub-tasks accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-3362) Estimate compaction based on diff to previous compacted head state
[
https://issues.apache.org/jira/browse/OAK-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249867#comment-15249867
]
Michael Dürig commented on OAK-3362:
Moved this to {{oak-segment-next}}. However, not sure whether we should invest
much here. I would actually propose to remove the current estimation approach
entirely as is expensive wrt. IO, CPU and cache coherence. If we want to keep
an estimation step we need IMO come up with a cheap way (at least 2 orders of
magnitude cheaper than compaction).
> Estimate compaction based on diff to previous compacted head state
> --
>
> Key: OAK-3362
> URL: https://issues.apache.org/jira/browse/OAK-3362
> Project: Jackrabbit Oak
> Issue Type: New Feature
> Components: segment-next
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
>Priority: Minor
> Labels: compaction, gc
> Fix For: 1.6
>
>
> Food for thought: try to base the compaction estimation on a diff between the
> latest compacted state and the current state.
> Pros
> * estimation duration would be proportional to number of changes on the
> current head state
> * using the size on disk as a reference, we could actually stop the
> estimation early when we go over the gc threshold.
> * data collected during this diff could in theory be passed as input to the
> compactor so it could focus on compacting a specific subtree
> Cons
> * need to keep a reference to a previous compacted state. post-startup and
> pre-compaction this might prove difficult (except maybe if we only persist
> the revision similar to what the async indexer is doing currently)
> * coming up with a threshold for running compaction might prove difficult
> * diff might be costly, but still cheaper than the current full diff
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3362) Estimate compaction based on diff to previous compacted head state
[ https://issues.apache.org/jira/browse/OAK-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3362: --- Component/s: (was: segmentmk) segment-next > Estimate compaction based on diff to previous compacted head state > -- > > Key: OAK-3362 > URL: https://issues.apache.org/jira/browse/OAK-3362 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-next >Reporter: Alex Parvulescu >Assignee: Alex Parvulescu >Priority: Minor > Labels: compaction, gc > Fix For: 1.6 > > > Food for thought: try to base the compaction estimation on a diff between the > latest compacted state and the current state. > Pros > * estimation duration would be proportional to number of changes on the > current head state > * using the size on disk as a reference, we could actually stop the > estimation early when we go over the gc threshold. > * data collected during this diff could in theory be passed as input to the > compactor so it could focus on compacting a specific subtree > Cons > * need to keep a reference to a previous compacted state. post-startup and > pre-compaction this might prove difficult (except maybe if we only persist > the revision similar to what the async indexer is doing currently) > * coming up with a threshold for running compaction might prove difficult > * diff might be costly, but still cheaper than the current full diff -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2683) the "hitting the observation queue limit" problem
[ https://issues.apache.org/jira/browse/OAK-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2683: --- Component/s: (was: segmentmk) segment-next > the "hitting the observation queue limit" problem > - > > Key: OAK-2683 > URL: https://issues.apache.org/jira/browse/OAK-2683 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: core, mongomk, segment-next >Reporter: Stefan Egli > Labels: observation, resilience > > There are several tickets in this area: > * OAK-2587: threading with observation being too eagar causing observation > queue to grow > * OAK-2669: avoiding diffing from mongo by using persistent cache instead. > * OAK-2349: which might be a duplicate or at least similar to 2669.. > * OAK-2562: diffcache is inefficient > Yet I think it makes sense to create this summarizing ticket, about > describing again what happens when the observation queue hits the limit - and > eventually about how this can be improved > Consider the following scenario (also compare with OAK-2587 - but that one > focused more on eagerness of threading): > * rate of incoming commits is large and starts to generate many changes into > the observation queues, hence those queue become somewhat filled/loaded > * depending on the underlying nodestore used the calculation of diffs is more > or less expensive - but at least for mongomk it is important that the diff > can be served from the cache > ** in case of mongomk it can happen that diffs are no longer found in the > cache and thus require a round-trip to mongo - which is magnitudes slower > than via cache of course. this would result in the queue to start increasing > even faster as dequeuing becomes slower now. > ** not sure about tarmk - I believe it should always be fast there > * so based on the above, there can be a situation where the queue grows and > hits the configured limit > * if this limit is reached, the current mechanism is to collapse any > subsequent change into one-big-marked-as-external-event change, lets call > this a collapsed-change. > * this collapsed-change now becomes part of the normal queue and eventually > would 'walk down the queue' and be processed normally - hence opening a high > chance that yet a new collapsed-change is created should the queue just hit > the limit again. and this game can now be played for a while, resulting in > the queue to contain many/mostly such collapse-changes. > * there is now an additional assumption in that the diffing of such collapses > is more expensive than normal diffing - plus it is almost guaranteed that the > diff cannot for example be shared between observation listeners, since the > exact 'collapse borders' depends on timing of each of the listeners' queues - > ie the collapse diffs are unique thus not cachable.. > * so as a result: once you have those collapse-diffs you can almost not get > rid of them - they are heavy to process - hence dequeuing is very slow > * at the same time, there is always likely some commits happening in a > typical system, eg with sling on top you have sling discovery which does > heartbeats every now and then. So there's always new commits that add to the > load. > * this will hence create a situation where quite a small additional commit > rate can keep all the queues filled - due to the fact that the queue is full > with 'heavy collapse diffs' that have to be calculated for each and every > listener (of which you could have eg 150-200) individually. > So again, possible solutions for this: > * OAK-2669: tune diffing via persistent cache > * OAK-2587: have more threads to remain longer 'in the cache zone' > * tune your input speed explicitly to avoid filling the observation queues > (this would be specific to your use-case of course, but can be seen as > explicitly throttling on the input side) > * increase the relevant caches to the max > * but I think we will come up with yet a broader improvement of this > observation queue limit problem by either > ** doing flow control - eg via the commit rate limiter (also see OAK-1659) > ** moving out handling of observation changes to a messaging subsystem - be > it to handle local events only (since handling external events makes the > system problematic wrt scalability if not done right) - also see > [corresponding suggestion on dev > list|http://markmail.org/message/b5trr6csyn4zzuj7] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-1576) SegmentMK: Implement refined conflict resolution for addExistingNode conflicts
[ https://issues.apache.org/jira/browse/OAK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-1576: --- Component/s: (was: segmentmk) segment-next > SegmentMK: Implement refined conflict resolution for addExistingNode conflicts > -- > > Key: OAK-1576 > URL: https://issues.apache.org/jira/browse/OAK-1576 > Project: Jackrabbit Oak > Issue Type: Sub-task > Components: segment-next >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: concurrency, resilience, scalability > > Implement refined conflict resolution for addExistingNode conflicts as > defined in the parent issue for the SegementMK. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2405) Monitoring to track old NodeStates
[
https://issues.apache.org/jira/browse/OAK-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-2405:
---
Component/s: (was: segmentmk)
segment-next
> Monitoring to track old NodeStates
> --
>
> Key: OAK-2405
> URL: https://issues.apache.org/jira/browse/OAK-2405
> Project: Jackrabbit Oak
> Issue Type: Sub-task
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: gc, monitoring, tooling
> Fix For: 1.6
>
>
> We should add some monitoring that allows us to track "old" node states,
> which potentially block revision gc.
> Possible approaches:
> * Add monitoring too old revisions (root node states) along with the stack
> traces from where they have been acquired.
> * Include RecordId of root node state in the {{SessionMBean}}.
> * Add additional tooling on top of the {{SessionMBean}} to make it easier to
> make sense of the wealth of information provided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2498) Root record references provide too little context for parsing a segment
[ https://issues.apache.org/jira/browse/OAK-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2498: --- Component/s: (was: segmentmk) segment-next > Root record references provide too little context for parsing a segment > --- > > Key: OAK-2498 > URL: https://issues.apache.org/jira/browse/OAK-2498 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-next >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: tools > Fix For: 1.6 > > > According to the [documentation | > http://jackrabbit.apache.org/oak/docs/nodestore/segmentmk.html] the root > record references in a segment header provide enough context for parsing all > records within this segment without any external information. > Turns out this is not true: if a root record reference turns e.g. to a list > record. The items in that list are record ids of unknown type. So even though > those records might live in the same segment, we can't parse them as we don't > know their type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2403) Improve monitoring capabilities for TarMk revision gc
[ https://issues.apache.org/jira/browse/OAK-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2403: --- Component/s: (was: segmentmk) segment-next > Improve monitoring capabilities for TarMk revision gc > - > > Key: OAK-2403 > URL: https://issues.apache.org/jira/browse/OAK-2403 > Project: Jackrabbit Oak > Issue Type: Task > Components: segment-next >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: gc, monitoring, tooling > Fix For: 1.6 > > > Container devoted to improving monitoring of the TarMk revision garbage > collection process. The overall goal is to make it more transparent what > revision gc does, how it performs, why it failed etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2849) Improve revision gc on SegmentMK
[ https://issues.apache.org/jira/browse/OAK-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2849: --- Component/s: (was: segmentmk) segment-next > Improve revision gc on SegmentMK > > > Key: OAK-2849 > URL: https://issues.apache.org/jira/browse/OAK-2849 > Project: Jackrabbit Oak > Issue Type: Epic > Components: segment-next >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: compaction, gc > Attachments: SegmentCompactionIT-conflicts.png > > > This is a container issue for the ongoing effort to improve revision gc of > the SegmentMK. > I'm exploring > * ways to make the reference graph as exact as possible and necessary: it > should not contain segments that are not referenceable any more and but must > contain all segments that are referenceable. > * ways to segregate the reference graph reducing dependencies between certain > set of segments as much as possible. > * Reducing the number of in memory references and their impact on gc as much > as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3036) DocumentRootBuilder: revisit update.limit default
[ https://issues.apache.org/jira/browse/OAK-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3036: --- Component/s: (was: segmentmk) segment-next > DocumentRootBuilder: revisit update.limit default > - > > Key: OAK-3036 > URL: https://issues.apache.org/jira/browse/OAK-3036 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: mongomk, rdbmk, segment-next >Reporter: Julian Reschke > Labels: performance > > update.limit decides whether a commit is persisted using a branch or not. The > default is 1 (and can be overridden using the system property). > A typical call pattern in JCR is to persist batches of ~1024 nodes. These > translate to more than 1 changes (see PackageImportIT), due to JCR > properties, and also indexing commit hooks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3350) Incremental compaction
[ https://issues.apache.org/jira/browse/OAK-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3350: --- Component/s: (was: segmentmk) segment-next > Incremental compaction > -- > > Key: OAK-3350 > URL: https://issues.apache.org/jira/browse/OAK-3350 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-next >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: compaction, gc > > This is OAK-3349 taken to the extreme: given a segment that is almost not > referenced any more we could just rewrite the still referenced content. That > is, say a segment contains two properties reachable from the current root > node state and all its remaining content is not reachable from the root node > state. In that case we could rewrite these two properties and create a new > root node state referencing the rewritten properties. This would effectively > make the segment eligible for being gc-ed. > Such an approach would start from segments that are sparse and compact these > instead of compacting everything as we currently do, which might cause a lot > of copying around stuff that already is compact. The challenging part here is > probably finding the segments that are sparse as this involves inverting the > reference graph. > Todo: Asses feasibility and impact, implement prototype. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2849) Improve revision gc on SegmentMK
[ https://issues.apache.org/jira/browse/OAK-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2849: --- Labels: compaction gc (was: candidate_oak_1_0 candidate_oak_1_2 compaction gc) > Improve revision gc on SegmentMK > > > Key: OAK-2849 > URL: https://issues.apache.org/jira/browse/OAK-2849 > Project: Jackrabbit Oak > Issue Type: Epic > Components: segment-next >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: compaction, gc > Attachments: SegmentCompactionIT-conflicts.png > > > This is a container issue for the ongoing effort to improve revision gc of > the SegmentMK. > I'm exploring > * ways to make the reference graph as exact as possible and necessary: it > should not contain segments that are not referenceable any more and but must > contain all segments that are referenceable. > * ways to segregate the reference graph reducing dependencies between certain > set of segments as much as possible. > * Reducing the number of in memory references and their impact on gc as much > as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3349) Partial compaction
[ https://issues.apache.org/jira/browse/OAK-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3349: --- Component/s: (was: segmentmk) segment-next > Partial compaction > -- > > Key: OAK-3349 > URL: https://issues.apache.org/jira/browse/OAK-3349 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: segment-next >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: compaction, gc > > On big repositories compaction can take quite a while to run as it needs to > create a full deep copy of the current root node state. For such cases it > could be beneficial if we could partially compact the repository thus > splitting full compaction over multiple cycles. > Partial compaction would run compaction on a sub-tree just like we now run it > on the full tree. Afterwards it would create a new root node state by > referencing the previous root node state replacing said sub-tree with the > compacted one. > Todo: Asses feasibility and impact, implement prototype. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3603) Evaluate skipping cleanup of a subset of tar files
[ https://issues.apache.org/jira/browse/OAK-3603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3603: --- Component/s: (was: segmentmk) segment-next > Evaluate skipping cleanup of a subset of tar files > -- > > Key: OAK-3603 > URL: https://issues.apache.org/jira/browse/OAK-3603 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-next >Reporter: Alex Parvulescu >Assignee: Alex Parvulescu > Labels: cleanup, gc > > Given the fact that tar readers are immutable (we only create new generations > of them once they reach a certain threshold of garbage) we can consider > coming up with a heuristic for skipping cleanup entirely for consequent > cleanup calls based on the same referenced id set (provided we can make this > set more stable, aka. OAK-2849). > Ex: for a specific input set a cleanup call on a tar reader might decide that > there's no enough garbage (some IO involved in reading through all existing > entries). if the following cleanup cycle would have the exact same input, it > doesn't make sense to recheck the tar file, we already know cleanup can be > skipped, moreover we can skip the older tar files too, as their input would > also not change. the gains increase the larger the number of tar files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3309) SegmentMK segment cache loader stats
[ https://issues.apache.org/jira/browse/OAK-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3309: --- Component/s: (was: segmentmk) segment-next > SegmentMK segment cache loader stats > > > Key: OAK-3309 > URL: https://issues.apache.org/jira/browse/OAK-3309 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-next >Reporter: Alex Parvulescu >Assignee: Alex Parvulescu > Labels: gc > Fix For: 1.6 > > Attachments: OAK-3309.patch > > > The existing Segment Cache has no loading-related stats, I'd like to see how > complicated it would be to add some. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3679) Rollback to timestamp
[ https://issues.apache.org/jira/browse/OAK-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3679: --- Component/s: (was: segmentmk) segment-next > Rollback to timestamp > - > > Key: OAK-3679 > URL: https://issues.apache.org/jira/browse/OAK-3679 > Project: Jackrabbit Oak > Issue Type: New Feature > Components: core, documentmk, segment-next >Reporter: Thomas Mueller > > We should have a feature to roll back to a certain point in time. The use > case are: > * undo a failed, large operation (for example upgrade, migration, installing > a package), > * on a copy of the repository, switch to an old state for reading old content > * recover from a corruption (for example corruption due to incorrect > "discovery" state, such as concurrent async index updates). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3696) Improve SegmentMK resilience
[ https://issues.apache.org/jira/browse/OAK-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3696: --- Component/s: (was: segmentmk) segment-next > Improve SegmentMK resilience > > > Key: OAK-3696 > URL: https://issues.apache.org/jira/browse/OAK-3696 > Project: Jackrabbit Oak > Issue Type: Epic > Components: segment-next >Reporter: Michael Marth >Assignee: Michael Dürig > Labels: resilience > > Epic for collection SegmentMK resilience improvements -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3693) Expose the internal state of the repository through indicators and checks
[ https://issues.apache.org/jira/browse/OAK-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3693: --- Component/s: (was: segmentmk) segment-next > Expose the internal state of the repository through indicators and checks > - > > Key: OAK-3693 > URL: https://issues.apache.org/jira/browse/OAK-3693 > Project: Jackrabbit Oak > Issue Type: Epic > Components: query, segment-next >Reporter: Valentin Olteanu > Labels: production > Fix For: 1.6 > > > This container groups all the issues related to areas where we could improve > the monitoring of an OAK repository. This can be achieved by exposing > different indicators of the internal state and adding checks for certain > properties. > Areas to improve: > * Async Indexing > * Revision GC -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3695) Expose ratio between waste and real data
[ https://issues.apache.org/jira/browse/OAK-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3695: --- Component/s: (was: segmentmk) segment-next > Expose ratio between waste and real data > > > Key: OAK-3695 > URL: https://issues.apache.org/jira/browse/OAK-3695 > Project: Jackrabbit Oak > Issue Type: Story > Components: segment-next >Reporter: Valentin Olteanu > Labels: gc > > As a user, I want to know the ratio (or precise absolute values) between > waste and real data on TarMK, so that I can decide if Revision GC needs to be > run. The measurement has to be done on a running repository and without > impacting the performance. > This would also help measure the efficiency of Revision GC and see the effect > of improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4014) The segment store should merge small TAR files into bigger ones
[ https://issues.apache.org/jira/browse/OAK-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-4014: --- Component/s: (was: segmentmk) segment-next > The segment store should merge small TAR files into bigger ones > --- > > Key: OAK-4014 > URL: https://issues.apache.org/jira/browse/OAK-4014 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-next >Reporter: Francesco Mari >Assignee: Francesco Mari > Fix For: 1.6 > > > The cleanup process removes unused segments from TAR files and writes new > generations of those TAR files without the removed segments. > In the long run, the size of some TAR file might be smaller than the maximum > size allowed for a TAR file. At the time this issue was created the default > maximum size of a TAR file is 256 MiB. > If there are many small TAR files, it should be possible to merge them in > bigger files. This way, we can reduce the total number of TAR files in the > segment store, and thus the number of open file descriptors that Oak has to > maintain. > A possible implementation for the merge operation is the following: > # Sort the list of TAR files by size, ascending. > # Pick TAR files for the sorted list until the sum of their sizes after the > merge is less than 256 MiB. > # Merge the picked up files into a new TAR file and marked the picked up > files for deletion. > # Continue picking up TAR files from the sorted list until the list is > exhausted or until it's only possible to pick a single TAR file. > The merge process can run in a background thread but it is important that it > doesn't conflict with the cleanup operation, since merge and cleanup both > change the representation of TAR files on the file system. Two possible > solutions to avoid conflicts are: > # Use a global lock for the whole set of TAR files. > # Use a lock per TAR file. The cleanup and merge processes have to agree on > the order to use when acquiring the lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3797) SegmentTracker#collectBlobReferences should retain fewer SegmentId instances
[
https://issues.apache.org/jira/browse/OAK-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-3797:
---
Component/s: (was: segmentmk)
segment-next
> SegmentTracker#collectBlobReferences should retain fewer SegmentId instances
>
>
> Key: OAK-3797
> URL: https://issues.apache.org/jira/browse/OAK-3797
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-next
>Reporter: Michael Dürig
> Labels: datastore, gc
> Fix For: 1.6
>
>
> {{SegmentTracker#collectBlobReferences}} currently keeps a queue of yet
> unprocessed {{SegmentId}} instances internally. This potentially impacts the
> system as those instances are also tracked in the segment tracker's segment
> id tables. I think we should improve the implementation to not retain so many
> {{SegmentId}} instances and rely on arrays of {{msb}}, {{lsb}} instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-4015) Expedite commits from the compactor
[ https://issues.apache.org/jira/browse/OAK-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249851#comment-15249851 ] Michael Dürig commented on OAK-4015: I think ultimately this should be implemented through a commit scheduler (OAK-4122). > Expedite commits from the compactor > --- > > Key: OAK-4015 > URL: https://issues.apache.org/jira/browse/OAK-4015 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-next >Reporter: Michael Dürig > Labels: compaction, gc, perfomance > Fix For: 1.6 > > Attachments: OAK-4015-histo.png, OAK-4015-wait-time.png > > > Concurrent commits during compaction cause those to be re-compacted. > Currently it seems that the compaction thread can end up waiting for some > time to acquire the commit lock [1], which in turn causes more commits to > pile up to be re-compacted. I think this could be improved by tweaking the > lock such that the compactor could jump ahead of the queue. I.e. use a lock > which can be acquired in expedited mode. > [1] SegmentNodeStore#commitSemaphore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4040) Some classes from o.a.j.o.plugins.segment.compaction should be exported
[
https://issues.apache.org/jira/browse/OAK-4040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4040:
---
Component/s: (was: segmentmk)
segment-next
> Some classes from o.a.j.o.plugins.segment.compaction should be exported
> ---
>
> Key: OAK-4040
> URL: https://issues.apache.org/jira/browse/OAK-4040
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Francesco Mari
> Fix For: 1.6
>
>
> Classes
> {{org.apache.jackrabbit.oak.plugins.segment.compaction.CompactionStrategy}}
> and
> {{org.apache.jackrabbit.oak.plugins.segment.compaction.CompactionStrategyMBean}}
> should be exported. The former is used in the public API of multiple classes
> from {{org.apache.jackrabbit.oak.plugins.segment.file}} and
> {{org.apache.jackrabbit.oak.plugins.segment}}, while the latter is used as
> interface type for a service registered in the whiteboard.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4015) Expedite commits from the compactor
[ https://issues.apache.org/jira/browse/OAK-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-4015: --- Component/s: (was: segmentmk) segment-next > Expedite commits from the compactor > --- > > Key: OAK-4015 > URL: https://issues.apache.org/jira/browse/OAK-4015 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-next >Reporter: Michael Dürig > Labels: compaction, gc, perfomance > Fix For: 1.6 > > Attachments: OAK-4015-histo.png, OAK-4015-wait-time.png > > > Concurrent commits during compaction cause those to be re-compacted. > Currently it seems that the compaction thread can end up waiting for some > time to acquire the commit lock [1], which in turn causes more commits to > pile up to be re-compacted. I think this could be improved by tweaking the > lock such that the compactor could jump ahead of the queue. I.e. use a lock > which can be acquired in expedited mode. > [1] SegmentNodeStore#commitSemaphore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-4121) CompactionMap#get not transitive across compaction map generations
[
https://issues.apache.org/jira/browse/OAK-4121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-4121.
Resolution: Won't Fix
This is fixed in {{oak-segment-next}}: RIP compaction map
> CompactionMap#get not transitive across compaction map generations
> --
>
> Key: OAK-4121
> URL: https://issues.apache.org/jira/browse/OAK-4121
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: compaction, gc
>
> {{CompactionMap#get(RecordId before)}} searches through the compaction maps
> until it finds one containing {{before}} returning its value. However that
> one might already have been compacted again an be present as key in a later
> compaction map generation.
> A correct implementation of {{CompactionMap#get(RecordId before)}} should
> consider the transitive closure over all maps starting at {{before}}. Note
> however that in this case we would also need to stop removing keys from the
> compaction map after cleanup as this would break transitivity again. (See
> http://svn.apache.org/viewvc?view=revision=1673791)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4122) Replace the commit semaphore in the segment node store with a scheduler
[
https://issues.apache.org/jira/browse/OAK-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4122:
---
Component/s: (was: segmentmk)
segment-next
> Replace the commit semaphore in the segment node store with a scheduler
> ---
>
> Key: OAK-4122
> URL: https://issues.apache.org/jira/browse/OAK-4122
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Francesco Mari
> Labels: performance, scalability, throughput
> Fix For: 1.6
>
>
> {{SegmentNodeStore}} currently uses a semaphore to coordinate concurrent
> commits thus relying on the scheduling algorithm of that implementation and
> ultimately of the JVM for in what order commits are processed.
> I think it would be beneficial to replace that semaphore with an explicit
> queue of pending commit. This would allow us to implement a proper scheduler
> optimising for e.g. minimal system load, maximal throughput or minimal
> latency etc. A scheduler could e.g. give precedence to big commits and order
> commits along the order of its base revisions, which would decrease the
> amount of work to be done in rebasing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4138) Decouple revision cleanup from the flush thread
[ https://issues.apache.org/jira/browse/OAK-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-4138: --- Component/s: (was: segmentmk) segment-next > Decouple revision cleanup from the flush thread > --- > > Key: OAK-4138 > URL: https://issues.apache.org/jira/browse/OAK-4138 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-next >Reporter: Michael Dürig > Labels: resilience > Fix For: 1.6 > > > I suggest we decouple revision cleanup from the flush thread. With large > repositories where cleanup can take several minutes to complete it blocks the > flush thread from updating the journal and the persisted head thus resulting > in larger then necessary data loss in case of a crash. > /cc [~alex.parvulescu] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4201) Add an index of binary references in a tar file
[
https://issues.apache.org/jira/browse/OAK-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4201:
---
Component/s: (was: segmentmk)
segment-next
> Add an index of binary references in a tar file
> ---
>
> Key: OAK-4201
> URL: https://issues.apache.org/jira/browse/OAK-4201
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-next
>Reporter: Chetan Mehrotra
> Fix For: 1.6
>
>
> Currently for Blob GC in case of segment {{SegmentBlobReferenceRetriever}}
> goes through all tar files and extracts the binary references. This has 2
> issues
> # Logic has go through all the segments in all tar files
> # All segments get loaded in memory once which would affect normal system
> performance
> This process can be optimized if we also write a file entry in tar (similar
> to gph i.e. graph and idx i.e. index files) which has entries of all binary
> references referred to in any segment present in that tar file. Then GC logic
> would just have read this file and avoid scanning all the segments
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4246) Update segment tooling to choose target store
[
https://issues.apache.org/jira/browse/OAK-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4246:
---
Component/s: (was: segmentmk)
segment-next
> Update segment tooling to choose target store
> -
>
> Key: OAK-4246
> URL: https://issues.apache.org/jira/browse/OAK-4246
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-next
>Reporter: Michael Dürig
> Fix For: 1.6
>
>
> We need to add command line options segment specific tooling so users could
> chose between {{oak-segment}} and {{oak-segment-next}}. {{oak-segment}}
> should be the default until deprecated, where {{oak-segment-next}} should be
> made the default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-1905) SegmentMK: Arch segment(s)
[ https://issues.apache.org/jira/browse/OAK-1905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-1905: --- Component/s: (was: segmentmk) (was: core) segment-next > SegmentMK: Arch segment(s) > -- > > Key: OAK-1905 > URL: https://issues.apache.org/jira/browse/OAK-1905 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-next >Reporter: Jukka Zitting >Priority: Minor > > There are a lot of constants and other commonly occurring name, values and > other data in a typical repository. To optimize storage space and access > speed, it would be useful to place such data in one or more constant "arch > segments" that are always cached in memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2113) TarMK cold standby: ensure client integrity
[ https://issues.apache.org/jira/browse/OAK-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2113: --- Component/s: (was: segmentmk) segment-next > TarMK cold standby: ensure client integrity > > > Key: OAK-2113 > URL: https://issues.apache.org/jira/browse/OAK-2113 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-next >Reporter: Manfred Baedke >Priority: Minor > > TarMK cold standby needs measures to ensure the integrity of each segment on > the slave and that all segments that are reachable on the master from a given > segment are available on the slave. > To ensure the integrity of a given segment on the slave, we can just use the > checksum stored in the segment, so there is no need to change the > communication protocol for this. > To ensure that all segments that are reachable from a given segment are the > same on the master and on the slave, we need a new request to calculate a > suitable checksum on the master and send it back to the slave. > If missing or broken segments are detected, the slave will pull them from the > master again. > Both measures combined should be scheduled to run regularly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4165) Too verbose logging of forward references in FileStore.cleanup
[
https://issues.apache.org/jira/browse/OAK-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4165:
---
Component/s: (was: segmentmk)
segment-next
> Too verbose logging of forward references in FileStore.cleanup
> --
>
> Key: OAK-4165
> URL: https://issues.apache.org/jira/browse/OAK-4165
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: cleanup, gc, logging
> Fix For: 1.6
>
> Attachments: OAK_4165.patch
>
>
> {{FileStore.cleanup}} logs the segment id of any forward reference found when
> including those in the reference graph. The logged information can amount to
> several MBs impacting normal operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4211) FileAccess.Mapped leaks file channels
[
https://issues.apache.org/jira/browse/OAK-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4211:
---
Component/s: (was: segmentmk)
segment-next
> FileAccess.Mapped leaks file channels
> --
>
> Key: OAK-4211
> URL: https://issues.apache.org/jira/browse/OAK-4211
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Michael Dürig
> Fix For: 1.6
>
> Attachments: OAK-4211.patch
>
>
> {{FileAccess.Mapped}} seems to leak {{FileChannnel}} instances. The file
> channels are acquired in the constructor but never release.
> FYI [~alex.parvulescu], [~frm]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2795) TarMK revision cleanup message too verbose.
[
https://issues.apache.org/jira/browse/OAK-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-2795:
---
Component/s: (was: segmentmk)
segment-next
> TarMK revision cleanup message too verbose.
> ---
>
> Key: OAK-2795
> URL: https://issues.apache.org/jira/browse/OAK-2795
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-next
>Reporter: Zygmunt Wiercioch
>Priority: Minor
> Labels: gc
>
> The segment GC message can be thousands of lines long on a system which
> experienced a lot of activity. For example, in my test, I had:
> {noformat}
> org.apache.jackrabbit.oak.plugins.segment.file.TarReader-GC Cleaned segments
> from data8a.tar:
> 4b6882e7-babe-4854-a966-c3a630c338c6, 506a2eb3-92a6-4b83-a320-5cc5ac304302,
> f179e4b1-acec-4a5d-a686-bf5e97828563, 8c6b3a4d-c327-4701-affb-fcd055c4aa67,
> {noformat}
> followed by approximately 40,000 more lines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3602) Remove the SegmentWriter segmentId from the cleanup set
[
https://issues.apache.org/jira/browse/OAK-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-3602.
Resolution: Won't Fix
This is fixed in {{oak-segment-next}} by the new cleanup strategy implemented
within OAK-3348.
> Remove the SegmentWriter segmentId from the cleanup set
> ---
>
> Key: OAK-3602
> URL: https://issues.apache.org/jira/browse/OAK-3602
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segmentmk
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
>Priority: Minor
> Labels: cleanup, gc
>
> It looks like the current head's segment id (coming from the SegmentWriter)
> is always passed to the cleanup's referenced segment ids set, even though it
> cannot be cleaned (similar to the TarWriter situation) so I propose removing
> it early from the mentioned set.
> benefits include making the cleanup set more stable (head changes quite often
> so the cleanup set is more volatile) which will help in figuring out if we
> still need to clean a specific tar file or not based on previous cleanup runs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4257) Findbug issues (mostly NPEs)
[ https://issues.apache.org/jira/browse/OAK-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alfusainey Jallow updated OAK-4257: --- Attachment: findbugs-trivial-issues.patch > Findbug issues (mostly NPEs) > > > Key: OAK-4257 > URL: https://issues.apache.org/jira/browse/OAK-4257 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: auth-external, authorization-cug, remoting, security, > upgrade >Reporter: Alfusainey Jallow >Priority: Trivial > Attachments: findbugs-trivial-issues.patch > > > Sub task addressing trivial issues reported by the findbugs static analyzer. > the issues addressed are mostly NPEs plus a few in oak-security -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4106) Reclaimed size reported by FileStore.cleanup is off
[ https://issues.apache.org/jira/browse/OAK-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-4106: --- Component/s: (was: segmentmk) segment-next > Reclaimed size reported by FileStore.cleanup is off > --- > > Key: OAK-4106 > URL: https://issues.apache.org/jira/browse/OAK-4106 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-next >Reporter: Michael Dürig >Priority: Minor > Labels: cleanup, gc > Fix For: 1.6 > > > The current implementation simply reports the difference between the > repository size before cleanup to the size after cleanup. As cleanup runs > concurrently to other commits, the size increase contributed by those is not > accounted for. In the extreme case where cleanup cannot reclaim anything this > can even result in negative values being reported. > We should either change the wording of the respective log message and speak > of before and after sizes or adjust our calculation of reclaimed size > (preferred). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4097) Add metric for FileStore journal writes
[ https://issues.apache.org/jira/browse/OAK-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-4097: --- Component/s: (was: segmentmk) segment-next > Add metric for FileStore journal writes > --- > > Key: OAK-4097 > URL: https://issues.apache.org/jira/browse/OAK-4097 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-next >Reporter: Chetan Mehrotra >Assignee: Chetan Mehrotra >Priority: Minor > Fix For: 1.6 > > > TarMK flush thread should run every 5 secs and flush the current root head to > journal.log. It would be good to have a metric to capture the number of runs > per minute > This would help in confirming if flush is working at expected frequency or > delay in acquiring locks is causing some delays -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4127) Cleanup creates new generation of tar file without removing any segments
[ https://issues.apache.org/jira/browse/OAK-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-4127: --- Component/s: (was: segmentmk) segment-next > Cleanup creates new generation of tar file without removing any segments > - > > Key: OAK-4127 > URL: https://issues.apache.org/jira/browse/OAK-4127 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-next >Reporter: Michael Dürig >Priority: Minor > Labels: cleanup, gc > Fix For: 1.6 > > > On some deployments I have seen tar files with a quite hight generation > post-fix (e.g. 'v'). From the log files I could deduce that this particular > tar file was rewritten multiple times without actually any segment being > removed. > I assume this is caused by the 25% [gain threshold | > https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/TarReader.java#L789] > not taking the sizes contributed by the index and the graph entries into > account. > We should try to come up with a test case validating above hypothesis. A fix > should then be relatively straight forward: either include the sizes of these > two entries in the calculation or skip further clean cycles if a file size > drops below a certain size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-4202) leakage of temp files
[ https://issues.apache.org/jira/browse/OAK-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249830#comment-15249830 ] Vikas Saurabh commented on OAK-4202: [~reschke], this should be cleaned up fine now. > leakage of temp files > - > > Key: OAK-4202 > URL: https://issues.apache.org/jira/browse/OAK-4202 > Project: Jackrabbit Oak > Issue Type: Task >Reporter: Julian Reschke >Priority: Minor > Attachments: leftovers-1739224.txt, leftovers-1739267.txt, > leftovers-1739428.txt, leftovers-1739696.txt, leftovers-1739725.txt, > leftovers-1739889.txt, leftovers-1739902.txt, leftovers-1739931.txt, > leftovers-1739958.txt, leftovers.txt > > > Running the tests currently leaves a ton of temporary files. I assume in > general these are caused by bugs in test cases, but they could also indicate > problems in the actual code. > Will attach a detailed list separately and open sub-tasks accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-4236) SegmentNodeStoreConfigTest#testDeadlock in oak-pojosr leaves out tmp file
[
https://issues.apache.org/jira/browse/OAK-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh resolved OAK-4236.
Resolution: Fixed
Fix Version/s: 1.5.2
Committed [r1740140|https://svn.apache.org/r1740140] to have non-mmap seg store
in both configs.
> SegmentNodeStoreConfigTest#testDeadlock in oak-pojosr leaves out tmp file
> -
>
> Key: OAK-4236
> URL: https://issues.apache.org/jira/browse/OAK-4236
> Project: Jackrabbit Oak
> Issue Type: Technical task
>Reporter: Vikas Saurabh
>Assignee: Chetan Mehrotra
>Priority: Minor
> Fix For: 1.5.2
>
> Attachments: unit-tests.log
>
>
> Leftover temp folder contains {{repository/segmentstore/data0a.tar}}...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4146) Improve tarmkrecovery docs
[
https://issues.apache.org/jira/browse/OAK-4146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4146:
---
Component/s: segment-next
> Improve tarmkrecovery docs
> --
>
> Key: OAK-4146
> URL: https://issues.apache.org/jira/browse/OAK-4146
> Project: Jackrabbit Oak
> Issue Type: Documentation
> Components: run, segment-next, segmentmk
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
>Priority: Minor
>
> Add some helper steps on output and what you can actually do with it:
> {quote}
> 1. Run tarmkrecovery command
> {code:none}
> nohup java -Xmx2048m -jar oak-run-*.jar tarmkrecovery repository/segmentstore
> &> tarmkrecovery.log &
> {code}
> 2. Take the output of the tarmkrecovery, take the top 10 items output
> (excluding "Current head revision line") then reverse the order of those and
> format them to journal.log file format (revision:offset root) and put those
> values in a fresh journal.log in that format
> For example:
> {code:none}
> 6ee64a26-491e-4630-ac2e-bdad1f27e73a:257016 root
> 5ee64a26-491e-4630-ac2e-bdad1f27e73b:257111 root
> {code}
> 3. After setting up the new journal.log then run this command on the
> segmentstore
> {code:none}
> nohup java -Xmx2048m -jar oak-run-*.jar check -p repository/segmentstore -d
> &> check.log &
> {code}
> 4. That command will give you output of which of those 10 items in the
> journal.log are good. Now remove all lines from the journal that come after
> the last known good revision.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-1553) More sophisticated conflict resolution when concurrently adding nodes
[
https://issues.apache.org/jira/browse/OAK-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-1553:
---
Component/s: (was: segmentmk)
segment-next
> More sophisticated conflict resolution when concurrently adding nodes
> -
>
> Key: OAK-1553
> URL: https://issues.apache.org/jira/browse/OAK-1553
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk, segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: concurrency, technical_debt
> Attachments: OAK-1553.patch
>
>
> {{MicroKernel.rebase}} currently specifies: "addExistingNode: A node has been
> added that is different from a node of them same name that has been added to
> the trunk."
> This is somewhat troublesome in the case where the same node with different
> but non conflicting child items is added concurrently:
> {code}
> f.add("fo").add("u1"); commit();
> f.add("fo").add("u2"); commit();
> {code}
> currently fails with a conflict because {{fo}} is not the same node for the
> both cases. See discussion http://markmail.org/message/flst4eiqvbp4gi3z
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-4257) Findbug issues (mostly NPEs)
Alfusainey Jallow created OAK-4257: -- Summary: Findbug issues (mostly NPEs) Key: OAK-4257 URL: https://issues.apache.org/jira/browse/OAK-4257 Project: Jackrabbit Oak Issue Type: Technical task Components: auth-external, authorization-cug, remoting, security, upgrade Reporter: Alfusainey Jallow Priority: Trivial Sub task addressing trivial issues reported by the findbugs static analyzer. the issues addressed are mostly NPEs plus a few in oak-security -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-1558) Expose FileStoreBackupRestoreMBean for supported NodeStores
[
https://issues.apache.org/jira/browse/OAK-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-1558:
---
Component/s: (was: segmentmk)
segment-next
> Expose FileStoreBackupRestoreMBean for supported NodeStores
> ---
>
> Key: OAK-1558
> URL: https://issues.apache.org/jira/browse/OAK-1558
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: mongomk, segment-next
>Reporter: Michael Dürig
> Labels: monitoring
>
> {{NodeStore}} implementations should expose the
> {{FileStoreBackupRestoreMBean}} in order to be interoperable with
> {{RepositoryManagementMBean}}. See OAK-1160.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-3753) Test failure: HeavyWriteIT
[
https://issues.apache.org/jira/browse/OAK-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-3753.
Resolution: Cannot Reproduce
Fix Version/s: (was: 1.6)
> Test failure: HeavyWriteIT
> --
>
> Key: OAK-3753
> URL: https://issues.apache.org/jira/browse/OAK-3753
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
> Environment: Jenkins, Ubuntu:
> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Critical
> Labels: ci, jenkins
>
> {{org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT}} failed on Jenkins:
> {noformat}
> heavyWrite[usePersistedMap:
> false](org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT) Time elapsed:
> 106.519 sec <<< ERROR!
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:134)
> at
> org.apache.jackrabbit.oak.plugins.segment.Segment.(Segment.java:214)
> at
> org.apache.jackrabbit.oak.plugins.segment.Segment.(Segment.java:198)
> at
> org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:1177)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.readSegment(SegmentTracker.java:224)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:149)
> at
> org.apache.jackrabbit.oak.plugins.segment.RecordId.getSegment(RecordId.java:88)
> at
> org.apache.jackrabbit.oak.plugins.segment.Segment.readTemplate(Segment.java:506)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getTemplate(SegmentNodeState.java:79)
> at
> org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getChildNode(SegmentNodeState.java:381)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$UnconnectedHead.update(MemoryNodeBuilder.java:651)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$ConnectedHead.update(MemoryNodeBuilder.java:729)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.head(MemoryNodeBuilder.java:171)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.access$300(MemoryNodeBuilder.java:88)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$UnconnectedHead.update(MemoryNodeBuilder.java:650)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder$ConnectedHead.update(MemoryNodeBuilder.java:729)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.head(MemoryNodeBuilder.java:171)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.exists(MemoryNodeBuilder.java:273)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setProperty(MemoryNodeBuilder.java:506)
> at
> org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setProperty(MemoryNodeBuilder.java:515)
> at
> org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT.createProperties(HeavyWriteIT.java:156)
> at
> org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT.createNodes(HeavyWriteIT.java:148)
> at
> org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT.createNodes(HeavyWriteIT.java:149)
> at
> org.apache.jackrabbit.oak.plugins.segment.HeavyWriteIT.heavyWrite(HeavyWriteIT.java:129)
> {noformat}
> Seen at build 597
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2896) Putting many elements into a map results in many small segments.
[
https://issues.apache.org/jira/browse/OAK-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-2896:
---
Component/s: (was: segmentmk)
segment-next
> Putting many elements into a map results in many small segments.
> -
>
> Key: OAK-2896
> URL: https://issues.apache.org/jira/browse/OAK-2896
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Critical
> Labels: performance
> Attachments: OAK-2896.png, OAK-2896.xlsx, size-dist.png
>
>
> There is an issue with how the HAMT implementation
> ({{SegmentWriter.writeMap()}} interacts with the 256 segment references limit
> when putting many entries into the map: This limit gets regularly reached
> once the maps contains about 200k entries. At that points segments get
> prematurely flushed resulting in more segments, thus more references and thus
> even smaller segments. It is common for segments to be as small as 7k with a
> tar file containing up to 35k segments. This is problematic as at this point
> handling of the segment graph becomes expensive, both memory and CPU wise. I
> have seen persisted segment graphs as big as 35M where the usual size is a
> couple of ks.
> As the HAMT map is used for storing children of a node this might have an
> advert effect on nodes with many child nodes.
> The following code can be used to reproduce the issue:
> {code}
> SegmentWriter writer = new SegmentWriter(segmentStore, getTracker(), V_11);
> MapRecord baseMap = null;
> for (;;) {
> Map map = newHashMap();
> for (int k = 0; k < 1000; k++) {
> RecordId stringId =
> writer.writeString(String.valueOf(rnd.nextLong()));
> map.put(String.valueOf(rnd.nextLong()), stringId);
> }
> Stopwatch w = Stopwatch.createStarted();
> baseMap = writer.writeMap(baseMap, map);
> System.out.println(baseMap.size() + " " + w.elapsed());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4247) Deprecate oak-segment
[
https://issues.apache.org/jira/browse/OAK-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4247:
---
Component/s: segment-next
> Deprecate oak-segment
> -
>
> Key: OAK-4247
> URL: https://issues.apache.org/jira/browse/OAK-4247
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-next, segmentmk
>Reporter: Michael Dürig
>Priority: Blocker
> Fix For: 1.6
>
>
> Before the next major release we need to deprecate {{oak-segment}} and make
> {{oak-segment-next}} the new default implementation:
> * Deprecate all classes in {{oak-segment}}
> * Update documentation to reflect this change
> * Update tooling to target {{oak-segment-next}} (See OAK-4246).
> * Update dependencies of upstream modules / projects from {{oak-segment}} to
> {{oak-segment-next}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3893) SegmentWriter records cache could use thinner keys
[
https://issues.apache.org/jira/browse/OAK-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-3893:
---
Component/s: (was: segmentmk)
segment-next
> SegmentWriter records cache could use thinner keys
> --
>
> Key: OAK-3893
> URL: https://issues.apache.org/jira/browse/OAK-3893
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: segment-next
>Reporter: Alex Parvulescu
>Assignee: Alex Parvulescu
>Priority: Minor
> Attachments: OAK-3893.patch
>
>
> The SegmentWriter keeps a records deduplication cache ('records' map) that
> maintains 2 types of mappings:
> * template -> recordid
> * strings -> recordid
> For the first one (template-> recordid) we can come up with a thinner
> representation of a template (a hash function that is fast and not very
> collision prone) so we don't have to keep a reference to each template object.
> Same applies for second one, similar to what is happening in the StringsCache
> now, we could keep the string value up to a certain size and beyond that,
> hash it and use that for the deduplication map.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3468) Replace BackgroundThread with Scheduler
[ https://issues.apache.org/jira/browse/OAK-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3468: --- Component/s: (was: segmentmk) segment-next > Replace BackgroundThread with Scheduler > --- > > Key: OAK-3468 > URL: https://issues.apache.org/jira/browse/OAK-3468 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: segment-next >Reporter: Michael Dürig > Labels: technical_debt > > I think we should replace the background thread with some kind of a > scheduler. The goal would be to decouple threading from scheduling. IMO > threads should not be managed by the application but by the container. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3690) Decouple SegmentBufferWriter from SegmentStore
[
https://issues.apache.org/jira/browse/OAK-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-3690:
---
Component/s: (was: segmentmk)
segment-next
> Decouple SegmentBufferWriter from SegmentStore
> --
>
> Key: OAK-3690
> URL: https://issues.apache.org/jira/browse/OAK-3690
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: segment-next
>Reporter: Michael Dürig
> Labels: technical_debt
>
> Currently {{SegmentBufferWriter.flush()}} directly calls
> {{SegmentStore.writeSegment()}} once the current segment does not have enough
> space for the next record. We should try to cut this dependency as
> {{SegmentBufferWriter}} should only be concerned with providing buffers for
> segments. Actually writing these to the store should be handled by a higher
> level component.
> A number of deadlock (e.g. (OAK-2560, OAK-3179, OAK-3264) we have seen is one
> manifestation of this troublesome dependency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4102) Break cyclic dependency of FileStore and SegmentTracker
[
https://issues.apache.org/jira/browse/OAK-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4102:
---
Component/s: (was: segmentmk)
segment-next
> Break cyclic dependency of FileStore and SegmentTracker
> ---
>
> Key: OAK-4102
> URL: https://issues.apache.org/jira/browse/OAK-4102
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: segment-next
>Reporter: Michael Dürig
> Labels: technical_debt
> Fix For: 1.6
>
>
> {{SegmentTracker}} and {{FileStore}} are mutually dependent on each other.
> This is problematic and makes initialising instances of these classes
> difficult: the {{FileStore}} constructor e.g. passes a not fully initialised
> instance to the {{SegmentTracker}}, which in turn writes an initial node
> state to the store. Notably using the not fully initialised {{FileStore}}
> instance!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4103) Replace journal.log with an in place journal
[
https://issues.apache.org/jira/browse/OAK-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4103:
---
Component/s: (was: segmentmk)
segment-next
> Replace journal.log with an in place journal
>
>
> Key: OAK-4103
> URL: https://issues.apache.org/jira/browse/OAK-4103
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: segment-next
>Reporter: Michael Dürig
>Priority: Minor
> Labels: resilience
> Fix For: 1.6
>
>
> Instead of writing the current head revision to the {{journal.log}} file we
> could make it an integral part of the node states: as OAK-3804 demonstrates
> we already have very good heuristics to reconstruct a lost journal. If we add
> the right annotations to the root node states this could replace the current
> approach. The latter is problematic as it relies on the flush thread properly
> and timely updating {{journal.log}}. See e.g. OAK-3303.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-3869) Refactor RecordWriter.write to always return a RecordId
[
https://issues.apache.org/jira/browse/OAK-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-3869:
---
Component/s: (was: segmentmk)
segment-next
> Refactor RecordWriter.write to always return a RecordId
> ---
>
> Key: OAK-3869
> URL: https://issues.apache.org/jira/browse/OAK-3869
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: technical_debt
>
> I think it would be cleaner if {{RecordId.write}} would always return a
> {{RecordId}} instead of depending on its type parametrisation and would like
> to refactor it to that respect..
> This is also a pre-requisite for my work on OAK-3348 and might also be for
> OAK-3864.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4105) Implement FileStore.size through FileStore.approximateSize
[
https://issues.apache.org/jira/browse/OAK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4105:
---
Component/s: (was: segmentmk)
segment-next
> Implement FileStore.size through FileStore.approximateSize
> --
>
> Key: OAK-4105
> URL: https://issues.apache.org/jira/browse/OAK-4105
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: segment-next
>Reporter: Michael Dürig
>Priority: Minor
> Labels: resilience
> Fix For: 1.6
>
>
> {{FileStore.size()}} is prone to lock contention and should not be called too
> often. As OAK-2879 already introduced an approach for tracking the current
> size of the file store without having to lock, we might as well promote his
> to be "the official" implementation.
> [~frm] WDYT?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2833) Refactor TarMK
[ https://issues.apache.org/jira/browse/OAK-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2833: --- Component/s: (was: segmentmk) segment-next > Refactor TarMK > -- > > Key: OAK-2833 > URL: https://issues.apache.org/jira/browse/OAK-2833 > Project: Jackrabbit Oak > Issue Type: Task > Components: segment-next >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: technical_debt > > Container issue for refactoring the TarMK to make it more testable, > maintainable, extensible and less entangled. > For example the segment format should be readable, writeable through > standalone means so tests, tools and production code can share this code. > Currently there is a lot of code duplication involved here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-1879) Store record offset as short
[ https://issues.apache.org/jira/browse/OAK-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-1879: --- Component/s: (was: segmentmk) segment-next > Store record offset as short > > > Key: OAK-1879 > URL: https://issues.apache.org/jira/browse/OAK-1879 > Project: Jackrabbit Oak > Issue Type: Technical task > Components: segment-next >Reporter: Alex Parvulescu >Assignee: Alex Parvulescu > > The offset value is actually a short but the RecordId keeps it as an int. > This is a leftover from the initial design and it would be beneficial to > refactor those parts of the code to use a short instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4104) Refactor reading records from segments
[
https://issues.apache.org/jira/browse/OAK-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4104:
---
Component/s: (was: segmentmk)
segment-next
> Refactor reading records from segments
> --
>
> Key: OAK-4104
> URL: https://issues.apache.org/jira/browse/OAK-4104
> Project: Jackrabbit Oak
> Issue Type: Technical task
> Components: segment-next
>Reporter: Michael Dürig
> Labels: technical_debt
> Fix For: 1.6
>
>
> We should refactor how records (e.g. node states) are read from segments.
> Currently this is scattered and replicated across various places. All of
> which hard coding certain indexes into a byte buffer (see calls to
> {{Record.getOffset}} for how bad this is).
> The current implementation makes it very hard to maintain the code and evolve
> the segment format. We should optimally have one place per segment version
> defining the format as a single source of truth which is then reused by the
> various parts in of the SegmentMK, tooling and tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4243) Oak Segment Next Module
[
https://issues.apache.org/jira/browse/OAK-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4243:
---
Component/s: (was: segmentmk)
segment-next
> Oak Segment Next Module
> ---
>
> Key: OAK-4243
> URL: https://issues.apache.org/jira/browse/OAK-4243
> Project: Jackrabbit Oak
> Issue Type: Epic
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Fix For: 1.6
>
>
> There is a couple of issues requiring us to change the segment format in a
> non compatible way (OAK-3348, OAK-2896, OAK-2498, OAK-4201).
> We should introduce a new module here
> * to minimise ripple effect on concurrent development work in other parts of
> Oak and upstream projects
> * to be able to cleanly migrate existing repositories via a side grading
> * to cleanly separate breaking changes from the existing code base
> The plan is roughly to:
> * Create new module (called {{oak-segment-next}} for now, will discuss names
> later)
> * Apply patch prepared for OAK-3348
> * Discuss and decide on final name and refactor accordingly
> * Reactor affected tooling such that the targeted segment store can be
> specified via an option. Keep default at {{oak-segment}}.
> * Once sufficiently stabilised, deprecate {{oak-segment}}. Make this one the
> default in switch the default target for tooling.
> * Define and implement migration path
> I will create respective issues as needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4236) SegmentNodeStoreConfigTest#testDeadlock in oak-pojosr leaves out tmp file
[
https://issues.apache.org/jira/browse/OAK-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Saurabh updated OAK-4236:
---
Attachment: unit-tests.log
Attaching [^unit-tests.log]. There are few observations:
* File store gets closed timely
* Delete fails even if I pause execution atTempFolder.delete
Making second SegmentNodeStore's as to have non-mmap access allows to clean up
nicely. So, both the following diffs get the cleanup through:
{code}
//which first wait for nodeStoreLatch and then on NodeStoreTracker lock
createConfig([
'org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService': [
-cache: 200
+cache: 200,
+"tarmk.mode": 32
]
])
{code}
OR
{code}
//1. Get NodeStore created
createConfig([
'org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService': [
-cache: 256
+cache: 256,
+"tarmk.mode": 32
]
])
getServiceWithWait(NodeStore.class)
//which first wait for nodeStoreLatch and then on NodeStoreTracker lock
createConfig([
'org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService': [
-cache: 200
+cache: 200,
+"tarmk.mode": 32
]
])
{code}
BTW, have file access for first config and mmap access for second _doesn't_
work:
{code}
//1. Get NodeStore created
createConfig([
'org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService': [
-cache: 256
+cache: 256,
+"tarmk.mode": 32
]
])
getServiceWithWait(NodeStore.class)
{code}
That indicated that quite possibly the issues is with locked mmapped file
access. I tried 2 System.gc() at the end of method to force clean-up of weak
refs if any... but that didn't work as well
{code}
assertNull("Deadlock detected",
ManagementFactory.getThreadMXBean().findDeadlockedThreads())
allWellLatch.await()
tracker.close()
+
+System.gc();
+System.gc();
}
@Override
{code}
> SegmentNodeStoreConfigTest#testDeadlock in oak-pojosr leaves out tmp file
> -
>
> Key: OAK-4236
> URL: https://issues.apache.org/jira/browse/OAK-4236
> Project: Jackrabbit Oak
> Issue Type: Technical task
>Reporter: Vikas Saurabh
>Assignee: Chetan Mehrotra
>Priority: Minor
> Attachments: unit-tests.log
>
>
> Leftover temp folder contains {{repository/segmentstore/data0a.tar}}...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4256) CLONE - Cross gc sessions might introduce references to pre-compacted segments
[ https://issues.apache.org/jira/browse/OAK-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-4256: --- Component/s: (was: segment-next) segmentmk > CLONE - Cross gc sessions might introduce references to pre-compacted segments > -- > > Key: OAK-4256 > URL: https://issues.apache.org/jira/browse/OAK-4256 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segmentmk >Reporter: Michael Dürig >Assignee: Michael Dürig >Priority: Critical > Labels: cleanup, compaction, gc > Attachments: OAK-3348-1.patch, OAK-3348-2.patch, OAK-3348.patch, > SCIT.patch, cleanup-time.png, compaction-time.png, cross-gc-refs.pdf, > image.png, repo-size.png > > > I suspect that certain write operations during compaction can cause > references from compacted segments to pre-compacted ones. This would > effectively prevent the pre-compacted segments from getting evicted in > subsequent cleanup phases. > The scenario is as follows: > * A session is opened and a lot of content is written to it such that the > update limit is exceeded. This causes the changes to be written to disk. > * Revision gc runs causing a new, compacted root node state to be written to > disk. > * The session saves its changes. This causes rebasing of its changes onto the > current root (the compacted one). At this point any node that has been added > will be added again in the sub-tree rooted at the current root. Such nodes > however might have been written to disk *before* revision gc ran and might > thus be contained in pre-compacted segments. As I suspect the node-add > operation in the rebasing process *not* to create a deep copy of such nodes > but to rather create a *reference* to them, a reference to a pre-compacted > segment is introduced here. > Going forward we need to validate above hypothesis, assess its impact if > necessary come up with a solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (OAK-4256) CLONE - Cross gc sessions might introduce references to pre-compacted segments
[
https://issues.apache.org/jira/browse/OAK-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-4256.
Resolution: Won't Fix
Fix Version/s: (was: 1.5.2)
(was: 1.6)
Fix will only go into {{oak-segment-next}}
> CLONE - Cross gc sessions might introduce references to pre-compacted segments
> --
>
> Key: OAK-4256
> URL: https://issues.apache.org/jira/browse/OAK-4256
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Critical
> Labels: cleanup, compaction, gc
> Attachments: OAK-3348-1.patch, OAK-3348-2.patch, OAK-3348.patch,
> SCIT.patch, cleanup-time.png, compaction-time.png, cross-gc-refs.pdf,
> image.png, repo-size.png
>
>
> I suspect that certain write operations during compaction can cause
> references from compacted segments to pre-compacted ones. This would
> effectively prevent the pre-compacted segments from getting evicted in
> subsequent cleanup phases.
> The scenario is as follows:
> * A session is opened and a lot of content is written to it such that the
> update limit is exceeded. This causes the changes to be written to disk.
> * Revision gc runs causing a new, compacted root node state to be written to
> disk.
> * The session saves its changes. This causes rebasing of its changes onto the
> current root (the compacted one). At this point any node that has been added
> will be added again in the sub-tree rooted at the current root. Such nodes
> however might have been written to disk *before* revision gc ran and might
> thus be contained in pre-compacted segments. As I suspect the node-add
> operation in the rebasing process *not* to create a deep copy of such nodes
> but to rather create a *reference* to them, a reference to a pre-compacted
> segment is introduced here.
> Going forward we need to validate above hypothesis, assess its impact if
> necessary come up with a solution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-4256) CLONE - Cross gc sessions might introduce references to pre-compacted segments
Michael Dürig created OAK-4256: -- Summary: CLONE - Cross gc sessions might introduce references to pre-compacted segments Key: OAK-4256 URL: https://issues.apache.org/jira/browse/OAK-4256 Project: Jackrabbit Oak Issue Type: Improvement Components: segment-next Reporter: Michael Dürig Assignee: Michael Dürig Priority: Critical Fix For: 1.6, 1.5.2 Attachments: OAK-3348-1.patch, OAK-3348-2.patch, OAK-3348.patch, SCIT.patch, cleanup-time.png, compaction-time.png, cross-gc-refs.pdf, image.png, repo-size.png I suspect that certain write operations during compaction can cause references from compacted segments to pre-compacted ones. This would effectively prevent the pre-compacted segments from getting evicted in subsequent cleanup phases. The scenario is as follows: * A session is opened and a lot of content is written to it such that the update limit is exceeded. This causes the changes to be written to disk. * Revision gc runs causing a new, compacted root node state to be written to disk. * The session saves its changes. This causes rebasing of its changes onto the current root (the compacted one). At this point any node that has been added will be added again in the sub-tree rooted at the current root. Such nodes however might have been written to disk *before* revision gc ran and might thus be contained in pre-compacted segments. As I suspect the node-add operation in the rebasing process *not* to create a deep copy of such nodes but to rather create a *reference* to them, a reference to a pre-compacted segment is introduced here. Going forward we need to validate above hypothesis, assess its impact if necessary come up with a solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-3348) Cross gc sessions might introduce references to pre-compacted segments
[ https://issues.apache.org/jira/browse/OAK-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-3348: --- Component/s: (was: segmentmk) segment-next > Cross gc sessions might introduce references to pre-compacted segments > -- > > Key: OAK-3348 > URL: https://issues.apache.org/jira/browse/OAK-3348 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: segment-next >Reporter: Michael Dürig >Assignee: Michael Dürig >Priority: Critical > Labels: cleanup, compaction, gc > Fix For: 1.6, 1.5.2 > > Attachments: OAK-3348-1.patch, OAK-3348-2.patch, OAK-3348.patch, > SCIT.patch, cleanup-time.png, compaction-time.png, cross-gc-refs.pdf, > image.png, repo-size.png > > > I suspect that certain write operations during compaction can cause > references from compacted segments to pre-compacted ones. This would > effectively prevent the pre-compacted segments from getting evicted in > subsequent cleanup phases. > The scenario is as follows: > * A session is opened and a lot of content is written to it such that the > update limit is exceeded. This causes the changes to be written to disk. > * Revision gc runs causing a new, compacted root node state to be written to > disk. > * The session saves its changes. This causes rebasing of its changes onto the > current root (the compacted one). At this point any node that has been added > will be added again in the sub-tree rooted at the current root. Such nodes > however might have been written to disk *before* revision gc ran and might > thus be contained in pre-compacted segments. As I suspect the node-add > operation in the rebasing process *not* to create a deep copy of such nodes > but to rather create a *reference* to them, a reference to a pre-compacted > segment is introduced here. > Going forward we need to validate above hypothesis, assess its impact if > necessary come up with a solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-4244) Create new module oak-segment-next
[
https://issues.apache.org/jira/browse/OAK-4244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4244:
---
Component/s: (was: segmentmk)
segment-next
> Create new module oak-segment-next
> --
>
> Key: OAK-4244
> URL: https://issues.apache.org/jira/browse/OAK-4244
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Fix For: 1.6
>
>
> Create a new module {{oak-segment-next}} starting from a copy of
> {{oak-segment}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4245) Decide on a final name for oak-segment-next
[
https://issues.apache.org/jira/browse/OAK-4245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4245:
---
Component/s: (was: segmentmk)
segment-next
> Decide on a final name for oak-segment-next
> ---
>
> Key: OAK-4245
> URL: https://issues.apache.org/jira/browse/OAK-4245
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Blocker
> Fix For: 1.5.2
>
>
> We should come up with a definite and final name for {{oak-segment-name}}.
> Making this a blocker to avoid releasing with the current WIP name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-4244) Create new module oak-segment-next
[
https://issues.apache.org/jira/browse/OAK-4244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-4244.
Resolution: Fixed
Fix Version/s: (was: 1.5.2)
1.6
Fixed at http://svn.apache.org/viewvc?rev=1740107=rev
> Create new module oak-segment-next
> --
>
> Key: OAK-4244
> URL: https://issues.apache.org/jira/browse/OAK-4244
> Project: Jackrabbit Oak
> Issue Type: Task
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Fix For: 1.6
>
>
> Create a new module {{oak-segment-next}} starting from a copy of
> {{oak-segment}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-4255) CLONE - FileStore.containsSegment returns alway true (almost)
[
https://issues.apache.org/jira/browse/OAK-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-4255.
Resolution: Fixed
Fixed at http://svn.apache.org/viewvc?rev=1740139=rev
> CLONE - FileStore.containsSegment returns alway true (almost)
> -
>
> Key: OAK-4255
> URL: https://issues.apache.org/jira/browse/OAK-4255
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Critical
> Labels: compaction, gc, stability
> Fix For: 1.6
>
>
> {{FileStore.containsSegment()}} looks
> [funky|https://github.com/mduerig/jackrabbit-oak/blob/36cb3bf6e5078e3afa75581fb789eeca7b5df2e2/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java#L1197-L1197].
> This "optimisation" causes it to always return {{true}}.
> {{containsSegment}} is used for deduplication and revision gc. The current
> implementation causes {{SNFE}} exceptions once gc is effective (as I
> experienced while working on OAK-3348).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-4054) FileStore.containsSegment returns alway true (almost)
[
https://issues.apache.org/jira/browse/OAK-4054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-4054.
Resolution: Won't Fix
Fix Version/s: (was: 1.6)
Fix will only go into {{oak-segment-next}}
> FileStore.containsSegment returns alway true (almost)
> -
>
> Key: OAK-4054
> URL: https://issues.apache.org/jira/browse/OAK-4054
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Critical
> Labels: compaction, gc, stability
>
> {{FileStore.containsSegment()}} looks
> [funky|https://github.com/mduerig/jackrabbit-oak/blob/36cb3bf6e5078e3afa75581fb789eeca7b5df2e2/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java#L1197-L1197].
> This "optimisation" causes it to always return {{true}}.
> {{containsSegment}} is used for deduplication and revision gc. The current
> implementation causes {{SNFE}} exceptions once gc is effective (as I
> experienced while working on OAK-3348).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4255) CLONE - FileStore.containsSegment returns alway true (almost)
[
https://issues.apache.org/jira/browse/OAK-4255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4255:
---
Component/s: (was: segmentmk)
segment-next
> CLONE - FileStore.containsSegment returns alway true (almost)
> -
>
> Key: OAK-4255
> URL: https://issues.apache.org/jira/browse/OAK-4255
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
>Priority: Critical
> Labels: compaction, gc, stability
> Fix For: 1.6
>
>
> {{FileStore.containsSegment()}} looks
> [funky|https://github.com/mduerig/jackrabbit-oak/blob/36cb3bf6e5078e3afa75581fb789eeca7b5df2e2/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java#L1197-L1197].
> This "optimisation" causes it to always return {{true}}.
> {{containsSegment}} is used for deduplication and revision gc. The current
> implementation causes {{SNFE}} exceptions once gc is effective (as I
> experienced while working on OAK-3348).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4254) CLONE - BackgroundThread should log and re-throw instances of Error
[
https://issues.apache.org/jira/browse/OAK-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4254:
---
Component/s: (was: segmentmk)
segment-next
> CLONE - BackgroundThread should log and re-throw instances of Error
> ---
>
> Key: OAK-4254
> URL: https://issues.apache.org/jira/browse/OAK-4254
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4,
> resilience
> Fix For: 1.6
>
>
> If the run method of a {{BackgroundThread}} instance hits an {{Error}} it
> dies silently. Instead it should log an re-throw the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-4254) CLONE - BackgroundThread should log and re-throw instances of Error
[
https://issues.apache.org/jira/browse/OAK-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-4254.
Resolution: Fixed
Assignee: Michael Dürig
Fixed at http://svn.apache.org/viewvc?rev=1740099=rev
> CLONE - BackgroundThread should log and re-throw instances of Error
> ---
>
> Key: OAK-4254
> URL: https://issues.apache.org/jira/browse/OAK-4254
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4,
> resilience
> Fix For: 1.6
>
>
> If the run method of a {{BackgroundThread}} instance hits an {{Error}} it
> dies silently. Instead it should log an re-throw the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-4255) CLONE - FileStore.containsSegment returns alway true (almost)
Michael Dürig created OAK-4255:
--
Summary: CLONE - FileStore.containsSegment returns alway true
(almost)
Key: OAK-4255
URL: https://issues.apache.org/jira/browse/OAK-4255
Project: Jackrabbit Oak
Issue Type: Bug
Components: segmentmk
Reporter: Michael Dürig
Assignee: Michael Dürig
Priority: Critical
Fix For: 1.6
{{FileStore.containsSegment()}} looks
[funky|https://github.com/mduerig/jackrabbit-oak/blob/36cb3bf6e5078e3afa75581fb789eeca7b5df2e2/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java#L1197-L1197].
This "optimisation" causes it to always return {{true}}.
{{containsSegment}} is used for deduplication and revision gc. The current
implementation causes {{SNFE}} exceptions once gc is effective (as I
experienced while working on OAK-3348).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-4253) CLONE - TarReader#loadGraph wrongly detects segment graph as corrupt
[
https://issues.apache.org/jira/browse/OAK-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-4253:
---
Fix Version/s: 1.6
> CLONE - TarReader#loadGraph wrongly detects segment graph as corrupt
> -
>
> Key: OAK-4253
> URL: https://issues.apache.org/jira/browse/OAK-4253
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-next
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: cleanup, gc
> Fix For: 1.6
>
> Attachments: OAK-4147.patch
>
>
> {{org.apache.jackrabbit.oak.plugins.segment.file.TarReader#loadGraph}}
> sometimes detects a segment graph as corrupt although it isn't. This results
> in cleanup rewriting the tar file (all over again).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
