[jira] [Commented] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508450#comment-14508450
]
Chetan Mehrotra commented on OAK-2799:
--
Patch looks good. Can we also add some test around this that close is done
properly so that future refactoring honour the contract regarding clone handling
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2, 1.0.14
>
> Attachments: OAK-2799.0.patch
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2783) Make LDAP connection pool 'testOnBorrow' configurable
[
https://issues.apache.org/jira/browse/OAK-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tobias Bocanegra updated OAK-2783:
--
Fix Version/s: 1.0.14
> Make LDAP connection pool 'testOnBorrow' configurable
> -
>
> Key: OAK-2783
> URL: https://issues.apache.org/jira/browse/OAK-2783
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: auth-ldap
>Affects Versions: 1.2
>Reporter: Tobias Bocanegra
>Assignee: Tobias Bocanegra
>Priority: Minor
> Fix For: 1.3.0, 1.0.14
>
>
> Depending of the LDAP server configuration, it fails to connect as the server
> doesn't allow the connection validation query.
> It fails on
> {quote}
> Caused by: java.util.NoSuchElementException: Could not create a validated
> object, cause: ValidateObject failed
> at
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1233)
> at
> org.apache.directory.ldap.client.api.LdapConnectionPool.getConnection(LdapConnectionPool.java:56)
> at
> org.apache.jackrabbit.oak.security.authentication.ldap.impl.LdapIdentityProvider.connect(LdapIdentityProvider.java:532)
> ... 92 common frames omitted
> {quote}
> Based on customer analyze of Oak code this is the reason it fails:
> {quote}
> I think I have found a solution for the problem. While the system is
> initializing the connection it tries to validate the connection. This is the
> reason for the strange search request:
> SearchRequest
> baseDn : ''
> filter : '(objectClass=*)'
> scope : base object
> Because such kind of requests are not allowed in the client's ldap system the
> connection is being rejected (as invalid). It is configurable if the
> connection should be validated. The class
> org.apache.jackrabbit.oak.security.authentication.ldap.impl.LdapIdentityProvider
> contains this code
> if (config.getAdminPoolConfig().getMaxActive() != 0) {
> adminPool = new LdapConnectionPool(adminConnectionFactory);
> adminPool.setTestOnBorrow(true);
> adminPool.setMaxActive(config.getAdminPoolConfig().getMaxActive());
> adminPool.setWhenExhaustedAction(GenericObjectPool.WHEN_EXHAUSTED_BLOCK);
> }
> A solution for our Problem would most probably be to change the
> connectionPool configuration adminPool.setTestOnBorrow(false);
> This Parameter comes sadly not from the identity provider configuration.
> Is there a way to change this this parameter without creating an own
> implementation of the identity provider?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2783) Make LDAP connection pool 'testOnBorrow' configurable
[
https://issues.apache.org/jira/browse/OAK-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507993#comment-14507993
]
Tobias Bocanegra commented on OAK-2783:
---
also fixed in 1.0 branch, r1675378
> Make LDAP connection pool 'testOnBorrow' configurable
> -
>
> Key: OAK-2783
> URL: https://issues.apache.org/jira/browse/OAK-2783
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: auth-ldap
>Affects Versions: 1.2
>Reporter: Tobias Bocanegra
>Assignee: Tobias Bocanegra
>Priority: Minor
> Fix For: 1.3.0, 1.0.14
>
>
> Depending of the LDAP server configuration, it fails to connect as the server
> doesn't allow the connection validation query.
> It fails on
> {quote}
> Caused by: java.util.NoSuchElementException: Could not create a validated
> object, cause: ValidateObject failed
> at
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1233)
> at
> org.apache.directory.ldap.client.api.LdapConnectionPool.getConnection(LdapConnectionPool.java:56)
> at
> org.apache.jackrabbit.oak.security.authentication.ldap.impl.LdapIdentityProvider.connect(LdapIdentityProvider.java:532)
> ... 92 common frames omitted
> {quote}
> Based on customer analyze of Oak code this is the reason it fails:
> {quote}
> I think I have found a solution for the problem. While the system is
> initializing the connection it tries to validate the connection. This is the
> reason for the strange search request:
> SearchRequest
> baseDn : ''
> filter : '(objectClass=*)'
> scope : base object
> Because such kind of requests are not allowed in the client's ldap system the
> connection is being rejected (as invalid). It is configurable if the
> connection should be validated. The class
> org.apache.jackrabbit.oak.security.authentication.ldap.impl.LdapIdentityProvider
> contains this code
> if (config.getAdminPoolConfig().getMaxActive() != 0) {
> adminPool = new LdapConnectionPool(adminConnectionFactory);
> adminPool.setTestOnBorrow(true);
> adminPool.setMaxActive(config.getAdminPoolConfig().getMaxActive());
> adminPool.setWhenExhaustedAction(GenericObjectPool.WHEN_EXHAUSTED_BLOCK);
> }
> A solution for our Problem would most probably be to change the
> connectionPool configuration adminPool.setTestOnBorrow(false);
> This Parameter comes sadly not from the identity provider configuration.
> Is there a way to change this this parameter without creating an own
> implementation of the identity provider?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (OAK-2640) export org.apache.jackrabbit.oak.plugins.atomic
[
https://issues.apache.org/jira/browse/OAK-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruce Edge updated OAK-2640:
Comment: was deleted
(was: Probably a noob comment here, but I'm running into
Unable to resolve
osgi.wiring.package=org.apache.jackrabbit.oak.plugins.atomic)(version>=1.0.0)(!(version>=2.0.0
I see from http://svn.apache.org/r1667247 that the version is "1.0"
Is it possible that the version in
/jackrabbit/oak/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/atomic/package-info.java
should be 1.0.0 rather than 1.0?
)
> export org.apache.jackrabbit.oak.plugins.atomic
> ---
>
> Key: OAK-2640
> URL: https://issues.apache.org/jira/browse/OAK-2640
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
>Affects Versions: 1.1.6, 1.1.7
>Reporter: Davide Giannella
>Assignee: Davide Giannella
>Priority: Blocker
> Fix For: 1.1.8
>
>
> The package {{org.apache.jackrabbit.oak.plugins.atomic}} is not exported for
> OSGi environments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2640) export org.apache.jackrabbit.oak.plugins.atomic
[
https://issues.apache.org/jira/browse/OAK-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507605#comment-14507605
]
Bruce Edge commented on OAK-2640:
-
Probably a noob comment here, but I'm running into
Unable to resolve
osgi.wiring.package=org.apache.jackrabbit.oak.plugins.atomic)(version>=1.0.0)(!(version>=2.0.0
I see from http://svn.apache.org/r1667247 that the version is "1.0"
Is it possible that the version in
/jackrabbit/oak/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/atomic/package-info.java
should be 1.0.0 rather than 1.0?
> export org.apache.jackrabbit.oak.plugins.atomic
> ---
>
> Key: OAK-2640
> URL: https://issues.apache.org/jira/browse/OAK-2640
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
>Affects Versions: 1.1.6, 1.1.7
>Reporter: Davide Giannella
>Assignee: Davide Giannella
>Priority: Blocker
> Fix For: 1.1.8
>
>
> The package {{org.apache.jackrabbit.oak.plugins.atomic}} is not exported for
> OSGi environments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2803) RepositoryUpgrade.copy() doesn't create correct templates for residual item definitions
[ https://issues.apache.org/jira/browse/OAK-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke updated OAK-2803: Affects Version/s: (was: 1.4) > RepositoryUpgrade.copy() doesn't create correct templates for residual item > definitions > --- > > Key: OAK-2803 > URL: https://issues.apache.org/jira/browse/OAK-2803 > Project: Jackrabbit Oak > Issue Type: Bug > Components: upgrade >Affects Versions: 1.2.1 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.3.0, 1.2.2 > > > Since OAK-2454 RepositoryUpgrade.copy() uses the NodeTypeManager to register > node types, but does not properly set the name "*" for residual child > definition templates, causing a NullPointerException at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2803) RepositoryUpgrade.copy() doesn't create correct templates for residual item definitions
[ https://issues.apache.org/jira/browse/OAK-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke updated OAK-2803: Fix Version/s: (was: 1.4) 1.3.0 > RepositoryUpgrade.copy() doesn't create correct templates for residual item > definitions > --- > > Key: OAK-2803 > URL: https://issues.apache.org/jira/browse/OAK-2803 > Project: Jackrabbit Oak > Issue Type: Bug > Components: upgrade >Affects Versions: 1.2.1 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.3.0, 1.2.2 > > > Since OAK-2454 RepositoryUpgrade.copy() uses the NodeTypeManager to register > node types, but does not properly set the name "*" for residual child > definition templates, causing a NullPointerException at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2803) RepositoryUpgrade.copy() doesn't create correct templates for residual item definitions
[ https://issues.apache.org/jira/browse/OAK-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507506#comment-14507506 ] Manfred Baedke commented on OAK-2803: - Certainly [~mmarth], it's just so confusing that trunk is on 1.4-SNAPSHOT. > RepositoryUpgrade.copy() doesn't create correct templates for residual item > definitions > --- > > Key: OAK-2803 > URL: https://issues.apache.org/jira/browse/OAK-2803 > Project: Jackrabbit Oak > Issue Type: Bug > Components: upgrade >Affects Versions: 1.2.1, 1.4 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.4, 1.2.2 > > > Since OAK-2454 RepositoryUpgrade.copy() uses the NodeTypeManager to register > node types, but does not properly set the name "*" for residual child > definition templates, causing a NullPointerException at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2803) RepositoryUpgrade.copy() doesn't create correct templates for residual item definitions
[ https://issues.apache.org/jira/browse/OAK-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507417#comment-14507417 ] Michael Marth commented on OAK-2803: [~baedke], this seems fixed in trunk - I believe the fixVersion should be 1.3.0 rather than 1.4 then (?) > RepositoryUpgrade.copy() doesn't create correct templates for residual item > definitions > --- > > Key: OAK-2803 > URL: https://issues.apache.org/jira/browse/OAK-2803 > Project: Jackrabbit Oak > Issue Type: Bug > Components: upgrade >Affects Versions: 1.2.1, 1.4 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.4, 1.2.2 > > > Since OAK-2454 RepositoryUpgrade.copy() uses the NodeTypeManager to register > node types, but does not properly set the name "*" for residual child > definition templates, causing a NullPointerException at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2804) Conditional remove on DocumentStore
[
https://issues.apache.org/jira/browse/OAK-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507219#comment-14507219
]
Marcel Reutegger commented on OAK-2804:
---
Added a new method:
{noformat}
/**
* Batch remove documents with given keys and corresponding conditions. Keys
* for documents that do not exist are simply ignored. A document is only
* removed if the corresponding conditions are met. If this method fails
* with an exception, then only some of the documents may have been removed.
*
* @param the document type
* @param collection the collection.
* @param toRemove the keys of the documents to remove with the
* corresponding conditions.
* @return the number of removed documents.
*/
int remove(Collection collection,
Map> toRemove);
{noformat}
Implementations for {{MemoryDocumentStore}} and {{MongoDocumentStore}} is done
already. {{RDBDocumentStore}} is still missing.
Work in progress committed to trunk: http://svn.apache.org/r1675382
> Conditional remove on DocumentStore
> ---
>
> Key: OAK-2804
> URL: https://issues.apache.org/jira/browse/OAK-2804
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core, mongomk
>Reporter: Marcel Reutegger
>Assignee: Marcel Reutegger
> Fix For: 1.3.0
>
>
> The DocumentStore API allows for conditional inserts (only add document if
> not present yet) and updates (using findAndModify() with a condition), but it
> doesn't allow you to remove a document given some conditions are met.
> This feature is required to make sure the VersionGarbageCollector does not
> remove document that are modified concurrently. See OAK-2778.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506957#comment-14506957
]
Tommaso Teofili edited comment on OAK-2799 at 4/22/15 2:44 PM:
---
attached first patch.
{{OakIndexInput}} "original" (not cloned) instances have a {{WeakIdentityMap}}
holding references to clones once they get created.
Such clones get closed upon closing of the original instance.
An {{AlreadyClosedException}} is raised if {{OakIndexInput}} are already closed.
IntelliJ keeps telling me we should use {{super#clone}} in
{{OakIndexInput#clone}} and that would be inline with the usage of
{{WeakIdentityMap}} (using reference equality instead of {{#toString}} for key
comparison), otherwise it's probably safe to switch to a {{WeakHashMap}}, even
if that impl is not concurrent and therefore may require _synchronizing_ on
that or wrapping it with a concurrent version.
/cc [~chetanm], [~alexparvulescu], [~tmueller]
was (Author: teofili):
attached first patch.
{{OakIndexInput}} "original" (not cloned) instances have a {{WeakIdentityMap}}
holding references to clones once they get created.
Such clones get closed upon closing of the original instance.
An {{AlreadyClosedException}} is raised if {{OakIndexInput}} are already closed.
IntelliJ keeps telling me we should use {{super#clone}} in
{{OakIndexInput#clone}} and that would be inline with the usage of
{{WeakIdentityMap}} (using reference equality instead of {{#toString}} for key
comparison), otherwise it's probably safe to switch to a {{WeakHashMap}}, even
if that impl is not concurrent and therefore may require _synchronizing_ on
that or wrapping it with a concurrent version.
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2, 1.0.14
>
> Attachments: OAK-2799.0.patch
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2804) Conditional remove on DocumentStore
Marcel Reutegger created OAK-2804: - Summary: Conditional remove on DocumentStore Key: OAK-2804 URL: https://issues.apache.org/jira/browse/OAK-2804 Project: Jackrabbit Oak Issue Type: Improvement Components: core, mongomk Reporter: Marcel Reutegger Assignee: Marcel Reutegger Fix For: 1.3.0 The DocumentStore API allows for conditional inserts (only add document if not present yet) and updates (using findAndModify() with a condition), but it doesn't allow you to remove a document given some conditions are met. This feature is required to make sure the VersionGarbageCollector does not remove document that are modified concurrently. See OAK-2778. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2801) Clear excess references before cleanup
[
https://issues.apache.org/jira/browse/OAK-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-2801:
---
Description:
{{FileStore#cleanup}} would be more efficient when getting rid of as much
references as possibly beforehand. Excess references are contributed by the
current {{TarWriter}} instance and segment cache in {{SegmentTracker}}.
Those excess references turn out to be especially harmful with many concurrent
writers continuously writing to the repository. Starting with a certain write
load clean up will become completely blocked.
was:{{FileStore#cleanup}} would be more efficient when getting rid of as much
references as possibly beforehand. Excess references are contributed by the
current {{TarWriter}} instance and segment cache in {{SegmentTracker}}.
> Clear excess references before cleanup
> --
>
> Key: OAK-2801
> URL: https://issues.apache.org/jira/browse/OAK-2801
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: compaction, gc
> Fix For: 1.3.0
>
> Attachments: OAK-2801.patch, with-OAK-2801.png, without-OAK-2801.png
>
>
> {{FileStore#cleanup}} would be more efficient when getting rid of as much
> references as possibly beforehand. Excess references are contributed by the
> current {{TarWriter}} instance and segment cache in {{SegmentTracker}}.
> Those excess references turn out to be especially harmful with many
> concurrent writers continuously writing to the repository. Starting with a
> certain write load clean up will become completely blocked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-2801) Clear excess references before cleanup
[
https://issues.apache.org/jira/browse/OAK-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507109#comment-14507109
]
Michael Dürig edited comment on OAK-2801 at 4/22/15 1:54 PM:
-
Graphs showing the file store size and the depth and weight of the compaction
map with and without the patch. Top graph shows file store size in bytes,
middle graph shows compaction map weight in bytes and bottom graph shows
compaction map depths.
The data has been collected through the {{SegmentCompactionITMBean}} while
running {{SegmentCompactionIT}} with 10 readers, 10 references and 20 writers.
Without the patch there is no cleanup taking place. This is caused by too many
concurrent writers keeping too many references around. With the patch cleanup
takes place regularly.
was (Author: mduerig):
Graphs showing the file store size and the depth and width of the compaction
map with and without the patch.
The data has been collected through the {{SegmentCompactionITMBean}} while
running {{SegmentCompactionIT}} with 10 readers, 10 references and 20 writers.
Without the patch there is no cleanup taking place. This is caused by too many
concurrent writers keeping too many references around. With the patch cleanup
takes place regularly.
> Clear excess references before cleanup
> --
>
> Key: OAK-2801
> URL: https://issues.apache.org/jira/browse/OAK-2801
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: compaction, gc
> Fix For: 1.3.0
>
> Attachments: OAK-2801.patch, with-OAK-2801.png, without-OAK-2801.png
>
>
> {{FileStore#cleanup}} would be more efficient when getting rid of as much
> references as possibly beforehand. Excess references are contributed by the
> current {{TarWriter}} instance and segment cache in {{SegmentTracker}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2801) Clear excess references before cleanup
[
https://issues.apache.org/jira/browse/OAK-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-2801:
---
Attachment: without-OAK-2801.png
with-OAK-2801.png
Graphs showing the file store size and the depth and width of the compaction
map with and without the patch.
The data has been collected through the {{SegmentCompactionITMBean}} while
running {{SegmentCompactionIT}} with 10 readers, 10 references and 20 writers.
Without the patch there is no cleanup taking place. This is caused by too many
concurrent writers keeping too many references around. With the patch cleanup
takes place regularly.
> Clear excess references before cleanup
> --
>
> Key: OAK-2801
> URL: https://issues.apache.org/jira/browse/OAK-2801
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: compaction, gc
> Fix For: 1.3.0
>
> Attachments: OAK-2801.patch, with-OAK-2801.png, without-OAK-2801.png
>
>
> {{FileStore#cleanup}} would be more efficient when getting rid of as much
> references as possibly beforehand. Excess references are contributed by the
> current {{TarWriter}} instance and segment cache in {{SegmentTracker}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-2726) Avoid repository traversal for trivial node type changes
[
https://issues.apache.org/jira/browse/OAK-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke resolved OAK-2726.
-
Resolution: Fixed
> Avoid repository traversal for trivial node type changes
>
>
> Key: OAK-2726
> URL: https://issues.apache.org/jira/browse/OAK-2726
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.2.1
>Reporter: Marcel Reutegger
>Assignee: Julian Reschke
> Fix For: 1.4, 1.2.2
>
> Attachments: OAK-2726.diff
>
>
> The {{TypeEditor}} in oak-core checks the repository content when a node type
> changes to make sure the content still conforms to the updated node type. For
> a repository with a lot of nodes, this check can take quite a bit of time.
> Jackrabbit has an optimization in place for trivial node type changes. E.g.
> it will not check the content if a non-mandatory item is added to an existing
> node type definition. This optimization could be implemented in Oak as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-2803) RepositoryUpgrade.copy() doesn't create correct templates for residual item definitions
[ https://issues.apache.org/jira/browse/OAK-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke resolved OAK-2803. - Resolution: Fixed > RepositoryUpgrade.copy() doesn't create correct templates for residual item > definitions > --- > > Key: OAK-2803 > URL: https://issues.apache.org/jira/browse/OAK-2803 > Project: Jackrabbit Oak > Issue Type: Bug > Components: upgrade >Affects Versions: 1.2.1, 1.4 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.4, 1.2.2 > > > Since OAK-2454 RepositoryUpgrade.copy() uses the NodeTypeManager to register > node types, but does not properly set the name "*" for residual child > definition templates, causing a NullPointerException at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2726) Avoid repository traversal for trivial node type changes
[
https://issues.apache.org/jira/browse/OAK-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-2726:
Affects Version/s: 1.2.1
> Avoid repository traversal for trivial node type changes
>
>
> Key: OAK-2726
> URL: https://issues.apache.org/jira/browse/OAK-2726
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.2.1
>Reporter: Marcel Reutegger
>Assignee: Julian Reschke
> Fix For: 1.4, 1.2.2
>
> Attachments: OAK-2726.diff
>
>
> The {{TypeEditor}} in oak-core checks the repository content when a node type
> changes to make sure the content still conforms to the updated node type. For
> a repository with a lot of nodes, this check can take quite a bit of time.
> Jackrabbit has an optimization in place for trivial node type changes. E.g.
> it will not check the content if a non-mandatory item is added to an existing
> node type definition. This optimization could be implemented in Oak as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2726) Avoid repository traversal for trivial node type changes
[
https://issues.apache.org/jira/browse/OAK-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julian Reschke updated OAK-2726:
Fix Version/s: 1.2.2
> Avoid repository traversal for trivial node type changes
>
>
> Key: OAK-2726
> URL: https://issues.apache.org/jira/browse/OAK-2726
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.2.1
>Reporter: Marcel Reutegger
>Assignee: Julian Reschke
> Fix For: 1.4, 1.2.2
>
> Attachments: OAK-2726.diff
>
>
> The {{TypeEditor}} in oak-core checks the repository content when a node type
> changes to make sure the content still conforms to the updated node type. For
> a repository with a lot of nodes, this check can take quite a bit of time.
> Jackrabbit has an optimization in place for trivial node type changes. E.g.
> it will not check the content if a non-mandatory item is added to an existing
> node type definition. This optimization could be implemented in Oak as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2803) RepositoryUpgrade.copy() doesn't create correct templates for residual item definitions
[ https://issues.apache.org/jira/browse/OAK-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507034#comment-14507034 ] Manfred Baedke commented on OAK-2803: - Fixed in 1.4-SNAPSHOT with r1675354. Merged into 1.2.2-SNAPSHOT with r1675356 > RepositoryUpgrade.copy() doesn't create correct templates for residual item > definitions > --- > > Key: OAK-2803 > URL: https://issues.apache.org/jira/browse/OAK-2803 > Project: Jackrabbit Oak > Issue Type: Bug > Components: upgrade >Affects Versions: 1.2.1, 1.4 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.4, 1.2.2 > > > Since OAK-2454 RepositoryUpgrade.copy() uses the NodeTypeManager to register > node types, but does not properly set the name "*" for residual child > definition templates, causing a NullPointerException at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2802) avoid NodeTypeDefDiff code duplication
[ https://issues.apache.org/jira/browse/OAK-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2802: --- Labels: technical_debt (was: ) > avoid NodeTypeDefDiff code duplication > -- > > Key: OAK-2802 > URL: https://issues.apache.org/jira/browse/OAK-2802 > Project: Jackrabbit Oak > Issue Type: Task > Components: core >Affects Versions: 1.4 >Reporter: Julian Reschke >Priority: Minor > Labels: technical_debt > Fix For: 1.3.0 > > > org.apache.jackrabbit.oak.plugins.nodetype.NodeTypeDefDiff copies a huge > amount of code from > org.apache.jackrabbit.spi.commons.nodetype.NodeTypeDefDiff (the latter > working on QNodeTypeDefinitions, not NodeTypeDefinitions) > Figure out how to avoid the code duplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2802) avoid NodeTypeDefDiff code duplication
[ https://issues.apache.org/jira/browse/OAK-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig updated OAK-2802: --- Fix Version/s: 1.3.0 > avoid NodeTypeDefDiff code duplication > -- > > Key: OAK-2802 > URL: https://issues.apache.org/jira/browse/OAK-2802 > Project: Jackrabbit Oak > Issue Type: Task > Components: core >Affects Versions: 1.4 >Reporter: Julian Reschke >Priority: Minor > Labels: technical_debt > Fix For: 1.3.0 > > > org.apache.jackrabbit.oak.plugins.nodetype.NodeTypeDefDiff copies a huge > amount of code from > org.apache.jackrabbit.spi.commons.nodetype.NodeTypeDefDiff (the latter > working on QNodeTypeDefinitions, not NodeTypeDefinitions) > Figure out how to avoid the code duplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2803) RepositoryUpgrade.copy() doesn't create correct templates for residual item definitions
[ https://issues.apache.org/jira/browse/OAK-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manfred Baedke updated OAK-2803: Description: Since OAK-2454 RepositoryUpgrade.copy() uses the NodeTypeManager to register node types, but does not properly set the name "*" for residual child definition templates, causing a NullPointerException at runtime. > RepositoryUpgrade.copy() doesn't create correct templates for residual item > definitions > --- > > Key: OAK-2803 > URL: https://issues.apache.org/jira/browse/OAK-2803 > Project: Jackrabbit Oak > Issue Type: Bug > Components: upgrade >Affects Versions: 1.2.1, 1.4 >Reporter: Manfred Baedke >Assignee: Manfred Baedke > Fix For: 1.4, 1.2.2 > > > Since OAK-2454 RepositoryUpgrade.copy() uses the NodeTypeManager to register > node types, but does not properly set the name "*" for residual child > definition templates, causing a NullPointerException at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OAK-2803) RepositoryUpgrade.copy() doesn't create correct templates for residual item definitions
Manfred Baedke created OAK-2803: --- Summary: RepositoryUpgrade.copy() doesn't create correct templates for residual item definitions Key: OAK-2803 URL: https://issues.apache.org/jira/browse/OAK-2803 Project: Jackrabbit Oak Issue Type: Bug Components: upgrade Affects Versions: 1.2.1, 1.4 Reporter: Manfred Baedke Assignee: Manfred Baedke Fix For: 1.4, 1.2.2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506957#comment-14506957
]
Tommaso Teofili edited comment on OAK-2799 at 4/22/15 12:10 PM:
attached first patch.
{{OakIndexInput}} "original" (not cloned) instances have a {{WeakIdentityMap}}
holding references to clones once they get created.
Such clones get closed upon closing of the original instance.
An {{AlreadyClosedException}} is raised if {{OakIndexInput}} are already closed.
IntelliJ keeps telling me we should use {{super#clone}} in
{{OakIndexInput#clone}} and that would be inline with the usage of
{{WeakIdentityMap}} (using reference equality instead of {{#toString}} for key
comparison), otherwise it's probably safe to switch to a {{WeakHashMap}}, even
if that impl is not concurrent and therefore may require _synchronizing_ on
that or wrapping it with a concurrent version.
was (Author: teofili):
attached first patch.
{{OakIndexInput}} "original" (not cloned) instances have a {{WeakIdentityMap}}
holding references to clones once they get created.
Such clones get closed upon closing of the original instance.
An {{AlreadyClosedException}} are raised if {{OakIndexInput}} are already
closed.
IntelliJ keeps telling me we should use {{super#clone}} in
{{OakIndexInput#clone}} and that would be inline with the usage of
{{WeakIdentityMap}} (using reference equality instead of {{#toString}} for key
comparison), otherwise it's probably safe to switch to a {{WeakHashMap}}, even
if that impl is not concurrent and therefore may require _synchronizing_ on
that or wrapping it with a concurrent version.
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2, 1.0.14
>
> Attachments: OAK-2799.0.patch
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommaso Teofili updated OAK-2799:
-
Fix Version/s: 1.0.14
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2, 1.0.14
>
> Attachments: OAK-2799.0.patch
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommaso Teofili updated OAK-2799:
-
Attachment: OAK-2799.0.patch
attached first patch.
{{OakIndexInput}} "original" (not cloned) instances have a {{WeakIdentityMap}}
holding references to clones once they get created.
Such clones get closed upon closing of the original instance.
An {{AlreadyClosedException}} are raised if {{OakIndexInput}} are already
closed.
IntelliJ keeps telling me we should use {{super#clone}} in
{{OakIndexInput#clone}} and that would be inline with the usage of
{{WeakIdentityMap}} (using reference equality instead of {{#toString}} for key
comparison), otherwise it's probably safe to switch to a {{WeakHashMap}}, even
if that impl is not concurrent and therefore may require _synchronizing_ on
that or wrapping it with a concurrent version.
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2
>
> Attachments: OAK-2799.0.patch
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2802) avoid NodeTypeDefDiff code duplication
Julian Reschke created OAK-2802: --- Summary: avoid NodeTypeDefDiff code duplication Key: OAK-2802 URL: https://issues.apache.org/jira/browse/OAK-2802 Project: Jackrabbit Oak Issue Type: Task Components: core Affects Versions: 1.4 Reporter: Julian Reschke Priority: Minor org.apache.jackrabbit.oak.plugins.nodetype.NodeTypeDefDiff copies a huge amount of code from org.apache.jackrabbit.spi.commons.nodetype.NodeTypeDefDiff (the latter working on QNodeTypeDefinitions, not NodeTypeDefinitions) Figure out how to avoid the code duplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OAK-2801) Clear excess references before cleanup
[
https://issues.apache.org/jira/browse/OAK-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig updated OAK-2801:
---
Attachment: OAK-2801.patch
Patch clearing the segment cache and creating a new {{TarWriter}} instance
before cleanup. Note: this uncovered an issue with the special casing of bulk
segments in {{TarReader.cleanup}}, which might cause a
{{SegmentNotFoundException}}.
[~alex.parvulescu], WDYT?
> Clear excess references before cleanup
> --
>
> Key: OAK-2801
> URL: https://issues.apache.org/jira/browse/OAK-2801
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: compaction, gc
> Fix For: 1.3.0
>
> Attachments: OAK-2801.patch
>
>
> {{FileStore#cleanup}} would be more efficient when getting rid of as much
> references as possibly beforehand. Excess references are contributed by the
> current {{TarWriter}} instance and segment cache in {{SegmentTracker}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-2798) OakIndexInput should unset OakIndexFile's byte[] upon close
[
https://issues.apache.org/jira/browse/OAK-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommaso Teofili resolved OAK-2798.
--
Resolution: Fixed
Assignee: Tommaso Teofili
fixed in r1675332 (trunk), r1675334 (branch 1.2), r1675336 (branch 1.0)
> OakIndexInput should unset OakIndexFile's byte[] upon close
> ---
>
> Key: OAK-2798
> URL: https://issues.apache.org/jira/browse/OAK-2798
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2, 1.0.14
>
> Attachments: OAK-2798.patch
>
>
> {{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
> {{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
> _byte[]_ is not unset.
> This can lead to memory leak as when IndexInputs gets closed, e.g. by
> {{StoredFieldsReaders}} such references won't be properly cleared (with
> probably a higher impact in cases where _copyOnRead = true_).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2801) Clear excess references before cleanup
Michael Dürig created OAK-2801:
--
Summary: Clear excess references before cleanup
Key: OAK-2801
URL: https://issues.apache.org/jira/browse/OAK-2801
Project: Jackrabbit Oak
Issue Type: Improvement
Components: segmentmk
Reporter: Michael Dürig
Assignee: Michael Dürig
Fix For: 1.3.0
{{FileStore#cleanup}} would be more efficient when getting rid of as much
references as possibly beforehand. Excess references are contributed by the
current {{TarWriter}} instance and segment cache in {{SegmentTracker}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommaso Teofili updated OAK-2799:
-
Comment: was deleted
(was: fixed in r1675332 (trunk) and r1675334 (branch 1.2))
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommaso Teofili reopened OAK-2799:
--
clicked resolve on the wrong issue, reopening
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommaso Teofili resolved OAK-2799.
--
Resolution: Fixed
Assignee: Tommaso Teofili
fixed in r1675332 (trunk) and r1675334 (branch 1.2)
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommaso Teofili updated OAK-2799:
-
Fix Version/s: 1.3.0
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2798) OakIndexInput should unset OakIndexFile's byte[] upon close
[
https://issues.apache.org/jira/browse/OAK-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506834#comment-14506834
]
Tommaso Teofili commented on OAK-2798:
--
yes, I'll file a separate patch in OAK-2799 for that.
> OakIndexInput should unset OakIndexFile's byte[] upon close
> ---
>
> Key: OAK-2798
> URL: https://issues.apache.org/jira/browse/OAK-2798
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2, 1.0.14
>
> Attachments: OAK-2798.patch
>
>
> {{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
> {{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
> _byte[]_ is not unset.
> This can lead to memory leak as when IndexInputs gets closed, e.g. by
> {{StoredFieldsReaders}} such references won't be properly cleared (with
> probably a higher impact in cases where _copyOnRead = true_).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-2800) Make contributions to reference graph from TarWriter less conservative
[
https://issues.apache.org/jira/browse/OAK-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Dürig resolved OAK-2800.
Resolution: Fixed
Fixed in trunk at http://svn.apache.org/r1675319
> Make contributions to reference graph from TarWriter less conservative
> ---
>
> Key: OAK-2800
> URL: https://issues.apache.org/jira/browse/OAK-2800
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: segmentmk
>Reporter: Michael Dürig
>Assignee: Michael Dürig
> Labels: compaction, gc
> Fix For: 1.3.0
>
>
> {{TarWriter#cleanup}} currently adds all its references to the initial
> elements of the reference graph. It would be sufficient though to just add
> the references in the tar writer's own graph.
> At the same time I'd like to rename that method from {{cleanup}} to
> {{collectReferences}}, which better reflects its semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2798) OakIndexInput should unset OakIndexFile's byte[] upon close
[
https://issues.apache.org/jira/browse/OAK-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506791#comment-14506791
]
Chetan Mehrotra commented on OAK-2798:
--
Patch looks fine to me. May be we add support for AlreadyClosedException etc
but I think that can be done as part of OAK-2799
> OakIndexInput should unset OakIndexFile's byte[] upon close
> ---
>
> Key: OAK-2798
> URL: https://issues.apache.org/jira/browse/OAK-2798
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2, 1.0.14
>
> Attachments: OAK-2798.patch
>
>
> {{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
> {{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
> _byte[]_ is not unset.
> This can lead to memory leak as when IndexInputs gets closed, e.g. by
> {{StoredFieldsReaders}} such references won't be properly cleared (with
> probably a higher impact in cases where _copyOnRead = true_).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2798) OakIndexInput should unset OakIndexFile's byte[] upon close
[
https://issues.apache.org/jira/browse/OAK-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chetan Mehrotra updated OAK-2798:
-
Fix Version/s: 1.0.14
1.3.0
> OakIndexInput should unset OakIndexFile's byte[] upon close
> ---
>
> Key: OAK-2798
> URL: https://issues.apache.org/jira/browse/OAK-2798
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.3.0, 1.2.2, 1.0.14
>
> Attachments: OAK-2798.patch
>
>
> {{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
> {{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
> _byte[]_ is not unset.
> This can lead to memory leak as when IndexInputs gets closed, e.g. by
> {{StoredFieldsReaders}} such references won't be properly cleared (with
> probably a higher impact in cases where _copyOnRead = true_).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2800) Make contributions to reference graph from TarWriter less conservative
Michael Dürig created OAK-2800:
--
Summary: Make contributions to reference graph from TarWriter less
conservative
Key: OAK-2800
URL: https://issues.apache.org/jira/browse/OAK-2800
Project: Jackrabbit Oak
Issue Type: Improvement
Components: segmentmk
Reporter: Michael Dürig
Assignee: Michael Dürig
Fix For: 1.3.0
{{TarWriter#cleanup}} currently adds all its references to the initial elements
of the reference graph. It would be sufficient though to just add the
references in the tar writer's own graph.
At the same time I'd like to rename that method from {{cleanup}} to
{{collectReferences}}, which better reflects its semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2798) OakIndexInput should unset OakIndexFile's byte[] upon close
[
https://issues.apache.org/jira/browse/OAK-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommaso Teofili updated OAK-2798:
-
Attachment: OAK-2798.patch
attached patch unsetting blob and data for OakIndexInput and OakIndexOutput (as
I think the same applies there).
> OakIndexInput should unset OakIndexFile's byte[] upon close
> ---
>
> Key: OAK-2798
> URL: https://issues.apache.org/jira/browse/OAK-2798
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.2.2
>
> Attachments: OAK-2798.patch
>
>
> {{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
> {{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
> _byte[]_ is not unset.
> This can lead to memory leak as when IndexInputs gets closed, e.g. by
> {{StoredFieldsReaders}} such references won't be properly cleared (with
> probably a higher impact in cases where _copyOnRead = true_).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (OAK-2734) Compaction does not finish on repository with continuous writes
[ https://issues.apache.org/jira/browse/OAK-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Dürig resolved OAK-2734. Resolution: Fixed Fixed in trunk at http://svn.apache.org/r1675311 > Compaction does not finish on repository with continuous writes > > > Key: OAK-2734 > URL: https://issues.apache.org/jira/browse/OAK-2734 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segmentmk >Reporter: Michael Dürig >Assignee: Michael Dürig > Labels: compaction, gc > Fix For: 1.3.0 > > > A repository with continuous writes can keep the compactor from completing > causing the repository size to grow indefinitely. > This effect is caused by the compactor trying to catch up with changes that > occurred while compacting. I.e. compacting them on top of the already > compacted head. When there is a steady stream of incoming changes it can > happen that the compactor never actually catches up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506744#comment-14506744
]
Tommaso Teofili commented on OAK-2799:
--
bq. we would need to have weak reference map etc to be kept
yes, I think so too.
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.2.2
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2798) OakIndexInput should unset OakIndexFile's byte[] upon close
[
https://issues.apache.org/jira/browse/OAK-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506739#comment-14506739
]
Alex Parvulescu commented on OAK-2798:
--
sounds good to me. should we also clear the _data_ blob list then?
> OakIndexInput should unset OakIndexFile's byte[] upon close
> ---
>
> Key: OAK-2798
> URL: https://issues.apache.org/jira/browse/OAK-2798
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.2.2
>
>
> {{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
> {{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
> _byte[]_ is not unset.
> This can lead to memory leak as when IndexInputs gets closed, e.g. by
> {{StoredFieldsReaders}} such references won't be properly cleared (with
> probably a higher impact in cases where _copyOnRead = true_).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506731#comment-14506731
]
Chetan Mehrotra commented on OAK-2799:
--
bq. but I also think that the original instance should close the cloned
instances
hmm yes looks like that would need to be done and then we would need to have
weak reference map etc to be kept. No way to avoid that then
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.2.2
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2798) OakIndexInput should unset OakIndexFile's byte[] upon close
[
https://issues.apache.org/jira/browse/OAK-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506725#comment-14506725
]
Chetan Mehrotra commented on OAK-2798:
--
+1 to unset the array
bq. This can lead to memory leak as when IndexInputs gets closed, e.g. by
StoredFieldsReaders such references won't be properly cleared (with probably a
higher impact in cases where copyOnRead = true).
Did not understand how it would have higher impact for copyOnRead as in that
case I expect that OakIndexFile would not be used much. IndexCopier would use
it for copy but then would not keep a reference to the input instance. So it
would eventually get GC
> OakIndexInput should unset OakIndexFile's byte[] upon close
> ---
>
> Key: OAK-2798
> URL: https://issues.apache.org/jira/browse/OAK-2798
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.2.2
>
>
> {{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
> {{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
> _byte[]_ is not unset.
> This can lead to memory leak as when IndexInputs gets closed, e.g. by
> {{StoredFieldsReaders}} such references won't be properly cleared (with
> probably a higher impact in cases where _copyOnRead = true_).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommaso Teofili updated OAK-2799:
-
Assignee: (was: Tommaso Teofili)
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.2.2
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (OAK-2798) OakIndexInput should unset OakIndexFile's byte[] upon close
[
https://issues.apache.org/jira/browse/OAK-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506700#comment-14506700
]
Tommaso Teofili edited comment on OAK-2798 at 4/22/15 9:33 AM:
---
[~alexparvulescu], [~chetanm] WDYT?
was (Author: teofili):
[~alexparvulescu] WDYT?
> OakIndexInput should unset OakIndexFile's byte[] upon close
> ---
>
> Key: OAK-2798
> URL: https://issues.apache.org/jira/browse/OAK-2798
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.2.2
>
>
> {{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
> {{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
> _byte[]_ is not unset.
> This can lead to memory leak as when IndexInputs gets closed, e.g. by
> {{StoredFieldsReaders}} such references won't be properly cleared (with
> probably a higher impact in cases where _copyOnRead = true_).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2799) OakIndexInput cloned instances are not closed
[
https://issues.apache.org/jira/browse/OAK-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506718#comment-14506718
]
Tommaso Teofili commented on OAK-2799:
--
[~alexparvulescu], [~chetanm] WDYT?
> OakIndexInput cloned instances are not closed
> -
>
> Key: OAK-2799
> URL: https://issues.apache.org/jira/browse/OAK-2799
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.2.2
>
>
> Related to the inspections I was doing for OAK-2798 I also noticed that we
> don't fully comply with the {{IndexInput}} javadoc [1] as the cloned
> instances should throw the given exception if original is closed, but I also
> think that the original instance should close the cloned instances, see also
> [ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
> [1] : {code}
> /** Abstract base class for input from a file in a {@link Directory}. A
> * random-access input stream. Used for all Lucene index input operations.
> *
> * {@code IndexInput} may only be used from one thread, because it is not
> * thread safe (it keeps internal state like file position). To allow
> * multithreaded use, every {@code IndexInput} instance must be cloned before
> * used in another thread. Subclasses must therefore implement {@link
> #clone()},
> * returning a new {@code IndexInput} which operates on the same underlying
> * resource, but positioned independently. Lucene never closes cloned
> * {@code IndexInput}s, it will only do this on the original one.
> * The original instance must take care that cloned instances throw
> * {@link AlreadyClosedException} when the original one is closed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2799) OakIndexInput cloned instances are not closed
Tommaso Teofili created OAK-2799:
Summary: OakIndexInput cloned instances are not closed
Key: OAK-2799
URL: https://issues.apache.org/jira/browse/OAK-2799
Project: Jackrabbit Oak
Issue Type: Bug
Components: lucene
Affects Versions: 1.2.1
Reporter: Tommaso Teofili
Assignee: Tommaso Teofili
Fix For: 1.2.2
Related to the inspections I was doing for OAK-2798 I also noticed that we
don't fully comply with the {{IndexInput}} javadoc [1] as the cloned instances
should throw the given exception if original is closed, but I also think that
the original instance should close the cloned instances, see also
[ByteBufferIndexInput#close|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java#L271].
[1] : {code}
/** Abstract base class for input from a file in a {@link Directory}. A
* random-access input stream. Used for all Lucene index input operations.
*
* {@code IndexInput} may only be used from one thread, because it is not
* thread safe (it keeps internal state like file position). To allow
* multithreaded use, every {@code IndexInput} instance must be cloned before
* used in another thread. Subclasses must therefore implement {@link #clone()},
* returning a new {@code IndexInput} which operates on the same underlying
* resource, but positioned independently. Lucene never closes cloned
* {@code IndexInput}s, it will only do this on the original one.
* The original instance must take care that cloned instances throw
* {@link AlreadyClosedException} when the original one is closed.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2798) OakIndexInput should unset OakIndexFile's byte[] upon close
[
https://issues.apache.org/jira/browse/OAK-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommaso Teofili updated OAK-2798:
-
Description:
{{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
{{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
_byte[]_ is not unset.
This can lead to memory leak as when IndexInputs gets closed, e.g. by
{{StoredFieldsReaders}} such references won't be properly cleared (with
probably a higher impact in cases where _copyOnRead = true_).
was:
{{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
{{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
_byte[]_ is not unset.
This can lead to memory leak as when IndexInputs gets closed, e.g. by
{{StoredFieldsReaders}} such references won't be properly cleared.
> OakIndexInput should unset OakIndexFile's byte[] upon close
> ---
>
> Key: OAK-2798
> URL: https://issues.apache.org/jira/browse/OAK-2798
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.2.2
>
>
> {{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
> {{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
> _byte[]_ is not unset.
> This can lead to memory leak as when IndexInputs gets closed, e.g. by
> {{StoredFieldsReaders}} such references won't be properly cleared (with
> probably a higher impact in cases where _copyOnRead = true_).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2797) Closeable aspect of Analyzer should be accounted for
[
https://issues.apache.org/jira/browse/OAK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506702#comment-14506702
]
Chetan Mehrotra commented on OAK-2797:
--
Looking at [Analyzer
code|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/analysis/Analyzer.java]
it uses a {{CloseableThreadLocal}}
{code}
public abstract class Analyzer implements Closeable {
private final ReuseStrategy reuseStrategy;
// non final as it gets nulled if closed; pkg private for access by
ReuseStrategy's final helper methods:
CloseableThreadLocal storedValue = new CloseableThreadLocal();
/** Frees persistent resources used by this Analyzer */
@Override
public void close() {
if (storedValue != null) {
storedValue.close();
storedValue = null;
}
}
}
{code}
And checking the docs for
[CloseableThreadLocal|https://github.com/apache/lucene-solr/blob/lucene_solr_4_7_1/lucene/core/src/java/org/apache/lucene/util/CloseableThreadLocal.java]
which states that it should not be closed *until all threads are done using
the instance* it appears that we need to close the {{Analyzer}} only upon
shutdown to free up resources. So possibly in
{{LuceneIndexProviderService#deactivate}}
{code}
* We can not rely on {@link ThreadLocal#remove()} as it
* only removes the value for the caller thread, whereas
* {@link #close} takes care of all
* threads. You should not call {@link #close} until all
* threads are done using the instance.
*
* @lucene.internal
*/
public class CloseableThreadLocal implements Closeable {
{code}
> Closeable aspect of Analyzer should be accounted for
>
>
> Key: OAK-2797
> URL: https://issues.apache.org/jira/browse/OAK-2797
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Reporter: Chetan Mehrotra
> Fix For: 1.3.0
>
>
> Lucene {{Analyzer}} implements {{Closeable}} [1] interface and internally it
> has a ThreadLocal storage of some persistent resource
> So far in oak-lucene we do not take care of closing any analyzer. In fact we
> use a singleton Analyzer in all cases. Opening this bug to think about this
> aspect and see if our usage of Analyzer follows the best practices
> [1]
> http://lucene.apache.org/core/4_7_0/core/org/apache/lucene/analysis/Analyzer.html#close%28%29
> /cc [~teofili] [~alex.parvulescu]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2798) OakIndexInput should unset OakIndexFile's byte[] upon close
[
https://issues.apache.org/jira/browse/OAK-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506700#comment-14506700
]
Tommaso Teofili commented on OAK-2798:
--
[~alexparvulescu] WDYT?
> OakIndexInput should unset OakIndexFile's byte[] upon close
> ---
>
> Key: OAK-2798
> URL: https://issues.apache.org/jira/browse/OAK-2798
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
>Assignee: Tommaso Teofili
> Fix For: 1.2.2
>
>
> {{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
> {{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
> _byte[]_ is not unset.
> This can lead to memory leak as when IndexInputs gets closed, e.g. by
> {{StoredFieldsReaders}} such references won't be properly cleared.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OAK-2798) OakIndexInput should unset OakIndexFile's byte[] upon close
[
https://issues.apache.org/jira/browse/OAK-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tommaso Teofili updated OAK-2798:
-
Assignee: (was: Tommaso Teofili)
> OakIndexInput should unset OakIndexFile's byte[] upon close
> ---
>
> Key: OAK-2798
> URL: https://issues.apache.org/jira/browse/OAK-2798
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.2.1
>Reporter: Tommaso Teofili
> Fix For: 1.2.2
>
>
> {{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
> {{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
> _byte[]_ is not unset.
> This can lead to memory leak as when IndexInputs gets closed, e.g. by
> {{StoredFieldsReaders}} such references won't be properly cleared.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2798) OakIndexInput should unset OakIndexFile's byte[] upon close
Tommaso Teofili created OAK-2798:
Summary: OakIndexInput should unset OakIndexFile's byte[] upon
close
Key: OAK-2798
URL: https://issues.apache.org/jira/browse/OAK-2798
Project: Jackrabbit Oak
Issue Type: Bug
Components: lucene
Affects Versions: 1.2.1
Reporter: Tommaso Teofili
Assignee: Tommaso Teofili
Fix For: 1.2.2
{{OakIndexInput}}, being an extension of Lucene's {{IndexInput}} implements
{{Closable}}, however upon {{#close}} the referenced {{OakIndexFile}}'s
_byte[]_ is not unset.
This can lead to memory leak as when IndexInputs gets closed, e.g. by
{{StoredFieldsReaders}} such references won't be properly cleared.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2797) Closeable aspect of Analyzer should be accounted for
Chetan Mehrotra created OAK-2797:
Summary: Closeable aspect of Analyzer should be accounted for
Key: OAK-2797
URL: https://issues.apache.org/jira/browse/OAK-2797
Project: Jackrabbit Oak
Issue Type: Bug
Components: lucene
Reporter: Chetan Mehrotra
Fix For: 1.3.0
Lucene {{Analyzer}} implements {{Closeable}} [1] interface and internally it
has a ThreadLocal storage of some persistent resource
So far in oak-lucene we do not take care of closing any analyzer. In fact we
use a singleton Analyzer in all cases. Opening this bug to think about this
aspect and see if our usage of Analyzer follows the best practices
[1]
http://lucene.apache.org/core/4_7_0/core/org/apache/lucene/analysis/Analyzer.html#close%28%29
/cc [~teofili] [~alex.parvulescu]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2796) Persistent cache: old files not removed sometimes
[
https://issues.apache.org/jira/browse/OAK-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506585#comment-14506585
]
Thomas Mueller commented on OAK-2796:
-
Something to test / verify: the old files should be removed after a restart at
the latest.
> Persistent cache: old files not removed sometimes
> -
>
> Key: OAK-2796
> URL: https://issues.apache.org/jira/browse/OAK-2796
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: cache
>Affects Versions: 1.0, 1.2
>Reporter: Thomas Mueller
>Assignee: Thomas Mueller
> Fix For: 1.2.2, 1.0.14
>
>
> When using Windows (not sure if that's relevant), the cache directory grows
> and the following messages are logged:
> {noformat}
> org.apache.jackrabbit.oak.plugins.document.persistentCache.MapFactory Could
> not close or compact the store: java.lang.IllegalStateException: This map is
> closed [1.4.185/4]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (OAK-2796) Persistent cache: old files not removed sometimes
Thomas Mueller created OAK-2796:
---
Summary: Persistent cache: old files not removed sometimes
Key: OAK-2796
URL: https://issues.apache.org/jira/browse/OAK-2796
Project: Jackrabbit Oak
Issue Type: Bug
Components: cache
Affects Versions: 1.2, 1.0
Reporter: Thomas Mueller
Assignee: Thomas Mueller
Fix For: 1.2.2, 1.0.14
When using Windows (not sure if that's relevant), the cache directory grows and
the following messages are logged:
{noformat}
org.apache.jackrabbit.oak.plugins.document.persistentCache.MapFactory Could not
close or compact the store: java.lang.IllegalStateException: This map is closed
[1.4.185/4]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OAK-2569) LuceneIndex#loadDocs can throw IllegalArgumentException
[
https://issues.apache.org/jira/browse/OAK-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506551#comment-14506551
]
Chetan Mehrotra commented on OAK-2569:
--
bq. if many documents were added and removed at the same time, some of them
might not be returned. That means the query result would be wrong (the result
could miss data that was there in both the old and the new revision).
With current flow that would not happen. What I was suggesting that if
exception occurs then perform search again without offset. This would lead to
duplicate results but then those would be filtered out due to {{seenPaths}}.
Query would still be performed again with newer index so there should not be a
possibility of wrong result.
bq. We would need to hold on to IndexSearcher from within Cursor
The problem with that would be that long running Cursor would prevent the
IndexNode to switch to newer version. Currently when a change in index is
detected then IndexNode waits for existing searchers to be retuned back so that
it can close the current directory and then open a new one. Now once a Cursor
is returned it would not be possible to enforce timely close of cursor and it
would hold up the switch to newer index. Hence it needs to be avoided
> LuceneIndex#loadDocs can throw IllegalArgumentException
> ---
>
> Key: OAK-2569
> URL: https://issues.apache.org/jira/browse/OAK-2569
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: lucene
>Affects Versions: 1.1.6, 1.2.1
>Reporter: Alex Parvulescu
> Fix For: 1.3.0, 1.2.2
>
>
> It looks like in some cases a IllegalArgumentException will be thrown if the
> _nextBatchSize_ passed to the _searchAfter_ method call is smaller than the
> current searcher limit (_lastDoc.doc_).
> Stack trace for the failure:
> {code}
> java.lang.IllegalArgumentException: after.doc exceeds the number of documents
> in the reader: after.doc=387045 limit=386661
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:442)
> at org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:243)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex$1.loadDocs(LuceneIndex.java:352)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex$1.computeNext(LuceneIndex.java:292)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex$1.computeNext(LuceneIndex.java:283)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex$LucenePathCursor$1.hasNext(LuceneIndex.java:1055)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.jackrabbit.oak.spi.query.Cursors$PathCursor.hasNext(Cursors.java:198)
> at
> org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndex$LucenePathCursor.hasNext(LuceneIndex.java:1076)
> at
> org.apache.jackrabbit.oak.plugins.index.aggregate.AggregationCursor.fetchNext(AggregationCursor.java:88)
> at
> org.apache.jackrabbit.oak.plugins.index.aggregate.AggregationCursor.hasNext(AggregationCursor.java:75)
> at
> org.apache.jackrabbit.oak.spi.query.Cursors$IntersectionCursor.fetchNext(Cursors.java:389)
> at
> org.apache.jackrabbit.oak.spi.query.Cursors$IntersectionCursor.hasNext(Cursors.java:381)
> at
> org.apache.jackrabbit.oak.spi.query.Cursors$IntersectionCursor.fetchNext(Cursors.java:389)
> at
> org.apache.jackrabbit.oak.spi.query.Cursors$IntersectionCursor.hasNext(Cursors.java:381)
> at
> org.apache.jackrabbit.oak.spi.query.Cursors$IntersectionCursor.fetchNext(Cursors.java:389)
> at
> org.apache.jackrabbit.oak.spi.query.Cursors$IntersectionCursor.hasNext(Cursors.java:381)
> at
> org.apache.jackrabbit.oak.query.ast.SelectorImpl.next(SelectorImpl.java:401)
> at
> org.apache.jackrabbit.oak.query.QueryImpl$RowIterator.fetchNext(QueryImpl.java:664)
> at
> org.apache.jackrabbit.oak.query.QueryImpl$RowIterator.hasNext(QueryImpl.java:689)
> at
> org.apache.jackrabbit.oak.query.FilterIterators$SortIterator.init(FilterIterators.java:203)
> at
> org.apache.jackrabbit.oak.query.FilterIterators$SortIterator.hasNext(FilterIterators.java:237)
> at
> org.apache.jackrabbit.oak.jcr.query.QueryResultImpl$1.fetch(QueryResultImpl.java:108)
> at
> org.apache.jackrabbit.oak.jcr.query.QueryResultImpl$1.(QueryResultImpl.java:104)
> at
> org.apache.jackrabbit.oak.jcr.query.QueryResultImpl.getRows(QueryResultImpl.java:91)
> at org.apache.jackrabbit.commons.query.G
