[
https://issues.apache.org/jira/browse/DIRSERVER-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199979#comment-17199979
]
li edited comment on DIRSERVER-2255 at 9/22/20, 10:37 AM:
----------------------------------------------------------
[~elecharny]
Hi Emmanuel,
About this replication issue, I find that this bug would be trigged when
the slave has network issue
e.g.
1. the socket will be reset by firewall, because the replication socket
is a long TCP connection, there are some firewall policy which could be
triggered to reset the TCP connection in some case.
2. slave instance be shutdown or restart
When the slave replication socket reconnect, master server will create
another ads-dsReplicaId,. Because the change log still be written to old
dsReplicaId's REPL_EVENT_LOG and this change log does not be consumed by slave,
so the memory will be exhausted.
So I guess maybe there are some bug in the creation of the
ads-dsReplicaId, or the creation of the ads-dsReplicaId without checking.
was (Author: jingwen.li):
[~elecharny]
Hi Emmanuel,
About this replication issue, I find that this bug would be trigged when
the slave has network issue
e.g.
1. the socket will be reset by firewall, because the replication socket
is a long TCP connection, there are some firewall policy which could be
triggered to reset the TCP connection in some case.
2. slave instance be shutdown or restart
When the slave replication socket reconnect, master server will create
another ads-dsReplicaId,. Because the change log still be written to old
dsReplicaId's REPL_EVENT_LOG and this change log is not be consumed by slave,
so the memory will be exhausted.
So I guess maybe there are some bug in the creation of the
ads-dsReplicaId, or the creation of the ads-dsReplicaId without checking.
> java.lang.OutOfMemoryError: Java heap space
> -------------------------------------------
>
> Key: DIRSERVER-2255
> URL: https://issues.apache.org/jira/browse/DIRSERVER-2255
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: replication
> Affects Versions: 2.0.0.AM25
> Environment: Linux CentOS 7
> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
> Reporter: li
> Priority: Major
> Attachments: 1.png, 9-instances-SyncReplSearchListener .png,
> conf.zip, image-2020-09-22-09-08-56-660.png,
> image-2020-09-22-09-13-55-503.png, image-2020-09-22-09-18-18-047.png,
> wrapper.conf
>
>
> INFO | jvm 1 | 2018/10/29 12:01:45 | Exception in thread "Thread-4"
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> java.util.Arrays.copyOf(Arrays.java:3236)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> java.nio.charset.CharsetEncoder.replaceWith(CharsetEncoder.java:301)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:207)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:233)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:558)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:554)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> sun.nio.cs.UTF_8.newEncoder(UTF_8.java:72)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> java.lang.StringCoding.encode(StringCoding.java:348)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> java.lang.String.getBytes(String.java:941)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.api.util.Strings.getBytesUtf8(Strings.java:1626)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.api.ldap.model.entry.Value.<init>(Value.java:261)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.apply(DefaultAttribute.java:1828)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.api.ldap.model.entry.DefaultEntry.readExternal(DefaultEntry.java:2443)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.ldap.replication.ReplicaEventMessageSerializer.deserialize(ReplicaEventMessageSerializer.java:126)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.deserialize(BPage.java:1188)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.deserialize(BPage.java:81)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.recman.BaseRecordManager.fetch(BaseRecordManager.java:329)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.loadBPage(BPage.java:949)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.childBPage(BPage.java:938)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.insert(BPage.java:380)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.insert(BPage.java:381)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.insert(BPage.java:381)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.insert(BPage.java:381)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BPage.insert(BPage.java:381)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> jdbm.btree.BTree.insert(BTree.java:302)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.put(JdbmTable.java:541)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.ldap.replication.provider.ReplicaEventLog.log(ReplicaEventLog.java:176)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.ldap.replication.provider.SyncReplSearchListener.entryModified(SyncReplSearchListener.java:314)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.core.event.EventInterceptor.fire(EventInterceptor.java:169)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.core.event.EventInterceptor.modify(EventInterceptor.java:333)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:508)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at
> org.apache.directory.server.core.subtree.SubentryInterceptor.modify(SubentryInterceptor.java:1249)
> INFO | jvm 1 | 2018/10/29 12:01:47 | Exception in thread
> "pool-13-thread-1" java.lang.OutOfMemoryError: GC overhead limit exceeded
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.api.ldap.model.entry.Value.clone(Value.java:425)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.api.ldap.model.name.Ava.getValue(Ava.java:728)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.api.ldap.model.name.Rdn.<init>(Rdn.java:455)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.api.ldap.model.name.Dn.<init>(Dn.java:197)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.api.ldap.model.entry.DefaultEntry.normalizeDn(DefaultEntry.java:519)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.api.ldap.model.entry.DefaultEntry.<init>(DefaultEntry.java:300)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.directory.ldap.client.api.LdapNetworkConnection.messageReceived(LdapNetworkConnection.java:2757)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1019)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1141)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:437)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:256)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1141)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:643)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:539)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.access$1200(AbstractPollingIoProcessor.java:68)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1242)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1231)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:683)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at
> java.lang.Thread.run(Thread.java:748)
> INFO | jvm 1 | 2018/10/29 12:50:50 | Exception in thread
> "pool-7-thread-5835" java.lang.OutOfMemoryError: GC overhead limit exceeded
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> java.io.BufferedReader.<init>(BufferedReader.java:105)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> java.io.BufferedReader.<init>(BufferedReader.java:116)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.api.ldap.model.schema.parsers.OpenLdapSchemaParser.parseAttributeType(OpenLdapSchemaParser.java:2599)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.api.ldap.model.schema.parsers.AttributeTypeDescriptionSchemaParser.parse(AttributeTypeDescriptionSchemaParser.java:81)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.api.ldap.model.schema.syntaxCheckers.AttributeTypeDescriptionSyntaxChecker.isValidSyntax(AttributeTypeDescriptionSyntaxChecker.java:163)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.api.ldap.model.entry.Value.<init>(Value.java:285)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.createStringValue(DefaultAttribute.java:449)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.add(DefaultAttribute.java:1104)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.shared.SchemaService.generateAttributeTypes(SchemaService.java:144)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.shared.SchemaService.generateSchemaSubentry(SchemaService.java:258)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.shared.SchemaService.getSubschemaEntry(SchemaService.java:432)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.schema.SchemaInterceptor.search(SchemaInterceptor.java:1519)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.authz.DefaultAuthorizationInterceptor.search(DefaultAuthorizationInterceptor.java:354)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.authz.AciAuthorizationInterceptor.search(AciAuthorizationInterceptor.java:1247)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.authn.AuthenticationInterceptor.search(AuthenticationInterceptor.java:1394)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.normalization.NormalizationInterceptor.search(NormalizationInterceptor.java:414)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.DefaultOperationManager.search(DefaultOperationManager.java:1781)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.core.shared.DefaultCoreSession.search(DefaultCoreSession.java:1193)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.doSimpleSearch(SearchRequestHandler.java:810)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handleIgnoringReferrals(SearchRequestHandler.java:1164)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handleWithReferrals(SearchRequestHandler.java:1262)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handle(SearchRequestHandler.java:212)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handle(SearchRequestHandler.java:92)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:207)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:56)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:243)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.directory.server.ldap.LdapProtocolHandler.messageReceived(LdapProtocolHandler.java:223)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at
> org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1019)
> STATUS | wrapper | 2018/10/29 12:56:37 | TERM trapped. Shutting down.
> INFO | jvm 1 | 2018/10/29 12:56:46 | Error in WrapperListener.stop
> callback. java.lang.OutOfMemoryError: Java heap space
> INFO | jvm 1 | 2018/10/29 12:56:46 | java.lang.OutOfMemoryError: Java
> heap space
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> java.util.HashMap.resize(HashMap.java:704)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> java.util.HashMap.putVal(HashMap.java:663)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> java.util.HashMap.put(HashMap.java:612)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> jdbm.recman.RecordFile.commit(RecordFile.java:314)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> jdbm.recman.PageManager.close(PageManager.java:329)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> jdbm.recman.BaseRecordManager.close(BaseRecordManager.java:164)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> org.apache.directory.server.ldap.replication.provider.ReplicaEventLog.stop(ReplicaEventLog.java:226)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> org.apache.directory.server.ldap.replication.provider.SyncReplRequestHandler.stop(SyncReplRequestHandler.java:274)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> org.apache.directory.server.ldap.LdapServer.stop(LdapServer.java:668)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> org.apache.directory.server.ApacheDsService.stop(ApacheDsService.java:610)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.stop(ApacheDsTanukiWrapper.java:205)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at
> org.tanukisoftware.wrapper.WrapperManager$13.run(WrapperManager.java:3134)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org
For additional commands, e-mail: dev-h...@directory.apache.org
