[ https://issues.apache.org/jira/browse/OAK-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15545205#comment-15545205 ]
Francesco Mari commented on OAK-4884: ------------------------------------- In r1763269 I made bind, connect and close synchronous for the standby client, server and proxy. I also added a bunch of debug messages along the way and reorganised some code. I moved NetworkErrorProxy to the org.apache.jackrabbit.oak.segment.standby package, so that its debug messages are logged as part of our tests. I also refined the useProxy method in BrokenNetworkTest regarding the acquisition and releasing of the standby client, server and proxy. I am already aware that this change will increase the build time of one or two minutes, but I am going to keep it this way as long as we are seeing intermittent failures in the cold standby tests. > Test failure: > org.apache.jackrabbit.oak.segment.standby.ExternalSharedStoreIT.testProxyFlippedIntermediateByteChange2 > --------------------------------------------------------------------------------------------------------------------- > > Key: OAK-4884 > URL: https://issues.apache.org/jira/browse/OAK-4884 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-tar > Reporter: Francesco Mari > Assignee: Francesco Mari > Fix For: Segment Tar 0.0.14 > > > The test > {{org.apache.jackrabbit.oak.segment.standby.ExternalSharedStoreIT.testProxyFlippedIntermediateByteChange2}} > fails intermittently with the following error: > {noformat} > expected: org.apache.jackrabbit.oak.segment.SegmentNodeState<{ root = { ... } > }> but was: org.apache.jackrabbit.oak.segment.SegmentNodeState<{ root = { ... > } }> > {noformat} > The cause of the issue seems to be the proxy holding to a port after it's > been asked to disconnect: > {noformat} > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) ~[na:1.7.0_80] > at sun.nio.ch.Net.bind(Net.java:463) ~[na:1.7.0_80] > at sun.nio.ch.Net.bind(Net.java:455) ~[na:1.7.0_80] > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) > ~[na:1.7.0_80] > at > io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:505) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:490) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:965) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:210) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:353) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408) > ~[netty-common-4.0.41.Final.jar:4.0.41.Final] > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) > ~[netty-common-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > ~[netty-common-4.0.41.Final.jar:4.0.41.Final] > at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80] > {noformat} > After that, the client fails when trying to connect to the server through the > proxy: > {noformat} > 09:39:11.795 ERROR [main] StandbyClientSync.java:160 Failed > synchronizing state. > io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection > refused: /127.0.0.1:41866 > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > ~[na:1.7.0_80] > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) > ~[na:1.7.0_80] > at > io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:552) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:466) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) > ~[netty-transport-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) > ~[netty-common-4.0.41.Final.jar:4.0.41.Final] > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > ~[netty-common-4.0.41.Final.jar:4.0.41.Final] > at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)