[jira] [Commented] (OAK-4917) DefaultStandbyReferencesReader does not re-attempt to read segment upon failure
[
https://issues.apache.org/jira/browse/OAK-4917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562611#comment-15562611
]
Timothee Maret commented on OAK-4917:
-
[~alexparvulescu], [~frm], [~mduerig] could you review the patch ?
> DefaultStandbyReferencesReader does not re-attempt to read segment upon
> failure
>
>
> Key: OAK-4917
> URL: https://issues.apache.org/jira/browse/OAK-4917
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: Segment Tar 0.0.14
>Reporter: Timothee Maret
>Assignee: Timothee Maret
> Fix For: Segment Tar 0.0.16
>
> Attachments: OAK-4917.patch
>
>
> This following logs occurred while running integration testing.
> {code}
> 07.10.2016 16:50:30.882 *DEBUG* [nioEventLoopGroup-3-2]
> org.apache.jackrabbit.oak.segment.standby.codec.GetHeadResponseEncoder
> Sending head 9d91bcd2-c4e0-4411-a0e0-d358fff02afc.0020 to client
> qastandby1
> 07.10.2016 16:50:30.885 *DEBUG* [nioEventLoopGroup-3-2]
> org.apache.jackrabbit.oak.segment.standby.codec.RequestDecoder Parsed 'get
> references' message
> 07.10.2016 16:50:30.885 *DEBUG* [nioEventLoopGroup-3-2]
> org.apache.jackrabbit.oak.segment.standby.server.GetReferencesRequestHandler
> Reading references of segment 9d91bcd2-c4e0-4411-a0e0-d358fff02afc for client
> qastandby1
> 07.10.2016 16:50:30.886 *WARN* [nioEventLoopGroup-3-2]
> org.apache.jackrabbit.oak.segment.standby.server.DefaultStandbyReferencesReader
> Unable to read segment 9d91bcd2-c4e0-4411-a0e0-d358fff02afc
> org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment
> 9d91bcd2-c4e0-4411-a0e0-d358fff02afc not found
> at
> org.apache.jackrabbit.oak.segment.file.FileStore$18.call(FileStore.java:1364)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore$18.call(FileStore.java:1304)
> at
> org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1013)
> at
> org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:974)
> at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:285)
> at
> org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:92)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1304)
> at
> org.apache.jackrabbit.oak.segment.standby.server.DefaultStandbyReferencesReader.readSegment(DefaultStandbyReferencesReader.java:66)
> at
> org.apache.jackrabbit.oak.segment.standby.server.DefaultStandbyReferencesReader.readReferences(DefaultStandbyReferencesReader.java:49)
> at
> org.apache.jackrabbit.oak.segment.standby.server.GetReferencesRequestHandler.channelRead0(GetReferencesRequestHandler.java:41)
> at
> org.apache.jackrabbit.oak.segment.standby.server.GetReferencesRequestHandler.channelRead0(GetReferencesRequestHandler.java:27)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:108)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:108)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:108)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
>
[jira] [Commented] (OAK-4917) DefaultStandbyReferencesReader does not re-attempt to read segment upon failure
[
https://issues.apache.org/jira/browse/OAK-4917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562576#comment-15562576
]
Timothee Maret commented on OAK-4917:
-
Discussing this with [~alexparvulescu], it seems this is expected and in fact
the cold stdby code deal with it by re-trying to load in a loop with a delay.
This looping behaviour is already implemented in the
{{DefaultStandbySegmentReader}} [0]. However the
{{DefaultStandbyReferencesReader}} does not contains the loop, causing the
issue.
[0]
https://github.com/apache/jackrabbit-oak/blob/066edc9eb5ef500339de0ebf7859e7ee9dd3ab4b/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/standby/server/DefaultStandbySegmentReader.java
> DefaultStandbyReferencesReader does not re-attempt to read segment upon
> failure
>
>
> Key: OAK-4917
> URL: https://issues.apache.org/jira/browse/OAK-4917
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: Segment Tar 0.0.14
>Reporter: Timothee Maret
> Fix For: Segment Tar 0.0.16
>
>
> This following logs occurred while running integration testing.
> {code}
> 07.10.2016 16:50:30.882 *DEBUG* [nioEventLoopGroup-3-2]
> org.apache.jackrabbit.oak.segment.standby.codec.GetHeadResponseEncoder
> Sending head 9d91bcd2-c4e0-4411-a0e0-d358fff02afc.0020 to client
> qastandby1
> 07.10.2016 16:50:30.885 *DEBUG* [nioEventLoopGroup-3-2]
> org.apache.jackrabbit.oak.segment.standby.codec.RequestDecoder Parsed 'get
> references' message
> 07.10.2016 16:50:30.885 *DEBUG* [nioEventLoopGroup-3-2]
> org.apache.jackrabbit.oak.segment.standby.server.GetReferencesRequestHandler
> Reading references of segment 9d91bcd2-c4e0-4411-a0e0-d358fff02afc for client
> qastandby1
> 07.10.2016 16:50:30.886 *WARN* [nioEventLoopGroup-3-2]
> org.apache.jackrabbit.oak.segment.standby.server.DefaultStandbyReferencesReader
> Unable to read segment 9d91bcd2-c4e0-4411-a0e0-d358fff02afc
> org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment
> 9d91bcd2-c4e0-4411-a0e0-d358fff02afc not found
> at
> org.apache.jackrabbit.oak.segment.file.FileStore$18.call(FileStore.java:1364)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore$18.call(FileStore.java:1304)
> at
> org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1013)
> at
> org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:974)
> at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:285)
> at
> org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:92)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1304)
> at
> org.apache.jackrabbit.oak.segment.standby.server.DefaultStandbyReferencesReader.readSegment(DefaultStandbyReferencesReader.java:66)
> at
> org.apache.jackrabbit.oak.segment.standby.server.DefaultStandbyReferencesReader.readReferences(DefaultStandbyReferencesReader.java:49)
> at
> org.apache.jackrabbit.oak.segment.standby.server.GetReferencesRequestHandler.channelRead0(GetReferencesRequestHandler.java:41)
> at
> org.apache.jackrabbit.oak.segment.standby.server.GetReferencesRequestHandler.channelRead0(GetReferencesRequestHandler.java:27)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:108)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:108)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
>
[jira] [Commented] (OAK-4917) DefaultStandbyReferencesReader does not re-attempt to read segment upon failure
[
https://issues.apache.org/jira/browse/OAK-4917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562564#comment-15562564
]
Timothee Maret commented on OAK-4917:
-
Looking a bit further, both readers ({{DefaultStandbyHeadReader}} and
{{DefaultStandbyReferencesReader}}) do go through the
{{org.apache.jackrabbit.oak.segment.file.FileStore}} in order to fetch data.
However, both methods seem to use different caches.
The root cause of this issue seems to be those two cache not being in sync.
{{org.apache.jackrabbit.oak.segment.file.FileStore#getHead}} does use the
CachingSegmentReader segment Reader.
{{org.apache.jackrabbit.oak.segment.file.FileStore#readSegment}} does use the
SegmentCache segment cache in the FileStore.
> DefaultStandbyReferencesReader does not re-attempt to read segment upon
> failure
>
>
> Key: OAK-4917
> URL: https://issues.apache.org/jira/browse/OAK-4917
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
>Affects Versions: Segment Tar 0.0.14
>Reporter: Timothee Maret
> Fix For: Segment Tar 0.0.16
>
>
> This following logs occurred while running integration testing.
> {code}
> 07.10.2016 16:50:30.882 *DEBUG* [nioEventLoopGroup-3-2]
> org.apache.jackrabbit.oak.segment.standby.codec.GetHeadResponseEncoder
> Sending head 9d91bcd2-c4e0-4411-a0e0-d358fff02afc.0020 to client
> qastandby1
> 07.10.2016 16:50:30.885 *DEBUG* [nioEventLoopGroup-3-2]
> org.apache.jackrabbit.oak.segment.standby.codec.RequestDecoder Parsed 'get
> references' message
> 07.10.2016 16:50:30.885 *DEBUG* [nioEventLoopGroup-3-2]
> org.apache.jackrabbit.oak.segment.standby.server.GetReferencesRequestHandler
> Reading references of segment 9d91bcd2-c4e0-4411-a0e0-d358fff02afc for client
> qastandby1
> 07.10.2016 16:50:30.886 *WARN* [nioEventLoopGroup-3-2]
> org.apache.jackrabbit.oak.segment.standby.server.DefaultStandbyReferencesReader
> Unable to read segment 9d91bcd2-c4e0-4411-a0e0-d358fff02afc
> org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment
> 9d91bcd2-c4e0-4411-a0e0-d358fff02afc not found
> at
> org.apache.jackrabbit.oak.segment.file.FileStore$18.call(FileStore.java:1364)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore$18.call(FileStore.java:1304)
> at
> org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1013)
> at
> org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:974)
> at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:285)
> at
> org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:92)
> at
> org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1304)
> at
> org.apache.jackrabbit.oak.segment.standby.server.DefaultStandbyReferencesReader.readSegment(DefaultStandbyReferencesReader.java:66)
> at
> org.apache.jackrabbit.oak.segment.standby.server.DefaultStandbyReferencesReader.readReferences(DefaultStandbyReferencesReader.java:49)
> at
> org.apache.jackrabbit.oak.segment.standby.server.GetReferencesRequestHandler.channelRead0(GetReferencesRequestHandler.java:41)
> at
> org.apache.jackrabbit.oak.segment.standby.server.GetReferencesRequestHandler.channelRead0(GetReferencesRequestHandler.java:27)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:108)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:108)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)
> at
>
