[jira] [Commented] (HBASE-12981) FSHLog: UNEXPECTED java.lang.ArrayIndexOutOfBoundsException: -4
[
https://issues.apache.org/jira/browse/HBASE-12981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309957#comment-14309957
]
Lars Hofhansl commented on HBASE-12981:
---
{code}
+ idx = Math.abs(idx);
{code}
Whoa, that can make things much worse, no?
> FSHLog: UNEXPECTED java.lang.ArrayIndexOutOfBoundsException: -4
> ---
>
> Key: HBASE-12981
> URL: https://issues.apache.org/jira/browse/HBASE-12981
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 0.98.10
>Reporter: stack
>Assignee: stack
> Fix For: 0.98.11
>
> Attachments: 12981.0.98.txt
>
>
> A user reported the below. It happens after the RS has been running a while.
> 015-01-20 22:33:23,031 ERROR org.apache.hadoop.hbase.regionserver.wal.FSHLog:
> UNEXPECTED
> java.lang.ArrayIndexOutOfBoundsException: -4
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1149)
> at java.lang.Thread.run(Thread.java:745)
> 2015-01-20 22:33:23,035 INFO org.apache.hadoop.hbase.regionserver.wal.FSHLog:
> regionserver60020-WAL.AsyncWriter exiting
> ##
> Similarly on Node 23 - on 12-20-2014 05:13:
> 2014-12-20 05:13:40,715 ERROR
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: UNEXPECTED
> java.lang.ArrayIndexOutOfBoundsException: -3
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1149)
> at java.lang.Thread.run(Thread.java:745)
> ###
> Looking in code, I can't see how this could come about other than our write
> seqid ran over the top of a long (unlikely). I think this a 0.98 issue since
> 1.0+ is different here. It does:
> int index = Math.abs(this.syncRunnerIndex++) % this.syncRunners.length;
> I'm going to add logging of the circumstance that produces a negative index
> and then defense against our using negative indices; there could be more
> going on in here, more than I can see.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12981) FSHLog: UNEXPECTED java.lang.ArrayIndexOutOfBoundsException: -4
[ https://issues.apache.org/jira/browse/HBASE-12981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310062#comment-14310062 ] Enis Soztutar commented on HBASE-12981: --- Could this be HBASE-11200 ? We have seen this in action using earlier 0.98 versions. > FSHLog: UNEXPECTED java.lang.ArrayIndexOutOfBoundsException: -4 > --- > > Key: HBASE-12981 > URL: https://issues.apache.org/jira/browse/HBASE-12981 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.98.10 >Reporter: stack >Assignee: stack > Fix For: 0.98.11 > > Attachments: 12981.0.98.txt > > > A user reported the below. It happens after the RS has been running a while. > 015-01-20 22:33:23,031 ERROR org.apache.hadoop.hbase.regionserver.wal.FSHLog: > UNEXPECTED > java.lang.ArrayIndexOutOfBoundsException: -4 > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1149) > at java.lang.Thread.run(Thread.java:745) > 2015-01-20 22:33:23,035 INFO org.apache.hadoop.hbase.regionserver.wal.FSHLog: > regionserver60020-WAL.AsyncWriter exiting > ## > Similarly on Node 23 - on 12-20-2014 05:13: > 2014-12-20 05:13:40,715 ERROR > org.apache.hadoop.hbase.regionserver.wal.FSHLog: UNEXPECTED > java.lang.ArrayIndexOutOfBoundsException: -3 > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1149) > at java.lang.Thread.run(Thread.java:745) > ### > Looking in code, I can't see how this could come about other than our write > seqid ran over the top of a long (unlikely). I think this a 0.98 issue since > 1.0+ is different here. It does: > int index = Math.abs(this.syncRunnerIndex++) % this.syncRunners.length; > I'm going to add logging of the circumstance that produces a negative index > and then defense against our using negative indices; there could be more > going on in here, more than I can see. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12981) FSHLog: UNEXPECTED java.lang.ArrayIndexOutOfBoundsException: -4
[
https://issues.apache.org/jira/browse/HBASE-12981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310096#comment-14310096
]
Sean Busbey commented on HBASE-12981:
-
{quote}
{code}
+ idx = Math.abs(idx);
{code}
Whoa, that can make things much worse, no?
{quote}
At this point idx should be in the range (-asyncSyncers.length,
asyncSyncers.length) so taking the absolute value should be okay.
Usually I'd do a < 0 check and add the devisor, but I don't think it makes a
difference unless this is a very hot code path.
> FSHLog: UNEXPECTED java.lang.ArrayIndexOutOfBoundsException: -4
> ---
>
> Key: HBASE-12981
> URL: https://issues.apache.org/jira/browse/HBASE-12981
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 0.98.10
>Reporter: stack
>Assignee: stack
> Fix For: 0.98.11
>
> Attachments: 12981.0.98.txt
>
>
> A user reported the below. It happens after the RS has been running a while.
> 015-01-20 22:33:23,031 ERROR org.apache.hadoop.hbase.regionserver.wal.FSHLog:
> UNEXPECTED
> java.lang.ArrayIndexOutOfBoundsException: -4
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1149)
> at java.lang.Thread.run(Thread.java:745)
> 2015-01-20 22:33:23,035 INFO org.apache.hadoop.hbase.regionserver.wal.FSHLog:
> regionserver60020-WAL.AsyncWriter exiting
> ##
> Similarly on Node 23 - on 12-20-2014 05:13:
> 2014-12-20 05:13:40,715 ERROR
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: UNEXPECTED
> java.lang.ArrayIndexOutOfBoundsException: -3
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1149)
> at java.lang.Thread.run(Thread.java:745)
> ###
> Looking in code, I can't see how this could come about other than our write
> seqid ran over the top of a long (unlikely). I think this a 0.98 issue since
> 1.0+ is different here. It does:
> int index = Math.abs(this.syncRunnerIndex++) % this.syncRunners.length;
> I'm going to add logging of the circumstance that produces a negative index
> and then defense against our using negative indices; there could be more
> going on in here, more than I can see.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12981) FSHLog: UNEXPECTED java.lang.ArrayIndexOutOfBoundsException: -4
[ https://issues.apache.org/jira/browse/HBASE-12981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310116#comment-14310116 ] stack commented on HBASE-12981: --- Thanks [~enis] I think you are right. I was looking at wrong 0.98 version. This is a patched 0.98.1 (CDH5.1.3). It doesn't have HBASE-11200. Let me close this as a duplicate (Thanks for reviews [~lhofhansl] and [~busbey] -- agree w/ Sean). > FSHLog: UNEXPECTED java.lang.ArrayIndexOutOfBoundsException: -4 > --- > > Key: HBASE-12981 > URL: https://issues.apache.org/jira/browse/HBASE-12981 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.98.10 >Reporter: stack >Assignee: stack > Fix For: 0.98.11 > > Attachments: 12981.0.98.txt > > > A user reported the below. It happens after the RS has been running a while. > 015-01-20 22:33:23,031 ERROR org.apache.hadoop.hbase.regionserver.wal.FSHLog: > UNEXPECTED > java.lang.ArrayIndexOutOfBoundsException: -4 > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1149) > at java.lang.Thread.run(Thread.java:745) > 2015-01-20 22:33:23,035 INFO org.apache.hadoop.hbase.regionserver.wal.FSHLog: > regionserver60020-WAL.AsyncWriter exiting > ## > Similarly on Node 23 - on 12-20-2014 05:13: > 2014-12-20 05:13:40,715 ERROR > org.apache.hadoop.hbase.regionserver.wal.FSHLog: UNEXPECTED > java.lang.ArrayIndexOutOfBoundsException: -3 > at > org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1149) > at java.lang.Thread.run(Thread.java:745) > ### > Looking in code, I can't see how this could come about other than our write > seqid ran over the top of a long (unlikely). I think this a 0.98 issue since > 1.0+ is different here. It does: > int index = Math.abs(this.syncRunnerIndex++) % this.syncRunners.length; > I'm going to add logging of the circumstance that produces a negative index > and then defense against our using negative indices; there could be more > going on in here, more than I can see. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12981) FSHLog: UNEXPECTED java.lang.ArrayIndexOutOfBoundsException: -4
[
https://issues.apache.org/jira/browse/HBASE-12981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310226#comment-14310226
]
Hadoop QA commented on HBASE-12981:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697132/12981.0.98.txt
against 0.98 branch at commit 75148385ee5b2065992aea19a810436196576f20.
ATTACHMENT ID: 12697132
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
23 warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 42 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.TestChoreService.testCorePoolIncrease(TestChoreService.java:395)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/patchReleaseAuditWarnings.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12722//console
This message is automatically generated.
> FSHLog: UNEXPECTED java.lang.ArrayIndexOutOfBoundsException: -4
> ---
>
> Key: HBASE-12981
> URL: https://issues.apache.org/jira/browse/HBASE-12981
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 0.98.1
>Reporter: stack
>Assignee: stack
> Attachments: 12981.0.98.txt
>
>
> A user reported the below. It happens after the RS has been running a while.
> 015-01-20 22:33:23,031 ERROR org.apache.hadoop.hbase.regionserver.wal.FSHLog:
> UNEXPECTED
> java.lang.ArrayIndexOutOfBoundsException: -4
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1149)
> at java.lang.
