[jira] [Commented] (HBASE-10549) when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop.
[
https://issues.apache.org/jira/browse/HBASE-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13919306#comment-13919306
]
Hadoop QA commented on HBASE-10549:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632481/HBASE-10549-trunk.patch
against trunk revision .
ATTACHMENT ID: 12632481
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8876//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8876//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8876//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8876//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8876//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8876//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8876//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8876//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8876//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8876//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8876//console
This message is automatically generated.
> when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop.
>
>
> Key: HBASE-10549
> URL: https://issues.apache.org/jira/browse/HBASE-10549
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.94.11
>Reporter: yuanxinen
> Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
>
> Attachments: HBASE-10549-trunk.patch
>
>
> First,I explan my test steps.
> 1.importtsv
> 2.split the region
> 3.delete the region info from .META.(make a hole)
> 4.LoadIncrementalHFiles (this step will hung up in an infinite loop)
> I check the log,there are two issues
> 1.it create _tmp folder in an infinite loop.
> hdfs://hacluster/output3/i/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/test_table,136.bottom
> 2.when slpliting the hfile,it put the first line data(1211) into two
> files(top and bottom)
> Input
> File=hdfs://hacluster/output3/i/3ac6ec287c644a8fb72d96b13e31f576,outFile=hdfs://hacluster/output3/i/_tmp/test_table,2.top,KeyValue=1211/i:value/1390469306407/Put/vlen=1/ts=0
> Input
> File=hdfs://hacluster/output3/i/3ac6ec287c644a8fb72d96b13e31f576,outFile=hdfs://hacluster/output3/i/_tmp/test_table,2.bottom,KeyValue=1211/i:value/1390469306407/Put/vlen=1/ts=0
> and then I check the code.
> So I think before spliting the hfile,we should check the consistency of
> startkey and endkey,if something wrong,we should throw the exception,and stop
> LoadIncrementalHFiles.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10549) when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop.
[ https://issues.apache.org/jira/browse/HBASE-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915376#comment-13915376 ] yuanxinen commented on HBASE-10549: --- yes,I am preparing the patch,and I will submit it. > when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop. > > > Key: HBASE-10549 > URL: https://issues.apache.org/jira/browse/HBASE-10549 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.11 >Reporter: yuanxinen > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10549) when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop.
[ https://issues.apache.org/jira/browse/HBASE-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904636#comment-13904636 ] Nick Dimiduk commented on HBASE-10549: -- Thanks for the report [~Auraro]! Would you mind attaching a patch vs trunk for the sake of our QA bot? We can work with it from there to ensure the fix is backported to all active versions. We would also appreciate an updated test that covers this scenario. TestLoadIncrementalHFiles and TestLoadIncrementalHFilesSplitRecovery look like a good place to start. > when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop. > > > Key: HBASE-10549 > URL: https://issues.apache.org/jira/browse/HBASE-10549 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.11 >Reporter: yuanxinen > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10549) when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop.
[
https://issues.apache.org/jira/browse/HBASE-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903010#comment-13903010
]
yuanxinen commented on HBASE-10549:
---
---add check code---
if (!(Bytes.compareTo(startEndKeys.getSecond()[idx],
startEndKeys.getFirst()[idx + 1]) == 0)){
throw new Exception("Find a inconsistenty in a table,please use
hbck tool to fix it first.");
}
-
boolean lastKeyInRange =
Bytes.compareTo(last, startEndKeys.getSecond()[idx]) < 0
|| Bytes.equals(startEndKeys.getSecond()[idx],
HConstants.EMPTY_BYTE_ARRAY);
if (!lastKeyInRange) {
List lqis =
this.splitStoreFile(item, table,
startEndKeys.getFirst()[indexForCallable], startEndKeys
.getSecond()[indexForCallable]);
return lqis;
}
> when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop.
>
>
> Key: HBASE-10549
> URL: https://issues.apache.org/jira/browse/HBASE-10549
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.94.11
>Reporter: yuanxinen
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HBASE-10549) when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop.
[ https://issues.apache.org/jira/browse/HBASE-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13903007#comment-13903007 ] yuanxinen commented on HBASE-10549: --- First,I explan my test steps. 1.importtsv 2.split the region 3.delete the region info from .META.(make a hole) 4.LoadIncrementalHFiles (this step will hung up in an infinite loop) I check the log,there are two issues 1.it create _tmp folder in an infinite loop. hdfs://hacluster/output3/i/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/test_table,136.bottom 2.when slpliting the hfile,it put the first line data(1211) into two files(top and bottom) Input File=hdfs://hacluster/output3/i/3ac6ec287c644a8fb72d96b13e31f576,outFile=hdfs://hacluster/output3/i/_tmp/test_table,2.top,KeyValue=1211/i:value/1390469306407/Put/vlen=1/ts=0 Input File=hdfs://hacluster/output3/i/3ac6ec287c644a8fb72d96b13e31f576,outFile=hdfs://hacluster/output3/i/_tmp/test_table,2.bottom,KeyValue=1211/i:value/1390469306407/Put/vlen=1/ts=0 and then I check the code. So I think before spliting the hfile,we should check the consistency of startkey and endkey,if something wrong,we should throw the exception,and stop LoadIncrementalHFiles. > when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop. > > > Key: HBASE-10549 > URL: https://issues.apache.org/jira/browse/HBASE-10549 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.11 >Reporter: yuanxinen > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10549) when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop.
[ https://issues.apache.org/jira/browse/HBASE-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902517#comment-13902517 ] Jerry He commented on HBASE-10549: -- Can you give more details, e.g. what is a hole? > when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop. > > > Key: HBASE-10549 > URL: https://issues.apache.org/jira/browse/HBASE-10549 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.11 >Reporter: yuanxinen > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10549) when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop.
[ https://issues.apache.org/jira/browse/HBASE-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902368#comment-13902368 ] yuanxinen commented on HBASE-10549: --- when there is a hole,it will loop to split the hfile,and create more _tmp in hdfs,this is dangerous. I think if there is a hole,it should stop splitting and throw an exception. > when there is a hole,LoadIncrementalHFiles will hung up in an infinite loop. > > > Key: HBASE-10549 > URL: https://issues.apache.org/jira/browse/HBASE-10549 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.11 >Reporter: yuanxinen >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
