[jira] [Commented] (HBASE-4348) Add metrics for regions in transition
[ https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232484#comment-13232484 ] jirapos...@reviews.apache.org commented on HBASE-4348: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4402/ --- (Updated 2012-03-19 06:48:19.158231) Review request for hbase. Summary (updated) --- This patch is for addinf Region in transition metrics to the HMaster metrics system. It also adds these metrics in the master ui, in the Region in transition section. I have attached the proposed new format in the jira 4348. This addresses bug HBase-4348. https://issues.apache.org/jira/browse/HBase-4348 Diffs - src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon 0dc0691 src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java ae468ca src/main/java/org/apache/hadoop/hbase/master/HMaster.java c4b4d30 src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java 83abc52 src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java d68ce33 Diff: https://reviews.apache.org/r/4402/diff Testing --- Ran on a 5 node cluster and kill region servers randomly to observe the changes in the RIT metrics as emitted out by the Master's mxbean; mvn test passes without any failure. Thanks, Himanshu > Add metrics for regions in transition > - > > Key: HBASE-4348 > URL: https://issues.apache.org/jira/browse/HBASE-4348 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: Himanshu Vashishtha >Priority: Minor > Labels: noob > Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, > RITs.png, RegionInTransitions2.png, metrics-v2.patch > > > The following metrics would be useful for monitoring the master: > - the number of regions in transition > - the number of regions in transition that have been in transition for more > than a minute > - how many seconds has the oldest region-in-transition been in transition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5599) The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_I
[ https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fulin wang updated HBASE-5599: -- Description: The hbck tool can not fix the six scenarios. 1. Version file does not exist in root dir. Fix: I try to create a version file by 'FSUtils.setVersion' method. 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any region server. Fix: I get region info form the hdfs file, this region info write to '.META.' table. 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] Fix: I get region info form the hdfs file, this region info write to '.META.' table. 4. [REGIONNAME] should not be deployed according to META, but is deployed on [SERVERNAME] Fix: Close this region. 5. First region should start with an empty key. You need to create a new region and regioninfo in HDFS to plug the hole. Fix: The region info is not in hdfs and .META., so it create a empty region for this error. 6. There is a hole in the region chain between [KEY] and [KEY]. You need to create a new regioninfo and region dir in hdfs to plug the hole. Fix: The region info is not in hdfs and .META., so it create a empty region for this hole. was: The hbkc tool can not fix the six scenarios. 1. Version file does not exist in root dir. Fix: I try to create a version file by 'FSUtils.setVersion' method. 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any region server. Fix: I get region info form the hdfs file, this region info write to '.META.' table. 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] Fix: I get region info form the hdfs file, this region info write to '.META.' table. 4. [REGIONNAME] should not be deployed according to META, but is deployed on [SERVERNAME] Fix: Close this region. 5. First region should start with an empty key. You need to create a new region and regioninfo in HDFS to plug the hole. Fix: The region info is not in hdfs and .META., so it create a empty region for this error. 6. There is a hole in the region chain between [KEY] and [KEY]. You need to create a new regioninfo and region dir in hdfs to plug the hole. Fix: The region info is not in hdfs and .META., so it create a empty region for this hole. Summary: The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. (was: The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN.) > The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, > NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, > FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. > - > > Key: HBASE-5599 > URL: https://issues.apache.org/jira/browse/HBASE-5599 > Project: HBase > Issue Type: New Feature > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Fix For: 0.90.6 > > Attachments: hbase-5599-0.90.patch > > > The hbck tool can not fix the six scenarios. > 1. Version file does not exist in root dir. >Fix: I try to create a version file by 'FSUtils.setVersion' method. > > 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any > region server. >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 4. [REGIONNAME] should not be deployed according to META, but is deployed on > [SERVERNAME] >Fix: Close this region. > > 5. First region should start with an empty key. You need to create a new > region and regioninfo in HDFS to plug the hole. >Fix: The region info is not in hdfs and .META., so it create a empty > region for this error. > 6. There is a hole in the region chain between [KEY] and [KEY]. You need to > create a new regioninfo and region dir in hdfs to plug the hole. > Fix: The region info is not in hdfs and .META., so it create a empty region > for this hole. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5599) The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE
[ https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232476#comment-13232476 ] ramkrishna.s.vasudevan commented on HBASE-5599: --- @Fulin If you feel it is not present in HBASE-5128 pls add a comment in that JIRA. Jonathan can take a look at it and we can take your patch there and integrate it to 0.90? Is that fine with you? > The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, > NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, > FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. > - > > Key: HBASE-5599 > URL: https://issues.apache.org/jira/browse/HBASE-5599 > Project: HBase > Issue Type: New Feature > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Fix For: 0.90.6 > > Attachments: hbase-5599-0.90.patch > > > The hbkc tool can not fix the six scenarios. > 1. Version file does not exist in root dir. >Fix: I try to create a version file by 'FSUtils.setVersion' method. > > 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any > region server. >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 4. [REGIONNAME] should not be deployed according to META, but is deployed on > [SERVERNAME] >Fix: Close this region. > > 5. First region should start with an empty key. You need to create a new > region and regioninfo in HDFS to plug the hole. >Fix: The region info is not in hdfs and .META., so it create a empty > region for this error. > 6. There is a hole in the region chain between [KEY] and [KEY]. You need to > create a new regioninfo and region dir in hdfs to plug the hole. > Fix: The region info is not in hdfs and .META., so it create a empty region > for this hole. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5599) The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE
[ https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232475#comment-13232475 ] fulin wang commented on HBASE-5599: --- Yes, I referring to HBASE-5128. I want to know when Jonathan H will check in 0.90? > The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, > NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, > FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. > - > > Key: HBASE-5599 > URL: https://issues.apache.org/jira/browse/HBASE-5599 > Project: HBase > Issue Type: New Feature > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Fix For: 0.90.6 > > Attachments: hbase-5599-0.90.patch > > > The hbkc tool can not fix the six scenarios. > 1. Version file does not exist in root dir. >Fix: I try to create a version file by 'FSUtils.setVersion' method. > > 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any > region server. >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 4. [REGIONNAME] should not be deployed according to META, but is deployed on > [SERVERNAME] >Fix: Close this region. > > 5. First region should start with an empty key. You need to create a new > region and regioninfo in HDFS to plug the hole. >Fix: The region info is not in hdfs and .META., so it create a empty > region for this error. > 6. There is a hole in the region chain between [KEY] and [KEY]. You need to > create a new regioninfo and region dir in hdfs to plug the hole. > Fix: The region info is not in hdfs and .META., so it create a empty region > for this hole. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4348) Add metrics for regions in transition
[ https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-4348: --- Attachment: RegionInTransitions2.png > Add metrics for regions in transition > - > > Key: HBASE-4348 > URL: https://issues.apache.org/jira/browse/HBASE-4348 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: Himanshu Vashishtha >Priority: Minor > Labels: noob > Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, > RITs.png, RegionInTransitions2.png, metrics-v2.patch > > > The following metrics would be useful for monitoring the master: > - the number of regions in transition > - the number of regions in transition that have been in transition for more > than a minute > - how many seconds has the oldest region-in-transition been in transition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4348) Add metrics for regions in transition
[ https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232468#comment-13232468 ] Himanshu Vashishtha commented on HBASE-4348: Thanks for the review Jon. I incorporated your suggestions, and have tested this new patch on a 5 node cluster and randomly killing regions servers to bring some regions in transition, and noting the change in metrics. I wonder how the metrics for RIT can be tested by some simple unit tests. Attached is the new UI for region in transition section on the master UI. I will upload a review request on the board soon. > Add metrics for regions in transition > - > > Key: HBASE-4348 > URL: https://issues.apache.org/jira/browse/HBASE-4348 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: Himanshu Vashishtha >Priority: Minor > Labels: noob > Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, > RITs.png, metrics-v2.patch > > > The following metrics would be useful for monitoring the master: > - the number of regions in transition > - the number of regions in transition that have been in transition for more > than a minute > - how many seconds has the oldest region-in-transition been in transition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232467#comment-13232467 ] Hadoop QA commented on HBASE-5593: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518870/HBASE-5593.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1224//console This message is automatically generated. > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232466#comment-13232466 ] Zhihong Yu commented on HBASE-5593: --- @Stack, @Ram: Please comment on latest patch. > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang updated HBASE-5593: - Attachment: HBASE-5593.patch > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang updated HBASE-5593: - Status: Patch Available (was: Open) Changed name of test as per Ted's comment. > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang updated HBASE-5593: - Attachment: (was: HBASE-5593-3.patch) > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang updated HBASE-5593: - Attachment: (was: HBASE-5593-2.patch) > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang updated HBASE-5593: - Attachment: (was: HBASE-5593.patch) > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5593: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518868/HBASE-5593-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1223//console This message is automatically generated.) > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593-2.patch, HBASE-5593-3.patch, HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5593: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518698/HBASE-5593.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1206//console This message is automatically generated.) > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593-2.patch, HBASE-5593-3.patch, HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5599) The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE
[ https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232457#comment-13232457 ] ramkrishna.s.vasudevan commented on HBASE-5599: --- @Fulin Yes, pls follow as Ted suggested. > The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, > NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, > FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. > - > > Key: HBASE-5599 > URL: https://issues.apache.org/jira/browse/HBASE-5599 > Project: HBase > Issue Type: New Feature > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Fix For: 0.90.6 > > Attachments: hbase-5599-0.90.patch > > > The hbkc tool can not fix the six scenarios. > 1. Version file does not exist in root dir. >Fix: I try to create a version file by 'FSUtils.setVersion' method. > > 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any > region server. >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 4. [REGIONNAME] should not be deployed according to META, but is deployed on > [SERVERNAME] >Fix: Close this region. > > 5. First region should start with an empty key. You need to create a new > region and regioninfo in HDFS to plug the hole. >Fix: The region info is not in hdfs and .META., so it create a empty > region for this error. > 6. There is a hole in the region chain between [KEY] and [KEY]. You need to > create a new regioninfo and region dir in hdfs to plug the hole. > Fix: The region info is not in hdfs and .META., so it create a empty region > for this hole. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232458#comment-13232458 ] Zhihong Yu commented on HBASE-5593: --- @David: Sorry for the comment about test category - I forgot this is for 0.90 only. How about naming testMasterOnlineRegions() testCorrectHostnameFormat() ? You can append ForOnlineRegions at the end if you want. > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593-2.patch, HBASE-5593-3.patch, HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5593: -- Status: Open (was: Patch Available) > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593-2.patch, HBASE-5593-3.patch, HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232454#comment-13232454 ] David S. Wang commented on HBASE-5593: -- Ted, The patch is meant to be applied to 0.90.x branch only. The QA robot tries to apply to trunk even though I marked this bug as only for 0.90.x, so it will fail the application. Also, I do not see any "@Category" tags in 0.90.x branch tests, though I see them in trunk. Do you still want me to add this tag? I will change the name of the test to testMasterOnlineRegionsCorrectFormat and upload a new patch with that change as soon as I get a response to the two aforementioned observations. > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593-2.patch, HBASE-5593-3.patch, HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-3996: -- Fix Version/s: (was: 0.94.0) 0.96.0 Hadoop Flags: Reviewed > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner >Assignee: Eran Kutner > Fix For: 0.96.0 > > Attachments: 3996-v2.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu reassigned HBASE-3996: - Assignee: Eran Kutner > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner >Assignee: Eran Kutner > Fix For: 0.96.0 > > Attachments: 3996-v2.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5599) The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE
[ https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232451#comment-13232451 ] Zhihong Yu commented on HBASE-5599: --- @Fulin: Ramkrishna was referring to HBASE-5128. Jonathan H would prepare a port of HBASE-5128 to 0.90 Please participate in the review of HBASE-5128 so that your fix becomes part of it. > The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, > NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, > FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. > - > > Key: HBASE-5599 > URL: https://issues.apache.org/jira/browse/HBASE-5599 > Project: HBase > Issue Type: New Feature > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Fix For: 0.90.6 > > Attachments: hbase-5599-0.90.patch > > > The hbkc tool can not fix the six scenarios. > 1. Version file does not exist in root dir. >Fix: I try to create a version file by 'FSUtils.setVersion' method. > > 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any > region server. >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 4. [REGIONNAME] should not be deployed according to META, but is deployed on > [SERVERNAME] >Fix: Close this region. > > 5. First region should start with an empty key. You need to create a new > region and regioninfo in HDFS to plug the hole. >Fix: The region info is not in hdfs and .META., so it create a empty > region for this error. > 6. There is a hole in the region chain between [KEY] and [KEY]. You need to > create a new regioninfo and region dir in hdfs to plug the hole. > Fix: The region info is not in hdfs and .META., so it create a empty region > for this hole. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[
https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232449#comment-13232449
]
Zhihong Yu commented on HBASE-5593:
---
One more thing, please label TestStrings class with:
{code}
@Category(SmallTests.class)
{code}
> Reverse DNS resolution in regionServerStartup() does not strip trailing dot
> ---
>
> Key: HBASE-5593
> URL: https://issues.apache.org/jira/browse/HBASE-5593
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.90.6
>Reporter: David S. Wang
>Assignee: David S. Wang
> Fix For: 0.90.7
>
> Attachments: HBASE-5593-2.patch, HBASE-5593-3.patch, HBASE-5593.patch
>
>
> HBASE-4109 covered the removal of trailing dots in PTR records from reverse
> DNS lookups. We seem to have missed a case in HMaster#regionServerStartup().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232448#comment-13232448 ] Zhihong Yu commented on HBASE-5593: --- Thanks for the quick turn-around. Was the patch generated based on latest trunk ? Please fix patch application conflict. For testMasterOnlineRegions(), can we give it a better name ? It only tests whether the name of host conforms to certain criterion. > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593-2.patch, HBASE-5593-3.patch, HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5568) Multi concurrent flushcache() for one region could cause data loss
[ https://issues.apache.org/jira/browse/HBASE-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232446#comment-13232446 ] Zhihong Yu commented on HBASE-5568: --- Integrated 0.92 patch to 0.92 branch. > Multi concurrent flushcache() for one region could cause data loss > -- > > Key: HBASE-5568 > URL: https://issues.apache.org/jira/browse/HBASE-5568 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0 > > Attachments: HBASE-5568-90.patch, HBASE-5568-92v2.patch, > HBASE-5568.patch, HBASE-5568.patch, HBASE-5568v2.patch > > > We could call HRegion#flushcache() concurrently now through > HRegionServer#splitRegion or HRegionServer#flushRegion by HBaseAdmin. > However, we find if HRegion#internalFlushcache() is called concurrently by > multi thread, HRegion.memstoreSize will be calculated wrong. > At the end of HRegion#internalFlushcache(), we will do > this.addAndGetGlobalMemstoreSize(-flushsize), but the flushsize may not the > actual memsize which flushed to hdfs. It cause HRegion.memstoreSize is > negative and prevent next flush if we close this region. > Logs in RS for region e9d827913a056e696c39bc569ea3 > 2012-03-11 16:31:36,690 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 > f99f., current region memstore size 128.0m > 2012-03-11 16:31:37,999 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf1/8162481165586107427, entries=153106, sequenceid=619316544, > memsize=59.6m, filesize=31.2m > 2012-03-11 16:31:38,830 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 > f99f., current region memstore size 134.8m > 2012-03-11 16:31:39,458 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf2/3425971951499794221, entries=230183, sequenceid=619316544, > memsize=68.5m, filesize=26.6m > 2012-03-11 16:31:39,459 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Finished memstore flush of ~128.1m for region > writetest1,,1331454657410.e9d827913a > 056e696c39bc569ea3f99f. in 2769ms, sequenceid=619316544, compaction > requested=false > 2012-03-11 16:31:39,459 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 > f99f., current region memstore size 6.8m > 2012-03-11 16:31:39,529 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf1/1811012969998104626, entries=8002, sequenceid=619332759, > memsize=3.1m, filesize=1.6m > 2012-03-11 16:31:39,640 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf2/770333473623552048, entries=12231, sequenceid=619332759, > memsize=3.6m, filesize=1.4m > 2012-03-11 16:31:39,641 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Finished memstore flush of ~134.8m for region > writetest1,,1331454657410.e9d827913a > 056e696c39bc569ea3f99f. in 811ms, sequenceid=619332759, compaction > requested=true > 2012-03-11 16:31:39,707 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf1/5656568849587368557, entries=119, sequenceid=619332979, > memsize=47.4k, filesize=25.6k > 2012-03-11 16:31:39,775 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf2/794343845650987521, entries=157, sequenceid=619332979, > memsize=47.8k, filesize=19.3k > 2012-03-11 16:31:39,777 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Finished memstore flush of ~6.8m for region > writetest1,,1331454657410.e9d827913a05 > 6e696c39bc569ea3f99f. in 318ms, sequenceid=619332979, compaction > requested=true -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232447#comment-13232447 ] Hadoop QA commented on HBASE-5593: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518868/HBASE-5593-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1223//console This message is automatically generated. > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593-2.patch, HBASE-5593-3.patch, HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang updated HBASE-5593: - Status: Patch Available (was: Open) > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593-2.patch, HBASE-5593-3.patch, HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang updated HBASE-5593: - Attachment: HBASE-5593-3.patch Fixed as per Ted's comments, thanks. > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593-2.patch, HBASE-5593-3.patch, HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5568) Multi concurrent flushcache() for one region could cause data loss
[ https://issues.apache.org/jira/browse/HBASE-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232444#comment-13232444 ] ramkrishna.s.vasudevan commented on HBASE-5568: --- Sorry Ted. I did not notice that. :( > Multi concurrent flushcache() for one region could cause data loss > -- > > Key: HBASE-5568 > URL: https://issues.apache.org/jira/browse/HBASE-5568 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0 > > Attachments: HBASE-5568-90.patch, HBASE-5568-92v2.patch, > HBASE-5568.patch, HBASE-5568.patch, HBASE-5568v2.patch > > > We could call HRegion#flushcache() concurrently now through > HRegionServer#splitRegion or HRegionServer#flushRegion by HBaseAdmin. > However, we find if HRegion#internalFlushcache() is called concurrently by > multi thread, HRegion.memstoreSize will be calculated wrong. > At the end of HRegion#internalFlushcache(), we will do > this.addAndGetGlobalMemstoreSize(-flushsize), but the flushsize may not the > actual memsize which flushed to hdfs. It cause HRegion.memstoreSize is > negative and prevent next flush if we close this region. > Logs in RS for region e9d827913a056e696c39bc569ea3 > 2012-03-11 16:31:36,690 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 > f99f., current region memstore size 128.0m > 2012-03-11 16:31:37,999 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf1/8162481165586107427, entries=153106, sequenceid=619316544, > memsize=59.6m, filesize=31.2m > 2012-03-11 16:31:38,830 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 > f99f., current region memstore size 134.8m > 2012-03-11 16:31:39,458 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf2/3425971951499794221, entries=230183, sequenceid=619316544, > memsize=68.5m, filesize=26.6m > 2012-03-11 16:31:39,459 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Finished memstore flush of ~128.1m for region > writetest1,,1331454657410.e9d827913a > 056e696c39bc569ea3f99f. in 2769ms, sequenceid=619316544, compaction > requested=false > 2012-03-11 16:31:39,459 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 > f99f., current region memstore size 6.8m > 2012-03-11 16:31:39,529 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf1/1811012969998104626, entries=8002, sequenceid=619332759, > memsize=3.1m, filesize=1.6m > 2012-03-11 16:31:39,640 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf2/770333473623552048, entries=12231, sequenceid=619332759, > memsize=3.6m, filesize=1.4m > 2012-03-11 16:31:39,641 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Finished memstore flush of ~134.8m for region > writetest1,,1331454657410.e9d827913a > 056e696c39bc569ea3f99f. in 811ms, sequenceid=619332759, compaction > requested=true > 2012-03-11 16:31:39,707 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf1/5656568849587368557, entries=119, sequenceid=619332979, > memsize=47.4k, filesize=25.6k > 2012-03-11 16:31:39,775 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf2/794343845650987521, entries=157, sequenceid=619332979, > memsize=47.8k, filesize=19.3k > 2012-03-11 16:31:39,777 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Finished memstore flush of ~6.8m for region > writetest1,,1331454657410.e9d827913a05 > 6e696c39bc569ea3f99f. in 318ms, sequenceid=619332979, compaction > requested=true -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-5568) Multi concurrent flushcache() for one region could cause data loss
[ https://issues.apache.org/jira/browse/HBASE-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu reopened HBASE-5568: --- Patch for 0.92 hasn't been integrated. > Multi concurrent flushcache() for one region could cause data loss > -- > > Key: HBASE-5568 > URL: https://issues.apache.org/jira/browse/HBASE-5568 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0 > > Attachments: HBASE-5568-90.patch, HBASE-5568-92v2.patch, > HBASE-5568.patch, HBASE-5568.patch, HBASE-5568v2.patch > > > We could call HRegion#flushcache() concurrently now through > HRegionServer#splitRegion or HRegionServer#flushRegion by HBaseAdmin. > However, we find if HRegion#internalFlushcache() is called concurrently by > multi thread, HRegion.memstoreSize will be calculated wrong. > At the end of HRegion#internalFlushcache(), we will do > this.addAndGetGlobalMemstoreSize(-flushsize), but the flushsize may not the > actual memsize which flushed to hdfs. It cause HRegion.memstoreSize is > negative and prevent next flush if we close this region. > Logs in RS for region e9d827913a056e696c39bc569ea3 > 2012-03-11 16:31:36,690 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 > f99f., current region memstore size 128.0m > 2012-03-11 16:31:37,999 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf1/8162481165586107427, entries=153106, sequenceid=619316544, > memsize=59.6m, filesize=31.2m > 2012-03-11 16:31:38,830 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 > f99f., current region memstore size 134.8m > 2012-03-11 16:31:39,458 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf2/3425971951499794221, entries=230183, sequenceid=619316544, > memsize=68.5m, filesize=26.6m > 2012-03-11 16:31:39,459 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Finished memstore flush of ~128.1m for region > writetest1,,1331454657410.e9d827913a > 056e696c39bc569ea3f99f. in 2769ms, sequenceid=619316544, compaction > requested=false > 2012-03-11 16:31:39,459 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 > f99f., current region memstore size 6.8m > 2012-03-11 16:31:39,529 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf1/1811012969998104626, entries=8002, sequenceid=619332759, > memsize=3.1m, filesize=1.6m > 2012-03-11 16:31:39,640 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf2/770333473623552048, entries=12231, sequenceid=619332759, > memsize=3.6m, filesize=1.4m > 2012-03-11 16:31:39,641 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Finished memstore flush of ~134.8m for region > writetest1,,1331454657410.e9d827913a > 056e696c39bc569ea3f99f. in 811ms, sequenceid=619332759, compaction > requested=true > 2012-03-11 16:31:39,707 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf1/5656568849587368557, entries=119, sequenceid=619332979, > memsize=47.4k, filesize=25.6k > 2012-03-11 16:31:39,775 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf2/794343845650987521, entries=157, sequenceid=619332979, > memsize=47.8k, filesize=19.3k > 2012-03-11 16:31:39,777 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Finished memstore flush of ~6.8m for region > writetest1,,1331454657410.e9d827913a05 > 6e696c39bc569ea3f99f. in 318ms, sequenceid=619332979, compaction > requested=true -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[
https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232441#comment-13232441
]
Zhihong Yu commented on HBASE-3996:
---
I think it would be better for Eran to create the review request.
@Eran:
For the new public classes such as MultiTableInputFormatBase, please add
@InterfaceAudience annotation.
For TestMultiTableInputFormat, please label it:
{code}
@Category(LargeTests.class)
{code}
Thanks
> Support multiple tables and scanners as input to the mapper in map/reduce jobs
> --
>
> Key: HBASE-3996
> URL: https://issues.apache.org/jira/browse/HBASE-3996
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Eran Kutner
> Fix For: 0.94.0
>
> Attachments: 3996-v2.txt, HBase-3996.patch
>
>
> It seems that in many cases feeding data from multiple tables or multiple
> scanners on a single table can save a lot of time when running map/reduce
> jobs.
> I propose a new MultiTableInputFormat class that would allow doing this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5594) Unable to stop a master that's waiting on -ROOT- during initialization
[
https://issues.apache.org/jira/browse/HBASE-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232440#comment-13232440
]
ramkrishna.s.vasudevan commented on HBASE-5594:
---
@J-D
This same as HBASE-4951. I tried to give a patch there some half baked one.
But needed some more changes to make it fully done.
Can you confirm if the two are duplictes if so one we can invalidate?
> Unable to stop a master that's waiting on -ROOT- during initialization
> --
>
> Key: HBASE-5594
> URL: https://issues.apache.org/jira/browse/HBASE-5594
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.92.1
>Reporter: Jean-Daniel Cryans
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
>
> We just had a case where the master (that was just restarted) was having a
> hard time assigning -ROOT- (all the PRI handlers were full already) so we
> tried to shutdown the cluster and even though all the RS closed down properly
> the master kept running being blocked on:
> {noformat}
> "master-sv4r20s12,10302,1331916142866" prio=10 tid=0x7f3708008800
> nid=0x4b20 in Object.wait() [0x7f370d1d]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0006030be3f8> (a
> org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
> - locked <0x0006030be3f8> (a
> org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
> - locked <0x0006030be3f8> (a
> org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:313)
> at
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:571)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:501)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:336)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> I haven't checked the 0.90 code, we got this on 0.92.1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5568) Multi concurrent flushcache() for one region could cause data loss
[ https://issues.apache.org/jira/browse/HBASE-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5568: -- Resolution: Fixed Status: Resolved (was: Patch Available) Resolving as committed to all versions. > Multi concurrent flushcache() for one region could cause data loss > -- > > Key: HBASE-5568 > URL: https://issues.apache.org/jira/browse/HBASE-5568 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0 > > Attachments: HBASE-5568-90.patch, HBASE-5568-92v2.patch, > HBASE-5568.patch, HBASE-5568.patch, HBASE-5568v2.patch > > > We could call HRegion#flushcache() concurrently now through > HRegionServer#splitRegion or HRegionServer#flushRegion by HBaseAdmin. > However, we find if HRegion#internalFlushcache() is called concurrently by > multi thread, HRegion.memstoreSize will be calculated wrong. > At the end of HRegion#internalFlushcache(), we will do > this.addAndGetGlobalMemstoreSize(-flushsize), but the flushsize may not the > actual memsize which flushed to hdfs. It cause HRegion.memstoreSize is > negative and prevent next flush if we close this region. > Logs in RS for region e9d827913a056e696c39bc569ea3 > 2012-03-11 16:31:36,690 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 > f99f., current region memstore size 128.0m > 2012-03-11 16:31:37,999 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf1/8162481165586107427, entries=153106, sequenceid=619316544, > memsize=59.6m, filesize=31.2m > 2012-03-11 16:31:38,830 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 > f99f., current region memstore size 134.8m > 2012-03-11 16:31:39,458 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf2/3425971951499794221, entries=230183, sequenceid=619316544, > memsize=68.5m, filesize=26.6m > 2012-03-11 16:31:39,459 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Finished memstore flush of ~128.1m for region > writetest1,,1331454657410.e9d827913a > 056e696c39bc569ea3f99f. in 2769ms, sequenceid=619316544, compaction > requested=false > 2012-03-11 16:31:39,459 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Started memstore flush for > writetest1,,1331454657410.e9d827913a056e696c39bc569ea3 > f99f., current region memstore size 6.8m > 2012-03-11 16:31:39,529 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf1/1811012969998104626, entries=8002, sequenceid=619332759, > memsize=3.1m, filesize=1.6m > 2012-03-11 16:31:39,640 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf2/770333473623552048, entries=12231, sequenceid=619332759, > memsize=3.6m, filesize=1.4m > 2012-03-11 16:31:39,641 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Finished memstore flush of ~134.8m for region > writetest1,,1331454657410.e9d827913a > 056e696c39bc569ea3f99f. in 811ms, sequenceid=619332759, compaction > requested=true > 2012-03-11 16:31:39,707 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf1/5656568849587368557, entries=119, sequenceid=619332979, > memsize=47.4k, filesize=25.6k > 2012-03-11 16:31:39,775 INFO org.apache.hadoop.hbase.regionserver.Store: > Added > hdfs://dw74.kgb.sqa.cm4:9700/hbase-func1/writetest1/e9d827913a056e696c39bc569e > a3f99f/cf2/794343845650987521, entries=157, sequenceid=619332979, > memsize=47.8k, filesize=19.3k > 2012-03-11 16:31:39,777 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Finished memstore flush of ~6.8m for region > writetest1,,1331454657410.e9d827913a05 > 6e696c39bc569ea3f99f. in 318ms, sequenceid=619332979, compaction > requested=true -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[
https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232439#comment-13232439
]
Zhihong Yu commented on HBASE-5593:
---
Thanks for adding the test.
{code}
+ * Copyright 2012 The Apache Software Foundation
{code}
Year isn't needed above.
{code}
+assertEquals(Strings.domainNamePointerToHostName("hbase.apache.org."),
"hbase.apache.org");
{code}
Please wrap long line above.
> Reverse DNS resolution in regionServerStartup() does not strip trailing dot
> ---
>
> Key: HBASE-5593
> URL: https://issues.apache.org/jira/browse/HBASE-5593
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.90.6
>Reporter: David S. Wang
>Assignee: David S. Wang
> Fix For: 0.90.7
>
> Attachments: HBASE-5593-2.patch, HBASE-5593.patch
>
>
> HBASE-4109 covered the removal of trailing dots in PTR records from reverse
> DNS lookups. We seem to have missed a case in HMaster#regionServerStartup().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5599) The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE
[ https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232436#comment-13232436 ] ramkrishna.s.vasudevan commented on HBASE-5599: --- Is uber-hbck doing this? > The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, > NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, > FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. > - > > Key: HBASE-5599 > URL: https://issues.apache.org/jira/browse/HBASE-5599 > Project: HBase > Issue Type: New Feature > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Fix For: 0.90.6 > > Attachments: hbase-5599-0.90.patch > > > The hbkc tool can not fix the six scenarios. > 1. Version file does not exist in root dir. >Fix: I try to create a version file by 'FSUtils.setVersion' method. > > 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any > region server. >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 4. [REGIONNAME] should not be deployed according to META, but is deployed on > [SERVERNAME] >Fix: Close this region. > > 5. First region should start with an empty key. You need to create a new > region and regioninfo in HDFS to plug the hole. >Fix: The region info is not in hdfs and .META., so it create a empty > region for this error. > 6. There is a hole in the region chain between [KEY] and [KEY]. You need to > create a new regioninfo and region dir in hdfs to plug the hole. > Fix: The region info is not in hdfs and .META., so it create a empty region > for this hole. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5153) Add retry logic in HConnectionImplementation#resetZooKeeperTrackers
[ https://issues.apache.org/jira/browse/HBASE-5153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5153: -- Resolution: Fixed Fix Version/s: (was: 0.90.7) 0.90.6 Status: Resolved (was: Patch Available) This got committed to 0.90.6. Sorry for not resolving it at that time. > Add retry logic in HConnectionImplementation#resetZooKeeperTrackers > --- > > Key: HBASE-5153 > URL: https://issues.apache.org/jira/browse/HBASE-5153 > Project: HBase > Issue Type: Bug > Components: client >Affects Versions: 0.90.4 >Reporter: Jieshan Bean >Assignee: Jieshan Bean > Fix For: 0.90.6 > > Attachments: 5153-92.txt, 5153-trunk-minimal.txt, 5153-trunk-v2.txt, > 5153-trunk.txt, 5153-trunk.txt, HBASE-5153-V2.patch, HBASE-5153-V3.patch, > HBASE-5153-V4-90.patch, HBASE-5153-V5-90.patch, > HBASE-5153-V6-90-minorchange.patch, HBASE-5153-V6-90.txt, > HBASE-5153-trunk-v2.patch, HBASE-5153-trunk.patch, HBASE-5153.patch, > HBASE-5153_addendum_0.90_1.patch, HBase-5153-90-addendum.patch, > TestResults-hbase5153.out > > > HBASE-4893 is related to this issue. In that issue, we know, if multi-threads > share a same connection, once this connection got abort in one thread, the > other threads will got a > "HConnectionManager$HConnectionImplementation@18fb1f7 closed" exception. > It solve the problem of "stale connection can't removed". But the orignal > HTable instance cann't be continue to use. The connection in HTable should be > recreated. > Actually, there's two aproach to solve this: > 1. In user code, once catch an IOE, close connection and re-create HTable > instance. We can use this as a workaround. > 2. In HBase Client side, catch this exception, and re-create connection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5595) Fix NoSuchMethodException in 0.92 when running on local filesystem
[
https://issues.apache.org/jira/browse/HBASE-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan reassigned HBASE-5595:
-
Assignee: Uma Maheswara Rao G (was: Uma Mahesh)
> Fix NoSuchMethodException in 0.92 when running on local filesystem
> --
>
> Key: HBASE-5595
> URL: https://issues.apache.org/jira/browse/HBASE-5595
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Uma Maheswara Rao G
>Priority: Critical
> Fix For: 0.92.2
>
> Attachments: HBASE-5595.patch
>
>
> Fix this ugly exception that shows when running 0.92.1 when on local
> filesystem:
> {code}
> 2012-03-16 10:54:48,351 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
> getNumCurrentReplicas--HDFS-826 not available;
> hdfs_out=org.apache.hadoop.fs.FSDataOutputStream@301abf87
> java.lang.NoSuchMethodException:
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.getNumCurrentReplicas()
> at java.lang.Class.getDeclaredMethod(Class.java:1937)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.getGetNumCurrentReplicas(HLog.java:425)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.(HLog.java:408)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.(HLog.java:331)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateHLog(HRegionServer.java:1229)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1218)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:937)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:648)
> at java.lang.Thread.run(Thread.java:680)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5490) Move the enum RS_ZK_REGION_FAILED_OPEN to the last of the enum list in 0.90 EventHandler
[ https://issues.apache.org/jira/browse/HBASE-5490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan resolved HBASE-5490. --- Resolution: Fixed Fix Version/s: (was: 0.90.7) 0.90.6 Assignee: ramkrishna.s.vasudevan This is already committed to 0.90.6. Changing it to 0.90.6 > Move the enum RS_ZK_REGION_FAILED_OPEN to the last of the enum list in 0.90 > EventHandler > > > Key: HBASE-5490 > URL: https://issues.apache.org/jira/browse/HBASE-5490 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.90.6 > > Attachments: 5490-v2.txt, HBASE-5490.patch > > > The new state that was added RS_ZK_REGION_FAILED_OPEN was failing the > rolling restart. > So move the new enum to the end of the list. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5595) Fix NoSuchMethodException in 0.92 when running on local filesystem
[
https://issues.apache.org/jira/browse/HBASE-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan reassigned HBASE-5595:
-
Assignee: Uma Mahesh
Thanks for the patch Uma. Added you to the contributor list.
> Fix NoSuchMethodException in 0.92 when running on local filesystem
> --
>
> Key: HBASE-5595
> URL: https://issues.apache.org/jira/browse/HBASE-5595
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Uma Mahesh
>Priority: Critical
> Fix For: 0.92.2
>
> Attachments: HBASE-5595.patch
>
>
> Fix this ugly exception that shows when running 0.92.1 when on local
> filesystem:
> {code}
> 2012-03-16 10:54:48,351 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
> getNumCurrentReplicas--HDFS-826 not available;
> hdfs_out=org.apache.hadoop.fs.FSDataOutputStream@301abf87
> java.lang.NoSuchMethodException:
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.getNumCurrentReplicas()
> at java.lang.Class.getDeclaredMethod(Class.java:1937)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.getGetNumCurrentReplicas(HLog.java:425)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.(HLog.java:408)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.(HLog.java:331)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateHLog(HRegionServer.java:1229)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1218)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:937)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:648)
> at java.lang.Thread.run(Thread.java:680)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang updated HBASE-5593: - Attachment: HBASE-5593-2.patch New patch with unit tests. I successfully built and ran mvn test -Dtests=TestMaster and mvn test -Dtests=TestStrings. > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593-2.patch, HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232429#comment-13232429 ] Lars Hofhansl commented on HBASE-3996: -- Even v2 has a bunch of formatting/whitespace changes. While those are good changes, they kinda obscure the interesting changes. Any chance to put this up on RB (in that case with the whitespace fixes)? > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner > Fix For: 0.94.0 > > Attachments: 3996-v2.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5599) The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_I
[ https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fulin wang updated HBASE-5599: -- Attachment: hbase-5599-0.90.patch > The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, > NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, > FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. > - > > Key: HBASE-5599 > URL: https://issues.apache.org/jira/browse/HBASE-5599 > Project: HBase > Issue Type: New Feature > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Fix For: 0.90.6 > > Attachments: hbase-5599-0.90.patch > > > The hbkc tool can not fix the six scenarios. > 1. Version file does not exist in root dir. >Fix: I try to create a version file by 'FSUtils.setVersion' method. > > 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any > region server. >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 4. [REGIONNAME] should not be deployed according to META, but is deployed on > [SERVERNAME] >Fix: Close this region. > > 5. First region should start with an empty key. You need to create a new > region and regioninfo in HDFS to plug the hole. >Fix: The region info is not in hdfs and .META., so it create a empty > region for this error. > 6. There is a hole in the region chain between [KEY] and [KEY]. You need to > create a new regioninfo and region dir in hdfs to plug the hole. > Fix: The region info is not in hdfs and .META., so it create a empty region > for this hole. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5599) The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_I
[ https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fulin wang updated HBASE-5599: -- Status: Open (was: Patch Available) > The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, > NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, > FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. > - > > Key: HBASE-5599 > URL: https://issues.apache.org/jira/browse/HBASE-5599 > Project: HBase > Issue Type: New Feature > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Fix For: 0.90.6 > > > The hbkc tool can not fix the six scenarios. > 1. Version file does not exist in root dir. >Fix: I try to create a version file by 'FSUtils.setVersion' method. > > 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any > region server. >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 4. [REGIONNAME] should not be deployed according to META, but is deployed on > [SERVERNAME] >Fix: Close this region. > > 5. First region should start with an empty key. You need to create a new > region and regioninfo in HDFS to plug the hole. >Fix: The region info is not in hdfs and .META., so it create a empty > region for this error. > 6. There is a hole in the region chain between [KEY] and [KEY]. You need to > create a new regioninfo and region dir in hdfs to plug the hole. > Fix: The region info is not in hdfs and .META., so it create a empty region > for this hole. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5599) The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_I
[ https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fulin wang updated HBASE-5599: -- Status: Patch Available (was: Open) > The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, > NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, > FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. > - > > Key: HBASE-5599 > URL: https://issues.apache.org/jira/browse/HBASE-5599 > Project: HBase > Issue Type: New Feature > Components: hbck >Affects Versions: 0.90.6 >Reporter: fulin wang > Fix For: 0.90.6 > > > The hbkc tool can not fix the six scenarios. > 1. Version file does not exist in root dir. >Fix: I try to create a version file by 'FSUtils.setVersion' method. > > 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any > region server. >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] >Fix: I get region info form the hdfs file, this region info write to > '.META.' table. > > 4. [REGIONNAME] should not be deployed according to META, but is deployed on > [SERVERNAME] >Fix: Close this region. > > 5. First region should start with an empty key. You need to create a new > region and regioninfo in HDFS to plug the hole. >Fix: The region info is not in hdfs and .META., so it create a empty > region for this error. > 6. There is a hole in the region chain between [KEY] and [KEY]. You need to > create a new regioninfo and region dir in hdfs to plug the hole. > Fix: The region info is not in hdfs and .META., so it create a empty region > for this hole. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5593) Reverse DNS resolution in regionServerStartup() does not strip trailing dot
[ https://issues.apache.org/jira/browse/HBASE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang updated HBASE-5593: - Status: Open (was: Patch Available) > Reverse DNS resolution in regionServerStartup() does not strip trailing dot > --- > > Key: HBASE-5593 > URL: https://issues.apache.org/jira/browse/HBASE-5593 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang > Fix For: 0.90.7 > > Attachments: HBASE-5593.patch > > > HBASE-4109 covered the removal of trailing dots in PTR records from reverse > DNS lookups. We seem to have missed a case in HMaster#regionServerStartup(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5599) The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_I
The hbkc tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, NOT_IN_HDFS_OR_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. - Key: HBASE-5599 URL: https://issues.apache.org/jira/browse/HBASE-5599 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.90.6 Reporter: fulin wang Fix For: 0.90.6 The hbkc tool can not fix the six scenarios. 1. Version file does not exist in root dir. Fix: I try to create a version file by 'FSUtils.setVersion' method. 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any region server. Fix: I get region info form the hdfs file, this region info write to '.META.' table. 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] Fix: I get region info form the hdfs file, this region info write to '.META.' table. 4. [REGIONNAME] should not be deployed according to META, but is deployed on [SERVERNAME] Fix: Close this region. 5. First region should start with an empty key. You need to create a new region and regioninfo in HDFS to plug the hole. Fix: The region info is not in hdfs and .META., so it create a empty region for this error. 6. There is a hole in the region chain between [KEY] and [KEY]. You need to create a new regioninfo and region dir in hdfs to plug the hole. Fix: The region info is not in hdfs and .META., so it create a empty region for this hole. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-3996: -- Status: Open (was: Patch Available) > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner > Fix For: 0.94.0 > > Attachments: 3996-v2.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-3996: -- Attachment: 3996-v2.txt Patch v2 is smaller than Eran's patch because I didn't apply trivial formatting changes. > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner > Fix For: 0.94.0 > > Attachments: 3996-v2.txt, HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[
https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232418#comment-13232418
]
Zhihong Yu commented on HBASE-3996:
---
There is one compilation error:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile)
on project hbase: Compilation failure
[ERROR]
/Users/zhihyu/trunk-hbase/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java:[85,7]
cannot find symbol
[ERROR] symbol : method init()
[ERROR] location: class org.apache.hadoop.hbase.mapreduce.TableRecordReader
{code}
If I change the init() call to the following:
{code}
trr.initialize(split, context);
{code}
compiler complains that InterruptedException isn't handled.
@Eran:
Please address the above based on modified patch and make sure
TestMultiTableInputFormat passes.
> Support multiple tables and scanners as input to the mapper in map/reduce jobs
> --
>
> Key: HBASE-3996
> URL: https://issues.apache.org/jira/browse/HBASE-3996
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Eran Kutner
> Fix For: 0.94.0
>
> Attachments: HBase-3996.patch
>
>
> It seems that in many cases feeding data from multiple tables or multiple
> scanners on a single table can save a lot of time when running map/reduce
> jobs.
> I propose a new MultiTableInputFormat class that would allow doing this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[
https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232416#comment-13232416
]
Zhihong Yu commented on HBASE-3996:
---
@Eran:
There was unnecessary formatting in your patch:
{code}
+ if (jars.isEmpty())
+ return;
{code}
The convention is to put 'return' on the same line or use curly braces
otherwise.
> Support multiple tables and scanners as input to the mapper in map/reduce jobs
> --
>
> Key: HBASE-3996
> URL: https://issues.apache.org/jira/browse/HBASE-3996
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Eran Kutner
> Fix For: 0.94.0
>
> Attachments: HBase-3996.patch
>
>
> It seems that in many cases feeding data from multiple tables or multiple
> scanners on a single table can save a lot of time when running map/reduce
> jobs.
> I propose a new MultiTableInputFormat class that would allow doing this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232414#comment-13232414 ] Zhihong Yu commented on HBASE-3996: --- I am manually resolving the conflicts since the patch is one month old. @Eran: HbaseObjectWritable supports Scan. Is there special reason why String form of Scan is stored in TableSplit.java ? > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner > Fix For: 0.94.0 > > Attachments: HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232408#comment-13232408 ] Bohdan Mushkevych commented on HBASE-3996: -- It is so unfortunate that such useful functionality is gathering dust on shelves. > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner > Fix For: 0.94.0 > > Attachments: HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232405#comment-13232405 ] Hadoop QA commented on HBASE-3996: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12514793/HBase-3996.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1222//console This message is automatically generated. > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner > Fix For: 0.94.0 > > Attachments: HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-3996: -- Status: Patch Available (was: Open) > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner > Fix For: 0.94.0 > > Attachments: HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232393#comment-13232393
]
Hadoop QA commented on HBASE-5569:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518856/5569-v4.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 160 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1221//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1221//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1221//console
This message is automatically generated.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569-v4.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232390#comment-13232390
]
nkeywal commented on HBASE-5569:
Right now it's still running well. I'm doing the test on a small server, with a
4 core Intel Xeon E3-1220.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569-v4.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232386#comment-13232386
]
Lars Hofhansl commented on HBASE-5569:
--
@N.: Note that the patch also reduces the number of threads for the
test[Multi]RowMutationMultipleThreads and increases the rate of flushes per
thread.
This made it (far) more likely on my home machine to fail, might be different
on your machine.
I should note that on my home machine both test fail every time now on my home
machine, but do not with this patch.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569-v4.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5569:
-
Status: Patch Available (was: Open)
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569-v4.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5569:
-
Attachment: 5569-v4.txt
Same patch with fixes for TestKeyValue and TestCompaction.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569-v4.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232384#comment-13232384
]
Lars Hofhansl commented on HBASE-5569:
--
Thanks N.!
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5569:
-
Status: Open (was: Patch Available)
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232382#comment-13232382
]
nkeywal commented on HBASE-5569:
I've got the testRowMutationMultiThreads running currently on the patch v3. No
issue so far. I will make it run 5000 times, previously it always failed before
1000 iterations.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232383#comment-13232383
]
Lars Hofhansl commented on HBASE-5569:
--
TestCompaction.testMajorCompactingToNoOutput fails because the first scanner in
the test was not closed, then the compaction was done. Hence the compaction
could not remove the deleted rows, because a scanner is still (potentially)
using them.
The test is easily fixed (need to close the first scanner), but we need to
think about whether this is the design we want.
This *is* the same behavior we have with HBASE-2856 for expired rows (TTL or
too many version): If a scanner is open with an earlier readpoint these will
not be collected.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232380#comment-13232380
]
Lars Hofhansl commented on HBASE-5569:
--
Thanks Stack.
TestKeyValue is a simple fix (because I changed the output of KV.toString()).
TestCompaction looks worrisome, checking it out now.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232366#comment-13232366
]
stack commented on HBASE-5569:
--
Nice work Lars. Will review/test tomorrow.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232365#comment-13232365
]
Hadoop QA commented on HBASE-5569:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518850/5569-v3.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 160 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestCompaction
org.apache.hadoop.hbase.TestKeyValue
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1220//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1220//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1220//console
This message is automatically generated.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5569:
-
Attachment: 5569-v3.txt
New patch.
Also adds code to show the memstoreTS in KV.toString.
The number of loops on the TestAtomicOperation was reduced and the number of a
flushes increased.
Please have a careful look.
If possible if some other folks could run TestAtomicOperation in a loop for a
while that would be very helpful (considering that this problem did not occur
at all on my work machine).
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5569:
-
Status: Patch Available (was: Reopened)
Getting a test run.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569-v3.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232359#comment-13232359
]
Lars Hofhansl commented on HBASE-5569:
--
This
{code}
if (includeDeleteMarker
&& kv.getMemstoreTS() <= maxReadPointToTrackVersions) {
this.deletes.add(bytes, offset, qualLength, timestamp, type);
}
{code}
Fixes the issue. Note that maxReadPointToTrackVersions is actually the minimum
readpoint of any scanner still operating in the region and it is *only* set
during compaction.
I think this correct because of the following:
All delete markers precede the KVs they affect. So by not adding the delete
marker it is guarantees that no KVs will be removed during flush that might
still be in use. It also removes this race condition between scanner and
flushes.
So my previous fix was almost correct (in thought at least). I had believed it
to be correct, because I had not been able - not even a single time - to
reproduce this on my work machine.
I'll attach a patch soon.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232354#comment-13232354 ] Bohdan Mushkevych commented on HBASE-3996: -- Eran - thank you for updated patch. Zhihong Yu - have you had a chance to apply it on TRUNK? > Support multiple tables and scanners as input to the mapper in map/reduce jobs > -- > > Key: HBASE-3996 > URL: https://issues.apache.org/jira/browse/HBASE-3996 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Eran Kutner > Fix For: 0.94.0 > > Attachments: HBase-3996.patch > > > It seems that in many cases feeding data from multiple tables or multiple > scanners on a single table can save a lot of time when running map/reduce > jobs. > I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232349#comment-13232349
]
Lars Hofhansl commented on HBASE-5569:
--
Here's my theory...
In ScanQueryMatcher we have this:
{code}
byte type = kv.getType();
if (kv.isDelete()) {
if (!keepDeletedCells) {
...
this.deletes.add(bytes, offset, qualLength, timestamp, type);
}
...
} else if (!this.deletes.isEmpty()) {
DeleteResult deleteResult = deletes.isDeleted(bytes, offset, qualLength,
timestamp);
...
}
{code}
And in StoreScanner.resetScannerStack
{code}
// Reset the state of the Query Matcher and set to top row.
// Only reset and call setRow if the row changes; avoids confusing the
// query matcher if scanning intra-row.
...
if ((matcher.row == null) || !kv.matchingRow(matcher.row)) {
matcher.reset();
matcher.setRow(kv.getRow());
}
{code}
So, the SQM might already have a delete registered, or might miss a delete.
With KEEP_DELETED_CELLS that race does not happen, because deletes are simply
not registered.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5595) Fix NoSuchMethodException in 0.92 when running on local filesystem
[
https://issues.apache.org/jira/browse/HBASE-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-5595:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the patch Uma. Just what the doctor ordered. Applied to 0.92, 0.94
and trunk
> Fix NoSuchMethodException in 0.92 when running on local filesystem
> --
>
> Key: HBASE-5595
> URL: https://issues.apache.org/jira/browse/HBASE-5595
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Critical
> Fix For: 0.92.2
>
> Attachments: HBASE-5595.patch
>
>
> Fix this ugly exception that shows when running 0.92.1 when on local
> filesystem:
> {code}
> 2012-03-16 10:54:48,351 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
> getNumCurrentReplicas--HDFS-826 not available;
> hdfs_out=org.apache.hadoop.fs.FSDataOutputStream@301abf87
> java.lang.NoSuchMethodException:
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.getNumCurrentReplicas()
> at java.lang.Class.getDeclaredMethod(Class.java:1937)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.getGetNumCurrentReplicas(HLog.java:425)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.(HLog.java:408)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.(HLog.java:331)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateHLog(HRegionServer.java:1229)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1218)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:937)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:648)
> at java.lang.Thread.run(Thread.java:680)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5569) Do not collect deleted KVs when they are still in use by a scanner.
[
https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232341#comment-13232341
]
Lars Hofhansl commented on HBASE-5569:
--
I spent a lot of more time looking at this. I thought it might be due to the
flushes being executed in parallel by multiple threads, but synchronizing this
part made the failure more likely!
Doing this and increasing the frequency of flushes reproduces the problem
multiple times on every test run now, which it good.
But... My initial hunch was correct. When I enable KEEP_DELETED_CELLS on the
store the problem goes away!
Hence this definitely has to do with collection of deletes and delete markers.
> Do not collect deleted KVs when they are still in use by a scanner.
> ---
>
> Key: HBASE-5569
> URL: https://issues.apache.org/jira/browse/HBASE-5569
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5569-v2.txt, 5569.txt,
> TestAtomicOperation-output.trunk_120313.rar
>
>
> I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads
> fails rarely.
> The solution is similar to HBASE-2856, where expired KVs are not collected
> when in use by a scanner.
> ---
> What I pieced together so far is that it is the *scanning* side that has
> problems sometimes.
> Every time I see a assertion failure in the log I see this before:
> {quote}
> 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499):
> Storescanner.peek() is changed where before =
> rowB/colfamily11:qual1/75366/Put/vlen=6,and after =
> rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0
> {quote}
> The order of if the Put and Delete is sometimes reversed.
> The test threads should always see exactly one KV, if the "before" was the
> Put the thread see 0 KVs, if the "before" was the Delete the threads see 2
> KVs.
> This debug message comes from StoreScanner to checkReseek. It seems we still
> some consistency issue with scanning sometimes :(
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5521) Move compression/decompression to an encoder specific encoding context
[ https://issues.apache.org/jira/browse/HBASE-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232334#comment-13232334 ] Phabricator commented on HBASE-5521: heyongqiang has commented on the revision "HBASE-5521 [jira] Move compression/decompression to an encoder specific encoding context". i think Mikhail already did +1 on this. The last few updates are just to address tedyu's comments REVISION DETAIL https://reviews.facebook.net/D2097 > Move compression/decompression to an encoder specific encoding context > -- > > Key: HBASE-5521 > URL: https://issues.apache.org/jira/browse/HBASE-5521 > Project: HBase > Issue Type: Improvement >Reporter: He Yongqiang >Assignee: He Yongqiang > Fix For: 0.96.0 > > Attachments: HBASE-5521.1.patch, HBASE-5521.D2097.1.patch, > HBASE-5521.D2097.10.patch, HBASE-5521.D2097.2.patch, > HBASE-5521.D2097.3.patch, HBASE-5521.D2097.4.patch, HBASE-5521.D2097.5.patch, > HBASE-5521.D2097.6.patch, HBASE-5521.D2097.7.patch, HBASE-5521.D2097.8.patch, > HBASE-5521.D2097.9.patch > > > As part of working on HBASE-5313, we want to add a new columnar > encoder/decoder. It makes sense to move compression to be part of > encoder/decoder: > 1) a scanner for a columnar encoded block can do lazy decompression to a > specific part of a key value object > 2) avoid an extra bytes copy from encoder to hblock-writer. > If there is no encoder specified for a writer, the HBlock.Writer will use a > default compression-context to do something very similar to today's code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5341) Push the security 0.92 profile to maven repo
[
https://issues.apache.org/jira/browse/HBASE-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232333#comment-13232333
]
stack commented on HBASE-5341:
--
I tried doing this but each time, though it built the security artifacts under
the target dir, it'd go upload to nexus the insecures. Here is command I was
using:
{code}
% mvn -Papache-release,security release:perform
{code}
Need to add in some move switches it seems for release plugin when security
profile enabled. Any input?
> Push the security 0.92 profile to maven repo
>
>
> Key: HBASE-5341
> URL: https://issues.apache.org/jira/browse/HBASE-5341
> Project: HBase
> Issue Type: Improvement
> Components: build, security
>Affects Versions: 0.92.1, 0.94.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Blocker
> Fix For: 0.92.2
>
>
> Hbase 0.92.0 was released with two artifacts, plain and security. The
> security code is built with -Psecurity. There are two tarballs, but only the
> plain jar in maven repo at repository.a.o.
> I see no reason to do a separate artifact for the security related code,
> since 0.92 already depends on secure Hadoop 1.0.0, and all of the security
> related code is not loaded by default. In this issue, I propose, we merge the
> code under /security to src/ and remove the maven profile.
> Edit: after some discussion, and the plans for modularizing the build to
> include a security module, we changed the issue description to push the
> security jars in 0.92.1 to maven repo.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5206) Port HBASE-5155 to 0.92, 0.94, and TRUNK
[ https://issues.apache.org/jira/browse/HBASE-5206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5206: -- Resolution: Fixed Status: Resolved (was: Patch Available) Resolving as committed. > Port HBASE-5155 to 0.92, 0.94, and TRUNK > > > Key: HBASE-5206 > URL: https://issues.apache.org/jira/browse/HBASE-5206 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.2, 0.94.0, 0.96.0 >Reporter: Zhihong Yu >Assignee: Ashutosh Jindal > Fix For: 0.92.2, 0.94.0, 0.96.0 > > Attachments: 5206_92_1.patch, 5206_92_latest_1.patch, > 5206_92_latest_2.patch, 5206_92_latest_3.patch, 5206_trunk_1.patch, > 5206_trunk_latest_1.patch, 5206_trunk_latest_2.patch, > 5206_trunk_latest_3.patch > > > This JIRA ports HBASE-5155 (ServerShutDownHandler And Disable/Delete should > not happen parallely leading to recreation of regions that were deleted) to > 0.92 and TRUNK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
