[jira] [Updated] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6205: Attachment: HBASE-6205v4.patch Updating patch v4 as Ted's comments in review board Support an option to keep data of dropped table for some time - Key: HBASE-6205 URL: https://issues.apache.org/jira/browse/HBASE-6205 Project: HBase Issue Type: New Feature Affects Versions: 0.94.0, 0.96.0 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6205.patch, HBASE-6205v2.patch, HBASE-6205v3.patch, HBASE-6205v4.patch User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, e.g. 1 day In the patch: We make a new dir named .trashtables in the rood dir. In the DeleteTableHandler, we move files in dropped table's dir to trash table dir instead of deleting them directly. And Create new class TrashCleaner which will clean dropped tables if it is time out with a period check. Default keep time for dropped tables is 1 day, and check period is 1 hour. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397286#comment-13397286 ] ramkrishna.s.vasudevan commented on HBASE-6228: --- @Chunhui So this happens when master restart and SSH goes in parallel only? Fixup daughters twice cause daughter region assigned twice --- Key: HBASE-6228 URL: https://issues.apache.org/jira/browse/HBASE-6228 Project: HBase Issue Type: Bug Components: master Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6228.patch First, how fixup daughters twice happen? 1.we will fixupDaughters at the last of HMaster#finishInitialization 2.ServerShutdownHandler will fixupDaughters when reassigning region through ServerShutdownHandler#processDeadRegion When fixupDaughters, we will added daughters to .META., but it coudn't prevent the above case, because FindDaughterVisitor. The detail is as the following: Suppose region A is a splitted parent region, and its daughter region B is missing 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B to .META. with serverName=null, and assign the daughter. 2.Then, Master's initialization thread will also find the daughter region B is missing and assign it. It is because FindDaughterVisitor consider daughter is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4246) Cluster with too many regions cannot withstand some master failover scenarios
[ https://issues.apache.org/jira/browse/HBASE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397290#comment-13397290 ] Laxman commented on HBASE-4246: --- This may come in latest version also as we didn't change the znode hierarchy of the unassigned regions. As mentioned in linked issue, there is a cap on packet length. We can't read/write huge data in a single packet. IMO, to resolve this we need to do *either of the following*. * In HBASE: We can use hierarchical structure. HDFS datanode follows similar strategy. It keeps block files in different sub directories to avoid FS lookup latency. * In ZooKeeper: Increase the limit. What is reasonable? We have tried this out in some other project but it has the side effects. When we tried read/write huge data from ZooKeeper, clients occassionally gets disconnected. This is sequential request processing. Please check out the related discussions @ http://mail-archives.apache.org/mod_mbox/zookeeper-user/201007.mbox/%3cc85a33ec.3a46a%25maha...@yahoo-inc.com%3E Following JIRA and discussion also applicable in current scenario. http://mail-archives.apache.org/mod_mbox/zookeeper-user/201104.mbox/%3cffa3bdb6-1c83-42b9-b2a0-767513462...@me.com%3E https://issues.apache.org/jira/browse/ZOOKEEPER-1049 Cluster with too many regions cannot withstand some master failover scenarios - Key: HBASE-4246 URL: https://issues.apache.org/jira/browse/HBASE-4246 Project: HBase Issue Type: Bug Components: master, zookeeper Affects Versions: 0.90.4 Reporter: Todd Lipcon Priority: Critical Fix For: 0.96.0 We ran into the following sequence of events: - master startup failed after only ROOT had been assigned (for another reason) - restarted the master without restarting other servers. Since there was at least one region assigned, it went through the failover code path - master scanned META and inserted every region into /hbase/unassigned in ZK. - then, it called listChildren on the /hbase/unassigned znode, and crashed with Packet len6080218 is out of range! since the IPC response was larger than the default maximum. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397291#comment-13397291 ] chunhui shen commented on HBASE-6228: - @ram So this happens when master restart and SSH goes in parallel only? Yes, as the current logic, we only fixup daughter in the two places. Fixup daughters twice cause daughter region assigned twice --- Key: HBASE-6228 URL: https://issues.apache.org/jira/browse/HBASE-6228 Project: HBase Issue Type: Bug Components: master Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6228.patch First, how fixup daughters twice happen? 1.we will fixupDaughters at the last of HMaster#finishInitialization 2.ServerShutdownHandler will fixupDaughters when reassigning region through ServerShutdownHandler#processDeadRegion When fixupDaughters, we will added daughters to .META., but it coudn't prevent the above case, because FindDaughterVisitor. The detail is as the following: Suppose region A is a splitted parent region, and its daughter region B is missing 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B to .META. with serverName=null, and assign the daughter. 2.Then, Master's initialization thread will also find the daughter region B is missing and assign it. It is because FindDaughterVisitor consider daughter is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6227) SSH and cluster startup causes data loss
[ https://issues.apache.org/jira/browse/HBASE-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397293#comment-13397293 ] Hadoop QA commented on HBASE-6227: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532648/HBASE-6227v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2197//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2197//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2197//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2197//console This message is automatically generated. SSH and cluster startup causes data loss - Key: HBASE-6227 URL: https://issues.apache.org/jira/browse/HBASE-6227 Project: HBase Issue Type: Bug Components: master Reporter: chunhui shen Assignee: chunhui shen Attachments: HBASE-6227.patch, HBASE-6227v2.patch In AssignmentManager#processDeadServersAndRegionsInTransition, if servershutdownhandler is processing and master consider it cluster startup, master will assign all user regions, however, servershutdownhandler has not completed splitting logs. Let me describe it in more detail. Suppose there are two regionservers A1 and B1, A1 carried META and ROOT 1.master restart and completed assignRootAndMeta 2.A1 and B1 are both restarted, new regionservers are A2 and B2 3.SSH which processed for A1 completed assigning ROOT and META 4.master do rebuilding user regions and no regions added to master's region list 5.master consider it as a cluster startup, and assign all user regions 6.SSH which processed for B1 start to split B1's logs 7.All regions' data carried by B1 would loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5505) if max file size set to 256M, Per table hbase.hregion.max.filesize setting(at htable creation) overwritten by value set in hbase-site.conf(which might not be 256MB)
[
https://issues.apache.org/jira/browse/HBASE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397292#comment-13397292
]
steven zhuang commented on HBASE-5505:
--
after a try, I think its too expensive to fix this minor bug, too many files
are involved.
better we leave it this way?
if max file size set to 256M, Per table hbase.hregion.max.filesize setting(at
htable creation) overwritten by value set in hbase-site.conf(which might not
be 256MB)
Key: HBASE-5505
URL: https://issues.apache.org/jira/browse/HBASE-5505
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.4, 0.92.0
Reporter: steven zhuang
seems to me that when set to DEFAULT_MAX_FILE_SIZE, the max region file size
set at Htable creation time may later be overwritten by value set in
hbase-site.conf, i.e. no Per table hbase.hregion.max.filesize at all.
in this case, on Web UI in the table description, max file is still 256M, but
we can find region file much bigger on HDFS.
In 0.92.0, o.a.h.hbase.regionserver.ConstantSizeRegionSplitPolicy, around
line 33:
long maxFileSize = region.getTableDesc().getMaxFileSize();
// By default we split region if a file
HConstants.DEFAULT_MAX_FILE_SIZE.
if (maxFileSize == HConstants.DEFAULT_MAX_FILE_SIZE) { //zx: here
setting is over written by configuaration from hbase-site.conf
maxFileSize = getConf().getLong(hbase.hregion.max.filesize,
HConstants.DEFAULT_MAX_FILE_SIZE);
}
In 0.90.4(cdh3u3) or earlier version, this is found in
o.a.h.hbase.regionserver.Store, around line 188:
long maxFileSize = info.getTableDesc().getMaxFileSize();
if (maxFileSize == HConstants.DEFAULT_MAX_FILE_SIZE) {
maxFileSize = conf.getLong(hbase.hregion.max.filesize,
HConstants.DEFAULT_MAX_FILE_SIZE);
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397297#comment-13397297 ] ramkrishna.s.vasudevan commented on HBASE-6228: --- So as i see the code if in the normal case where SSH tries to fix up, if SSH sees there is no servername i.e the servername is null, it will try to fixup daughter and assign it. As i see the patch now we are trying to remove code where we get ServerName as null. So if only SSH is going on and there is no master restart scenario how this patch will work? Sorry if my question is dumb? Fixup daughters twice cause daughter region assigned twice --- Key: HBASE-6228 URL: https://issues.apache.org/jira/browse/HBASE-6228 Project: HBase Issue Type: Bug Components: master Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6228.patch First, how fixup daughters twice happen? 1.we will fixupDaughters at the last of HMaster#finishInitialization 2.ServerShutdownHandler will fixupDaughters when reassigning region through ServerShutdownHandler#processDeadRegion When fixupDaughters, we will added daughters to .META., but it coudn't prevent the above case, because FindDaughterVisitor. The detail is as the following: Suppose region A is a splitted parent region, and its daughter region B is missing 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B to .META. with serverName=null, and assign the daughter. 2.Then, Master's initialization thread will also find the daughter region B is missing and assign it. It is because FindDaughterVisitor consider daughter is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397301#comment-13397301 ] chunhui shen commented on HBASE-6228: - bq.So if only SSH is going on and there is no master restart scenario how this patch will work? In current logic, we will add daughter to META with serverName in the SplitTransaction. And we won't added daughter with serverName==null except fixup daughter. However, master may die after added daughter but before assign it, and master won't assign it when restart So I think we should added daughter with parent's serverName. Thanks ram to find the hole Fixup daughters twice cause daughter region assigned twice --- Key: HBASE-6228 URL: https://issues.apache.org/jira/browse/HBASE-6228 Project: HBase Issue Type: Bug Components: master Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6228.patch First, how fixup daughters twice happen? 1.we will fixupDaughters at the last of HMaster#finishInitialization 2.ServerShutdownHandler will fixupDaughters when reassigning region through ServerShutdownHandler#processDeadRegion When fixupDaughters, we will added daughters to .META., but it coudn't prevent the above case, because FindDaughterVisitor. The detail is as the following: Suppose region A is a splitted parent region, and its daughter region B is missing 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B to .META. with serverName=null, and assign the daughter. 2.Then, Master's initialization thread will also find the daughter region B is missing and assign it. It is because FindDaughterVisitor consider daughter is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397305#comment-13397305 ] Hadoop QA commented on HBASE-6205: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532653/HBASE-6205v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2198//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2198//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2198//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2198//console This message is automatically generated. Support an option to keep data of dropped table for some time - Key: HBASE-6205 URL: https://issues.apache.org/jira/browse/HBASE-6205 Project: HBase Issue Type: New Feature Affects Versions: 0.94.0, 0.96.0 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6205.patch, HBASE-6205v2.patch, HBASE-6205v3.patch, HBASE-6205v4.patch User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, e.g. 1 day In the patch: We make a new dir named .trashtables in the rood dir. In the DeleteTableHandler, we move files in dropped table's dir to trash table dir instead of deleting them directly. And Create new class TrashCleaner which will clean dropped tables if it is time out with a period check. Default keep time for dropped tables is 1 day, and check period is 1 hour. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4064) Two concurrent unassigning of the same region caused the endless loop of Region has been PENDING_CLOSE for too long...
[ https://issues.apache.org/jira/browse/HBASE-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397332#comment-13397332 ] steven zhuang commented on HBASE-4064: -- hi, Ramkrishna, sorry for the late response. This turn of event came because we wanted to update the hbase jar to enable our table-balancer without turning the whole cluster off. Here is what I did: 1. turn off first half RS of the cluster, update the jars 2. turn on the first half RS, (used hbase-daemon.sh stop on the RS which holds the META region coz it won't go down ) 3. turn off the second half RS of the cluster, update the jars, 4. turn on the second half RS At this time, I found the ROOT region is in transition, and status was CLOSING although the RS hold it was down . So I tried to restart the whole cluster, the cluster cannot be started now, coz the RS which master believes holds the ROOT region has no region served, master can only give out NotServingRegionException. I closed all the RS, the started the master again, only to find the master trying to reach the same ROOT-holding RS which was down then, and gave ServerNotRunningYetException after dozens of retries. I guess I restarted the half cluster too soon after I closed it, so root region was still in transition when the original RS restarted, and later I force closed the whole cluster, but status in ZK cluster stayed. so later when I restarted the cluster, the master tried to close the ROOT region instead of start it, but that time, the region is not on any RS, master then fall into a (dead?) loop. the log files is all messed up, with splitlog and other crappy exception log, and is way too big(even just the grepped part) to upload, so hope my description helps. Two concurrent unassigning of the same region caused the endless loop of Region has been PENDING_CLOSE for too long... Key: HBASE-4064 URL: https://issues.apache.org/jira/browse/HBASE-4064 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.3 Reporter: Jieshan Bean Fix For: 0.90.7 Attachments: HBASE-4064-v1.patch, HBASE-4064_branch90V2.patch, disableflow.png 1. If there is a rubbish RegionState object with PENDING_CLOSE in regionsInTransition(The RegionState was remained by some exception which should be removed, that's why I called it as rubbish object), but the region is not currently assigned anywhere, TimeoutMonitor will fall into an endless loop: 2011-06-27 10:32:21,326 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. state=PENDING_CLOSE, ts=1309141555301 2011-06-27 10:32:21,326 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been PENDING_CLOSE for too long, running forced unassign again on region=test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. 2011-06-27 10:32:21,438 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. (offlining) 2011-06-27 10:32:21,441 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. but it is not currently assigned anywhere 2011-06-27 10:32:31,207 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. state=PENDING_CLOSE, ts=1309141555301 2011-06-27 10:32:31,207 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been PENDING_CLOSE for too long, running forced unassign again on region=test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. 2011-06-27 10:32:31,215 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. (offlining) 2011-06-27 10:32:31,215 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. but it is not currently assigned anywhere 2011-06-27 10:32:41,164 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. state=PENDING_CLOSE, ts=1309141555301 2011-06-27 10:32:41,164 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been PENDING_CLOSE for too long, running forced unassign again on region=test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. 2011-06-27 10:32:41,172 DEBUG
[jira] [Updated] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6228: Attachment: HBASE-6228v2.patch Updating patch： added daughter with parent region's serverName when fixup daughters Fixup daughters twice cause daughter region assigned twice --- Key: HBASE-6228 URL: https://issues.apache.org/jira/browse/HBASE-6228 Project: HBase Issue Type: Bug Components: master Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6228.patch, HBASE-6228v2.patch First, how fixup daughters twice happen? 1.we will fixupDaughters at the last of HMaster#finishInitialization 2.ServerShutdownHandler will fixupDaughters when reassigning region through ServerShutdownHandler#processDeadRegion When fixupDaughters, we will added daughters to .META., but it coudn't prevent the above case, because FindDaughterVisitor. The detail is as the following: Suppose region A is a splitted parent region, and its daughter region B is missing 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B to .META. with serverName=null, and assign the daughter. 2.Then, Master's initialization thread will also find the daughter region B is missing and assign it. It is because FindDaughterVisitor consider daughter is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397339#comment-13397339 ] Hadoop QA commented on HBASE-6228: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532666/HBASE-6228v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2199//console This message is automatically generated. Fixup daughters twice cause daughter region assigned twice --- Key: HBASE-6228 URL: https://issues.apache.org/jira/browse/HBASE-6228 Project: HBase Issue Type: Bug Components: master Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6228.patch, HBASE-6228v2.patch First, how fixup daughters twice happen? 1.we will fixupDaughters at the last of HMaster#finishInitialization 2.ServerShutdownHandler will fixupDaughters when reassigning region through ServerShutdownHandler#processDeadRegion When fixupDaughters, we will added daughters to .META., but it coudn't prevent the above case, because FindDaughterVisitor. The detail is as the following: Suppose region A is a splitted parent region, and its daughter region B is missing 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B to .META. with serverName=null, and assign the daughter. 2.Then, Master's initialization thread will also find the daughter region B is missing and assign it. It is because FindDaughterVisitor consider daughter is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6228: Attachment: HBASE-6228v2.patch Reattaching patch v2 Fixup daughters twice cause daughter region assigned twice --- Key: HBASE-6228 URL: https://issues.apache.org/jira/browse/HBASE-6228 Project: HBase Issue Type: Bug Components: master Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6228.patch, HBASE-6228v2.patch, HBASE-6228v2.patch First, how fixup daughters twice happen? 1.we will fixupDaughters at the last of HMaster#finishInitialization 2.ServerShutdownHandler will fixupDaughters when reassigning region through ServerShutdownHandler#processDeadRegion When fixupDaughters, we will added daughters to .META., but it coudn't prevent the above case, because FindDaughterVisitor. The detail is as the following: Suppose region A is a splitted parent region, and its daughter region B is missing 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B to .META. with serverName=null, and assign the daughter. 2.Then, Master's initialization thread will also find the daughter region B is missing and assign it. It is because FindDaughterVisitor consider daughter is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397365#comment-13397365 ] Hadoop QA commented on HBASE-6228: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532668/HBASE-6228v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2200//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2200//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2200//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2200//console This message is automatically generated. Fixup daughters twice cause daughter region assigned twice --- Key: HBASE-6228 URL: https://issues.apache.org/jira/browse/HBASE-6228 Project: HBase Issue Type: Bug Components: master Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6228.patch, HBASE-6228v2.patch, HBASE-6228v2.patch First, how fixup daughters twice happen? 1.we will fixupDaughters at the last of HMaster#finishInitialization 2.ServerShutdownHandler will fixupDaughters when reassigning region through ServerShutdownHandler#processDeadRegion When fixupDaughters, we will added daughters to .META., but it coudn't prevent the above case, because FindDaughterVisitor. The detail is as the following: Suppose region A is a splitted parent region, and its daughter region B is missing 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B to .META. with serverName=null, and assign the daughter. 2.Then, Master's initialization thread will also find the daughter region B is missing and assign it. It is because FindDaughterVisitor consider daughter is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6109) Improve RIT performances during assignment on large clusters
[ https://issues.apache.org/jira/browse/HBASE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-6109: --- Resolution: Fixed Status: Resolved (was: Patch Available) Improve RIT performances during assignment on large clusters Key: HBASE-6109 URL: https://issues.apache.org/jira/browse/HBASE-6109 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.96.0 Attachments: 6109.v19.patch, 6109.v21.patch, 6109.v23.patch, 6109.v24.patch, 6109.v7.patch The main points in this patch are: - lowering the number of copy of the RIT list - lowering the number of synchronization - synchronizing on a region rather than on everything It also contains: - some fixes around the RIT notification: the list was sometimes modified without a corresponding 'notify'. - some tests flakiness correction, actually unrelated to this patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5924) In the client code, don't wait for all the requests to be executed before resubmitting a request in error.
[ https://issues.apache.org/jira/browse/HBASE-5924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-5924: --- Resolution: Fixed Status: Resolved (was: Patch Available) In the client code, don't wait for all the requests to be executed before resubmitting a request in error. -- Key: HBASE-5924 URL: https://issues.apache.org/jira/browse/HBASE-5924 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 Attachments: 5924.v11.patch, 5924.v14.patch, 5924.v19.patch, 5924.v5.patch, 5924.v9.patch The client (in the function HConnectionManager#processBatchCallback) works in two steps: - make the requests - collect the failures and successes and prepare for retry It means that when there is an immediate error (region moved, split, dead server, ...) we still wait for all the initial requests to be executed before submitting again the failed request. If we have a scenario with all the requests taking 5 seconds we have a final execution time of: 5 (initial requests) + 1 (wait time) + 5 (final request) = 11s. We could improve this by analyzing immediately the results. This would lead us, for the scenario mentioned above, to 6 seconds. So we could have a performance improvement of nearly 50% in many cases, and much more than 50% if the request execution time is different. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6156) Improve multiop performances in HTable#flushCommits
[
https://issues.apache.org/jira/browse/HBASE-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nkeywal resolved HBASE-6156.
Resolution: Fixed
Fixed in HBASE-5924
Improve multiop performances in HTable#flushCommits
---
Key: HBASE-6156
URL: https://issues.apache.org/jira/browse/HBASE-6156
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Fix For: 0.96.0
This code:
{noformat}
@Override
public void flushCommits() throws IOException {
try {
Object[] results = new Object[writeBuffer.size()];
try {
this.connection.processBatch(writeBuffer, tableName, pool, results);
} catch (InterruptedException e) {
throw new IOException(e);
} finally {
// mutate list so that it is empty for complete success, or contains
// only failed records results are returned in the same order as the
// requests in list walk the list backwards, so we can remove from
list
// without impacting the indexes of earlier members
for (int i = results.length - 1; i=0; i--) {
if (results[i] instanceof Result) {
// successful Puts are removed from the list here.
writeBuffer.remove(i);
}
}
}
} finally {
if (clearBufferOnFail) {
writeBuffer.clear();
currentWriteBufferSize = 0;
} else {
// the write buffer was adjusted by processBatchOfPuts
currentWriteBufferSize = 0;
for (Put aPut : writeBuffer) {
currentWriteBufferSize += aPut.heapSize();
}
}
}
}
{noformat}
Can be improved by:
- not iterating on the list if clearBufferOnFail is set
- not iterating the the list of there are no error
- iterating on the list only once instead of two when we really have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6195) Increment data will be lost when the memstore is flushed
[
https://issues.apache.org/jira/browse/HBASE-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397406#comment-13397406
]
Xing Shi commented on HBASE-6195:
-
@Ted, I find that if there is no flushcache, there could be three increments in
one millisecond(depends on the machine).
I made all the timestamps (variable now) the same value, to simulate that in
the same millisecond, it happends lots of increment.
Also I start the thread to flushcache. And the test failed:
{code}
Results :
Failed tests:
testParallelIncrementWithMemStoreFlush(org.apache.hadoop.hbase.regionserver.TestHRegion):
expected:2000 but was:146
{code}
Follow is my suspect：
Maybe it is because the increment read the old value（v1） from snapshot, and
write the new value（v1+1） to memstore, after this if the flush cache is still
not completed, another increment read the value also from snapshot(v1), and
write the new value(v1+1) to memstore, so the increment operation takes no
effect.
If the above is right, I think we can read the value from memstore fist for the
same timestamp(I didn't know how to), or make the timestamp to nanoTime to
decrease the probability?
Increment data will be lost when the memstore is flushed
Key: HBASE-6195
URL: https://issues.apache.org/jira/browse/HBASE-6195
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: Xing Shi
Assignee: ShiXing
Fix For: 0.96.0, 0.94.1
Attachments: 6195-trunk-V7.patch, 6195.addendum,
HBASE-6195-trunk-V2.patch, HBASE-6195-trunk-V3.patch,
HBASE-6195-trunk-V4.patch, HBASE-6195-trunk-V5.patch,
HBASE-6195-trunk-V6.patch, HBASE-6195-trunk.patch
There are two problems in increment() now:
First:
I see that the timestamp(the variable now) in HRegion's Increment() is
generated before got the rowLock, so when there are multi-thread increment
the same row, although it generate earlier, it may got the lock later.
Because increment just store one version, so till now, the result will still
be right.
When the region is flushing, these increment will read the kv from snapshot
and memstore with whose timestamp is larger, and write it back to memstore.
If the snapshot's timestamp larger than the memstore, the increment will got
the old data and then do the increment, it's wrong.
Secondly:
Also there is a risk in increment. Because it writes the memstore first and
then HLog, so if it writes HLog failed, the client will also read the
incremented value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5953) Expose the current state of the balancerSwitch
[
https://issues.apache.org/jira/browse/HBASE-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397422#comment-13397422
]
Anoop Sam John commented on HBASE-5953:
---
{code}
+ private boolean isBalancerOn;
...
+ public boolean getIsBalancerOn() {
+return isBalancerOn;
+ }
{code}
Can we name the variable as balancerOn and method name as isBalancerOn()
I have seen in some other JIRA that Stack was mentioning abt we maintain the
convention of is... () for the getters of boolean variables.
Expose the current state of the balancerSwitch
--
Key: HBASE-5953
URL: https://issues.apache.org/jira/browse/HBASE-5953
Project: HBase
Issue Type: Improvement
Components: master
Affects Versions: 0.96.0
Reporter: David S. Wang
Assignee: Gregory Chanan
Priority: Minor
Attachments: HBASE-5953.patch
The balancerSwitch as in both 0.90 and 0.92.1 is implemented in such a way
that it is impossible to retrieve its value without changing it:
/**
Turn the load balancer on or off.
@param b If true, enable balancer. If false, disable balancer.
@return Previous balancer value
*/
public boolean balanceSwitch(final boolean b);
It would be nice to have a way to just get the current state.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5630) hbck should disable the balancer using synchronousBalanceSwitch.
[ https://issues.apache.org/jira/browse/HBASE-5630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397444#comment-13397444 ] Anoop Sam John commented on HBASE-5630: --- The old balanceSwith() in Master only is exposed via HBaseAdmin now. So for this we need to expose the new way of balance switch in HBaseAdmin right? Also one more thing. A Master restart in between the HBCK run will have some impacts on the HBCK? hbck should disable the balancer using synchronousBalanceSwitch. Key: HBASE-5630 URL: https://issues.apache.org/jira/browse/HBASE-5630 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Priority: Minor hbck disable the balancer using admin.balanceSwith(bool) when it would be preferable to use the newer synchronusBalanceSwitch method found in 0.94 and trunk branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5918) Master will block forever when startup if root server died between assign root and assign meta
[
https://issues.apache.org/jira/browse/HBASE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397512#comment-13397512
]
ramkrishna.s.vasudevan commented on HBASE-5918:
---
This issue needs to be committed. There are few more like this that needs to
be integrated I feel.
Master will block forever when startup if root server died between assign
root and assign meta
--
Key: HBASE-5918
URL: https://issues.apache.org/jira/browse/HBASE-5918
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.1
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: HBASE-5918.patch, HBASE-5918.patch, HBASE-5918V2.patch
When master is initializing, if root server died between assign root and
assign meta, master will block at
HMaster#assignRootAndMeta:{code}assignmentManager.assignMeta();
this.catalogTracker.waitForMeta();{code}
because ServerShutdownHandler is disabled,
So we should enable ServerShutdownHandler after called
assignmentManager.assignMeta();
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397544#comment-13397544 ] Zhihong Ted Yu commented on HBASE-6228: --- +1 on patch v2. Fixup daughters twice cause daughter region assigned twice --- Key: HBASE-6228 URL: https://issues.apache.org/jira/browse/HBASE-6228 Project: HBase Issue Type: Bug Components: master Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0 Attachments: HBASE-6228.patch, HBASE-6228v2.patch, HBASE-6228v2.patch First, how fixup daughters twice happen? 1.we will fixupDaughters at the last of HMaster#finishInitialization 2.ServerShutdownHandler will fixupDaughters when reassigning region through ServerShutdownHandler#processDeadRegion When fixupDaughters, we will added daughters to .META., but it coudn't prevent the above case, because FindDaughterVisitor. The detail is as the following: Suppose region A is a splitted parent region, and its daughter region B is missing 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B to .META. with serverName=null, and assign the daughter. 2.Then, Master's initialization thread will also find the daughter region B is missing and assign it. It is because FindDaughterVisitor consider daughter is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6156) Improve multiop performances in HTable#flushCommits
[
https://issues.apache.org/jira/browse/HBASE-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397556#comment-13397556
]
nkeywal commented on HBASE-6156:
Small status as of June.
* Improvements identified
Failure detection time: performed by ZK, with a timeout. With the default
value, we needed 90 seconds before starting to act on a software or hardware
issue.
Recovery time - server side: split in two parts: reassigning the regions of a
dead RS to a new RS, replaying the WAL. Must be as fast as possible.
Recovery time - client side: errors should be transparent for the user code.
On the client side, we must as well limit the time lost on errors to a minimum.
Planned rolling restart: just make this as fast and less disruptive as
possible
Other possible changes. detailed below.
* Status
Failure detection time: software crash - done
Done in HBASE-5844, HBASE-5926
Failure detection time: hardware issue - not started
1) as much as possible, it should be handled by ZooKeeper and not HBase, see
open Jira as ZOOKEEPER-702, ZOOKEEPER-922, ...
2) we need to make easy for a monitoring tool to tag a RS or Master as dead.
This way, specialized HW tools could point out dead RS. Jira to open.
Recovery time - Server: in progress
1) bulk assignment: To be retested, there are many just-closed JIRA on this
(HBASE-5998, HBASE-6109, HBASE-5970, ...). A lot of work by many people. There
are still possible improvements (HBASE-6058, ...)
2) Log replay: To be retested, there are many just-closed JIRA on this
(HBASE-6134, ...).
Recovery time - Client - done
1) The RS now returns the new RS to the client after a region move
(HBASE-5992, HBASE-5877)
2) Client retries sooner on errors (HBASE-5924).
3) In the future, it could be interesting to share the region location in ZK
with the client. It's not reasonable today as it could lead to have too many
connection to ZK. ZOOKEEPER-1147 is an open JIRA on this.
Planned rolling restart performances - in progress
Benefits from the modifications in the client mentioned above.
To do: analyze move performances to make it faster if possible.
Other possible changes
- Restart the server immediately on software crash: done in HBASE-5939
- Reuse the same assignment on software crash: not planned
- Use spare hardware to reuse the same assignment on hardware failure: not
planned
- Multiple RS for the same region (excluded in the initial document: hbase
architecture change previously discussed by Gary H./Andy P.): not planned
Improve multiop performances in HTable#flushCommits
---
Key: HBASE-6156
URL: https://issues.apache.org/jira/browse/HBASE-6156
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Fix For: 0.96.0
This code:
{noformat}
@Override
public void flushCommits() throws IOException {
try {
Object[] results = new Object[writeBuffer.size()];
try {
this.connection.processBatch(writeBuffer, tableName, pool, results);
} catch (InterruptedException e) {
throw new IOException(e);
} finally {
// mutate list so that it is empty for complete success, or contains
// only failed records results are returned in the same order as the
// requests in list walk the list backwards, so we can remove from
list
// without impacting the indexes of earlier members
for (int i = results.length - 1; i=0; i--) {
if (results[i] instanceof Result) {
// successful Puts are removed from the list here.
writeBuffer.remove(i);
}
}
}
} finally {
if (clearBufferOnFail) {
writeBuffer.clear();
currentWriteBufferSize = 0;
} else {
// the write buffer was adjusted by processBatchOfPuts
currentWriteBufferSize = 0;
for (Put aPut : writeBuffer) {
currentWriteBufferSize += aPut.heapSize();
}
}
}
}
{noformat}
Can be improved by:
- not iterating on the list if clearBufferOnFail is set
- not iterating the the list of there are no error
- iterating on the list only once instead of two when we really have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5843) Improve HBase MTTR - Mean Time To Recover
[ https://issues.apache.org/jira/browse/HBASE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397559#comment-13397559 ] nkeywal commented on HBASE-5843: Small status as of June. * Improvements identified Failure detection time: performed by ZK, with a timeout. With the default value, we needed 90 seconds before starting to act on a software or hardware issue. Recovery time - server side: split in two parts: reassigning the regions of a dead RS to a new RS, replaying the WAL. Must be as fast as possible. Recovery time - client side: errors should be transparent for the user code. On the client side, we must as well limit the time lost on errors to a minimum. Planned rolling restart: just make this as fast and less disruptive as possible Other possible changes. detailed below. * Status Failure detection time: software crash - done Done in HBASE-5844, HBASE-5926 Failure detection time: hardware issue - not started 1) as much as possible, it should be handled by ZooKeeper and not HBase, see open Jira as ZOOKEEPER-702, ZOOKEEPER-922, ... 2) we need to make easy for a monitoring tool to tag a RS or Master as dead. This way, specialized HW tools could point out dead RS. Jira to open. Recovery time - Server: in progress 1) bulk assignment: To be retested, there are many just-closed JIRA on this (HBASE-5998, HBASE-6109, HBASE-5970, ...). A lot of work by many people. There are still possible improvements (HBASE-6058, ...) 2) Log replay: To be retested, there are many just-closed JIRA on this (HBASE-6134, ...). Recovery time - Client - done 1) The RS now returns the new RS to the client after a region move (HBASE-5992, HBASE-5877) 2) Client retries sooner on errors (HBASE-5924). 3) In the future, it could be interesting to share the region location in ZK with the client. It's not reasonable today as it could lead to have too many connection to ZK. ZOOKEEPER-1147 is an open JIRA on this. Planned rolling restart performances - in progress Benefits from the modifications in the client mentioned above. To do: analyze move performances to make it faster if possible. Other possible changes Restart the server immediately on software crash: done in HBASE-5939 Reuse the same assignment on software crash: not planned Use spare hardware to reuse the same assignment on hardware failure: not planned Multiple RS for the same region (excluded in the initial document: hbase architecture change previously discussed by Gary H./Andy P.): not planned Improve HBase MTTR - Mean Time To Recover - Key: HBASE-5843 URL: https://issues.apache.org/jira/browse/HBASE-5843 Project: HBase Issue Type: Umbrella Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal A part of the approach is described here: https://docs.google.com/document/d/1z03xRoZrIJmg7jsWuyKYl6zNournF_7ZHzdi0qz_B4c/edit The ideal target is: - failure impact client applications only by an added delay to execute a query, whatever the failure. - this delay is always inferior to 1 second. We're not going to achieve that immediately... Priority will be given to the most frequent issues. Short term: - software crash - standard administrative tasks as stop/start of a cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6156) Improve multiop performances in HTable#flushCommits
[
https://issues.apache.org/jira/browse/HBASE-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397562#comment-13397562
]
nkeywal commented on HBASE-6156:
It seems I can't update my comments... The comment above is for hbase-5843.
Improve multiop performances in HTable#flushCommits
---
Key: HBASE-6156
URL: https://issues.apache.org/jira/browse/HBASE-6156
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Fix For: 0.96.0
This code:
{noformat}
@Override
public void flushCommits() throws IOException {
try {
Object[] results = new Object[writeBuffer.size()];
try {
this.connection.processBatch(writeBuffer, tableName, pool, results);
} catch (InterruptedException e) {
throw new IOException(e);
} finally {
// mutate list so that it is empty for complete success, or contains
// only failed records results are returned in the same order as the
// requests in list walk the list backwards, so we can remove from
list
// without impacting the indexes of earlier members
for (int i = results.length - 1; i=0; i--) {
if (results[i] instanceof Result) {
// successful Puts are removed from the list here.
writeBuffer.remove(i);
}
}
}
} finally {
if (clearBufferOnFail) {
writeBuffer.clear();
currentWriteBufferSize = 0;
} else {
// the write buffer was adjusted by processBatchOfPuts
currentWriteBufferSize = 0;
for (Put aPut : writeBuffer) {
currentWriteBufferSize += aPut.heapSize();
}
}
}
}
{noformat}
Can be improved by:
- not iterating on the list if clearBufferOnFail is set
- not iterating the the list of there are no error
- iterating on the list only once instead of two when we really have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5362) ZK connection leak
[ https://issues.apache.org/jira/browse/HBASE-5362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397565#comment-13397565 ] Ionut Ignatescu commented on HBASE-5362: Any update on this?Any fix? ZK connection leak -- Key: HBASE-5362 URL: https://issues.apache.org/jira/browse/HBASE-5362 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.90.3 Reporter: Ionut Ignatescu I have an application like a scheduler that starts periodically some MR jobs that reads from one HBase table and write in more tables, changing the row key and the columns list. I have encounted a problem with this: after each MR job, I have +1 connections open to ZK and after a period of time, I have received IOException. I saw that I am not the first that had had this problem, but I didn't see any fix. No in HBase 0.90.4, nor in HBase 0.92. To fix this issue I wrote a custom TableInputFormat that forces to close the connection to ZK after each getSplits call. Also, the initTableMapperJob was overwritten. I found an open JIRA issue here: https://issues.apache.org/jira/browse/HBASE-3792 Regards, Ionut I. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6244) Restful get and scan don't need to add column
Xing Shi created HBASE-6244:
---
Summary: Restful get and scan don't need to add column
Key: HBASE-6244
URL: https://issues.apache.org/jira/browse/HBASE-6244
Project: HBase
Issue Type: Improvement
Components: rest
Reporter: Xing Shi
Priority: Minor
Now, the RowResultGenerator and the ScanerResultGenerator will fit the column
family if the request doesn't contain any column info.
{code}
if (rowspec.hasColumns()) {
//addColumn for get or scan
} else {
for (HColumnDescriptor family:
table.getTableDescriptor().getFamilies()) {
scan/get.addFamily(family.getName());
}
}
{code}
The table.getTableDescriptor() will cost 10+ milliseconds in our hbase cluster
each request. We can remove these code because the server will auto add the
columns.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6244) Restful get and scan don't need to add columns if no column specified
[
https://issues.apache.org/jira/browse/HBASE-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xing Shi updated HBASE-6244:
Summary: Restful get and scan don't need to add columns if no column
specified (was: Restful get and scan don't need to add column)
Restful get and scan don't need to add columns if no column specified
-
Key: HBASE-6244
URL: https://issues.apache.org/jira/browse/HBASE-6244
Project: HBase
Issue Type: Improvement
Components: rest
Reporter: Xing Shi
Priority: Minor
Now, the RowResultGenerator and the ScanerResultGenerator will fit the column
family if the request doesn't contain any column info.
{code}
if (rowspec.hasColumns()) {
//addColumn for get or scan
} else {
for (HColumnDescriptor family:
table.getTableDescriptor().getFamilies()) {
scan/get.addFamily(family.getName());
}
}
{code}
The table.getTableDescriptor() will cost 10+ milliseconds in our hbase
cluster each request. We can remove these code because the server will auto
add the columns.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5918) Master will block forever at startup if root server dies between assigning root and assigning meta
[
https://issues.apache.org/jira/browse/HBASE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Ted Yu updated HBASE-5918:
--
Summary: Master will block forever at startup if root server dies between
assigning root and assigning meta (was: Master will block forever when startup
if root server died between assign root and assign meta)
Master will block forever at startup if root server dies between assigning
root and assigning meta
--
Key: HBASE-5918
URL: https://issues.apache.org/jira/browse/HBASE-5918
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.1
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: HBASE-5918.patch, HBASE-5918.patch, HBASE-5918V2.patch
When master is initializing, if root server died between assign root and
assign meta, master will block at
HMaster#assignRootAndMeta:{code}assignmentManager.assignMeta();
this.catalogTracker.waitForMeta();{code}
because ServerShutdownHandler is disabled,
So we should enable ServerShutdownHandler after called
assignmentManager.assignMeta();
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5918) Master will block forever at startup if root server dies between assigning root and assigning meta
[
https://issues.apache.org/jira/browse/HBASE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397583#comment-13397583
]
Zhihong Ted Yu commented on HBASE-5918:
---
Patch v2 integrated to trunk.
Thanks for the patch, Chunhui.
Thanks for the review, Stack and Ram.
Master will block forever at startup if root server dies between assigning
root and assigning meta
--
Key: HBASE-5918
URL: https://issues.apache.org/jira/browse/HBASE-5918
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.1
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: HBASE-5918.patch, HBASE-5918.patch, HBASE-5918V2.patch
When master is initializing, if root server died between assign root and
assign meta, master will block at
HMaster#assignRootAndMeta:{code}assignmentManager.assignMeta();
this.catalogTracker.waitForMeta();{code}
because ServerShutdownHandler is disabled,
So we should enable ServerShutdownHandler after called
assignmentManager.assignMeta();
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5918) Master will block forever at startup if root server dies between assigning root and assigning meta
[
https://issues.apache.org/jira/browse/HBASE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397635#comment-13397635
]
Hudson commented on HBASE-5918:
---
Integrated in HBase-TRUNK #3048 (See
[https://builds.apache.org/job/HBase-TRUNK/3048/])
HBASE-5918 Master will block forever at startup if root server dies between
assigning root and assigning meta (Chunhui) (Revision 1352161)
Result = SUCCESS
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Master will block forever at startup if root server dies between assigning
root and assigning meta
--
Key: HBASE-5918
URL: https://issues.apache.org/jira/browse/HBASE-5918
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.1
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: HBASE-5918.patch, HBASE-5918.patch, HBASE-5918V2.patch
When master is initializing, if root server died between assign root and
assign meta, master will block at
HMaster#assignRootAndMeta:{code}assignmentManager.assignMeta();
this.catalogTracker.waitForMeta();{code}
because ServerShutdownHandler is disabled,
So we should enable ServerShutdownHandler after called
assignmentManager.assignMeta();
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6164) Correct the bug in block encoding usage in bulkload
[ https://issues.apache.org/jira/browse/HBASE-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-6164: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and 0.94 Thanks for the patch Anoop. Thanks for the review Ted. Correct the bug in block encoding usage in bulkload --- Key: HBASE-6164 URL: https://issues.apache.org/jira/browse/HBASE-6164 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.96.0, 0.94.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.96.0, 0.94.1 Attachments: 6164_94.patch, 6164_Trunk.patch Address the issue raised under HBASE-6040 https://issues.apache.org/jira/browse/HBASE-6040?focusedCommentId=13289334page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13289334 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6222) Add per-KeyValue Security
[ https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397665#comment-13397665 ] Eric Newton commented on HBASE-6222: For clarification, Accumulo does not support negation in the visibility expressions. ! may be reserved for future use, but expressions are limited to terms, or | and parens for precedence. Visibility expressions are not required: the metadata table does not use them, for example. The visibility needs to be part of the key in order to have different values at different visibilities. @Jonathan - enjoyed your talk @hadoop summit. Add per-KeyValue Security - Key: HBASE-6222 URL: https://issues.apache.org/jira/browse/HBASE-6222 Project: HBase Issue Type: New Feature Components: security Reporter: stack Saw an interesting article: http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14 The Senate Armed Services Committee version of the fiscal 2013 national defense authorization act (S. 3254) would require DoD agencies to foreswear the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO certifies that there exists either no viable commercial open source database with security features comparable to [Accumulo] (such as the HBase or Cassandra databases)... Not sure what a 'commercial open source database' is, and I'm not sure whats going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' like Accumulo's, we might put ourselves in the running for federal contributions? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5918) Master will block forever at startup if root server dies between assigning root and assigning meta
[
https://issues.apache.org/jira/browse/HBASE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5918:
--
Attachment: HBASE-5918_0.94
Patch for 0.94.
Master will block forever at startup if root server dies between assigning
root and assigning meta
--
Key: HBASE-5918
URL: https://issues.apache.org/jira/browse/HBASE-5918
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.1
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: HBASE-5918.patch, HBASE-5918.patch, HBASE-5918V2.patch,
HBASE-5918_0.94
When master is initializing, if root server died between assign root and
assign meta, master will block at
HMaster#assignRootAndMeta:{code}assignmentManager.assignMeta();
this.catalogTracker.waitForMeta();{code}
because ServerShutdownHandler is disabled,
So we should enable ServerShutdownHandler after called
assignmentManager.assignMeta();
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5918) Master will block forever at startup if root server dies between assigning root and assigning meta
[
https://issues.apache.org/jira/browse/HBASE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5918:
--
Resolution: Fixed
Fix Version/s: 0.94.1
0.96.0
Status: Resolved (was: Patch Available)
Resolving after committing it to 0.94.
Master will block forever at startup if root server dies between assigning
root and assigning meta
--
Key: HBASE-5918
URL: https://issues.apache.org/jira/browse/HBASE-5918
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.1
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.1
Attachments: HBASE-5918.patch, HBASE-5918.patch, HBASE-5918V2.patch,
HBASE-5918_0.94
When master is initializing, if root server died between assign root and
assign meta, master will block at
HMaster#assignRootAndMeta:{code}assignmentManager.assignMeta();
this.catalogTracker.waitForMeta();{code}
because ServerShutdownHandler is disabled,
So we should enable ServerShutdownHandler after called
assignmentManager.assignMeta();
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5843) Improve HBase MTTR - Mean Time To Recover
[ https://issues.apache.org/jira/browse/HBASE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397674#comment-13397674 ] ramkrishna.s.vasudevan commented on HBASE-5843: --- @N Just for update, I feel HBASE-6060(under progress) may also be related to MTTR. Improve HBase MTTR - Mean Time To Recover - Key: HBASE-5843 URL: https://issues.apache.org/jira/browse/HBASE-5843 Project: HBase Issue Type: Umbrella Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal A part of the approach is described here: https://docs.google.com/document/d/1z03xRoZrIJmg7jsWuyKYl6zNournF_7ZHzdi0qz_B4c/edit The ideal target is: - failure impact client applications only by an added delay to execute a query, whatever the failure. - this delay is always inferior to 1 second. We're not going to achieve that immediately... Priority will be given to the most frequent issues. Short term: - software crash - standard administrative tasks as stop/start of a cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6245) Upon page refresh new UI should return to the previously selected tab
Andrew Purtell created HBASE-6245: - Summary: Upon page refresh new UI should return to the previously selected tab Key: HBASE-6245 URL: https://issues.apache.org/jira/browse/HBASE-6245 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Andrew Purtell Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6245) Upon page refresh new UI should return to the previously selected tab
[ https://issues.apache.org/jira/browse/HBASE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6245: -- Component/s: regionserver master Tags: noob Upon page refresh new UI should return to the previously selected tab - Key: HBASE-6245 URL: https://issues.apache.org/jira/browse/HBASE-6245 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6243) New UI should space detailed latency columns equally
[ https://issues.apache.org/jira/browse/HBASE-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6243: -- Component/s: regionserver master Tags: noob New UI should space detailed latency columns equally Key: HBASE-6243 URL: https://issues.apache.org/jira/browse/HBASE-6243 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Priority: Minor Spacing between the columns of the detailed latencies tab should be roughly equal. Round latencies to two digits right of decimal point. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6242) New UI should color task list entries
[ https://issues.apache.org/jira/browse/HBASE-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6242: -- Component/s: regionserver master Tags: noob New UI should color task list entries - Key: HBASE-6242 URL: https://issues.apache.org/jira/browse/HBASE-6242 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Priority: Minor The old UI changed the background color of tasklist entries according to their final status: green if successful, yellow if aborted, red if failed. Bring this back in the new UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6195) Increment data will be lost when the memstore is flushed
[
https://issues.apache.org/jira/browse/HBASE-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397696#comment-13397696
]
Zhihong Ted Yu commented on HBASE-6195:
---
nanoTime is used to calculate duration. It wouldn't fit the following:
{code}
KeyValue newKV = new KeyValue(row, family.getKey(), column.getKey(),
now, Bytes.toBytes(amount));
{code}
Store doesn't provide access to its memstore directly.
Increment data will be lost when the memstore is flushed
Key: HBASE-6195
URL: https://issues.apache.org/jira/browse/HBASE-6195
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: Xing Shi
Assignee: ShiXing
Fix For: 0.96.0, 0.94.1
Attachments: 6195-trunk-V7.patch, 6195.addendum,
HBASE-6195-trunk-V2.patch, HBASE-6195-trunk-V3.patch,
HBASE-6195-trunk-V4.patch, HBASE-6195-trunk-V5.patch,
HBASE-6195-trunk-V6.patch, HBASE-6195-trunk.patch
There are two problems in increment() now:
First:
I see that the timestamp(the variable now) in HRegion's Increment() is
generated before got the rowLock, so when there are multi-thread increment
the same row, although it generate earlier, it may got the lock later.
Because increment just store one version, so till now, the result will still
be right.
When the region is flushing, these increment will read the kv from snapshot
and memstore with whose timestamp is larger, and write it back to memstore.
If the snapshot's timestamp larger than the memstore, the increment will got
the old data and then do the increment, it's wrong.
Secondly:
Also there is a risk in increment. Because it writes the memstore first and
then HLog, so if it writes HLog failed, the client will also read the
incremented value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6242) New UI should color task list entries
[ https://issues.apache.org/jira/browse/HBASE-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397698#comment-13397698 ] Elliott Clark commented on HBASE-6242: -- Ooops, missed that. Nice catch. New UI should color task list entries - Key: HBASE-6242 URL: https://issues.apache.org/jira/browse/HBASE-6242 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Priority: Minor The old UI changed the background color of tasklist entries according to their final status: green if successful, yellow if aborted, red if failed. Bring this back in the new UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6246) Admin.move() without specifying destination calls am.unassign() and does not go through AccessController.
ramkrishna.s.vasudevan created HBASE-6246:
-
Summary: Admin.move() without specifying destination calls
am.unassign() and does not go through AccessController.
Key: HBASE-6246
URL: https://issues.apache.org/jira/browse/HBASE-6246
Project: HBase
Issue Type: Bug
Components: security
Reporter: ramkrishna.s.vasudevan
{code}
if (destServerName == null || destServerName.length == 0) {
LOG.info(Passed destination servername is null/empty so +
choosing a server at random);
this.assignmentManager.clearRegionPlan(hri);
// Unassign will reassign it elsewhere choosing random server.
this.assignmentManager.unassign(hri);
{code}
I think we should go through security to see if there is sufficient permissions
to do this operation?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6247) [REST] HTablePool.putTable is deprecated
Andrew Purtell created HBASE-6247: - Summary: [REST] HTablePool.putTable is deprecated Key: HBASE-6247 URL: https://issues.apache.org/jira/browse/HBASE-6247 Project: HBase Issue Type: Improvement Components: rest Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor HTablePool.putTable is deprecated, use returnTable instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6227) SSH and cluster startup causes data loss
[ https://issues.apache.org/jira/browse/HBASE-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397750#comment-13397750 ] Zhihong Ted Yu commented on HBASE-6227: --- +1 on patch v2. SSH and cluster startup causes data loss - Key: HBASE-6227 URL: https://issues.apache.org/jira/browse/HBASE-6227 Project: HBase Issue Type: Bug Components: master Reporter: chunhui shen Assignee: chunhui shen Attachments: HBASE-6227.patch, HBASE-6227v2.patch In AssignmentManager#processDeadServersAndRegionsInTransition, if servershutdownhandler is processing and master consider it cluster startup, master will assign all user regions, however, servershutdownhandler has not completed splitting logs. Let me describe it in more detail. Suppose there are two regionservers A1 and B1, A1 carried META and ROOT 1.master restart and completed assignRootAndMeta 2.A1 and B1 are both restarted, new regionservers are A2 and B2 3.SSH which processed for A1 completed assigning ROOT and META 4.master do rebuilding user regions and no regions added to master's region list 5.master consider it as a cluster startup, and assign all user regions 6.SSH which processed for B1 start to split B1's logs 7.All regions' data carried by B1 would loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6229) AM.assign() should not set table state to ENABLED directly.
[ https://issues.apache.org/jira/browse/HBASE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397761#comment-13397761 ] Zhihong Ted Yu commented on HBASE-6229: --- Integrated to 0.94 and trunk. Thanks for the patch, Rajesh. Thanks for the review Laxman and Ram. AM.assign() should not set table state to ENABLED directly. --- Key: HBASE-6229 URL: https://issues.apache.org/jira/browse/HBASE-6229 Project: HBase Issue Type: Bug Affects Versions: 0.92.2, 0.94.1 Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.96.0, 0.94.1, 0.92.3 Attachments: 6229_trunk_2.patch, HBASE-6229_94.patch, HBASE-6229_94_1.patch, HBASE-6229_94_2.patch, HBASE-6229_trunk.patch, HBASE-6229_trunk_1.patch In case of assign from EnableTableHandler table state is ENABLING. Any how EnableTableHandler will set ENABLED after assigning all the the table regions. If we try to set to ENABLED directly then client api may think ENABLE table is completed. When we have a case like all the regions are added directly into META and we call assignRegion then we need to make the table ENABLED. Hence in such case the table will not be in ENABLING or ENABLED state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5352) ACL improvements
[ https://issues.apache.org/jira/browse/HBASE-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397772#comment-13397772 ] Laxman commented on HBASE-5352: --- Patch available for: HBASE-6209, HBASE-6238, HBASE-6192 Please review. ACL improvements Key: HBASE-5352 URL: https://issues.apache.org/jira/browse/HBASE-5352 Project: HBase Issue Type: Improvement Components: security Affects Versions: 0.92.1, 0.94.0 Reporter: Enis Soztutar Assignee: Enis Soztutar In this issue I would like to open discussion for a few minor ACL related improvements. The proposed changes are as follows: 1. Introduce something like AccessControllerProtocol.checkPermissions(Permission[] permissions) API, so that clients can check access rights before carrying out the operations. We need this kind of operation for HCATALOG-245, which introduces authorization providers for hbase over hcat. We cannot use getUserPermissions() since it requires ADMIN permissions on the global/table level. 2. getUserPermissions(tableName)/grant/revoke and drop/modify table operations should not check for global CREATE/ADMIN rights, but table CREATE/ADMIN rights. The reasoning is that if a user is able to admin or read from a table, she should be able to read the table's permissions. We can choose whether we want only READ or ADMIN permissions for getUserPermission(). Since we check for global permissions first for table permissions, configuring table access using global permissions will continue to work. 3. Grant/Revoke global permissions - HBASE-5342 (included for completeness) From all 3, we may want to backport the first one to 0.92 since without it, Hive/Hcatalog cannot use Hbase's authorization mechanism effectively. I will create subissues and convert HBASE-5342 to a subtask when we get some feedback, and opinions for going further. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6229) AM.assign() should not set table state to ENABLED directly.
[
https://issues.apache.org/jira/browse/HBASE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397774#comment-13397774
]
Zhihong Ted Yu commented on HBASE-6229:
---
I ran TestHBaseFsck manually with patch v2 and it passed.
{code}
Running org.apache.hadoop.hbase.util.TestHBaseFsck
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 243.228 sec
Results :
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] --- maven-surefire-plugin:2.10:test (secondPartTestsExecution) @
hbase-server ---
[INFO] Tests are skipped.
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 4:28.120s
{code}
AM.assign() should not set table state to ENABLED directly.
---
Key: HBASE-6229
URL: https://issues.apache.org/jira/browse/HBASE-6229
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.2, 0.94.1
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.96.0, 0.94.1, 0.92.3
Attachments: 6229_trunk_2.patch, HBASE-6229_94.patch,
HBASE-6229_94_1.patch, HBASE-6229_94_2.patch, HBASE-6229_trunk.patch,
HBASE-6229_trunk_1.patch
In case of assign from EnableTableHandler table state is ENABLING. Any how
EnableTableHandler will set ENABLED after assigning all the the table
regions. If we try to set to ENABLED directly then client api may think
ENABLE table is completed. When we have a case like all the regions are added
directly into META and we call assignRegion then we need to make the table
ENABLED. Hence in such case the table will not be in ENABLING or ENABLED
state.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6164) Correct the bug in block encoding usage in bulkload
[ https://issues.apache.org/jira/browse/HBASE-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397778#comment-13397778 ] Hudson commented on HBASE-6164: --- Integrated in HBase-0.94 #263 (See [https://builds.apache.org/job/HBase-0.94/263/]) HBASE-6164 Correct the bug in block encoding usage in bulkload Submitted by:Anoop Reviewed by:Ted (Revision 1352221) Result = SUCCESS ramkrishna : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java Correct the bug in block encoding usage in bulkload --- Key: HBASE-6164 URL: https://issues.apache.org/jira/browse/HBASE-6164 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.96.0, 0.94.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.96.0, 0.94.1 Attachments: 6164_94.patch, 6164_Trunk.patch Address the issue raised under HBASE-6040 https://issues.apache.org/jira/browse/HBASE-6040?focusedCommentId=13289334page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13289334 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5918) Master will block forever at startup if root server dies between assigning root and assigning meta
[
https://issues.apache.org/jira/browse/HBASE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397779#comment-13397779
]
Hudson commented on HBASE-5918:
---
Integrated in HBase-0.94 #263 (See
[https://builds.apache.org/job/HBase-0.94/263/])
HBASE-5918 Master will block forever at startup if root server dies between
assigning root and assigning meta
Submitted by:Chunhui
Reviewed by:Stack, Ted, Ram (Revision 1352229)
Result = SUCCESS
ramkrishna :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Master will block forever at startup if root server dies between assigning
root and assigning meta
--
Key: HBASE-5918
URL: https://issues.apache.org/jira/browse/HBASE-5918
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.1
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.1
Attachments: HBASE-5918.patch, HBASE-5918.patch, HBASE-5918V2.patch,
HBASE-5918_0.94
When master is initializing, if root server died between assign root and
assign meta, master will block at
HMaster#assignRootAndMeta:{code}assignmentManager.assignMeta();
this.catalogTracker.waitForMeta();{code}
because ServerShutdownHandler is disabled,
So we should enable ServerShutdownHandler after called
assignmentManager.assignMeta();
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6246) Admin.move() without specifying destination calls am.unassign() and does not go through AccessController.
[
https://issues.apache.org/jira/browse/HBASE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laxman updated HBASE-6246:
--
Component/s: coprocessors
Affects Version/s: 0.94.1
0.94.0
Labels: coprocessors security (was: )
Seems to be 0.94 specific issue. Code looks good in trunk.
Admin.move() without specifying destination calls am.unassign() and does not
go through AccessController.
-
Key: HBASE-6246
URL: https://issues.apache.org/jira/browse/HBASE-6246
Project: HBase
Issue Type: Bug
Components: coprocessors, security
Affects Versions: 0.94.0, 0.94.1
Reporter: ramkrishna.s.vasudevan
Labels: coprocessors, security
{code}
if (destServerName == null || destServerName.length == 0) {
LOG.info(Passed destination servername is null/empty so +
choosing a server at random);
this.assignmentManager.clearRegionPlan(hri);
// Unassign will reassign it elsewhere choosing random server.
this.assignmentManager.unassign(hri);
{code}
I think we should go through security to see if there is sufficient
permissions to do this operation?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5991) Introduce sequential ZNode based read/write locks
[ https://issues.apache.org/jira/browse/HBASE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397811#comment-13397811 ] Zhihong Ted Yu commented on HBASE-5991: --- @Alex: Are you working on this issue ? Thanks Introduce sequential ZNode based read/write locks -- Key: HBASE-5991 URL: https://issues.apache.org/jira/browse/HBASE-5991 Project: HBase Issue Type: Improvement Reporter: Alex Feinberg Assignee: Alex Feinberg This is a continuation of HBASE-5494: Currently table-level write locks have been implemented using non-sequential ZNodes as part of HBASE-5494 and committed to 89-fb branch. This issue is to track converting the table-level locks to sequential ZNodes and supporting read-write locks, as to solve the issue of preventing schema changes during region splits or merges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-5547: -- Attachment: hbase-5447-v8.patch Patch from Jesse. Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates Attachments: hbase-5447-v8.patch, java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397815#comment-13397815 ] Hadoop QA commented on HBASE-5547: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532749/hbase-5447-v8.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2201//console This message is automatically generated. Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates Attachments: hbase-5447-v8.patch, java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6229) AM.assign() should not set table state to ENABLED directly.
[ https://issues.apache.org/jira/browse/HBASE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397821#comment-13397821 ] Hudson commented on HBASE-6229: --- Integrated in HBase-0.94 #264 (See [https://builds.apache.org/job/HBase-0.94/264/]) HBASE-6229 AM.assign() should not set table state to ENABLED directly (Rajesh) (Revision 1352263) Result = SUCCESS tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java AM.assign() should not set table state to ENABLED directly. --- Key: HBASE-6229 URL: https://issues.apache.org/jira/browse/HBASE-6229 Project: HBase Issue Type: Bug Affects Versions: 0.92.2, 0.94.1 Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.96.0, 0.94.1, 0.92.3 Attachments: 6229_trunk_2.patch, HBASE-6229_94.patch, HBASE-6229_94_1.patch, HBASE-6229_94_2.patch, HBASE-6229_trunk.patch, HBASE-6229_trunk_1.patch In case of assign from EnableTableHandler table state is ENABLING. Any how EnableTableHandler will set ENABLED after assigning all the the table regions. If we try to set to ENABLED directly then client api may think ENABLE table is completed. When we have a case like all the regions are added directly into META and we call assignRegion then we need to make the table ENABLED. Hence in such case the table will not be in ENABLING or ENABLED state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5953) Expose the current state of the balancerSwitch
[ https://issues.apache.org/jira/browse/HBASE-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397824#comment-13397824 ] Gregory Chanan commented on HBASE-5953: --- Sounds fine to me, Anoop. Do we have a link (a JIRA is fine) where we discuss conventions like this? Expose the current state of the balancerSwitch -- Key: HBASE-5953 URL: https://issues.apache.org/jira/browse/HBASE-5953 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.96.0 Reporter: David S. Wang Assignee: Gregory Chanan Priority: Minor Attachments: HBASE-5953.patch The balancerSwitch as in both 0.90 and 0.92.1 is implemented in such a way that it is impossible to retrieve its value without changing it: /** Turn the load balancer on or off. @param b If true, enable balancer. If false, disable balancer. @return Previous balancer value */ public boolean balanceSwitch(final boolean b); It would be nice to have a way to just get the current state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397848#comment-13397848 ] Zhihong Ted Yu commented on HBASE-5547: --- @Jesse: HConstants.java has moved to hbase-common Can you rebase your patch ? Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates Attachments: hbase-5447-v8.patch, java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6248) Jetty init may fail if directory name contains master
Dave Revell created HBASE-6248: -- Summary: Jetty init may fail if directory name contains master Key: HBASE-6248 URL: https://issues.apache.org/jira/browse/HBASE-6248 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.0 Reporter: Dave Revell Assignee: Dave Revell Priority: Minor InfoServer.getWebAppsPath() unsafely assumes that any instance of the string master in the full path to hbase-webapps can be truncated. This breaks in the case where hbase is installed in a directory such as /my/hbasemaster/. The result is that Jetty initialization will fail since the master determines an incorrect path to hbase-webapps. The master still runs but the web UI returns 503. I have a patch for this problem that I'll upload soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6248) Jetty init may fail if directory name contains master
[ https://issues.apache.org/jira/browse/HBASE-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Revell updated HBASE-6248: --- Attachment: HBASE-6248-0.94.0-0.diff Patch v1 attached Jetty init may fail if directory name contains master --- Key: HBASE-6248 URL: https://issues.apache.org/jira/browse/HBASE-6248 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.0 Reporter: Dave Revell Assignee: Dave Revell Priority: Minor Attachments: HBASE-6248-0.94.0-0.diff InfoServer.getWebAppsPath() unsafely assumes that any instance of the string master in the full path to hbase-webapps can be truncated. This breaks in the case where hbase is installed in a directory such as /my/hbasemaster/. The result is that Jetty initialization will fail since the master determines an incorrect path to hbase-webapps. The master still runs but the web UI returns 503. I have a patch for this problem that I'll upload soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HBASE-6248) Jetty init may fail if directory name contains master
[ https://issues.apache.org/jira/browse/HBASE-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-6248 started by Dave Revell. Jetty init may fail if directory name contains master --- Key: HBASE-6248 URL: https://issues.apache.org/jira/browse/HBASE-6248 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.0 Reporter: Dave Revell Assignee: Dave Revell Priority: Minor Attachments: HBASE-6248-0.94.0-0.diff InfoServer.getWebAppsPath() unsafely assumes that any instance of the string master in the full path to hbase-webapps can be truncated. This breaks in the case where hbase is installed in a directory such as /my/hbasemaster/. The result is that Jetty initialization will fail since the master determines an incorrect path to hbase-webapps. The master still runs but the web UI returns 503. I have a patch for this problem that I'll upload soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6248) Jetty init may fail if directory name contains master
[ https://issues.apache.org/jira/browse/HBASE-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Revell updated HBASE-6248: --- Status: Patch Available (was: In Progress) Jetty init may fail if directory name contains master --- Key: HBASE-6248 URL: https://issues.apache.org/jira/browse/HBASE-6248 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.0 Reporter: Dave Revell Assignee: Dave Revell Priority: Minor Attachments: HBASE-6248-0.94.0-0.diff InfoServer.getWebAppsPath() unsafely assumes that any instance of the string master in the full path to hbase-webapps can be truncated. This breaks in the case where hbase is installed in a directory such as /my/hbasemaster/. The result is that Jetty initialization will fail since the master determines an incorrect path to hbase-webapps. The master still runs but the web UI returns 503. I have a patch for this problem that I'll upload soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5547: --- Attachment: hbase-5447-v8.patch Attaching rebased version. Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates Attachments: hbase-5447-v8.patch, hbase-5447-v8.patch, java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6244) Restful get and scan don't need to add columns if no column specified
[
https://issues.apache.org/jira/browse/HBASE-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-6244:
--
Attachment: 6244-0.94.patch
6244.patch
Restful get and scan don't need to add columns if no column specified
-
Key: HBASE-6244
URL: https://issues.apache.org/jira/browse/HBASE-6244
Project: HBase
Issue Type: Improvement
Components: rest
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Xing Shi
Priority: Minor
Attachments: 6244-0.94.patch, 6244.patch
Now, the RowResultGenerator and the ScanerResultGenerator will fit the column
family if the request doesn't contain any column info.
{code}
if (rowspec.hasColumns()) {
//addColumn for get or scan
} else {
for (HColumnDescriptor family:
table.getTableDescriptor().getFamilies()) {
scan/get.addFamily(family.getName());
}
}
{code}
The table.getTableDescriptor() will cost 10+ milliseconds in our hbase
cluster each request. We can remove these code because the server will auto
add the columns.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6244) [REST] Result generators do not need to query table schema
[
https://issues.apache.org/jira/browse/HBASE-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-6244:
--
Affects Version/s: 0.94.1
0.96.0
0.92.2
Assignee: Andrew Purtell
Summary: [REST] Result generators do not need to query table
schema (was: Restful get and scan don't need to add columns if no column
specified)
[REST] Result generators do not need to query table schema
--
Key: HBASE-6244
URL: https://issues.apache.org/jira/browse/HBASE-6244
Project: HBase
Issue Type: Improvement
Components: rest
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Xing Shi
Assignee: Andrew Purtell
Priority: Minor
Attachments: 6244-0.94.patch, 6244.patch
Now, the RowResultGenerator and the ScanerResultGenerator will fit the column
family if the request doesn't contain any column info.
{code}
if (rowspec.hasColumns()) {
//addColumn for get or scan
} else {
for (HColumnDescriptor family:
table.getTableDescriptor().getFamilies()) {
scan/get.addFamily(family.getName());
}
}
{code}
The table.getTableDescriptor() will cost 10+ milliseconds in our hbase
cluster each request. We can remove these code because the server will auto
add the columns.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6229) AM.assign() should not set table state to ENABLED directly.
[ https://issues.apache.org/jira/browse/HBASE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397874#comment-13397874 ] Hudson commented on HBASE-6229: --- Integrated in HBase-TRUNK #3050 (See [https://builds.apache.org/job/HBase-TRUNK/3050/]) HBASE-6229 AM.assign() should not set table state to ENABLED directly (Rajesh) (Revision 1352262) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java AM.assign() should not set table state to ENABLED directly. --- Key: HBASE-6229 URL: https://issues.apache.org/jira/browse/HBASE-6229 Project: HBase Issue Type: Bug Affects Versions: 0.92.2, 0.94.1 Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.96.0, 0.94.1, 0.92.3 Attachments: 6229_trunk_2.patch, HBASE-6229_94.patch, HBASE-6229_94_1.patch, HBASE-6229_94_2.patch, HBASE-6229_trunk.patch, HBASE-6229_trunk_1.patch In case of assign from EnableTableHandler table state is ENABLING. Any how EnableTableHandler will set ENABLED after assigning all the the table regions. If we try to set to ENABLED directly then client api may think ENABLE table is completed. When we have a case like all the regions are added directly into META and we call assignRegion then we need to make the table ENABLED. Hence in such case the table will not be in ENABLING or ENABLED state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5843) Improve HBase MTTR - Mean Time To Recover
[ https://issues.apache.org/jira/browse/HBASE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397876#comment-13397876 ] Andrew Purtell commented on HBASE-5843: --- HBASE-5844 and HBASE-5926 are not in 0.94, can we port these back? Seems like pretty self contained changes. Improve HBase MTTR - Mean Time To Recover - Key: HBASE-5843 URL: https://issues.apache.org/jira/browse/HBASE-5843 Project: HBase Issue Type: Umbrella Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal A part of the approach is described here: https://docs.google.com/document/d/1z03xRoZrIJmg7jsWuyKYl6zNournF_7ZHzdi0qz_B4c/edit The ideal target is: - failure impact client applications only by an added delay to execute a query, whatever the failure. - this delay is always inferior to 1 second. We're not going to achieve that immediately... Priority will be given to the most frequent issues. Short term: - software crash - standard administrative tasks as stop/start of a cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6244) [REST] Result generators do not need to query table schema
[
https://issues.apache.org/jira/browse/HBASE-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-6244.
---
Resolution: Fixed
Fix Version/s: 0.94.1
0.96.0
0.92.2
Committed trivial patch to trunk, 0.94, and 0.92 branches. All unit tests pass
locally for trunk. All REST unit tests pass locally for 0.94.
Many thanks for pointing out this issue, Xing!
[REST] Result generators do not need to query table schema
--
Key: HBASE-6244
URL: https://issues.apache.org/jira/browse/HBASE-6244
Project: HBase
Issue Type: Improvement
Components: rest
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Xing Shi
Assignee: Andrew Purtell
Priority: Minor
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: 6244-0.94.patch, 6244.patch
Now, the RowResultGenerator and the ScanerResultGenerator will fit the column
family if the request doesn't contain any column info.
{code}
if (rowspec.hasColumns()) {
//addColumn for get or scan
} else {
for (HColumnDescriptor family:
table.getTableDescriptor().getFamilies()) {
scan/get.addFamily(family.getName());
}
}
{code}
The table.getTableDescriptor() will cost 10+ milliseconds in our hbase
cluster each request. We can remove these code because the server will auto
add the columns.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6244) [REST] Result generators do not need to query table schema
[
https://issues.apache.org/jira/browse/HBASE-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397877#comment-13397877
]
Andrew Purtell edited comment on HBASE-6244 at 6/20/12 9:17 PM:
Committed trivial patch to trunk, 0.94, and 0.92 branches. All unit tests pass
locally for trunk. All REST unit tests pass locally for 0.94.
Many thanks for pointing out this issue, Xing!
Edit: ... and REST tests pass for 0.92.
was (Author: apurtell):
Committed trivial patch to trunk, 0.94, and 0.92 branches. All unit tests
pass locally for trunk. All REST unit tests pass locally for 0.94.
Many thanks for pointing out this issue, Xing!
[REST] Result generators do not need to query table schema
--
Key: HBASE-6244
URL: https://issues.apache.org/jira/browse/HBASE-6244
Project: HBase
Issue Type: Improvement
Components: rest
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Xing Shi
Assignee: Andrew Purtell
Priority: Minor
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: 6244-0.94.patch, 6244.patch
Now, the RowResultGenerator and the ScanerResultGenerator will fit the column
family if the request doesn't contain any column info.
{code}
if (rowspec.hasColumns()) {
//addColumn for get or scan
} else {
for (HColumnDescriptor family:
table.getTableDescriptor().getFamilies()) {
scan/get.addFamily(family.getName());
}
}
{code}
The table.getTableDescriptor() will cost 10+ milliseconds in our hbase
cluster each request. We can remove these code because the server will auto
add the columns.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6247) [REST] HTablePool.putTable is deprecated
[ https://issues.apache.org/jira/browse/HBASE-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6247: -- Attachment: 6247-0.94.patch 6247.patch [REST] HTablePool.putTable is deprecated Key: HBASE-6247 URL: https://issues.apache.org/jira/browse/HBASE-6247 Project: HBase Issue Type: Improvement Components: rest Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: 6247-0.94.patch, 6247.patch HTablePool.putTable is deprecated, use returnTable instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6247) [REST] HTablePool.putTable is deprecated
[ https://issues.apache.org/jira/browse/HBASE-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-6247. --- Resolution: Fixed Fix Version/s: 0.94.1 0.96.0 0.92.2 Committed trivial patch to trunk, 0.94, and 0.92 branches. All REST tests pass locally. [REST] HTablePool.putTable is deprecated Key: HBASE-6247 URL: https://issues.apache.org/jira/browse/HBASE-6247 Project: HBase Issue Type: Improvement Components: rest Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: 6247-0.94.patch, 6247.patch HTablePool.putTable is deprecated, use returnTable instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6058) Use ZK 3.4 API 'multi' in bulk assignment
[ https://issues.apache.org/jira/browse/HBASE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397917#comment-13397917 ] Jesse Yates commented on HBASE-6058: I think the consensus is that we don't move until 3.4 is considered stable. I've got a couple other patches up for other various zkutil things, for when we do move. Hopefully this happens around 0.96, especially since Patrick Hunt was saying that there haven't been any major issues around 3.4 (particularly around the multi stuff). Use ZK 3.4 API 'multi' in bulk assignment - Key: HBASE-6058 URL: https://issues.apache.org/jira/browse/HBASE-6058 Project: HBase Issue Type: Improvement Components: master, zookeeper Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We use async API today. This is already much much faster than the sync API. Still, it makes sense to use the 'multi' function: this will decrease the network zookeeper load at startup/rolling restart. On a 500 nodes cluster, we see 3 that 3 seconds are spent on updating ZK per bulk assignment. This should cut it in half (+ the benefits on the network/zk load). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6224) add Pre and Post coprocessor hooks for BulkLoad
[ https://issues.apache.org/jira/browse/HBASE-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6224: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532357/HBASE-6224.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2180//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2180//console This message is automatically generated.) add Pre and Post coprocessor hooks for BulkLoad --- Key: HBASE-6224 URL: https://issues.apache.org/jira/browse/HBASE-6224 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-6224.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6224) add Pre and Post coprocessor hooks for BulkLoad
[ https://issues.apache.org/jira/browse/HBASE-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397918#comment-13397918 ] Andrew Purtell commented on HBASE-6224: --- This is the patch on https://reviews.apache.org/r/5380/ right? +1 if so. add Pre and Post coprocessor hooks for BulkLoad --- Key: HBASE-6224 URL: https://issues.apache.org/jira/browse/HBASE-6224 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-6224.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6224) add Pre and Post coprocessor hooks for BulkLoad
[ https://issues.apache.org/jira/browse/HBASE-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6224: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532357/HBASE-6224.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2179//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2179//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2179//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2179//console This message is automatically generated.) add Pre and Post coprocessor hooks for BulkLoad --- Key: HBASE-6224 URL: https://issues.apache.org/jira/browse/HBASE-6224 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-6224.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6058) Use ZK 3.4 API 'multi' in bulk assignment
[ https://issues.apache.org/jira/browse/HBASE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397923#comment-13397923 ] Andrew Purtell commented on HBASE-6058: --- We run 3.4 in production and have had no issues. Multi has not had much use though, relative to the remainder of the ZK API. IMO, 0.96 is a good target for use of it, and we should get such usage in soon enough so we can have plenty of experience with it before cutting a release that uses it. Use ZK 3.4 API 'multi' in bulk assignment - Key: HBASE-6058 URL: https://issues.apache.org/jira/browse/HBASE-6058 Project: HBase Issue Type: Improvement Components: master, zookeeper Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We use async API today. This is already much much faster than the sync API. Still, it makes sense to use the 'multi' function: this will decrease the network zookeeper load at startup/rolling restart. On a 500 nodes cluster, we see 3 that 3 seconds are spent on updating ZK per bulk assignment. This should cut it in half (+ the benefits on the network/zk load). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6116) Allow parallel HDFS writes for HLogs.
[ https://issues.apache.org/jira/browse/HBASE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397933#comment-13397933 ] Andrew Purtell commented on HBASE-6116: --- I ported HDFS-1783 and this patch to Hadoop 2.0.1 and HBase 0.94. A stressful test on a 5 slave cluster with LoadTestTool is still stable after 8 hours. I only have EC2 resources so am not sure a relative performance benchmark would be that meaningful, but I could try. Allow parallel HDFS writes for HLogs. - Key: HBASE-6116 URL: https://issues.apache.org/jira/browse/HBASE-6116 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Attachments: 6116-v1.txt In HDFS-1783 I adapted Dhrubas changes to be used in Hadoop trunk. This issue will include the necessary reflection changes to optionally enable this for the WALs in HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6249) Apply ACLs to new bulk load hooks
Andrew Purtell created HBASE-6249: - Summary: Apply ACLs to new bulk load hooks Key: HBASE-6249 URL: https://issues.apache.org/jira/browse/HBASE-6249 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell HBASE-6224 adds coprocessor hooks for bulk loading. This should require table WRITE permission. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6244) [REST] Result generators do not need to query table schema
[
https://issues.apache.org/jira/browse/HBASE-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397944#comment-13397944
]
Hudson commented on HBASE-6244:
---
Integrated in HBase-0.94 #265 (See
[https://builds.apache.org/job/HBase-0.94/265/])
HBASE-6244. [REST] Result generators do not need to query table schema
(Revision 1352326)
Result = FAILURE
apurtell :
Files :
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RowResultGenerator.java
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/ScannerResultGenerator.java
[REST] Result generators do not need to query table schema
--
Key: HBASE-6244
URL: https://issues.apache.org/jira/browse/HBASE-6244
Project: HBase
Issue Type: Improvement
Components: rest
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Xing Shi
Assignee: Andrew Purtell
Priority: Minor
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: 6244-0.94.patch, 6244.patch
Now, the RowResultGenerator and the ScanerResultGenerator will fit the column
family if the request doesn't contain any column info.
{code}
if (rowspec.hasColumns()) {
//addColumn for get or scan
} else {
for (HColumnDescriptor family:
table.getTableDescriptor().getFamilies()) {
scan/get.addFamily(family.getName());
}
}
{code}
The table.getTableDescriptor() will cost 10+ milliseconds in our hbase
cluster each request. We can remove these code because the server will auto
add the columns.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6248) Jetty init may fail if directory name contains master
[
https://issues.apache.org/jira/browse/HBASE-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397946#comment-13397946
]
Zhihong Ted Yu commented on HBASE-6248:
---
{code}
+} else {
+ return p;
+}
{code}
The else statement is not needed.
Can you tell us the result of running test suite ? Hadoop QA didn't process
your patch.
Jetty init may fail if directory name contains master
---
Key: HBASE-6248
URL: https://issues.apache.org/jira/browse/HBASE-6248
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.0
Reporter: Dave Revell
Assignee: Dave Revell
Priority: Minor
Attachments: HBASE-6248-0.94.0-0.diff
InfoServer.getWebAppsPath() unsafely assumes that any instance of the string
master in the full path to hbase-webapps can be truncated. This breaks in
the case where hbase is installed in a directory such as /my/hbasemaster/.
The result is that Jetty initialization will fail since the master determines
an incorrect path to hbase-webapps. The master still runs but the web UI
returns 503.
I have a patch for this problem that I'll upload soon.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6209) ACL Corrections for AccessControllerProtocol apis
[ https://issues.apache.org/jira/browse/HBASE-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6209: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and 0.94 branch. TestAccessController passes locally. Thanks for the patch Laxman! ACL Corrections for AccessControllerProtocol apis - Key: HBASE-6209 URL: https://issues.apache.org/jira/browse/HBASE-6209 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Laxman Assignee: Laxman Labels: acl, security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6209.patch APIs provided in AccessController are authorized against global-admin permissions. Instead we need to check for table-admin level permissions. Edit: Append operation also has no authorization check. We can update it together. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6238) Grant on META not taking effect
[ https://issues.apache.org/jira/browse/HBASE-6238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6238: -- Summary: Grant on META not taking effect (was: Grant on Meta Table is not taking affect.) Grant on META not taking effect --- Key: HBASE-6238 URL: https://issues.apache.org/jira/browse/HBASE-6238 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Laxman Assignee: Laxman Labels: acl, security Attachments: HBASE-6238.patch User is not able to perform authorized operations on Meta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6238) Grant on META not taking effect
[ https://issues.apache.org/jira/browse/HBASE-6238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6238: -- Resolution: Fixed Fix Version/s: 0.94.1 0.96.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and 0.94 branch. TestAccessController passes locally. Many thanks for the patch Laxman! Grant on META not taking effect --- Key: HBASE-6238 URL: https://issues.apache.org/jira/browse/HBASE-6238 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Laxman Assignee: Laxman Labels: acl, security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6238.patch User is not able to perform authorized operations on Meta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6247) [REST] HTablePool.putTable is deprecated
[ https://issues.apache.org/jira/browse/HBASE-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397965#comment-13397965 ] Hudson commented on HBASE-6247: --- Integrated in HBase-TRUNK #3051 (See [https://builds.apache.org/job/HBase-TRUNK/3051/]) HBASE-6247. [REST] HTablePool.putTable is deprecated (Revision 1352348) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RowResultGenerator.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/ScannerResultGenerator.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java [REST] HTablePool.putTable is deprecated Key: HBASE-6247 URL: https://issues.apache.org/jira/browse/HBASE-6247 Project: HBase Issue Type: Improvement Components: rest Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: 6247-0.94.patch, 6247.patch HTablePool.putTable is deprecated, use returnTable instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6244) [REST] Result generators do not need to query table schema
[
https://issues.apache.org/jira/browse/HBASE-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397966#comment-13397966
]
Hudson commented on HBASE-6244:
---
Integrated in HBase-TRUNK #3051 (See
[https://builds.apache.org/job/HBase-TRUNK/3051/])
HBASE-6244. [REST] Result generators do not need to query table schema
(Revision 1352325)
Result = FAILURE
apurtell :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RowResultGenerator.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/ScannerResultGenerator.java
[REST] Result generators do not need to query table schema
--
Key: HBASE-6244
URL: https://issues.apache.org/jira/browse/HBASE-6244
Project: HBase
Issue Type: Improvement
Components: rest
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Xing Shi
Assignee: Andrew Purtell
Priority: Minor
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: 6244-0.94.patch, 6244.patch
Now, the RowResultGenerator and the ScanerResultGenerator will fit the column
family if the request doesn't contain any column info.
{code}
if (rowspec.hasColumns()) {
//addColumn for get or scan
} else {
for (HColumnDescriptor family:
table.getTableDescriptor().getFamilies()) {
scan/get.addFamily(family.getName());
}
}
{code}
The table.getTableDescriptor() will cost 10+ milliseconds in our hbase
cluster each request. We can remove these code because the server will auto
add the columns.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6209) ACL Corrections for AccessControllerProtocol apis
[ https://issues.apache.org/jira/browse/HBASE-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397988#comment-13397988 ] Hudson commented on HBASE-6209: --- Integrated in HBase-TRUNK #3052 (See [https://builds.apache.org/job/HBase-TRUNK/3052/]) HBASE-6209. ACL Corrections for AccessControllerProtocol (Laxman) (Revision 1352354) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java ACL Corrections for AccessControllerProtocol apis - Key: HBASE-6209 URL: https://issues.apache.org/jira/browse/HBASE-6209 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Laxman Assignee: Laxman Labels: acl, security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6209.patch APIs provided in AccessController are authorized against global-admin permissions. Instead we need to check for table-admin level permissions. Edit: Append operation also has no authorization check. We can update it together. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6238) Grant on META not taking effect
[ https://issues.apache.org/jira/browse/HBASE-6238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397987#comment-13397987 ] Hudson commented on HBASE-6238: --- Integrated in HBase-TRUNK #3052 (See [https://builds.apache.org/job/HBase-TRUNK/3052/]) HBASE-6238. Grant on META not taking effect (Laxman) (Revision 1352356) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java Grant on META not taking effect --- Key: HBASE-6238 URL: https://issues.apache.org/jira/browse/HBASE-6238 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Laxman Assignee: Laxman Labels: acl, security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6238.patch User is not able to perform authorized operations on Meta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6244) [REST] Result generators do not need to query table schema
[
https://issues.apache.org/jira/browse/HBASE-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397991#comment-13397991
]
Hudson commented on HBASE-6244:
---
Integrated in HBase-0.92 #455 (See
[https://builds.apache.org/job/HBase-0.92/455/])
HBASE-6244. [REST] Result generators do not need to query table schema
(Revision 1352327)
Result = SUCCESS
apurtell :
Files :
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/rest/RowResultGenerator.java
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/rest/ScannerResultGenerator.java
[REST] Result generators do not need to query table schema
--
Key: HBASE-6244
URL: https://issues.apache.org/jira/browse/HBASE-6244
Project: HBase
Issue Type: Improvement
Components: rest
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Xing Shi
Assignee: Andrew Purtell
Priority: Minor
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: 6244-0.94.patch, 6244.patch
Now, the RowResultGenerator and the ScanerResultGenerator will fit the column
family if the request doesn't contain any column info.
{code}
if (rowspec.hasColumns()) {
//addColumn for get or scan
} else {
for (HColumnDescriptor family:
table.getTableDescriptor().getFamilies()) {
scan/get.addFamily(family.getName());
}
}
{code}
The table.getTableDescriptor() will cost 10+ milliseconds in our hbase
cluster each request. We can remove these code because the server will auto
add the columns.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5630) hbck should disable the balancer using synchronousBalanceSwitch.
[ https://issues.apache.org/jira/browse/HBASE-5630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan reassigned HBASE-5630: - Assignee: Gregory Chanan hbck should disable the balancer using synchronousBalanceSwitch. Key: HBASE-5630 URL: https://issues.apache.org/jira/browse/HBASE-5630 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Assignee: Gregory Chanan Priority: Minor hbck disable the balancer using admin.balanceSwith(bool) when it would be preferable to use the newer synchronusBalanceSwitch method found in 0.94 and trunk branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6247) [REST] HTablePool.putTable is deprecated
[ https://issues.apache.org/jira/browse/HBASE-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13397992#comment-13397992 ] Hudson commented on HBASE-6247: --- Integrated in HBase-0.94 #266 (See [https://builds.apache.org/job/HBase-0.94/266/]) HBASE-6247. [REST] HTablePool.putTable is deprecated (Revision 1352349) Result = FAILURE apurtell : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RowResultGenerator.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/ScannerResultGenerator.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java [REST] HTablePool.putTable is deprecated Key: HBASE-6247 URL: https://issues.apache.org/jira/browse/HBASE-6247 Project: HBase Issue Type: Improvement Components: rest Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: 6247-0.94.patch, 6247.patch HTablePool.putTable is deprecated, use returnTable instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6247) [REST] HTablePool.putTable is deprecated
[ https://issues.apache.org/jira/browse/HBASE-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398012#comment-13398012 ] Hudson commented on HBASE-6247: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #61 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/61/]) HBASE-6247. [REST] HTablePool.putTable is deprecated (Revision 1352348) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RowResultGenerator.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/ScannerResultGenerator.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java [REST] HTablePool.putTable is deprecated Key: HBASE-6247 URL: https://issues.apache.org/jira/browse/HBASE-6247 Project: HBase Issue Type: Improvement Components: rest Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: 6247-0.94.patch, 6247.patch HTablePool.putTable is deprecated, use returnTable instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6229) AM.assign() should not set table state to ENABLED directly.
[ https://issues.apache.org/jira/browse/HBASE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398011#comment-13398011 ] Hudson commented on HBASE-6229: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #61 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/61/]) HBASE-6229 AM.assign() should not set table state to ENABLED directly (Rajesh) (Revision 1352262) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java AM.assign() should not set table state to ENABLED directly. --- Key: HBASE-6229 URL: https://issues.apache.org/jira/browse/HBASE-6229 Project: HBase Issue Type: Bug Affects Versions: 0.92.2, 0.94.1 Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.96.0, 0.94.1, 0.92.3 Attachments: 6229_trunk_2.patch, HBASE-6229_94.patch, HBASE-6229_94_1.patch, HBASE-6229_94_2.patch, HBASE-6229_trunk.patch, HBASE-6229_trunk_1.patch In case of assign from EnableTableHandler table state is ENABLING. Any how EnableTableHandler will set ENABLED after assigning all the the table regions. If we try to set to ENABLED directly then client api may think ENABLE table is completed. When we have a case like all the regions are added directly into META and we call assignRegion then we need to make the table ENABLED. Hence in such case the table will not be in ENABLING or ENABLED state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6238) Grant on META not taking effect
[ https://issues.apache.org/jira/browse/HBASE-6238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398010#comment-13398010 ] Hudson commented on HBASE-6238: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #61 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/61/]) HBASE-6238. Grant on META not taking effect (Laxman) (Revision 1352356) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java Grant on META not taking effect --- Key: HBASE-6238 URL: https://issues.apache.org/jira/browse/HBASE-6238 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Laxman Assignee: Laxman Labels: acl, security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6238.patch User is not able to perform authorized operations on Meta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6244) [REST] Result generators do not need to query table schema
[
https://issues.apache.org/jira/browse/HBASE-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398013#comment-13398013
]
Hudson commented on HBASE-6244:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #61 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/61/])
HBASE-6244. [REST] Result generators do not need to query table schema
(Revision 1352325)
Result = FAILURE
apurtell :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RowResultGenerator.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/ScannerResultGenerator.java
[REST] Result generators do not need to query table schema
--
Key: HBASE-6244
URL: https://issues.apache.org/jira/browse/HBASE-6244
Project: HBase
Issue Type: Improvement
Components: rest
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Xing Shi
Assignee: Andrew Purtell
Priority: Minor
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: 6244-0.94.patch, 6244.patch
Now, the RowResultGenerator and the ScanerResultGenerator will fit the column
family if the request doesn't contain any column info.
{code}
if (rowspec.hasColumns()) {
//addColumn for get or scan
} else {
for (HColumnDescriptor family:
table.getTableDescriptor().getFamilies()) {
scan/get.addFamily(family.getName());
}
}
{code}
The table.getTableDescriptor() will cost 10+ milliseconds in our hbase
cluster each request. We can remove these code because the server will auto
add the columns.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5918) Master will block forever at startup if root server dies between assigning root and assigning meta
[
https://issues.apache.org/jira/browse/HBASE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398015#comment-13398015
]
Hudson commented on HBASE-5918:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #61 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/61/])
HBASE-5918 Master will block forever at startup if root server dies between
assigning root and assigning meta (Chunhui) (Revision 1352161)
Result = FAILURE
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Master will block forever at startup if root server dies between assigning
root and assigning meta
--
Key: HBASE-5918
URL: https://issues.apache.org/jira/browse/HBASE-5918
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.1
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.1
Attachments: HBASE-5918.patch, HBASE-5918.patch, HBASE-5918V2.patch,
HBASE-5918_0.94
When master is initializing, if root server died between assign root and
assign meta, master will block at
HMaster#assignRootAndMeta:{code}assignmentManager.assignMeta();
this.catalogTracker.waitForMeta();{code}
because ServerShutdownHandler is disabled,
So we should enable ServerShutdownHandler after called
assignmentManager.assignMeta();
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6209) ACL Corrections for AccessControllerProtocol apis
[ https://issues.apache.org/jira/browse/HBASE-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398014#comment-13398014 ] Hudson commented on HBASE-6209: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #61 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/61/]) HBASE-6209. ACL Corrections for AccessControllerProtocol (Laxman) (Revision 1352354) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java ACL Corrections for AccessControllerProtocol apis - Key: HBASE-6209 URL: https://issues.apache.org/jira/browse/HBASE-6209 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Laxman Assignee: Laxman Labels: acl, security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6209.patch APIs provided in AccessController are authorized against global-admin permissions. Instead we need to check for table-admin level permissions. Edit: Append operation also has no authorization check. We can update it together. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6248) Jetty init may fail if directory name contains master
[ https://issues.apache.org/jira/browse/HBASE-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Revell updated HBASE-6248: --- Attachment: HBASE-6248-0.94.0-1.diff New patch ending in -1 removes the else statement, per Zhihong's style feedback. All tests pass when running on 0.94.0 except for org.apache.hadoop.hbase.coprocessor.TestClassLoading, but this also fails on vanilla 0.94.0 so it wasn't due to my patch. Jetty init may fail if directory name contains master --- Key: HBASE-6248 URL: https://issues.apache.org/jira/browse/HBASE-6248 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.0 Reporter: Dave Revell Assignee: Dave Revell Priority: Minor Attachments: HBASE-6248-0.94.0-0.diff, HBASE-6248-0.94.0-1.diff InfoServer.getWebAppsPath() unsafely assumes that any instance of the string master in the full path to hbase-webapps can be truncated. This breaks in the case where hbase is installed in a directory such as /my/hbasemaster/. The result is that Jetty initialization will fail since the master determines an incorrect path to hbase-webapps. The master still runs but the web UI returns 503. I have a patch for this problem that I'll upload soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6238) Grant on META not taking effect
[ https://issues.apache.org/jira/browse/HBASE-6238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398023#comment-13398023 ] Hudson commented on HBASE-6238: --- Integrated in HBase-0.94 #267 (See [https://builds.apache.org/job/HBase-0.94/267/]) HBASE-6238. Grant on META not taking effect (Laxman) (Revision 1352358) Result = FAILURE apurtell : Files : * /hbase/branches/0.94/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java Grant on META not taking effect --- Key: HBASE-6238 URL: https://issues.apache.org/jira/browse/HBASE-6238 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Laxman Assignee: Laxman Labels: acl, security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6238.patch User is not able to perform authorized operations on Meta. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6209) ACL Corrections for AccessControllerProtocol apis
[ https://issues.apache.org/jira/browse/HBASE-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398024#comment-13398024 ] Hudson commented on HBASE-6209: --- Integrated in HBase-0.94 #267 (See [https://builds.apache.org/job/HBase-0.94/267/]) HBASE-6209. ACL Corrections for AccessControllerProtocol (Laxman) (Revision 1352355) Result = FAILURE apurtell : Files : * /hbase/branches/0.94/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * /hbase/branches/0.94/security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java ACL Corrections for AccessControllerProtocol apis - Key: HBASE-6209 URL: https://issues.apache.org/jira/browse/HBASE-6209 Project: HBase Issue Type: Sub-task Components: security Affects Versions: 0.94.0, 0.96.0, 0.94.1 Reporter: Laxman Assignee: Laxman Labels: acl, security Fix For: 0.96.0, 0.94.1 Attachments: HBASE-6209.patch APIs provided in AccessController are authorized against global-admin permissions. Instead we need to check for table-admin level permissions. Edit: Append operation also has no authorization check. We can update it together. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6224) add Pre and Post coprocessor hooks for BulkLoad
[ https://issues.apache.org/jira/browse/HBASE-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6224: -- Component/s: coprocessors Affects Version/s: 0.94.1 0.96.0 add Pre and Post coprocessor hooks for BulkLoad --- Key: HBASE-6224 URL: https://issues.apache.org/jira/browse/HBASE-6224 Project: HBase Issue Type: Sub-task Components: coprocessors Affects Versions: 0.96.0, 0.94.1 Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-6224.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6248) Jetty init may fail if directory name contains master
[ https://issues.apache.org/jira/browse/HBASE-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398027#comment-13398027 ] Elliott Clark commented on HBASE-6248: -- Thanks for this. That part of the jetty init was pretty junky. Jetty init may fail if directory name contains master --- Key: HBASE-6248 URL: https://issues.apache.org/jira/browse/HBASE-6248 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.0 Reporter: Dave Revell Assignee: Dave Revell Priority: Minor Attachments: HBASE-6248-0.94.0-0.diff, HBASE-6248-0.94.0-1.diff InfoServer.getWebAppsPath() unsafely assumes that any instance of the string master in the full path to hbase-webapps can be truncated. This breaks in the case where hbase is installed in a directory such as /my/hbasemaster/. The result is that Jetty initialization will fail since the master determines an incorrect path to hbase-webapps. The master still runs but the web UI returns 503. I have a patch for this problem that I'll upload soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
