[jira] [Commented] (HBASE-6580) Deprecate HTablePool in favor of HConnection.getTable(...)
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729128#comment-13729128 ] Hadoop QA commented on HBASE-6580: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595855/6580-trunk-v4.txt against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 9 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 4 warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6604//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6604//console This message is automatically generated. Deprecate HTablePool in favor of HConnection.getTable(...) -- Key: HBASE-6580 URL: https://issues.apache.org/jira/browse/HBASE-6580 Project: HBase Issue Type: Improvement Affects Versions: 0.94.6, 0.95.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: 6580-trunk.txt, 6580-trunk-v2.txt, 6580-trunk-v3.txt, 6580-trunk-v4.txt, HBASE-6580_v1.patch, HBASE-6580_v2.patch Update: I now propose deprecating HTablePool and instead introduce a getTable method on HConnection and allow HConnection to manage the ThreadPool. Initial proposal: Here I propose a very simple TablePool. It could be called LightHTablePool (or something - if you have a better name). Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it. In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used. It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9108) LoadTestTool need to have a way to ignore keys which were failed during write.
[ https://issues.apache.org/jira/browse/HBASE-9108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729300#comment-13729300 ] gautam commented on HBASE-9108: --- The sanity tests have gone fine here. Can we expect this to be checked in into 94.x mainline soon? LoadTestTool need to have a way to ignore keys which were failed during write. --- Key: HBASE-9108 URL: https://issues.apache.org/jira/browse/HBASE-9108 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.95.0, 0.95.1, 0.94.9, 0.94.10 Reporter: gautam Assignee: gautam Priority: Critical Attachments: HBASE-9108.patch._0.94, HBASE-9108.patch._trunk Original Estimate: 48h Remaining Estimate: 48h While running the chaosmonkey integration tests, it is found that write sometimes fails when the cluster components are restarted/stopped/killed etc.. The data key which was being put, using the LoadTestTool, is added to the failed key set, and at the end of the test, this failed key set is checked for any entries to assert failures. While doing fail-over testing, it is expected that some of the keys may go un-written. The point here is to validate that whatever gets into hbase for an unstable cluster really goes in, and hence read should be 100% for whatever keys went in successfully. Currently LoadTestTool has strict checks to validate every key being written or not. In case any keys is not written, it fails. I wanted to loosen this constraint by allowing users to pass in a set of exceptions they expect when doing put/write operations over hbase. If one of these expected exception set is thrown while writing key to hbase, the failed key would be ignored, and hence wont even be considered again for subsequent write as well as read. This can be passed to the load test tool as csv list parameter -allowed_write_exceptions, or it can be passed through hbase-site.xml by writing a value for test.ignore.exceptions.during.write Here is the usage: -allowed_write_exceptions java.io.EOFException,org.apache.hadoop.hbase.NotServingRegionException,org.apache.hadoop.hbase.client.NoServerForRegionException,org.apache.hadoop.hbase.ipc.ServerNotRunningYetException Hence, by doing this the existing integration tests can also make use of this change by passing it as property in hbase-site.xml, as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8778) Region assigments scan table directory making them slow for huge tables
[ https://issues.apache.org/jira/browse/HBASE-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729505#comment-13729505 ] Matteo Bertozzi commented on HBASE-8778: +1, v5 looks good to me Region assigments scan table directory making them slow for huge tables --- Key: HBASE-8778 URL: https://issues.apache.org/jira/browse/HBASE-8778 Project: HBase Issue Type: Improvement Reporter: Dave Latham Assignee: Dave Latham Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: 8778-dirmodtime.txt, HBASE-8778-0.94.5.patch, HBASE-8778-0.94.5-v2.patch, HBASE-8778.patch, HBASE-8778-v2.patch, HBASE-8778-v3.patch, HBASE-8778-v4.patch, HBASE-8778-v5.patch On a table with 130k regions it takes about 3 seconds for a region server to open a region once it has been assigned. Watching the threads for a region server running 0.94.5 that is opening many such regions shows the thread opening the reigon in code like this: {noformat} PRI IPC Server handler 4 on 60020 daemon prio=10 tid=0x2aaac07e9000 nid=0x6566 runnable [0x4c46d000] java.lang.Thread.State: RUNNABLE at java.lang.String.indexOf(String.java:1521) at java.net.URI$Parser.scan(URI.java:2912) at java.net.URI$Parser.parse(URI.java:3004) at java.net.URI.init(URI.java:736) at org.apache.hadoop.fs.Path.initialize(Path.java:145) at org.apache.hadoop.fs.Path.init(Path.java:126) at org.apache.hadoop.fs.Path.init(Path.java:50) at org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:215) at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:252) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:311) at org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:159) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:842) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:867) at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1168) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:269) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:255) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoModtime(FSTableDescriptors.java:368) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:155) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:126) at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2834) at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2807) at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) {noformat} To open the region, the region server first loads the latest HTableDescriptor. Since HBASE-4553 HTableDescriptor's are stored in the file system at /hbase/tableDir/.tableinfo.sequenceNum. The file with the largest sequenceNum is the current descriptor. This is done so that the current descirptor is updated atomically. However, since the filename is not known in advance FSTableDescriptors it has to do a FileSystem.listStatus operation which has to list all files in the directory to find it. The directory also contains all the region directories, so in our case it has to load 130k FileStatus objects. Even using a globStatus matching function still transfers all the objects to the client before performing the pattern matching. Furthermore HDFS uses a default of transferring 1000 directory entries in each RPC call, so it requires 130 roundtrips to the namenode to fetch all the directory entries. Consequently, to reassign all the regions of a table (or a constant fraction thereof) requires time proportional to the square of the number of regions. In our case, if a region server fails with 200 such regions, it takes 10+ minutes for them all to be reassigned, after the zk expiration and log splitting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see:
[jira] [Commented] (HBASE-8409) Security support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729552#comment-13729552 ] Matteo Bertozzi commented on HBASE-8409: Do you plan to add a migration step to the NamespaceUpgrade tool? migrating the _acl_ table to hbase._acl_? How do you handle the row keys migration? just a simple scan, put new values and delete the old ones? Security support for namespaces --- Key: HBASE-8409 URL: https://issues.apache.org/jira/browse/HBASE-8409 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Attachments: HBASE-8049_trunk.patch This task adds the security piece to the namespace feature. The work related to migration of the existing acl table to the new namespace is remaining and will be completed in the follow up patch. Permissions can be granted to a namespace by the hbase admin, by appending '@' to the namespace name. A user with write or admin permissions on a given namespace can create tables in that namespace. The other privileges (R, X, C ) do not have any special meaning w.r.t namespaces. Any users of hbase can list tables in a namespace. The following commands can only be executed by HBase admins. 1. Grant privileges for user on Namespace. 2. Revoke privileges for user on Namespace Grant Command: hbase grant 'tenant-A' 'W' '@N1' In the above example, the command will grant the user 'tenant-A' write privileges for a namespace named N1. Revoke Command: hbase revoke 'tenant-A''@N1' In the above example, the command will revoke all privileges from user 'tenant-A' for namespace named N1. Lets see an example on how privileges work with namespaces. User Mike request for a namespace named hbase_perf with the hbase admin. whoami: hbase hbase shell namespace_create 'hbase_perf' hbase shell grant 'mike', 'W', '@hbase_perf' Mike creates two tables table20 and table50 in the above workspace. whoami: mike hbase shell create 'hbase_perf.table20', 'family1' hbase shell create 'hbase_perf.table50', 'family1' Note: As Mike was able to create tables 'hbase_perf.table20', 'hbase_perf.table50', he becomes the owner of those tables. This means he has RWXCA perms on those tables. Another team member of Mike, Alice wants also to share the same workspace hbase_perf. HBase admin grants Alice also permission to create tables in hbase_perf namespace. whoami: hbase hbase shell grant 'alice', 'W', '@hbase_perf' Now Alice can create new tables under hbase_perf namespace, but cannot read,write,alter,delete existing tables in the namespace. whoami: alice hbase shell namespace_list_tables 'hbase_perf' hbase_perf.table20 hbase_perf.table50 hbase shell scan 'hbase_perf.table20' AccessDeniedException If Alice wants to read or write to existing tables in the hbase_perf namespace, hbase admins need to explicitly grant permission. whoami: hbase hbase shell grant 'alice', 'RW', 'hbase_perf.table20' hbase shell grant 'alice', 'RW', 'hbase_perf.table50' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8760) possible loss of data in snapshot taken after region split
[ https://issues.apache.org/jira/browse/HBASE-8760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729572#comment-13729572 ] Ted Yu commented on HBASE-8760: --- {code} + LOG.warn(msg); + //throw new CorruptedSnapshotException(msg, snapshot); {code} Remove the commented out code ? possible loss of data in snapshot taken after region split -- Key: HBASE-8760 URL: https://issues.apache.org/jira/browse/HBASE-8760 Project: HBase Issue Type: Bug Components: snapshots Affects Versions: 0.94.8, 0.95.1 Reporter: Jerry He Fix For: 0.98.0, 0.95.2, 0.94.12 Attachments: HBase-8760-0.94.8.patch, HBase-8760-0.94.8-v1.patch, HBASE-8760-thz-v0.patch, HBASE-8760-thz-v1.patch Right after a region split but before the daughter regions are compacted, we have two daughter regions containing Reference files to the parent hfiles. If we take snapshot right at the moment, the snapshot will succeed, but it will only contain the daughter Reference files. Since there is no hold on the parent hfiles, they will be deleted by the HFile Cleaner after they are no longer needed by the daughter regions soon after. A minimum we need to do is the keep these parent hfiles from being deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9108) LoadTestTool need to have a way to ignore keys which were failed during write.
[ https://issues.apache.org/jira/browse/HBASE-9108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729646#comment-13729646 ] Ted Yu commented on HBASE-9108: --- {code} +LOG.error(Class: [ + className + ] in not an instance of Exception.); {code} typo: 'in not' {code} + public boolean ignoredKey(long k) { {code} The above can be package private. {code} + protected static final String OPT_ALLOWED_WRITE_EXCEPTIONS = allowed_write_exceptions; {code} Can we use a shorter string for option ? LoadTestTool need to have a way to ignore keys which were failed during write. --- Key: HBASE-9108 URL: https://issues.apache.org/jira/browse/HBASE-9108 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.95.0, 0.95.1, 0.94.9, 0.94.10 Reporter: gautam Assignee: gautam Priority: Critical Attachments: HBASE-9108.patch._0.94, HBASE-9108.patch._trunk Original Estimate: 48h Remaining Estimate: 48h While running the chaosmonkey integration tests, it is found that write sometimes fails when the cluster components are restarted/stopped/killed etc.. The data key which was being put, using the LoadTestTool, is added to the failed key set, and at the end of the test, this failed key set is checked for any entries to assert failures. While doing fail-over testing, it is expected that some of the keys may go un-written. The point here is to validate that whatever gets into hbase for an unstable cluster really goes in, and hence read should be 100% for whatever keys went in successfully. Currently LoadTestTool has strict checks to validate every key being written or not. In case any keys is not written, it fails. I wanted to loosen this constraint by allowing users to pass in a set of exceptions they expect when doing put/write operations over hbase. If one of these expected exception set is thrown while writing key to hbase, the failed key would be ignored, and hence wont even be considered again for subsequent write as well as read. This can be passed to the load test tool as csv list parameter -allowed_write_exceptions, or it can be passed through hbase-site.xml by writing a value for test.ignore.exceptions.during.write Here is the usage: -allowed_write_exceptions java.io.EOFException,org.apache.hadoop.hbase.NotServingRegionException,org.apache.hadoop.hbase.client.NoServerForRegionException,org.apache.hadoop.hbase.ipc.ServerNotRunningYetException Hence, by doing this the existing integration tests can also make use of this change by passing it as property in hbase-site.xml, as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729655#comment-13729655 ] Nick Dimiduk commented on HBASE-9091: - [~stack] Since we're not running on flintstones computers, my preference is to use a single class that wraps both a range in a byte[] and a position marker, a la {{ByteBuffer}}. [~mcorgan] Having ready through Guava's {{Precondition}} code, I'm not as concerned about the performance impact of the additional checks. I don't have a microbenchmark to quantify the lack of concern, but the code describes the pains taken to shape the condition checking statements for the benefit of the JIT, at least not in the face of giving helpful feedback in a user-facing API. [~mcorgan] Thanks for outlining your two use-cases above, that's helpful. I think my intended use-case is an extension of #2, where position tracking is the added feature. I can't think of a case where a user would want to build a compound rowkey across multiple threads. I'll assemble a new patch that leaves {{ByteRange}} intact, extending it with a subclass that adds the position tracking. I will omit the precondition checking for the time being, so long as we all understand that (1) omission opens the user up to easily detected bugs and (2) adding it later will impact a (admittedly brand new, {{Evolving}}) public API. I will not update the other two patches until we've found something here that's agreeable. I'd rather ship 8201 and 8693 as they are than miss the window because of this bit of API sugar. Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9099) logReplay could trigger double region assignment
[ https://issues.apache.org/jira/browse/HBASE-9099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-9099: - Resolution: Fixed Fix Version/s: 0.95.2 0.98.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) logReplay could trigger double region assignment Key: HBASE-9099 URL: https://issues.apache.org/jira/browse/HBASE-9099 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.2 Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 0.98.0, 0.95.2 Attachments: hbase-9099.patch, hbase-9099-v1.patch The symptom is the first region assignment submitted in SSH is in progress while when am.waitOnRegionToClearRegionsInTransition times out we will re-submitted another SSH which will invoke another region assignment for the region. It will cause the region get stuck in RIT status. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9099) logReplay could trigger double region assignment
[ https://issues.apache.org/jira/browse/HBASE-9099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729686#comment-13729686 ] Jeffrey Zhong commented on HBASE-9099: -- Integrated the v1 patch into 0.95 and trunk branch. Thanks! logReplay could trigger double region assignment Key: HBASE-9099 URL: https://issues.apache.org/jira/browse/HBASE-9099 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.2 Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Attachments: hbase-9099.patch, hbase-9099-v1.patch The symptom is the first region assignment submitted in SSH is in progress while when am.waitOnRegionToClearRegionsInTransition times out we will re-submitted another SSH which will invoke another region assignment for the region. It will cause the region get stuck in RIT status. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) Deprecate HTablePool in favor of HConnection.getTable(...)
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729695#comment-13729695 ] Lars Hofhansl commented on HBASE-6580: -- TestZKInterProcessReadWriteLock passes locally. Deprecate HTablePool in favor of HConnection.getTable(...) -- Key: HBASE-6580 URL: https://issues.apache.org/jira/browse/HBASE-6580 Project: HBase Issue Type: Improvement Affects Versions: 0.94.6, 0.95.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: 6580-trunk.txt, 6580-trunk-v2.txt, 6580-trunk-v3.txt, 6580-trunk-v4.txt, HBASE-6580_v1.patch, HBASE-6580_v2.patch Update: I now propose deprecating HTablePool and instead introduce a getTable method on HConnection and allow HConnection to manage the ThreadPool. Initial proposal: Here I propose a very simple TablePool. It could be called LightHTablePool (or something - if you have a better name). Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it. In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used. It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8960) TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes
[ https://issues.apache.org/jira/browse/HBASE-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729699#comment-13729699 ] Jeffrey Zhong commented on HBASE-8960: -- [~saint@gmail.com] A good suggestion. I'm thinking to extend the tool to include stats for skipped ones as well so we can cover the test suite timeout situations. The end result is printing stats of failed and skipped tests in last N runs instead of just failed ones. TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes -- Key: HBASE-8960 URL: https://issues.apache.org/jira/browse/HBASE-8960 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jeffrey Zhong Priority: Minor Fix For: 0.95.2 Attachments: hbase-8960-addendum-2.patch, hbase-8960-addendum.patch, hbase-8960.patch http://54.241.6.143/job/HBase-0.95-Hadoop-2/org.apache.hbase$hbase-server/634/testReport/junit/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testLogReplayForDisablingTable/ {noformat} java.lang.AssertionError: expected:1000 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testLogReplayForDisablingTable(TestDistributedLogSplitting.java:797) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8322) Re-enable hbase checksums by default
[ https://issues.apache.org/jira/browse/HBASE-8322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729703#comment-13729703 ] Jean-Daniel Cryans commented on HBASE-8322: --- [~enis] I was wondering, why did you pick CRC32C? It's not available in Hadoop 1 and it's what's failing. CRC32 is fine though. Re-enable hbase checksums by default Key: HBASE-8322 URL: https://issues.apache.org/jira/browse/HBASE-8322 Project: HBase Issue Type: Improvement Components: Filesystem Integration Reporter: Enis Soztutar Assignee: Jean-Daniel Cryans Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: hbase-8322_v1.patch, HBASE-8322-v2.patch Double checksumming issues in HBase level checksums(HBASE-5074) has been fixed in HBASE-6868. However, that patch also disables hbase checksums by default. I think we should re-enable it by default, and document the interaction with shortcircuit reads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication
[ https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729707#comment-13729707 ] Jean-Daniel Cryans commented on HBASE-7709: --- Ah ok that's a fancy setup you got there. Sounds ok to me. Infinite loop possible in Master/Master replication --- Key: HBASE-7709 URL: https://issues.apache.org/jira/browse/HBASE-7709 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.94.6, 0.95.1 Reporter: Lars Hofhansl Fix For: 0.98.0, 0.95.2, 0.94.12 We just discovered the following scenario: # Cluster A and B are setup in master/master replication # By accident we had Cluster C replicate to Cluster A. Now all edit originating from C will be bouncing between A and B. Forever! The reason is that when the edit come in from C the cluster ID is already set and won't be reset. We have a couple of options here: # Optionally only support master/master (not cycles of more than two clusters). In that case we can always reset the cluster ID in the ReplicationSource. That means that now cycles 2 will have the data cycle forever. This is the only option that requires no changes in the HLog format. # Instead of a single cluster id per edit maintain a (unordered) set of cluster id that have seen this edit. Then in ReplicationSource we drop any edit that the sink has seen already. The is the cleanest approach, but it might need a lot of data stored per edit if there are many clusters involved. # Maintain a configurable counter of the maximum cycle side we want to support. Could default to 10 (even maybe even just). Store a hop-count in the WAL and the ReplicationSource increases that hop-count on each hop. If we're over the max, just drop the edit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8939) Hanging unit tests
[ https://issues.apache.org/jira/browse/HBASE-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729706#comment-13729706 ] stack commented on HBASE-8939: -- Failed again: http://54.241.6.143/job/HBase-0.95-Hadoop-2/759/org.apache.hbase$hbase-server/ Hanging unit tests -- Key: HBASE-8939 URL: https://issues.apache.org/jira/browse/HBASE-8939 Project: HBase Issue Type: Bug Components: test Reporter: stack Fix For: 0.95.2 Attachments: 8939.txt We have hanging tests. Here's a few from this morning's review: {code} durruti:0.95 stack$ ./dev-support/findHangingTest.sh https://builds.apache.org/job/hbase-0.95-on-hadoop2/176/consoleText % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 3300k0 3300k0 0 508k 0 --:--:-- 0:00:06 --:--:-- 621k Hanging test: Running org.apache.hadoop.hbase.TestIOFencing Hanging test: Running org.apache.hadoop.hbase.regionserver.wal.TestLogRolling {code} And... {code} durruti:0.95 stack$ ./dev-support/findHangingTest.sh http://54.241.6.143/job/HBase-TRUNK-Hadoop-2/396/consoleText % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 779k0 779k0 0 538k 0 --:--:-- 0:00:01 --:--:-- 559k Hanging test: Running org.apache.hadoop.hbase.TestIOFencing Hanging test: Running org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort Hanging test: Running org.apache.hadoop.hbase.client.TestFromClientSide3 {code} and {code} durruti:0.95 stack$ ./dev-support/findHangingTest.sh http://54.241.6.143/job/HBase-0.95/607/consoleText % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 445k0 445k0 0 490k 0 --:--:-- --:--:-- --:--:-- 522k Hanging test: Running org.apache.hadoop.hbase.replication.TestReplicationDisableInactivePeer Hanging test: Running org.apache.hadoop.hbase.master.TestAssignmentManager Hanging test: Running org.apache.hadoop.hbase.util.TestHBaseFsck Hanging test: Running org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary Hanging test: Running org.apache.hadoop.hbase.IntegrationTestDataIngestSlowDeterministic {code} and... {code} durruti:0.95 stack$ ./dev-support/findHangingTest.sh http://54.241.6.143/job/HBase-0.95-Hadoop-2/607/consoleText % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 781k0 781k0 0 240k 0 --:--:-- 0:00:03 --:--:-- 244k Hanging test: Running org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint Hanging test: Running org.apache.hadoop.hbase.client.TestFromClientSide Hanging test: Running org.apache.hadoop.hbase.TestIOFencing Hanging test: Running org.apache.hadoop.hbase.master.TestMasterFailoverBalancerPersistence Hanging test: Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9072) TestSnapshotFromMaster.testSnapshotHFileArchiving fails
[ https://issues.apache.org/jira/browse/HBASE-9072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729709#comment-13729709 ] stack commented on HBASE-9072: -- Failed here this morning: http://54.241.6.143/job/HBase-0.95/org.apache.hbase$hbase-server/753/testReport/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/ TestSnapshotFromMaster.testSnapshotHFileArchiving fails --- Key: HBASE-9072 URL: https://issues.apache.org/jira/browse/HBASE-9072 Project: HBase Issue Type: Bug Components: test Reporter: stack http://54.241.6.143/job/HBase-TRUNK-Hadoop-2/org.apache.hbase$hbase-server/428/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/ {code} Failed 4600 actions: SocketTimeoutException: 4600 times, Stacktrace at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:794) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:756) at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1312) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297) org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 4600 actions: SocketTimeoutException: 4600 times, at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:157) at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:145) at org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:700) at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:827) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:794) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:756) at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1312) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runners.Suite.runChild(Suite.java:127) at org.junit.runners.Suite.runChild(Suite.java:26) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) {code} [~mbertozzi] Opinion? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7980) TestZKInterProcessReadWriteLock fails occasionally in QA test run
[ https://issues.apache.org/jira/browse/HBASE-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729711#comment-13729711 ] stack commented on HBASE-7980: -- Failed here: https://builds.apache.org/job/PreCommit-HBASE-Build/6604//testReport/ Anyone want to check these out? TestZKInterProcessReadWriteLock fails occasionally in QA test run - Key: HBASE-7980 URL: https://issues.apache.org/jira/browse/HBASE-7980 Project: HBase Issue Type: Bug Components: test Reporter: Ted Yu Fix For: 0.95.2 {code} testReadLockExcludesWriters(org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock) Time elapsed: 0.003 sec ERROR! java.lang.Exception: test timed out after 3000 milliseconds at sun.misc.Unsafe.park(Native Method) {code} You can find the test output here: https://builds.apache.org/job/PreCommit-HBASE-Build/4634/artifact/trunk/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock-output.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) Deprecate HTablePool in favor of HConnection.getTable(...)
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729718#comment-13729718 ] stack commented on HBASE-6580: -- HBASE-7980 TestZKInterProcessReadWriteLock is known flakey (I should just disable it since no one is looking at it). Patch lgtm on skim. Hurray HTablePool is deprecated! Deprecate HTablePool in favor of HConnection.getTable(...) -- Key: HBASE-6580 URL: https://issues.apache.org/jira/browse/HBASE-6580 Project: HBase Issue Type: Improvement Affects Versions: 0.94.6, 0.95.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: 6580-trunk.txt, 6580-trunk-v2.txt, 6580-trunk-v3.txt, 6580-trunk-v4.txt, HBASE-6580_v1.patch, HBASE-6580_v2.patch Update: I now propose deprecating HTablePool and instead introduce a getTable method on HConnection and allow HConnection to manage the ThreadPool. Initial proposal: Here I propose a very simple TablePool. It could be called LightHTablePool (or something - if you have a better name). Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it. In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used. It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9095) AssignmentManager's handleRegion should respect the single threaded nature of the processing
[ https://issues.apache.org/jira/browse/HBASE-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-9095: --- Attachment: 9095-2.txt I had to do some more changes to have the tests work (TestMaster and TestTestZKBasedOpenCloseRegion). I was trying to make the tests work with Mockito/spy but it didn't work and at some point I gave up. This approach is not as elegant but works. AssignmentManager's handleRegion should respect the single threaded nature of the processing Key: HBASE-9095 URL: https://issues.apache.org/jira/browse/HBASE-9095 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.95.2 Attachments: 9095-1 (1).txt, 9095-1.txt, 9095-1.txt, 9095-1.txt, 9095-2.txt While debugging a case where a region was getting opened on a RegionServer and then closed soon after (and then never re-opened anywhere thereafter), it seemed like the processing in handleRegion to do with deletion of ZK nodes should be non-asynchronous. This achieves two things: 1. The synchronous deletion prevents more than one processing on the same event data twice. Assuming that we do get more than one notification (on let's say, region OPENED event), the subsequent processing(s) in handleRegion for the same znode would end up with a zookeeper node not found exception. The return value of the data read would be null and that's already handled. If it is asynchronous, it leads to issues like - master opens a region on a certain RegionServer and soon after it sends that RegionServer a close for the same region, and then the znode is deleted. 2. The deletion is currently handled in an executor service. This is problematic since by design the events for a given region should be processed in order. By delegating a part of the processing to executor service we are somewhat violating this contract since there is no guarantee of the ordering in the executor service executions... Thanks to [~jeffreyz] and [~enis] for the discussions on this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9079) FilterList getNextKeyHint skips rows that should be included in the results
[ https://issues.apache.org/jira/browse/HBASE-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729738#comment-13729738 ] Viral Bajaria commented on HBASE-9079: -- [~lhofhansl] Can you review the patch when you get a chance ? I have already deployed this to my production cluster and have not had any issues. FilterList getNextKeyHint skips rows that should be included in the results --- Key: HBASE-9079 URL: https://issues.apache.org/jira/browse/HBASE-9079 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.94.10 Reporter: Viral Bajaria Attachments: HBASE-9079-0.94.patch, HBASE-9079-trunk.patch I hit a weird issue/bug and am able to reproduce the error consistently. The problem arises when FilterList has two filters where each implements the getNextKeyHint method. The way the current implementation works is, StoreScanner will call matcher.getNextKeyHint() whenever it gets a SEEK_NEXT_USING_HINT. This in turn will call filter.getNextKeyHint() which at this stage is of type FilterList. The implementation in FilterList iterates through all the filters and keeps the max KeyValue that it sees. All is fine if you wrap filters in FilterList in which only one of them implements getNextKeyHint. but if multiple of them implement then that's where things get weird. For example: - create two filters: one is FuzzyRowFilter and second is ColumnRangeFilter. Both of them implement getNextKeyHint - wrap them in FilterList with MUST_PASS_ALL - FuzzyRowFilter will seek to the correct first row and then pass it to ColumnRangeFilter which will return the SEEK_NEXT_USING_HINT code. - Now in FilterList when getNextKeyHint is called, it calls the one on FuzzyRow first which basically says what the next row should be. While in reality we want the ColumnRangeFilter to give the seek hint. - The above behavior skips data that should be returned, which I have verified by using a RowFilter with RegexStringComparator. I updated the FilterList to maintain state on which filter returns the SEEK_NEXT_USING_HINT and in getNextKeyHint, I invoke the method on the saved filter and reset that state. I tested it with my current queries and it works fine but I need to run the entire test suite to make sure I have not introduced any regression. In addition to that I need to figure out what should be the behavior when the opeation is MUST_PASS_ONE, but I doubt it should be any different. Is my understanding of it being a bug correct ? Or am I trivializing it and ignoring something very important ? If it's tough to wrap your head around the explanation, then I can open a JIRA and upload a patch against 0.94 head. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8409) Security support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729736#comment-13729736 ] stack commented on HBASE-8409: -- Can the new acl table be called hbase.acl instead of hbase._acl_? bq. The work related to migration of the existing acl table to the new namespace is remaining and will be completed in the follow up patch. As per our Matteo, how is the above going? bq. Permissions can be granted to a namespace by the hbase admin, by appending '@' to the namespace name. What is the '@' about? Seems awkward? Is this because our current grant didn't include ns: {code} grant user permissions [ table [ column family [ column qualifier ] ] ] {code} ... the '@' is how we get it in there? Can we do better? Thanks. Security support for namespaces --- Key: HBASE-8409 URL: https://issues.apache.org/jira/browse/HBASE-8409 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Attachments: HBASE-8049_trunk.patch This task adds the security piece to the namespace feature. The work related to migration of the existing acl table to the new namespace is remaining and will be completed in the follow up patch. Permissions can be granted to a namespace by the hbase admin, by appending '@' to the namespace name. A user with write or admin permissions on a given namespace can create tables in that namespace. The other privileges (R, X, C ) do not have any special meaning w.r.t namespaces. Any users of hbase can list tables in a namespace. The following commands can only be executed by HBase admins. 1. Grant privileges for user on Namespace. 2. Revoke privileges for user on Namespace Grant Command: hbase grant 'tenant-A' 'W' '@N1' In the above example, the command will grant the user 'tenant-A' write privileges for a namespace named N1. Revoke Command: hbase revoke 'tenant-A''@N1' In the above example, the command will revoke all privileges from user 'tenant-A' for namespace named N1. Lets see an example on how privileges work with namespaces. User Mike request for a namespace named hbase_perf with the hbase admin. whoami: hbase hbase shell namespace_create 'hbase_perf' hbase shell grant 'mike', 'W', '@hbase_perf' Mike creates two tables table20 and table50 in the above workspace. whoami: mike hbase shell create 'hbase_perf.table20', 'family1' hbase shell create 'hbase_perf.table50', 'family1' Note: As Mike was able to create tables 'hbase_perf.table20', 'hbase_perf.table50', he becomes the owner of those tables. This means he has RWXCA perms on those tables. Another team member of Mike, Alice wants also to share the same workspace hbase_perf. HBase admin grants Alice also permission to create tables in hbase_perf namespace. whoami: hbase hbase shell grant 'alice', 'W', '@hbase_perf' Now Alice can create new tables under hbase_perf namespace, but cannot read,write,alter,delete existing tables in the namespace. whoami: alice hbase shell namespace_list_tables 'hbase_perf' hbase_perf.table20 hbase_perf.table50 hbase shell scan 'hbase_perf.table20' AccessDeniedException If Alice wants to read or write to existing tables in the hbase_perf namespace, hbase admins need to explicitly grant permission. whoami: hbase hbase shell grant 'alice', 'RW', 'hbase_perf.table20' hbase shell grant 'alice', 'RW', 'hbase_perf.table50' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8409) Security support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8409: - Priority: Critical (was: Major) ns can't go in w/o this fixed (migration and '@' thingy) Security support for namespaces --- Key: HBASE-8409 URL: https://issues.apache.org/jira/browse/HBASE-8409 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Priority: Critical Attachments: HBASE-8049_trunk.patch This task adds the security piece to the namespace feature. The work related to migration of the existing acl table to the new namespace is remaining and will be completed in the follow up patch. Permissions can be granted to a namespace by the hbase admin, by appending '@' to the namespace name. A user with write or admin permissions on a given namespace can create tables in that namespace. The other privileges (R, X, C ) do not have any special meaning w.r.t namespaces. Any users of hbase can list tables in a namespace. The following commands can only be executed by HBase admins. 1. Grant privileges for user on Namespace. 2. Revoke privileges for user on Namespace Grant Command: hbase grant 'tenant-A' 'W' '@N1' In the above example, the command will grant the user 'tenant-A' write privileges for a namespace named N1. Revoke Command: hbase revoke 'tenant-A''@N1' In the above example, the command will revoke all privileges from user 'tenant-A' for namespace named N1. Lets see an example on how privileges work with namespaces. User Mike request for a namespace named hbase_perf with the hbase admin. whoami: hbase hbase shell namespace_create 'hbase_perf' hbase shell grant 'mike', 'W', '@hbase_perf' Mike creates two tables table20 and table50 in the above workspace. whoami: mike hbase shell create 'hbase_perf.table20', 'family1' hbase shell create 'hbase_perf.table50', 'family1' Note: As Mike was able to create tables 'hbase_perf.table20', 'hbase_perf.table50', he becomes the owner of those tables. This means he has RWXCA perms on those tables. Another team member of Mike, Alice wants also to share the same workspace hbase_perf. HBase admin grants Alice also permission to create tables in hbase_perf namespace. whoami: hbase hbase shell grant 'alice', 'W', '@hbase_perf' Now Alice can create new tables under hbase_perf namespace, but cannot read,write,alter,delete existing tables in the namespace. whoami: alice hbase shell namespace_list_tables 'hbase_perf' hbase_perf.table20 hbase_perf.table50 hbase shell scan 'hbase_perf.table20' AccessDeniedException If Alice wants to read or write to existing tables in the hbase_perf namespace, hbase admins need to explicitly grant permission. whoami: hbase hbase shell grant 'alice', 'RW', 'hbase_perf.table20' hbase shell grant 'alice', 'RW', 'hbase_perf.table50' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9095) AssignmentManager's handleRegion should respect the single threaded nature of the processing
[ https://issues.apache.org/jira/browse/HBASE-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729739#comment-13729739 ] Devaraj Das commented on HBASE-9095: bq. Since these handlers no longer run in executors, are there executors we start in the RS that are now unused? Not sure on this. The executor service is used during the master recovery (look at AM.processRegionsInTransition) and I am not sure I want to touch that part in this patch... AssignmentManager's handleRegion should respect the single threaded nature of the processing Key: HBASE-9095 URL: https://issues.apache.org/jira/browse/HBASE-9095 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.95.2 Attachments: 9095-1 (1).txt, 9095-1.txt, 9095-1.txt, 9095-1.txt, 9095-2.txt While debugging a case where a region was getting opened on a RegionServer and then closed soon after (and then never re-opened anywhere thereafter), it seemed like the processing in handleRegion to do with deletion of ZK nodes should be non-asynchronous. This achieves two things: 1. The synchronous deletion prevents more than one processing on the same event data twice. Assuming that we do get more than one notification (on let's say, region OPENED event), the subsequent processing(s) in handleRegion for the same znode would end up with a zookeeper node not found exception. The return value of the data read would be null and that's already handled. If it is asynchronous, it leads to issues like - master opens a region on a certain RegionServer and soon after it sends that RegionServer a close for the same region, and then the znode is deleted. 2. The deletion is currently handled in an executor service. This is problematic since by design the events for a given region should be processed in order. By delegating a part of the processing to executor service we are somewhat violating this contract since there is no guarantee of the ordering in the executor service executions... Thanks to [~jeffreyz] and [~enis] for the discussions on this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7525) A canary monitoring program specifically for regionserver
[ https://issues.apache.org/jira/browse/HBASE-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729743#comment-13729743 ] stack commented on HBASE-7525: -- [~takeshi.miao] Pardon us for overlooking this addition. Please rebase (I think the scan instead of get for the first key in table has been addressed) and lets get it in. A canary monitoring program specifically for regionserver - Key: HBASE-7525 URL: https://issues.apache.org/jira/browse/HBASE-7525 Project: HBase Issue Type: New Feature Components: monitoring Affects Versions: 0.94.0 Reporter: takeshi.miao Priority: Minor Fix For: 0.95.0 Attachments: HBASE-7525-0.95-v0.patch, HBASE-7525-0.95-v1.patch, HBASE-7525-0.95-v3.patch, HBASE-7525-v0.patch, RegionServerCanary.java *Motivation* This ticket is to provide a canary monitoring tool specifically for HRegionserver, details as follows 1. This tool is required by operation team due to they thought that the canary for each region of a HBase is too many for them, so I implemented this coarse-granular one based on the original o.a.h.h.tool.Canary for them 2. And this tool is implemented by multi-threading, which means the each Get request sent by a thread. the reason I use this way is due to we suffered the region server hung issue by now the root cause is still not clear. so this tool can help operation team to detect hung region server if any. *example* 1. the tool docs ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -help Usage: [opts] [regionServerName 1 [regionServrName 2...]] regionServerName - FQDN serverName, can use linux command:hostname -f to check your serverName where [-opts] are: -help Show this help and exit. -eUse regionServerName as regular expression which means the regionServerName is regular expression pattern -f B stop whole program if first error occurs, default is true -t N timeout for a check, default is 60 (milisecs) -daemonContinuous check at defined intervals. -interval N Interval between checks (sec) 2. Will send a request to each regionserver in a HBase cluster ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary 3. Will send a request to a regionserver by given name ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary rs1.domainname 4. Will send a request to regionserver(s) by given regular-expression /opt/trend/circus-opstool/bin/hbase-canary-monitor-each-regionserver.sh -e rs1.domainname.pattern // another example ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -e tw-poc-tm-puppet-hdn[0-9]\{1,2\}.client.tw.trendnet.org 5. Will send a request to a regionserver and also set a timeout limit for this test // query regionserver:rs1.domainname with timeout limit 10sec // -f false, means that will not exit this program even test failed ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -f false -t 1 rs1.domainname // echo 1 if timeout echo $? 6. Will run as daemon mode, which means it will send request to each regionserver periodically ./bin/hbase org.apache.hadoop.hbase.tool.RegionServerCanary -daemon -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9095) AssignmentManager's handleRegion should respect the single threaded nature of the processing
[ https://issues.apache.org/jira/browse/HBASE-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729744#comment-13729744 ] Jimmy Xiang commented on HBASE-9095: Currently, the executor service is still needed. We can get rid of it later on when we remove the timeout monitor. AssignmentManager's handleRegion should respect the single threaded nature of the processing Key: HBASE-9095 URL: https://issues.apache.org/jira/browse/HBASE-9095 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.95.2 Attachments: 9095-1 (1).txt, 9095-1.txt, 9095-1.txt, 9095-1.txt, 9095-2.txt While debugging a case where a region was getting opened on a RegionServer and then closed soon after (and then never re-opened anywhere thereafter), it seemed like the processing in handleRegion to do with deletion of ZK nodes should be non-asynchronous. This achieves two things: 1. The synchronous deletion prevents more than one processing on the same event data twice. Assuming that we do get more than one notification (on let's say, region OPENED event), the subsequent processing(s) in handleRegion for the same znode would end up with a zookeeper node not found exception. The return value of the data read would be null and that's already handled. If it is asynchronous, it leads to issues like - master opens a region on a certain RegionServer and soon after it sends that RegionServer a close for the same region, and then the znode is deleted. 2. The deletion is currently handled in an executor service. This is problematic since by design the events for a given region should be processed in order. By delegating a part of the processing to executor service we are somewhat violating this contract since there is no guarantee of the ordering in the executor service executions... Thanks to [~jeffreyz] and [~enis] for the discussions on this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9091: Attachment: 0001-HBASE-9091-Introduce-PositionedByteRange.patch Adds the new subclass described previously. Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9072) TestSnapshotFromMaster.testSnapshotHFileArchiving fails
[ https://issues.apache.org/jira/browse/HBASE-9072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-9072: --- Assignee: Matteo Bertozzi Status: Patch Available (was: Open) TestSnapshotFromMaster.testSnapshotHFileArchiving fails --- Key: HBASE-9072 URL: https://issues.apache.org/jira/browse/HBASE-9072 Project: HBase Issue Type: Bug Components: test Reporter: stack Assignee: Matteo Bertozzi Attachments: HBASE-9072-fix0.patch http://54.241.6.143/job/HBase-TRUNK-Hadoop-2/org.apache.hbase$hbase-server/428/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/ {code} Failed 4600 actions: SocketTimeoutException: 4600 times, Stacktrace at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:794) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:756) at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1312) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297) org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 4600 actions: SocketTimeoutException: 4600 times, at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:157) at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:145) at org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:700) at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:827) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:794) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:756) at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1312) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runners.Suite.runChild(Suite.java:127) at org.junit.runners.Suite.runChild(Suite.java:26) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) {code} [~mbertozzi] Opinion? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9072) TestSnapshotFromMaster.testSnapshotHFileArchiving fails
[ https://issues.apache.org/jira/browse/HBASE-9072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-9072: --- Attachment: HBASE-9072-fix0.patch This is a different error, from the one above. The test looks flaky by design, it relies on a strange voodoo of the data inserted is always 4 files... attached a -fix0 patch, that ensures the 4 files flush/compaction... but now it relies on the compaction to compact all the store files... so as soon we change the compaction algorithm this test is flaky again. TestSnapshotFromMaster.testSnapshotHFileArchiving fails --- Key: HBASE-9072 URL: https://issues.apache.org/jira/browse/HBASE-9072 Project: HBase Issue Type: Bug Components: test Reporter: stack Attachments: HBASE-9072-fix0.patch http://54.241.6.143/job/HBase-TRUNK-Hadoop-2/org.apache.hbase$hbase-server/428/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/ {code} Failed 4600 actions: SocketTimeoutException: 4600 times, Stacktrace at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:794) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:756) at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1312) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297) org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 4600 actions: SocketTimeoutException: 4600 times, at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:157) at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:145) at org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:700) at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:827) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:794) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:756) at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1312) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runners.Suite.runChild(Suite.java:127) at org.junit.runners.Suite.runChild(Suite.java:26) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) {code} [~mbertozzi] Opinion? -- This message is automatically
[jira] [Commented] (HBASE-8960) TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes
[ https://issues.apache.org/jira/browse/HBASE-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729772#comment-13729772 ] Jeffrey Zhong commented on HBASE-8960: -- There is one more time out situation happened on http://54.241.6.143/job/HBase-0.95-Hadoop-2/759/. Let me split TestDistributedLogSplitting into two test suites one with replay on and the rest with replay off to see if we can make things better and possibly get some error messages. TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes -- Key: HBASE-8960 URL: https://issues.apache.org/jira/browse/HBASE-8960 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jeffrey Zhong Priority: Minor Fix For: 0.95.2 Attachments: hbase-8960-addendum-2.patch, hbase-8960-addendum.patch, hbase-8960.patch http://54.241.6.143/job/HBase-0.95-Hadoop-2/org.apache.hbase$hbase-server/634/testReport/junit/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testLogReplayForDisablingTable/ {noformat} java.lang.AssertionError: expected:1000 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testLogReplayForDisablingTable(TestDistributedLogSplitting.java:797) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9072) TestSnapshotFromMaster.testSnapshotHFileArchiving fails
[ https://issues.apache.org/jira/browse/HBASE-9072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729803#comment-13729803 ] stack commented on HBASE-9072: -- [~mbertozzi] Thanks boss for taking a look. I'd rather just disable a crappy test. TestSnapshotFromMaster.testSnapshotHFileArchiving fails --- Key: HBASE-9072 URL: https://issues.apache.org/jira/browse/HBASE-9072 Project: HBase Issue Type: Bug Components: test Reporter: stack Assignee: Matteo Bertozzi Attachments: HBASE-9072-fix0.patch http://54.241.6.143/job/HBase-TRUNK-Hadoop-2/org.apache.hbase$hbase-server/428/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/ {code} Failed 4600 actions: SocketTimeoutException: 4600 times, Stacktrace at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:794) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:756) at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1312) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297) org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 4600 actions: SocketTimeoutException: 4600 times, at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:157) at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:145) at org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:700) at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:827) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:794) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:756) at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1312) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runners.Suite.runChild(Suite.java:127) at org.junit.runners.Suite.runChild(Suite.java:26) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) {code} [~mbertozzi] Opinion? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9126) Make HFile MIN VERSION as 2
Anoop Sam John created HBASE-9126: - Summary: Make HFile MIN VERSION as 2 Key: HBASE-9126 URL: https://issues.apache.org/jira/browse/HBASE-9126 Project: HBase Issue Type: Improvement Affects Versions: 0.95.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Removed the HFile V1 support from version95. We can make the min version supported as 2? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8960) TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes
[ https://issues.apache.org/jira/browse/HBASE-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729809#comment-13729809 ] stack commented on HBASE-8960: -- [~jeffreyz] Thanks boss. TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes -- Key: HBASE-8960 URL: https://issues.apache.org/jira/browse/HBASE-8960 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jeffrey Zhong Priority: Minor Fix For: 0.95.2 Attachments: hbase-8960-addendum-2.patch, hbase-8960-addendum.patch, hbase-8960.patch http://54.241.6.143/job/HBase-0.95-Hadoop-2/org.apache.hbase$hbase-server/634/testReport/junit/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testLogReplayForDisablingTable/ {noformat} java.lang.AssertionError: expected:1000 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testLogReplayForDisablingTable(TestDistributedLogSplitting.java:797) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9126) Make HFile MIN VERSION as 2
[ https://issues.apache.org/jira/browse/HBASE-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9126: -- Status: Patch Available (was: Open) Make HFile MIN VERSION as 2 --- Key: HBASE-9126 URL: https://issues.apache.org/jira/browse/HBASE-9126 Project: HBase Issue Type: Improvement Affects Versions: 0.95.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9126.patch Removed the HFile V1 support from version95. We can make the min version supported as 2? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9126) Make HFile MIN VERSION as 2
[ https://issues.apache.org/jira/browse/HBASE-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9126: -- Attachment: HBASE-9126.patch Mostly code removal Make HFile MIN VERSION as 2 --- Key: HBASE-9126 URL: https://issues.apache.org/jira/browse/HBASE-9126 Project: HBase Issue Type: Improvement Affects Versions: 0.95.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9126.patch Removed the HFile V1 support from version95. We can make the min version supported as 2? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) Deprecate HTablePool in favor of HConnection.getTable(...)
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6580: - Attachment: 6580-trunk-v5.txt Added getTable(String, ExecutorService) API. Added simple test to TestHCM to exercise new APIs. Should be getting close. Deprecate HTablePool in favor of HConnection.getTable(...) -- Key: HBASE-6580 URL: https://issues.apache.org/jira/browse/HBASE-6580 Project: HBase Issue Type: Improvement Affects Versions: 0.94.6, 0.95.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: 6580-trunk.txt, 6580-trunk-v2.txt, 6580-trunk-v3.txt, 6580-trunk-v4.txt, 6580-trunk-v5.txt, HBASE-6580_v1.patch, HBASE-6580_v2.patch Update: I now propose deprecating HTablePool and instead introduce a getTable method on HConnection and allow HConnection to manage the ThreadPool. Initial proposal: Here I propose a very simple TablePool. It could be called LightHTablePool (or something - if you have a better name). Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it. In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used. It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729813#comment-13729813 ] stack commented on HBASE-9091: -- [~ndimiduk] So, this is not 'api sugar', is it? Because it will show through in the API? We need this for the other stuff to go in (unless we want flintstone computing)? Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729846#comment-13729846 ] Nick Dimiduk commented on HBASE-9091: - Yes, it does show through on the API. I consider it sugar at this point because the same features are supported using {{byte[], int}} vs {{PositionedByteRange}}. The former will be a little slower because of the extra traversal and pushes that much more into user code, but the features are identical. That's why I suggest the APIs could be extended later to support {{PositionedByteRange}} if we can't get consensus today. [~mcorgan], [~giacomotaylor], [~nkeywal], [~enis]: you've all expressed opinions about these API. Do these two classes look reasonable to you? Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9095) AssignmentManager's handleRegion should respect the single threaded nature of the processing
[ https://issues.apache.org/jira/browse/HBASE-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729844#comment-13729844 ] Hadoop QA commented on HBASE-9095: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12596163/9095-2.txt against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 6 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6605//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6605//console This message is automatically generated. AssignmentManager's handleRegion should respect the single threaded nature of the processing Key: HBASE-9095 URL: https://issues.apache.org/jira/browse/HBASE-9095 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.95.2 Attachments: 9095-1 (1).txt, 9095-1.txt, 9095-1.txt, 9095-1.txt, 9095-2.txt While debugging a case where a region was getting opened on a RegionServer and then closed soon after (and then never re-opened anywhere thereafter), it seemed like the processing in handleRegion to do with deletion of ZK nodes should be non-asynchronous. This achieves two things: 1. The synchronous deletion prevents more than one processing on the same event data twice. Assuming that we do get more than one notification (on let's say, region OPENED event), the subsequent processing(s) in handleRegion for the same znode would end up with a zookeeper node not found exception. The return value of the data read would be null and that's already handled. If it is asynchronous, it leads to issues like - master opens a region on a certain RegionServer and soon after it sends that RegionServer a close for the same region, and then the znode is deleted. 2. The deletion is currently handled in an executor service. This is problematic since by design the events for a given region should be processed in order. By delegating a part of the processing to executor service we are somewhat violating this contract since there is no guarantee of the ordering in the executor service executions... Thanks to [~jeffreyz] and [~enis] for the discussions on this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see:
[jira] [Commented] (HBASE-6580) Deprecate HTablePool in favor of HConnection.getTable(...)
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729849#comment-13729849 ] Lars Hofhansl commented on HBASE-6580: -- I might want to move this to 0.94.12, so that we can do proper review/testing on this (I would like to spin 0.94.11 today). Deprecate HTablePool in favor of HConnection.getTable(...) -- Key: HBASE-6580 URL: https://issues.apache.org/jira/browse/HBASE-6580 Project: HBase Issue Type: Improvement Affects Versions: 0.94.6, 0.95.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: 6580-trunk.txt, 6580-trunk-v2.txt, 6580-trunk-v3.txt, 6580-trunk-v4.txt, 6580-trunk-v5.txt, HBASE-6580_v1.patch, HBASE-6580_v2.patch Update: I now propose deprecating HTablePool and instead introduce a getTable method on HConnection and allow HConnection to manage the ThreadPool. Initial proposal: Here I propose a very simple TablePool. It could be called LightHTablePool (or something - if you have a better name). Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it. In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used. It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9127) HFileContext
Andrew Purtell created HBASE-9127: - Summary: HFileContext Key: HBASE-9127 URL: https://issues.apache.org/jira/browse/HBASE-9127 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell We can roll up at least some of the state we are leaking between the Store layer and the HFile layer by introducing an IO context for passing between the two. This idea has come up in other discussions. Here I am calling it 'HFileContext' because the particulars are regarding how to configure HFile readers and writers. This will be easier to maintain than various and sundry method parameters and (duplicated) instance variables sprinkled about, and will make adding or modifying persistence features easier and less disruptive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) Deprecate HTablePool in favor of HConnection.getTable(...)
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6580: - Fix Version/s: (was: 0.94.11) 0.94.12 Moving to 0.94.12. Preparing for 0.94.11 Deprecate HTablePool in favor of HConnection.getTable(...) -- Key: HBASE-6580 URL: https://issues.apache.org/jira/browse/HBASE-6580 Project: HBase Issue Type: Improvement Affects Versions: 0.94.6, 0.95.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.98.0, 0.95.2, 0.94.12 Attachments: 6580-trunk.txt, 6580-trunk-v2.txt, 6580-trunk-v3.txt, 6580-trunk-v4.txt, 6580-trunk-v5.txt, HBASE-6580_v1.patch, HBASE-6580_v2.patch Update: I now propose deprecating HTablePool and instead introduce a getTable method on HConnection and allow HConnection to manage the ThreadPool. Initial proposal: Here I propose a very simple TablePool. It could be called LightHTablePool (or something - if you have a better name). Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it. In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used. It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8496) Implement tags and the internals of how a tag should look like
[ https://issues.apache.org/jira/browse/HBASE-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729855#comment-13729855 ] ramkrishna.s.vasudevan commented on HBASE-8496: --- Pls note that we would not be persisting any tag related information on the HFileBlock. HFileBlock would not be modified in terms of what is persisted. In the recent patch that we have worked up on we are just enclosing some of the state variables into a POJO. So specifically for tags there are no modifications on HfileBlock. So there is no need for subclassing HFileBlock or add a new version to the hfileblock. Based on the Encoding/Decoding context state the Encoder and decoding logic of the algos would handle tags. In the previous version of the patches the includeMemstoreTS and the includeTags created changes to the interface apis. So as to avoid this we enclosed these things inside the Context. Iin the recent patches the context would have the POJO mentioned above). So from the context we would be able to get the various decision points like to include tags or not, includememstoreTS or not. The encoding apis carry the context but the actual seekers that does the actual decoding does not use contexts. So we have made changes such that the encoding logic and the seekers use the context related objects. Implement tags and the internals of how a tag should look like -- Key: HBASE-8496 URL: https://issues.apache.org/jira/browse/HBASE-8496 Project: HBase Issue Type: New Feature Affects Versions: 0.98.0, 0.95.2 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical Attachments: Comparison.pdf, HBASE-8496_2.patch, HBASE-8496.patch, Tag design.pdf, Tag design_updated.pdf, Tag_In_KV_Buffer_For_reference.patch The intent of this JIRA comes from HBASE-7897. This would help us to decide on the structure and format of how the tags should look like. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9127) HFileContext
[ https://issues.apache.org/jira/browse/HBASE-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9127: -- Attachment: 0003-Move-data-block-encoder-into-HFileContext.patch 0002-Move-compression-algorithm-into-HFileContext.patch 0001-Introduce-HFileContext.patch Attached are three patches as a thought experiment. They are based on 0.94 but taking the same actions in later versions would be similar. 0001 - Introduces HFileContext and moves CacheConfig into it. 0002 - Moves compression algorithm selection into HFileContext. 0003 - Moves in cache and on disk data block encoder selection into HFileContext. This one is not tested, so a few unit tests may need more tweaks. Next steps could be: - Move BloomType into HFileContext - Move the MVCC read point into HFileContext - Test if the additional indirection has a performance impact. - Consider what more of HFileContext state can be made final/immutable to facilitate inlining. This is interesting wherever a reader wants to reconfigure based on file trailer or other metadata discovered when opening the HFile. - CacheConfig could be fully rolled into HFileContext. - In the HFile V3 work we have a patch that, like in the existing case with memstoreTS, does whole file optimization if no tags will be stored (just like if memstoreTS would be 0 for all KVs in an HFile), which breaks encapsulation in the Store and Compactor. Putting this into HFileContext (at some future time) would fix that. - HBASE-7544 could be reworked to use this instead of sprinkling additional method parameters throughout Store and HFile. HFileContext Key: HBASE-9127 URL: https://issues.apache.org/jira/browse/HBASE-9127 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Attachments: 0001-Introduce-HFileContext.patch, 0002-Move-compression-algorithm-into-HFileContext.patch, 0003-Move-data-block-encoder-into-HFileContext.patch We can roll up at least some of the state we are leaking between the Store layer and the HFile layer by introducing an IO context for passing between the two. This idea has come up in other discussions. Here I am calling it 'HFileContext' because the particulars are regarding how to configure HFile readers and writers. This will be easier to maintain than various and sundry method parameters and (duplicated) instance variables sprinkled about, and will make adding or modifying persistence features easier and less disruptive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9118) Optimization in HFile V3 when no tags are present in a file
[ https://issues.apache.org/jira/browse/HBASE-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729857#comment-13729857 ] ramkrishna.s.vasudevan commented on HBASE-9118: --- We could use HBASE-9127(something similar) and refactor how mvcc works along with the tags optimization. Optimization in HFile V3 when no tags are present in a file --- Key: HBASE-9118 URL: https://issues.apache.org/jira/browse/HBASE-9118 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0, 0.95.2 Reporter: Anoop Sam John Assignee: Anoop Sam John Now with V3 we will write tags length (short) along with every KV after value bytes. When no tags at all present it will be wasting 2 bytes for every KV. We can avoid this. During flush let the tags length be written. In fileInfo we can add info like max tags length. During compaction if all the files undergoing the compaction having a 0 max tags length, we can avoid writing 2 bytes (0) along with every KV.. Note : Similar optimization available with mvcc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9095) AssignmentManager's handleRegion should respect the single threaded nature of the processing
[ https://issues.apache.org/jira/browse/HBASE-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729870#comment-13729870 ] ramkrishna.s.vasudevan commented on HBASE-9095: --- Ok to do this in 0.94 also to avoid many of the concurrency issues? Need to see the code to see how much it helps though? What [~jxiang], [~jeffreyz] feel? AssignmentManager's handleRegion should respect the single threaded nature of the processing Key: HBASE-9095 URL: https://issues.apache.org/jira/browse/HBASE-9095 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.95.2 Attachments: 9095-1 (1).txt, 9095-1.txt, 9095-1.txt, 9095-1.txt, 9095-2.txt While debugging a case where a region was getting opened on a RegionServer and then closed soon after (and then never re-opened anywhere thereafter), it seemed like the processing in handleRegion to do with deletion of ZK nodes should be non-asynchronous. This achieves two things: 1. The synchronous deletion prevents more than one processing on the same event data twice. Assuming that we do get more than one notification (on let's say, region OPENED event), the subsequent processing(s) in handleRegion for the same znode would end up with a zookeeper node not found exception. The return value of the data read would be null and that's already handled. If it is asynchronous, it leads to issues like - master opens a region on a certain RegionServer and soon after it sends that RegionServer a close for the same region, and then the znode is deleted. 2. The deletion is currently handled in an executor service. This is problematic since by design the events for a given region should be processed in order. By delegating a part of the processing to executor service we are somewhat violating this contract since there is no guarantee of the ordering in the executor service executions... Thanks to [~jeffreyz] and [~enis] for the discussions on this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9099) logReplay could trigger double region assignment
[ https://issues.apache.org/jira/browse/HBASE-9099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729876#comment-13729876 ] Hudson commented on HBASE-9099: --- SUCCESS: Integrated in HBase-TRUNK #4343 (See [https://builds.apache.org/job/HBase-TRUNK/4343/]) HBASE-9099: logReplay could trigger double region assignment (jeffreyz: rev 1510615) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java logReplay could trigger double region assignment Key: HBASE-9099 URL: https://issues.apache.org/jira/browse/HBASE-9099 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.2 Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 0.98.0, 0.95.2 Attachments: hbase-9099.patch, hbase-9099-v1.patch The symptom is the first region assignment submitted in SSH is in progress while when am.waitOnRegionToClearRegionsInTransition times out we will re-submitted another SSH which will invoke another region assignment for the region. It will cause the region get stuck in RIT status. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9095) AssignmentManager's handleRegion should respect the single threaded nature of the processing
[ https://issues.apache.org/jira/browse/HBASE-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729882#comment-13729882 ] Jimmy Xiang commented on HBASE-9095: For 0.94, just one thread is used to process assignment events. Will this slow down assignments? For trunk, it is not a problem since we have up to 20 zk event workers by default. AssignmentManager's handleRegion should respect the single threaded nature of the processing Key: HBASE-9095 URL: https://issues.apache.org/jira/browse/HBASE-9095 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.95.2 Attachments: 9095-1 (1).txt, 9095-1.txt, 9095-1.txt, 9095-1.txt, 9095-2.txt While debugging a case where a region was getting opened on a RegionServer and then closed soon after (and then never re-opened anywhere thereafter), it seemed like the processing in handleRegion to do with deletion of ZK nodes should be non-asynchronous. This achieves two things: 1. The synchronous deletion prevents more than one processing on the same event data twice. Assuming that we do get more than one notification (on let's say, region OPENED event), the subsequent processing(s) in handleRegion for the same znode would end up with a zookeeper node not found exception. The return value of the data read would be null and that's already handled. If it is asynchronous, it leads to issues like - master opens a region on a certain RegionServer and soon after it sends that RegionServer a close for the same region, and then the znode is deleted. 2. The deletion is currently handled in an executor service. This is problematic since by design the events for a given region should be processed in order. By delegating a part of the processing to executor service we are somewhat violating this contract since there is no guarantee of the ordering in the executor service executions... Thanks to [~jeffreyz] and [~enis] for the discussions on this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9126) Make HFile MIN VERSION as 2
[ https://issues.apache.org/jira/browse/HBASE-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729887#comment-13729887 ] stack commented on HBASE-9126: -- +1 Makes sense. Make HFile MIN VERSION as 2 --- Key: HBASE-9126 URL: https://issues.apache.org/jira/browse/HBASE-9126 Project: HBase Issue Type: Improvement Affects Versions: 0.95.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9126.patch Removed the HFile V1 support from version95. We can make the min version supported as 2? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9072) TestSnapshotFromMaster.testSnapshotHFileArchiving fails
[ https://issues.apache.org/jira/browse/HBASE-9072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729894#comment-13729894 ] Hadoop QA commented on HBASE-9072: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12596174/HBASE-9072-fix0.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 6 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6606//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6606//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6606//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6606//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6606//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6606//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6606//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6606//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6606//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6606//console This message is automatically generated. TestSnapshotFromMaster.testSnapshotHFileArchiving fails --- Key: HBASE-9072 URL: https://issues.apache.org/jira/browse/HBASE-9072 Project: HBase Issue Type: Bug Components: test Reporter: stack Assignee: Matteo Bertozzi Attachments: HBASE-9072-fix0.patch http://54.241.6.143/job/HBase-TRUNK-Hadoop-2/org.apache.hbase$hbase-server/428/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/ {code} Failed 4600 actions: SocketTimeoutException: 4600 times, Stacktrace at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:794) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:756) at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1312) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297) org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 4600 actions: SocketTimeoutException: 4600 times, at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:157) at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:145) at org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:700) at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:827) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:794) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:756) at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1312) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
[jira] [Commented] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729898#comment-13729898 ] stack commented on HBASE-9091: -- Fix: {code} + * The state of a {@code ByteRange} is not synchronized and does to support + * concurrent access. {code} This seems superfluous: + public ByteRange(int capacity, int offset, int length) { Nice language ...it becomes a husk... Is this sentence missing a tail? + * Designed to be a mutable alternative to lgtm Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9095) AssignmentManager's handleRegion should respect the single threaded nature of the processing
[ https://issues.apache.org/jira/browse/HBASE-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729897#comment-13729897 ] Devaraj Das commented on HBASE-9095: Although a couple of you gave +1s before, mind taking a look at the patch once to see if it still is okay (main changes are to do with the unit test fixes). AssignmentManager's handleRegion should respect the single threaded nature of the processing Key: HBASE-9095 URL: https://issues.apache.org/jira/browse/HBASE-9095 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.95.2 Attachments: 9095-1 (1).txt, 9095-1.txt, 9095-1.txt, 9095-1.txt, 9095-2.txt While debugging a case where a region was getting opened on a RegionServer and then closed soon after (and then never re-opened anywhere thereafter), it seemed like the processing in handleRegion to do with deletion of ZK nodes should be non-asynchronous. This achieves two things: 1. The synchronous deletion prevents more than one processing on the same event data twice. Assuming that we do get more than one notification (on let's say, region OPENED event), the subsequent processing(s) in handleRegion for the same znode would end up with a zookeeper node not found exception. The return value of the data read would be null and that's already handled. If it is asynchronous, it leads to issues like - master opens a region on a certain RegionServer and soon after it sends that RegionServer a close for the same region, and then the znode is deleted. 2. The deletion is currently handled in an executor service. This is problematic since by design the events for a given region should be processed in order. By delegating a part of the processing to executor service we are somewhat violating this contract since there is no guarantee of the ordering in the executor service executions... Thanks to [~jeffreyz] and [~enis] for the discussions on this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8960) TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes
[ https://issues.apache.org/jira/browse/HBASE-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729902#comment-13729902 ] stack commented on HBASE-8960: -- Thanks [~jeffreyz] Failed this morning: http://54.241.6.143/job/HBase-0.95-Hadoop-2/759/org.apache.hbase$hbase-server/ TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes -- Key: HBASE-8960 URL: https://issues.apache.org/jira/browse/HBASE-8960 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jeffrey Zhong Priority: Minor Fix For: 0.95.2 Attachments: hbase-8960-addendum-2.patch, hbase-8960-addendum.patch, hbase-8960.patch http://54.241.6.143/job/HBase-0.95-Hadoop-2/org.apache.hbase$hbase-server/634/testReport/junit/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testLogReplayForDisablingTable/ {noformat} java.lang.AssertionError: expected:1000 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testLogReplayForDisablingTable(TestDistributedLogSplitting.java:797) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9055) HBaseAdmin#isTableEnabled() should return false for non-existent table
[ https://issues.apache.org/jira/browse/HBASE-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-9055: -- Resolution: Fixed Status: Resolved (was: Patch Available) HBaseAdmin#isTableEnabled() should return false for non-existent table --- Key: HBASE-9055 URL: https://issues.apache.org/jira/browse/HBASE-9055 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.98.0 Attachments: 9055-v1.txt, 9055-v2.txt, 9055-v3.txt Currently HBaseAdmin#isTableEnabled() returns true for a table which doesn't exist. We should check table existence. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9064) test-patch.sh would silently fail if compilation against hadoop 1.0 fails
[ https://issues.apache.org/jira/browse/HBASE-9064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-9064: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) test-patch.sh would silently fail if compilation against hadoop 1.0 fails - Key: HBASE-9064 URL: https://issues.apache.org/jira/browse/HBASE-9064 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Attachments: 9064-v1.txt {code} if [[ $? != 0 ]] ; then JIRA_COMMENT=$JIRA_COMMENT {color:red}-1 hadoop1.0{color}. The patch failed to compile against the hadoop 1.0 profile. cleanupAndExit 1 fi {code} There is currently no post back to JIRA when there is compilation error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8409) Security support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729904#comment-13729904 ] Francis Liu commented on HBASE-8409: Yes, that's the plan. The table migration will be part of 8408 just to keep the patches more independent. Yep it'll be done in NamespaceUpgrade. I don't plan to scan since I know the table names. I was hoping I could get by just deleting the old keys and writing the new ones. Also will have to rewrite .regioninfo since the table name change. All of this has to be done will the system is offline. Does this sound feasible? Vandana is on vacation this week. So I'll take care of the work in the interest of expediency. Security support for namespaces --- Key: HBASE-8409 URL: https://issues.apache.org/jira/browse/HBASE-8409 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Priority: Critical Attachments: HBASE-8049_trunk.patch This task adds the security piece to the namespace feature. The work related to migration of the existing acl table to the new namespace is remaining and will be completed in the follow up patch. Permissions can be granted to a namespace by the hbase admin, by appending '@' to the namespace name. A user with write or admin permissions on a given namespace can create tables in that namespace. The other privileges (R, X, C ) do not have any special meaning w.r.t namespaces. Any users of hbase can list tables in a namespace. The following commands can only be executed by HBase admins. 1. Grant privileges for user on Namespace. 2. Revoke privileges for user on Namespace Grant Command: hbase grant 'tenant-A' 'W' '@N1' In the above example, the command will grant the user 'tenant-A' write privileges for a namespace named N1. Revoke Command: hbase revoke 'tenant-A''@N1' In the above example, the command will revoke all privileges from user 'tenant-A' for namespace named N1. Lets see an example on how privileges work with namespaces. User Mike request for a namespace named hbase_perf with the hbase admin. whoami: hbase hbase shell namespace_create 'hbase_perf' hbase shell grant 'mike', 'W', '@hbase_perf' Mike creates two tables table20 and table50 in the above workspace. whoami: mike hbase shell create 'hbase_perf.table20', 'family1' hbase shell create 'hbase_perf.table50', 'family1' Note: As Mike was able to create tables 'hbase_perf.table20', 'hbase_perf.table50', he becomes the owner of those tables. This means he has RWXCA perms on those tables. Another team member of Mike, Alice wants also to share the same workspace hbase_perf. HBase admin grants Alice also permission to create tables in hbase_perf namespace. whoami: hbase hbase shell grant 'alice', 'W', '@hbase_perf' Now Alice can create new tables under hbase_perf namespace, but cannot read,write,alter,delete existing tables in the namespace. whoami: alice hbase shell namespace_list_tables 'hbase_perf' hbase_perf.table20 hbase_perf.table50 hbase shell scan 'hbase_perf.table20' AccessDeniedException If Alice wants to read or write to existing tables in the hbase_perf namespace, hbase admins need to explicitly grant permission. whoami: hbase hbase shell grant 'alice', 'RW', 'hbase_perf.table20' hbase shell grant 'alice', 'RW', 'hbase_perf.table50' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9099) logReplay could trigger double region assignment
[ https://issues.apache.org/jira/browse/HBASE-9099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729906#comment-13729906 ] Hudson commented on HBASE-9099: --- SUCCESS: Integrated in hbase-0.95 #404 (See [https://builds.apache.org/job/hbase-0.95/404/]) HBASE-9099: logReplay could trigger double region assignment (jeffreyz: rev 1510616) * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java logReplay could trigger double region assignment Key: HBASE-9099 URL: https://issues.apache.org/jira/browse/HBASE-9099 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.2 Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 0.98.0, 0.95.2 Attachments: hbase-9099.patch, hbase-9099-v1.patch The symptom is the first region assignment submitted in SSH is in progress while when am.waitOnRegionToClearRegionsInTransition times out we will re-submitted another SSH which will invoke another region assignment for the region. It will cause the region get stuck in RIT status. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8409) Security support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729914#comment-13729914 ] Francis Liu commented on HBASE-8409: {quote} Can the new acl table be called hbase.acl instead of hbase.acl? {quote} I assuming rename _acl_ = hbase:acl? That's the plan. {quote} What is the '@' about? Seems awkward? Is this because our current grant didn't include ns: {quote} It's to disambiguate granting per table and granting per namespace. The same paradigm (prefix with '@') is used to disambiguate user and group in the grant command. We could also go with grant user permissions namespace: ie grant 'foo', 'A', 'my_ns:' To disambiguate NS from table. We couldn't do this before since we were using '.'. Does this sound ok? Security support for namespaces --- Key: HBASE-8409 URL: https://issues.apache.org/jira/browse/HBASE-8409 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Priority: Critical Attachments: HBASE-8049_trunk.patch This task adds the security piece to the namespace feature. The work related to migration of the existing acl table to the new namespace is remaining and will be completed in the follow up patch. Permissions can be granted to a namespace by the hbase admin, by appending '@' to the namespace name. A user with write or admin permissions on a given namespace can create tables in that namespace. The other privileges (R, X, C ) do not have any special meaning w.r.t namespaces. Any users of hbase can list tables in a namespace. The following commands can only be executed by HBase admins. 1. Grant privileges for user on Namespace. 2. Revoke privileges for user on Namespace Grant Command: hbase grant 'tenant-A' 'W' '@N1' In the above example, the command will grant the user 'tenant-A' write privileges for a namespace named N1. Revoke Command: hbase revoke 'tenant-A''@N1' In the above example, the command will revoke all privileges from user 'tenant-A' for namespace named N1. Lets see an example on how privileges work with namespaces. User Mike request for a namespace named hbase_perf with the hbase admin. whoami: hbase hbase shell namespace_create 'hbase_perf' hbase shell grant 'mike', 'W', '@hbase_perf' Mike creates two tables table20 and table50 in the above workspace. whoami: mike hbase shell create 'hbase_perf.table20', 'family1' hbase shell create 'hbase_perf.table50', 'family1' Note: As Mike was able to create tables 'hbase_perf.table20', 'hbase_perf.table50', he becomes the owner of those tables. This means he has RWXCA perms on those tables. Another team member of Mike, Alice wants also to share the same workspace hbase_perf. HBase admin grants Alice also permission to create tables in hbase_perf namespace. whoami: hbase hbase shell grant 'alice', 'W', '@hbase_perf' Now Alice can create new tables under hbase_perf namespace, but cannot read,write,alter,delete existing tables in the namespace. whoami: alice hbase shell namespace_list_tables 'hbase_perf' hbase_perf.table20 hbase_perf.table50 hbase shell scan 'hbase_perf.table20' AccessDeniedException If Alice wants to read or write to existing tables in the hbase_perf namespace, hbase admins need to explicitly grant permission. whoami: hbase hbase shell grant 'alice', 'RW', 'hbase_perf.table20' hbase shell grant 'alice', 'RW', 'hbase_perf.table50' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8409) Security support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729919#comment-13729919 ] stack commented on HBASE-8409: -- I forgot that group is done w/ a '@'. So there is precedent. Might be ok as-is then. On rewriting .regioninfo, FYI HBASE-8778 moves stuff around and is about to go in. Security support for namespaces --- Key: HBASE-8409 URL: https://issues.apache.org/jira/browse/HBASE-8409 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Priority: Critical Attachments: HBASE-8049_trunk.patch This task adds the security piece to the namespace feature. The work related to migration of the existing acl table to the new namespace is remaining and will be completed in the follow up patch. Permissions can be granted to a namespace by the hbase admin, by appending '@' to the namespace name. A user with write or admin permissions on a given namespace can create tables in that namespace. The other privileges (R, X, C ) do not have any special meaning w.r.t namespaces. Any users of hbase can list tables in a namespace. The following commands can only be executed by HBase admins. 1. Grant privileges for user on Namespace. 2. Revoke privileges for user on Namespace Grant Command: hbase grant 'tenant-A' 'W' '@N1' In the above example, the command will grant the user 'tenant-A' write privileges for a namespace named N1. Revoke Command: hbase revoke 'tenant-A''@N1' In the above example, the command will revoke all privileges from user 'tenant-A' for namespace named N1. Lets see an example on how privileges work with namespaces. User Mike request for a namespace named hbase_perf with the hbase admin. whoami: hbase hbase shell namespace_create 'hbase_perf' hbase shell grant 'mike', 'W', '@hbase_perf' Mike creates two tables table20 and table50 in the above workspace. whoami: mike hbase shell create 'hbase_perf.table20', 'family1' hbase shell create 'hbase_perf.table50', 'family1' Note: As Mike was able to create tables 'hbase_perf.table20', 'hbase_perf.table50', he becomes the owner of those tables. This means he has RWXCA perms on those tables. Another team member of Mike, Alice wants also to share the same workspace hbase_perf. HBase admin grants Alice also permission to create tables in hbase_perf namespace. whoami: hbase hbase shell grant 'alice', 'W', '@hbase_perf' Now Alice can create new tables under hbase_perf namespace, but cannot read,write,alter,delete existing tables in the namespace. whoami: alice hbase shell namespace_list_tables 'hbase_perf' hbase_perf.table20 hbase_perf.table50 hbase shell scan 'hbase_perf.table20' AccessDeniedException If Alice wants to read or write to existing tables in the hbase_perf namespace, hbase admins need to explicitly grant permission. whoami: hbase hbase shell grant 'alice', 'RW', 'hbase_perf.table20' hbase shell grant 'alice', 'RW', 'hbase_perf.table50' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9072) TestSnapshotFromMaster.testSnapshotHFileArchiving fails
[ https://issues.apache.org/jira/browse/HBASE-9072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729920#comment-13729920 ] Ted Yu commented on HBASE-9072: --- +1 {code} +waitForTableToBeOnline(util, table.getTableName()); {code} The above method waits for flushes. Maybe rename the method to reflect this waiting. TestSnapshotFromMaster.testSnapshotHFileArchiving fails --- Key: HBASE-9072 URL: https://issues.apache.org/jira/browse/HBASE-9072 Project: HBase Issue Type: Bug Components: test Reporter: stack Assignee: Matteo Bertozzi Attachments: HBASE-9072-fix0.patch http://54.241.6.143/job/HBase-TRUNK-Hadoop-2/org.apache.hbase$hbase-server/428/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/ {code} Failed 4600 actions: SocketTimeoutException: 4600 times, Stacktrace at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:794) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:756) at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1312) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297) org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 4600 actions: SocketTimeoutException: 4600 times, at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:157) at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:145) at org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:700) at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:827) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:794) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:756) at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1312) at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runners.Suite.runChild(Suite.java:127) at org.junit.runners.Suite.runChild(Suite.java:26) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) {code} [~mbertozzi] Opinion? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) Deprecate HTablePool in favor of HConnection.getTable(...)
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729925#comment-13729925 ] Lars Hofhansl commented on HBASE-6580: -- In the fact the 0.94 patch will be quite a bit different. Deprecate HTablePool in favor of HConnection.getTable(...) -- Key: HBASE-6580 URL: https://issues.apache.org/jira/browse/HBASE-6580 Project: HBase Issue Type: Improvement Affects Versions: 0.94.6, 0.95.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.98.0, 0.95.2, 0.94.12 Attachments: 6580-trunk.txt, 6580-trunk-v2.txt, 6580-trunk-v3.txt, 6580-trunk-v4.txt, 6580-trunk-v5.txt, HBASE-6580_v1.patch, HBASE-6580_v2.patch Update: I now propose deprecating HTablePool and instead introduce a getTable method on HConnection and allow HConnection to manage the ThreadPool. Initial proposal: Here I propose a very simple TablePool. It could be called LightHTablePool (or something - if you have a better name). Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it. In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used. It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8760) possible loss of data in snapshot taken after region split
[ https://issues.apache.org/jira/browse/HBASE-8760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-8760: --- Attachment: HBASE-8760-thz-v2.patch possible loss of data in snapshot taken after region split -- Key: HBASE-8760 URL: https://issues.apache.org/jira/browse/HBASE-8760 Project: HBase Issue Type: Bug Components: snapshots Affects Versions: 0.94.8, 0.95.1 Reporter: Jerry He Fix For: 0.98.0, 0.95.2, 0.94.12 Attachments: HBase-8760-0.94.8.patch, HBase-8760-0.94.8-v1.patch, HBASE-8760-thz-v0.patch, HBASE-8760-thz-v1.patch, HBASE-8760-thz-v2.patch Right after a region split but before the daughter regions are compacted, we have two daughter regions containing Reference files to the parent hfiles. If we take snapshot right at the moment, the snapshot will succeed, but it will only contain the daughter Reference files. Since there is no hold on the parent hfiles, they will be deleted by the HFile Cleaner after they are no longer needed by the daughter regions soon after. A minimum we need to do is the keep these parent hfiles from being deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8760) possible loss of data in snapshot taken after region split
[ https://issues.apache.org/jira/browse/HBASE-8760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729934#comment-13729934 ] Matteo Bertozzi commented on HBASE-8760: v2 has the fix requested above, no real code changes. I want to test this patch more, with all the 3 cases above (normal table, clone table, clone of a clone) [~jerryjch] are you testing the patch on a cluster or just looking at the code? possible loss of data in snapshot taken after region split -- Key: HBASE-8760 URL: https://issues.apache.org/jira/browse/HBASE-8760 Project: HBase Issue Type: Bug Components: snapshots Affects Versions: 0.94.8, 0.95.1 Reporter: Jerry He Fix For: 0.98.0, 0.95.2, 0.94.12 Attachments: HBase-8760-0.94.8.patch, HBase-8760-0.94.8-v1.patch, HBASE-8760-thz-v0.patch, HBASE-8760-thz-v1.patch, HBASE-8760-thz-v2.patch Right after a region split but before the daughter regions are compacted, we have two daughter regions containing Reference files to the parent hfiles. If we take snapshot right at the moment, the snapshot will succeed, but it will only contain the daughter Reference files. Since there is no hold on the parent hfiles, they will be deleted by the HFile Cleaner after they are no longer needed by the daughter regions soon after. A minimum we need to do is the keep these parent hfiles from being deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9095) AssignmentManager's handleRegion should respect the single threaded nature of the processing
[ https://issues.apache.org/jira/browse/HBASE-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729951#comment-13729951 ] Jimmy Xiang commented on HBASE-9095: +1. Probably we can change how the unit test does verifications later on, so that we don't need the special counter/tracker. AssignmentManager's handleRegion should respect the single threaded nature of the processing Key: HBASE-9095 URL: https://issues.apache.org/jira/browse/HBASE-9095 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.95.2 Attachments: 9095-1 (1).txt, 9095-1.txt, 9095-1.txt, 9095-1.txt, 9095-2.txt While debugging a case where a region was getting opened on a RegionServer and then closed soon after (and then never re-opened anywhere thereafter), it seemed like the processing in handleRegion to do with deletion of ZK nodes should be non-asynchronous. This achieves two things: 1. The synchronous deletion prevents more than one processing on the same event data twice. Assuming that we do get more than one notification (on let's say, region OPENED event), the subsequent processing(s) in handleRegion for the same znode would end up with a zookeeper node not found exception. The return value of the data read would be null and that's already handled. If it is asynchronous, it leads to issues like - master opens a region on a certain RegionServer and soon after it sends that RegionServer a close for the same region, and then the znode is deleted. 2. The deletion is currently handled in an executor service. This is problematic since by design the events for a given region should be processed in order. By delegating a part of the processing to executor service we are somewhat violating this contract since there is no guarantee of the ordering in the executor service executions... Thanks to [~jeffreyz] and [~enis] for the discussions on this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729952#comment-13729952 ] Matt Corgan commented on HBASE-9091: Looks good Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9091: Attachment: 0001-HBASE-9091-Introduce-PositionedByteRange.patch Addressing Stack's comments. Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9091: Hadoop Flags: Reviewed Status: Patch Available (was: Open) Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9091: Status: Open (was: Patch Available) Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9095) AssignmentManager's handleRegion should respect the single threaded nature of the processing
[ https://issues.apache.org/jira/browse/HBASE-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729975#comment-13729975 ] Devaraj Das commented on HBASE-9095: Thanks [~jxiang]. I'll commit the patch in an hour or so unless I hear any objection. AssignmentManager's handleRegion should respect the single threaded nature of the processing Key: HBASE-9095 URL: https://issues.apache.org/jira/browse/HBASE-9095 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.95.2 Attachments: 9095-1 (1).txt, 9095-1.txt, 9095-1.txt, 9095-1.txt, 9095-2.txt While debugging a case where a region was getting opened on a RegionServer and then closed soon after (and then never re-opened anywhere thereafter), it seemed like the processing in handleRegion to do with deletion of ZK nodes should be non-asynchronous. This achieves two things: 1. The synchronous deletion prevents more than one processing on the same event data twice. Assuming that we do get more than one notification (on let's say, region OPENED event), the subsequent processing(s) in handleRegion for the same znode would end up with a zookeeper node not found exception. The return value of the data read would be null and that's already handled. If it is asynchronous, it leads to issues like - master opens a region on a certain RegionServer and soon after it sends that RegionServer a close for the same region, and then the znode is deleted. 2. The deletion is currently handled in an executor service. This is problematic since by design the events for a given region should be processed in order. By delegating a part of the processing to executor service we are somewhat violating this contract since there is no guarantee of the ordering in the executor service executions... Thanks to [~jeffreyz] and [~enis] for the discussions on this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729981#comment-13729981 ] stack commented on HBASE-8408: -- Trying the patch, LoadTestTool is only hunk that would not apply (Easy fix of an import). Also has prob w/ b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java This pretty brutal in the shell: {code} hbase(main):002:0 scan '.META.' ERROR: java.lang.IllegalArgumentException: Illegal first character 46 at 0. Namespaces can only start with alphanumeric characters': i.e. [a-zA-Z_0-9]: .META. Here is some help for this command: Scan a table; pass table name and optionally a dictionary of scanner specifications. Scanner specifications may include one or more of: TIMERANGE, FILTER, LIMIT, STARTROW, STOPROW, TIMESTAMP, MAXLENGTH, or COLUMNS, CACHE {code} Should probably give folks a bit of a nicer letdown. If I do 'list', I do not see system tables and the help gives me no clue they exist. I suppose this is how it always was. Should I be able to do this? hbase(main):013:0 create 'hbase:stack', 'stack' 0 row(s) in 0.1790 seconds I created a system table named 'stack'? The 'create' command needs updating so it has some namespace stuff going on? Doesn't it? It has nothing on it currently. Implement namespace --- Key: HBASE-8408 URL: https://issues.apache.org/jira/browse/HBASE-8408 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-8015_11.patch, HBASE-8015_12.patch, HBASE-8015_13.patch, HBASE-8015_14.patch, HBASE-8015_15.patch, HBASE-8015_16.patch, HBASE-8015_1.patch, HBASE-8015_2.patch, HBASE-8015_3.patch, HBASE-8015_4.patch, HBASE-8015_5.patch, HBASE-8015_6.patch, HBASE-8015_7.patch, HBASE-8015_8.patch, HBASE-8015_9.patch, HBASE-8015.patch, TestNamespaceMigration.tgz, TestNamespaceUpgrade.tgz -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729986#comment-13729986 ] Hadoop QA commented on HBASE-9091: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12596172/0001-HBASE-9091-Introduce-PositionedByteRange.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 16 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 4 warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6607//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6607//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6607//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6607//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6607//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6607//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6607//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6607//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6607//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6607//console This message is automatically generated. Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7980) TestZKInterProcessReadWriteLock fails occasionally in QA test run
[ https://issues.apache.org/jira/browse/HBASE-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-7980: - Assignee: Enis Soztutar TestZKInterProcessReadWriteLock fails occasionally in QA test run - Key: HBASE-7980 URL: https://issues.apache.org/jira/browse/HBASE-7980 Project: HBase Issue Type: Bug Components: test Reporter: Ted Yu Assignee: Enis Soztutar Fix For: 0.95.2 {code} testReadLockExcludesWriters(org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock) Time elapsed: 0.003 sec ERROR! java.lang.Exception: test timed out after 3000 milliseconds at sun.misc.Unsafe.park(Native Method) {code} You can find the test output here: https://builds.apache.org/job/PreCommit-HBASE-Build/4634/artifact/trunk/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock-output.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7980) TestZKInterProcessReadWriteLock fails occasionally in QA test run
[ https://issues.apache.org/jira/browse/HBASE-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-7980: - Attachment: hbase-7980_v1.patch There seems to be a typo with the timeout. I'll commit the attached patch shortly. TestZKInterProcessReadWriteLock fails occasionally in QA test run - Key: HBASE-7980 URL: https://issues.apache.org/jira/browse/HBASE-7980 Project: HBase Issue Type: Bug Components: test Reporter: Ted Yu Assignee: Enis Soztutar Fix For: 0.95.2 Attachments: hbase-7980_v1.patch {code} testReadLockExcludesWriters(org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock) Time elapsed: 0.003 sec ERROR! java.lang.Exception: test timed out after 3000 milliseconds at sun.misc.Unsafe.park(Native Method) {code} You can find the test output here: https://builds.apache.org/job/PreCommit-HBASE-Build/4634/artifact/trunk/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock-output.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729993#comment-13729993 ] Nick Dimiduk commented on HBASE-9091: - [~mcorgan] You have a comment in there about possibly extending this to be an interface with different backing implementations. How serious are you about that? Introducing this change places this class front-and-center for users' consumption and this will limit our ability to make changes in the future. Hat-tip to [~enis] for bringing this up. Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729994#comment-13729994 ] stack commented on HBASE-8408: -- If I take a snapshot running in local mode, my data dir goes away. Do you see that [~toffer] Implement namespace --- Key: HBASE-8408 URL: https://issues.apache.org/jira/browse/HBASE-8408 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-8015_11.patch, HBASE-8015_12.patch, HBASE-8015_13.patch, HBASE-8015_14.patch, HBASE-8015_15.patch, HBASE-8015_16.patch, HBASE-8015_1.patch, HBASE-8015_2.patch, HBASE-8015_3.patch, HBASE-8015_4.patch, HBASE-8015_5.patch, HBASE-8015_6.patch, HBASE-8015_7.patch, HBASE-8015_8.patch, HBASE-8015_9.patch, HBASE-8015.patch, TestNamespaceMigration.tgz, TestNamespaceUpgrade.tgz -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7980) TestZKInterProcessReadWriteLock fails occasionally in QA test run
[ https://issues.apache.org/jira/browse/HBASE-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar resolved HBASE-7980. -- Resolution: Fixed Fix Version/s: 0.98.0 TestZKInterProcessReadWriteLock fails occasionally in QA test run - Key: HBASE-7980 URL: https://issues.apache.org/jira/browse/HBASE-7980 Project: HBase Issue Type: Bug Components: test Reporter: Ted Yu Assignee: Enis Soztutar Priority: Trivial Fix For: 0.98.0, 0.95.2 Attachments: hbase-7980_v1.patch {code} testReadLockExcludesWriters(org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock) Time elapsed: 0.003 sec ERROR! java.lang.Exception: test timed out after 3000 milliseconds at sun.misc.Unsafe.park(Native Method) {code} You can find the test output here: https://builds.apache.org/job/PreCommit-HBASE-Build/4634/artifact/trunk/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock-output.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9126) Make HFile MIN VERSION as 2
[ https://issues.apache.org/jira/browse/HBASE-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730001#comment-13730001 ] Ted Yu commented on HBASE-9126: --- Looks like the following check can also be simplified: {code} if (majorVersion 2 || (majorVersion == 2 minorVersion = PBUF_TRAILER_MINOR_VERSION)) { if (majorVersion 2 || (majorVersion == 2 minorVersion = PBUF_TRAILER_MINOR_VERSION)) { src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java {code} Make HFile MIN VERSION as 2 --- Key: HBASE-9126 URL: https://issues.apache.org/jira/browse/HBASE-9126 Project: HBase Issue Type: Improvement Affects Versions: 0.95.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9126.patch Removed the HFile V1 support from version95. We can make the min version supported as 2? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7980) TestZKInterProcessReadWriteLock fails occasionally in QA test run
[ https://issues.apache.org/jira/browse/HBASE-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-7980: - Priority: Trivial (was: Major) TestZKInterProcessReadWriteLock fails occasionally in QA test run - Key: HBASE-7980 URL: https://issues.apache.org/jira/browse/HBASE-7980 Project: HBase Issue Type: Bug Components: test Reporter: Ted Yu Assignee: Enis Soztutar Priority: Trivial Fix For: 0.95.2 Attachments: hbase-7980_v1.patch {code} testReadLockExcludesWriters(org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock) Time elapsed: 0.003 sec ERROR! java.lang.Exception: test timed out after 3000 milliseconds at sun.misc.Unsafe.park(Native Method) {code} You can find the test output here: https://builds.apache.org/job/PreCommit-HBASE-Build/4634/artifact/trunk/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock-output.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7980) TestZKInterProcessReadWriteLock fails occasionally in QA test run
[ https://issues.apache.org/jira/browse/HBASE-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13729997#comment-13729997 ] Enis Soztutar commented on HBASE-7980: -- I committed the patch. Let me resolve this. Please reopen if you see the failure again. TestZKInterProcessReadWriteLock fails occasionally in QA test run - Key: HBASE-7980 URL: https://issues.apache.org/jira/browse/HBASE-7980 Project: HBase Issue Type: Bug Components: test Reporter: Ted Yu Assignee: Enis Soztutar Priority: Trivial Fix For: 0.95.2 Attachments: hbase-7980_v1.patch {code} testReadLockExcludesWriters(org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock) Time elapsed: 0.003 sec ERROR! java.lang.Exception: test timed out after 3000 milliseconds at sun.misc.Unsafe.park(Native Method) {code} You can find the test output here: https://builds.apache.org/job/PreCommit-HBASE-Build/4634/artifact/trunk/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock-output.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8760) possible loss of data in snapshot taken after region split
[ https://issues.apache.org/jira/browse/HBASE-8760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730002#comment-13730002 ] Jerry He commented on HBASE-8760: - I had seen the problem on a 0.94 live cluster. It happened when there was heavy write load on the cluster while the snapshot was taken. Later to re-create the problem, I had to suspend the compaction thread manually so that right after region split the new regions would not be compacted right away. I have not got a chance to do testing on this patch yet. possible loss of data in snapshot taken after region split -- Key: HBASE-8760 URL: https://issues.apache.org/jira/browse/HBASE-8760 Project: HBase Issue Type: Bug Components: snapshots Affects Versions: 0.94.8, 0.95.1 Reporter: Jerry He Fix For: 0.98.0, 0.95.2, 0.94.12 Attachments: HBase-8760-0.94.8.patch, HBase-8760-0.94.8-v1.patch, HBASE-8760-thz-v0.patch, HBASE-8760-thz-v1.patch, HBASE-8760-thz-v2.patch Right after a region split but before the daughter regions are compacted, we have two daughter regions containing Reference files to the parent hfiles. If we take snapshot right at the moment, the snapshot will succeed, but it will only contain the daughter Reference files. Since there is no hold on the parent hfiles, they will be deleted by the HFile Cleaner after they are no longer needed by the daughter regions soon after. A minimum we need to do is the keep these parent hfiles from being deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9128) Drop the '.' prefix from names under hbase.rootdir after namespaces goes in
stack created HBASE-9128: Summary: Drop the '.' prefix from names under hbase.rootdir after namespaces goes in Key: HBASE-9128 URL: https://issues.apache.org/jira/browse/HBASE-9128 Project: HBase Issue Type: Sub-task Reporter: stack Priority: Critical Change it so instead of hbase.rootdir looking like this: {code} drwxr-xr-x 5 stack staff 170 Aug 5 14:43 .data -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.id.crc -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.version.crc drwxr-xr-x 3 stack staff 102 Aug 5 14:43 .logs drwxr-xr-x 2 stack staff 68 Aug 5 14:43 .oldlogs drwxr-xr-x 3 stack staff 102 Aug 5 14:43 .tmp -rwxrwxrwx 1 stack staff 42 Aug 5 14:43 hbase.id -rwxrwxrwx 1 stack staff7 Aug 5 14:43 hbase.version {code} It looks like this: {code} drwxr-xr-x 5 stack staff 170 Aug 5 14:43 data -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.id.crc -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.version.crc drwxr-xr-x 3 stack staff 102 Aug 5 14:43 logs drwxr-xr-x 2 stack staff 68 Aug 5 14:43 oldlogs drwxr-xr-x 3 stack staff 102 Aug 5 14:43 tmp -rwxrwxrwx 1 stack staff 42 Aug 5 14:43 hbase.id -rwxrwxrwx 1 stack staff7 Aug 5 14:43 hbase.version {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9128) Drop the '.' prefix from names under hbase.rootdir after namespaces goes in
[ https://issues.apache.org/jira/browse/HBASE-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730008#comment-13730008 ] Enis Soztutar commented on HBASE-9128: -- Sounds good. The thing that came up in NS review was that if the user had a table named data, logs, etc the migration should handle this accordingly. Drop the '.' prefix from names under hbase.rootdir after namespaces goes in --- Key: HBASE-9128 URL: https://issues.apache.org/jira/browse/HBASE-9128 Project: HBase Issue Type: Sub-task Reporter: stack Priority: Critical Change it so instead of hbase.rootdir looking like this: {code} drwxr-xr-x 5 stack staff 170 Aug 5 14:43 .data -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.id.crc -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.version.crc drwxr-xr-x 3 stack staff 102 Aug 5 14:43 .logs drwxr-xr-x 2 stack staff 68 Aug 5 14:43 .oldlogs drwxr-xr-x 3 stack staff 102 Aug 5 14:43 .tmp -rwxrwxrwx 1 stack staff 42 Aug 5 14:43 hbase.id -rwxrwxrwx 1 stack staff7 Aug 5 14:43 hbase.version {code} It looks like this: {code} drwxr-xr-x 5 stack staff 170 Aug 5 14:43 data -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.id.crc -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.version.crc drwxr-xr-x 3 stack staff 102 Aug 5 14:43 logs drwxr-xr-x 2 stack staff 68 Aug 5 14:43 oldlogs drwxr-xr-x 3 stack staff 102 Aug 5 14:43 tmp -rwxrwxrwx 1 stack staff 42 Aug 5 14:43 hbase.id -rwxrwxrwx 1 stack staff7 Aug 5 14:43 hbase.version {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730009#comment-13730009 ] stack commented on HBASE-8408: -- [~toffer] nvm. I did ls and the dot dirs did not show. My bad (sorry if I gave you a heart attack). Implement namespace --- Key: HBASE-8408 URL: https://issues.apache.org/jira/browse/HBASE-8408 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-8015_11.patch, HBASE-8015_12.patch, HBASE-8015_13.patch, HBASE-8015_14.patch, HBASE-8015_15.patch, HBASE-8015_16.patch, HBASE-8015_1.patch, HBASE-8015_2.patch, HBASE-8015_3.patch, HBASE-8015_4.patch, HBASE-8015_5.patch, HBASE-8015_6.patch, HBASE-8015_7.patch, HBASE-8015_8.patch, HBASE-8015_9.patch, HBASE-8015.patch, TestNamespaceMigration.tgz, TestNamespaceUpgrade.tgz -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730012#comment-13730012 ] stack commented on HBASE-8408: -- +1 Seems to work. Implement namespace --- Key: HBASE-8408 URL: https://issues.apache.org/jira/browse/HBASE-8408 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-8015_11.patch, HBASE-8015_12.patch, HBASE-8015_13.patch, HBASE-8015_14.patch, HBASE-8015_15.patch, HBASE-8015_16.patch, HBASE-8015_1.patch, HBASE-8015_2.patch, HBASE-8015_3.patch, HBASE-8015_4.patch, HBASE-8015_5.patch, HBASE-8015_6.patch, HBASE-8015_7.patch, HBASE-8015_8.patch, HBASE-8015_9.patch, HBASE-8015.patch, TestNamespaceMigration.tgz, TestNamespaceUpgrade.tgz -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730021#comment-13730021 ] stack commented on HBASE-9091: -- +1 caveat question to [~mcorgan] Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8322) Re-enable hbase checksums by default
[ https://issues.apache.org/jira/browse/HBASE-8322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730025#comment-13730025 ] Enis Soztutar commented on HBASE-8322: -- bq. why did you pick CRC32C? It's not available in Hadoop 1 and it's what's failing. I did not realize CRC32C is hadoop2 only. We should default to CRC32 for sure. Generally speaking, CRC32C polynomial is said to be better. Re-enable hbase checksums by default Key: HBASE-8322 URL: https://issues.apache.org/jira/browse/HBASE-8322 Project: HBase Issue Type: Improvement Components: Filesystem Integration Reporter: Enis Soztutar Assignee: Jean-Daniel Cryans Priority: Critical Fix For: 0.98.0, 0.95.2 Attachments: hbase-8322_v1.patch, HBASE-8322-v2.patch Double checksumming issues in HBase level checksums(HBASE-5074) has been fixed in HBASE-6868. However, that patch also disables hbase checksums by default. I think we should re-enable it by default, and document the interaction with shortcircuit reads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9128) Drop the '.' prefix from names under hbase.rootdir after namespaces goes in
[ https://issues.apache.org/jira/browse/HBASE-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730046#comment-13730046 ] stack commented on HBASE-9128: -- Good point. How would migration 'handle' it? The table move into default ns has to happen before we do this renaming? Drop the '.' prefix from names under hbase.rootdir after namespaces goes in --- Key: HBASE-9128 URL: https://issues.apache.org/jira/browse/HBASE-9128 Project: HBase Issue Type: Sub-task Reporter: stack Priority: Critical Change it so instead of hbase.rootdir looking like this: {code} drwxr-xr-x 5 stack staff 170 Aug 5 14:43 .data -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.id.crc -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.version.crc drwxr-xr-x 3 stack staff 102 Aug 5 14:43 .logs drwxr-xr-x 2 stack staff 68 Aug 5 14:43 .oldlogs drwxr-xr-x 3 stack staff 102 Aug 5 14:43 .tmp -rwxrwxrwx 1 stack staff 42 Aug 5 14:43 hbase.id -rwxrwxrwx 1 stack staff7 Aug 5 14:43 hbase.version {code} It looks like this: {code} drwxr-xr-x 5 stack staff 170 Aug 5 14:43 data -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.id.crc -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.version.crc drwxr-xr-x 3 stack staff 102 Aug 5 14:43 logs drwxr-xr-x 2 stack staff 68 Aug 5 14:43 oldlogs drwxr-xr-x 3 stack staff 102 Aug 5 14:43 tmp -rwxrwxrwx 1 stack staff 42 Aug 5 14:43 hbase.id -rwxrwxrwx 1 stack staff7 Aug 5 14:43 hbase.version {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9091: Attachment: 0001-HBASE-9091-Introduce-PositionedByteRange.patch *sigh* javadoc. Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8803) region_mover.rb should move multiple regions at a time
[ https://issues.apache.org/jira/browse/HBASE-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730051#comment-13730051 ] Lars Hofhansl commented on HBASE-8803: -- Could we just instantiate an ExecutorService with N max threads and an unbounded waiting queue, and then submit all move jobs onto that service, and lastly shutdown and awaitTermination? region_mover.rb should move multiple regions at a time -- Key: HBASE-8803 URL: https://issues.apache.org/jira/browse/HBASE-8803 Project: HBase Issue Type: Bug Components: Usability Affects Versions: 0.98.0, 0.94.8, 0.95.1 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Critical Fix For: 0.98.0, 0.95.2, 0.94.12 Attachments: HBASE-8803-v0-trunk.patch, HBASE-8803-v1-0.94.patch, HBASE-8803-v1-trunk.patch, HBASE-8803-v2-0.94.patch, HBASE-8803-v2-0.94.patch, HBASE-8803-v3-0.94.patch, HBASE-8803-v4-0.94.patch Original Estimate: 48h Remaining Estimate: 48h When there is many regions in a cluster, rolling_restart can take hours because region_mover is moving the regions one by one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9129) Scanner prefetching leaks scanners.
Elliott Clark created HBASE-9129: Summary: Scanner prefetching leaks scanners. Key: HBASE-9129 URL: https://issues.apache.org/jira/browse/HBASE-9129 Project: HBase Issue Type: Bug Reporter: Elliott Clark Priority: Blocker Running YCSB workload-e on 0.95 basically hangs the whole cluster. YCSB opens a scanner for 100 rows. # YCSB Calls next # Pre-fetching starts # YCSB Closes scanner # Pre-fetching re-adds the scanner. So the end result is: {noformat} scan-prefetch-2-thread-45 daemon prio=10 tid=0x7f7e406ec800 nid=0x40bc runnable [0x7f75ffefd000] java.lang.Thread.State: RUNNABLE at org.apache.hadoop.hbase.regionserver.Leases$Lease.equals(Leases.java:272) at java.util.PriorityQueue.indexOf(PriorityQueue.java:342) at java.util.PriorityQueue.remove(PriorityQueue.java:360) at java.util.concurrent.DelayQueue.remove(DelayQueue.java:476) at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:232) - locked 0x7f774455a660 (a java.util.concurrent.DelayQueue) at org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:269) at org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:260) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) scan-prefetch-2-thread-44 daemon prio=10 tid=0x7f7e4c1ba800 nid=0x40bb waiting on condition [0x7f7605b79000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7f774455ad78 (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) scan-prefetch-2-thread-43 daemon prio=10 tid=0x7f7e38cbc800 nid=0x40ba waiting on condition [0x7f7609ab8000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7f774455ad78 (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722){noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9129) Scanner prefetching leaks scanners.
[ https://issues.apache.org/jira/browse/HBASE-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730065#comment-13730065 ] Elliott Clark commented on HBASE-9129: -- [~jxiang] Did the backport of 89-fb's patch. Should we consider backing this out and going with only a client side pre-fetching ? Scanner prefetching leaks scanners. --- Key: HBASE-9129 URL: https://issues.apache.org/jira/browse/HBASE-9129 Project: HBase Issue Type: Bug Reporter: Elliott Clark Priority: Blocker Running YCSB workload-e on 0.95 basically hangs the whole cluster. YCSB opens a scanner for 100 rows. # YCSB Calls next # Pre-fetching starts # YCSB Closes scanner # Pre-fetching re-adds the scanner. So the end result is: {noformat} scan-prefetch-2-thread-45 daemon prio=10 tid=0x7f7e406ec800 nid=0x40bc runnable [0x7f75ffefd000] java.lang.Thread.State: RUNNABLE at org.apache.hadoop.hbase.regionserver.Leases$Lease.equals(Leases.java:272) at java.util.PriorityQueue.indexOf(PriorityQueue.java:342) at java.util.PriorityQueue.remove(PriorityQueue.java:360) at java.util.concurrent.DelayQueue.remove(DelayQueue.java:476) at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:232) - locked 0x7f774455a660 (a java.util.concurrent.DelayQueue) at org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:269) at org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:260) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) scan-prefetch-2-thread-44 daemon prio=10 tid=0x7f7e4c1ba800 nid=0x40bb waiting on condition [0x7f7605b79000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7f774455ad78 (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) scan-prefetch-2-thread-43 daemon prio=10 tid=0x7f7e38cbc800 nid=0x40ba waiting on condition [0x7f7609ab8000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7f774455ad78 (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722){noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9130) Invert the name of namespace commands to match the snapshots and replication form
stack created HBASE-9130: Summary: Invert the name of namespace commands to match the snapshots and replication form Key: HBASE-9130 URL: https://issues.apache.org/jira/browse/HBASE-9130 Project: HBase Issue Type: Sub-task Reporter: stack Priority: Critical In our shell, we are inconsistent in the way that we color/group commands. The majority though seem to be COMMAND '-' GROUP as in list_snapshots and list_peers for snapshotting and replication respectively wherease namespaces does 'namespaces_list'. If we could start over, I'd have gone w/ the namespaces format but we are where we are. lets fix this before we release post ns commit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9129) Scanner prefetching leaks scanners.
[ https://issues.apache.org/jira/browse/HBASE-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-9129: - Fix Version/s: 0.95.2 Scanner prefetching leaks scanners. --- Key: HBASE-9129 URL: https://issues.apache.org/jira/browse/HBASE-9129 Project: HBase Issue Type: Bug Reporter: Elliott Clark Priority: Blocker Fix For: 0.95.2 Running YCSB workload-e on 0.95 basically hangs the whole cluster. YCSB opens a scanner for 100 rows. # YCSB Calls next # Pre-fetching starts # YCSB Closes scanner # Pre-fetching re-adds the scanner. So the end result is: {noformat} scan-prefetch-2-thread-45 daemon prio=10 tid=0x7f7e406ec800 nid=0x40bc runnable [0x7f75ffefd000] java.lang.Thread.State: RUNNABLE at org.apache.hadoop.hbase.regionserver.Leases$Lease.equals(Leases.java:272) at java.util.PriorityQueue.indexOf(PriorityQueue.java:342) at java.util.PriorityQueue.remove(PriorityQueue.java:360) at java.util.concurrent.DelayQueue.remove(DelayQueue.java:476) at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:232) - locked 0x7f774455a660 (a java.util.concurrent.DelayQueue) at org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:269) at org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:260) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) scan-prefetch-2-thread-44 daemon prio=10 tid=0x7f7e4c1ba800 nid=0x40bb waiting on condition [0x7f7605b79000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7f774455ad78 (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) scan-prefetch-2-thread-43 daemon prio=10 tid=0x7f7e38cbc800 nid=0x40ba waiting on condition [0x7f7609ab8000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7f774455ad78 (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722){noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8760) possible loss of data in snapshot taken after region split
[ https://issues.apache.org/jira/browse/HBASE-8760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730075#comment-13730075 ] Hadoop QA commented on HBASE-8760: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12596212/HBASE-8760-thz-v2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6608//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6608//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6608//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6608//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6608//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6608//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6608//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6608//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6608//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6608//console This message is automatically generated. possible loss of data in snapshot taken after region split -- Key: HBASE-8760 URL: https://issues.apache.org/jira/browse/HBASE-8760 Project: HBase Issue Type: Bug Components: snapshots Affects Versions: 0.94.8, 0.95.1 Reporter: Jerry He Fix For: 0.98.0, 0.95.2, 0.94.12 Attachments: HBase-8760-0.94.8.patch, HBase-8760-0.94.8-v1.patch, HBASE-8760-thz-v0.patch, HBASE-8760-thz-v1.patch, HBASE-8760-thz-v2.patch Right after a region split but before the daughter regions are compacted, we have two daughter regions containing Reference files to the parent hfiles. If we take snapshot right at the moment, the snapshot will succeed, but it will only contain the daughter Reference files. Since there is no hold on the parent hfiles, they will be deleted by the HFile Cleaner after they are no longer needed by the daughter regions soon after. A minimum we need to do is the keep these parent hfiles from being deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-8718) shell balancer is broke in 0.95
[ https://issues.apache.org/jira/browse/HBASE-8718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-8718. -- Resolution: Cannot Reproduce shell balancer is broke in 0.95 --- Key: HBASE-8718 URL: https://issues.apache.org/jira/browse/HBASE-8718 Project: HBase Issue Type: Bug Components: shell Reporter: stack Fix For: 0.95.2 {code} hbase(main):005:0 balancer ERROR: com.google.protobuf.ServiceException: java.net.SocketTimeoutException: Call to sss-1/10.20.195.21:6 failed because java.net.SocketTimeoutException: 6 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.20.195.21:47282 remote=sss-1/10.20.195.21:6] Here is some help for this command: Trigger the cluster balancer. Returns true if balancer ran and was able to tell the region servers to unassign all the regions to balance (the re-assignment itself is async). Otherwise false (Will not run if regions in transition). {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9128) Drop the '.' prefix from names under hbase.rootdir after namespaces goes in
[ https://issues.apache.org/jira/browse/HBASE-9128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730078#comment-13730078 ] Enis Soztutar commented on HBASE-9128: -- bq. How would migration 'handle' it? The problem was that, when we filter out table directories using HConstants.HBASE_NON_USER_TABLE_DIRS, we might ignore tables named logs, data if we just change the constants. We can easily circumvent this, but hard coding the old reserved names in the migration script, and handle filtering out table directories from ls output for hbase.rootdir differently. Drop the '.' prefix from names under hbase.rootdir after namespaces goes in --- Key: HBASE-9128 URL: https://issues.apache.org/jira/browse/HBASE-9128 Project: HBase Issue Type: Sub-task Reporter: stack Priority: Critical Change it so instead of hbase.rootdir looking like this: {code} drwxr-xr-x 5 stack staff 170 Aug 5 14:43 .data -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.id.crc -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.version.crc drwxr-xr-x 3 stack staff 102 Aug 5 14:43 .logs drwxr-xr-x 2 stack staff 68 Aug 5 14:43 .oldlogs drwxr-xr-x 3 stack staff 102 Aug 5 14:43 .tmp -rwxrwxrwx 1 stack staff 42 Aug 5 14:43 hbase.id -rwxrwxrwx 1 stack staff7 Aug 5 14:43 hbase.version {code} It looks like this: {code} drwxr-xr-x 5 stack staff 170 Aug 5 14:43 data -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.id.crc -rw-r--r-- 1 stack staff 12 Aug 5 14:43 .hbase.version.crc drwxr-xr-x 3 stack staff 102 Aug 5 14:43 logs drwxr-xr-x 2 stack staff 68 Aug 5 14:43 oldlogs drwxr-xr-x 3 stack staff 102 Aug 5 14:43 tmp -rwxrwxrwx 1 stack staff 42 Aug 5 14:43 hbase.id -rwxrwxrwx 1 stack staff7 Aug 5 14:43 hbase.version {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8496) Implement tags and the internals of how a tag should look like
[ https://issues.apache.org/jira/browse/HBASE-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730092#comment-13730092 ] stack commented on HBASE-8496: -- On the design doc: + A whole byte to keep the type when the number of types will be small seems profligate in our base type? + Two bytes of length ditto. You could save 50% putting type and length together in a short? Or is the tag length, the overall tags length? What else changed in the design doc? Implement tags and the internals of how a tag should look like -- Key: HBASE-8496 URL: https://issues.apache.org/jira/browse/HBASE-8496 Project: HBase Issue Type: New Feature Affects Versions: 0.98.0, 0.95.2 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical Attachments: Comparison.pdf, HBASE-8496_2.patch, HBASE-8496.patch, Tag design.pdf, Tag design_updated.pdf, Tag_In_KV_Buffer_For_reference.patch The intent of this JIRA comes from HBASE-7897. This would help us to decide on the structure and format of how the tags should look like. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9129) Scanner prefetching leaks scanners.
[ https://issues.apache.org/jira/browse/HBASE-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730096#comment-13730096 ] Jimmy Xiang commented on HBASE-9129: It's backported from 89-fb, but with some change in the server side since the code base is different. It's ok with me to back it out since the performance improvement is not very big. Scanner prefetching leaks scanners. --- Key: HBASE-9129 URL: https://issues.apache.org/jira/browse/HBASE-9129 Project: HBase Issue Type: Bug Reporter: Elliott Clark Priority: Blocker Fix For: 0.95.2 Running YCSB workload-e on 0.95 basically hangs the whole cluster. YCSB opens a scanner for 100 rows. # YCSB Calls next # Pre-fetching starts # YCSB Closes scanner # Pre-fetching re-adds the scanner. So the end result is: {noformat} scan-prefetch-2-thread-45 daemon prio=10 tid=0x7f7e406ec800 nid=0x40bc runnable [0x7f75ffefd000] java.lang.Thread.State: RUNNABLE at org.apache.hadoop.hbase.regionserver.Leases$Lease.equals(Leases.java:272) at java.util.PriorityQueue.indexOf(PriorityQueue.java:342) at java.util.PriorityQueue.remove(PriorityQueue.java:360) at java.util.concurrent.DelayQueue.remove(DelayQueue.java:476) at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:232) - locked 0x7f774455a660 (a java.util.concurrent.DelayQueue) at org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:269) at org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:260) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) scan-prefetch-2-thread-44 daemon prio=10 tid=0x7f7e4c1ba800 nid=0x40bb waiting on condition [0x7f7605b79000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7f774455ad78 (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) scan-prefetch-2-thread-43 daemon prio=10 tid=0x7f7e38cbc800 nid=0x40ba waiting on condition [0x7f7609ab8000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7f774455ad78 (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722){noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9095) AssignmentManager's handleRegion should respect the single threaded nature of the processing
[ https://issues.apache.org/jira/browse/HBASE-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-9095: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to 0.95 and trunk. AssignmentManager's handleRegion should respect the single threaded nature of the processing Key: HBASE-9095 URL: https://issues.apache.org/jira/browse/HBASE-9095 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.95.2 Attachments: 9095-1 (1).txt, 9095-1.txt, 9095-1.txt, 9095-1.txt, 9095-2.txt While debugging a case where a region was getting opened on a RegionServer and then closed soon after (and then never re-opened anywhere thereafter), it seemed like the processing in handleRegion to do with deletion of ZK nodes should be non-asynchronous. This achieves two things: 1. The synchronous deletion prevents more than one processing on the same event data twice. Assuming that we do get more than one notification (on let's say, region OPENED event), the subsequent processing(s) in handleRegion for the same znode would end up with a zookeeper node not found exception. The return value of the data read would be null and that's already handled. If it is asynchronous, it leads to issues like - master opens a region on a certain RegionServer and soon after it sends that RegionServer a close for the same region, and then the znode is deleted. 2. The deletion is currently handled in an executor service. This is problematic since by design the events for a given region should be processed in order. By delegating a part of the processing to executor service we are somewhat violating this contract since there is no guarantee of the ordering in the executor service executions... Thanks to [~jeffreyz] and [~enis] for the discussions on this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication
[ https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730116#comment-13730116 ] stack commented on HBASE-7709: -- What is to be done on this for 0.95.2? Infinite loop possible in Master/Master replication --- Key: HBASE-7709 URL: https://issues.apache.org/jira/browse/HBASE-7709 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.94.6, 0.95.1 Reporter: Lars Hofhansl Fix For: 0.98.0, 0.95.2, 0.94.12 We just discovered the following scenario: # Cluster A and B are setup in master/master replication # By accident we had Cluster C replicate to Cluster A. Now all edit originating from C will be bouncing between A and B. Forever! The reason is that when the edit come in from C the cluster ID is already set and won't be reset. We have a couple of options here: # Optionally only support master/master (not cycles of more than two clusters). In that case we can always reset the cluster ID in the ReplicationSource. That means that now cycles 2 will have the data cycle forever. This is the only option that requires no changes in the HLog format. # Instead of a single cluster id per edit maintain a (unordered) set of cluster id that have seen this edit. Then in ReplicationSource we drop any edit that the sink has seen already. The is the cleanest approach, but it might need a lot of data stored per edit if there are many clusters involved. # Maintain a configurable counter of the maximum cycle side we want to support. Could default to 10 (even maybe even just). Store a hop-count in the WAL and the ReplicationSource increases that hop-count on each hop. If we're over the max, just drop the edit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9131) Add admin-level documention about configuration and usage of the Bucket Cache
Jonathan Hsieh created HBASE-9131: - Summary: Add admin-level documention about configuration and usage of the Bucket Cache Key: HBASE-9131 URL: https://issues.apache.org/jira/browse/HBASE-9131 Project: HBase Issue Type: Bug Reporter: Jonathan Hsieh HBASE-7404 added the bucket cache but its configuration settings are currently undocumented. Without documentation developers would be the only ones aware of the feature. Specifically documentation about slide 23 from http://www.slideshare.net/cloudera/operations-session-4 would be great to add! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7404) Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
[ https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730126#comment-13730126 ] Jonathan Hsieh commented on HBASE-7404: --- I've filed HBASE-9131 -- it would be great if someone who worked on this code added documentation at an admin-level to the ref guide. Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE -- Key: HBASE-7404 URL: https://issues.apache.org/jira/browse/HBASE-7404 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.95.0 Attachments: 7404-trunk-v10.patch, 7404-trunk-v11.patch, 7404-trunk-v12.patch, 7404-trunk-v13.patch, 7404-trunk-v13.txt, 7404-trunk-v14.patch, BucketCache.pdf, hbase-7404-94v2.patch, HBASE-7404-backport-0.94.patch, hbase-7404-trunkv2.patch, hbase-7404-trunkv9.patch, Introduction of Bucket Cache.pdf First, thanks @neil from Fusion-IO share the source code. Usage: 1.Use bucket cache as main memory cache, configured as the following: –hbase.bucketcache.ioengine heap –hbase.bucketcache.size 0.4 (size for bucket cache, 0.4 is a percentage of max heap size) 2.Use bucket cache as a secondary cache, configured as the following: –hbase.bucketcache.ioengine file:/disk1/hbase/cache.data(The file path where to store the block data) –hbase.bucketcache.size 1024 (size for bucket cache, unit is MB, so 1024 means 1GB) –hbase.bucketcache.combinedcache.enabled false (default value being true) See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and org.apache.hadoop.hbase.io.hfile.bucket.BucketCache What's Bucket Cache? It could greatly decrease CMS and heap fragment by GC It support a large cache space for High Read Performance by using high speed disk like Fusion-io 1.An implementation of block cache like LruBlockCache 2.Self manage blocks' storage position through Bucket Allocator 3.The cached blocks could be stored in the memory or file system 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC. 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space How about SlabCache? We have studied and test SlabCache first, but the result is bad, because: 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better 3.Direct heap performance is not good as heap, and maybe cause OOM, so we recommend using heap engine See more in the attachment and in the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira