[jira] [Commented] (HBASE-9093) Hbase client API: Restore the writeToWal method
[ https://issues.apache.org/jira/browse/HBASE-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724914#comment-13724914 ] Lars Hofhansl commented on HBASE-9093: -- Wait. 0.94 does support the new API as well. By this logic we cannot have any API changes not even in the supposed singularity. Hbase client API: Restore the writeToWal method --- Key: HBASE-9093 URL: https://issues.apache.org/jira/browse/HBASE-9093 Project: HBase Issue Type: Bug Components: Client, Usability Affects Versions: 0.95.0 Reporter: Hari Shreedharan Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9093.patch, HBASE-9093.patch, HBASE-9093.patch The writeToWal is used by downstream projects like Flume to disable writes to WAL to improve performance when durability is not strictly required. But renaming this method to setDurability forces us to use reflection to support hbase versions 95 - which in turn hits performance, as this method needs to be called on every single write. I recommend adding the old method back as deprecated and removing it once hbase-95/96 becomes the popular version used in prod. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8960) TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes
[ https://issues.apache.org/jira/browse/HBASE-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724923#comment-13724923 ] Hudson commented on HBASE-8960: --- SUCCESS: Integrated in hbase-0.95 #387 (See [https://builds.apache.org/job/hbase-0.95/387/]) HBASE-8960: TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes - Addendum2 (jeffreyz: rev 1508712) * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes -- Key: HBASE-8960 URL: https://issues.apache.org/jira/browse/HBASE-8960 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jeffrey Zhong Priority: Minor Fix For: 0.95.2 Attachments: hbase-8960-addendum-2.patch, hbase-8960-addendum.patch, hbase-8960.patch http://54.241.6.143/job/HBase-0.95-Hadoop-2/org.apache.hbase$hbase-server/634/testReport/junit/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testLogReplayForDisablingTable/ {noformat} java.lang.AssertionError: expected:1000 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testLogReplayForDisablingTable(TestDistributedLogSplitting.java:797) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-9093) Hbase client API: Restore the writeToWal method
[ https://issues.apache.org/jira/browse/HBASE-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reopened HBASE-9093: -- Changes? Are we not just restoring old APIs? Hbase client API: Restore the writeToWal method --- Key: HBASE-9093 URL: https://issues.apache.org/jira/browse/HBASE-9093 Project: HBase Issue Type: Bug Components: Client, Usability Affects Versions: 0.95.0 Reporter: Hari Shreedharan Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9093.patch, HBASE-9093.patch, HBASE-9093.patch The writeToWal is used by downstream projects like Flume to disable writes to WAL to improve performance when durability is not strictly required. But renaming this method to setDurability forces us to use reflection to support hbase versions 95 - which in turn hits performance, as this method needs to be called on every single write. I recommend adding the old method back as deprecated and removing it once hbase-95/96 becomes the popular version used in prod. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9093) Hbase client API: Restore the writeToWal method
[ https://issues.apache.org/jira/browse/HBASE-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724933#comment-13724933 ] Lars Hofhansl commented on HBASE-9093: -- I meant we cannot make forward changes. Guess my point was that 0.94 does support the new API, and we should use the singularity to let us make API incompatible changes. So the issue in flume would be that the *older* 0.94 (and 0.92) versions cannot be supported without reflection. If that is major hassle, by all means lets keep the old API around, we just have to weigh that against generally improving the API. Hbase client API: Restore the writeToWal method --- Key: HBASE-9093 URL: https://issues.apache.org/jira/browse/HBASE-9093 Project: HBase Issue Type: Bug Components: Client, Usability Affects Versions: 0.95.0 Reporter: Hari Shreedharan Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9093.patch, HBASE-9093.patch, HBASE-9093.patch The writeToWal is used by downstream projects like Flume to disable writes to WAL to improve performance when durability is not strictly required. But renaming this method to setDurability forces us to use reflection to support hbase versions 95 - which in turn hits performance, as this method needs to be called on every single write. I recommend adding the old method back as deprecated and removing it once hbase-95/96 becomes the popular version used in prod. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9084) HBase admin flush has a data loss risk even after HBASE-7671
[ https://issues.apache.org/jira/browse/HBASE-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724934#comment-13724934 ] chunhui shen commented on HBASE-9084: - Seems have such a risk. What about using the following fix: {code} try { boolean result = internalFlushcache(status); if (coprocessorHost != null) { status.setStatus(Running post-flush coprocessor hooks); coprocessorHost.postFlush(); } status.markComplete(Flush successful); return result; }catch(DroppedSnapshotException ex){ if(rsServices!=null){ rsServices.abort(Replay of HLog required. Forcing server shutdown, ex); } return false; } {code} I think the above code could fix the hole HBase admin flush has a data loss risk even after HBASE-7671 Key: HBASE-9084 URL: https://issues.apache.org/jira/browse/HBASE-9084 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.1, 0.94.10 Reporter: Liang Xie Assignee: Liang Xie Priority: Critical Attachments: HBASE-9084-0.94.txt, HBASE-9084.txt see https://issues.apache.org/jira/browse/HBASE-7671?focusedCommentId=13722148page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13722148 will attach a simple patch soon -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9084) HBase admin flush has a data loss risk even after HBASE-7671
[ https://issues.apache.org/jira/browse/HBASE-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724940#comment-13724940 ] chunhui shen commented on HBASE-9084: - It means move the call abort code to HRegion, thus due to this.rsServices != null this.rsServices.isAborted() is not be set should be impossible. Correct me if something I miss HBase admin flush has a data loss risk even after HBASE-7671 Key: HBASE-9084 URL: https://issues.apache.org/jira/browse/HBASE-9084 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.1, 0.94.10 Reporter: Liang Xie Assignee: Liang Xie Priority: Critical Attachments: HBASE-9084-0.94.txt, HBASE-9084.txt see https://issues.apache.org/jira/browse/HBASE-7671?focusedCommentId=13722148page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13722148 will attach a simple patch soon -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7266) [89-fb] Using pread for non-compaction read request
[ https://issues.apache.org/jira/browse/HBASE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724944#comment-13724944 ] Chao Shi commented on HBASE-7266: - We are experiencing this too. This happens when a large number of concurrent scan requests on a single region. Because of the lock (on seek+read), only one handler thread is effectively working. Our scan range is very small (just a few adjacent rows, fits into 1 block). [89-fb] Using pread for non-compaction read request --- Key: HBASE-7266 URL: https://issues.apache.org/jira/browse/HBASE-7266 Project: HBase Issue Type: Improvement Reporter: Liyin Tang There are 2 kinds of read operations in HBase: pread and seek+read. Pread, positional read, is stateless and create a new connection between the DFSClient and DataNode for each operation. While seek+read is to seek to a specific postion and prefetch blocks from data nodes. The benefit of seek+read is that it will cache the prefetch result but the downside is it is stateful and needs to synchronized. So far, both compaction and scan are using seek+read, which caused some resource contention. So using the pread for the scan request can avoid the resource contention. In addition, the region server is able to do the prefetch for the scan request (HBASE-6874) so that it won't be necessary to let the DFSClient to prefetch the data any more. I will run through the scan benchmark (with no block cache) with verify the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9084) HBase admin flush has a data loss risk even after HBASE-7671
[ https://issues.apache.org/jira/browse/HBASE-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724945#comment-13724945 ] Liang Xie commented on HBASE-9084: -- emm, imho, it doesn't work. data loss means the second flush(issued by hbaseadmin flush side) was successful, but didn't flush its own memstore snapshot, just flushed the previous failed old snapshot, then the second flush's data was lost silently... so i thought if just add this catch statement is not enough. or maybe i didn't get u. HBase admin flush has a data loss risk even after HBASE-7671 Key: HBASE-9084 URL: https://issues.apache.org/jira/browse/HBASE-9084 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.1, 0.94.10 Reporter: Liang Xie Assignee: Liang Xie Priority: Critical Attachments: HBASE-9084-0.94.txt, HBASE-9084.txt see https://issues.apache.org/jira/browse/HBASE-7671?focusedCommentId=13722148page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13722148 will attach a simple patch soon -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8960) TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes
[ https://issues.apache.org/jira/browse/HBASE-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724948#comment-13724948 ] Hudson commented on HBASE-8960: --- SUCCESS: Integrated in HBase-TRUNK #4323 (See [https://builds.apache.org/job/HBase-TRUNK/4323/]) HBASE-8960: TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes - Addendum2 (jeffreyz: rev 1508711) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes -- Key: HBASE-8960 URL: https://issues.apache.org/jira/browse/HBASE-8960 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jeffrey Zhong Priority: Minor Fix For: 0.95.2 Attachments: hbase-8960-addendum-2.patch, hbase-8960-addendum.patch, hbase-8960.patch http://54.241.6.143/job/HBase-0.95-Hadoop-2/org.apache.hbase$hbase-server/634/testReport/junit/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testLogReplayForDisablingTable/ {noformat} java.lang.AssertionError: expected:1000 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testLogReplayForDisablingTable(TestDistributedLogSplitting.java:797) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9084) HBase admin flush has a data loss risk even after HBASE-7671
[ https://issues.apache.org/jira/browse/HBASE-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724949#comment-13724949 ] chunhui shen commented on HBASE-9084: - bq.the second flush(issued by hbaseadmin flush side) was successful It won't be successful, since we have aborted the server HBase admin flush has a data loss risk even after HBASE-7671 Key: HBASE-9084 URL: https://issues.apache.org/jira/browse/HBASE-9084 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.1, 0.94.10 Reporter: Liang Xie Assignee: Liang Xie Priority: Critical Attachments: HBASE-9084-0.94.txt, HBASE-9084.txt see https://issues.apache.org/jira/browse/HBASE-7671?focusedCommentId=13722148page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13722148 will attach a simple patch soon -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9084) HBase admin flush has a data loss risk even after HBASE-7671
[ https://issues.apache.org/jira/browse/HBASE-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724953#comment-13724953 ] chunhui shen commented on HBASE-9084: - the second flush(issued by hbaseadmin flush side) won't work by {code} if (this.rsServices != null this.rsServices.isAborted()) { // Don't flush when server aborting, it's unsafe throw new IOException(Aborting flush because server is abortted...); } {code} or {code} synchronized (writestate) { if (!writestate.flushing writestate.writesEnabled) { this.writestate.flushing = true; } else { if (LOG.isDebugEnabled()) { LOG.debug(NOT flushing memstore for region + this + , flushing= + writestate.flushing + , writesEnabled= + writestate.writesEnabled); } status.abort(Not flushing since + (writestate.flushing ? already flushing : writes not enabled)); return false; } } {code} HBase admin flush has a data loss risk even after HBASE-7671 Key: HBASE-9084 URL: https://issues.apache.org/jira/browse/HBASE-9084 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.1, 0.94.10 Reporter: Liang Xie Assignee: Liang Xie Priority: Critical Attachments: HBASE-9084-0.94.txt, HBASE-9084.txt see https://issues.apache.org/jira/browse/HBASE-7671?focusedCommentId=13722148page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13722148 will attach a simple patch soon -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9093) Hbase client API: Restore the writeToWal method
[ https://issues.apache.org/jira/browse/HBASE-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724952#comment-13724952 ] Hari Shreedharan commented on HBASE-9093: - [~lhofhansl] - Thanks for your feedback. I am not against removing/upgrading/updating APIs. The only thing I am requesting is to deprecate the old API and keep them around long enough for downstream projects/users to upgrade to the versions of HBase where these APIs are removed. In this case, if setWriteToWal method was deprecated in 0.94 and then removed in 0.96, we would have made every effort to make sure we use the new setDurability API by the time HBase 0.96 is released (hopefully by then, most users would be using at least 0.94 where both APIs would have been available). From the javadoc, it does not look like hbase 0.94 supported the setDurability API though. (http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/Put.html - is there a newer one?) Hbase client API: Restore the writeToWal method --- Key: HBASE-9093 URL: https://issues.apache.org/jira/browse/HBASE-9093 Project: HBase Issue Type: Bug Components: Client, Usability Affects Versions: 0.95.0 Reporter: Hari Shreedharan Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9093.patch, HBASE-9093.patch, HBASE-9093.patch The writeToWal is used by downstream projects like Flume to disable writes to WAL to improve performance when durability is not strictly required. But renaming this method to setDurability forces us to use reflection to support hbase versions 95 - which in turn hits performance, as this method needs to be called on every single write. I recommend adding the old method back as deprecated and removing it once hbase-95/96 becomes the popular version used in prod. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8488) HBase transitive dependencies not being pulled in when building apps like Flume which depend on HBase
[ https://issues.apache.org/jira/browse/HBASE-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725014#comment-13725014 ] stack commented on HBASE-8488: -- Here is pointer to downstream project up on github https://github.com/saintstack/hbase-downstreamer HBase transitive dependencies not being pulled in when building apps like Flume which depend on HBase - Key: HBASE-8488 URL: https://issues.apache.org/jira/browse/HBASE-8488 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.95.0 Reporter: Roshan Naik Assignee: stack Priority: Blocker Fix For: 0.98.0, 0.95.2 Attachments: client.tgz Here is a snippet of the errors seen when building against Hbase {code} [WARNING] Invalid POM for org.apache.hbase:hbase-common:jar:0.97.0-SNAPSHOT, transitive dependencies (if any) will not be available, enable debug logging for more details: Some problems were encountered while processing the POMs: [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for org.apache.hbase:${compat.module}:jar with value '${compat.module}' does not match a valid id pattern. @ org.apache.hbase:hbase:0.97.0-SNAPSHOT, /Users/rnaik/.m2/repository/org/apache/hbase/hbase/0.97.0-SNAPSHOT/hbase-0.97.0-SNAPSHOT.pom, line 982, column 21 [ERROR] 'dependencyManagement.dependencies.dependency.artifactId' for org.apache.hbase:${compat.module}:test-jar with value '${compat.module}' does not match a valid id pattern. @ org.apache.hbase:hbase:0.97.0-SNAPSHOT, /Users/rnaik/.m2/repository/org/apache/hbase/hbase/0.97.0-SNAPSHOT/hbase-0.97.0-SNAPSHOT.pom, line 987, column 21 {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7634) Replication handling of changes to peer clusters is inefficient
[ https://issues.apache.org/jira/browse/HBASE-7634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabriel Reid updated HBASE-7634: Attachment: HBASE-7634.v5.patch Updated patch based on reviewboard comments from J-D No real functional changes (other than increased configurability). Most changes are formatting and whitespace. Replication handling of changes to peer clusters is inefficient --- Key: HBASE-7634 URL: https://issues.apache.org/jira/browse/HBASE-7634 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.95.2 Reporter: Gabriel Reid Attachments: HBASE-7634.patch, HBASE-7634.v2.patch, HBASE-7634.v3.patch, HBASE-7634.v4.patch, HBASE-7634.v5.patch The current handling of changes to the region servers in a replication peer cluster is currently quite inefficient. The list of region servers that are being replicated to is only updated if there are a large number of issues encountered while replicating. This can cause it to take quite a while to recognize that a number of the regionserver in a peer cluster are no longer available. A potentially bigger problem is that if a replication peer cluster is started with a small number of regionservers, and then more region servers are added after replication has started, the additional region servers will never be used for replication (unless there are failures on the in-use regionservers). Part of the current issue is that the retry code in ReplicationSource#shipEdits checks a randomly-chosen replication peer regionserver (in ReplicationSource#isSlaveDown) to see if it is up after a replication write has failed on a different randonly-chosen replication peer. If the peer is seen as not down, another randomly-chosen peer is used for writing. A second part of the issue is that changes to the list of region servers in a peer cluster are not detected at all, and are only picked up if a certain number of failures have occurred when trying to ship edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8224) Add '-hadoop1' or '-hadoop2' to our version string
[ https://issues.apache.org/jira/browse/HBASE-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725026#comment-13725026 ] stack commented on HBASE-8224: -- So our poms in repo, even if local repo, include the compat.module variable. It is not being interpolated. Here is a build of hbase against hadoop2 profile: {code} durruti:hbase stack$ grep -r 'compat.module' . ./hbase/0.95.2-SNAPSHOT/hbase-0.95.2-SNAPSHOT.pom: artifactId${compat.module}/artifactId ./hbase/0.95.2-SNAPSHOT/hbase-0.95.2-SNAPSHOT.pom: artifactId${compat.module}/artifactId ./hbase/0.95.2-SNAPSHOT/hbase-0.95.2-SNAPSHOT.pom: compat.modulehbase-hadoop1-compat/compat.module ./hbase/0.95.2-SNAPSHOT/hbase-0.95.2-SNAPSHOT.pom:!-- Need to set this for the Hadoop 1 compat module -- ./hbase/0.95.2-SNAPSHOT/hbase-0.95.2-SNAPSHOT.pom: compat.modulehbase-hadoop1-compat/compat.module ./hbase/0.95.2-SNAPSHOT/hbase-0.95.2-SNAPSHOT.pom: compat.modulehbase-hadoop2-compat/compat.module ./hbase-assembly/0.95.2-SNAPSHOT/hbase-assembly-0.95.2-SNAPSHOT.pom: artifactId${compat.module}/artifactId ./hbase-assembly/0.95.2-SNAPSHOT/hbase-assembly-0.95.2-SNAPSHOT.pom: artifactId${compat.module}/artifactId ./hbase-it/0.95.2-SNAPSHOT/hbase-it-0.95.2-SNAPSHOT.pom: artifactId${compat.module}/artifactId ./hbase-it/0.95.2-SNAPSHOT/hbase-it-0.95.2-SNAPSHOT.pom: artifactId${compat.module}/artifactId ./hbase-prefix-tree/0.95.2-SNAPSHOT/hbase-prefix-tree-0.95.2-SNAPSHOT.pom: artifactId${compat.module}/artifactId ./hbase-server/0.95.2-SNAPSHOT/hbase-server-0.95.2-SNAPSHOT.pom: artifactId${compat.module}/artifactId ./hbase-server/0.95.2-SNAPSHOT/hbase-server-0.95.2-SNAPSHOT.pom: artifactId${compat.module}/artifactId {code} I've been playing w/ replacing the above variable and then building w/ new poms. Here is my little script: {code} $ hadoop=hadoop2; for i in `find . -name pom.xml`; do echo $i; fn=pom.xml.${hadoop}; sed s/\${compat.module}/hbase-${hadoop}-compat/;s/relativePath\.\./..\/relativePatch$fn/;s/\(module[^]*\)/\1\/$fn/ $i $i.${hadoop}; done {code} This puts a file beside the original pom.xml named pom.xml.hadoop2. This file has a couple of substitutions done on original pom (I tried to put it into target dir so I did not mod the original srcs but mvn wants the modules under it -- and then child modules want to ref the parent -- I suppose I could make this work w/ more hackery). Now building I do this: mvn clean install -DskipTests -Dhadoop.profile=2.0 -f pom.xml.hadoop2 And repo seems better. My downstream project is failing still though it now no longer has hadoop1 on refernces or fail because it wants hadoop1 dependency. It is failing here: {code} 4 --- 3 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.67 sec FAILURE! 2 testSpinUpMiniHBaseCluster(org.hbase.downstreamer.TestHBaseMiniCluster) Time elapsed: 0.631 sec ERROR! 1 java.lang.UnsupportedOperationException: Not implemented by the DistributedFileSystem FileSystem implementation ... {code} ... which uusually means I was compiled against wrong version Looking. Add '-hadoop1' or '-hadoop2' to our version string -- Key: HBASE-8224 URL: https://issues.apache.org/jira/browse/HBASE-8224 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.95.2 Attachments: 8224-adding.classifiers.txt, hbase-8224-proto1.patch So we can publish both the hadoop1 and the hadoop2 jars to a maven repository, and so we can publish two packages, one for hadoop1 and one for hadoop2, given how maven works, our only alternative (to the best of my knowledge and after consulting others) is by amending the version string to include hadoop1 or hadoop2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9094) Rationalize dependencies; fix analysis complaints about used but non-declared dependencies
[ https://issues.apache.org/jira/browse/HBASE-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-9094: - Attachment: dep3.txt Some more fixup that comes of building against hadoop2. There are others still remaining -- stuff we reference but we do not declare it in poms. Will try again to fix these but meantime let this run against hadoopqa. Rationalize dependencies; fix analysis complaints about used but non-declared dependencies -- Key: HBASE-9094 URL: https://issues.apache.org/jira/browse/HBASE-9094 Project: HBase Issue Type: Sub-task Components: build Reporter: stack Assignee: stack Fix For: 0.98.0, 0.95.2 Attachments: dep2.txt, dep3.txt, dep.txt Do a cleanup pass through our dependency set so downstreamers get good picture of what they need to include looking at pom. Poking with dependency plugin, found the following issues: + hbase-common is missing listing of a bunch of commons libs + Some of our classes make use of slf4j for no good reason. zk, thrift, netty, and yammer, use slf4j but no need for us to be in the slf4j managing game... so I undid our use of it and stop its transitive include everywhere. + hbase-examples and hbase-it do not declare includes like hbase-client, hbase-protocol, etc. + hbase-hadoop1-compat depended on log4j but doesn't use it. + hbase-prefix-tree depends on hadoop but does not declare it (uses WritiableUtil and RawComparator -- just two imports... uses the Audience annotations which we could just remove but still these two critical includes) + hbase-server wasn't declaring its use of commons-*. Also added excludes of transitive include of log4j by zk and yammer. Got rid of slf4j as a dependency. + Add declarations for used libs such as httpclient and commons-math to top-level pom. + Removed setting versions on hadoop1 and hadoop2 profile for slf4j mess. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8768) Improve bulk load performance by moving key value construction from map phase to reduce phase.
[ https://issues.apache.org/jira/browse/HBASE-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajeshbabu updated HBASE-8768: -- Attachment: HBASE-8768_v4.patch Patch addressing Ted's comments. [~yuzhih...@gmail.com] Thanks for review. bq. Why is a new map created in each iteration of the loop The next set will be created only after reaching configured thresold. Its not for each iteration. Handled all other comments in the current patch. Improve bulk load performance by moving key value construction from map phase to reduce phase. -- Key: HBASE-8768 URL: https://issues.apache.org/jira/browse/HBASE-8768 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.98.0, 0.95.2 Attachments: HBASE-8768_v2.patch, HBASE-8768_v3.patch, HBASE-8768_v4.patch, HBase_Bulkload_Performance_Improvement.pdf ImportTSV bulkloading approach uses MapReduce framework. Existing mapper and reducer classes used by ImportTSV are TsvImporterMapper.java and PutSortReducer.java. ImportTSV tool parses the tab(by default) seperated values from the input files and Mapper class generates the PUT objects for each row using the Key value pairs created from the parsed text. PutSortReducer then uses the partions based on the regions and sorts the Put objects for each region. Overheads we can see in the above approach: == 1) keyvalue construction for each parsed value in the line adding extra data like rowkey,columnfamily,qualifier which will increase around 5x extra data to be shuffled in reduce phase. We can calculate data size to shuffled as below {code} Data to be shuffled = nl*nt*(rl+cfl+cql+vall+tsl+30) {code} If we move keyvalue construction to reduce phase we datasize to be shuffle will be which is very less compared to above. {code} Data to be shuffled = nl*nt*vall {code} nl - Number of lines in the raw file nt - Number of tabs or columns including row key. rl - row length which will be different for each line. cfl - column family length which will be different for each family cql - qualifier length tsl - timestamp length. vall- each parsed value length. 30 bytes for kv size,number of families etc. 2) In mapper side we are creating put objects by adding all keyvalues constructed for each line and in reducer we will again collect keyvalues from put and sort them. Instead we can directly create and sort keyvalues in reducer. Solution: We can improve bulk load performance by moving the key value construction from mapper to reducer so that Mapper just sends the raw text for each row to the Reducer. Reducer then parses the records for rows and create and sort the key value pairs before writing to HFiles. Conclusion: === The above suggestions will improve map phase performance by avoiding keyvalue construction and reduce phase performance by avoiding excess data to be shuffled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8548) postOpen hook called twice
[ https://issues.apache.org/jira/browse/HBASE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-8548: --- Attachment: 8548.v1.patch postOpen hook called twice -- Key: HBASE-8548 URL: https://issues.apache.org/jira/browse/HBASE-8548 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.0 Environment: CentOS Reporter: Roger Ruiz-Carrillo Attachments: 8548.v1.patch, HRegion_HBASE-8548-0.95.patch postOpen hook is called twice when a region is initializing: Once at the end of the body of the initializeRegionInternals() method of the HRegion class. Once at the end initializeRegionStores() method of the HRegion class; initializeRegionStores() is called inside initializeRegionInternals() and as such causes the postOpen hook to be called twice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8548) postOpen hook called twice
[ https://issues.apache.org/jira/browse/HBASE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725054#comment-13725054 ] Nicolas Liochon commented on HBASE-8548: I made an error and committed the fix for this bug while committing HBASE-6295. I don't know why the tests didn't pass in this jira. So the patch here only contains a test. I've checked that the test does not pass with the 0.95 version from the 20th of June (git 4bfe077). I would like to commit it on 0.95 and trunk. postOpen hook called twice -- Key: HBASE-8548 URL: https://issues.apache.org/jira/browse/HBASE-8548 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.0 Environment: CentOS Reporter: Roger Ruiz-Carrillo Attachments: 8548.v1.patch, HRegion_HBASE-8548-0.95.patch postOpen hook is called twice when a region is initializing: Once at the end of the body of the initializeRegionInternals() method of the HRegion class. Once at the end initializeRegionStores() method of the HRegion class; initializeRegionStores() is called inside initializeRegionInternals() and as such causes the postOpen hook to be called twice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8548) postOpen hook called twice
[ https://issues.apache.org/jira/browse/HBASE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-8548: --- Assignee: Nicolas Liochon Release Note: (was: Affects only 0.95 (0.97 seems ok)) Status: Patch Available (was: Open) postOpen hook called twice -- Key: HBASE-8548 URL: https://issues.apache.org/jira/browse/HBASE-8548 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.0 Environment: CentOS Reporter: Roger Ruiz-Carrillo Assignee: Nicolas Liochon Attachments: 8548.v1.patch, HRegion_HBASE-8548-0.95.patch postOpen hook is called twice when a region is initializing: Once at the end of the body of the initializeRegionInternals() method of the HRegion class. Once at the end initializeRegionStores() method of the HRegion class; initializeRegionStores() is called inside initializeRegionInternals() and as such causes the postOpen hook to be called twice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9101) Addendum to pluggable RpcScheduler
Chao Shi created HBASE-9101: --- Summary: Addendum to pluggable RpcScheduler Key: HBASE-9101 URL: https://issues.apache.org/jira/browse/HBASE-9101 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi This patch fixes the review comments from [~stack] and a small fix: - Make RpcScheduler fully pluggable. One can write its own implementation and add it to classpath and specify it by config hbase.region.server.rpc.scheduler.factory.class. - Add unit tests and fix that RpcScheduler.stop is not called (discovered by tests) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9101) Addendum to pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-9101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Shi updated HBASE-9101: Attachment: hbase-9101.patch try hudson Addendum to pluggable RpcScheduler -- Key: HBASE-9101 URL: https://issues.apache.org/jira/browse/HBASE-9101 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Attachments: hbase-9101.patch This patch fixes the review comments from [~stack] and a small fix: - Make RpcScheduler fully pluggable. One can write its own implementation and add it to classpath and specify it by config hbase.region.server.rpc.scheduler.factory.class. - Add unit tests and fix that RpcScheduler.stop is not called (discovered by tests) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9101) Addendum to pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-9101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Shi updated HBASE-9101: Status: Patch Available (was: Open) Addendum to pluggable RpcScheduler -- Key: HBASE-9101 URL: https://issues.apache.org/jira/browse/HBASE-9101 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Attachments: hbase-9101.patch This patch fixes the review comments from [~stack] and a small fix: - Make RpcScheduler fully pluggable. One can write its own implementation and add it to classpath and specify it by config hbase.region.server.rpc.scheduler.factory.class. - Add unit tests and fix that RpcScheduler.stop is not called (discovered by tests) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9101) Addendum to pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-9101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725072#comment-13725072 ] Chao Shi commented on HBASE-9101: - https://reviews.apache.org/r/13106/ Addendum to pluggable RpcScheduler -- Key: HBASE-9101 URL: https://issues.apache.org/jira/browse/HBASE-9101 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Attachments: hbase-9101.patch This patch fixes the review comments from [~stack] and a small fix: - Make RpcScheduler fully pluggable. One can write its own implementation and add it to classpath and specify it by config hbase.region.server.rpc.scheduler.factory.class. - Add unit tests and fix that RpcScheduler.stop is not called (discovered by tests) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8884) Pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725073#comment-13725073 ] Chao Shi commented on HBASE-8884: - Hi stack, I opened another issue (HBASE-9101) and posted a patch to fix the issue you mentioned. Please have a look. Thanks! Pluggable RpcScheduler -- Key: HBASE-8884 URL: https://issues.apache.org/jira/browse/HBASE-8884 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-8884.patch, hbase-8884-v2.patch, hbase-8884-v3.patch, hbase-8884-v4.patch, hbase-8884-v5.patch, hbase-8884-v6.patch, hbase-8884-v7.patch, hbase-8884-v8.patch Today, the RPC scheduling mechanism is pretty simple: it execute requests in isolated thread-pools based on their priority. In the current implementation, all normal get/put requests are using the same pool. We'd like to add some per-user or per-region level isolation, so that a misbehaved user/region will not saturate the thread-pool and cause DoS to others easily. The idea is similar to FairScheduler in MR. The current scheduling code is not standalone and is mixed with others (Connection#processRequest). The issue is the first step to extract it to an interface, so that people are free to write and test their own implementations. This patch doesn't make it completely pluggable yet, as some parameters are pass from constructor. This is because HMaster and HRegionServer both use RpcServer and they have different thread-pool size config. Let me know if you have a solution to this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9094) Rationalize dependencies; fix analysis complaints about used but non-declared dependencies
[ https://issues.apache.org/jira/browse/HBASE-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725078#comment-13725078 ] Lars Francke commented on HBASE-9094: - I'd be happy to take a look at this tomorrow. Rationalize dependencies; fix analysis complaints about used but non-declared dependencies -- Key: HBASE-9094 URL: https://issues.apache.org/jira/browse/HBASE-9094 Project: HBase Issue Type: Sub-task Components: build Reporter: stack Assignee: stack Fix For: 0.98.0, 0.95.2 Attachments: dep2.txt, dep3.txt, dep.txt Do a cleanup pass through our dependency set so downstreamers get good picture of what they need to include looking at pom. Poking with dependency plugin, found the following issues: + hbase-common is missing listing of a bunch of commons libs + Some of our classes make use of slf4j for no good reason. zk, thrift, netty, and yammer, use slf4j but no need for us to be in the slf4j managing game... so I undid our use of it and stop its transitive include everywhere. + hbase-examples and hbase-it do not declare includes like hbase-client, hbase-protocol, etc. + hbase-hadoop1-compat depended on log4j but doesn't use it. + hbase-prefix-tree depends on hadoop but does not declare it (uses WritiableUtil and RawComparator -- just two imports... uses the Audience annotations which we could just remove but still these two critical includes) + hbase-server wasn't declaring its use of commons-*. Also added excludes of transitive include of log4j by zk and yammer. Got rid of slf4j as a dependency. + Add declarations for used libs such as httpclient and commons-math to top-level pom. + Removed setting versions on hadoop1 and hadoop2 profile for slf4j mess. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9090) cleanup snapshot tests setup/teardown code
[ https://issues.apache.org/jira/browse/HBASE-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-9090: --- Summary: cleanup snapshot tests setup/teardown code (was: cleanup snapshots setup/teardown code ) cleanup snapshot tests setup/teardown code --- Key: HBASE-9090 URL: https://issues.apache.org/jira/browse/HBASE-9090 Project: HBase Issue Type: Test Components: snapshots, test Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9090-v0.patch, HBASE-9090-v1.patch There's a lot of duplicated code around createTable() and loadTable() and other some stuff are done slightly different in each test (e.g. the snapshot deletion in the teardown) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7634) Replication handling of changes to peer clusters is inefficient
[ https://issues.apache.org/jira/browse/HBASE-7634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725081#comment-13725081 ] Hadoop QA commented on HBASE-7634: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595145/HBASE-7634.v5.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1 warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6538//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6538//console This message is automatically generated. Replication handling of changes to peer clusters is inefficient --- Key: HBASE-7634 URL: https://issues.apache.org/jira/browse/HBASE-7634 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.95.2 Reporter: Gabriel Reid Attachments: HBASE-7634.patch, HBASE-7634.v2.patch, HBASE-7634.v3.patch, HBASE-7634.v4.patch, HBASE-7634.v5.patch The current handling of changes to the region servers in a replication peer cluster is currently quite inefficient. The list of region servers that are being replicated to is only updated if there are a large number of issues encountered while replicating. This can cause it to take quite a while to recognize that a number of the regionserver in a peer cluster are no longer available. A potentially bigger problem is that if a replication peer cluster is started with a small number of regionservers, and then more region servers are added after replication has started, the additional region servers will never be used for replication (unless there are failures on the in-use regionservers). Part of the current issue is that the retry code in ReplicationSource#shipEdits checks a randomly-chosen replication peer regionserver (in ReplicationSource#isSlaveDown) to see if it is up after a replication write has failed on a different randonly-chosen replication peer. If the peer is seen as not down, another randomly-chosen peer is used for writing. A second part of the issue is that changes to the list of region servers in a peer cluster are not detected at all, and are only picked up if a certain number of failures have occurred when trying to ship edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9094) Rationalize dependencies; fix analysis complaints about used but non-declared dependencies
[ https://issues.apache.org/jira/browse/HBASE-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725085#comment-13725085 ] Hadoop QA commented on HBASE-9094: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595146/dep3.txt against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6539//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6539//console This message is automatically generated. Rationalize dependencies; fix analysis complaints about used but non-declared dependencies -- Key: HBASE-9094 URL: https://issues.apache.org/jira/browse/HBASE-9094 Project: HBase Issue Type: Sub-task Components: build Reporter: stack Assignee: stack Fix For: 0.98.0, 0.95.2 Attachments: dep2.txt, dep3.txt, dep.txt Do a cleanup pass through our dependency set so downstreamers get good picture of what they need to include looking at pom. Poking with dependency plugin, found the following issues: + hbase-common is missing listing of a bunch of commons libs + Some of our classes make use of slf4j for no good reason. zk, thrift, netty, and yammer, use slf4j but no need for us to be in the slf4j managing game... so I undid our use of it and stop its transitive include everywhere. + hbase-examples and hbase-it do not declare includes like hbase-client, hbase-protocol, etc. + hbase-hadoop1-compat depended on log4j but doesn't use it. + hbase-prefix-tree depends on hadoop but does not declare it (uses WritiableUtil and RawComparator -- just two imports... uses the Audience annotations which we could just remove but still these two critical includes) + hbase-server wasn't declaring its use of commons-*. Also added excludes of transitive include of log4j by zk and yammer. Got rid of slf4j as a dependency. + Add declarations for used libs such as httpclient and commons-math to top-level pom. + Removed setting versions on hadoop1 and hadoop2 profile for slf4j mess. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9090) cleanup snapshot tests setup/teardown code
[ https://issues.apache.org/jira/browse/HBASE-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-9090: --- Resolution: Fixed Fix Version/s: 0.94.11 Status: Resolved (was: Patch Available) committed to 0.94, 0.95 and trunk, thanks! cleanup snapshot tests setup/teardown code --- Key: HBASE-9090 URL: https://issues.apache.org/jira/browse/HBASE-9090 Project: HBase Issue Type: Test Components: snapshots, test Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9090-v0.patch, HBASE-9090-v1.patch There's a lot of duplicated code around createTable() and loadTable() and other some stuff are done slightly different in each test (e.g. the snapshot deletion in the teardown) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8768) Improve bulk load performance by moving key value construction from map phase to reduce phase.
[ https://issues.apache.org/jira/browse/HBASE-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725136#comment-13725136 ] Hadoop QA commented on HBASE-8768: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595148/HBASE-8768_v4.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 6 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6540//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6540//console This message is automatically generated. Improve bulk load performance by moving key value construction from map phase to reduce phase. -- Key: HBASE-8768 URL: https://issues.apache.org/jira/browse/HBASE-8768 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.98.0, 0.95.2 Attachments: HBASE-8768_v2.patch, HBASE-8768_v3.patch, HBASE-8768_v4.patch, HBase_Bulkload_Performance_Improvement.pdf ImportTSV bulkloading approach uses MapReduce framework. Existing mapper and reducer classes used by ImportTSV are TsvImporterMapper.java and PutSortReducer.java. ImportTSV tool parses the tab(by default) seperated values from the input files and Mapper class generates the PUT objects for each row using the Key value pairs created from the parsed text. PutSortReducer then uses the partions based on the regions and sorts the Put objects for each region. Overheads we can see in the above approach: == 1) keyvalue construction for each parsed value in the line adding extra data like rowkey,columnfamily,qualifier which will increase around 5x extra data to be shuffled in reduce phase. We can calculate data size to shuffled as below {code} Data to be shuffled = nl*nt*(rl+cfl+cql+vall+tsl+30) {code} If we move keyvalue construction to reduce phase we datasize to be shuffle will be which is very less compared to above. {code} Data to be shuffled = nl*nt*vall {code} nl - Number of lines in the raw file nt - Number of tabs or columns including row key. rl - row length which will be different for each line. cfl - column family length which will be different for each family cql - qualifier length tsl - timestamp length. vall- each parsed value length. 30 bytes for kv size,number of families etc. 2) In mapper side we are
[jira] [Commented] (HBASE-8548) postOpen hook called twice
[ https://issues.apache.org/jira/browse/HBASE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725140#comment-13725140 ] Hadoop QA commented on HBASE-8548: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595151/8548.v1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 6 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6541//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6541//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6541//console This message is automatically generated. postOpen hook called twice -- Key: HBASE-8548 URL: https://issues.apache.org/jira/browse/HBASE-8548 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.0 Environment: CentOS Reporter: Roger Ruiz-Carrillo Assignee: Nicolas Liochon Attachments: 8548.v1.patch, HRegion_HBASE-8548-0.95.patch postOpen hook is called twice when a region is initializing: Once at the end of the body of the initializeRegionInternals() method of the HRegion class. Once at the end initializeRegionStores() method of the HRegion class; initializeRegionStores() is called inside initializeRegionInternals() and as such causes the postOpen hook to be called twice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7634) Replication handling of changes to peer clusters is inefficient
[ https://issues.apache.org/jira/browse/HBASE-7634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabriel Reid updated HBASE-7634: Attachment: HBASE-7634.v6.patch Now without javadoc warnings Replication handling of changes to peer clusters is inefficient --- Key: HBASE-7634 URL: https://issues.apache.org/jira/browse/HBASE-7634 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.95.2 Reporter: Gabriel Reid Attachments: HBASE-7634.patch, HBASE-7634.v2.patch, HBASE-7634.v3.patch, HBASE-7634.v4.patch, HBASE-7634.v5.patch, HBASE-7634.v6.patch The current handling of changes to the region servers in a replication peer cluster is currently quite inefficient. The list of region servers that are being replicated to is only updated if there are a large number of issues encountered while replicating. This can cause it to take quite a while to recognize that a number of the regionserver in a peer cluster are no longer available. A potentially bigger problem is that if a replication peer cluster is started with a small number of regionservers, and then more region servers are added after replication has started, the additional region servers will never be used for replication (unless there are failures on the in-use regionservers). Part of the current issue is that the retry code in ReplicationSource#shipEdits checks a randomly-chosen replication peer regionserver (in ReplicationSource#isSlaveDown) to see if it is up after a replication write has failed on a different randonly-chosen replication peer. If the peer is seen as not down, another randomly-chosen peer is used for writing. A second part of the issue is that changes to the list of region servers in a peer cluster are not detected at all, and are only picked up if a certain number of failures have occurred when trying to ship edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9090) cleanup snapshot tests setup/teardown code
[ https://issues.apache.org/jira/browse/HBASE-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725147#comment-13725147 ] Hudson commented on HBASE-9090: --- SUCCESS: Integrated in hbase-0.95-on-hadoop2 #210 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/210/]) HBASE-9090 cleanup snapshot tests setup/teardown code (mbertozzi: rev 1508804) * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java cleanup snapshot tests setup/teardown code --- Key: HBASE-9090 URL: https://issues.apache.org/jira/browse/HBASE-9090 Project: HBase Issue Type: Test Components: snapshots, test Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9090-v0.patch, HBASE-9090-v1.patch There's a lot of duplicated code around createTable() and loadTable() and other some stuff are done slightly different in each test (e.g. the snapshot deletion in the teardown) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8960) TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes
[ https://issues.apache.org/jira/browse/HBASE-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725148#comment-13725148 ] Hudson commented on HBASE-8960: --- SUCCESS: Integrated in hbase-0.95-on-hadoop2 #210 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/210/]) HBASE-8960: TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes - Addendum2 (jeffreyz: rev 1508712) * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes -- Key: HBASE-8960 URL: https://issues.apache.org/jira/browse/HBASE-8960 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jeffrey Zhong Priority: Minor Fix For: 0.95.2 Attachments: hbase-8960-addendum-2.patch, hbase-8960-addendum.patch, hbase-8960.patch http://54.241.6.143/job/HBase-0.95-Hadoop-2/org.apache.hbase$hbase-server/634/testReport/junit/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testLogReplayForDisablingTable/ {noformat} java.lang.AssertionError: expected:1000 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testLogReplayForDisablingTable(TestDistributedLogSplitting.java:797) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9093) Hbase client API: Restore the writeToWal method
[ https://issues.apache.org/jira/browse/HBASE-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725146#comment-13725146 ] Hudson commented on HBASE-9093: --- SUCCESS: Integrated in hbase-0.95-on-hadoop2 #210 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/210/]) HBASE-9093 Hbase client API: Restore the writeToWal method (stack: rev 1508703) * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java * /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestPutWriteToWal.java Hbase client API: Restore the writeToWal method --- Key: HBASE-9093 URL: https://issues.apache.org/jira/browse/HBASE-9093 Project: HBase Issue Type: Bug Components: Client, Usability Affects Versions: 0.95.0 Reporter: Hari Shreedharan Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9093.patch, HBASE-9093.patch, HBASE-9093.patch The writeToWal is used by downstream projects like Flume to disable writes to WAL to improve performance when durability is not strictly required. But renaming this method to setDurability forces us to use reflection to support hbase versions 95 - which in turn hits performance, as this method needs to be called on every single write. I recommend adding the old method back as deprecated and removing it once hbase-95/96 becomes the popular version used in prod. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9101) Addendum to pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-9101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725152#comment-13725152 ] Hadoop QA commented on HBASE-9101: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595152/hbase-9101.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.TestCheckTestClasses Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6542//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6542//console This message is automatically generated. Addendum to pluggable RpcScheduler -- Key: HBASE-9101 URL: https://issues.apache.org/jira/browse/HBASE-9101 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Attachments: hbase-9101.patch This patch fixes the review comments from [~stack] and a small fix: - Make RpcScheduler fully pluggable. One can write its own implementation and add it to classpath and specify it by config hbase.region.server.rpc.scheduler.factory.class. - Add unit tests and fix that RpcScheduler.stop is not called (discovered by tests) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9090) cleanup snapshot tests setup/teardown code
[ https://issues.apache.org/jira/browse/HBASE-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725159#comment-13725159 ] Hudson commented on HBASE-9090: --- SUCCESS: Integrated in HBase-0.94-security #238 (See [https://builds.apache.org/job/HBase-0.94-security/238/]) HBASE-9090 cleanup snapshot tests setup/teardown code (mbertozzi: rev 1508811) * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java cleanup snapshot tests setup/teardown code --- Key: HBASE-9090 URL: https://issues.apache.org/jira/browse/HBASE-9090 Project: HBase Issue Type: Test Components: snapshots, test Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9090-v0.patch, HBASE-9090-v1.patch There's a lot of duplicated code around createTable() and loadTable() and other some stuff are done slightly different in each test (e.g. the snapshot deletion in the teardown) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8768) Improve bulk load performance by moving key value construction from map phase to reduce phase.
[ https://issues.apache.org/jira/browse/HBASE-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725174#comment-13725174 ] Jean-Marc Spaggiari commented on HBASE-8768: Do we have any numbers regarding the improvements? Like it's 10% faster% 50% faster? Improve bulk load performance by moving key value construction from map phase to reduce phase. -- Key: HBASE-8768 URL: https://issues.apache.org/jira/browse/HBASE-8768 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.98.0, 0.95.2 Attachments: HBASE-8768_v2.patch, HBASE-8768_v3.patch, HBASE-8768_v4.patch, HBase_Bulkload_Performance_Improvement.pdf ImportTSV bulkloading approach uses MapReduce framework. Existing mapper and reducer classes used by ImportTSV are TsvImporterMapper.java and PutSortReducer.java. ImportTSV tool parses the tab(by default) seperated values from the input files and Mapper class generates the PUT objects for each row using the Key value pairs created from the parsed text. PutSortReducer then uses the partions based on the regions and sorts the Put objects for each region. Overheads we can see in the above approach: == 1) keyvalue construction for each parsed value in the line adding extra data like rowkey,columnfamily,qualifier which will increase around 5x extra data to be shuffled in reduce phase. We can calculate data size to shuffled as below {code} Data to be shuffled = nl*nt*(rl+cfl+cql+vall+tsl+30) {code} If we move keyvalue construction to reduce phase we datasize to be shuffle will be which is very less compared to above. {code} Data to be shuffled = nl*nt*vall {code} nl - Number of lines in the raw file nt - Number of tabs or columns including row key. rl - row length which will be different for each line. cfl - column family length which will be different for each family cql - qualifier length tsl - timestamp length. vall- each parsed value length. 30 bytes for kv size,number of families etc. 2) In mapper side we are creating put objects by adding all keyvalues constructed for each line and in reducer we will again collect keyvalues from put and sort them. Instead we can directly create and sort keyvalues in reducer. Solution: We can improve bulk load performance by moving the key value construction from mapper to reducer so that Mapper just sends the raw text for each row to the Reducer. Reducer then parses the records for rows and create and sort the key value pairs before writing to HFiles. Conclusion: === The above suggestions will improve map phase performance by avoiding keyvalue construction and reduce phase performance by avoiding excess data to be shuffled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8768) Improve bulk load performance by moving key value construction from map phase to reduce phase.
[ https://issues.apache.org/jira/browse/HBASE-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725184#comment-13725184 ] rajeshbabu commented on HBASE-8768: --- [~jmspaggi] bq. Do we have any numbers regarding the improvements? Like it's 10% faster% 50% faster? performance results included in this pdf. https://issues.apache.org/jira/secure/attachment/12594382/HBase_Bulkload_Performance_Improvement.pdf Improve bulk load performance by moving key value construction from map phase to reduce phase. -- Key: HBASE-8768 URL: https://issues.apache.org/jira/browse/HBASE-8768 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.98.0, 0.95.2 Attachments: HBASE-8768_v2.patch, HBASE-8768_v3.patch, HBASE-8768_v4.patch, HBase_Bulkload_Performance_Improvement.pdf ImportTSV bulkloading approach uses MapReduce framework. Existing mapper and reducer classes used by ImportTSV are TsvImporterMapper.java and PutSortReducer.java. ImportTSV tool parses the tab(by default) seperated values from the input files and Mapper class generates the PUT objects for each row using the Key value pairs created from the parsed text. PutSortReducer then uses the partions based on the regions and sorts the Put objects for each region. Overheads we can see in the above approach: == 1) keyvalue construction for each parsed value in the line adding extra data like rowkey,columnfamily,qualifier which will increase around 5x extra data to be shuffled in reduce phase. We can calculate data size to shuffled as below {code} Data to be shuffled = nl*nt*(rl+cfl+cql+vall+tsl+30) {code} If we move keyvalue construction to reduce phase we datasize to be shuffle will be which is very less compared to above. {code} Data to be shuffled = nl*nt*vall {code} nl - Number of lines in the raw file nt - Number of tabs or columns including row key. rl - row length which will be different for each line. cfl - column family length which will be different for each family cql - qualifier length tsl - timestamp length. vall- each parsed value length. 30 bytes for kv size,number of families etc. 2) In mapper side we are creating put objects by adding all keyvalues constructed for each line and in reducer we will again collect keyvalues from put and sort them. Instead we can directly create and sort keyvalues in reducer. Solution: We can improve bulk load performance by moving the key value construction from mapper to reducer so that Mapper just sends the raw text for each row to the Reducer. Reducer then parses the records for rows and create and sort the key value pairs before writing to HFiles. Conclusion: === The above suggestions will improve map phase performance by avoiding keyvalue construction and reduce phase performance by avoiding excess data to be shuffled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8768) Improve bulk load performance by moving key value construction from map phase to reduce phase.
[ https://issues.apache.org/jira/browse/HBASE-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725186#comment-13725186 ] Jean-Marc Spaggiari commented on HBASE-8768: Awesome! Thanks a lot [~rajesh23] Improve bulk load performance by moving key value construction from map phase to reduce phase. -- Key: HBASE-8768 URL: https://issues.apache.org/jira/browse/HBASE-8768 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.98.0, 0.95.2 Attachments: HBASE-8768_v2.patch, HBASE-8768_v3.patch, HBASE-8768_v4.patch, HBase_Bulkload_Performance_Improvement.pdf ImportTSV bulkloading approach uses MapReduce framework. Existing mapper and reducer classes used by ImportTSV are TsvImporterMapper.java and PutSortReducer.java. ImportTSV tool parses the tab(by default) seperated values from the input files and Mapper class generates the PUT objects for each row using the Key value pairs created from the parsed text. PutSortReducer then uses the partions based on the regions and sorts the Put objects for each region. Overheads we can see in the above approach: == 1) keyvalue construction for each parsed value in the line adding extra data like rowkey,columnfamily,qualifier which will increase around 5x extra data to be shuffled in reduce phase. We can calculate data size to shuffled as below {code} Data to be shuffled = nl*nt*(rl+cfl+cql+vall+tsl+30) {code} If we move keyvalue construction to reduce phase we datasize to be shuffle will be which is very less compared to above. {code} Data to be shuffled = nl*nt*vall {code} nl - Number of lines in the raw file nt - Number of tabs or columns including row key. rl - row length which will be different for each line. cfl - column family length which will be different for each family cql - qualifier length tsl - timestamp length. vall- each parsed value length. 30 bytes for kv size,number of families etc. 2) In mapper side we are creating put objects by adding all keyvalues constructed for each line and in reducer we will again collect keyvalues from put and sort them. Instead we can directly create and sort keyvalues in reducer. Solution: We can improve bulk load performance by moving the key value construction from mapper to reducer so that Mapper just sends the raw text for each row to the Reducer. Reducer then parses the records for rows and create and sort the key value pairs before writing to HFiles. Conclusion: === The above suggestions will improve map phase performance by avoiding keyvalue construction and reduce phase performance by avoiding excess data to be shuffled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7634) Replication handling of changes to peer clusters is inefficient
[ https://issues.apache.org/jira/browse/HBASE-7634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725190#comment-13725190 ] Hadoop QA commented on HBASE-7634: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595161/HBASE-7634.v6.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6543//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6543//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6543//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6543//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6543//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6543//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6543//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6543//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6543//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6543//console This message is automatically generated. Replication handling of changes to peer clusters is inefficient --- Key: HBASE-7634 URL: https://issues.apache.org/jira/browse/HBASE-7634 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.95.2 Reporter: Gabriel Reid Attachments: HBASE-7634.patch, HBASE-7634.v2.patch, HBASE-7634.v3.patch, HBASE-7634.v4.patch, HBASE-7634.v5.patch, HBASE-7634.v6.patch The current handling of changes to the region servers in a replication peer cluster is currently quite inefficient. The list of region servers that are being replicated to is only updated if there are a large number of issues encountered while replicating. This can cause it to take quite a while to recognize that a number of the regionserver in a peer cluster are no longer available. A potentially bigger problem is that if a replication peer cluster is started with a small number of regionservers, and then more region servers are added after replication has started, the additional region servers will never be used for replication (unless there are failures on the in-use regionservers). Part of the current issue is that the retry code in ReplicationSource#shipEdits checks a randomly-chosen replication peer regionserver (in ReplicationSource#isSlaveDown) to see if it is up after a replication write has failed on a different randonly-chosen replication peer. If the peer is seen as not down, another randomly-chosen peer is used for writing. A second part of the issue is that changes to the list of region servers in a peer cluster are not detected at all, and are only picked up if a certain number of failures have occurred when trying to ship edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9090) cleanup snapshot tests setup/teardown code
[ https://issues.apache.org/jira/browse/HBASE-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725192#comment-13725192 ] Hudson commented on HBASE-9090: --- FAILURE: Integrated in HBase-0.94 #1085 (See [https://builds.apache.org/job/HBase-0.94/1085/]) HBASE-9090 cleanup snapshot tests setup/teardown code (mbertozzi: rev 1508811) * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java cleanup snapshot tests setup/teardown code --- Key: HBASE-9090 URL: https://issues.apache.org/jira/browse/HBASE-9090 Project: HBase Issue Type: Test Components: snapshots, test Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9090-v0.patch, HBASE-9090-v1.patch There's a lot of duplicated code around createTable() and loadTable() and other some stuff are done slightly different in each test (e.g. the snapshot deletion in the teardown) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9090) cleanup snapshot tests setup/teardown code
[ https://issues.apache.org/jira/browse/HBASE-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725208#comment-13725208 ] Hudson commented on HBASE-9090: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #645 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/645/]) HBASE-9090 cleanup snapshot tests setup/teardown code (mbertozzi: rev 1508801) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java cleanup snapshot tests setup/teardown code --- Key: HBASE-9090 URL: https://issues.apache.org/jira/browse/HBASE-9090 Project: HBase Issue Type: Test Components: snapshots, test Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9090-v0.patch, HBASE-9090-v1.patch There's a lot of duplicated code around createTable() and loadTable() and other some stuff are done slightly different in each test (e.g. the snapshot deletion in the teardown) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8960) TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes
[ https://issues.apache.org/jira/browse/HBASE-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725209#comment-13725209 ] Hudson commented on HBASE-8960: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #645 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/645/]) HBASE-8960: TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes - Addendum2 (jeffreyz: rev 1508711) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes -- Key: HBASE-8960 URL: https://issues.apache.org/jira/browse/HBASE-8960 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jeffrey Zhong Priority: Minor Fix For: 0.95.2 Attachments: hbase-8960-addendum-2.patch, hbase-8960-addendum.patch, hbase-8960.patch http://54.241.6.143/job/HBase-0.95-Hadoop-2/org.apache.hbase$hbase-server/634/testReport/junit/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testLogReplayForDisablingTable/ {noformat} java.lang.AssertionError: expected:1000 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testLogReplayForDisablingTable(TestDistributedLogSplitting.java:797) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9093) Hbase client API: Restore the writeToWal method
[ https://issues.apache.org/jira/browse/HBASE-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725207#comment-13725207 ] Hudson commented on HBASE-9093: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #645 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/645/]) HBASE-9093 Hbase client API: Restore the writeToWal method (stack: rev 1508702) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java * /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestPutWriteToWal.java Hbase client API: Restore the writeToWal method --- Key: HBASE-9093 URL: https://issues.apache.org/jira/browse/HBASE-9093 Project: HBase Issue Type: Bug Components: Client, Usability Affects Versions: 0.95.0 Reporter: Hari Shreedharan Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9093.patch, HBASE-9093.patch, HBASE-9093.patch The writeToWal is used by downstream projects like Flume to disable writes to WAL to improve performance when durability is not strictly required. But renaming this method to setDurability forces us to use reflection to support hbase versions 95 - which in turn hits performance, as this method needs to be called on every single write. I recommend adding the old method back as deprecated and removing it once hbase-95/96 becomes the popular version used in prod. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9090) cleanup snapshot tests setup/teardown code
[ https://issues.apache.org/jira/browse/HBASE-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725217#comment-13725217 ] Hudson commented on HBASE-9090: --- SUCCESS: Integrated in HBase-TRUNK #4324 (See [https://builds.apache.org/job/HBase-TRUNK/4324/]) HBASE-9090 cleanup snapshot tests setup/teardown code (mbertozzi: rev 1508801) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java cleanup snapshot tests setup/teardown code --- Key: HBASE-9090 URL: https://issues.apache.org/jira/browse/HBASE-9090 Project: HBase Issue Type: Test Components: snapshots, test Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9090-v0.patch, HBASE-9090-v1.patch There's a lot of duplicated code around createTable() and loadTable() and other some stuff are done slightly different in each test (e.g. the snapshot deletion in the teardown) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9090) cleanup snapshot tests setup/teardown code
[ https://issues.apache.org/jira/browse/HBASE-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725223#comment-13725223 ] Hudson commented on HBASE-9090: --- SUCCESS: Integrated in hbase-0.95 #388 (See [https://builds.apache.org/job/hbase-0.95/388/]) HBASE-9090 cleanup snapshot tests setup/teardown code (mbertozzi: rev 1508804) * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java cleanup snapshot tests setup/teardown code --- Key: HBASE-9090 URL: https://issues.apache.org/jira/browse/HBASE-9090 Project: HBase Issue Type: Test Components: snapshots, test Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9090-v0.patch, HBASE-9090-v1.patch There's a lot of duplicated code around createTable() and loadTable() and other some stuff are done slightly different in each test (e.g. the snapshot deletion in the teardown) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8548) postOpen hook called twice
[ https://issues.apache.org/jira/browse/HBASE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725306#comment-13725306 ] stack commented on HBASE-8548: -- Excellent [~nkeywal] +1 on commit postOpen hook called twice -- Key: HBASE-8548 URL: https://issues.apache.org/jira/browse/HBASE-8548 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.0 Environment: CentOS Reporter: Roger Ruiz-Carrillo Assignee: Nicolas Liochon Attachments: 8548.v1.patch, HRegion_HBASE-8548-0.95.patch postOpen hook is called twice when a region is initializing: Once at the end of the body of the initializeRegionInternals() method of the HRegion class. Once at the end initializeRegionStores() method of the HRegion class; initializeRegionStores() is called inside initializeRegionInternals() and as such causes the postOpen hook to be called twice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8548) postOpen hook called twice
[ https://issues.apache.org/jira/browse/HBASE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-8548: --- Resolution: Fixed Fix Version/s: 0.95.2 0.98.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk 0.95, thanks for the review, Stack. postOpen hook called twice -- Key: HBASE-8548 URL: https://issues.apache.org/jira/browse/HBASE-8548 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.0 Environment: CentOS Reporter: Roger Ruiz-Carrillo Assignee: Nicolas Liochon Fix For: 0.98.0, 0.95.2 Attachments: 8548.v1.patch, HRegion_HBASE-8548-0.95.patch postOpen hook is called twice when a region is initializing: Once at the end of the body of the initializeRegionInternals() method of the HRegion class. Once at the end initializeRegionStores() method of the HRegion class; initializeRegionStores() is called inside initializeRegionInternals() and as such causes the postOpen hook to be called twice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9087) Handlers being blocked during reads
[ https://issues.apache.org/jira/browse/HBASE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725346#comment-13725346 ] Elliott Clark commented on HBASE-9087: -- Running a ycsb workload e benchmark on this right now. Handlers being blocked during reads --- Key: HBASE-9087 URL: https://issues.apache.org/jira/browse/HBASE-9087 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 0.94.7, 0.95.1 Reporter: Pablo Medina Assignee: Elliott Clark Priority: Critical Attachments: HBASE-9087-0.patch, HBASE-9087-1.patch I'm having a lot of handlers (90 - 300 aprox) being blocked when reading rows. They are blocked during changedReaderObserver registration. Lars Hofhansl suggests to change the implementation of changedReaderObserver from CopyOnWriteList to ConcurrentHashMap. Here is a stack trace: IPC Server handler 99 on 60020 daemon prio=10 tid=0x41c84000 nid=0x2244 waiting on condition [0x7ff51fefd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xc5c13ae8 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:553) at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221) at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:1085) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2077) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3755) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1804) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1796) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1771) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4776) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700) at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8768) Improve bulk load performance by moving key value construction from map phase to reduce phase.
[ https://issues.apache.org/jira/browse/HBASE-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725367#comment-13725367 ] Ted Yu commented on HBASE-8768: --- Integrated to trunk. Thanks for the patch, Rajeshbabu. Thanks for the reviews. Improve bulk load performance by moving key value construction from map phase to reduce phase. -- Key: HBASE-8768 URL: https://issues.apache.org/jira/browse/HBASE-8768 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.98.0, 0.95.2 Attachments: HBASE-8768_v2.patch, HBASE-8768_v3.patch, HBASE-8768_v4.patch, HBase_Bulkload_Performance_Improvement.pdf ImportTSV bulkloading approach uses MapReduce framework. Existing mapper and reducer classes used by ImportTSV are TsvImporterMapper.java and PutSortReducer.java. ImportTSV tool parses the tab(by default) seperated values from the input files and Mapper class generates the PUT objects for each row using the Key value pairs created from the parsed text. PutSortReducer then uses the partions based on the regions and sorts the Put objects for each region. Overheads we can see in the above approach: == 1) keyvalue construction for each parsed value in the line adding extra data like rowkey,columnfamily,qualifier which will increase around 5x extra data to be shuffled in reduce phase. We can calculate data size to shuffled as below {code} Data to be shuffled = nl*nt*(rl+cfl+cql+vall+tsl+30) {code} If we move keyvalue construction to reduce phase we datasize to be shuffle will be which is very less compared to above. {code} Data to be shuffled = nl*nt*vall {code} nl - Number of lines in the raw file nt - Number of tabs or columns including row key. rl - row length which will be different for each line. cfl - column family length which will be different for each family cql - qualifier length tsl - timestamp length. vall- each parsed value length. 30 bytes for kv size,number of families etc. 2) In mapper side we are creating put objects by adding all keyvalues constructed for each line and in reducer we will again collect keyvalues from put and sort them. Instead we can directly create and sort keyvalues in reducer. Solution: We can improve bulk load performance by moving the key value construction from mapper to reducer so that Mapper just sends the raw text for each row to the Reducer. Reducer then parses the records for rows and create and sort the key value pairs before writing to HFiles. Conclusion: === The above suggestions will improve map phase performance by avoiding keyvalue construction and reduce phase performance by avoiding excess data to be shuffled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9097) Set HBASE_CLASSPATH before rest of the classpath
[ https://issues.apache.org/jira/browse/HBASE-9097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725376#comment-13725376 ] Lars Hofhansl commented on HBASE-9097: -- +1 from me. I would like to find out about the previous rationale of placing the HBASE_CLASSPATH last, though. Set HBASE_CLASSPATH before rest of the classpath Key: HBASE-9097 URL: https://issues.apache.org/jira/browse/HBASE-9097 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.98.0, 0.95.2, 0.94.11 Reporter: Jesse Yates Assignee: Jesse Yates Attachments: hbase-9097-v0.patch We encountered this when one of the hadoop test jars (specifically hadoop-mapreduce-client-jobclient-2.0.0-cdh4.3.0-tests.jar, but that's beside the point) had an hdfs-site.xml. This clobbered the hdfs-site.xml that we included on the classpath via HBASE_CLASSPATH in hbase-env.sh, meaning the master didn't start in HA NN mode, because the proxy-provider wasn't found in the hdfs-site.xml from the test jar (even though it was in our config file) because that was the first resolution of that file. This should be a fairly simple fix in bin/hbase, but has some potentially wide-ranging effects on existing installs that just 'happen' to work. Generally, I'd expect things set on the HBASE_CLASSPATH to take precedence over anything else when starting the hbase daemon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-9101) Addendum to pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-9101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-9101: - Assignee: Chao Shi Addendum to pluggable RpcScheduler -- Key: HBASE-9101 URL: https://issues.apache.org/jira/browse/HBASE-9101 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Attachments: hbase-9101.patch This patch fixes the review comments from [~stack] and a small fix: - Make RpcScheduler fully pluggable. One can write its own implementation and add it to classpath and specify it by config hbase.region.server.rpc.scheduler.factory.class. - Add unit tests and fix that RpcScheduler.stop is not called (discovered by tests) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9101) Addendum to pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-9101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-9101: -- Description: This patch fixes the review comments from [~stack] and a small fix: - Make RpcScheduler fully pluggable. One can write his/her own implementation and add it to classpath and specify it by config hbase.region.server.rpc.scheduler.factory.class. - Add unit tests and fix that RpcScheduler.stop is not called (discovered by tests) was: This patch fixes the review comments from [~stack] and a small fix: - Make RpcScheduler fully pluggable. One can write its own implementation and add it to classpath and specify it by config hbase.region.server.rpc.scheduler.factory.class. - Add unit tests and fix that RpcScheduler.stop is not called (discovered by tests) Addendum to pluggable RpcScheduler -- Key: HBASE-9101 URL: https://issues.apache.org/jira/browse/HBASE-9101 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-9101.patch This patch fixes the review comments from [~stack] and a small fix: - Make RpcScheduler fully pluggable. One can write his/her own implementation and add it to classpath and specify it by config hbase.region.server.rpc.scheduler.factory.class. - Add unit tests and fix that RpcScheduler.stop is not called (discovered by tests) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7266) [89-fb] Using pread for non-compaction read request
[ https://issues.apache.org/jira/browse/HBASE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725387#comment-13725387 ] Lars Hofhansl commented on HBASE-7266: -- In 0.94+ I fixed this by falling back to pread if the lock istream is taken (HBASE-7336). [89-fb] Using pread for non-compaction read request --- Key: HBASE-7266 URL: https://issues.apache.org/jira/browse/HBASE-7266 Project: HBase Issue Type: Improvement Reporter: Liyin Tang There are 2 kinds of read operations in HBase: pread and seek+read. Pread, positional read, is stateless and create a new connection between the DFSClient and DataNode for each operation. While seek+read is to seek to a specific postion and prefetch blocks from data nodes. The benefit of seek+read is that it will cache the prefetch result but the downside is it is stateful and needs to synchronized. So far, both compaction and scan are using seek+read, which caused some resource contention. So using the pread for the scan request can avoid the resource contention. In addition, the region server is able to do the prefetch for the scan request (HBASE-6874) so that it won't be necessary to let the DFSClient to prefetch the data any more. I will run through the scan benchmark (with no block cache) with verify the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9101) Addendum to pluggable RpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-9101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-9101: -- Fix Version/s: 0.98.0 Addendum to pluggable RpcScheduler -- Key: HBASE-9101 URL: https://issues.apache.org/jira/browse/HBASE-9101 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0 Attachments: hbase-9101.patch This patch fixes the review comments from [~stack] and a small fix: - Make RpcScheduler fully pluggable. One can write its own implementation and add it to classpath and specify it by config hbase.region.server.rpc.scheduler.factory.class. - Add unit tests and fix that RpcScheduler.stop is not called (discovered by tests) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9093) Hbase client API: Restore the writeToWal method
[ https://issues.apache.org/jira/browse/HBASE-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725390#comment-13725390 ] Lars Hofhansl commented on HBASE-9093: -- Seems we need to regenerate the 0.94 API docs on the site. 0.94 has the new api since 0.94.7 (HBASE-7801) Hbase client API: Restore the writeToWal method --- Key: HBASE-9093 URL: https://issues.apache.org/jira/browse/HBASE-9093 Project: HBase Issue Type: Bug Components: Client, Usability Affects Versions: 0.95.0 Reporter: Hari Shreedharan Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9093.patch, HBASE-9093.patch, HBASE-9093.patch The writeToWal is used by downstream projects like Flume to disable writes to WAL to improve performance when durability is not strictly required. But renaming this method to setDurability forces us to use reflection to support hbase versions 95 - which in turn hits performance, as this method needs to be called on every single write. I recommend adding the old method back as deprecated and removing it once hbase-95/96 becomes the popular version used in prod. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9087) Handlers being blocked during reads
[ https://issues.apache.org/jira/browse/HBASE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9087: - Fix Version/s: 0.94.11 0.95.2 0.98.0 Handlers being blocked during reads --- Key: HBASE-9087 URL: https://issues.apache.org/jira/browse/HBASE-9087 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 0.94.7, 0.95.1 Reporter: Pablo Medina Assignee: Elliott Clark Priority: Critical Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9087-0.patch, HBASE-9087-1.patch I'm having a lot of handlers (90 - 300 aprox) being blocked when reading rows. They are blocked during changedReaderObserver registration. Lars Hofhansl suggests to change the implementation of changedReaderObserver from CopyOnWriteList to ConcurrentHashMap. Here is a stack trace: IPC Server handler 99 on 60020 daemon prio=10 tid=0x41c84000 nid=0x2244 waiting on condition [0x7ff51fefd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xc5c13ae8 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:553) at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221) at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:1085) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2077) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3755) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1804) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1796) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1771) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4776) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700) at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9093) Hbase client API: Restore the writeToWal method
[ https://issues.apache.org/jira/browse/HBASE-9093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725394#comment-13725394 ] Lars Hofhansl commented on HBASE-9093: -- And setWriteToWal is deprecated in 0.94 (but again, that is not visible from the docs on the site). Hbase client API: Restore the writeToWal method --- Key: HBASE-9093 URL: https://issues.apache.org/jira/browse/HBASE-9093 Project: HBase Issue Type: Bug Components: Client, Usability Affects Versions: 0.95.0 Reporter: Hari Shreedharan Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9093.patch, HBASE-9093.patch, HBASE-9093.patch The writeToWal is used by downstream projects like Flume to disable writes to WAL to improve performance when durability is not strictly required. But renaming this method to setDurability forces us to use reflection to support hbase versions 95 - which in turn hits performance, as this method needs to be called on every single write. I recommend adding the old method back as deprecated and removing it once hbase-95/96 becomes the popular version used in prod. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8548) postOpen hook called twice
[ https://issues.apache.org/jira/browse/HBASE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725398#comment-13725398 ] Hudson commented on HBASE-8548: --- SUCCESS: Integrated in HBase-TRUNK #4325 (See [https://builds.apache.org/job/HBase-TRUNK/4325/]) HBASE-8548 postOpen hook called twice (nkeywal: rev 1508895) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java postOpen hook called twice -- Key: HBASE-8548 URL: https://issues.apache.org/jira/browse/HBASE-8548 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.0 Environment: CentOS Reporter: Roger Ruiz-Carrillo Assignee: Nicolas Liochon Fix For: 0.98.0, 0.95.2 Attachments: 8548.v1.patch, HRegion_HBASE-8548-0.95.patch postOpen hook is called twice when a region is initializing: Once at the end of the body of the initializeRegionInternals() method of the HRegion class. Once at the end initializeRegionStores() method of the HRegion class; initializeRegionStores() is called inside initializeRegionInternals() and as such causes the postOpen hook to be called twice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7634) Replication handling of changes to peer clusters is inefficient
[ https://issues.apache.org/jira/browse/HBASE-7634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725400#comment-13725400 ] Jean-Daniel Cryans commented on HBASE-7634: --- Alright I'm +1, thanks for tackling this Gabriel. [~ctrezzo] did you want to review before I commit? [~lhofhansl] is this too big of a change for 0.94? It seems the backport shouldn't be too bad. Replication handling of changes to peer clusters is inefficient --- Key: HBASE-7634 URL: https://issues.apache.org/jira/browse/HBASE-7634 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.95.2 Reporter: Gabriel Reid Attachments: HBASE-7634.patch, HBASE-7634.v2.patch, HBASE-7634.v3.patch, HBASE-7634.v4.patch, HBASE-7634.v5.patch, HBASE-7634.v6.patch The current handling of changes to the region servers in a replication peer cluster is currently quite inefficient. The list of region servers that are being replicated to is only updated if there are a large number of issues encountered while replicating. This can cause it to take quite a while to recognize that a number of the regionserver in a peer cluster are no longer available. A potentially bigger problem is that if a replication peer cluster is started with a small number of regionservers, and then more region servers are added after replication has started, the additional region servers will never be used for replication (unless there are failures on the in-use regionservers). Part of the current issue is that the retry code in ReplicationSource#shipEdits checks a randomly-chosen replication peer regionserver (in ReplicationSource#isSlaveDown) to see if it is up after a replication write has failed on a different randonly-chosen replication peer. If the peer is seen as not down, another randomly-chosen peer is used for writing. A second part of the issue is that changes to the list of region servers in a peer cluster are not detected at all, and are only picked up if a certain number of failures have occurred when trying to ship edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9087) Handlers being blocked during reads
[ https://issues.apache.org/jira/browse/HBASE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725406#comment-13725406 ] Pablo Medina commented on HBASE-9087: - Elliot, did you run that benchmark ? did it improve the performance under concurrency? Handlers being blocked during reads --- Key: HBASE-9087 URL: https://issues.apache.org/jira/browse/HBASE-9087 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 0.94.7, 0.95.1 Reporter: Pablo Medina Assignee: Elliott Clark Priority: Critical Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9087-0.patch, HBASE-9087-1.patch I'm having a lot of handlers (90 - 300 aprox) being blocked when reading rows. They are blocked during changedReaderObserver registration. Lars Hofhansl suggests to change the implementation of changedReaderObserver from CopyOnWriteList to ConcurrentHashMap. Here is a stack trace: IPC Server handler 99 on 60020 daemon prio=10 tid=0x41c84000 nid=0x2244 waiting on condition [0x7ff51fefd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xc5c13ae8 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:553) at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221) at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:1085) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2077) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3755) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1804) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1796) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1771) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4776) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700) at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7634) Replication handling of changes to peer clusters is inefficient
[ https://issues.apache.org/jira/browse/HBASE-7634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725409#comment-13725409 ] Lars Hofhansl commented on HBASE-7634: -- Hmm... That's a tough one. The logic can clearly improved upon, but this is bigg'ish change with new uses of ZK. I'd -0 unless somebody makes a strong case for this in 0.94. Replication handling of changes to peer clusters is inefficient --- Key: HBASE-7634 URL: https://issues.apache.org/jira/browse/HBASE-7634 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.95.2 Reporter: Gabriel Reid Attachments: HBASE-7634.patch, HBASE-7634.v2.patch, HBASE-7634.v3.patch, HBASE-7634.v4.patch, HBASE-7634.v5.patch, HBASE-7634.v6.patch The current handling of changes to the region servers in a replication peer cluster is currently quite inefficient. The list of region servers that are being replicated to is only updated if there are a large number of issues encountered while replicating. This can cause it to take quite a while to recognize that a number of the regionserver in a peer cluster are no longer available. A potentially bigger problem is that if a replication peer cluster is started with a small number of regionservers, and then more region servers are added after replication has started, the additional region servers will never be used for replication (unless there are failures on the in-use regionservers). Part of the current issue is that the retry code in ReplicationSource#shipEdits checks a randomly-chosen replication peer regionserver (in ReplicationSource#isSlaveDown) to see if it is up after a replication write has failed on a different randonly-chosen replication peer. If the peer is seen as not down, another randomly-chosen peer is used for writing. A second part of the issue is that changes to the list of region servers in a peer cluster are not detected at all, and are only picked up if a certain number of failures have occurred when trying to ship edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9085) Integration Tests fails because of bug in teardown phase where the cluster state is not being restored properly.
[ https://issues.apache.org/jira/browse/HBASE-9085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gautam updated HBASE-9085: -- Issue Type: Bug (was: Test) Integration Tests fails because of bug in teardown phase where the cluster state is not being restored properly. Key: HBASE-9085 URL: https://issues.apache.org/jira/browse/HBASE-9085 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.95.0, 0.94.9, 0.94.10 Reporter: gautam Assignee: gautam Fix For: 0.98.0, 0.95.2, 0.94.10 Attachments: HBASE-9085.patch._0.94, HBASE-9085.patch._0.95_or_trunk I was running the following test over a Distributed Cluster: bin/hbase org.apache.hadoop.hbase.IntegrationTestsDriver IntegrationTestDataIngestSlowDeterministic The IntegrationTestingUtility.restoreCluster() is called in the teardown phase of the test. For a distributed cluster, it ends up calling DistributedHBaseCluster.restoreClusterStatus, which does the task of restoring the cluster back to original state. The restore steps done here, does not solve one specific case: When the initial HBase Master is currently down, and the current HBase Master is different from the initial one. You get into this flow: //check whether current master has changed if (!ServerName.isSameHostnameAndPort(initial.getMaster(), current.getMaster())) { . } In the above code path, the current backup masters are stopped, and the current active master is also stopped. At this point, for the aforementioned usecase, none of the Hbase Masters would be available, hence the subsequent attempts to do any operation over the cluster would fail, resulting in Test Failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9087) Handlers being blocked during reads
[ https://issues.apache.org/jira/browse/HBASE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725414#comment-13725414 ] Lars Hofhansl commented on HBASE-9087: -- I'm very curious as well :) Handlers being blocked during reads --- Key: HBASE-9087 URL: https://issues.apache.org/jira/browse/HBASE-9087 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 0.94.7, 0.95.1 Reporter: Pablo Medina Assignee: Elliott Clark Priority: Critical Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9087-0.patch, HBASE-9087-1.patch I'm having a lot of handlers (90 - 300 aprox) being blocked when reading rows. They are blocked during changedReaderObserver registration. Lars Hofhansl suggests to change the implementation of changedReaderObserver from CopyOnWriteList to ConcurrentHashMap. Here is a stack trace: IPC Server handler 99 on 60020 daemon prio=10 tid=0x41c84000 nid=0x2244 waiting on condition [0x7ff51fefd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xc5c13ae8 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:553) at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221) at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:1085) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2077) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3755) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1804) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1796) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1771) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4776) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700) at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9087) Handlers being blocked during reads
[ https://issues.apache.org/jira/browse/HBASE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725422#comment-13725422 ] Pablo Medina commented on HBASE-9087: - btw. In the meanwhile... do you guys know what is a 'proper' the number of handlers?. I know that 'proper' means different things in different use cases but have you seen any region server serving requests using 1k handlers or more? It is that common scenario? Handlers being blocked during reads --- Key: HBASE-9087 URL: https://issues.apache.org/jira/browse/HBASE-9087 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 0.94.7, 0.95.1 Reporter: Pablo Medina Assignee: Elliott Clark Priority: Critical Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9087-0.patch, HBASE-9087-1.patch I'm having a lot of handlers (90 - 300 aprox) being blocked when reading rows. They are blocked during changedReaderObserver registration. Lars Hofhansl suggests to change the implementation of changedReaderObserver from CopyOnWriteList to ConcurrentHashMap. Here is a stack trace: IPC Server handler 99 on 60020 daemon prio=10 tid=0x41c84000 nid=0x2244 waiting on condition [0x7ff51fefd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xc5c13ae8 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:553) at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221) at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:1085) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2077) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3755) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1804) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1796) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1771) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4776) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700) at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9087) Handlers being blocked during reads
[ https://issues.apache.org/jira/browse/HBASE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725428#comment-13725428 ] Elliott Clark commented on HBASE-9087: -- Running the benchmarks but they are tied in with integration tests so they will take 5 hours or so. I hope to have results by the end of the day. Handlers being blocked during reads --- Key: HBASE-9087 URL: https://issues.apache.org/jira/browse/HBASE-9087 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 0.94.7, 0.95.1 Reporter: Pablo Medina Assignee: Elliott Clark Priority: Critical Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9087-0.patch, HBASE-9087-1.patch I'm having a lot of handlers (90 - 300 aprox) being blocked when reading rows. They are blocked during changedReaderObserver registration. Lars Hofhansl suggests to change the implementation of changedReaderObserver from CopyOnWriteList to ConcurrentHashMap. Here is a stack trace: IPC Server handler 99 on 60020 daemon prio=10 tid=0x41c84000 nid=0x2244 waiting on condition [0x7ff51fefd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xc5c13ae8 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:553) at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221) at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:1085) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2077) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3755) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1804) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1796) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1771) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4776) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700) at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9087) Handlers being blocked during reads
[ https://issues.apache.org/jira/browse/HBASE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725436#comment-13725436 ] Lars Hofhansl commented on HBASE-9087: -- You want to be able to keep both CPU and Disks busy. So one should have at least as many handlers as CPU threads and disk spindles. Beyond that it is trial and error. We have 12 core CPU (24 HW threads) and 6 disk drives and have set the handler count to 50. Handlers being blocked during reads --- Key: HBASE-9087 URL: https://issues.apache.org/jira/browse/HBASE-9087 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 0.94.7, 0.95.1 Reporter: Pablo Medina Assignee: Elliott Clark Priority: Critical Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9087-0.patch, HBASE-9087-1.patch I'm having a lot of handlers (90 - 300 aprox) being blocked when reading rows. They are blocked during changedReaderObserver registration. Lars Hofhansl suggests to change the implementation of changedReaderObserver from CopyOnWriteList to ConcurrentHashMap. Here is a stack trace: IPC Server handler 99 on 60020 daemon prio=10 tid=0x41c84000 nid=0x2244 waiting on condition [0x7ff51fefd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xc5c13ae8 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:553) at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221) at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:1085) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2077) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3755) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1804) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1796) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1771) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4776) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700) at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9102) HFile block pre-loading for large sequential scan
Liyin Tang created HBASE-9102: - Summary: HFile block pre-loading for large sequential scan Key: HBASE-9102 URL: https://issues.apache.org/jira/browse/HBASE-9102 Project: HBase Issue Type: Improvement Affects Versions: 0.89-fb Reporter: Liyin Tang Assignee: Liyin Tang The current HBase scan model cannot take full advantage of the aggrediate disk throughput, especially for the large sequential scan cases. And for the large sequential scan, it is easy to predict what the next block to read in advance so that it can pre-load and decompress/decoded these data blocks from HDFS into block cache right before the current read point. Therefore, this jira is to optimized the large sequential scan performance by pre-loading the HFile blocks into the block cache in a stream fashion so that the scan query can read from the cache directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8548) postOpen hook called twice
[ https://issues.apache.org/jira/browse/HBASE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725437#comment-13725437 ] Hudson commented on HBASE-8548: --- SUCCESS: Integrated in hbase-0.95 #389 (See [https://builds.apache.org/job/hbase-0.95/389/]) HBASE-8548 postOpen hook called twice (nkeywal: rev 1508897) * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java postOpen hook called twice -- Key: HBASE-8548 URL: https://issues.apache.org/jira/browse/HBASE-8548 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.95.0 Environment: CentOS Reporter: Roger Ruiz-Carrillo Assignee: Nicolas Liochon Fix For: 0.98.0, 0.95.2 Attachments: 8548.v1.patch, HRegion_HBASE-8548-0.95.patch postOpen hook is called twice when a region is initializing: Once at the end of the body of the initializeRegionInternals() method of the HRegion class. Once at the end initializeRegionStores() method of the HRegion class; initializeRegionStores() is called inside initializeRegionInternals() and as such causes the postOpen hook to be called twice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7266) [89-fb] Using pread for non-compaction read request
[ https://issues.apache.org/jira/browse/HBASE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725443#comment-13725443 ] Liyin Tang commented on HBASE-7266: --- Chao,we have switched all the read operation to pread in the 89-fb branch. There are 2 followup tasks for the pread. 1) The DFSClient maintains a connection pool instead of creating new connection for each pread operation. 2) HBase will actively pre-load the next several blocks in a stream fashion for large sequential scans (HBASE-9102) [89-fb] Using pread for non-compaction read request --- Key: HBASE-7266 URL: https://issues.apache.org/jira/browse/HBASE-7266 Project: HBase Issue Type: Improvement Reporter: Liyin Tang There are 2 kinds of read operations in HBase: pread and seek+read. Pread, positional read, is stateless and create a new connection between the DFSClient and DataNode for each operation. While seek+read is to seek to a specific postion and prefetch blocks from data nodes. The benefit of seek+read is that it will cache the prefetch result but the downside is it is stateful and needs to synchronized. So far, both compaction and scan are using seek+read, which caused some resource contention. So using the pread for the scan request can avoid the resource contention. In addition, the region server is able to do the prefetch for the scan request (HBASE-6874) so that it won't be necessary to let the DFSClient to prefetch the data any more. I will run through the scan benchmark (with no block cache) with verify the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725442#comment-13725442 ] Matt Corgan commented on HBASE-9091: On second thought I'm not as worried about prefix-tree performance since it hasn't been released yet and therefore has no established baseline. I'll probably go back after .96 is released to make further performance improvements and can evaluate the impact then. I can easily dig up the old version of ByteRange if there was an impact, but there's also a chance that ByteRange isn't even the fastest strategy for prefix-tree. I'm still worried about mutability. The class was designed to be used similarly to String, but for bytes instead of chars. Recycling of the object was the major difference and was to be used with care. If people start using this new version with mutable position all around the code base, we're going to run into nasty bugs where methods disagree on who is responsible for position. Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Sub-task Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.95.2 Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9102) HFile block pre-loading for large sequential scan
[ https://issues.apache.org/jira/browse/HBASE-9102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725446#comment-13725446 ] Lars Hofhansl commented on HBASE-9102: -- Should we handle this from the client instead. Part of the problem is that the network pipe is not kept full, so by triggering prefetching from the client we can solve both problems. HFile block pre-loading for large sequential scan - Key: HBASE-9102 URL: https://issues.apache.org/jira/browse/HBASE-9102 Project: HBase Issue Type: Improvement Affects Versions: 0.89-fb Reporter: Liyin Tang Assignee: Liyin Tang The current HBase scan model cannot take full advantage of the aggrediate disk throughput, especially for the large sequential scan cases. And for the large sequential scan, it is easy to predict what the next block to read in advance so that it can pre-load and decompress/decoded these data blocks from HDFS into block cache right before the current read point. Therefore, this jira is to optimized the large sequential scan performance by pre-loading the HFile blocks into the block cache in a stream fashion so that the scan query can read from the cache directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8089) Add type support
[ https://issues.apache.org/jira/browse/HBASE-8089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725451#comment-13725451 ] Matt Corgan commented on HBASE-8089: Nick - what other dependencies does your whole new type library have on HBase besides ByteRange? It would be grand if it were a standalone jar that could be used by other projects without importing hbase-specific libs (which then drag in other dependencies). All of this functionality is really cool and is more likely to gain adoption if it's as easy as possible to drop in existing projects. Add type support Key: HBASE-8089 URL: https://issues.apache.org/jira/browse/HBASE-8089 Project: HBase Issue Type: New Feature Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.95.2 Attachments: HBASE-8089-types.txt, HBASE-8089-types.txt, HBASE-8089-types.txt, HBASE-8089-types.txt, hbase data types WIP.pdf This proposal outlines an improvement to HBase that provides for a set of types, above and beyond the existing byte-bucket strategy. This is intended to reduce user-level duplication of effort, provide better support for 3rd-party integration, and provide an overall improved experience for developers using HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9102) HFile block pre-loading for large sequential scan
[ https://issues.apache.org/jira/browse/HBASE-9102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725453#comment-13725453 ] Vladimir Rodionov commented on HBASE-9102: -- This is what, actually, OS does and all modern disk controllers do on a lower lvel. If short-circuit reads are enabled and HBase cluster has good HDFS locality and data is compacted you will get dick blocks pre-fetching for free. HFile block pre-loading for large sequential scan - Key: HBASE-9102 URL: https://issues.apache.org/jira/browse/HBASE-9102 Project: HBase Issue Type: Improvement Affects Versions: 0.89-fb Reporter: Liyin Tang Assignee: Liyin Tang The current HBase scan model cannot take full advantage of the aggrediate disk throughput, especially for the large sequential scan cases. And for the large sequential scan, it is easy to predict what the next block to read in advance so that it can pre-load and decompress/decoded these data blocks from HDFS into block cache right before the current read point. Therefore, this jira is to optimized the large sequential scan performance by pre-loading the HFile blocks into the block cache in a stream fashion so that the scan query can read from the cache directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9087) Handlers being blocked during reads
[ https://issues.apache.org/jira/browse/HBASE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725455#comment-13725455 ] Elliott Clark commented on HBASE-9087: -- Requests are queued as they are decoded off the wire. SO you can have lots of requests coming in, however only 50 will be actively worked on at a time. Handlers being blocked during reads --- Key: HBASE-9087 URL: https://issues.apache.org/jira/browse/HBASE-9087 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 0.94.7, 0.95.1 Reporter: Pablo Medina Assignee: Elliott Clark Priority: Critical Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9087-0.patch, HBASE-9087-1.patch I'm having a lot of handlers (90 - 300 aprox) being blocked when reading rows. They are blocked during changedReaderObserver registration. Lars Hofhansl suggests to change the implementation of changedReaderObserver from CopyOnWriteList to ConcurrentHashMap. Here is a stack trace: IPC Server handler 99 on 60020 daemon prio=10 tid=0x41c84000 nid=0x2244 waiting on condition [0x7ff51fefd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xc5c13ae8 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:553) at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221) at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:1085) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2077) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3755) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1804) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1796) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1771) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4776) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700) at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9087) Handlers being blocked during reads
[ https://issues.apache.org/jira/browse/HBASE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725449#comment-13725449 ] Pablo Medina commented on HBASE-9087: - So you can not handle more than 50 requests at a time?. Handlers being blocked during reads --- Key: HBASE-9087 URL: https://issues.apache.org/jira/browse/HBASE-9087 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 0.94.7, 0.95.1 Reporter: Pablo Medina Assignee: Elliott Clark Priority: Critical Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9087-0.patch, HBASE-9087-1.patch I'm having a lot of handlers (90 - 300 aprox) being blocked when reading rows. They are blocked during changedReaderObserver registration. Lars Hofhansl suggests to change the implementation of changedReaderObserver from CopyOnWriteList to ConcurrentHashMap. Here is a stack trace: IPC Server handler 99 on 60020 daemon prio=10 tid=0x41c84000 nid=0x2244 waiting on condition [0x7ff51fefd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xc5c13ae8 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:553) at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221) at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:1085) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2077) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3755) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1804) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1796) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1771) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4776) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700) at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9087) Handlers being blocked during reads
[ https://issues.apache.org/jira/browse/HBASE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725461#comment-13725461 ] Pablo Medina commented on HBASE-9087: - Right. But if you have free cpu cycles and let's say that you have a high block cache hit ratio (almost all the data is in memory) you should consider increasing the handlers so you can use those free cycles to increase your performance, right?. I guess that 50 handlers in Lars scenario consumes almost all its cpu / disk bandwith. Handlers being blocked during reads --- Key: HBASE-9087 URL: https://issues.apache.org/jira/browse/HBASE-9087 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 0.94.7, 0.95.1 Reporter: Pablo Medina Assignee: Elliott Clark Priority: Critical Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9087-0.patch, HBASE-9087-1.patch I'm having a lot of handlers (90 - 300 aprox) being blocked when reading rows. They are blocked during changedReaderObserver registration. Lars Hofhansl suggests to change the implementation of changedReaderObserver from CopyOnWriteList to ConcurrentHashMap. Here is a stack trace: IPC Server handler 99 on 60020 daemon prio=10 tid=0x41c84000 nid=0x2244 waiting on condition [0x7ff51fefd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xc5c13ae8 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:553) at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221) at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:1085) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2077) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3755) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1804) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1796) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1771) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4776) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700) at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9029) Backport HBASE-8706 Some improvement in snapshot to 0.94
[ https://issues.apache.org/jira/browse/HBASE-9029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725469#comment-13725469 ] Jerry He commented on HBASE-9029: - Hi, guys Should we put this in 0.94? Backport HBASE-8706 Some improvement in snapshot to 0.94 Key: HBASE-9029 URL: https://issues.apache.org/jira/browse/HBASE-9029 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.94.9 Reporter: Jerry He Assignee: Jerry He Priority: Minor Fix For: 0.94.11 Attachments: HBase-9029-0.94.patch 'HBASE-8706 Some improvement in snapshot' has some good parameter tuning and improvement for snapshot handling, making snapshot more robust. It will nice to put it in 0.94. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9049) Generalize ServerCallable creation to support custom callables
[ https://issues.apache.org/jira/browse/HBASE-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-9049: --- Attachment: hbase-9049-trunk-v3.patch Attaching patch that fixes TestAsyncProcess locally and does a little cleanup in RpcCallerFactory (better config name, makes the configuration final). Committing today, pending HadoopQA Generalize ServerCallable creation to support custom callables -- Key: HBASE-9049 URL: https://issues.apache.org/jira/browse/HBASE-9049 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.2, 0.94.11 Reporter: Jesse Yates Assignee: Jesse Yates Attachments: hbase-9049-trunk-v0.patch, hbase-9049-trunk-v1.patch, hbase-9049-trunk-v2.patch, hbase-9049-trunk-v3.patch Currently, sever callables are instantiated via direct calls. Instead, we can use a single factory and that allows more specialized callable implementations, for instance, using a circuit-breaker pattern (or the Hystrix implementation!) to minimize attempts to contact the server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725488#comment-13725488 ] Nick Dimiduk commented on HBASE-9091: - For your mutability concerns, I'm not sure what to tell you other than that you're talking to a clojure developer :) It wasn't clear to me from the API that the class is intended to be immutable. The presence of independent setXXX methods for offset and length threw me off. I'm up for a discussion about the API here. I agree in that ByteBuffer is confusing in many cases. How does the documentation I added, describing the class's three responsibilities, inform our choices? Is that the right way to think about this class? Perhaps there's some Java trickery with which I'm less familiar that can make this class more palatable? Would thread-local variables help here? As for responsibility of consumers, that's a contract that must be maintained by consumers of the class. The methods in OrderedBytes and DataType can be made more explicit about their handling of the position marker. In many places, I'm careful to document expectations about changes the position marker or the backing array, but I can review for that. The existing prefix code assumed position didn't exist, so it's safe from this addition, at least at present. Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Sub-task Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.95.2 Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9049) Generalize ServerCallable creation to support custom callables
[ https://issues.apache.org/jira/browse/HBASE-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725507#comment-13725507 ] Hadoop QA commented on HBASE-9049: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595222/hbase-9049-trunk-v3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 6 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1 warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestSnapshotFromAdmin org.apache.hadoop.hbase.client.TestClientNoCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6544//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6544//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6544//console This message is automatically generated. Generalize ServerCallable creation to support custom callables -- Key: HBASE-9049 URL: https://issues.apache.org/jira/browse/HBASE-9049 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.2, 0.94.11 Reporter: Jesse Yates Assignee: Jesse Yates Attachments: hbase-9049-trunk-v0.patch, hbase-9049-trunk-v1.patch, hbase-9049-trunk-v2.patch, hbase-9049-trunk-v3.patch Currently, sever callables are instantiated via direct calls. Instead, we can use a single factory and that allows more specialized callable implementations, for instance, using a circuit-breaker pattern (or the Hystrix implementation!) to minimize attempts to contact the server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9102) HFile block pre-loading for large sequential scan
[ https://issues.apache.org/jira/browse/HBASE-9102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725515#comment-13725515 ] Liyin Tang commented on HBASE-9102: --- It is true that OS cached the compressed/encoded blocks and the DFSClient non-pread operation is also able to pre-load all the bytes up to that DFS block. And this feature is to pre-load (decompress/decoded) these data blocks in additional to the OS cache/disk read-ahead. Also the scan prefetch is currently implemented in the RegionScanner level. I think it is a good idea to implement some prefetch logic in the HBase client as well. HFile block pre-loading for large sequential scan - Key: HBASE-9102 URL: https://issues.apache.org/jira/browse/HBASE-9102 Project: HBase Issue Type: Improvement Affects Versions: 0.89-fb Reporter: Liyin Tang Assignee: Liyin Tang The current HBase scan model cannot take full advantage of the aggrediate disk throughput, especially for the large sequential scan cases. And for the large sequential scan, it is easy to predict what the next block to read in advance so that it can pre-load and decompress/decoded these data blocks from HDFS into block cache right before the current read point. Therefore, this jira is to optimized the large sequential scan performance by pre-loading the HFile blocks into the block cache in a stream fashion so that the scan query can read from the cache directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9087) Handlers being blocked during reads
[ https://issues.apache.org/jira/browse/HBASE-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725531#comment-13725531 ] Lars Hofhansl commented on HBASE-9087: -- That's why you make sure to have at least as many handlers as CPU threads, and few more to handle io waits. Having way more threads than CPU threads and spindles is counter productive and you're better off queuing request. Handlers being blocked during reads --- Key: HBASE-9087 URL: https://issues.apache.org/jira/browse/HBASE-9087 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 0.94.7, 0.95.1 Reporter: Pablo Medina Assignee: Elliott Clark Priority: Critical Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9087-0.patch, HBASE-9087-1.patch I'm having a lot of handlers (90 - 300 aprox) being blocked when reading rows. They are blocked during changedReaderObserver registration. Lars Hofhansl suggests to change the implementation of changedReaderObserver from CopyOnWriteList to ConcurrentHashMap. Here is a stack trace: IPC Server handler 99 on 60020 daemon prio=10 tid=0x41c84000 nid=0x2244 waiting on condition [0x7ff51fefd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xc5c13ae8 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:553) at java.util.concurrent.CopyOnWriteArraySet.add(CopyOnWriteArraySet.java:221) at org.apache.hadoop.hbase.regionserver.Store.addChangedReaderObserver(Store.java:1085) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138) at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2077) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3755) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1804) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1796) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1771) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4776) at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750) at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700) at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9085) Integration Tests fails because of bug in teardown phase where the cluster state is not being restored properly.
[ https://issues.apache.org/jira/browse/HBASE-9085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9085: - Fix Version/s: (was: 0.94.10) 0.94.11 Integration Tests fails because of bug in teardown phase where the cluster state is not being restored properly. Key: HBASE-9085 URL: https://issues.apache.org/jira/browse/HBASE-9085 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.95.0, 0.94.9, 0.94.10 Reporter: gautam Assignee: gautam Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9085.patch._0.94, HBASE-9085.patch._0.95_or_trunk I was running the following test over a Distributed Cluster: bin/hbase org.apache.hadoop.hbase.IntegrationTestsDriver IntegrationTestDataIngestSlowDeterministic The IntegrationTestingUtility.restoreCluster() is called in the teardown phase of the test. For a distributed cluster, it ends up calling DistributedHBaseCluster.restoreClusterStatus, which does the task of restoring the cluster back to original state. The restore steps done here, does not solve one specific case: When the initial HBase Master is currently down, and the current HBase Master is different from the initial one. You get into this flow: //check whether current master has changed if (!ServerName.isSameHostnameAndPort(initial.getMaster(), current.getMaster())) { . } In the above code path, the current backup masters are stopped, and the current active master is also stopped. At this point, for the aforementioned usecase, none of the Hbase Masters would be available, hence the subsequent attempts to do any operation over the cluster would fail, resulting in Test Failure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7634) Replication handling of changes to peer clusters is inefficient
[ https://issues.apache.org/jira/browse/HBASE-7634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725546#comment-13725546 ] Chris Trezzo commented on HBASE-7634: - No additional comments past JD's. +1 for trunk/0.95 Thanks Gabriel. Replication handling of changes to peer clusters is inefficient --- Key: HBASE-7634 URL: https://issues.apache.org/jira/browse/HBASE-7634 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.95.2 Reporter: Gabriel Reid Attachments: HBASE-7634.patch, HBASE-7634.v2.patch, HBASE-7634.v3.patch, HBASE-7634.v4.patch, HBASE-7634.v5.patch, HBASE-7634.v6.patch The current handling of changes to the region servers in a replication peer cluster is currently quite inefficient. The list of region servers that are being replicated to is only updated if there are a large number of issues encountered while replicating. This can cause it to take quite a while to recognize that a number of the regionserver in a peer cluster are no longer available. A potentially bigger problem is that if a replication peer cluster is started with a small number of regionservers, and then more region servers are added after replication has started, the additional region servers will never be used for replication (unless there are failures on the in-use regionservers). Part of the current issue is that the retry code in ReplicationSource#shipEdits checks a randomly-chosen replication peer regionserver (in ReplicationSource#isSlaveDown) to see if it is up after a replication write has failed on a different randonly-chosen replication peer. If the peer is seen as not down, another randomly-chosen peer is used for writing. A second part of the issue is that changes to the list of region servers in a peer cluster are not detected at all, and are only picked up if a certain number of failures have occurred when trying to ship edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9049) Generalize ServerCallable creation to support custom callables
[ https://issues.apache.org/jira/browse/HBASE-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-9049: --- Attachment: hbase-9049-trunk-v4.patch Wow, rough day. Fix for tests locally. Also includes line length and javadoc fix. Come on HadoopQA! Generalize ServerCallable creation to support custom callables -- Key: HBASE-9049 URL: https://issues.apache.org/jira/browse/HBASE-9049 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.2, 0.94.11 Reporter: Jesse Yates Assignee: Jesse Yates Attachments: hbase-9049-trunk-v0.patch, hbase-9049-trunk-v1.patch, hbase-9049-trunk-v2.patch, hbase-9049-trunk-v3.patch, hbase-9049-trunk-v4.patch Currently, sever callables are instantiated via direct calls. Instead, we can use a single factory and that allows more specialized callable implementations, for instance, using a circuit-breaker pattern (or the Hystrix implementation!) to minimize attempts to contact the server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9103) Print lines that are longer than allowed in HadoopQA output.
Jesse Yates created HBASE-9103: -- Summary: Print lines that are longer than allowed in HadoopQA output. Key: HBASE-9103 URL: https://issues.apache.org/jira/browse/HBASE-9103 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.98.0, 0.95.2, 0.94.11 Its a little annoying to not know which lines are longer - helpful to find the ones that are over. Generally, this will be a small number of lines that the formatter didn't get quite right, so massive logging statements should be rare -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8768) Improve bulk load performance by moving key value construction from map phase to reduce phase.
[ https://issues.apache.org/jira/browse/HBASE-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725593#comment-13725593 ] Hudson commented on HBASE-8768: --- SUCCESS: Integrated in HBase-TRUNK #4326 (See [https://builds.apache.org/job/HBase-TRUNK/4326/]) HBASE-8768 Improve bulk load performance by moving key value construction from map phase to reduce phase (Rajeshbabu) (tedyu: rev 1508941) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterTextMapper.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsvParser.java Improve bulk load performance by moving key value construction from map phase to reduce phase. -- Key: HBASE-8768 URL: https://issues.apache.org/jira/browse/HBASE-8768 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.98.0, 0.95.2 Attachments: HBASE-8768_v2.patch, HBASE-8768_v3.patch, HBASE-8768_v4.patch, HBase_Bulkload_Performance_Improvement.pdf ImportTSV bulkloading approach uses MapReduce framework. Existing mapper and reducer classes used by ImportTSV are TsvImporterMapper.java and PutSortReducer.java. ImportTSV tool parses the tab(by default) seperated values from the input files and Mapper class generates the PUT objects for each row using the Key value pairs created from the parsed text. PutSortReducer then uses the partions based on the regions and sorts the Put objects for each region. Overheads we can see in the above approach: == 1) keyvalue construction for each parsed value in the line adding extra data like rowkey,columnfamily,qualifier which will increase around 5x extra data to be shuffled in reduce phase. We can calculate data size to shuffled as below {code} Data to be shuffled = nl*nt*(rl+cfl+cql+vall+tsl+30) {code} If we move keyvalue construction to reduce phase we datasize to be shuffle will be which is very less compared to above. {code} Data to be shuffled = nl*nt*vall {code} nl - Number of lines in the raw file nt - Number of tabs or columns including row key. rl - row length which will be different for each line. cfl - column family length which will be different for each family cql - qualifier length tsl - timestamp length. vall- each parsed value length. 30 bytes for kv size,number of families etc. 2) In mapper side we are creating put objects by adding all keyvalues constructed for each line and in reducer we will again collect keyvalues from put and sort them. Instead we can directly create and sort keyvalues in reducer. Solution: We can improve bulk load performance by moving the key value construction from mapper to reducer so that Mapper just sends the raw text for each row to the Reducer. Reducer then parses the records for rows and create and sort the key value pairs before writing to HFiles. Conclusion: === The above suggestions will improve map phase performance by avoiding keyvalue construction and reduce phase performance by avoiding excess data to be shuffled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9034) hbase-daemon.sh swallows start up errors
[ https://issues.apache.org/jira/browse/HBASE-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725594#comment-13725594 ] Hudson commented on HBASE-9034: --- SUCCESS: Integrated in HBase-TRUNK #4326 (See [https://builds.apache.org/job/HBase-TRUNK/4326/]) HBASE-9034 hbase-daemon.sh swallows start up errors -- ADD (eclark: rev 1508944) * /hbase/trunk/bin/hbase-daemon.sh hbase-daemon.sh swallows start up errors Key: HBASE-9034 URL: https://issues.apache.org/jira/browse/HBASE-9034 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.98.0, 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9034-0.patch, HBASE-9034-ADD-1.patch, HBASE-9034-ADD.patch Fix it so that start up errors go into .out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9103) Print lines that are longer than allowed in HadoopQA output.
[ https://issues.apache.org/jira/browse/HBASE-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-9103: --- Attachment: hbase-9103-v0.patch Simple awk line that dumps to a variable. Print lines that are longer than allowed in HadoopQA output. Key: HBASE-9103 URL: https://issues.apache.org/jira/browse/HBASE-9103 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: hbase-9103-v0.patch Its a little annoying to not know which lines are longer - helpful to find the ones that are over. Generally, this will be a small number of lines that the formatter didn't get quite right, so massive logging statements should be rare -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9103) Print lines that are longer than allowed in HadoopQA output.
[ https://issues.apache.org/jira/browse/HBASE-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-9103: --- Status: Patch Available (was: Open) Print lines that are longer than allowed in HadoopQA output. Key: HBASE-9103 URL: https://issues.apache.org/jira/browse/HBASE-9103 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: hbase-9103-v0.patch Its a little annoying to not know which lines are longer - helpful to find the ones that are over. Generally, this will be a small number of lines that the formatter didn't get quite right, so massive logging statements should be rare -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8966) Add YCSB-like summary statistics to LoadTestTool
[ https://issues.apache.org/jira/browse/HBASE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-8966: -- Attachment: 8966-0.94.patch Here's an updated patch for 0.94 where the reader and writer thread pools keep separate reservoir samples using an array of AtomicLongs and then reply them into DescriptiveStatistics at the end for printing. Just messing around, don't take it too seriously. Add YCSB-like summary statistics to LoadTestTool Key: HBASE-8966 URL: https://issues.apache.org/jira/browse/HBASE-8966 Project: HBase Issue Type: Improvement Components: test Reporter: Andrew Purtell Priority: Trivial Attachments: 8966-0.94.patch, 8966-0.94.patch, 8966.patch LoadTestTool does not provide summary statistics after completion. Provide them. Make them similar to YCSB summary output for post parsing convenience. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8369) MapReduce over snapshot files
[ https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725620#comment-13725620 ] Matteo Bertozzi commented on HBASE-8369: [~enis] any update on this? or the latest patch is still trunk-v3, 94-v5 MapReduce over snapshot files - Key: HBASE-8369 URL: https://issues.apache.org/jira/browse/HBASE-8369 Project: HBase Issue Type: New Feature Components: mapreduce, snapshots Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.98.0, 0.95.2 Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch, HBASE-8369-0.94_v3.patch, HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch, HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch, HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch The idea is to add an InputFormat, which can run the mapreduce job over snapshot files directly bypassing hbase server layer. The IF is similar in usage to TableInputFormat, taking a Scan object from the user, but instead of running from an online table, it runs from a table snapshot. We do one split per region in the snapshot, and open an HRegion inside the RecordReader. A RegionScanner is used internally for doing the scan without any HRegionServer bits. Users have been asking and searching for ways to run MR jobs by reading directly from hfiles, so this allows new use cases if reading from stale data is ok: - Take snapshots periodically, and run MR jobs only on snapshots. - Export snapshots to remote hdfs cluster, run the MR jobs at that cluster without HBase cluster. - (Future use case) Combine snapshot data with online hbase data: Scan from yesterday's snapshot, but read today's data from online hbase cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9034) hbase-daemon.sh swallows start up errors
[ https://issues.apache.org/jira/browse/HBASE-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725624#comment-13725624 ] Hudson commented on HBASE-9034: --- FAILURE: Integrated in hbase-0.95 #390 (See [https://builds.apache.org/job/hbase-0.95/390/]) HBASE-9034 hbase-daemon.sh swallows start up errors -- ADD (eclark: rev 1508945) * /hbase/branches/0.95/bin/hbase-daemon.sh hbase-daemon.sh swallows start up errors Key: HBASE-9034 URL: https://issues.apache.org/jira/browse/HBASE-9034 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.98.0, 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9034-0.patch, HBASE-9034-ADD-1.patch, HBASE-9034-ADD.patch Fix it so that start up errors go into .out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9049) Generalize ServerCallable creation to support custom callables
[ https://issues.apache.org/jira/browse/HBASE-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725644#comment-13725644 ] Hadoop QA commented on HBASE-9049: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12595234/hbase-9049-trunk-v4.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 6 new or modified tests. {color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop 1.0 profile. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6545//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6545//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6545//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6545//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6545//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6545//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6545//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6545//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6545//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6545//console This message is automatically generated. Generalize ServerCallable creation to support custom callables -- Key: HBASE-9049 URL: https://issues.apache.org/jira/browse/HBASE-9049 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.2, 0.94.11 Reporter: Jesse Yates Assignee: Jesse Yates Attachments: hbase-9049-trunk-v0.patch, hbase-9049-trunk-v1.patch, hbase-9049-trunk-v2.patch, hbase-9049-trunk-v3.patch, hbase-9049-trunk-v4.patch Currently, sever callables are instantiated via direct calls. Instead, we can use a single factory and that allows more specialized callable implementations, for instance, using a circuit-breaker pattern (or the Hystrix implementation!) to minimize attempts to contact the server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9103) Print lines that are longer than allowed in HadoopQA output.
[ https://issues.apache.org/jira/browse/HBASE-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725662#comment-13725662 ] Ted Yu commented on HBASE-9103: --- What about the long lines in code generated by thrift or protobuf ? Print lines that are longer than allowed in HadoopQA output. Key: HBASE-9103 URL: https://issues.apache.org/jira/browse/HBASE-9103 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: hbase-9103-v0.patch Its a little annoying to not know which lines are longer - helpful to find the ones that are over. Generally, this will be a small number of lines that the formatter didn't get quite right, so massive logging statements should be rare -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9103) Print lines that are longer than allowed in HadoopQA output.
[ https://issues.apache.org/jira/browse/HBASE-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725666#comment-13725666 ] Jesse Yates commented on HBASE-9103: Then I guess those will show up...seems still relatively rare that we are changing those things though. Print lines that are longer than allowed in HadoopQA output. Key: HBASE-9103 URL: https://issues.apache.org/jira/browse/HBASE-9103 Project: HBase Issue Type: Improvement Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: hbase-9103-v0.patch Its a little annoying to not know which lines are longer - helpful to find the ones that are over. Generally, this will be a small number of lines that the formatter didn't get quite right, so massive logging statements should be rare -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9049) Generalize ServerCallable creation to support custom callables
[ https://issues.apache.org/jira/browse/HBASE-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-9049: --- Resolution: Fixed Release Note: Support custom RpcRetryingCaller via a configurable factory. Status: Resolved (was: Patch Available) Committed to 0.95 and 0.98. Generalize ServerCallable creation to support custom callables -- Key: HBASE-9049 URL: https://issues.apache.org/jira/browse/HBASE-9049 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.2, 0.94.11 Reporter: Jesse Yates Assignee: Jesse Yates Attachments: hbase-9049-trunk-v0.patch, hbase-9049-trunk-v1.patch, hbase-9049-trunk-v2.patch, hbase-9049-trunk-v3.patch, hbase-9049-trunk-v4.patch Currently, sever callables are instantiated via direct calls. Instead, we can use a single factory and that allows more specialized callable implementations, for instance, using a circuit-breaker pattern (or the Hystrix implementation!) to minimize attempts to contact the server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira