[jira] [Reopened] (HBASE-12762) Region with no hfiles will have the highest locality cost in LocalityCostFunction
[
https://issues.apache.org/jira/browse/HBASE-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reopened HBASE-12762:
No fix version for 0.98 was set on this issue but it was committed there.
TestShell is failing on 0.98 since this change, please see
https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/728 and
https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/729
Region with no hfiles will have the highest locality cost in
LocalityCostFunction
-
Key: HBASE-12762
URL: https://issues.apache.org/jira/browse/HBASE-12762
Project: HBase
Issue Type: Improvement
Components: Balancer
Affects Versions: 0.99.2
Reporter: cuijianwei
Assignee: cuijianwei
Priority: Minor
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12762-trunk.patch
The locality cost of region will be computed in LocalityCostFunction.cost as:
{code}
double cost() {
...
int index = -1;
for (int j = 0; j regionLocations.length; j++) {
if (regionLocations[j] = 0 regionLocations[j] == serverIndex) {
index = j;
break;
}
}
if (index 0) {
cost += 1; // == region with no hfiles will have the highest cost
} else {
cost += (double) index / (double) regionLocations.length;
}
...
}
{code}
The region with no hfiles(such as empty region) will have the highest cost
which represents the worst case that region located in the server with no
locality for hfiles. However, this might be the best case because there are
no hlogs for the region. Although the absolute cost value won't affect the
balance process, will it be more reasonable to have zero cost for such
regions? such as:
{code}
...
if (index 0) {
if (regionLocation.length 0) { // == only consider regions with
hfiles
cost += 1;
}
} else {
cost += (double) index / (double) regionLocations.length;
}
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12762) Region with no hfiles will have the highest locality cost in LocalityCostFunction
[
https://issues.apache.org/jira/browse/HBASE-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260902#comment-14260902
]
Andrew Purtell commented on HBASE-12762:
My mistake, please pardon, fix version is fine, it's (phone) operator error on
my part. Test issue is real though. :-)
Region with no hfiles will have the highest locality cost in
LocalityCostFunction
-
Key: HBASE-12762
URL: https://issues.apache.org/jira/browse/HBASE-12762
Project: HBase
Issue Type: Improvement
Components: Balancer
Affects Versions: 0.99.2
Reporter: cuijianwei
Assignee: cuijianwei
Priority: Minor
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12762-trunk.patch
The locality cost of region will be computed in LocalityCostFunction.cost as:
{code}
double cost() {
...
int index = -1;
for (int j = 0; j regionLocations.length; j++) {
if (regionLocations[j] = 0 regionLocations[j] == serverIndex) {
index = j;
break;
}
}
if (index 0) {
cost += 1; // == region with no hfiles will have the highest cost
} else {
cost += (double) index / (double) regionLocations.length;
}
...
}
{code}
The region with no hfiles(such as empty region) will have the highest cost
which represents the worst case that region located in the server with no
locality for hfiles. However, this might be the best case because there are
no hlogs for the region. Although the absolute cost value won't affect the
balance process, will it be more reasonable to have zero cost for such
regions? such as:
{code}
...
if (index 0) {
if (regionLocation.length 0) { // == only consider regions with
hfiles
cost += 1;
}
} else {
cost += (double) index / (double) regionLocations.length;
}
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12772) TestPerColumnFamilyFlush failing
[
https://issues.apache.org/jira/browse/HBASE-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12772:
--
Attachment: 12772.addendum.txt
Flush the namespace table before we start adding edits so its hanging edit
won't get in way of test.
Rather than wait 4 seconds, wait on the WAL log count to change (should change
after successful flush).
Add timeouts on all tests.
Unable to test locally. Trying against hadoopqa.
TestPerColumnFamilyFlush failing
Key: HBASE-12772
URL: https://issues.apache.org/jira/browse/HBASE-12772
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 1.0.0
Reporter: stack
Attachments: 0001-HBASE-12772-TestPerColumnFamilyFlush-failing.patch,
12772.addendum.txt
On internal rig see this failing in two places:
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling
Failing for the past 1 build (Since Failed#653 )
Took 9 sec.
Error Message
expected:424 but was:205744
Stacktrace
java.lang.AssertionError: expected:424 but was:205744
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling(TestPerColumnFamilyFlush.java:483)
{code}
and
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay
Failing for the past 1 build (Since Failed#653 )
Took 25 ms.
Error Message
A mini-cluster is already running
Stacktrace
java.lang.IllegalStateException: A mini-cluster is already running
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:921)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:812)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:794)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:781)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplay(TestPerColumnFamilyFlush.java:337)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay(TestPerColumnFamilyFlush.java:418)
{code}
Opening issue to keep an eye on these tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12772) TestPerColumnFamilyFlush failing
[
https://issues.apache.org/jira/browse/HBASE-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12772:
--
Assignee: stack
Status: Patch Available (was: Open)
TestPerColumnFamilyFlush failing
Key: HBASE-12772
URL: https://issues.apache.org/jira/browse/HBASE-12772
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 1.0.0
Reporter: stack
Assignee: stack
Attachments: 0001-HBASE-12772-TestPerColumnFamilyFlush-failing.patch,
12772.addendum.txt
On internal rig see this failing in two places:
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling
Failing for the past 1 build (Since Failed#653 )
Took 9 sec.
Error Message
expected:424 but was:205744
Stacktrace
java.lang.AssertionError: expected:424 but was:205744
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling(TestPerColumnFamilyFlush.java:483)
{code}
and
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay
Failing for the past 1 build (Since Failed#653 )
Took 25 ms.
Error Message
A mini-cluster is already running
Stacktrace
java.lang.IllegalStateException: A mini-cluster is already running
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:921)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:812)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:794)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:781)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplay(TestPerColumnFamilyFlush.java:337)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay(TestPerColumnFamilyFlush.java:418)
{code}
Opening issue to keep an eye on these tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260911#comment-14260911
]
Andrew Purtell commented on HBASE-12728:
My 0.02, it shouldn't be too late for 1.0. Table is a fundamental API and we
should get it right for a 1.0 release. Let things slip if need be IMHO
buffered writes substantially less useful after removal of HTablePool
-
Key: HBASE-12728
URL: https://issues.apache.org/jira/browse/HBASE-12728
Project: HBase
Issue Type: Bug
Components: hbase
Affects Versions: 0.98.0
Reporter: Aaron Beppu
In previous versions of HBase, when use of HTablePool was encouraged, HTable
instances were long-lived in that pool, and for that reason, if autoFlush was
set to false, the table instance could accumulate a full buffer of writes
before a flush was triggered. Writes from the client to the cluster could
then be substantially larger and less frequent than without buffering.
However, when HTablePool was deprecated, the primary justification seems to
have been that creating HTable instances is cheap, so long as the connection
and executor service being passed to it are pre-provided. A use pattern was
encouraged where users should create a new HTable instance for every
operation, using an existing connection and executor service, and then close
the table. In this pattern, buffered writes are substantially less useful;
writes are as small and as frequent as they would have been with
autoflush=true, except the synchronous write is moved from the operation
itself to the table close call which immediately follows.
More concretely :
```
// Given these two helpers ...
private HTableInterface getAutoFlushTable(String tableName) throws
IOException {
// (autoflush is true by default)
return storedConnection.getTable(tableName, executorService);
}
private HTableInterface getBufferedTable(String tableName) throws IOException
{
HTableInterface table = getAutoFlushTable(tableName);
table.setAutoFlush(false);
return table;
}
// it's my contention that these two methods would behave almost identically,
// except the first will hit a synchronous flush during the put call,
and the second will
// flush during the (hidden) close call on table.
private void writeAutoFlushed(Put somePut) throws IOException {
try (HTableInterface table = getAutoFlushTable(tableName)) {
table.put(somePut); // will do synchronous flush
}
}
private void writeBuffered(Put somePut) throws IOException {
try (HTableInterface table = getBufferedTable(tableName)) {
table.put(somePut);
} // auto-close will trigger synchronous flush
}
```
For buffered writes to actually provide a performance benefit to users, one
of two things must happen:
- The writeBuffer itself shouldn't live, flush and die with the lifecycle of
it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
lifespan, this could cease to be an issue. However, if the same writeBuffer
is appended to by multiple tables, then some additional concurrency control
will be needed around it.
- Alternatively, there should be some pattern for having long-lived HTable
instances. However, since HTable is not thread-safe, we'd need multiple
instances, and a mechanism for leasing them out safely -- which sure sounds a
lot like the old HTablePool to me.
See discussion on mailing list here :
http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260920#comment-14260920
]
Andrew Purtell commented on HBASE-12728:
FWIW, I recently rewrote the YCSB client for HBase because it was setting auto
flush to off and using large write buffers to collect puts. Unfortunately
although producing 'excellent' write latency measurements this lead to multiple
threads flushing deep buffers more or less at the same time, resulting in long
periods of write unavailability. I'm sure that was an unintended consequence. I
believe that YCSB client code was written by HBase devs. An earlier era in any
case, but my point is devs very familiar with the API can get into trouble
never mind newcomers. Removing buffering from Table and moving it into
BufferedTable with suitable advice in javadoc there sounds like a good idea to
me.
buffered writes substantially less useful after removal of HTablePool
-
Key: HBASE-12728
URL: https://issues.apache.org/jira/browse/HBASE-12728
Project: HBase
Issue Type: Bug
Components: hbase
Affects Versions: 0.98.0
Reporter: Aaron Beppu
In previous versions of HBase, when use of HTablePool was encouraged, HTable
instances were long-lived in that pool, and for that reason, if autoFlush was
set to false, the table instance could accumulate a full buffer of writes
before a flush was triggered. Writes from the client to the cluster could
then be substantially larger and less frequent than without buffering.
However, when HTablePool was deprecated, the primary justification seems to
have been that creating HTable instances is cheap, so long as the connection
and executor service being passed to it are pre-provided. A use pattern was
encouraged where users should create a new HTable instance for every
operation, using an existing connection and executor service, and then close
the table. In this pattern, buffered writes are substantially less useful;
writes are as small and as frequent as they would have been with
autoflush=true, except the synchronous write is moved from the operation
itself to the table close call which immediately follows.
More concretely :
```
// Given these two helpers ...
private HTableInterface getAutoFlushTable(String tableName) throws
IOException {
// (autoflush is true by default)
return storedConnection.getTable(tableName, executorService);
}
private HTableInterface getBufferedTable(String tableName) throws IOException
{
HTableInterface table = getAutoFlushTable(tableName);
table.setAutoFlush(false);
return table;
}
// it's my contention that these two methods would behave almost identically,
// except the first will hit a synchronous flush during the put call,
and the second will
// flush during the (hidden) close call on table.
private void writeAutoFlushed(Put somePut) throws IOException {
try (HTableInterface table = getAutoFlushTable(tableName)) {
table.put(somePut); // will do synchronous flush
}
}
private void writeBuffered(Put somePut) throws IOException {
try (HTableInterface table = getBufferedTable(tableName)) {
table.put(somePut);
} // auto-close will trigger synchronous flush
}
```
For buffered writes to actually provide a performance benefit to users, one
of two things must happen:
- The writeBuffer itself shouldn't live, flush and die with the lifecycle of
it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
lifespan, this could cease to be an issue. However, if the same writeBuffer
is appended to by multiple tables, then some additional concurrency control
will be needed around it.
- Alternatively, there should be some pattern for having long-lived HTable
instances. However, since HTable is not thread-safe, we'd need multiple
instances, and a mechanism for leasing them out safely -- which sure sounds a
lot like the old HTablePool to me.
See discussion on mailing list here :
http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12768) Support enable cache_data_on_write in Shell while creating table
[ https://issues.apache.org/jira/browse/HBASE-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260923#comment-14260923 ] Andrew Purtell commented on HBASE-12768: +1 for 0.98 Support enable cache_data_on_write in Shell while creating table Key: HBASE-12768 URL: https://issues.apache.org/jira/browse/HBASE-12768 Project: HBase Issue Type: Improvement Components: shell Affects Versions: 1.0.0, 2.0.0, 0.94.27 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 1.0.0, 2.0.0, 0.98.10 Attachments: HBASE-12768.patch A simple approach to support cache_data_on_write while creating table in shell. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12768) Support enable cache_data_on_write in Shell while creating table
[ https://issues.apache.org/jira/browse/HBASE-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260926#comment-14260926 ] ramkrishna.s.vasudevan commented on HBASE-12768: Pushed to 0.98. Thanks @apurtell. Support enable cache_data_on_write in Shell while creating table Key: HBASE-12768 URL: https://issues.apache.org/jira/browse/HBASE-12768 Project: HBase Issue Type: Improvement Components: shell Affects Versions: 1.0.0, 2.0.0, 0.94.27 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 1.0.0, 2.0.0, 0.98.10 Attachments: HBASE-12768.patch A simple approach to support cache_data_on_write while creating table in shell. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12772) TestPerColumnFamilyFlush failing
[
https://issues.apache.org/jira/browse/HBASE-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260959#comment-14260959
]
Hadoop QA commented on HBASE-12772:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689451/12772.addendum.txt
against master branch at commit 0513a21dc8b86f57b5a6c1b742904821632f77f7.
ATTACHMENT ID: 12689451
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12254//console
This message is automatically generated.
TestPerColumnFamilyFlush failing
Key: HBASE-12772
URL: https://issues.apache.org/jira/browse/HBASE-12772
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 1.0.0
Reporter: stack
Assignee: stack
Attachments: 0001-HBASE-12772-TestPerColumnFamilyFlush-failing.patch,
12772.addendum.txt
On internal rig see this failing in two places:
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling
Failing for the past 1 build (Since Failed#653 )
Took 9 sec.
Error Message
expected:424 but was:205744
Stacktrace
java.lang.AssertionError: expected:424 but was:205744
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling(TestPerColumnFamilyFlush.java:483)
{code}
and
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay
Failing for the past 1 build (Since
[jira] [Commented] (HBASE-12775) CompressionTest ate my HFile (sigh!)
[
https://issues.apache.org/jira/browse/HBASE-12775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260963#comment-14260963
]
Aditya Kishore commented on HBASE-12775:
I agree but was trying to be consistent with what is already there. The program
exits with a SUCCESS on System.out if successful (though it does LOG in other
places).
May be replace all sysout and syserr with LOG.error() and LOG.info()?
CompressionTest ate my HFile (sigh!)
Key: HBASE-12775
URL: https://issues.apache.org/jira/browse/HBASE-12775
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.98.9, 0.99.2
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Fix For: 2.0.0
Attachments: HBASE-12775-CompressionTest-ate-my-HFile.patch
{{org.apache.hadoop.hbase.util.CompressionTest}} should abort execution if
the file specified on the command line exists. This will help careless (me)
or unsuspecting user to not loose data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12684: -- Attachment: HBASE-12684-DEBUG1.patch [~stack] Thanks! That file was exactly what I needed. :) Didn't know about that page with the build artifacts. It seems the fail is caused is a NullPointer in an area I thought could never be null. Created a new debug patch which probably fixes it and added a Netty ResourceLeakDetector since the log was complaining of ByteBuf leaks which I didn't have locally. Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG1.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[
https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260991#comment-14260991
]
Hadoop QA commented on HBASE-12684:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689481/HBASE-12684-DEBUG1.patch
against master branch at commit 0513a21dc8b86f57b5a6c1b742904821632f77f7.
ATTACHMENT ID: 12689481
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java:[403,35]
cannot find symbol
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:testCompile
(default-testCompile) on project hbase-server: Compilation failure
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java:[403,35]
cannot find symbol
[ERROR] symbol: variable done
[ERROR] location: class org.apache.hadoop.hbase.client.TestHCM
[ERROR] - [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn goals -rf :hbase-server
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12255//console
This message is automatically generated.
Add new AsyncRpcClient
--
Key: HBASE-12684
URL: https://issues.apache.org/jira/browse/HBASE-12684
Project: HBase
Issue Type: Improvement
Components: Client
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12684-DEBUG1.patch, HBASE-12684-v1.patch,
HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch,
HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch,
HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v2.patch,
HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch,
HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch,
HBASE-12684-v9.patch, HBASE-12684.patch
With the changes in HBASE-12597 it is possible to add new RpcClients. This
issue is about adding a new Async RpcClient which would enable HBase to do
non blocking protobuf service communication.
Besides delivering a new AsyncRpcClient I would also like to ask the question
what it would take to replace the current RpcClient? This would enable to
simplify async code in some next issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12684: -- Attachment: (was: HBASE-12684-DEBUG1.patch) Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12684: -- Attachment: HBASE-12684-DEBUG2.patch Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12768) Support enable cache_data_on_write in Shell while creating table
[ https://issues.apache.org/jira/browse/HBASE-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261015#comment-14261015 ] Hudson commented on HBASE-12768: SUCCESS: Integrated in HBase-0.98 #764 (See [https://builds.apache.org/job/HBase-0.98/764/]) HBASE-12768 - Support enable cache_data_on_write in Shell while creating (ramkrishna: rev 54b6830fb4f9c33f3cd35694598694ffcd39ade4) * hbase-shell/src/main/ruby/hbase/admin.rb Support enable cache_data_on_write in Shell while creating table Key: HBASE-12768 URL: https://issues.apache.org/jira/browse/HBASE-12768 Project: HBase Issue Type: Improvement Components: shell Affects Versions: 1.0.0, 2.0.0, 0.94.27 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 1.0.0, 2.0.0, 0.98.10 Attachments: HBASE-12768.patch A simple approach to support cache_data_on_write while creating table in shell. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12761) On region jump ClientScanners should get next row start key instead of a skip.
[ https://issues.apache.org/jira/browse/HBASE-12761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12761: -- Attachment: (was: HBASE-12761-v2.patch) On region jump ClientScanners should get next row start key instead of a skip. -- Key: HBASE-12761 URL: https://issues.apache.org/jira/browse/HBASE-12761 Project: HBase Issue Type: Improvement Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12761-v1.patch, HBASE-12761-v2.patch, HBASE-12761.patch While working on async scanner I had some trouble with the extra RPC calls that happen to let the Scanner advance 1 row so it skips the last already known row. This RPC call can be avoided by letting the start key be the last row with an appended 0. This saves quite some logic from the scanners and improves performance by saving extra RPC calls. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12740: -- Attachment: PROFILE_before_patch_test_fails.png PROFILE_after_patch.png [~dimaspivak] The main reason why I opened this issue was that this test crashes for me because it reaches my machines Java default max thread count of 2000. (I tried various stack overflow suggested options to raise it with no success) It caused many of these exceptions: java.lang.OutOfMemoryError: unable to create new native thread So I tried to lower this thread count to make it runnable so I could fix issues with the async client that this test was somehow pointing out. I see my very long runs were those which would succeed in the tests because HBase recovered of the exceptions by retrying and gave the server some time to timeout some connections. Those runs took minutes longer and when I fixed what causes it the test would run in around the same numbers you post. I think that the few milliseconds regression in your comparison is because the test now properly closes connections and regions and that takes up a bit more time. I have included 2 screenshots of profiling of the failing run with too much thread creation and the succeeding run with much less thread creation. Is it ok to still commit this patch so the test is runnable for those with lower thread limits? Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740.patch, PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12768) Support enable cache_data_on_write in Shell while creating table
[ https://issues.apache.org/jira/browse/HBASE-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261041#comment-14261041 ] Hudson commented on HBASE-12768: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #730 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/730/]) HBASE-12768 - Support enable cache_data_on_write in Shell while creating (ramkrishna: rev 54b6830fb4f9c33f3cd35694598694ffcd39ade4) * hbase-shell/src/main/ruby/hbase/admin.rb Support enable cache_data_on_write in Shell while creating table Key: HBASE-12768 URL: https://issues.apache.org/jira/browse/HBASE-12768 Project: HBase Issue Type: Improvement Components: shell Affects Versions: 1.0.0, 2.0.0, 0.94.27 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 1.0.0, 2.0.0, 0.98.10 Attachments: HBASE-12768.patch A simple approach to support cache_data_on_write while creating table in shell. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[
https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261067#comment-14261067
]
Hadoop QA commented on HBASE-12684:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689483/HBASE-12684-DEBUG2.patch
against master branch at commit 0513a21dc8b86f57b5a6c1b742904821632f77f7.
ATTACHMENT ID: 12689483
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12256//console
This message is automatically generated.
Add new AsyncRpcClient
--
Key: HBASE-12684
URL: https://issues.apache.org/jira/browse/HBASE-12684
Project: HBase
Issue Type: Improvement
Components: Client
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-v1.patch,
HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch,
HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch,
HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v2.patch,
HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch,
HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch,
HBASE-12684-v9.patch, HBASE-12684.patch
With the changes in HBASE-12597 it is possible to add new RpcClients. This
issue is about adding a new Async RpcClient which would enable HBase to do
non blocking protobuf service communication.
Besides delivering a new AsyncRpcClient I would also like to ask the question
what it would take to replace the current RpcClient? This would enable to
simplify async code in some next issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12781) Listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-12781:
-
Attachment: 12781-V1.patch
Patch for review.
Listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
--
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
-
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12781) Listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-12781:
-
Status: Patch Available (was: Open)
Listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
--
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
-
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12684: -- Attachment: HBASE-12684-DEBUG3.patch Debug patch 3 -Some logs for debugging (Although no test fails are reported it seems to have failed in the logs) -Probable fix for the ByteBuf leak. Maybe this was the cause? Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
[ https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-12270: Attachment: HBASE-12270-v2.diff Add tests for caching block on write with different block caches. [~stack] Add more tests if you wish. Thanks A bug in the bucket cache, with cache blocks on write enabled - Key: HBASE-12270 URL: https://issues.apache.org/jira/browse/HBASE-12270 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6.1 Environment: I can reproduce it on a simple 2 node cluster, one running the master and another running a RS. I was testing on ec2. I used the following configurations for the cluster. hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog gc:/tmp/hbase-regionserver-gc.log hbase-site: hbase.bucketcache.ioengine=offheap hbase.bucketcache.size=4196 hbase.rs.cacheblocksonwrite=true hfile.block.index.cacheonwrite=true hfile.block.bloom.cacheonwrite=true Reporter: Khaled Elmeleegy Assignee: Liu Shaohui Priority: Critical Fix For: 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12270-v1.diff, HBASE-12270-v2.diff, TestHBase.java, TestKey.java In my experiments, I have writers streaming their output to HBase. The reader powers a web page and does this scatter/gather, where it reads 1000 keys written last and passes them the the front end. With this workload, I get the exception below at the region server. Again, I am using HBAse (0.98.6.1). Any help is appreciated. 2014-10-10 15:06:44,173 ERROR [B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:236) at org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:176) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3758) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913) at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12783) Create efficient RegionLocator implementation
Solomon Duskis created HBASE-12783: -- Summary: Create efficient RegionLocator implementation Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261125#comment-14261125 ] Solomon Duskis commented on HBASE-12783: For now, I'm simply creating a new implementation of RegionLocator. We can slowly remove the implementation from HTable. Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12783.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12783: --- Attachment: HBASE-12783.patch Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12783.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261126#comment-14261126 ] Solomon Duskis commented on HBASE-12783: Hm... I guess I forgot tests. I'll work on that. Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12783.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261147#comment-14261147
]
Solomon Duskis commented on HBASE-12728:
Is HTableMultiplexer a good existing solution to replace the functionality of
HTable without autoflush?
buffered writes substantially less useful after removal of HTablePool
-
Key: HBASE-12728
URL: https://issues.apache.org/jira/browse/HBASE-12728
Project: HBase
Issue Type: Bug
Components: hbase
Affects Versions: 0.98.0
Reporter: Aaron Beppu
In previous versions of HBase, when use of HTablePool was encouraged, HTable
instances were long-lived in that pool, and for that reason, if autoFlush was
set to false, the table instance could accumulate a full buffer of writes
before a flush was triggered. Writes from the client to the cluster could
then be substantially larger and less frequent than without buffering.
However, when HTablePool was deprecated, the primary justification seems to
have been that creating HTable instances is cheap, so long as the connection
and executor service being passed to it are pre-provided. A use pattern was
encouraged where users should create a new HTable instance for every
operation, using an existing connection and executor service, and then close
the table. In this pattern, buffered writes are substantially less useful;
writes are as small and as frequent as they would have been with
autoflush=true, except the synchronous write is moved from the operation
itself to the table close call which immediately follows.
More concretely :
```
// Given these two helpers ...
private HTableInterface getAutoFlushTable(String tableName) throws
IOException {
// (autoflush is true by default)
return storedConnection.getTable(tableName, executorService);
}
private HTableInterface getBufferedTable(String tableName) throws IOException
{
HTableInterface table = getAutoFlushTable(tableName);
table.setAutoFlush(false);
return table;
}
// it's my contention that these two methods would behave almost identically,
// except the first will hit a synchronous flush during the put call,
and the second will
// flush during the (hidden) close call on table.
private void writeAutoFlushed(Put somePut) throws IOException {
try (HTableInterface table = getAutoFlushTable(tableName)) {
table.put(somePut); // will do synchronous flush
}
}
private void writeBuffered(Put somePut) throws IOException {
try (HTableInterface table = getBufferedTable(tableName)) {
table.put(somePut);
} // auto-close will trigger synchronous flush
}
```
For buffered writes to actually provide a performance benefit to users, one
of two things must happen:
- The writeBuffer itself shouldn't live, flush and die with the lifecycle of
it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
lifespan, this could cease to be an issue. However, if the same writeBuffer
is appended to by multiple tables, then some additional concurrency control
will be needed around it.
- Alternatively, there should be some pattern for having long-lived HTable
instances. However, since HTable is not thread-safe, we'd need multiple
instances, and a mechanism for leasing them out safely -- which sure sounds a
lot like the old HTablePool to me.
See discussion on mailing list here :
http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12781) Listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261170#comment-14261170
]
Hadoop QA commented on HBASE-12781:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689499/12781-V1.patch
against master branch at commit 0513a21dc8b86f57b5a6c1b742904821632f77f7.
ATTACHMENT ID: 12689499
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12258//console
This message is automatically generated.
Listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
--
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
-
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
In case when bind address is not passed through argument then it is binding
with any local address. It
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[
https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261178#comment-14261178
]
Hadoop QA commented on HBASE-12684:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689500/HBASE-12684-DEBUG3.patch
against master branch at commit 0513a21dc8b86f57b5a6c1b742904821632f77f7.
ATTACHMENT ID: 12689500
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12259//console
This message is automatically generated.
Add new AsyncRpcClient
--
Key: HBASE-12684
URL: https://issues.apache.org/jira/browse/HBASE-12684
Project: HBase
Issue Type: Improvement
Components: Client
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch,
HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch,
HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch,
HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch,
HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch,
HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch,
HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch
With the changes in HBASE-12597 it is possible to add new RpcClients. This
issue is about adding a new Async RpcClient which would enable HBase to do
non blocking protobuf service communication.
Besides delivering a new AsyncRpcClient I would also like to ask the question
what it would take to replace the current RpcClient? This would enable to
simplify async code in some next issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12781) Listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261184#comment-14261184
]
Ted Yu commented on HBASE-12781:
+1
Listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
--
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
-
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12684: -- Attachment: HBASE-12684-v17.patch Wahoo! Retry Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12684: -- Attachment: HBASE-12684-v18.patch Undid all the TestHCM and debug changes from the debug patch since it seems the root cause of all troubles is found. Lets see if it passes with the original timing. Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261193#comment-14261193 ] Jurriaan Mous commented on HBASE-12684: --- I submitted patch v18 with the fix :) Patch v17 will fail. It was DEBUG3 which was succeeding :) Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
[
https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261203#comment-14261203
]
Hadoop QA commented on HBASE-12270:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689504/HBASE-12270-v2.diff
against master branch at commit 0513a21dc8b86f57b5a6c1b742904821632f77f7.
ATTACHMENT ID: 12689504
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.sling.commons.testing.integration.SlingIntegrationTestClient.createNode(SlingIntegrationTestClient.java:186)
at
org.apache.sling.commons.testing.integration.SlingIntegrationTestClient.createNode(SlingIntegrationTestClient.java:119)
at
org.apache.sling.commons.testing.integration.SlingIntegrationTestClient.createNode(SlingIntegrationTestClient.java:109)
at
org.apache.sling.commons.testing.integration.HttpTestBase.slingServerReady(HttpTestBase.java:274)
at
org.apache.sling.commons.testing.integration.HttpTestBase.waitForSlingStartup(HttpTestBase.java:226)
at
org.apache.sling.commons.testing.integration.HttpTestBase.setUp(HttpTestBase.java:163)
at
org.apache.sling.launchpad.webapp.integrationtest.indexing.FullTextIndexingTest.setUp(FullTextIndexingTest.java:87)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12260//console
This message is automatically generated.
A bug in the bucket cache, with cache blocks on write enabled
-
Key: HBASE-12270
URL: https://issues.apache.org/jira/browse/HBASE-12270
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.11, 0.98.6.1
Environment: I can reproduce it on a simple 2 node cluster, one
running the master and another running a RS. I was testing on ec2.
I used the following configurations
[jira] [Updated] (HBASE-12772) TestPerColumnFamilyFlush failing
[
https://issues.apache.org/jira/browse/HBASE-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12772:
--
Attachment: 12772.addendum.v2.txt
Forgot to set new WAL upper bound globally.
TestPerColumnFamilyFlush failing
Key: HBASE-12772
URL: https://issues.apache.org/jira/browse/HBASE-12772
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 1.0.0
Reporter: stack
Assignee: stack
Attachments: 0001-HBASE-12772-TestPerColumnFamilyFlush-failing.patch,
12772.addendum.txt, 12772.addendum.v2.txt
On internal rig see this failing in two places:
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling
Failing for the past 1 build (Since Failed#653 )
Took 9 sec.
Error Message
expected:424 but was:205744
Stacktrace
java.lang.AssertionError: expected:424 but was:205744
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling(TestPerColumnFamilyFlush.java:483)
{code}
and
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay
Failing for the past 1 build (Since Failed#653 )
Took 25 ms.
Error Message
A mini-cluster is already running
Stacktrace
java.lang.IllegalStateException: A mini-cluster is already running
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:921)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:812)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:794)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:781)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplay(TestPerColumnFamilyFlush.java:337)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay(TestPerColumnFamilyFlush.java:418)
{code}
Opening issue to keep an eye on these tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
[ https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12270: -- Attachment: HBASE-12270-v2.patch Zombie found tests from other projects. Retry. A bug in the bucket cache, with cache blocks on write enabled - Key: HBASE-12270 URL: https://issues.apache.org/jira/browse/HBASE-12270 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6.1 Environment: I can reproduce it on a simple 2 node cluster, one running the master and another running a RS. I was testing on ec2. I used the following configurations for the cluster. hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog gc:/tmp/hbase-regionserver-gc.log hbase-site: hbase.bucketcache.ioengine=offheap hbase.bucketcache.size=4196 hbase.rs.cacheblocksonwrite=true hfile.block.index.cacheonwrite=true hfile.block.bloom.cacheonwrite=true Reporter: Khaled Elmeleegy Assignee: Liu Shaohui Priority: Critical Fix For: 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12270-v1.diff, HBASE-12270-v2.diff, HBASE-12270-v2.patch, TestHBase.java, TestKey.java In my experiments, I have writers streaming their output to HBase. The reader powers a web page and does this scatter/gather, where it reads 1000 keys written last and passes them the the front end. With this workload, I get the exception below at the region server. Again, I am using HBAse (0.98.6.1). Any help is appreciated. 2014-10-10 15:06:44,173 ERROR [B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:236) at org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:176) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3758) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913) at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261218#comment-14261218 ] stack commented on HBASE-12684: --- [~jurmous] Let me get a few perf runs in today. There is no config to turn it 'on' is there? Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12781) thrift2 listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12781:
---
Summary: thrift2 listen port will bind always to the passed command line
address (was: Listen port will bind always to the passed command line address)
thrift2 listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
--
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
-
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12781) thrift2 listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12781:
---
Fix Version/s: 1.1.0
0.98.10
2.0.0
Hadoop Flags: Reviewed
thrift2 listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 2.0.0, 0.98.10, 1.1.0
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
--
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
-
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12781) thrift2 listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12781:
---
Description:
In Thrift server, listen port will bind always to the address which is passed
through command line argument.
{code}
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
{code}
{code}
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
{code}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
was:
In Thrift server, listen port will bind always to the address which is passed
through command line argument.
--
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
-
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
thrift2 listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 2.0.0, 0.98.10, 1.1.0
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
{code}
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
{code}
{code}
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
{code}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12781) thrift2 listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12781:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the patch, Pankaj
thrift2 listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 2.0.0, 0.98.10, 1.1.0
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
{code}
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
{code}
{code}
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
{code}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12762) Region with no hfiles will have the highest locality cost in LocalityCostFunction
[
https://issues.apache.org/jira/browse/HBASE-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261228#comment-14261228
]
stack commented on HBASE-12762:
---
[~apurtell] I took a look and saw this:
14/12/30 07:49:47 INFO mortbay.log: Extract
jar:file:/home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/hbase-server/target/hbase-server-0.98.10-SNAPSHOT.jar!/hbase-webapps/regionserver
to /tmp/Jetty_0_0_0_0_38973_regionserver6v0l18/webapp
14/12/30 07:49:47 INFO mortbay.log: Started SelectChannelConnector@0.0.0.0:38973
14/12/30 07:49:47 INFO http.HttpServer: Added global filtersafety
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
14/12/30 07:49:47 INFO http.HttpServer: Port returned by
webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the
listener on 60010
14/12/30 07:49:47 FATAL master.HMaster: Unhandled exception. Starting shutdown.
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at
org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:602)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:666)
at java.lang.Thread.run(Thread.java:662)
And it just passed. Let me try it again to see if permanent failing going on.
Region with no hfiles will have the highest locality cost in
LocalityCostFunction
-
Key: HBASE-12762
URL: https://issues.apache.org/jira/browse/HBASE-12762
Project: HBase
Issue Type: Improvement
Components: Balancer
Affects Versions: 0.99.2
Reporter: cuijianwei
Assignee: cuijianwei
Priority: Minor
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12762-trunk.patch
The locality cost of region will be computed in LocalityCostFunction.cost as:
{code}
double cost() {
...
int index = -1;
for (int j = 0; j regionLocations.length; j++) {
if (regionLocations[j] = 0 regionLocations[j] == serverIndex) {
index = j;
break;
}
}
if (index 0) {
cost += 1; // == region with no hfiles will have the highest cost
} else {
cost += (double) index / (double) regionLocations.length;
}
...
}
{code}
The region with no hfiles(such as empty region) will have the highest cost
which represents the worst case that region located in the server with no
locality for hfiles. However, this might be the best case because there are
no hlogs for the region. Although the absolute cost value won't affect the
balance process, will it be more reasonable to have zero cost for such
regions? such as:
{code}
...
if (index 0) {
if (regionLocation.length 0) { // == only consider regions with
hfiles
cost += 1;
}
} else {
cost += (double) index / (double) regionLocations.length;
}
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261229#comment-14261229 ] Ted Yu commented on HBASE-12684: Great work, Jurriaan. Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
[ https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261239#comment-14261239 ] Ted Yu commented on HBASE-12270: +1 A bug in the bucket cache, with cache blocks on write enabled - Key: HBASE-12270 URL: https://issues.apache.org/jira/browse/HBASE-12270 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6.1 Environment: I can reproduce it on a simple 2 node cluster, one running the master and another running a RS. I was testing on ec2. I used the following configurations for the cluster. hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog gc:/tmp/hbase-regionserver-gc.log hbase-site: hbase.bucketcache.ioengine=offheap hbase.bucketcache.size=4196 hbase.rs.cacheblocksonwrite=true hfile.block.index.cacheonwrite=true hfile.block.bloom.cacheonwrite=true Reporter: Khaled Elmeleegy Assignee: Liu Shaohui Priority: Critical Fix For: 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12270-v1.diff, HBASE-12270-v2.diff, HBASE-12270-v2.patch, TestHBase.java, TestKey.java In my experiments, I have writers streaming their output to HBase. The reader powers a web page and does this scatter/gather, where it reads 1000 keys written last and passes them the the front end. With this workload, I get the exception below at the region server. Again, I am using HBAse (0.98.6.1). Any help is appreciated. 2014-10-10 15:06:44,173 ERROR [B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:236) at org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:176) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3758) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913) at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11869) Support snapshot owner
[ https://issues.apache.org/jira/browse/HBASE-11869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-11869: --- Resolution: Fixed Status: Resolved (was: Patch Available) This was integrated 3 months ago. Support snapshot owner -- Key: HBASE-11869 URL: https://issues.apache.org/jira/browse/HBASE-11869 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 Attachments: HBASE-11869-trunk-v1.diff, HBASE-11869-trunk-v3.diff, HBASE-11869-trunk-v4.diff In current codebase, the table snapshot operations only can be done by the global admin , not by the table admin. There is a multi-tenant hbase cluster, each table has different snapshot policies, eg: do snapshot per week, or snapshot after the new data are imported. We want to release the snapshot permission to each table admin. According to [~mbertozzi]'s suggestion, we implement the snapshot owner feature. * The user with table admin permission can create snapshot and the owner of this snapshot is this user. * The owner of snapshot can delete and restore the snapshot. * Only the user with global admin permission can clone a snapshot, for this operation creates a new table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261261#comment-14261261 ] Jurriaan Mous commented on HBASE-12684: --- In the current patch it is on by default because AsyncRpcClient is the default in RpcClientFactory. This way all the tests that use RpcClientFactory would use this new client. (The ones that don't have AsyncRpcClient duplicates) What the future default is is to be decided. It is configurable with hbase.rpc.client.impl. I am curious what the results are! You could possibly try out some thread pool tweaking with hbase.rpc.client.threads.max. It is by default 0 to let Netty decide. (Which is: Runtime.getRuntime().availableProcessors() * 2) Size of 1 made some tests fail. It seems in profiling that the new embedded thread pool ups the thread count a bit. But the total can come done in the future as the AsyncProcess could move to Netty channel threads. I did get some good runtime and CPU improvements on a few tests but I don't know what that represents in a normal cluster run. Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12784) TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes hangs
Ted Yu created HBASE-12784:
--
Summary:
TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes
hangs
Key: HBASE-12784
URL: https://issues.apache.org/jira/browse/HBASE-12784
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
I was running test suite against hadoop 2.7.0-SNAPSHOT and saw this:
{code}
testPreemptiveFastFailException50Times(org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil)
Time elapsed: 120.013 sec ERROR!
java.lang.Exception: test timed out after 12 milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException(TestFastFailWithoutTestUtil.java:450)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException50Times(TestFastFailWithoutTestUtil.java:338)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12784) TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes hangs
[
https://issues.apache.org/jira/browse/HBASE-12784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12784:
---
Attachment:
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil-output.txt
TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes
hangs
--
Key: HBASE-12784
URL: https://issues.apache.org/jira/browse/HBASE-12784
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
Attachments:
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil-output.txt
I was running test suite against hadoop 2.7.0-SNAPSHOT and saw this:
{code}
testPreemptiveFastFailException50Times(org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil)
Time elapsed: 120.013 sec ERROR!
java.lang.Exception: test timed out after 12 milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException(TestFastFailWithoutTestUtil.java:450)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException50Times(TestFastFailWithoutTestUtil.java:338)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12607) TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
[
https://issues.apache.org/jira/browse/HBASE-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261265#comment-14261265
]
Sean Busbey commented on HBASE-12607:
-
+1. It would be nice if we used a
[FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html]
to timeout the attempt to get the lock rather than wait the whole period and
then fail, but I'm fine with that being a follow-on.
TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
-
Key: HBASE-12607
URL: https://issues.apache.org/jira/browse/HBASE-12607
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Labels: hbck
Attachments: 12607-v1.txt, 12607-v2.txt,
org.apache.hadoop.hbase.util.TestHBaseFsck-output.txt
TestHBaseFsck#testParallelHbck consistently fails on both Linux and Mac.
I added more logging as shown below:
{code}
HBaseFsck h1 = hbck1.get();
HBaseFsck h2 = hbck2.get();
LOG.warn(h1: + h1);
LOG.warn(h2: + h2);
{code}
I saw the following in test output:
{code}
2014-12-01 14:58:44,728 WARN [main] util.TestHBaseFsck(576): h1:
org.apache.hadoop.hbase.util.HBaseFsck@72aa738e
2014-12-01 14:58:44,729 WARN [main] util.TestHBaseFsck(577): h2:
org.apache.hadoop.hbase.util.HBaseFsck@186e2ce6
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12607) TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
[
https://issues.apache.org/jira/browse/HBASE-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12607:
---
Fix Version/s: 1.1.0
2.0.0
Hadoop Flags: Reviewed
TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
-
Key: HBASE-12607
URL: https://issues.apache.org/jira/browse/HBASE-12607
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Labels: hbck
Fix For: 2.0.0, 1.1.0
Attachments: 12607-v1.txt, 12607-v2.txt,
org.apache.hadoop.hbase.util.TestHBaseFsck-output.txt
TestHBaseFsck#testParallelHbck consistently fails on both Linux and Mac.
I added more logging as shown below:
{code}
HBaseFsck h1 = hbck1.get();
HBaseFsck h2 = hbck2.get();
LOG.warn(h1: + h1);
LOG.warn(h2: + h2);
{code}
I saw the following in test output:
{code}
2014-12-01 14:58:44,728 WARN [main] util.TestHBaseFsck(576): h1:
org.apache.hadoop.hbase.util.HBaseFsck@72aa738e
2014-12-01 14:58:44,729 WARN [main] util.TestHBaseFsck(577): h2:
org.apache.hadoop.hbase.util.HBaseFsck@186e2ce6
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-3719) Workload has to drain before hlog can be rolled
[ https://issues.apache.org/jira/browse/HBASE-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261284#comment-14261284 ] Sean Busbey commented on HBASE-3719: client request to those regions that share the WAL that is rolling will wait, yes. I don't think we actually need to wait for the log close though, provided we had waited for a sync (that is, our persistence promise is already met at sync time). One of the suggestions I've seen in connection with HBASE-10278 is to always use a hot spare HDFS pipeline to do the rolling. I think that's a great idea and we could combine it with having a worker handle the closing outside of the critical section. That should go a long way to minimizing how much of a time hit clients take at roll time. Workload has to drain before hlog can be rolled --- Key: HBASE-3719 URL: https://issues.apache.org/jira/browse/HBASE-3719 Project: HBase Issue Type: Improvement Components: regionserver, wal Reporter: dhruba borthakur Assignee: dhruba borthakur In the current implementation, the regionserver blocks new transactions from occuring when the HLog is rolled. Closing the existing HLog sometimes takes more than a few seconds and during this time all new puts/increments are blocked. It will be nice if we can continue to write new transactions to the new HLog (but maybe not commit those transactions) while the old HLog is being closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12607) TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
[
https://issues.apache.org/jira/browse/HBASE-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12607:
---
Fix Version/s: 0.98.10
1.0.0
TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
-
Key: HBASE-12607
URL: https://issues.apache.org/jira/browse/HBASE-12607
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Labels: hbck
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: 12607-v1.txt, 12607-v2.txt,
org.apache.hadoop.hbase.util.TestHBaseFsck-output.txt
TestHBaseFsck#testParallelHbck consistently fails on both Linux and Mac.
I added more logging as shown below:
{code}
HBaseFsck h1 = hbck1.get();
HBaseFsck h2 = hbck2.get();
LOG.warn(h1: + h1);
LOG.warn(h2: + h2);
{code}
I saw the following in test output:
{code}
2014-12-01 14:58:44,728 WARN [main] util.TestHBaseFsck(576): h1:
org.apache.hadoop.hbase.util.HBaseFsck@72aa738e
2014-12-01 14:58:44,729 WARN [main] util.TestHBaseFsck(577): h2:
org.apache.hadoop.hbase.util.HBaseFsck@186e2ce6
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12607) TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
[
https://issues.apache.org/jira/browse/HBASE-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261290#comment-14261290
]
Ted Yu commented on HBASE-12607:
Thanks for the reviews, Ram and Sean.
TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
-
Key: HBASE-12607
URL: https://issues.apache.org/jira/browse/HBASE-12607
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Labels: hbck
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: 12607-v1.txt, 12607-v2.txt,
org.apache.hadoop.hbase.util.TestHBaseFsck-output.txt
TestHBaseFsck#testParallelHbck consistently fails on both Linux and Mac.
I added more logging as shown below:
{code}
HBaseFsck h1 = hbck1.get();
HBaseFsck h2 = hbck2.get();
LOG.warn(h1: + h1);
LOG.warn(h2: + h2);
{code}
I saw the following in test output:
{code}
2014-12-01 14:58:44,728 WARN [main] util.TestHBaseFsck(576): h1:
org.apache.hadoop.hbase.util.HBaseFsck@72aa738e
2014-12-01 14:58:44,729 WARN [main] util.TestHBaseFsck(577): h2:
org.apache.hadoop.hbase.util.HBaseFsck@186e2ce6
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12607) TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
[
https://issues.apache.org/jira/browse/HBASE-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12607:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
-
Key: HBASE-12607
URL: https://issues.apache.org/jira/browse/HBASE-12607
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Labels: hbck
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: 12607-v1.txt, 12607-v2.txt,
org.apache.hadoop.hbase.util.TestHBaseFsck-output.txt
TestHBaseFsck#testParallelHbck consistently fails on both Linux and Mac.
I added more logging as shown below:
{code}
HBaseFsck h1 = hbck1.get();
HBaseFsck h2 = hbck2.get();
LOG.warn(h1: + h1);
LOG.warn(h2: + h2);
{code}
I saw the following in test output:
{code}
2014-12-01 14:58:44,728 WARN [main] util.TestHBaseFsck(576): h1:
org.apache.hadoop.hbase.util.HBaseFsck@72aa738e
2014-12-01 14:58:44,729 WARN [main] util.TestHBaseFsck(577): h2:
org.apache.hadoop.hbase.util.HBaseFsck@186e2ce6
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
Ted Yu created HBASE-12785: -- Summary: Use FutureTask to timeout the attempt to get the lock for hbck Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Priority: Minor In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[
https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261306#comment-14261306
]
Hadoop QA commented on HBASE-12684:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689519/HBASE-12684-v18.patch
against master branch at commit 0513a21dc8b86f57b5a6c1b742904821632f77f7.
ATTACHMENT ID: 12689519
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
org.apache.hadoop.hbase.client.TestHCM
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12261//console
This message is automatically generated.
Add new AsyncRpcClient
--
Key: HBASE-12684
URL: https://issues.apache.org/jira/browse/HBASE-12684
Project: HBase
Issue Type: Improvement
Components: Client
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch,
HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch,
HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch,
HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch,
HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v2.patch,
HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch,
HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch,
HBASE-12684-v9.patch, HBASE-12684.patch
With the changes in HBASE-12597 it is possible to add new RpcClients. This
issue is about adding a new Async RpcClient which would enable HBase to do
non blocking protobuf service communication.
Besides delivering a new AsyncRpcClient I would also like to ask the question
what it would take to replace the current RpcClient? This would enable to
simplify async code in some next issues.
--
This message was sent
[jira] [Commented] (HBASE-12762) Region with no hfiles will have the highest locality cost in LocalityCostFunction
[
https://issues.apache.org/jira/browse/HBASE-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261315#comment-14261315
]
Andrew Purtell commented on HBASE-12762:
Thanks. I see later builds passed. I'll follow up on test port randomization.
Region with no hfiles will have the highest locality cost in
LocalityCostFunction
-
Key: HBASE-12762
URL: https://issues.apache.org/jira/browse/HBASE-12762
Project: HBase
Issue Type: Improvement
Components: Balancer
Affects Versions: 0.99.2
Reporter: cuijianwei
Assignee: cuijianwei
Priority: Minor
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12762-trunk.patch
The locality cost of region will be computed in LocalityCostFunction.cost as:
{code}
double cost() {
...
int index = -1;
for (int j = 0; j regionLocations.length; j++) {
if (regionLocations[j] = 0 regionLocations[j] == serverIndex) {
index = j;
break;
}
}
if (index 0) {
cost += 1; // == region with no hfiles will have the highest cost
} else {
cost += (double) index / (double) regionLocations.length;
}
...
}
{code}
The region with no hfiles(such as empty region) will have the highest cost
which represents the worst case that region located in the server with no
locality for hfiles. However, this might be the best case because there are
no hlogs for the region. Although the absolute cost value won't affect the
balance process, will it be more reasonable to have zero cost for such
regions? such as:
{code}
...
if (index 0) {
if (regionLocation.length 0) { // == only consider regions with
hfiles
cost += 1;
}
} else {
cost += (double) index / (double) regionLocations.length;
}
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-12762) Region with no hfiles will have the highest locality cost in LocalityCostFunction
[
https://issues.apache.org/jira/browse/HBASE-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-12762.
Resolution: Fixed
Region with no hfiles will have the highest locality cost in
LocalityCostFunction
-
Key: HBASE-12762
URL: https://issues.apache.org/jira/browse/HBASE-12762
Project: HBase
Issue Type: Improvement
Components: Balancer
Affects Versions: 0.99.2
Reporter: cuijianwei
Assignee: cuijianwei
Priority: Minor
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12762-trunk.patch
The locality cost of region will be computed in LocalityCostFunction.cost as:
{code}
double cost() {
...
int index = -1;
for (int j = 0; j regionLocations.length; j++) {
if (regionLocations[j] = 0 regionLocations[j] == serverIndex) {
index = j;
break;
}
}
if (index 0) {
cost += 1; // == region with no hfiles will have the highest cost
} else {
cost += (double) index / (double) regionLocations.length;
}
...
}
{code}
The region with no hfiles(such as empty region) will have the highest cost
which represents the worst case that region located in the server with no
locality for hfiles. However, this might be the best case because there are
no hlogs for the region. Although the absolute cost value won't affect the
balance process, will it be more reasonable to have zero cost for such
regions? such as:
{code}
...
if (index 0) {
if (regionLocation.length 0) { // == only consider regions with
hfiles
cost += 1;
}
} else {
cost += (double) index / (double) regionLocations.length;
}
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261316#comment-14261316
]
Carter commented on HBASE-12728:
[~sduskis] and I have been discussing this offline for a couple of days and
have come up with the following proposal. It will require a few JIRA
sub-tasks, but it's probably only a week's worth of work, plus reviews. In a
nutshell:
# Deprecate autoFlush methods (ie Put buffering) in {{HTable}}
# Remove all autoFlush methods from {{Table}}
# Create {{BufferedTable}} (outlined below), which will buffer Puts using
{{HTableMultiplexer}}
# Create {{BufferedConnection}} as a new factory class
# Have {{HTableMultiplexer}} implement {{Closeable}} (just fixing bad behavior)
# _Behavior change_: {{HTableMultiplexer}} flushes Puts by having
{{FlushWorker}} threads...
#* _OLD_: implement low-level logic against {{AsyncProcess}}
#* _NEW_: call {{Table#putListPut}}, thus removing duplicate code and
improving encapsulation
# _Behavior change_: When its queue is full, {{HTableMultiplexer}}...
#* _OLD_: immediately rejects all Puts
#* _NEW_: blocks for a configurable time in ms (can be 0) before rejecting Puts
# _Behavior change_: When an async Put fails...
#* _OLD_: the exception is thrown during a unrelated future Put operation
(confusing)
#* _NEW_: an exception is sent to the a listener provided by the client,
following the Observer pattern
These are the new classes:
{code:java}
public class BufferedConnection implements Connection {
private Connection c;
private HTableMultiplexer htm;
/* If listener is null, will log but won't notify an async exceptions */
public BufferedConnection(Connection c, ExceptionListener l) {
this.c = c;
this.htm = new HTableMultiplexer(..., c, l);
}
public BufferedTable getTable(TableName tn) {
return new BufferedTable(c.getTable(tn), htm);
}
/* getAdmin() and getRegionLocator(...) methods delegate to Connection */
}
public class BufferedTable implements Table {
private Table t;
private HTableMultiplexer htm;
public BufferedTable(Table t, HTableMultiplexer htm) { ... }
/* Puts go to htm.doPut(...), all methods delegate to t */
}
public interface ExceptionListener {
public void onException(RetriesExhaustedWithDetailsException e);
}
{code}
From a user standpoint it looks like this:
*Before*
{code:java}
Connection conn = ConnectionFactory.createConnection();
Table t = conn.getTable(TableName.valueOf(mytable));
t.setAutoFlushTo(false);
/* do stuff */
t.close();
conn.close();
{code}
*After*
{code:java}
Connection conn = new BufferedConnection(ConnectionFactory.createConnection());
Table t = conn.getTable(TableName.valueOf(mytable));
/* do stuff */
t.close();
conn.close();
{code}
In essence, a few new classes, a moderate amount of work in HTableMultiplexer,
and a few deprecation annotations in HTable. Let us know if this looks
acceptable and we'll create some subtasks and make it so.
buffered writes substantially less useful after removal of HTablePool
-
Key: HBASE-12728
URL: https://issues.apache.org/jira/browse/HBASE-12728
Project: HBase
Issue Type: Bug
Components: hbase
Affects Versions: 0.98.0
Reporter: Aaron Beppu
In previous versions of HBase, when use of HTablePool was encouraged, HTable
instances were long-lived in that pool, and for that reason, if autoFlush was
set to false, the table instance could accumulate a full buffer of writes
before a flush was triggered. Writes from the client to the cluster could
then be substantially larger and less frequent than without buffering.
However, when HTablePool was deprecated, the primary justification seems to
have been that creating HTable instances is cheap, so long as the connection
and executor service being passed to it are pre-provided. A use pattern was
encouraged where users should create a new HTable instance for every
operation, using an existing connection and executor service, and then close
the table. In this pattern, buffered writes are substantially less useful;
writes are as small and as frequent as they would have been with
autoflush=true, except the synchronous write is moved from the operation
itself to the table close call which immediately follows.
More concretely :
```
// Given these two helpers ...
private HTableInterface getAutoFlushTable(String tableName) throws
IOException {
// (autoflush is true by default)
return storedConnection.getTable(tableName, executorService);
}
private HTableInterface getBufferedTable(String tableName) throws IOException
{
HTableInterface table = getAutoFlushTable(tableName);
table.setAutoFlush(false);
return table;
}
// it's my contention that
[jira] [Commented] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
[
https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261324#comment-14261324
]
Hadoop QA commented on HBASE-12270:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689525/HBASE-12270-v2.patch
against master branch at commit 0513a21dc8b86f57b5a6c1b742904821632f77f7.
ATTACHMENT ID: 12689525
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12262//console
This message is automatically generated.
A bug in the bucket cache, with cache blocks on write enabled
-
Key: HBASE-12270
URL: https://issues.apache.org/jira/browse/HBASE-12270
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.11, 0.98.6.1
Environment: I can reproduce it on a simple 2 node cluster, one
running the master and another running a RS. I was testing on ec2.
I used the following configurations for the cluster.
hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G
-XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog
gc:/tmp/hbase-regionserver-gc.log
hbase-site:
hbase.bucketcache.ioengine=offheap
hbase.bucketcache.size=4196
hbase.rs.cacheblocksonwrite=true
hfile.block.index.cacheonwrite=true
hfile.block.bloom.cacheonwrite=true
Reporter: Khaled Elmeleegy
Assignee: Liu Shaohui
Priority: Critical
Fix For: 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12270-v1.diff, HBASE-12270-v2.diff,
HBASE-12270-v2.patch, TestHBase.java, TestKey.java
In my experiments, I have writers streaming their output to HBase. The reader
powers a web page and does this scatter/gather, where it reads 1000 keys
written last and passes them the the front end. With this
[jira] [Commented] (HBASE-12781) thrift2 listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261327#comment-14261327
]
Hudson commented on HBASE-12781:
FAILURE: Integrated in HBase-TRUNK #5976 (See
[https://builds.apache.org/job/HBase-TRUNK/5976/])
HBASE-12781 thrift2 listen port will bind always to the passed command line
address (Pankaj Kumar) (tedyu: rev 7310fa2ba7cc8546ee5d6638539457850f60eeab)
* hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
thrift2 listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 2.0.0, 0.98.10, 1.1.0
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
{code}
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
{code}
{code}
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
{code}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261330#comment-14261330 ] Dima Spivak commented on HBASE-12740: - :) I'm convinced. You're awesome, [~jurmous]. Just two minor nits: - Since poolSize is a constant, we should probably have it be a static final int and call it POOL_SIZE to make it stick out. - When constructing the scheduled thread pool executor, we should pass in POOL_SIZE instead of the hardcoded 7 (if I'm not mistaken). Other than that, great work. +1 Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740.patch, PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12781) thrift2 listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261336#comment-14261336
]
Hudson commented on HBASE-12781:
FAILURE: Integrated in HBase-1.1 #35 (See
[https://builds.apache.org/job/HBase-1.1/35/])
HBASE-12781 thrift2 listen port will bind always to the passed command line
address (Pankaj Kumar) (tedyu: rev bf94d64667fd5c5a98c39060fdec683252011658)
* hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
thrift2 listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 2.0.0, 0.98.10, 1.1.0
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
{code}
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
{code}
{code}
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
{code}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12784) TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes hangs
[
https://issues.apache.org/jira/browse/HBASE-12784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261337#comment-14261337
]
Dima Spivak commented on HBASE-12784:
-
Possibly related to HBASE-12771? Are you seeing this on master, [~tedyu]?
TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes
hangs
--
Key: HBASE-12784
URL: https://issues.apache.org/jira/browse/HBASE-12784
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
Attachments:
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil-output.txt
I was running test suite against hadoop 2.7.0-SNAPSHOT and saw this:
{code}
testPreemptiveFastFailException50Times(org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil)
Time elapsed: 120.013 sec ERROR!
java.lang.Exception: test timed out after 12 milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException(TestFastFailWithoutTestUtil.java:450)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException50Times(TestFastFailWithoutTestUtil.java:338)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12784) TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes hangs
[
https://issues.apache.org/jira/browse/HBASE-12784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261343#comment-14261343
]
Ted Yu commented on HBASE-12784:
I was running the test on branch-1
I can run the test on master later.
TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes
hangs
--
Key: HBASE-12784
URL: https://issues.apache.org/jira/browse/HBASE-12784
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
Attachments:
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil-output.txt
I was running test suite against hadoop 2.7.0-SNAPSHOT and saw this:
{code}
testPreemptiveFastFailException50Times(org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil)
Time elapsed: 120.013 sec ERROR!
java.lang.Exception: test timed out after 12 milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException(TestFastFailWithoutTestUtil.java:450)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException50Times(TestFastFailWithoutTestUtil.java:338)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12772) TestPerColumnFamilyFlush failing
[
https://issues.apache.org/jira/browse/HBASE-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261335#comment-14261335
]
Hadoop QA commented on HBASE-12772:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689524/12772.addendum.v2.txt
against master branch at commit 0513a21dc8b86f57b5a6c1b742904821632f77f7.
ATTACHMENT ID: 12689524
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12263//console
This message is automatically generated.
TestPerColumnFamilyFlush failing
Key: HBASE-12772
URL: https://issues.apache.org/jira/browse/HBASE-12772
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 1.0.0
Reporter: stack
Assignee: stack
Attachments: 0001-HBASE-12772-TestPerColumnFamilyFlush-failing.patch,
12772.addendum.txt, 12772.addendum.v2.txt
On internal rig see this failing in two places:
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling
Failing for the past 1 build (Since Failed#653 )
Took 9 sec.
Error Message
expected:424 but was:205744
Stacktrace
java.lang.AssertionError: expected:424 but was:205744
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling(TestPerColumnFamilyFlush.java:483)
{code}
and
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay
Failing for the past 1 build (Since Failed#653 )
Took 25 ms.
Error Message
A mini-cluster is
[jira] [Commented] (HBASE-12784) TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes hangs
[
https://issues.apache.org/jira/browse/HBASE-12784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261348#comment-14261348
]
Dima Spivak commented on HBASE-12784:
-
Okay, cool. [~stack] was seeing the test failing occasionally only on branch-1
(with different failure modes cited in that JIRA), so it might be a feature bug
in PFF as opposed to a test one. Let us know if you can reproduce on master.
TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes
hangs
--
Key: HBASE-12784
URL: https://issues.apache.org/jira/browse/HBASE-12784
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
Attachments:
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil-output.txt
I was running test suite against hadoop 2.7.0-SNAPSHOT and saw this:
{code}
testPreemptiveFastFailException50Times(org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil)
Time elapsed: 120.013 sec ERROR!
java.lang.Exception: test timed out after 12 milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException(TestFastFailWithoutTestUtil.java:450)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException50Times(TestFastFailWithoutTestUtil.java:338)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[ https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261351#comment-14261351 ] stack commented on HBASE-12785: --- Why not address Sean's comment in HBASE-12607 rather than open a new issue? Are you going to work on this [~tedyu]? This is a It would be nice... in how a test works. Seems very low priority and unless you are going to work on it now, just close (See @apurtell suggestion we do issue cleanup). Use FutureTask to timeout the attempt to get the lock for hbck -- Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Priority: Minor In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-12784) TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes hangs
[
https://issues.apache.org/jira/browse/HBASE-12784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-12784:
--
Assignee: Ted Yu
TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes
hangs
--
Key: HBASE-12784
URL: https://issues.apache.org/jira/browse/HBASE-12784
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Attachments:
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil-output.txt
I was running test suite against hadoop 2.7.0-SNAPSHOT and saw this:
{code}
testPreemptiveFastFailException50Times(org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil)
Time elapsed: 120.013 sec ERROR!
java.lang.Exception: test timed out after 12 milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException(TestFastFailWithoutTestUtil.java:450)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException50Times(TestFastFailWithoutTestUtil.java:338)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[ https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-12785: -- Assignee: Ted Yu Use FutureTask to timeout the attempt to get the lock for hbck -- Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12028) Abort the RegionServer, when it's handler threads die
[ https://issues.apache.org/jira/browse/HBASE-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12028: -- Summary: Abort the RegionServer, when it's handler threads die (was: Abort the RegionServer, when one of it's handler threads die) Abort the RegionServer, when it's handler threads die - Key: HBASE-12028 URL: https://issues.apache.org/jira/browse/HBASE-12028 Project: HBase Issue Type: Bug Components: regionserver Reporter: Sudarshan Kadambi Assignee: Alicia Ying Shu Attachments: Hbase-12028-v3.patch, Hbase-12028.patch, hbase-12028-v4.patch, hbase-12028-v5.patch Over in HBase-11813, a user identified an issue where in all the RPC handler threads would exit with StackOverflow errors due to an unchecked recursion-terminating condition. Our clusters demonstrated the same trace. While the patch posted for HBASE-11813 got our clusters to be merry again, the breakdown surfaced some larger issues. When the RegionServer had all it's RPC handler threads dead, it continued to have regions assigned it. Clearly, it wouldn't be able to serve reads and writes on those regions. A second issue was that when a user tried to disable or drop a table, the master would try to communicate to the regionserver for region unassignment. Since the same handler threads seem to be used for master - RS communication as well, the master ended up hanging on the RS indefinitely. Eventually, the master stopped responding to all table meta-operations. A handler thread should never exit, and if it does, it seems like the more prudent thing to do would be for the RS to abort. This way, at least recovery can be undertaken and the regions could be reassigned elsewhere. I also think that the master-RS communication should get its own exclusive threadpool, but I'll wait until this issue has been sufficiently discussed before opening an issue ticket for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[ https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261357#comment-14261357 ] Ted Yu commented on HBASE-12785: I am working on a patch. The fix for HBASE-12607 was not test fix - previous code depended on a bug which has been fixed in hadoop. Meaning, parallel execution of hbck against hadoop 2.6.0+ would not work as expected. Use FutureTask to timeout the attempt to get the lock for hbck -- Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12607) TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
[
https://issues.apache.org/jira/browse/HBASE-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261359#comment-14261359
]
Hudson commented on HBASE-12607:
SUCCESS: Integrated in HBase-1.0 #620 (See
[https://builds.apache.org/job/HBase-1.0/620/])
HBASE-12607 TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
(tedyu: rev 1f613bcceaf5c9ff4a67f0799306094173374730)
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
-
Key: HBASE-12607
URL: https://issues.apache.org/jira/browse/HBASE-12607
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Labels: hbck
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: 12607-v1.txt, 12607-v2.txt,
org.apache.hadoop.hbase.util.TestHBaseFsck-output.txt
TestHBaseFsck#testParallelHbck consistently fails on both Linux and Mac.
I added more logging as shown below:
{code}
HBaseFsck h1 = hbck1.get();
HBaseFsck h2 = hbck2.get();
LOG.warn(h1: + h1);
LOG.warn(h2: + h2);
{code}
I saw the following in test output:
{code}
2014-12-01 14:58:44,728 WARN [main] util.TestHBaseFsck(576): h1:
org.apache.hadoop.hbase.util.HBaseFsck@72aa738e
2014-12-01 14:58:44,729 WARN [main] util.TestHBaseFsck(577): h2:
org.apache.hadoop.hbase.util.HBaseFsck@186e2ce6
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12028) Abort the RegionServer, when it's handler threads die
[ https://issues.apache.org/jira/browse/HBASE-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12028: -- Release Note: Adds a configuration property hbase.regionserver.handler.abort.on.error.percent for aborting the region server when some of it's handler threads die. The default value is 0.5 causing an abort in the RS when half of it's handler threads die. A handler thread only dies in case of a serious software bug, or a non-recoverable Error (StackOverflow, OOM, etc) is thrown. These are possible values for the configuration: * -1 = Disable aborting * 0 = Abort if even a single handler has died * 0.x = Abort only when this percent of handlers have died * 1 = Abort only all of the handers have died Abort the RegionServer, when it's handler threads die - Key: HBASE-12028 URL: https://issues.apache.org/jira/browse/HBASE-12028 Project: HBase Issue Type: Bug Components: regionserver Reporter: Sudarshan Kadambi Assignee: Alicia Ying Shu Attachments: Hbase-12028-v3.patch, Hbase-12028.patch, hbase-12028-v4.patch, hbase-12028-v5.patch Over in HBase-11813, a user identified an issue where in all the RPC handler threads would exit with StackOverflow errors due to an unchecked recursion-terminating condition. Our clusters demonstrated the same trace. While the patch posted for HBASE-11813 got our clusters to be merry again, the breakdown surfaced some larger issues. When the RegionServer had all it's RPC handler threads dead, it continued to have regions assigned it. Clearly, it wouldn't be able to serve reads and writes on those regions. A second issue was that when a user tried to disable or drop a table, the master would try to communicate to the regionserver for region unassignment. Since the same handler threads seem to be used for master - RS communication as well, the master ended up hanging on the RS indefinitely. Eventually, the master stopped responding to all table meta-operations. A handler thread should never exit, and if it does, it seems like the more prudent thing to do would be for the RS to abort. This way, at least recovery can be undertaken and the regions could be reassigned elsewhere. I also think that the master-RS communication should get its own exclusive threadpool, but I'll wait until this issue has been sufficiently discussed before opening an issue ticket for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12028) Abort the RegionServer, when it's handler threads die
[ https://issues.apache.org/jira/browse/HBASE-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261361#comment-14261361 ] Enis Soztutar commented on HBASE-12028: --- Some offline discussions with Alicia, she reverted the RpcSchedulerFactory.Context change to enabled Phoenix to be able to compile with both 0.98 and 1.1+ versions in v5. +1 for the patch. Added some release notes to the issue. This will be ON by default. Will commit to branch-1+ unless objection. Abort the RegionServer, when it's handler threads die - Key: HBASE-12028 URL: https://issues.apache.org/jira/browse/HBASE-12028 Project: HBase Issue Type: Bug Components: regionserver Reporter: Sudarshan Kadambi Assignee: Alicia Ying Shu Attachments: Hbase-12028-v3.patch, Hbase-12028.patch, hbase-12028-v4.patch, hbase-12028-v5.patch Over in HBase-11813, a user identified an issue where in all the RPC handler threads would exit with StackOverflow errors due to an unchecked recursion-terminating condition. Our clusters demonstrated the same trace. While the patch posted for HBASE-11813 got our clusters to be merry again, the breakdown surfaced some larger issues. When the RegionServer had all it's RPC handler threads dead, it continued to have regions assigned it. Clearly, it wouldn't be able to serve reads and writes on those regions. A second issue was that when a user tried to disable or drop a table, the master would try to communicate to the regionserver for region unassignment. Since the same handler threads seem to be used for master - RS communication as well, the master ended up hanging on the RS indefinitely. Eventually, the master stopped responding to all table meta-operations. A handler thread should never exit, and if it does, it seems like the more prudent thing to do would be for the RS to abort. This way, at least recovery can be undertaken and the regions could be reassigned elsewhere. I also think that the master-RS communication should get its own exclusive threadpool, but I'll wait until this issue has been sufficiently discussed before opening an issue ticket for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[ https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261364#comment-14261364 ] stack commented on HBASE-12785: --- bq. I am working on a patch. Good. I won't close this as friviolous just yet. Use FutureTask to timeout the attempt to get the lock for hbck -- Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[ https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261364#comment-14261364 ] stack edited comment on HBASE-12785 at 12/30/14 7:07 PM: - bq. I am working on a patch. Good. I won't close this as friviolous just yet (What I don't get is that if you are working on it 'now', why did HBASE-12607 have to get committed before addressing Seans' comment rather than wait? Why create a JIRA just to address his comment?) was (Author: stack): bq. I am working on a patch. Good. I won't close this as friviolous just yet. Use FutureTask to timeout the attempt to get the lock for hbck -- Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12781) thrift2 listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261374#comment-14261374
]
Hudson commented on HBASE-12781:
SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #731 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/731/])
HBASE-12781 thrift2 listen port will bind always to the passed command line
address (Pankaj Kumar) (tedyu: rev 2885bd6a1538f28df2cf97ac5c34fbde6f359a9f)
* hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
thrift2 listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 2.0.0, 0.98.10, 1.1.0
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
{code}
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
{code}
{code}
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
{code}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12028) Abort the RegionServer, when it's handler threads die
[ https://issues.apache.org/jira/browse/HBASE-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261377#comment-14261377 ] Andrew Purtell commented on HBASE-12028: Are you intending a commit to 0.98 also? The key change is new constructors for passing in an Abortable to RPC schedulers, and existing constructors are retained and deprecated. This seems fine as long as default configuration is current behavior. Abort the RegionServer, when it's handler threads die - Key: HBASE-12028 URL: https://issues.apache.org/jira/browse/HBASE-12028 Project: HBase Issue Type: Bug Components: regionserver Reporter: Sudarshan Kadambi Assignee: Alicia Ying Shu Attachments: Hbase-12028-v3.patch, Hbase-12028.patch, hbase-12028-v4.patch, hbase-12028-v5.patch Over in HBase-11813, a user identified an issue where in all the RPC handler threads would exit with StackOverflow errors due to an unchecked recursion-terminating condition. Our clusters demonstrated the same trace. While the patch posted for HBASE-11813 got our clusters to be merry again, the breakdown surfaced some larger issues. When the RegionServer had all it's RPC handler threads dead, it continued to have regions assigned it. Clearly, it wouldn't be able to serve reads and writes on those regions. A second issue was that when a user tried to disable or drop a table, the master would try to communicate to the regionserver for region unassignment. Since the same handler threads seem to be used for master - RS communication as well, the master ended up hanging on the RS indefinitely. Eventually, the master stopped responding to all table meta-operations. A handler thread should never exit, and if it does, it seems like the more prudent thing to do would be for the RS to abort. This way, at least recovery can be undertaken and the regions could be reassigned elsewhere. I also think that the master-RS communication should get its own exclusive threadpool, but I'll wait until this issue has been sufficiently discussed before opening an issue ticket for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12781) thrift2 listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261380#comment-14261380
]
Hudson commented on HBASE-12781:
SUCCESS: Integrated in HBase-0.98 #765 (See
[https://builds.apache.org/job/HBase-0.98/765/])
HBASE-12781 thrift2 listen port will bind always to the passed command line
address (Pankaj Kumar) (tedyu: rev 2885bd6a1538f28df2cf97ac5c34fbde6f359a9f)
* hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
thrift2 listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 2.0.0, 0.98.10, 1.1.0
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
{code}
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
{code}
{code}
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
{code}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12772) TestPerColumnFamilyFlush failing
[
https://issues.apache.org/jira/browse/HBASE-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261393#comment-14261393
]
Dima Spivak commented on HBASE-12772:
-
Passed 25/25 times on my machine. +1 from me.
TestPerColumnFamilyFlush failing
Key: HBASE-12772
URL: https://issues.apache.org/jira/browse/HBASE-12772
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 1.0.0
Reporter: stack
Assignee: stack
Attachments: 0001-HBASE-12772-TestPerColumnFamilyFlush-failing.patch,
12772.addendum.txt, 12772.addendum.v2.txt
On internal rig see this failing in two places:
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling
Failing for the past 1 build (Since Failed#653 )
Took 9 sec.
Error Message
expected:424 but was:205744
Stacktrace
java.lang.AssertionError: expected:424 but was:205744
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling(TestPerColumnFamilyFlush.java:483)
{code}
and
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay
Failing for the past 1 build (Since Failed#653 )
Took 25 ms.
Error Message
A mini-cluster is already running
Stacktrace
java.lang.IllegalStateException: A mini-cluster is already running
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:921)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:812)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:794)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:781)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplay(TestPerColumnFamilyFlush.java:337)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay(TestPerColumnFamilyFlush.java:418)
{code}
Opening issue to keep an eye on these tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12772) TestPerColumnFamilyFlush failing
[
https://issues.apache.org/jira/browse/HBASE-12772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12772:
--
Resolution: Fixed
Fix Version/s: 1.1.0
2.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to branch-1 and master.
TestPerColumnFamilyFlush failing
Key: HBASE-12772
URL: https://issues.apache.org/jira/browse/HBASE-12772
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 1.0.0
Reporter: stack
Assignee: stack
Fix For: 2.0.0, 1.1.0
Attachments: 0001-HBASE-12772-TestPerColumnFamilyFlush-failing.patch,
12772.addendum.txt, 12772.addendum.v2.txt
On internal rig see this failing in two places:
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling
Failing for the past 1 build (Since Failed#653 )
Took 9 sec.
Error Message
expected:424 but was:205744
Stacktrace
java.lang.AssertionError: expected:424 but was:205744
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testFlushingWhenLogRolling(TestPerColumnFamilyFlush.java:483)
{code}
and
{code}
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay
Failing for the past 1 build (Since Failed#653 )
Took 25 ms.
Error Message
A mini-cluster is already running
Stacktrace
java.lang.IllegalStateException: A mini-cluster is already running
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:921)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:812)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:794)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:781)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplay(TestPerColumnFamilyFlush.java:337)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay(TestPerColumnFamilyFlush.java:418)
{code}
Opening issue to keep an eye on these tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12607) TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
[
https://issues.apache.org/jira/browse/HBASE-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261404#comment-14261404
]
Hudson commented on HBASE-12607:
FAILURE: Integrated in HBase-1.1 #36 (See
[https://builds.apache.org/job/HBase-1.1/36/])
HBASE-12607 TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
(tedyu: rev 034b10733d58f38db3dae89f830408e3fa71703b)
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
-
Key: HBASE-12607
URL: https://issues.apache.org/jira/browse/HBASE-12607
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Labels: hbck
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: 12607-v1.txt, 12607-v2.txt,
org.apache.hadoop.hbase.util.TestHBaseFsck-output.txt
TestHBaseFsck#testParallelHbck consistently fails on both Linux and Mac.
I added more logging as shown below:
{code}
HBaseFsck h1 = hbck1.get();
HBaseFsck h2 = hbck2.get();
LOG.warn(h1: + h1);
LOG.warn(h2: + h2);
{code}
I saw the following in test output:
{code}
2014-12-01 14:58:44,728 WARN [main] util.TestHBaseFsck(576): h1:
org.apache.hadoop.hbase.util.HBaseFsck@72aa738e
2014-12-01 14:58:44,729 WARN [main] util.TestHBaseFsck(577): h2:
org.apache.hadoop.hbase.util.HBaseFsck@186e2ce6
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12607) TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
[
https://issues.apache.org/jira/browse/HBASE-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261418#comment-14261418
]
Hudson commented on HBASE-12607:
SUCCESS: Integrated in HBase-TRUNK #5977 (See
[https://builds.apache.org/job/HBase-TRUNK/5977/])
HBASE-12607 TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
(tedyu: rev 57b1efaabed195c8dfed9823dcca80260cfea946)
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
-
Key: HBASE-12607
URL: https://issues.apache.org/jira/browse/HBASE-12607
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Labels: hbck
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: 12607-v1.txt, 12607-v2.txt,
org.apache.hadoop.hbase.util.TestHBaseFsck-output.txt
TestHBaseFsck#testParallelHbck consistently fails on both Linux and Mac.
I added more logging as shown below:
{code}
HBaseFsck h1 = hbck1.get();
HBaseFsck h2 = hbck2.get();
LOG.warn(h1: + h1);
LOG.warn(h2: + h2);
{code}
I saw the following in test output:
{code}
2014-12-01 14:58:44,728 WARN [main] util.TestHBaseFsck(576): h1:
org.apache.hadoop.hbase.util.HBaseFsck@72aa738e
2014-12-01 14:58:44,729 WARN [main] util.TestHBaseFsck(577): h2:
org.apache.hadoop.hbase.util.HBaseFsck@186e2ce6
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12719) Add test WAL provider to quantify FSHLog overhead in the absence of HDFS.
[ https://issues.apache.org/jira/browse/HBASE-12719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-12719: Attachment: HBASE-12719.1.patch.txt HBASE-12719_comparison_chart.tiff attaching an implementation as well as a chart showing the perf change going from disabled WAL to various amounts of HDFS interactions. if you look at HBASE-12719_comparison_chart you can see the disabled WAL (which is all noops) in red at the top. The cyan line is FSHLog but w/o any HDFS interactions. The default FSHLog and the new test driver but with all interactions enabled are at the bottom; they've drawn over each other so only one is apparent. Add test WAL provider to quantify FSHLog overhead in the absence of HDFS. - Key: HBASE-12719 URL: https://issues.apache.org/jira/browse/HBASE-12719 Project: HBase Issue Type: Improvement Components: test, wal Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12719.1.patch.txt, HBASE-12719_comparison_chart.tiff Discussion in HBASE-5699 included showing the max throughput for our WAL benchmark (using the DisabledWALProvider) compared to when we are actually dealing with synchronization and talking with HDFS. What we can't see in that comparison is how much of the (sizable) gap is due to the coordination done in FSHLog to deal with multi-threading and how much of it is due to talking to HDFS. Make a test-only provider that we can use to isolate the cost for HDFS appends, HDFS flush, and file rolling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12740: -- Attachment: HBASE-12740-v4.patch Nits are always welcome! :) I applied your suggestions. Thanks for reviewing! Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740-v4.patch, HBASE-12740.patch, PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[
https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261442#comment-14261442
]
Hadoop QA commented on HBASE-12740:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689556/HBASE-12740-v4.patch
against master branch at commit c6002ccd3873f3950df50bef7f4ac4c5760d595f.
ATTACHMENT ID: 12689556
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12265//console
This message is automatically generated.
Improve performance of TestHBaseFsck
Key: HBASE-12740
URL: https://issues.apache.org/jira/browse/HBASE-12740
Project: HBase
Issue Type: Bug
Components: util
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch,
HBASE-12740-v3.patch, HBASE-12740-v4.patch, HBASE-12740.patch,
PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png
TestHBaseFsck performs poor on my machine. It crashes because the threads
reach the 2000 thread limit on my machine. Looking at the code a lot of
optimization is possible and some API calls are used wrong. A lot of Admin
instances are created and never closed, lots of Tables are not closed,
ThreadPoolExecutors are not shut down and an unlimited thread pool which does
not recycle threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12719) Add test WAL provider to quantify FSHLog overhead in the absence of HDFS.
[ https://issues.apache.org/jira/browse/HBASE-12719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-12719: Attachment: HBASE-12719.2.patch.txt including the patch actually used to make the charts, version 1 was stale. :/ Add test WAL provider to quantify FSHLog overhead in the absence of HDFS. - Key: HBASE-12719 URL: https://issues.apache.org/jira/browse/HBASE-12719 Project: HBase Issue Type: Improvement Components: test, wal Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12719.1.patch.txt, HBASE-12719.2.patch.txt, HBASE-12719_comparison_chart.tiff Discussion in HBASE-5699 included showing the max throughput for our WAL benchmark (using the DisabledWALProvider) compared to when we are actually dealing with synchronization and talking with HDFS. What we can't see in that comparison is how much of the (sizable) gap is due to the coordination done in FSHLog to deal with multi-threading and how much of it is due to talking to HDFS. Make a test-only provider that we can use to isolate the cost for HDFS appends, HDFS flush, and file rolling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12719) Add test WAL provider to quantify FSHLog overhead in the absence of HDFS.
[ https://issues.apache.org/jira/browse/HBASE-12719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-12719: Status: Patch Available (was: In Progress) Add test WAL provider to quantify FSHLog overhead in the absence of HDFS. - Key: HBASE-12719 URL: https://issues.apache.org/jira/browse/HBASE-12719 Project: HBase Issue Type: Improvement Components: test, wal Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12719.1.patch.txt, HBASE-12719.2.patch.txt, HBASE-12719_comparison_chart.tiff Discussion in HBASE-5699 included showing the max throughput for our WAL benchmark (using the DisabledWALProvider) compared to when we are actually dealing with synchronization and talking with HDFS. What we can't see in that comparison is how much of the (sizable) gap is due to the coordination done in FSHLog to deal with multi-threading and how much of it is due to talking to HDFS. Make a test-only provider that we can use to isolate the cost for HDFS appends, HDFS flush, and file rolling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[ https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12785: --- Attachment: 12785-001.patch With this patch, testParallelHbck runs 20 seconds shorter against hadoop 2.6.0 compared with current code. Use FutureTask to timeout the attempt to get the lock for hbck -- Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Attachments: 12785-001.patch In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12740: -- Attachment: HBASE-12740-v4.patch Resubmitting since I see the previous patch was applied to master and retracted by [~stack]. I guess the next version needs to be checked by HadoopQA :) Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740-v4.patch, HBASE-12740-v4.patch, HBASE-12740.patch, PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[ https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12785: --- Attachment: 12785-001.patch Patch adjusted to current trunk. Use FutureTask to timeout the attempt to get the lock for hbck -- Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Attachments: 12785-001.patch In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[ https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12785: --- Attachment: (was: 12785-001.patch) Use FutureTask to timeout the attempt to get the lock for hbck -- Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Attachments: 12785-001.patch In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[ https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12785: --- Status: Patch Available (was: Open) Use FutureTask to timeout the attempt to get the lock for hbck -- Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Attachments: 12785-001.patch In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261473#comment-14261473 ] stack commented on HBASE-12740: --- Pardon me. I applied v3 with [~dimaspivak] items addressed. Was then trying to cherry-pick to branch-1 and beyond but was taking a little while (TestHBaseFsck runs are a bit of beast). I reverted my master work since [~jurmous] showed up w/ new version. Will go forward with v4. Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740-v4.patch, HBASE-12740-v4.patch, HBASE-12740.patch, PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12783: --- Status: Patch Available (was: Open) Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12783.patch, HBASE-12873B.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12783: --- Attachment: HBASE-12873B.patch I made a lot more changes. All regionlocator like functionality now ends up going to HRegionLocator one way or the other. Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12783.patch, HBASE-12873B.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261492#comment-14261492 ] Solomon Duskis commented on HBASE-12783: I had to make some other changes to convert concrete class uses to the new interfaces. I found a few HTables and etc that were not being closed, and closed them. Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12783.patch, HBASE-12873B.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12607) TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
[
https://issues.apache.org/jira/browse/HBASE-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261494#comment-14261494
]
Hudson commented on HBASE-12607:
FAILURE: Integrated in HBase-0.98 #766 (See
[https://builds.apache.org/job/HBase-0.98/766/])
HBASE-12607 TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
(tedyu: rev c1fcb8fe3304d578521731dc3f51ec5d7f33f420)
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
HBASE-12607 Addendum fixes compilation (tedyu: rev
e88d74d6e98e0da438333e7c22f96f17ac595a71)
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
TestHBaseFsck#testParallelHbck fails running against hadoop 2.6.0
-
Key: HBASE-12607
URL: https://issues.apache.org/jira/browse/HBASE-12607
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Labels: hbck
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: 12607-v1.txt, 12607-v2.txt,
org.apache.hadoop.hbase.util.TestHBaseFsck-output.txt
TestHBaseFsck#testParallelHbck consistently fails on both Linux and Mac.
I added more logging as shown below:
{code}
HBaseFsck h1 = hbck1.get();
HBaseFsck h2 = hbck2.get();
LOG.warn(h1: + h1);
LOG.warn(h2: + h2);
{code}
I saw the following in test output:
{code}
2014-12-01 14:58:44,728 WARN [main] util.TestHBaseFsck(576): h1:
org.apache.hadoop.hbase.util.HBaseFsck@72aa738e
2014-12-01 14:58:44,729 WARN [main] util.TestHBaseFsck(577): h2:
org.apache.hadoop.hbase.util.HBaseFsck@186e2ce6
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
