[jira] [Commented] (HBASE-7663) [Per-KV security] Visibility labels
[
https://issues.apache.org/jira/browse/HBASE-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822253#comment-13822253
]
Hadoop QA commented on HBASE-7663:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613788/HBASE-7663_V8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7857//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7857//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7857//console
This message is automatically generated.
[Per-KV security] Visibility labels
---
Key: HBASE-7663
URL: https://issues.apache.org/jira/browse/HBASE-7663
Project: HBase
Issue Type: Sub-task
Components: Coprocessors, security
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Assignee: Anoop Sam John
Fix For: 0.98.0
Attachments: HBASE-7663.patch, HBASE-7663_V2.patch,
HBASE-7663_V3.patch, HBASE-7663_V4.patch, HBASE-7663_V5.patch,
HBASE-7663_V6.patch, HBASE-7663_V7.patch, HBASE-7663_V8.patch
Implement Accumulo-style visibility labels. Consider the following design
principles:
- Coprocessor based implementation
- Minimal to no changes to core code
- Use KeyValue tags (HBASE-7448) to carry labels
- Use OperationWithAttributes# {get,set}Attribute for handling visibility
labels in the API
- Implement a new filter for evaluating visibility labels as KVs are streamed
through.
This approach would be consistent in deployment and API details with other
per-KV security work, supporting environments where they might be both be
employed, even stacked on some tables.
See the parent issue for more discussion.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-9969) Improve KeyValueHeap using loser tree
Chao Shi created HBASE-9969: --- Summary: Improve KeyValueHeap using loser tree Key: HBASE-9969 URL: https://issues.apache.org/jira/browse/HBASE-9969 Project: HBase Issue Type: Improvement Reporter: Chao Shi LoserTree is the better data structure than binary heap. It saves half of the comparisons on each next(), though the time complexity is on O(logN). Currently A scan or get will go through two KeyValueHeaps, one is merging KVs read from multiple HFiles in a single store, the other is merging results from multiple stores. This patch should improve the both cases whenever CPU is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). All of the optimization work is done in KeyValueHeap and does not change its public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9963) Remove the ReentrantReadWriteLock in the MemStore
[
https://issues.apache.org/jira/browse/HBASE-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Liochon updated HBASE-9963:
---
Attachment: 9963.v3.patch
Remove the ReentrantReadWriteLock in the MemStore
-
Key: HBASE-9963
URL: https://issues.apache.org/jira/browse/HBASE-9963
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments: 9963.v1.patch, 9963.v2.patch, 9963.v3.patch
If I'm not wrong, the MemStore is always used from the HStore. The code in
HStore puts a lock before calling MemStore. So the lock in Memstore is
useless.
For example, in HStore
{code}
@Override
public long upsert(IterableCell cells, long readpoint) throws IOException
{
this.lock.readLock().lock();
try {
return this.memstore.upsert(cells, readpoint);
} finally {
this.lock.readLock().unlock();
}
}
{code}
With this in MemStore
{code}
public long upsert(IterableCell cells, long readpoint) {
this.lock.readLock().lock(); // ==Am I useful?
try {
long size = 0;
for (Cell cell : cells) {
size += upsert(cell, readpoint);
}
return size;
} finally {
this.lock.readLock().unlock();
}
}
{code}
I've checked, all the locks in MemStore are backed by a lock in HStore, the
only exception beeing
{code}
void snapshot() {
this.memstore.snapshot();
}
{code}
And I would say it's a bug. If it's confirm ([~lhofhansl], what do you
think?), I will add a lock there and remove all of them in MemStore. They do
appear in the profiling.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9963) Remove the ReentrantReadWriteLock in the MemStore
[
https://issues.apache.org/jira/browse/HBASE-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Liochon updated HBASE-9963:
---
Status: Open (was: Patch Available)
Remove the ReentrantReadWriteLock in the MemStore
-
Key: HBASE-9963
URL: https://issues.apache.org/jira/browse/HBASE-9963
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.96.0, 0.98.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments: 9963.v1.patch, 9963.v2.patch, 9963.v3.patch
If I'm not wrong, the MemStore is always used from the HStore. The code in
HStore puts a lock before calling MemStore. So the lock in Memstore is
useless.
For example, in HStore
{code}
@Override
public long upsert(IterableCell cells, long readpoint) throws IOException
{
this.lock.readLock().lock();
try {
return this.memstore.upsert(cells, readpoint);
} finally {
this.lock.readLock().unlock();
}
}
{code}
With this in MemStore
{code}
public long upsert(IterableCell cells, long readpoint) {
this.lock.readLock().lock(); // ==Am I useful?
try {
long size = 0;
for (Cell cell : cells) {
size += upsert(cell, readpoint);
}
return size;
} finally {
this.lock.readLock().unlock();
}
}
{code}
I've checked, all the locks in MemStore are backed by a lock in HStore, the
only exception beeing
{code}
void snapshot() {
this.memstore.snapshot();
}
{code}
And I would say it's a bug. If it's confirm ([~lhofhansl], what do you
think?), I will add a lock there and remove all of them in MemStore. They do
appear in the profiling.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9963) Remove the ReentrantReadWriteLock in the MemStore
[
https://issues.apache.org/jira/browse/HBASE-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Liochon updated HBASE-9963:
---
Status: Patch Available (was: Open)
Tests work locally. v3 is a retry w/ the javadoc added.
Remove the ReentrantReadWriteLock in the MemStore
-
Key: HBASE-9963
URL: https://issues.apache.org/jira/browse/HBASE-9963
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.96.0, 0.98.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments: 9963.v1.patch, 9963.v2.patch, 9963.v3.patch
If I'm not wrong, the MemStore is always used from the HStore. The code in
HStore puts a lock before calling MemStore. So the lock in Memstore is
useless.
For example, in HStore
{code}
@Override
public long upsert(IterableCell cells, long readpoint) throws IOException
{
this.lock.readLock().lock();
try {
return this.memstore.upsert(cells, readpoint);
} finally {
this.lock.readLock().unlock();
}
}
{code}
With this in MemStore
{code}
public long upsert(IterableCell cells, long readpoint) {
this.lock.readLock().lock(); // ==Am I useful?
try {
long size = 0;
for (Cell cell : cells) {
size += upsert(cell, readpoint);
}
return size;
} finally {
this.lock.readLock().unlock();
}
}
{code}
I've checked, all the locks in MemStore are backed by a lock in HStore, the
only exception beeing
{code}
void snapshot() {
this.memstore.snapshot();
}
{code}
And I would say it's a bug. If it's confirm ([~lhofhansl], what do you
think?), I will add a lock there and remove all of them in MemStore. They do
appear in the profiling.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-3680) Publish more metrics about mslab
[ https://issues.apache.org/jira/browse/HBASE-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822261#comment-13822261 ] Asaf Mesika commented on HBASE-3680: I fix you should reset fix-version field, since it's not really fixed and confusing. Publish more metrics about mslab Key: HBASE-3680 URL: https://issues.apache.org/jira/browse/HBASE-3680 Project: HBase Issue Type: Improvement Affects Versions: 0.90.1 Reporter: Jean-Daniel Cryans Assignee: Todd Lipcon Fix For: 0.92.3 Attachments: hbase-3680.txt, hbase-3680.txt We have been using mslab on all our clusters for a while now and it seems it tends to OOME or send us into GC loops of death a lot more than it used to. For example, one RS with mslab enabled and 7GB of heap died out of OOME this afternoon; it had .55GB in the block cache and 2.03GB in the memstores which doesn't account for much... but it could be that because of mslab a lot of space was lost in those incomplete 2MB blocks and without metrics we can't really tell. Compactions were running at the time of the OOME and I see block cache activity. The average load on that cluster is 531. We should at least publish the total size of all those blocks and maybe even take actions based on that (like force flushing). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Shi updated HBASE-9969: Attachment: kvheap-benchmark.txt kvheap-benchmark.png hbase-9969.patch I ran two benchmarks: A) KeyValueHeapBenchmark class included in the patch It simply constructs a KeyValueHeap from several CollectionBackedKeyValueScanner and sees how many next/reseek calls per second. ||scanners|| lt-next || lt-reseek || pq-next || pq-reseek || |1|17543859.6|3058104|18181818.2|1798561.2| |2|11299435|5102040.8|11173184.4|3053435.1| |3|8547008.5|4854368.9|7915567.3|2859866.5| |4|7936507.9|4866180|5891016.2|2507837| |5|6711409.4|4739336.5|4748338.1|2296738.6| lt- denotes LoserTree based KeyValueHeap. pq- denotes PriorityQueue based KeyValueHeap. A complete result (with up to 19 scanners) is attached. B) ColumnPaginationFilter with offset=1M I ran a mini-cluster and put a huge number of columns on a single row. Thes columns are uniformly written to several HFiles. Then query using ColumnPaginationFilter with offset = 1M. Blocks are cached, so it is CPU intensive. Qualifiers and values are 4 byte integers. Row key is test_row. Blocks are not compressed. The below table shows how long the scan takes. || hfiles || lt || pq || | 1 | 749.8 ms | 986.69 ms | | 2 |1511.28 ms | 2190.97 ms | | 3 |2392.8 ms | 4029.8 ms | | 4 | 3318.8 ms | 5760.22 ms Improve KeyValueHeap using loser tree - Key: HBASE-9969 URL: https://issues.apache.org/jira/browse/HBASE-9969 Project: HBase Issue Type: Improvement Reporter: Chao Shi Attachments: hbase-9969.patch, kvheap-benchmark.png, kvheap-benchmark.txt LoserTree is the better data structure than binary heap. It saves half of the comparisons on each next(), though the time complexity is on O(logN). Currently A scan or get will go through two KeyValueHeaps, one is merging KVs read from multiple HFiles in a single store, the other is merging results from multiple stores. This patch should improve the both cases whenever CPU is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). All of the optimization work is done in KeyValueHeap and does not change its public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9811) ColumnPaginationFilter is slow when offset is large
[ https://issues.apache.org/jira/browse/HBASE-9811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822264#comment-13822264 ] Chao Shi commented on HBASE-9811: - HBASE-9969 is opened to improve performance of KeyValueHeap. ColumnPaginationFilter is slow when offset is large --- Key: HBASE-9811 URL: https://issues.apache.org/jira/browse/HBASE-9811 Project: HBase Issue Type: Bug Reporter: Chao Shi Hi there, we are trying to migrate a app from MySQL to HBase. One kind of the queries is pagination with large offset and small limit. We don't have too many such queries and so both MySQL and HBase should survive. (MySQL has no index for offset either.) When comparing the performance on both systems, we found something interest: write ~1M values in a single row, and query with offset = 1M. So all values should be scanned on RS side. When running the query on MySQL, the first query is pretty slow (more than 1 second) and then repeat the same query, it will become very low latency. HBase on the other hand, repeating the query does not help much (~1s forever). I can confirm that all data are in block cache and all the time is spent on in-memory data processing. (We have flushed data to disk.) I found reseek is the hot spot. It is caused by ColumnPaginationFilter returning NEXT_COL. If I replace this line by returning SKIP (which causes to call next rather than reseek), the latency is reduced to ~100ms. So I think there must be some room for optimization. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Shi updated HBASE-9969: Status: Patch Available (was: Open) Submit to hadoop QA. Improve KeyValueHeap using loser tree - Key: HBASE-9969 URL: https://issues.apache.org/jira/browse/HBASE-9969 Project: HBase Issue Type: Improvement Reporter: Chao Shi Attachments: hbase-9969.patch, kvheap-benchmark.png, kvheap-benchmark.txt LoserTree is the better data structure than binary heap. It saves half of the comparisons on each next(), though the time complexity is on O(logN). Currently A scan or get will go through two KeyValueHeaps, one is merging KVs read from multiple HFiles in a single store, the other is merging results from multiple stores. This patch should improve the both cases whenever CPU is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). All of the optimization work is done in KeyValueHeap and does not change its public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree
[
https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822285#comment-13822285
]
Hadoop QA commented on HBASE-9969:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613807/kvheap-benchmark.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7859//console
This message is automatically generated.
Improve KeyValueHeap using loser tree
-
Key: HBASE-9969
URL: https://issues.apache.org/jira/browse/HBASE-9969
Project: HBase
Issue Type: Improvement
Reporter: Chao Shi
Attachments: hbase-9969.patch, kvheap-benchmark.png,
kvheap-benchmark.txt
LoserTree is the better data structure than binary heap. It saves half of the
comparisons on each next(), though the time complexity is on O(logN).
Currently A scan or get will go through two KeyValueHeaps, one is merging KVs
read from multiple HFiles in a single store, the other is merging results
from multiple stores. This patch should improve the both cases whenever CPU
is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811).
All of the optimization work is done in KeyValueHeap and does not change its
public interfaces. The new code looks more cleaner and simpler to understand.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9869) Optimize HConnectionManager#getCachedLocation
[ https://issues.apache.org/jira/browse/HBASE-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9869: --- Attachment: 9869.v2.patch Optimize HConnectionManager#getCachedLocation - Key: HBASE-9869 URL: https://issues.apache.org/jira/browse/HBASE-9869 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9869.v1.patch, 9869.v1.patch, 9869.v2.patch It javadoc says: TODO: This method during writing consumes 15% of CPU doing lookup. This is still true, says Yourkit. With 0.96, we also spend more time in these methods. We retry more, and the AsyncProcess calls it in parallel. I don't have the patch for this yet, but I will spend some time on it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9869) Optimize HConnectionManager#getCachedLocation
[ https://issues.apache.org/jira/browse/HBASE-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9869: --- Status: Open (was: Patch Available) Optimize HConnectionManager#getCachedLocation - Key: HBASE-9869 URL: https://issues.apache.org/jira/browse/HBASE-9869 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9869.v1.patch, 9869.v1.patch, 9869.v2.patch It javadoc says: TODO: This method during writing consumes 15% of CPU doing lookup. This is still true, says Yourkit. With 0.96, we also spend more time in these methods. We retry more, and the AsyncProcess calls it in parallel. I don't have the patch for this yet, but I will spend some time on it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9869) Optimize HConnectionManager#getCachedLocation
[ https://issues.apache.org/jira/browse/HBASE-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9869: --- Status: Patch Available (was: Open) Optimize HConnectionManager#getCachedLocation - Key: HBASE-9869 URL: https://issues.apache.org/jira/browse/HBASE-9869 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9869.v1.patch, 9869.v1.patch, 9869.v2.patch It javadoc says: TODO: This method during writing consumes 15% of CPU doing lookup. This is still true, says Yourkit. With 0.96, we also spend more time in these methods. We retry more, and the AsyncProcess calls it in parallel. I don't have the patch for this yet, but I will spend some time on it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Status: Open (was: Patch Available) Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Status: Patch Available (was: Open) Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Attachment: 9959-trunk.v2.patch Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9963) Remove the ReentrantReadWriteLock in the MemStore
[
https://issues.apache.org/jira/browse/HBASE-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822307#comment-13822307
]
Hadoop QA commented on HBASE-9963:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613802/9963.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7858//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7858//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7858//console
This message is automatically generated.
Remove the ReentrantReadWriteLock in the MemStore
-
Key: HBASE-9963
URL: https://issues.apache.org/jira/browse/HBASE-9963
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments: 9963.v1.patch, 9963.v2.patch, 9963.v3.patch
If I'm not wrong, the MemStore is always used from the HStore. The code in
HStore puts a lock before calling MemStore. So the lock in Memstore is
useless.
For example, in HStore
{code}
@Override
public long upsert(IterableCell cells, long readpoint) throws IOException
{
this.lock.readLock().lock();
try {
return this.memstore.upsert(cells, readpoint);
} finally {
this.lock.readLock().unlock();
}
}
{code}
With this in MemStore
{code}
public long upsert(IterableCell cells, long readpoint) {
this.lock.readLock().lock(); // ==Am I useful?
try {
long size = 0;
for (Cell cell : cells) {
size += upsert(cell, readpoint);
}
return size;
} finally {
this.lock.readLock().unlock();
}
}
{code}
I've checked, all the locks in MemStore are backed by a lock in HStore, the
only exception beeing
{code}
void snapshot() {
this.memstore.snapshot();
}
{code}
And I would say it's a bug. If it's confirm ([~lhofhansl], what do you
think?), I will add a lock there and remove all of them in MemStore. They do
appear in the profiling.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822308#comment-13822308 ] Liang Xie commented on HBASE-8163: -- I did a apple-to-apple test with this patch. Here is the test result just for refer: Env: Xmx=Xms=8G,Xmn1G, memestore lower/upper limit = 0.2/0.3 each test began with empty table, then write 20million records each with three fields, and each field has 200 bytes: 1)original config(-XX:PretenureSizeThreshold=4m) YGC YGCTFGCFGCT GCT 6970 318.592 80.884 319.476 2)set hbase.hregion.memstore.mslab.chunksize to 4194320(-XX:PretenureSizeThreshold=4m) YGC YGCTFGCFGCT GCT 6973 253.891 80.522 254.413 3)set -XX:PretenureSizeThreshold=2097088(hbase.hregion.memstore.mslab.chunksize is by default 2M) 6960 260.642 81.427 262.069 4)set hbase.hregion.memstore.chunkpool.maxsize=0.6, means enable MemStoreChunkPool feature(log said maxCount=706), -XX:PretenureSizeThreshold=2097088(hbase.hregion.memstore.mslab.chunksize is by default 2M) 7028 258.598 20.401 258.999 To me, this MemStorechunkPool feature is useful for heavy FGC scenarios caused by write request. If the YGC is the big hurt by write request, personally i recommend to config hbase.hregion.memstore.mslab.chunksize or -XX:PretenureSizeThreshold, considering risk:) Hope this test result will be helpful for some guys MemStoreChunkPool: An improvement for JAVA GC when using MSLAB -- Key: HBASE-8163 URL: https://issues.apache.org/jira/browse/HBASE-8163 Project: HBase Issue Type: New Feature Components: Performance, regionserver Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.98.0, 0.95.0 Attachments: hbase-0.95-8163v6.patch, hbase-8163v1.patch, hbase-8163v2.patch, hbase-8163v3.patch, hbase-8163v4.patch, hbase-8163v5.patch, hbase-8163v6.patch *Usage:* Disable pool(default): configure hbase.hregion.memstore.chunkpool.maxsize as 0 Enable pool: configure hbase.hregion.memstore.chunkpool.maxsize as a percentage of global memstore size (between 0.0 and 1.0, recommend to set it as the gap between min and max sizes of memstore, e.g. 0.5) *Background*： When we use mslab,we will copy the keyvalue together in a structure called *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. *Problem*： When one chunk is full, we will create a new chunk, and then the old chunk will be reclaimed by JVM if no reference to it. Mostly the chunk object would be promoted when doing Young GC, cause increasing the cost of YGC When a Chunk object has no reference? It should meet two following condition: 1.Memstore which this chunk belongs to is flushed 2.No scanner is opening on the memstore which this chunk belongs to *Solution:* 1.Create a chunk pool to manage the no-reference chunks, instead of being reclaimed by JVM 2.When a Chunk has no reference, put it back to the pool 3.The pool has a max capacity, it will skip the chunks when achieve the max size 4.When we need new Chunk to store KeyValue, get it from the pool if exists, else create new one by pool, so we could be able to reuse the old chunks *Test results:* Environment: hbase-version:0.94 -Xms4G -Xmx4G -Xmn2G Row size=50 bytes, Value size=1024 bytes 50 concurrent theads per client, insert 10,000,000 rows Before: Avg write request per second:12953 After testing, final result of jstat -gcutil : YGC YGCT FGC FGCT GCT 747 36.503 48 2.492 38.995 After: Avg write request per second:14025 After testing, final result of jstat -gcutil : YGC YGCT FGC FGCT GCT 711 20.344 4 0.284 20.628 *Improvement: YGC 40+%; WPS 5%+* review board : https://reviews.apache.org/r/10056/ -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822310#comment-13822310 ] Liang Xie commented on HBASE-8163: -- Of cause, per theory, the chunkpool solution should be a little better than my recommend config solution, since it'll reuse the chunks to avoid more new byte[] invoke, more memset clear operation from hotspot. MemStoreChunkPool: An improvement for JAVA GC when using MSLAB -- Key: HBASE-8163 URL: https://issues.apache.org/jira/browse/HBASE-8163 Project: HBase Issue Type: New Feature Components: Performance, regionserver Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.98.0, 0.95.0 Attachments: hbase-0.95-8163v6.patch, hbase-8163v1.patch, hbase-8163v2.patch, hbase-8163v3.patch, hbase-8163v4.patch, hbase-8163v5.patch, hbase-8163v6.patch *Usage:* Disable pool(default): configure hbase.hregion.memstore.chunkpool.maxsize as 0 Enable pool: configure hbase.hregion.memstore.chunkpool.maxsize as a percentage of global memstore size (between 0.0 and 1.0, recommend to set it as the gap between min and max sizes of memstore, e.g. 0.5) *Background*： When we use mslab,we will copy the keyvalue together in a structure called *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. *Problem*： When one chunk is full, we will create a new chunk, and then the old chunk will be reclaimed by JVM if no reference to it. Mostly the chunk object would be promoted when doing Young GC, cause increasing the cost of YGC When a Chunk object has no reference? It should meet two following condition: 1.Memstore which this chunk belongs to is flushed 2.No scanner is opening on the memstore which this chunk belongs to *Solution:* 1.Create a chunk pool to manage the no-reference chunks, instead of being reclaimed by JVM 2.When a Chunk has no reference, put it back to the pool 3.The pool has a max capacity, it will skip the chunks when achieve the max size 4.When we need new Chunk to store KeyValue, get it from the pool if exists, else create new one by pool, so we could be able to reuse the old chunks *Test results:* Environment: hbase-version:0.94 -Xms4G -Xmx4G -Xmn2G Row size=50 bytes, Value size=1024 bytes 50 concurrent theads per client, insert 10,000,000 rows Before: Avg write request per second:12953 After testing, final result of jstat -gcutil : YGC YGCT FGC FGCT GCT 747 36.503 48 2.492 38.995 After: Avg write request per second:14025 After testing, final result of jstat -gcutil : YGC YGCT FGC FGCT GCT 711 20.344 4 0.284 20.628 *Improvement: YGC 40+%; WPS 5%+* review board : https://reviews.apache.org/r/10056/ -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9963) Remove the ReentrantReadWriteLock in the MemStore
[
https://issues.apache.org/jira/browse/HBASE-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Liochon updated HBASE-9963:
---
Attachment: 9963.96.v3.patch
Remove the ReentrantReadWriteLock in the MemStore
-
Key: HBASE-9963
URL: https://issues.apache.org/jira/browse/HBASE-9963
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments: 9963.96.v3.patch, 9963.v1.patch, 9963.v2.patch,
9963.v3.patch
If I'm not wrong, the MemStore is always used from the HStore. The code in
HStore puts a lock before calling MemStore. So the lock in Memstore is
useless.
For example, in HStore
{code}
@Override
public long upsert(IterableCell cells, long readpoint) throws IOException
{
this.lock.readLock().lock();
try {
return this.memstore.upsert(cells, readpoint);
} finally {
this.lock.readLock().unlock();
}
}
{code}
With this in MemStore
{code}
public long upsert(IterableCell cells, long readpoint) {
this.lock.readLock().lock(); // ==Am I useful?
try {
long size = 0;
for (Cell cell : cells) {
size += upsert(cell, readpoint);
}
return size;
} finally {
this.lock.readLock().unlock();
}
}
{code}
I've checked, all the locks in MemStore are backed by a lock in HStore, the
only exception beeing
{code}
void snapshot() {
this.memstore.snapshot();
}
{code}
And I would say it's a bug. If it's confirm ([~lhofhansl], what do you
think?), I will add a lock there and remove all of them in MemStore. They do
appear in the profiling.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9963) Remove the ReentrantReadWriteLock in the MemStore
[
https://issues.apache.org/jira/browse/HBASE-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Liochon updated HBASE-9963:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed, thanks for the review.
The javadoc warnings were not related to this patch, but I fixed them as well
in the commit.
Remove the ReentrantReadWriteLock in the MemStore
-
Key: HBASE-9963
URL: https://issues.apache.org/jira/browse/HBASE-9963
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments: 9963.96.v3.patch, 9963.v1.patch, 9963.v2.patch,
9963.v3.patch
If I'm not wrong, the MemStore is always used from the HStore. The code in
HStore puts a lock before calling MemStore. So the lock in Memstore is
useless.
For example, in HStore
{code}
@Override
public long upsert(IterableCell cells, long readpoint) throws IOException
{
this.lock.readLock().lock();
try {
return this.memstore.upsert(cells, readpoint);
} finally {
this.lock.readLock().unlock();
}
}
{code}
With this in MemStore
{code}
public long upsert(IterableCell cells, long readpoint) {
this.lock.readLock().lock(); // ==Am I useful?
try {
long size = 0;
for (Cell cell : cells) {
size += upsert(cell, readpoint);
}
return size;
} finally {
this.lock.readLock().unlock();
}
}
{code}
I've checked, all the locks in MemStore are backed by a lock in HStore, the
only exception beeing
{code}
void snapshot() {
this.memstore.snapshot();
}
{code}
And I would say it's a bug. If it's confirm ([~lhofhansl], what do you
think?), I will add a lock there and remove all of them in MemStore. They do
appear in the profiling.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9869) Optimize HConnectionManager#getCachedLocation
[
https://issues.apache.org/jira/browse/HBASE-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822357#comment-13822357
]
Hadoop QA commented on HBASE-9869:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613812/9869.v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7861//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7861//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7861//console
This message is automatically generated.
Optimize HConnectionManager#getCachedLocation
-
Key: HBASE-9869
URL: https://issues.apache.org/jira/browse/HBASE-9869
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9869.v1.patch, 9869.v1.patch, 9869.v2.patch
It javadoc says: TODO: This method during writing consumes 15% of CPU doing
lookup. This is still true, says Yourkit. With 0.96, we also spend more time
in these methods. We retry more, and the AsyncProcess calls it in parallel.
I don't have the patch for this yet, but I will spend some time on it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5945: - Status: Patch Available (was: Open) Reduce buffer copies in IPC server response path Key: HBASE-5945 URL: https://issues.apache.org/jira/browse/HBASE-5945 Project: HBase Issue Type: Improvement Components: IPC/RPC Affects Versions: 0.95.2 Reporter: Todd Lipcon Assignee: stack Fix For: 0.96.1 Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, 5945-in-progress.patch, 5945v2.txt, buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt The new PB code is sloppy with buffers and makes several needless copies. This increases GC time a lot. A few simple changes can cut this back down. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9869) Optimize HConnectionManager#getCachedLocation
[ https://issues.apache.org/jira/browse/HBASE-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822368#comment-13822368 ] Nicolas Liochon commented on HBASE-9869: bq. See any difference? Whats the math like? How large is the Map if 1M regions in it? We remove when region is bad but ones we don't access could stick around for ever. I suppose that serves us right if we don't access them. Folks like Lars are sensitive about clients being well-behaved especially when embedded in an apps server. I looked around for a simple LRU – the guava one – but we need SortedMap. My feeling is that if you have a table with 1 million regions you need to pay the price for this: i.e. have enough memory on the client. Let me do the math test it however. We can as well do something in the middle: do a LRU on the first map, the one on the tableName. Optimize HConnectionManager#getCachedLocation - Key: HBASE-9869 URL: https://issues.apache.org/jira/browse/HBASE-9869 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9869.v1.patch, 9869.v1.patch, 9869.v2.patch It javadoc says: TODO: This method during writing consumes 15% of CPU doing lookup. This is still true, says Yourkit. With 0.96, we also spend more time in these methods. We retry more, and the AsyncProcess calls it in parallel. I don't have the patch for this yet, but I will spend some time on it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Shi updated HBASE-9969: Attachment: hbase-9969.patch Repost the patch, as hadoop QA thought the benchmark result is the patch Improve KeyValueHeap using loser tree - Key: HBASE-9969 URL: https://issues.apache.org/jira/browse/HBASE-9969 Project: HBase Issue Type: Improvement Reporter: Chao Shi Attachments: hbase-9969.patch, hbase-9969.patch, kvheap-benchmark.png, kvheap-benchmark.txt LoserTree is the better data structure than binary heap. It saves half of the comparisons on each next(), though the time complexity is on O(logN). Currently A scan or get will go through two KeyValueHeaps, one is merging KVs read from multiple HFiles in a single store, the other is merging results from multiple stores. This patch should improve the both cases whenever CPU is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). All of the optimization work is done in KeyValueHeap and does not change its public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8143) HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
[
https://issues.apache.org/jira/browse/HBASE-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822373#comment-13822373
]
stack commented on HBASE-8143:
--
[~enis] A review boss please.
[~xieliang007] Looks like Lars got you over in HDFS-5461
HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
--
Key: HBASE-8143
URL: https://issues.apache.org/jira/browse/HBASE-8143
Project: HBase
Issue Type: Bug
Components: hadoop2
Affects Versions: 0.98.0, 0.94.7, 0.95.0
Reporter: Enis Soztutar
Assignee: stack
Priority: Critical
Fix For: 0.98.0, 0.96.1
Attachments: 8143.hbase-default.xml.txt, 8143doc.txt, 8143v2.txt,
OpenFileTest.java
We've run into an issue with HBase 0.94 on Hadoop2, with SSR turned on that
the memory usage of the HBase process grows to 7g, on an -Xmx3g, after some
time, this causes OOM for the RSs.
Upon further investigation, I've found out that we end up with 200 regions,
each having 3-4 store files open. Under hadoop2 SSR, BlockReaderLocal
allocates DirectBuffers, which is unlike HDFS 1 where there is no direct
buffer allocation.
It seems that there is no guards against the memory used by local buffers in
hdfs 2, and having a large number of open files causes multiple GB of memory
to be consumed from the RS process.
This issue is to further investigate what is going on. Whether we can limit
the memory usage in HDFS, or HBase, and/or document the setup.
Possible mitigation scenarios are:
- Turn off SSR for Hadoop 2
- Ensure that there is enough unallocated memory for the RS based on
expected # of store files
- Ensure that there is lower number of regions per region server (hence
number of open files)
Stack trace:
{code}
org.apache.hadoop.hbase.DroppedSnapshotException: region:
IntegrationTestLoadAndVerify,yC^P\xD7\x945\xD4,1363388517630.24655343d8d356ef708732f34cfe8946.
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1560)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1439)
at
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1380)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:449)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:215)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:63)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:632)
at java.nio.DirectByteBuffer.init(DirectByteBuffer.java:97)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at
org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer(DirectBufferPool.java:70)
at
org.apache.hadoop.hdfs.BlockReaderLocal.init(BlockReaderLocal.java:315)
at
org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:208)
at
org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:790)
at
org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:888)
at
org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:455)
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:645)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:689)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:312)
at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:543)
at
org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:589)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1261)
at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:512)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:603)
at
org.apache.hadoop.hbase.regionserver.Store.validateStoreFile(Store.java:1568)
at
org.apache.hadoop.hbase.regionserver.Store.commitFile(Store.java:845)
at
org.apache.hadoop.hbase.regionserver.Store.access$500(Store.java:109)
at
org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.commit(Store.java:2209)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1541)
{code}
--
This
[jira] [Created] (HBASE-9970) HBase BulkLoad, table is creating with the timestamp key also as a column to the table.
Y. SREENIVASULU REDDY created HBASE-9970:
Summary: HBase BulkLoad, table is creating with the timestamp key
also as a column to the table.
Key: HBASE-9970
URL: https://issues.apache.org/jira/browse/HBASE-9970
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.11
Reporter: Y. SREENIVASULU REDDY
Assignee: Y. SREENIVASULU REDDY
Fix For: 0.98.0, 0.96.1, 0.94.14
If BulkLoad job is running with out creating a table.
job itself will create the table if table is not found.
{quote}
if (!doesTableExist(tableName)) {
createTable(conf, tableName);
}
{quote}
if columns contains timestamp also then table is creating with defined columns
and timestamp key.
{quote}
eg: -Dimporttsv.columns=HBASE_ROW_KEY,HBASE_TS_KEY,d:num
{quote}
table is creating with the following columnFamilies.
'HBASE_TS_KEY' and 'd'
while iterating timestamp key also need to avoid while describing the column
descriptors.
{code}
private static void createTable(HBaseAdmin admin, String tableName, String[]
columns)
throws IOException {
HTableDescriptor htd = new HTableDescriptor(TableName.valueOf(tableName));
SetString cfSet = new HashSetString();
for (String aColumn : columns) {
if (TsvParser.ROWKEY_COLUMN_SPEC.equals(aColumn)) continue;
// we are only concerned with the first one (in case this is a cf:cq)
cfSet.add(aColumn.split(:, 2)[0]);
}
for (String cf : cfSet) {
HColumnDescriptor hcd = new HColumnDescriptor(Bytes.toBytes(cf));
htd.addFamily(hcd);
}
LOG.warn(format(Creating table '%s' with '%s' columns and default
descriptors.,
tableName, cfSet));
admin.createTable(htd);
}
{code}
{quote}
Index:
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
===
--- hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
(revision 1539967)
+++ hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
(working copy)
@@ -413,7 +413,8 @@
HTableDescriptor htd = new HTableDescriptor(TableName.valueOf(tableName));
SetString cfSet = new HashSetString();
for (String aColumn : columns) {
- if (TsvParser.ROWKEY_COLUMN_SPEC.equals(aColumn)) continue;
+ if (TsvParser.ROWKEY_COLUMN_SPEC.equals(aColumn)
+ || TsvParser.TIMESTAMPKEY_COLUMN_SPEC.equals(aColumn)) continue;
// we are only concerned with the first one (in case this is a cf:cq)
cfSet.add(aColumn.split(:, 2)[0]);
}
{quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[
https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822472#comment-13822472
]
Hadoop QA commented on HBASE-5945:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613551/5945v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:red}-1 hadoop2.0{color}. The patch failed to compile against the
hadoop 2.0 profile.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7862//console
This message is automatically generated.
Reduce buffer copies in IPC server response path
Key: HBASE-5945
URL: https://issues.apache.org/jira/browse/HBASE-5945
Project: HBase
Issue Type: Improvement
Components: IPC/RPC
Affects Versions: 0.95.2
Reporter: Todd Lipcon
Assignee: stack
Fix For: 0.96.1
Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch,
5945-in-progress.patch, 5945v2.txt, buffer-copies.txt, even-fewer-copies.txt,
hbase-5495.txt
The new PB code is sloppy with buffers and makes several needless copies.
This increases GC time a lot. A few simple changes can cut this back down.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9970) HBase BulkLoad, table is creating with the timestamp key also as a column to the table.
[
https://issues.apache.org/jira/browse/HBASE-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822484#comment-13822484
]
Anoop Sam John commented on HBASE-9970:
---
Create a patch for 94 and Trunk and attach here Sreenivas. Good catch.
HBase BulkLoad, table is creating with the timestamp key also as a column to
the table.
Key: HBASE-9970
URL: https://issues.apache.org/jira/browse/HBASE-9970
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.11
Reporter: Y. SREENIVASULU REDDY
Assignee: Y. SREENIVASULU REDDY
Fix For: 0.98.0, 0.96.1, 0.94.14
If BulkLoad job is running with out creating a table.
job itself will create the table if table is not found.
{quote}
if (!doesTableExist(tableName)) {
createTable(conf, tableName);
}
{quote}
if columns contains timestamp also then table is creating with defined
columns and timestamp key.
{quote}
eg: -Dimporttsv.columns=HBASE_ROW_KEY,HBASE_TS_KEY,d:num
{quote}
table is creating with the following columnFamilies.
'HBASE_TS_KEY' and 'd'
while iterating timestamp key also need to avoid while describing the column
descriptors.
{code}
private static void createTable(HBaseAdmin admin, String tableName, String[]
columns)
throws IOException {
HTableDescriptor htd = new HTableDescriptor(TableName.valueOf(tableName));
SetString cfSet = new HashSetString();
for (String aColumn : columns) {
if (TsvParser.ROWKEY_COLUMN_SPEC.equals(aColumn)) continue;
// we are only concerned with the first one (in case this is a cf:cq)
cfSet.add(aColumn.split(:, 2)[0]);
}
for (String cf : cfSet) {
HColumnDescriptor hcd = new HColumnDescriptor(Bytes.toBytes(cf));
htd.addFamily(hcd);
}
LOG.warn(format(Creating table '%s' with '%s' columns and default
descriptors.,
tableName, cfSet));
admin.createTable(htd);
}
{code}
{quote}
Index:
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
===
---
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
(revision 1539967)
+++
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
(working copy)
@@ -413,7 +413,8 @@
HTableDescriptor htd = new
HTableDescriptor(TableName.valueOf(tableName));
SetString cfSet = new HashSetString();
for (String aColumn : columns) {
- if (TsvParser.ROWKEY_COLUMN_SPEC.equals(aColumn)) continue;
+ if (TsvParser.ROWKEY_COLUMN_SPEC.equals(aColumn)
+ || TsvParser.TIMESTAMPKEY_COLUMN_SPEC.equals(aColumn)) continue;
// we are only concerned with the first one (in case this is a cf:cq)
cfSet.add(aColumn.split(:, 2)[0]);
}
{quote}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9849) [REST] Forbidden schema delete in read only mode
[ https://issues.apache.org/jira/browse/HBASE-9849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822500#comment-13822500 ] Hudson commented on HBASE-9849: --- SUCCESS: Integrated in HBase-0.94-security #335 (See [https://builds.apache.org/job/HBase-0.94-security/335/]) HBASE-9849 [REST] Forbidden schema delete in read only mode (Julian Zhou) (larsh: rev 1541642) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/rest/TestSchemaResource.java [REST] Forbidden schema delete in read only mode Key: HBASE-9849 URL: https://issues.apache.org/jira/browse/HBASE-9849 Project: HBase Issue Type: Bug Components: REST Affects Versions: 0.98.0, 0.94.14 Reporter: Julian Zhou Assignee: Julian Zhou Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: 9849-0.94-v0.patch, 9849-trunk-v0.patch If hbase.rest.readonly was set, all write operations should be forbidden via REST, right? So table schema deletion should also be forbidden in readonly mode? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-4654) [replication] Add a check to make sure we don't replicate to ourselves
[ https://issues.apache.org/jira/browse/HBASE-4654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822501#comment-13822501 ] Hudson commented on HBASE-4654: --- SUCCESS: Integrated in HBase-0.94-security #335 (See [https://builds.apache.org/job/HBase-0.94-security/335/]) HBASE-4654 [replication] Add a check to make sure we don't replicate to ourselves (Demai Ni) (larsh: rev 1541806) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java [replication] Add a check to make sure we don't replicate to ourselves -- Key: HBASE-4654 URL: https://issues.apache.org/jira/browse/HBASE-4654 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Assignee: Demai Ni Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: 4654-trunk.txt, HBASE-4654-0.94-v0.patch, HBASE-4654-0.96-v0.patch, HBASE-4654-trunk-v0.patch, HBASE-4654-trunk-v0.patch, HBASE-4654-trunk-v0.patch It's currently possible to add a peer for replication and point it to the local cluster, which I believe could very well happen for those like us that use only one ZK ensemble per DC so that only the root znode changes when you want to set up replication intra-DC. I don't think comparing just the cluster ID would be enough because you would normally use a different one for another cluster and nothing will block you from pointing elsewhere. Comparing the ZK ensemble address doesn't work either when you have multiple DNS entries that point at the same place. I think this could be resolved by looking up the master address in the relevant znode as it should be exactly the same thing in the case where you have the same cluster. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-4654) [replication] Add a check to make sure we don't replicate to ourselves
[ https://issues.apache.org/jira/browse/HBASE-4654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822507#comment-13822507 ] Hudson commented on HBASE-4654: --- SUCCESS: Integrated in HBase-0.94 #1201 (See [https://builds.apache.org/job/HBase-0.94/1201/]) HBASE-4654 [replication] Add a check to make sure we don't replicate to ourselves (Demai Ni) (larsh: rev 1541806) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java [replication] Add a check to make sure we don't replicate to ourselves -- Key: HBASE-4654 URL: https://issues.apache.org/jira/browse/HBASE-4654 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Assignee: Demai Ni Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: 4654-trunk.txt, HBASE-4654-0.94-v0.patch, HBASE-4654-0.96-v0.patch, HBASE-4654-trunk-v0.patch, HBASE-4654-trunk-v0.patch, HBASE-4654-trunk-v0.patch It's currently possible to add a peer for replication and point it to the local cluster, which I believe could very well happen for those like us that use only one ZK ensemble per DC so that only the root znode changes when you want to set up replication intra-DC. I don't think comparing just the cluster ID would be enough because you would normally use a different one for another cluster and nothing will block you from pointing elsewhere. Comparing the ZK ensemble address doesn't work either when you have multiple DNS entries that point at the same place. I think this could be resolved by looking up the master address in the relevant znode as it should be exactly the same thing in the case where you have the same cluster. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9849) [REST] Forbidden schema delete in read only mode
[ https://issues.apache.org/jira/browse/HBASE-9849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822506#comment-13822506 ] Hudson commented on HBASE-9849: --- SUCCESS: Integrated in HBase-0.94 #1201 (See [https://builds.apache.org/job/HBase-0.94/1201/]) HBASE-9849 [REST] Forbidden schema delete in read only mode (Julian Zhou) (larsh: rev 1541642) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/rest/TestSchemaResource.java [REST] Forbidden schema delete in read only mode Key: HBASE-9849 URL: https://issues.apache.org/jira/browse/HBASE-9849 Project: HBase Issue Type: Bug Components: REST Affects Versions: 0.98.0, 0.94.14 Reporter: Julian Zhou Assignee: Julian Zhou Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: 9849-0.94-v0.patch, 9849-trunk-v0.patch If hbase.rest.readonly was set, all write operations should be forbidden via REST, right? So table schema deletion should also be forbidden in readonly mode? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree
[
https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822529#comment-13822529
]
Hadoop QA commented on HBASE-9969:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613822/hbase-9969.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 2 release
audit warnings (more than the trunk's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7863//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7863//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7863//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7863//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7863//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7863//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7863//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7863//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7863//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7863//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7863//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7863//console
This message is automatically generated.
Improve KeyValueHeap using loser tree
-
Key: HBASE-9969
URL: https://issues.apache.org/jira/browse/HBASE-9969
Project: HBase
Issue Type: Improvement
Reporter: Chao Shi
Attachments: hbase-9969.patch, hbase-9969.patch,
kvheap-benchmark.png, kvheap-benchmark.txt
LoserTree is the better data structure than binary heap. It saves half of the
comparisons on each next(), though the time complexity is on O(logN).
Currently A scan or get will go through two KeyValueHeaps, one is merging KVs
read from multiple HFiles in a single store, the other is merging results
from multiple stores. This patch should improve the both cases whenever CPU
is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811).
All of the optimization work is done in KeyValueHeap and does not change its
public interfaces. The new code looks more cleaner and simpler to understand.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9962) Improve tag iteration
[ https://issues.apache.org/jira/browse/HBASE-9962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822548#comment-13822548 ] Hudson commented on HBASE-9962: --- SUCCESS: Integrated in HBase-TRUNK #4680 (See [https://builds.apache.org/job/HBase-TRUNK/4680/]) HBASE-9962. Improve tag iteration (apurtell: rev 1541772) * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/Tag.java Improve tag iteration - Key: HBASE-9962 URL: https://issues.apache.org/jira/browse/HBASE-9962 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.98.0 Attachments: 9962.patch The tag iterator was moved out of KeyValue into CellUtil and marked as for tests only. HBASE-7662 and HBASE-7663 will use it. The 'length' parameter was made into a short, which is inconvenient for most callers. The methods on Tag for getting tag data offset and length in the tag buffer were made default scope so it's impossible outside of the package to find the tag data in the backing buffer without calling Tag#asList, which might do some unwanted object allocations. Tags#asList also inconveniently uses short for 'length'. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9963) Remove the ReentrantReadWriteLock in the MemStore
[
https://issues.apache.org/jira/browse/HBASE-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822551#comment-13822551
]
Hudson commented on HBASE-9963:
---
SUCCESS: Integrated in HBase-TRUNK #4680 (See
[https://builds.apache.org/job/HBase-TRUNK/4680/])
HBASE-9963 Remove the ReentrantReadWriteLock in the MemStore (nkeywal: rev
1541880)
*
/hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
Remove the ReentrantReadWriteLock in the MemStore
-
Key: HBASE-9963
URL: https://issues.apache.org/jira/browse/HBASE-9963
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments: 9963.96.v3.patch, 9963.v1.patch, 9963.v2.patch,
9963.v3.patch
If I'm not wrong, the MemStore is always used from the HStore. The code in
HStore puts a lock before calling MemStore. So the lock in Memstore is
useless.
For example, in HStore
{code}
@Override
public long upsert(IterableCell cells, long readpoint) throws IOException
{
this.lock.readLock().lock();
try {
return this.memstore.upsert(cells, readpoint);
} finally {
this.lock.readLock().unlock();
}
}
{code}
With this in MemStore
{code}
public long upsert(IterableCell cells, long readpoint) {
this.lock.readLock().lock(); // ==Am I useful?
try {
long size = 0;
for (Cell cell : cells) {
size += upsert(cell, readpoint);
}
return size;
} finally {
this.lock.readLock().unlock();
}
}
{code}
I've checked, all the locks in MemStore are backed by a lock in HStore, the
only exception beeing
{code}
void snapshot() {
this.memstore.snapshot();
}
{code}
And I would say it's a bug. If it's confirm ([~lhofhansl], what do you
think?), I will add a lock there and remove all of them in MemStore. They do
appear in the profiling.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9849) [REST] Forbidden schema delete in read only mode
[ https://issues.apache.org/jira/browse/HBASE-9849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822547#comment-13822547 ] Hudson commented on HBASE-9849: --- SUCCESS: Integrated in HBase-TRUNK #4680 (See [https://builds.apache.org/job/HBase-TRUNK/4680/]) HBASE-9849 [REST] Forbidden schema delete in read only mode (Julian Zhou) (larsh: rev 1541644) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/TestSchemaResource.java [REST] Forbidden schema delete in read only mode Key: HBASE-9849 URL: https://issues.apache.org/jira/browse/HBASE-9849 Project: HBase Issue Type: Bug Components: REST Affects Versions: 0.98.0, 0.94.14 Reporter: Julian Zhou Assignee: Julian Zhou Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: 9849-0.94-v0.patch, 9849-trunk-v0.patch If hbase.rest.readonly was set, all write operations should be forbidden via REST, right? So table schema deletion should also be forbidden in readonly mode? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9710) Use the region name, not the encoded name, when region is not on current server
[
https://issues.apache.org/jira/browse/HBASE-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822550#comment-13822550
]
Hudson commented on HBASE-9710:
---
SUCCESS: Integrated in HBase-TRUNK #4680 (See
[https://builds.apache.org/job/HBase-TRUNK/4680/])
HBASE-9710 Use the region name, not the encoded name, when region is not on
current server (stack: rev 1541820)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Use the region name, not the encoded name, when region is not on current
server
---
Key: HBASE-9710
URL: https://issues.apache.org/jira/browse/HBASE-9710
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.95.2, 0.96.0
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments:
0001-Log-the-region-name-instead-of-the-encoded-region-na.patch, 9710v2.txt,
9710v2.txt, 9710v3.txt, 9710v4.txt
When we throw a {{RegionOpeningException}} or a {{NotServingRegionException}}
we put the encoded region name in the exception, which isn't super useful. I
propose putting the region name instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9907) Rig to fake a cluster so can profile client behaviors
[ https://issues.apache.org/jira/browse/HBASE-9907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822549#comment-13822549 ] Hudson commented on HBASE-9907: --- SUCCESS: Integrated in HBase-TRUNK #4680 (See [https://builds.apache.org/job/HBase-TRUNK/4680/]) HBASE-9907 Rig to fake a cluster so can profile client behaviors (stack: rev 1541703) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/protobuf/TestProtobufUtil.java Rig to fake a cluster so can profile client behaviors - Key: HBASE-9907 URL: https://issues.apache.org/jira/browse/HBASE-9907 Project: HBase Issue Type: Sub-task Components: Client Affects Versions: 0.96.0 Reporter: stack Assignee: stack Fix For: 0.98.0, 0.96.1 Attachments: 9907.txt, 9907.txt, 9907v2.txt, 9907v3.txt, 9907v3.txt, 9907v3.txt.1, 9907v4.096.txt, 9907v4.txt, 9907v4.txt Patch carried over from HBASE-9775 parent issue. Adds to the TestClientNoCluster#main a rig that allows faking many clients against a few servers and the opposite. Useful for studying client operation. Includes a few changes to pb makings to try and save on a few creations. Also has an edit of javadoc on how to create an HConnection and HTable trying to be more forceful about pointing you in right direction ([~lhofhansl] -- mind reviewing these javadoc changes?) I have a +1 already on this patch up in parent issue. Will run by hadoopqa to make sure all good before commit. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9810) Global memstore size will be calculated wrongly if replaying recovered edits throws exception
[
https://issues.apache.org/jira/browse/HBASE-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822554#comment-13822554
]
Hudson commented on HBASE-9810:
---
SUCCESS: Integrated in HBase-TRUNK #4680 (See
[https://builds.apache.org/job/HBase-TRUNK/4680/])
HBASE-9810 Global memstore size will be calculated wrongly if replaying
recovered edits throws exception (zjushch: rev 1541783)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Global memstore size will be calculated wrongly if replaying recovered edits
throws exception
-
Key: HBASE-9810
URL: https://issues.apache.org/jira/browse/HBASE-9810
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0, 0.96.1
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9810-trunk.patch
Recently we encountered such a case in 0.94-version:
Flush is triggered frequently because:
{noformat}DEBUG org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush
thread woke up because memory above low water=14.4g
{noformat}
But, the real global memstore size is about 1g.
It seems the global memstore size has been calculated wrongly.
Through the logs, I find the following root cause log:
{noformat}
ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
open of region=notifysub2_index,\x83\xDC^\xCD\xA3\x8A\x
E2\x8E\xE6\xAD!\xDC\xE8t\xED,1379148697072.46be7c2d71c555379278a7494df3015e.,
starting to roll back the global memstore size.
java.lang.NegativeArraySizeException
at org.apache.hadoop.hbase.KeyValue.getFamily(KeyValue.java:1096)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:2933)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:2811)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:583)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:499)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3939)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3887)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
Browse the code of this part, seems a critial bug about global memstore size
when replaying recovered edits.
(RegionServerAccounting#clearRegionReplayEditsSize is called for each edit
file, it means the roll back size is smaller than actual when calling
RegionServerAccounting#rollbackRegionReplayEditsSize)
Anyway, the solution is easy as the patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9757) Reenable fast region move in SlowDeterministicMonkey
[ https://issues.apache.org/jira/browse/HBASE-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822553#comment-13822553 ] Hudson commented on HBASE-9757: --- SUCCESS: Integrated in HBase-TRUNK #4680 (See [https://builds.apache.org/job/HBase-TRUNK/4680/]) HBASE-9757 Reenable fast region move in SlowDeterministicMonkey (jxiang: rev 1541811) * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/ChangeEncodingAction.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/SlowDeterministicMonkeyFactory.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java Reenable fast region move in SlowDeterministicMonkey Key: HBASE-9757 URL: https://issues.apache.org/jira/browse/HBASE-9757 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.98.0, 0.96.1 Attachments: 0.96-9757.patch, trunk-9757.patch, trunk-9757_v2.patch HBASE-9338 slows down the region move CM a little so that ITBLL is green for 0.96.0 RC. We should revert the change and make sure the test is still green. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9960) Fix javadoc for CellUtil#tagsIterator()
[
https://issues.apache.org/jira/browse/HBASE-9960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822555#comment-13822555
]
Hudson commented on HBASE-9960:
---
SUCCESS: Integrated in HBase-TRUNK #4680 (See
[https://builds.apache.org/job/HBase-TRUNK/4680/])
HBASE-9960. Fix javadoc for CellUtil#tagsIterator() (apurtell: rev 1541771)
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java
Fix javadoc for CellUtil#tagsIterator()
---
Key: HBASE-9960
URL: https://issues.apache.org/jira/browse/HBASE-9960
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Trivial
Fix For: 0.98.0
Attachments: 9960.txt
The @return tag has no arguments.
{code}
* @return
*/
public static IteratorTag tagsIterator(final byte[] tags, final int
offset, final short length) {
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9958) Remove some array copy, change lock scope in locateRegion
[ https://issues.apache.org/jira/browse/HBASE-9958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822546#comment-13822546 ] Hudson commented on HBASE-9958: --- SUCCESS: Integrated in HBase-TRUNK #4680 (See [https://builds.apache.org/job/HBase-TRUNK/4680/]) HBASE-9958 Remove some array copy, change lock scope in locateRegion (nkeywal: rev 1541688) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java Remove some array copy, change lock scope in locateRegion - Key: HBASE-9958 URL: https://issues.apache.org/jira/browse/HBASE-9958 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9958.v1.patch, 9958.v2.patch, 9958.v2.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[ https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822552#comment-13822552 ] Hudson commented on HBASE-9870: --- SUCCESS: Integrated in HBase-TRUNK #4680 (See [https://builds.apache.org/job/HBase-TRUNK/4680/]) HBASE-9870 HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format (jxiang: rev 1541629) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV3.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV3.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestLoadAndSwitchEncodeOnDisk.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java *
[jira] [Commented] (HBASE-6642) enable_all,disable_all,drop_all can call list command with regex directly.
[
https://issues.apache.org/jira/browse/HBASE-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822594#comment-13822594
]
Jean-Marc Spaggiari commented on HBASE-6642:
Faced the same issue recently. Might be good to fix that.
Now, since Java and Ruby are not sending back the same result for a regex, I
think we should the same API for the 2 filtering. To make sure it applies the
same way on the list. Doing that will allow us to keep the current feature, but
will fix it? Just a suggestion. Not sure if it's doable...
enable_all,disable_all,drop_all can call list command with regex directly.
Key: HBASE-6642
URL: https://issues.apache.org/jira/browse/HBASE-6642
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 0.94.6, 0.95.0
Reporter: Y. SREENIVASULU REDDY
Assignee: rajeshbabu
Attachments: HBASE-6642_trunk.patch
created few tables. then performing disable_all operation in shell prompt.
but it is not performing operation successfully.
{noformat}
hbase(main):043:0 disable_all '*'
table12
zk0113
zk0114
Disable the above 3 tables (y/n)?
y/
3 tables successfully disabled
just it is showing the message but operation is not success.
but the following way only performing successfully
hbase(main):043:0 disable_all '*.*'
table12
zk0113
zk0114
Disable the above 3 tables (y/n)?
y
3 tables successfully disabled
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree
[
https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822603#comment-13822603
]
Ted Yu commented on HBASE-9969:
---
Putting patch on review board would make reviewing easier.
For benchmark B, can you try ColumnPaginationFilter with smaller offset ?
Giving percentage improvement in the tables is desirable.
{code}
+ */
+public class LoserTreeT {
{code}
Please add annotation for audience.
{code}
+ * {@code tree[i]} where i 0 stores the index to greater value between
{@code value[tree[2*i]]}
+ * and {@code value[tree[2*i + 1]]}.
{code}
'value[' should be 'values[', right ?
{code}
+ * @return the index to the minimal elements.
{code}
elements - element
{code}
+ * Pushes next value from the stream that we previously taken the minimal
element from.
{code}
taken - took
{code}
+ * Passes {@code NULL} to value if the stream has been reached EOF.
{code}
Remove 'been'
{code}
+if (index != topIndex()) {
+ throw new IllegalArgumentException(Only the top index can be updated);
{code}
Consider including index and topIndex in the exception message.
{code}
+if (value == null values.get(index) != null) {
+ numOpenStreams--;
+ if (numOpenStreams 0) {
{code}
In what condition would numOpenStreams become negative ?
{code}
+throw new AssertionError(numOpenStreams is negative: +
numOpenStreams);
{code}
Throw IllegalStateException.
{code}
+ public ListInteger getOpenStreamsForTesting() {
{code}
The above is used in KeyValueHeap, consider renaming.
{code}
+ * from bottom up to the root. Once it loses, it stops there and the
winner continues to fight to up.
+ */
+ private void update(int i) {
{code}
'fight to up' - 'fight to top'
Please add @param for i.
KeyValueHeapBenchmark.java and TestLoserTree.java need license.
Improve KeyValueHeap using loser tree
-
Key: HBASE-9969
URL: https://issues.apache.org/jira/browse/HBASE-9969
Project: HBase
Issue Type: Improvement
Reporter: Chao Shi
Attachments: hbase-9969.patch, hbase-9969.patch,
kvheap-benchmark.png, kvheap-benchmark.txt
LoserTree is the better data structure than binary heap. It saves half of the
comparisons on each next(), though the time complexity is on O(logN).
Currently A scan or get will go through two KeyValueHeaps, one is merging KVs
read from multiple HFiles in a single store, the other is merging results
from multiple stores. This patch should improve the both cases whenever CPU
is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811).
All of the optimization work is done in KeyValueHeap and does not change its
public interfaces. The new code looks more cleaner and simpler to understand.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Status: Open (was: Patch Available) Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Status: Patch Available (was: Open) Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Attachment: 9959.v3.patch Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9959) Remove some array copy - server side
[
https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822643#comment-13822643
]
Hadoop QA commented on HBASE-9959:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613876/9959.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitTransaction
org.apache.hadoop.hbase.io.hfile.TestHFileWriterV3
org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransaction
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7864//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7864//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7864//console
This message is automatically generated.
Remove some array copy - server side
Key: HBASE-9959
URL: https://issues.apache.org/jira/browse/HBASE-9959
Project: HBase
Issue Type: Bug
Components: Protobufs, regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch,
9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-4654) [replication] Add a check to make sure we don't replicate to ourselves
[ https://issues.apache.org/jira/browse/HBASE-4654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822669#comment-13822669 ] Hudson commented on HBASE-4654: --- SUCCESS: Integrated in hbase-0.96 #189 (See [https://builds.apache.org/job/hbase-0.96/189/]) HBASE-4654 [replication] Add a check to make sure we don't replicate to ourselves (Demai Ni) (larsh: rev 1541805) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java [replication] Add a check to make sure we don't replicate to ourselves -- Key: HBASE-4654 URL: https://issues.apache.org/jira/browse/HBASE-4654 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Assignee: Demai Ni Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: 4654-trunk.txt, HBASE-4654-0.94-v0.patch, HBASE-4654-0.96-v0.patch, HBASE-4654-trunk-v0.patch, HBASE-4654-trunk-v0.patch, HBASE-4654-trunk-v0.patch It's currently possible to add a peer for replication and point it to the local cluster, which I believe could very well happen for those like us that use only one ZK ensemble per DC so that only the root znode changes when you want to set up replication intra-DC. I don't think comparing just the cluster ID would be enough because you would normally use a different one for another cluster and nothing will block you from pointing elsewhere. Comparing the ZK ensemble address doesn't work either when you have multiple DNS entries that point at the same place. I think this could be resolved by looking up the master address in the relevant znode as it should be exactly the same thing in the case where you have the same cluster. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9810) Global memstore size will be calculated wrongly if replaying recovered edits throws exception
[
https://issues.apache.org/jira/browse/HBASE-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822670#comment-13822670
]
Hudson commented on HBASE-9810:
---
SUCCESS: Integrated in hbase-0.96 #189 (See
[https://builds.apache.org/job/hbase-0.96/189/])
HBASE-9810 Global memstore size will be calculated wrongly if replaying
recovered edits throws exception (zjushch: rev 1541784)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Global memstore size will be calculated wrongly if replaying recovered edits
throws exception
-
Key: HBASE-9810
URL: https://issues.apache.org/jira/browse/HBASE-9810
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0, 0.96.1
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9810-trunk.patch
Recently we encountered such a case in 0.94-version:
Flush is triggered frequently because:
{noformat}DEBUG org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush
thread woke up because memory above low water=14.4g
{noformat}
But, the real global memstore size is about 1g.
It seems the global memstore size has been calculated wrongly.
Through the logs, I find the following root cause log:
{noformat}
ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
open of region=notifysub2_index,\x83\xDC^\xCD\xA3\x8A\x
E2\x8E\xE6\xAD!\xDC\xE8t\xED,1379148697072.46be7c2d71c555379278a7494df3015e.,
starting to roll back the global memstore size.
java.lang.NegativeArraySizeException
at org.apache.hadoop.hbase.KeyValue.getFamily(KeyValue.java:1096)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:2933)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:2811)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:583)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:499)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3939)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3887)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
Browse the code of this part, seems a critial bug about global memstore size
when replaying recovered edits.
(RegionServerAccounting#clearRegionReplayEditsSize is called for each edit
file, it means the roll back size is smaller than actual when calling
RegionServerAccounting#rollbackRegionReplayEditsSize)
Anyway, the solution is easy as the patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9958) Remove some array copy, change lock scope in locateRegion
[ https://issues.apache.org/jira/browse/HBASE-9958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822662#comment-13822662 ] Hudson commented on HBASE-9958: --- SUCCESS: Integrated in hbase-0.96 #189 (See [https://builds.apache.org/job/hbase-0.96/189/]) HBASE-9958 Remove some array copy, change lock scope in locateRegion (nkeywal: rev 1541691) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java Remove some array copy, change lock scope in locateRegion - Key: HBASE-9958 URL: https://issues.apache.org/jira/browse/HBASE-9958 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9958.v1.patch, 9958.v2.patch, 9958.v2.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9849) [REST] Forbidden schema delete in read only mode
[ https://issues.apache.org/jira/browse/HBASE-9849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822663#comment-13822663 ] Hudson commented on HBASE-9849: --- SUCCESS: Integrated in hbase-0.96 #189 (See [https://builds.apache.org/job/hbase-0.96/189/]) HBASE-9849 [REST] Forbidden schema delete in read only mode (Julian Zhou) (larsh: rev 1541643) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/TestSchemaResource.java [REST] Forbidden schema delete in read only mode Key: HBASE-9849 URL: https://issues.apache.org/jira/browse/HBASE-9849 Project: HBase Issue Type: Bug Components: REST Affects Versions: 0.98.0, 0.94.14 Reporter: Julian Zhou Assignee: Julian Zhou Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: 9849-0.94-v0.patch, 9849-trunk-v0.patch If hbase.rest.readonly was set, all write operations should be forbidden via REST, right? So table schema deletion should also be forbidden in readonly mode? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822667#comment-13822667
]
Hudson commented on HBASE-9870:
---
SUCCESS: Integrated in hbase-0.96 #189 (See
[https://builds.apache.org/job/hbase-0.96/189/])
HBASE-9870 HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
(jxiang: rev 1541627)
*
/hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
*
/hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestLoadAndSwitchEncodeOnDisk.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
* /hbase/branches/0.96/hbase-shell/src/main/ruby/hbase/admin.rb
* /hbase/branches/0.96/src/main/docbkx/shell.xml
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9870.patch, trunk-9870_v2.patch,
trunk-9870_v3.patch
In this method, we have
{code}
if
[jira] [Commented] (HBASE-9710) Use the region name, not the encoded name, when region is not on current server
[
https://issues.apache.org/jira/browse/HBASE-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822665#comment-13822665
]
Hudson commented on HBASE-9710:
---
SUCCESS: Integrated in hbase-0.96 #189 (See
[https://builds.apache.org/job/hbase-0.96/189/])
HBASE-9710 Use the region name, not the encoded name, when region is not on
current server (stack: rev 1541821)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Use the region name, not the encoded name, when region is not on current
server
---
Key: HBASE-9710
URL: https://issues.apache.org/jira/browse/HBASE-9710
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.95.2, 0.96.0
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments:
0001-Log-the-region-name-instead-of-the-encoded-region-na.patch, 9710v2.txt,
9710v2.txt, 9710v3.txt, 9710v4.txt
When we throw a {{RegionOpeningException}} or a {{NotServingRegionException}}
we put the encoded region name in the exception, which isn't super useful. I
propose putting the region name instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9757) Reenable fast region move in SlowDeterministicMonkey
[ https://issues.apache.org/jira/browse/HBASE-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822668#comment-13822668 ] Hudson commented on HBASE-9757: --- SUCCESS: Integrated in hbase-0.96 #189 (See [https://builds.apache.org/job/hbase-0.96/189/]) HBASE-9757 Reenable fast region move in SlowDeterministicMonkey (jxiang: rev 1541812) * /hbase/branches/0.96/hbase-common/src/main/resources/hbase-default.xml * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/ChangeEncodingAction.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/SlowDeterministicMonkeyFactory.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java Reenable fast region move in SlowDeterministicMonkey Key: HBASE-9757 URL: https://issues.apache.org/jira/browse/HBASE-9757 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.98.0, 0.96.1 Attachments: 0.96-9757.patch, trunk-9757.patch, trunk-9757_v2.patch HBASE-9338 slows down the region move CM a little so that ITBLL is green for 0.96.0 RC. We should revert the change and make sure the test is still green. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9907) Rig to fake a cluster so can profile client behaviors
[ https://issues.apache.org/jira/browse/HBASE-9907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822664#comment-13822664 ] Hudson commented on HBASE-9907: --- SUCCESS: Integrated in hbase-0.96 #189 (See [https://builds.apache.org/job/hbase-0.96/189/]) HBASE-9907 Rig to fake a cluster so can profile client behaviors (stack: rev 1541708) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/branches/0.96/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/protobuf/TestProtobufUtil.java Rig to fake a cluster so can profile client behaviors - Key: HBASE-9907 URL: https://issues.apache.org/jira/browse/HBASE-9907 Project: HBase Issue Type: Sub-task Components: Client Affects Versions: 0.96.0 Reporter: stack Assignee: stack Fix For: 0.98.0, 0.96.1 Attachments: 9907.txt, 9907.txt, 9907v2.txt, 9907v3.txt, 9907v3.txt, 9907v3.txt.1, 9907v4.096.txt, 9907v4.txt, 9907v4.txt Patch carried over from HBASE-9775 parent issue. Adds to the TestClientNoCluster#main a rig that allows faking many clients against a few servers and the opposite. Useful for studying client operation. Includes a few changes to pb makings to try and save on a few creations. Also has an edit of javadoc on how to create an HConnection and HTable trying to be more forceful about pointing you in right direction ([~lhofhansl] -- mind reviewing these javadoc changes?) I have a +1 already on this patch up in parent issue. Will run by hadoopqa to make sure all good before commit. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9963) Remove the ReentrantReadWriteLock in the MemStore
[
https://issues.apache.org/jira/browse/HBASE-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822666#comment-13822666
]
Hudson commented on HBASE-9963:
---
SUCCESS: Integrated in hbase-0.96 #189 (See
[https://builds.apache.org/job/hbase-0.96/189/])
HBASE-9963 Remove the ReentrantReadWriteLock in the MemStore (nkeywal: rev
1541882)
*
/hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
*
/hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
Remove the ReentrantReadWriteLock in the MemStore
-
Key: HBASE-9963
URL: https://issues.apache.org/jira/browse/HBASE-9963
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments: 9963.96.v3.patch, 9963.v1.patch, 9963.v2.patch,
9963.v3.patch
If I'm not wrong, the MemStore is always used from the HStore. The code in
HStore puts a lock before calling MemStore. So the lock in Memstore is
useless.
For example, in HStore
{code}
@Override
public long upsert(IterableCell cells, long readpoint) throws IOException
{
this.lock.readLock().lock();
try {
return this.memstore.upsert(cells, readpoint);
} finally {
this.lock.readLock().unlock();
}
}
{code}
With this in MemStore
{code}
public long upsert(IterableCell cells, long readpoint) {
this.lock.readLock().lock(); // ==Am I useful?
try {
long size = 0;
for (Cell cell : cells) {
size += upsert(cell, readpoint);
}
return size;
} finally {
this.lock.readLock().unlock();
}
}
{code}
I've checked, all the locks in MemStore are backed by a lock in HStore, the
only exception beeing
{code}
void snapshot() {
this.memstore.snapshot();
}
{code}
And I would say it's a bug. If it's confirm ([~lhofhansl], what do you
think?), I will add a lock there and remove all of them in MemStore. They do
appear in the profiling.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8438) Extend bin/hbase to print a mapreduce classpath
[ https://issues.apache.org/jira/browse/HBASE-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822674#comment-13822674 ] Nick Dimiduk commented on HBASE-8438: - I noticed the DEBUG output as well. It goes to stderr so I planned to pipe that to /dev/null from consuming shell scripts. I can look at changing the log level programmatically if you'd prefer. hadoop-mapreduce-client-core.jar comes from the default Job values (mapOutputKeyClass, mapOutputValueClass, etc). I introduce a new method in HBASE-9165 that excludes these classes. We can switch this patch over to use that method if you prefer. Mind giving that one a review as well? Extend bin/hbase to print a mapreduce classpath - Key: HBASE-8438 URL: https://issues.apache.org/jira/browse/HBASE-8438 Project: HBase Issue Type: Improvement Components: scripts Affects Versions: 0.94.6.1, 0.95.0 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.98.0, 0.96.1 Attachments: 0001-HBASE-8438-Extend-bin-hbase-to-print-a-minimal-class.patch, 0001-HBASE-8438-Extend-bin-hbase-to-print-a-minimal-class.patch, 0001-HBASE-8438-Extend-bin-hbase-to-print-a-minimal-class.patch, 0001-HBASE-8438-Extend-bin-hbase-to-print-a-minimal-class.patch, 0001-HBASE-8438-Extend-bin-hbase-to-print-a-minimal-class.patch, 0001-HBASE-8438-Extend-bin-hbase-to-print-a-minimal-class.patch, 0001-HBASE-8438-Extend-bin-hbase-to-print-a-minimal-class.patch, 0001-HBASE-8438-Extend-bin-hbase-to-print-a-minimal-class.patch For tools like pig and hive, blindly appending the full output of `bin/hbase classpath` to their own CLASSPATH is excessive. They already build CLASSPATH entries for hadoop. All they need from us is the delta entries, the dependencies we require w/o hadoop and all of it's transitive deps. This is also a kindness for Windows, where there's a shorter limit on the length of commandline arguments. See also HIVE-2055 for additional discussion. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9971) Port HBASE-9963 and part of HBASE-9958 to 0.94
Lars Hofhansl created HBASE-9971: Summary: Port HBASE-9963 and part of HBASE-9958 to 0.94 Key: HBASE-9971 URL: https://issues.apache.org/jira/browse/HBASE-9971 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Fix For: 0.94.14 Both of these have nice simple fixes that we should have in 0.94 as well. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9834) Minimize byte[] copies for 'smart' clients
[ https://issues.apache.org/jira/browse/HBASE-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822691#comment-13822691 ] Jesse Yates commented on HBASE-9834: Cool. Committing this afternoon (PST) unless there are any objections Minimize byte[] copies for 'smart' clients -- Key: HBASE-9834 URL: https://issues.apache.org/jira/browse/HBASE-9834 Project: HBase Issue Type: Bug Components: Client Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.14 Attachments: hbase-9834-0.94-v0.patch, hbase-9834-0.94-v1.patch, hbase-9834-0.94-v2.patch, hbase-9834-0.94-v3.patch 'Smart' clients (e.g. phoenix) that have in-depth knowledge of HBase often bemoan the extra byte[] copies that must be done when building multiple puts/deletes. We should provide a mechanism by which they can minimize these copies, but still remain wire compatible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9869) Optimize HConnectionManager#getCachedLocation
[ https://issues.apache.org/jira/browse/HBASE-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822692#comment-13822692 ] Nicolas Liochon commented on HBASE-9869: I've done some tests with TestClientNoCluster with 1 regions ||#clients|#puts|time without the patch|time with the patch|| ||1 client| 100 million| 94 seconds|65 seconds|| ||2 clients| 50 million each| 82 seconds|56 seconds|| ||5 clients| 20 million each| 105 seconds|66 seconds|| With 5 clients, we have 10 threads trying to insert as much as possible, so more clients means more context switches on more memory pressure (it's different if they have to wait for an answer from a server of course). I need to do more tests with more regions. But so far so good I would say. Optimize HConnectionManager#getCachedLocation - Key: HBASE-9869 URL: https://issues.apache.org/jira/browse/HBASE-9869 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9869.v1.patch, 9869.v1.patch, 9869.v2.patch It javadoc says: TODO: This method during writing consumes 15% of CPU doing lookup. This is still true, says Yourkit. With 0.96, we also spend more time in these methods. We retry more, and the AsyncProcess calls it in parallel. I don't have the patch for this yet, but I will spend some time on it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9972) Make HBase export metrics to JMX by default, instead of to NullContext
Gaurav Menghani created HBASE-9972: -- Summary: Make HBase export metrics to JMX by default, instead of to NullContext Key: HBASE-9972 URL: https://issues.apache.org/jira/browse/HBASE-9972 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 0.89-fb Reporter: Gaurav Menghani Fix For: 0.89-fb I was debugging something in the swift branch, and found that HBase doesn't export to JMX by default. The JMX server is being spun-up anyways in single node setup, we might as well export the metrics to it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HBASE-9972) Make HBase export metrics to JMX by default, instead of to NullContext
[ https://issues.apache.org/jira/browse/HBASE-9972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurav Menghani reassigned HBASE-9972: -- Assignee: Gaurav Menghani Make HBase export metrics to JMX by default, instead of to NullContext -- Key: HBASE-9972 URL: https://issues.apache.org/jira/browse/HBASE-9972 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 0.89-fb Reporter: Gaurav Menghani Assignee: Gaurav Menghani Fix For: 0.89-fb I was debugging something in the swift branch, and found that HBase doesn't export to JMX by default. The JMX server is being spun-up anyways in single node setup, we might as well export the metrics to it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree
[
https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822760#comment-13822760
]
Ted Yu commented on HBASE-9969:
---
{code}
* @return true if there are more keys, false if all scanners are done
*/
public boolean next(ListCell result, int limit) throws IOException {
...
+return loserTree.isEmpty();
}
{code}
Should the return value from loserTree.isEmpty() be negated ?
{code}
+ if (isLazy loserTree.getNumOfOpenStreams() 1) {
// If there is only one scanner left, we don't do lazy seek.
{code}
Please update comment above to match the condition.
Improve KeyValueHeap using loser tree
-
Key: HBASE-9969
URL: https://issues.apache.org/jira/browse/HBASE-9969
Project: HBase
Issue Type: Improvement
Reporter: Chao Shi
Attachments: hbase-9969.patch, hbase-9969.patch,
kvheap-benchmark.png, kvheap-benchmark.txt
LoserTree is the better data structure than binary heap. It saves half of the
comparisons on each next(), though the time complexity is on O(logN).
Currently A scan or get will go through two KeyValueHeaps, one is merging KVs
read from multiple HFiles in a single store, the other is merging results
from multiple stores. This patch should improve the both cases whenever CPU
is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811).
All of the optimization work is done in KeyValueHeap and does not change its
public interfaces. The new code looks more cleaner and simpler to understand.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8465) Auto-drop rollback snapshot for snapshot restore
[
https://issues.apache.org/jira/browse/HBASE-8465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822777#comment-13822777
]
Ted Yu commented on HBASE-8465:
---
{code}
+hbase-failsafe-{snapshot.name}-{restore.timestamp});
+ failSafeSnapshotSnapshotName = failSafeSnapshotSnapshotName
+.replace({snapshot.name}, snapshotName)
+.replace({table.name},
tableName.toString().replace(TableName.NAMESPACE_DELIM, '.'))
{code}
{table.name} doesn't appear in the default template. Please add comment in
hbase-default.xml
Auto-drop rollback snapshot for snapshot restore
Key: HBASE-8465
URL: https://issues.apache.org/jira/browse/HBASE-8465
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Matteo Bertozzi
Fix For: 0.98.0, 0.96.1
Attachments: 8465-trunk-v1.txt, 8465-trunk-v2.txt,
HBASE-8465-v3.patch, HBASE-8465-v4.patch
Below is an excerpt from snapshot restore javadoc:
{code}
* Restore the specified snapshot on the original table. (The table must be
disabled)
* Before restoring the table, a new snapshot with the current table state
is created.
* In case of failure, the table will be rolled back to the its original
state.
{code}
We can improve the handling of rollbackSnapshot in two ways:
1. give better name to the rollbackSnapshot (adding
{code}'-for-rollback-'{code}). Currently the name is of the form:
String rollbackSnapshot = snapshotName + - +
EnvironmentEdgeManager.currentTimeMillis();
2. drop rollbackSnapshot at the end of restoreSnapshot() if the restore is
successful. We can introduce new config param, named
'hbase.snapshot.restore.drop.rollback', to keep compatibility with current
behavior.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9973) [ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade to 0.96.x from 0.94.x or 0.92.x
[
https://issues.apache.org/jira/browse/HBASE-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Shulman updated HBASE-9973:
-
Labels: acl (was: )
[ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade
to 0.96.x from 0.94.x or 0.92.x
Key: HBASE-9973
URL: https://issues.apache.org/jira/browse/HBASE-9973
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.96.0, 0.96.1
Reporter: Aleksandr Shulman
Labels: acl
Fix For: 0.96.1
In our testing, we have uncovered that the ACL permissions for users with the
'A' credential do not hold after the upgrade to 0.96.x.
This is because in the ACL table, the entry for the admin user is a
permission on the '_acl_' table with permission 'A'. However, because of the
namespace transition, there is no longer an '_acl_' table. Therefore, that
entry in the hbase:acl table is no longer valid.
Example:
{code}hbase(main):002:0 scan 'hbase:acl'
ROW COLUMN+CELL
TestTablecolumn=l:hdfs, timestamp=1384454830701, value=RW
TestTablecolumn=l:root, timestamp=1384455875586, value=RWCA
_acl_column=l:root, timestamp=1384454767568, value=C
_acl_column=l:tableAdmin, timestamp=1384454788035, value=A
hbase:aclcolumn=l:root, timestamp=1384455875786, value=C
{code}
In this case, the following entry becomes meaningless:
{code} _acl_column=l:tableAdmin, timestamp=1384454788035,
value=A {code}
As a result,
Proposed fix:
I see the fix being relatively straightforward. As part of the migration,
change any entries in the '_acl_' table with key '_acl_' into a new row with
key 'hbase:acl', all else being the same. And the old entry would be deleted.
This can go into the standard migration script that we expect users to run.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-9973) [ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade to 0.96.x from 0.94.x or 0.92.x
Aleksandr Shulman created HBASE-9973:
Summary: [ACL]: Users with 'Admin' ACL permission will lose
permissions after upgrade to 0.96.x from 0.94.x or 0.92.x
Key: HBASE-9973
URL: https://issues.apache.org/jira/browse/HBASE-9973
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.96.0, 0.96.1
Reporter: Aleksandr Shulman
Fix For: 0.96.1
In our testing, we have uncovered that the ACL permissions for users with the
'A' credential do not hold after the upgrade to 0.96.x.
This is because in the ACL table, the entry for the admin user is a permission
on the '_acl_' table with permission 'A'. However, because of the namespace
transition, there is no longer an '_acl_' table. Therefore, that entry in the
hbase:acl table is no longer valid.
Example:
{code}hbase(main):002:0 scan 'hbase:acl'
ROW COLUMN+CELL
TestTablecolumn=l:hdfs, timestamp=1384454830701, value=RW
TestTablecolumn=l:root, timestamp=1384455875586, value=RWCA
_acl_column=l:root, timestamp=1384454767568, value=C
_acl_column=l:tableAdmin, timestamp=1384454788035, value=A
hbase:aclcolumn=l:root, timestamp=1384455875786, value=C
{code}
In this case, the following entry becomes meaningless:
{code} _acl_column=l:tableAdmin, timestamp=1384454788035,
value=A {code}
As a result,
Proposed fix:
I see the fix being relatively straightforward. As part of the migration,
change any entries in the '_acl_' table with key '_acl_' into a new row with
key 'hbase:acl', all else being the same. And the old entry would be deleted.
This can go into the standard migration script that we expect users to run.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9973) [ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade to 0.96.x from 0.94.x or 0.92.x
[
https://issues.apache.org/jira/browse/HBASE-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Shulman updated HBASE-9973:
-
Assignee: Himanshu Vashishtha
[ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade
to 0.96.x from 0.94.x or 0.92.x
Key: HBASE-9973
URL: https://issues.apache.org/jira/browse/HBASE-9973
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.96.0, 0.96.1
Reporter: Aleksandr Shulman
Assignee: Himanshu Vashishtha
Labels: acl
Fix For: 0.96.1
In our testing, we have uncovered that the ACL permissions for users with the
'A' credential do not hold after the upgrade to 0.96.x.
This is because in the ACL table, the entry for the admin user is a
permission on the '_acl_' table with permission 'A'. However, because of the
namespace transition, there is no longer an '_acl_' table. Therefore, that
entry in the hbase:acl table is no longer valid.
Example:
{code}hbase(main):002:0 scan 'hbase:acl'
ROW COLUMN+CELL
TestTablecolumn=l:hdfs, timestamp=1384454830701, value=RW
TestTablecolumn=l:root, timestamp=1384455875586, value=RWCA
_acl_column=l:root, timestamp=1384454767568, value=C
_acl_column=l:tableAdmin, timestamp=1384454788035, value=A
hbase:aclcolumn=l:root, timestamp=1384455875786, value=C
{code}
In this case, the following entry becomes meaningless:
{code} _acl_column=l:tableAdmin, timestamp=1384454788035,
value=A {code}
As a result,
Proposed fix:
I see the fix being relatively straightforward. As part of the migration,
change any entries in the '_acl_' table with key '_acl_' into a new row with
key 'hbase:acl', all else being the same. And the old entry would be deleted.
This can go into the standard migration script that we expect users to run.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9908) [WINDOWS] Fix filesystem / classloader related unit tests
[
https://issues.apache.org/jira/browse/HBASE-9908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822817#comment-13822817
]
Nick Dimiduk commented on HBASE-9908:
-
Cool. +1
[WINDOWS] Fix filesystem / classloader related unit tests
-
Key: HBASE-9908
URL: https://issues.apache.org/jira/browse/HBASE-9908
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9908_v1-addendum.patch, hbase-9908_v1.patch
Some of the unit tests related to classloasing and filesystem are failing on
windows.
{code}
org.apache.hadoop.hbase.coprocessor.TestClassLoading.testHBase3810
org.apache.hadoop.hbase.coprocessor.TestClassLoading.testClassLoadingFromLocalFS
org.apache.hadoop.hbase.coprocessor.TestClassLoading.testPrivateClassLoader
org.apache.hadoop.hbase.coprocessor.TestClassLoading.testClassLoadingFromRelativeLibDirInJar
org.apache.hadoop.hbase.coprocessor.TestClassLoading.testClassLoadingFromLibDirInJar
org.apache.hadoop.hbase.coprocessor.TestClassLoading.testClassLoadingFromHDFS
org.apache.hadoop.hbase.backup.TestHFileArchiving.testCleaningRace
org.apache.hadoop.hbase.regionserver.wal.TestDurability.testDurability
org.apache.hadoop.hbase.regionserver.wal.TestHLog.testMaintainOrderWithConcurrentWrites
org.apache.hadoop.hbase.security.access.TestAccessController.testBulkLoad
org.apache.hadoop.hbase.regionserver.TestHRegion.testRecoveredEditsReplayCompaction
org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait.testRecoveredEditsReplayCompaction
org.apache.hadoop.hbase.util.TestFSUtils.testRenameAndSetModifyTime
{code}
The root causes are:
- Using local file name for referring to hdfs paths (HBASE-6830)
- Classloader using the wrong file system
- StoreFile readers not being closed (for unfinished compaction)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9961) [WINDOWS] Multicast should bind to local address
[ https://issues.apache.org/jira/browse/HBASE-9961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822827#comment-13822827 ] Nicolas Liochon commented on HBASE-9961: +1 [WINDOWS] Multicast should bind to local address Key: HBASE-9961 URL: https://issues.apache.org/jira/browse/HBASE-9961 Project: HBase Issue Type: Bug Components: Client Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.98.0, 0.96.1 Attachments: hbase-9961_v1.patch, hbase-9961_v2.patch Binding to a multicast address (such as hbase.status.multicast.address.ip) seems to be the preferred method on most unix systems and linux(2,3). At least in RedHat, binding to multicast address might not filter out other traffic coming to the same port, but for different multi cast groups (2)]. However, on windows, you cannot bind to a non local (class D) address (1), which seems to be correct according to the spec. # http://msdn.microsoft.com/en-us/library/ms737550%28v=vs.85%29.aspx # https://bugzilla.redhat.com/show_bug.cgi?id=231899 # http://stackoverflow.com/questions/10692956/what-does-it-mean-to-bind-a-multicast-udp-socket # https://issues.jboss.org/browse/JGRP-515 The solution is to bind to mcast address on linux, but a local address on windows. TestHCM is also failing because of this. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HBASE-9834) Minimize byte[] copies for 'smart' clients
[ https://issues.apache.org/jira/browse/HBASE-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates resolved HBASE-9834. Resolution: Fixed Committed to 0.94. thanks for the reviews all. Minimize byte[] copies for 'smart' clients -- Key: HBASE-9834 URL: https://issues.apache.org/jira/browse/HBASE-9834 Project: HBase Issue Type: Bug Components: Client Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.14 Attachments: hbase-9834-0.94-v0.patch, hbase-9834-0.94-v1.patch, hbase-9834-0.94-v2.patch, hbase-9834-0.94-v3.patch 'Smart' clients (e.g. phoenix) that have in-depth knowledge of HBase often bemoan the extra byte[] copies that must be done when building multiple puts/deletes. We should provide a mechanism by which they can minimize these copies, but still remain wire compatible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9834) Minimize byte[] copies for 'smart' clients
[ https://issues.apache.org/jira/browse/HBASE-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822866#comment-13822866 ] stack commented on HBASE-9834: -- +1 Thanks for removing dup'd code. Minimize byte[] copies for 'smart' clients -- Key: HBASE-9834 URL: https://issues.apache.org/jira/browse/HBASE-9834 Project: HBase Issue Type: Bug Components: Client Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.94.14 Attachments: hbase-9834-0.94-v0.patch, hbase-9834-0.94-v1.patch, hbase-9834-0.94-v2.patch, hbase-9834-0.94-v3.patch 'Smart' clients (e.g. phoenix) that have in-depth knowledge of HBase often bemoan the extra byte[] copies that must be done when building multiple puts/deletes. We should provide a mechanism by which they can minimize these copies, but still remain wire compatible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9969: Assignee: Chao Shi Improve KeyValueHeap using loser tree - Key: HBASE-9969 URL: https://issues.apache.org/jira/browse/HBASE-9969 Project: HBase Issue Type: Improvement Reporter: Chao Shi Assignee: Chao Shi Attachments: hbase-9969.patch, hbase-9969.patch, kvheap-benchmark.png, kvheap-benchmark.txt LoserTree is the better data structure than binary heap. It saves half of the comparisons on each next(), though the time complexity is on O(logN). Currently A scan or get will go through two KeyValueHeaps, one is merging KVs read from multiple HFiles in a single store, the other is merging results from multiple stores. This patch should improve the both cases whenever CPU is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). All of the optimization work is done in KeyValueHeap and does not change its public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9958) Remove some array copy, change lock scope in locateRegion
[ https://issues.apache.org/jira/browse/HBASE-9958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822882#comment-13822882 ] Hudson commented on HBASE-9958: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #837 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/837/]) HBASE-9958 Remove some array copy, change lock scope in locateRegion (nkeywal: rev 1541688) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java Remove some array copy, change lock scope in locateRegion - Key: HBASE-9958 URL: https://issues.apache.org/jira/browse/HBASE-9958 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9958.v1.patch, 9958.v2.patch, 9958.v2.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9849) [REST] Forbidden schema delete in read only mode
[ https://issues.apache.org/jira/browse/HBASE-9849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822883#comment-13822883 ] Hudson commented on HBASE-9849: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #837 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/837/]) HBASE-9849 [REST] Forbidden schema delete in read only mode (Julian Zhou) (larsh: rev 1541644) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/TestSchemaResource.java [REST] Forbidden schema delete in read only mode Key: HBASE-9849 URL: https://issues.apache.org/jira/browse/HBASE-9849 Project: HBase Issue Type: Bug Components: REST Affects Versions: 0.98.0, 0.94.14 Reporter: Julian Zhou Assignee: Julian Zhou Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: 9849-0.94-v0.patch, 9849-trunk-v0.patch If hbase.rest.readonly was set, all write operations should be forbidden via REST, right? So table schema deletion should also be forbidden in readonly mode? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9710) Use the region name, not the encoded name, when region is not on current server
[
https://issues.apache.org/jira/browse/HBASE-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822886#comment-13822886
]
Hudson commented on HBASE-9710:
---
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #837 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/837/])
HBASE-9710 Use the region name, not the encoded name, when region is not on
current server (stack: rev 1541820)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Use the region name, not the encoded name, when region is not on current
server
---
Key: HBASE-9710
URL: https://issues.apache.org/jira/browse/HBASE-9710
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.95.2, 0.96.0
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments:
0001-Log-the-region-name-instead-of-the-encoded-region-na.patch, 9710v2.txt,
9710v2.txt, 9710v3.txt, 9710v4.txt
When we throw a {{RegionOpeningException}} or a {{NotServingRegionException}}
we put the encoded region name in the exception, which isn't super useful. I
propose putting the region name instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9810) Global memstore size will be calculated wrongly if replaying recovered edits throws exception
[
https://issues.apache.org/jira/browse/HBASE-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822890#comment-13822890
]
Hudson commented on HBASE-9810:
---
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #837 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/837/])
HBASE-9810 Global memstore size will be calculated wrongly if replaying
recovered edits throws exception (zjushch: rev 1541783)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Global memstore size will be calculated wrongly if replaying recovered edits
throws exception
-
Key: HBASE-9810
URL: https://issues.apache.org/jira/browse/HBASE-9810
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0, 0.96.1
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9810-trunk.patch
Recently we encountered such a case in 0.94-version:
Flush is triggered frequently because:
{noformat}DEBUG org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush
thread woke up because memory above low water=14.4g
{noformat}
But, the real global memstore size is about 1g.
It seems the global memstore size has been calculated wrongly.
Through the logs, I find the following root cause log:
{noformat}
ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
open of region=notifysub2_index,\x83\xDC^\xCD\xA3\x8A\x
E2\x8E\xE6\xAD!\xDC\xE8t\xED,1379148697072.46be7c2d71c555379278a7494df3015e.,
starting to roll back the global memstore size.
java.lang.NegativeArraySizeException
at org.apache.hadoop.hbase.KeyValue.getFamily(KeyValue.java:1096)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:2933)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:2811)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:583)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:499)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3939)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3887)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
Browse the code of this part, seems a critial bug about global memstore size
when replaying recovered edits.
(RegionServerAccounting#clearRegionReplayEditsSize is called for each edit
file, it means the roll back size is smaller than actual when calling
RegionServerAccounting#rollbackRegionReplayEditsSize)
Anyway, the solution is easy as the patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9960) Fix javadoc for CellUtil#tagsIterator()
[
https://issues.apache.org/jira/browse/HBASE-9960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822891#comment-13822891
]
Hudson commented on HBASE-9960:
---
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #837 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/837/])
HBASE-9960. Fix javadoc for CellUtil#tagsIterator() (apurtell: rev 1541771)
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java
Fix javadoc for CellUtil#tagsIterator()
---
Key: HBASE-9960
URL: https://issues.apache.org/jira/browse/HBASE-9960
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Trivial
Fix For: 0.98.0
Attachments: 9960.txt
The @return tag has no arguments.
{code}
* @return
*/
public static IteratorTag tagsIterator(final byte[] tags, final int
offset, final short length) {
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9907) Rig to fake a cluster so can profile client behaviors
[ https://issues.apache.org/jira/browse/HBASE-9907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822885#comment-13822885 ] Hudson commented on HBASE-9907: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #837 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/837/]) HBASE-9907 Rig to fake a cluster so can profile client behaviors (stack: rev 1541703) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/protobuf/TestProtobufUtil.java Rig to fake a cluster so can profile client behaviors - Key: HBASE-9907 URL: https://issues.apache.org/jira/browse/HBASE-9907 Project: HBase Issue Type: Sub-task Components: Client Affects Versions: 0.96.0 Reporter: stack Assignee: stack Fix For: 0.98.0, 0.96.1 Attachments: 9907.txt, 9907.txt, 9907v2.txt, 9907v3.txt, 9907v3.txt, 9907v3.txt.1, 9907v4.096.txt, 9907v4.txt, 9907v4.txt Patch carried over from HBASE-9775 parent issue. Adds to the TestClientNoCluster#main a rig that allows faking many clients against a few servers and the opposite. Useful for studying client operation. Includes a few changes to pb makings to try and save on a few creations. Also has an edit of javadoc on how to create an HConnection and HTable trying to be more forceful about pointing you in right direction ([~lhofhansl] -- mind reviewing these javadoc changes?) I have a +1 already on this patch up in parent issue. Will run by hadoopqa to make sure all good before commit. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9757) Reenable fast region move in SlowDeterministicMonkey
[ https://issues.apache.org/jira/browse/HBASE-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822889#comment-13822889 ] Hudson commented on HBASE-9757: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #837 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/837/]) HBASE-9757 Reenable fast region move in SlowDeterministicMonkey (jxiang: rev 1541811) * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/ChangeEncodingAction.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/SlowDeterministicMonkeyFactory.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java Reenable fast region move in SlowDeterministicMonkey Key: HBASE-9757 URL: https://issues.apache.org/jira/browse/HBASE-9757 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.98.0, 0.96.1 Attachments: 0.96-9757.patch, trunk-9757.patch, trunk-9757_v2.patch HBASE-9338 slows down the region move CM a little so that ITBLL is green for 0.96.0 RC. We should revert the change and make sure the test is still green. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[ https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822888#comment-13822888 ] Hudson commented on HBASE-9870: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #837 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/837/]) HBASE-9870 HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format (jxiang: rev 1541629) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV3.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV3.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestLoadAndSwitchEncodeOnDisk.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java *
[jira] [Commented] (HBASE-9963) Remove the ReentrantReadWriteLock in the MemStore
[
https://issues.apache.org/jira/browse/HBASE-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822887#comment-13822887
]
Hudson commented on HBASE-9963:
---
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #837 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/837/])
HBASE-9963 Remove the ReentrantReadWriteLock in the MemStore (nkeywal: rev
1541880)
*
/hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
* /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
Remove the ReentrantReadWriteLock in the MemStore
-
Key: HBASE-9963
URL: https://issues.apache.org/jira/browse/HBASE-9963
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments: 9963.96.v3.patch, 9963.v1.patch, 9963.v2.patch,
9963.v3.patch
If I'm not wrong, the MemStore is always used from the HStore. The code in
HStore puts a lock before calling MemStore. So the lock in Memstore is
useless.
For example, in HStore
{code}
@Override
public long upsert(IterableCell cells, long readpoint) throws IOException
{
this.lock.readLock().lock();
try {
return this.memstore.upsert(cells, readpoint);
} finally {
this.lock.readLock().unlock();
}
}
{code}
With this in MemStore
{code}
public long upsert(IterableCell cells, long readpoint) {
this.lock.readLock().lock(); // ==Am I useful?
try {
long size = 0;
for (Cell cell : cells) {
size += upsert(cell, readpoint);
}
return size;
} finally {
this.lock.readLock().unlock();
}
}
{code}
I've checked, all the locks in MemStore are backed by a lock in HStore, the
only exception beeing
{code}
void snapshot() {
this.memstore.snapshot();
}
{code}
And I would say it's a bug. If it's confirm ([~lhofhansl], what do you
think?), I will add a lock there and remove all of them in MemStore. They do
appear in the profiling.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9962) Improve tag iteration
[ https://issues.apache.org/jira/browse/HBASE-9962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822884#comment-13822884 ] Hudson commented on HBASE-9962: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #837 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/837/]) HBASE-9962. Improve tag iteration (apurtell: rev 1541772) * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/Tag.java Improve tag iteration - Key: HBASE-9962 URL: https://issues.apache.org/jira/browse/HBASE-9962 Project: HBase Issue Type: Bug Affects Versions: 0.98.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.98.0 Attachments: 9962.patch The tag iterator was moved out of KeyValue into CellUtil and marked as for tests only. HBASE-7662 and HBASE-7663 will use it. The 'length' parameter was made into a short, which is inconvenient for most callers. The methods on Tag for getting tag data offset and length in the tag buffer were made default scope so it's impossible outside of the package to find the tag data in the backing buffer without calling Tag#asList, which might do some unwanted object allocations. Tags#asList also inconveniently uses short for 'length'. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9969: Component/s: regionserver Improve KeyValueHeap using loser tree - Key: HBASE-9969 URL: https://issues.apache.org/jira/browse/HBASE-9969 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Chao Shi Assignee: Chao Shi Attachments: hbase-9969.patch, hbase-9969.patch, kvheap-benchmark.png, kvheap-benchmark.txt LoserTree is the better data structure than binary heap. It saves half of the comparisons on each next(), though the time complexity is on O(logN). Currently A scan or get will go through two KeyValueHeaps, one is merging KVs read from multiple HFiles in a single store, the other is merging results from multiple stores. This patch should improve the both cases whenever CPU is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). All of the optimization work is done in KeyValueHeap and does not change its public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9459) Backport 8534 Fix coverage for org.apache.hadoop.hbase.mapreduce to 0.94
[ https://issues.apache.org/jira/browse/HBASE-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822896#comment-13822896 ] Nick Dimiduk commented on HBASE-9459: - We have HBASE-9484 for backporting to 0.96. Sorry, I haven't gotten to it either. Backport 8534 Fix coverage for org.apache.hadoop.hbase.mapreduce to 0.94 -- Key: HBASE-9459 URL: https://issues.apache.org/jira/browse/HBASE-9459 Project: HBase Issue Type: Test Components: mapreduce, test Reporter: Nick Dimiduk Assignee: Ivan A. Veselovsky Fix For: 0.94.15 Attachments: HBASE-9459-0.94--n3.patch Do you want this test update backported? See HBASE-8534 for a 0.94 patch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9907) Rig to fake a cluster so can profile client behaviors
[ https://issues.apache.org/jira/browse/HBASE-9907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822909#comment-13822909 ] Hudson commented on HBASE-9907: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #119 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/119/]) HBASE-9907 Rig to fake a cluster so can profile client behaviors (stack: rev 1541708) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/branches/0.96/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/protobuf/TestProtobufUtil.java Rig to fake a cluster so can profile client behaviors - Key: HBASE-9907 URL: https://issues.apache.org/jira/browse/HBASE-9907 Project: HBase Issue Type: Sub-task Components: Client Affects Versions: 0.96.0 Reporter: stack Assignee: stack Fix For: 0.98.0, 0.96.1 Attachments: 9907.txt, 9907.txt, 9907v2.txt, 9907v3.txt, 9907v3.txt, 9907v3.txt.1, 9907v4.096.txt, 9907v4.txt, 9907v4.txt Patch carried over from HBASE-9775 parent issue. Adds to the TestClientNoCluster#main a rig that allows faking many clients against a few servers and the opposite. Useful for studying client operation. Includes a few changes to pb makings to try and save on a few creations. Also has an edit of javadoc on how to create an HConnection and HTable trying to be more forceful about pointing you in right direction ([~lhofhansl] -- mind reviewing these javadoc changes?) I have a +1 already on this patch up in parent issue. Will run by hadoopqa to make sure all good before commit. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9112) Custom TableInputFormat in initTableMapperJob throws ClassNoFoundException on TableMapper
[ https://issues.apache.org/jira/browse/HBASE-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9112: Fix Version/s: 0.96.1 0.98.0 Custom TableInputFormat in initTableMapperJob throws ClassNoFoundException on TableMapper - Key: HBASE-9112 URL: https://issues.apache.org/jira/browse/HBASE-9112 Project: HBase Issue Type: Bug Components: hadoop2, mapreduce Affects Versions: 0.94.6.1 Environment: CDH-4.3.0-1.cdh4.3.0.p0.22 Reporter: Debanjan Bhattacharyya Assignee: Nick Dimiduk Fix For: 0.98.0, 0.96.1 When using custom TableInputFormat in TableMapReduceUtil.initTableMapperJob in the following way TableMapReduceUtil.initTableMapperJob(mytable, MyScan, MyMapper.class, MyKey.class, MyValue.class, myJob,true, MyTableInputFormat.class); I get error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableMapper at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) If I do not use the last two parameters, there is no error. What is going wrong here? Thanks Regards -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822912#comment-13822912
]
Hudson commented on HBASE-9870:
---
SUCCESS: Integrated in hbase-0.96-hadoop2 #119 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/119/])
HBASE-9870 HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
(jxiang: rev 1541627)
*
/hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
*
/hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestLoadAndSwitchEncodeOnDisk.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
* /hbase/branches/0.96/hbase-shell/src/main/ruby/hbase/admin.rb
* /hbase/branches/0.96/src/main/docbkx/shell.xml
HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
-
Key: HBASE-9870
URL: https://issues.apache.org/jira/browse/HBASE-9870
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9870.patch, trunk-9870_v2.patch,
trunk-9870_v3.patch
In this method, we have
{code}
[jira] [Commented] (HBASE-9958) Remove some array copy, change lock scope in locateRegion
[ https://issues.apache.org/jira/browse/HBASE-9958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822907#comment-13822907 ] Hudson commented on HBASE-9958: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #119 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/119/]) HBASE-9958 Remove some array copy, change lock scope in locateRegion (nkeywal: rev 1541691) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java Remove some array copy, change lock scope in locateRegion - Key: HBASE-9958 URL: https://issues.apache.org/jira/browse/HBASE-9958 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9958.v1.patch, 9958.v2.patch, 9958.v2.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9963) Remove the ReentrantReadWriteLock in the MemStore
[
https://issues.apache.org/jira/browse/HBASE-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822911#comment-13822911
]
Hudson commented on HBASE-9963:
---
SUCCESS: Integrated in hbase-0.96-hadoop2 #119 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/119/])
HBASE-9963 Remove the ReentrantReadWriteLock in the MemStore (nkeywal: rev
1541882)
*
/hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
*
/hbase/branches/0.96/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
Remove the ReentrantReadWriteLock in the MemStore
-
Key: HBASE-9963
URL: https://issues.apache.org/jira/browse/HBASE-9963
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments: 9963.96.v3.patch, 9963.v1.patch, 9963.v2.patch,
9963.v3.patch
If I'm not wrong, the MemStore is always used from the HStore. The code in
HStore puts a lock before calling MemStore. So the lock in Memstore is
useless.
For example, in HStore
{code}
@Override
public long upsert(IterableCell cells, long readpoint) throws IOException
{
this.lock.readLock().lock();
try {
return this.memstore.upsert(cells, readpoint);
} finally {
this.lock.readLock().unlock();
}
}
{code}
With this in MemStore
{code}
public long upsert(IterableCell cells, long readpoint) {
this.lock.readLock().lock(); // ==Am I useful?
try {
long size = 0;
for (Cell cell : cells) {
size += upsert(cell, readpoint);
}
return size;
} finally {
this.lock.readLock().unlock();
}
}
{code}
I've checked, all the locks in MemStore are backed by a lock in HStore, the
only exception beeing
{code}
void snapshot() {
this.memstore.snapshot();
}
{code}
And I would say it's a bug. If it's confirm ([~lhofhansl], what do you
think?), I will add a lock there and remove all of them in MemStore. They do
appear in the profiling.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9710) Use the region name, not the encoded name, when region is not on current server
[
https://issues.apache.org/jira/browse/HBASE-9710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822910#comment-13822910
]
Hudson commented on HBASE-9710:
---
SUCCESS: Integrated in hbase-0.96-hadoop2 #119 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/119/])
HBASE-9710 Use the region name, not the encoded name, when region is not on
current server (stack: rev 1541821)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Use the region name, not the encoded name, when region is not on current
server
---
Key: HBASE-9710
URL: https://issues.apache.org/jira/browse/HBASE-9710
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.95.2, 0.96.0
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments:
0001-Log-the-region-name-instead-of-the-encoded-region-na.patch, 9710v2.txt,
9710v2.txt, 9710v3.txt, 9710v4.txt
When we throw a {{RegionOpeningException}} or a {{NotServingRegionException}}
we put the encoded region name in the exception, which isn't super useful. I
propose putting the region name instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9810) Global memstore size will be calculated wrongly if replaying recovered edits throws exception
[
https://issues.apache.org/jira/browse/HBASE-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822915#comment-13822915
]
Hudson commented on HBASE-9810:
---
SUCCESS: Integrated in hbase-0.96-hadoop2 #119 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/119/])
HBASE-9810 Global memstore size will be calculated wrongly if replaying
recovered edits throws exception (zjushch: rev 1541784)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Global memstore size will be calculated wrongly if replaying recovered edits
throws exception
-
Key: HBASE-9810
URL: https://issues.apache.org/jira/browse/HBASE-9810
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0, 0.96.1
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9810-trunk.patch
Recently we encountered such a case in 0.94-version:
Flush is triggered frequently because:
{noformat}DEBUG org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush
thread woke up because memory above low water=14.4g
{noformat}
But, the real global memstore size is about 1g.
It seems the global memstore size has been calculated wrongly.
Through the logs, I find the following root cause log:
{noformat}
ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
open of region=notifysub2_index,\x83\xDC^\xCD\xA3\x8A\x
E2\x8E\xE6\xAD!\xDC\xE8t\xED,1379148697072.46be7c2d71c555379278a7494df3015e.,
starting to roll back the global memstore size.
java.lang.NegativeArraySizeException
at org.apache.hadoop.hbase.KeyValue.getFamily(KeyValue.java:1096)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:2933)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:2811)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:583)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:499)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3939)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3887)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
Browse the code of this part, seems a critial bug about global memstore size
when replaying recovered edits.
(RegionServerAccounting#clearRegionReplayEditsSize is called for each edit
file, it means the roll back size is smaller than actual when calling
RegionServerAccounting#rollbackRegionReplayEditsSize)
Anyway, the solution is easy as the patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9849) [REST] Forbidden schema delete in read only mode
[ https://issues.apache.org/jira/browse/HBASE-9849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822908#comment-13822908 ] Hudson commented on HBASE-9849: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #119 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/119/]) HBASE-9849 [REST] Forbidden schema delete in read only mode (Julian Zhou) (larsh: rev 1541643) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/SchemaResource.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/TestSchemaResource.java [REST] Forbidden schema delete in read only mode Key: HBASE-9849 URL: https://issues.apache.org/jira/browse/HBASE-9849 Project: HBase Issue Type: Bug Components: REST Affects Versions: 0.98.0, 0.94.14 Reporter: Julian Zhou Assignee: Julian Zhou Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: 9849-0.94-v0.patch, 9849-trunk-v0.patch If hbase.rest.readonly was set, all write operations should be forbidden via REST, right? So table schema deletion should also be forbidden in readonly mode? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-4654) [replication] Add a check to make sure we don't replicate to ourselves
[ https://issues.apache.org/jira/browse/HBASE-4654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822914#comment-13822914 ] Hudson commented on HBASE-4654: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #119 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/119/]) HBASE-4654 [replication] Add a check to make sure we don't replicate to ourselves (Demai Ni) (larsh: rev 1541805) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java [replication] Add a check to make sure we don't replicate to ourselves -- Key: HBASE-4654 URL: https://issues.apache.org/jira/browse/HBASE-4654 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Assignee: Demai Ni Fix For: 0.98.0, 0.96.1, 0.94.14 Attachments: 4654-trunk.txt, HBASE-4654-0.94-v0.patch, HBASE-4654-0.96-v0.patch, HBASE-4654-trunk-v0.patch, HBASE-4654-trunk-v0.patch, HBASE-4654-trunk-v0.patch It's currently possible to add a peer for replication and point it to the local cluster, which I believe could very well happen for those like us that use only one ZK ensemble per DC so that only the root znode changes when you want to set up replication intra-DC. I don't think comparing just the cluster ID would be enough because you would normally use a different one for another cluster and nothing will block you from pointing elsewhere. Comparing the ZK ensemble address doesn't work either when you have multiple DNS entries that point at the same place. I think this could be resolved by looking up the master address in the relevant znode as it should be exactly the same thing in the case where you have the same cluster. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9757) Reenable fast region move in SlowDeterministicMonkey
[ https://issues.apache.org/jira/browse/HBASE-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822913#comment-13822913 ] Hudson commented on HBASE-9757: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #119 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/119/]) HBASE-9757 Reenable fast region move in SlowDeterministicMonkey (jxiang: rev 1541812) * /hbase/branches/0.96/hbase-common/src/main/resources/hbase-default.xml * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/ChangeEncodingAction.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/SlowDeterministicMonkeyFactory.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java Reenable fast region move in SlowDeterministicMonkey Key: HBASE-9757 URL: https://issues.apache.org/jira/browse/HBASE-9757 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.98.0, 0.96.1 Attachments: 0.96-9757.patch, trunk-9757.patch, trunk-9757_v2.patch HBASE-9338 slows down the region move CM a little so that ITBLL is green for 0.96.0 RC. We should revert the change and make sure the test is still green. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9854) initial documentation for stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-9854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822928#comment-13822928 ] Sergey Shelukhin commented on HBASE-9854: - Hmm... which part of doc should this go to? initial documentation for stripe compactions Key: HBASE-9854 URL: https://issues.apache.org/jira/browse/HBASE-9854 Project: HBase Issue Type: Sub-task Components: Compaction Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Initial documentation for stripe compactions (distill from attached docs, make up to date, put somewhere like book) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9854) initial documentation for stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-9854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822936#comment-13822936 ] Nick Dimiduk commented on HBASE-9854: - How about http://hbase.apache.org/book.html#compaction ? initial documentation for stripe compactions Key: HBASE-9854 URL: https://issues.apache.org/jira/browse/HBASE-9854 Project: HBase Issue Type: Sub-task Components: Compaction Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Initial documentation for stripe compactions (distill from attached docs, make up to date, put somewhere like book) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822944#comment-13822944 ] stack commented on HBASE-9969: -- [~stepinto] Very nice work. Thank you for digging in on this thorny area. That is a nice provable improvement (numbers look great). Thanks for putting up the graphic and the benchmarking tool. I agree the code is now cleaner. Improve KeyValueHeap using loser tree - Key: HBASE-9969 URL: https://issues.apache.org/jira/browse/HBASE-9969 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Chao Shi Assignee: Chao Shi Attachments: hbase-9969.patch, hbase-9969.patch, kvheap-benchmark.png, kvheap-benchmark.txt LoserTree is the better data structure than binary heap. It saves half of the comparisons on each next(), though the time complexity is on O(logN). Currently A scan or get will go through two KeyValueHeaps, one is merging KVs read from multiple HFiles in a single store, the other is merging results from multiple stores. This patch should improve the both cases whenever CPU is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). All of the optimization work is done in KeyValueHeap and does not change its public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
