[jira] [Updated] (HBASE-11144) Filter to support scan multiple row key ranges
[ https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiajia Li updated HBASE-11144: -- Attachment: HBASE_11144_V9.patch upload the HBASE_11144_V9.patch, according Ram's comment. > Filter to support scan multiple row key ranges > -- > > Key: HBASE-11144 > URL: https://issues.apache.org/jira/browse/HBASE-11144 > Project: HBase > Issue Type: Improvement > Components: Filters >Reporter: Jiajia Li >Assignee: Jiajia Li > Attachments: HBASE_11144_4.patch, HBASE_11144_V5.patch, > HBASE_11144_V6.patch, HBASE_11144_V7.patch, HBASE_11144_V9.patch, > MultiRowRangeFilter.patch, MultiRowRangeFilter2.patch, > MultiRowRangeFilter3.patch, hbase_11144_V8.patch > > > HBase is quite efficient when scanning only one small row key range. If user > needs to specify multiple row key ranges in one scan, the typical solutions > are: 1. through FilterList which is a list of row key Filters, 2. using the > SQL layer over HBase to join with two table, such as hive, phoenix etc. > However, both solutions are inefficient. Both of them can’t utilize the range > info to perform fast forwarding during scan which is quite time consuming. If > the number of ranges are quite big (e.g. millions), join is a proper solution > though it is slow. However, there are cases that user wants to specify a > small number of ranges to scan (e.g. <1000 ranges). Both solutions can’t > provide satisfactory performance in such case. > We provide this filter (MultiRowRangeFilter) to support such use case (scan > multiple row key ranges), which can construct the row key ranges from user > specified list and perform fast-forwarding during scan. Thus, the scan will > be quite efficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10869) Add FS_TYPE in HBaseTestingUtility configuration
[ https://issues.apache.org/jira/browse/HBASE-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Kabra updated HBASE-10869: --- Description: Motivation of doing this is two-folded: 1. local file system is more stable than `MiniDFSCluster`, so for cases only testing on HBase logic, using this may avoid unstable problems caused by DFS unstable problem. 1. Starting of local file systme is much faster. Currently only a few of the testcases is switch to LFS mode for testing. Some other diff may switch more. Implementaiton For `FS_TYPE_DFS`, same as before. For `FS_TYPE_LFS`, use `LocalFileSystem` was: Motivation of doing this is two-folded: 1. local file system is more stable than `MiniDFSCluster`, so for cases only testing on HBase logic, using this may avoid unstable problems caused by DFS unstable problem. 1. Starting of local file systme is much faster. Currently only a few of the testcases is switch to LFS mode for testing. Some other diff may switch more. Implementaiton For `FS_TYPE_DFS`, same as before. For `FS_TYPE_LFS`, use `LocalFileSyst > Add FS_TYPE in HBaseTestingUtility configuration > > > Key: HBASE-10869 > URL: https://issues.apache.org/jira/browse/HBASE-10869 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration >Affects Versions: 0.89-fb >Reporter: @deprecated Yi Deng >Priority: Minor > Labels: test > Fix For: 0.89-fb > > > Motivation of doing this is two-folded: > 1. local file system is more stable than `MiniDFSCluster`, so for cases only > testing on HBase logic, using this may avoid unstable problems caused by DFS > unstable problem. > 1. Starting of local file systme is much faster. > Currently only a few of the testcases is switch to LFS mode for testing. Some > other diff may switch more. > Implementaiton > For `FS_TYPE_DFS`, same as before. > For `FS_TYPE_LFS`, use `LocalFileSystem` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10869) Add FS_TYPE in HBaseTestingUtility configuration
[ https://issues.apache.org/jira/browse/HBASE-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Kabra updated HBASE-10869: --- Description: Motivation of doing this is two-folded: 1. local file system is more stable than `MiniDFSCluster`, so for cases only testing on HBase logic, using this may avoid unstable problems caused by DFS unstable problem. 1. Starting of local file systme is much faster. Currently only a few of the testcases is switch to LFS mode for testing. Some other diff may switch more. Implementaiton For `FS_TYPE_DFS`, same as before. For `FS_TYPE_LFS`, use `LocalFileSyst was: Motivation of doing this is two-folded: 1. local file system is more stable than `MiniDFSCluster`, so for cases only testing on HBase logic, using this may avoid unstable problems caused by DFS unstable problem. 1. Starting of local file systme is much faster. Currently only a few of the testcases is switch to LFS mode for testing. Some other diff may switch more. Implementaiton For `FS_TYPE_DFS`, same as before. For `FS_TYPE_LFS`, use `LocalFileSystem` > Add FS_TYPE in HBaseTestingUtility configuration > > > Key: HBASE-10869 > URL: https://issues.apache.org/jira/browse/HBASE-10869 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration >Affects Versions: 0.89-fb >Reporter: @deprecated Yi Deng >Priority: Minor > Labels: test > Fix For: 0.89-fb > > > Motivation of doing this is two-folded: > 1. local file system is more stable than `MiniDFSCluster`, so for cases only > testing on HBase logic, using this may avoid unstable problems caused by DFS > unstable problem. > 1. Starting of local file systme is much faster. > Currently only a few of the testcases is switch to LFS mode for testing. Some > other diff may switch more. > Implementaiton > For `FS_TYPE_DFS`, same as before. > For `FS_TYPE_LFS`, use `LocalFileSyst -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267308#comment-14267308 ] Andrew Purtell commented on HBASE-10528: See above Lars :-) > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 > > Attachments: 10528-1.0.addendum, HBASE-10528-0.94.patch, > HBASE-10528-0.98.patch, HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267302#comment-14267302
]
stack commented on HBASE-12728:
---
bq. Why? [The flush on BufferedTable is a noop, right?]
Because in the proposal, the BufferedTable is not the owner of the put buffer;
it is sharing it with other BufferedTable instances.
bq. I was thinking BufferedTable would be essentially what HTable is now.
We could do that. Its where we started. Only the BufferedTable would have to be
threadsafe (which HTable is not) or we'd need to add back a pool that gave out
these BufferedTables. BufferedTable would not be lightweight; each would be
carrying a fat buffer at least.
The latest suggestion has us taking buffering out of Table and explicitly
managing it separately. Cleaner and clearer as to what is going on.
bq. BufferedConnection would just return BufferedTables as opposed to HTables.
If multiple threads, they'd all share the one, thread-safe instance? Could
only flush the buffer when size/time limits broached or when
BufferedConnection#close was called (or add BufferedConnection#flush); calls to
BufferedTable#flush would have to be ignored. What happens when a thread calls
#close on a shared instance? We don't want one behavior if a Table instance
(i.e. create and close when done) and another when BufferedTable (don't close
it!)
> buffered writes substantially less useful after removal of HTablePool
> -
>
> Key: HBASE-12728
> URL: https://issues.apache.org/jira/browse/HBASE-12728
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 0.98.0
>Reporter: Aaron Beppu
>Assignee: Solomon Duskis
>Priority: Blocker
> Fix For: 1.0.0, 2.0.0, 1.1.0
>
>
> In previous versions of HBase, when use of HTablePool was encouraged, HTable
> instances were long-lived in that pool, and for that reason, if autoFlush was
> set to false, the table instance could accumulate a full buffer of writes
> before a flush was triggered. Writes from the client to the cluster could
> then be substantially larger and less frequent than without buffering.
> However, when HTablePool was deprecated, the primary justification seems to
> have been that creating HTable instances is cheap, so long as the connection
> and executor service being passed to it are pre-provided. A use pattern was
> encouraged where users should create a new HTable instance for every
> operation, using an existing connection and executor service, and then close
> the table. In this pattern, buffered writes are substantially less useful;
> writes are as small and as frequent as they would have been with
> autoflush=true, except the synchronous write is moved from the operation
> itself to the table close call which immediately follows.
> More concretely :
> ```
> // Given these two helpers ...
> private HTableInterface getAutoFlushTable(String tableName) throws
> IOException {
> // (autoflush is true by default)
> return storedConnection.getTable(tableName, executorService);
> }
> private HTableInterface getBufferedTable(String tableName) throws IOException
> {
> HTableInterface table = getAutoFlushTable(tableName);
> table.setAutoFlush(false);
> return table;
> }
> // it's my contention that these two methods would behave almost identically,
> // except the first will hit a synchronous flush during the put call,
> and the second will
> // flush during the (hidden) close call on table.
> private void writeAutoFlushed(Put somePut) throws IOException {
> try (HTableInterface table = getAutoFlushTable(tableName)) {
> table.put(somePut); // will do synchronous flush
> }
> }
> private void writeBuffered(Put somePut) throws IOException {
> try (HTableInterface table = getBufferedTable(tableName)) {
> table.put(somePut);
> } // auto-close will trigger synchronous flush
> }
> ```
> For buffered writes to actually provide a performance benefit to users, one
> of two things must happen:
> - The writeBuffer itself shouldn't live, flush and die with the lifecycle of
> it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
> lifespan, this could cease to be an issue. However, if the same writeBuffer
> is appended to by multiple tables, then some additional concurrency control
> will be needed around it.
> - Alternatively, there should be some pattern for having long-lived HTable
> instances. However, since HTable is not thread-safe, we'd need multiple
> instances, and a mechanism for leasing them out safely -- which sure sounds a
> lot like the old HTablePool to me.
> See discussion on mailing list here :
> http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhC
[jira] [Commented] (HBASE-11533) AsciiDoctor POC
[ https://issues.apache.org/jira/browse/HBASE-11533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267285#comment-14267285 ] stack commented on HBASE-11533: --- Tried it. Looks really nice. Src is more than markdown it seems but we can figure it; you just need to tell us what to do. I built the site but docbook still hooked up. Purge it! I'd be +1 for going over [~misty]. The [~jmhsieh] experience is troubling. Is later asciidoctor more stable you think? Nice work. > AsciiDoctor POC > --- > > Key: HBASE-11533 > URL: https://issues.apache.org/jira/browse/HBASE-11533 > Project: HBase > Issue Type: Sub-task > Components: build, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Minor > Fix For: 2.0.0 > > > Convert or create a subset of documentation in Asciidoc and integrate > Asciidoctor into the Maven build. http://asciidoctor.org/ > Get some folks to play around with it afterward. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267284#comment-14267284 ] Lars Hofhansl commented on HBASE-10528: --- +1 on 0.94. [~apurtell], you good for 0.98? > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 > > Attachments: 10528-1.0.addendum, HBASE-10528-0.94.patch, > HBASE-10528-0.98.patch, HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267283#comment-14267283
]
Lars Hofhansl commented on HBASE-12728:
---
BufferedConnection would just return BufferedTables as opposed to HTables.
> buffered writes substantially less useful after removal of HTablePool
> -
>
> Key: HBASE-12728
> URL: https://issues.apache.org/jira/browse/HBASE-12728
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 0.98.0
>Reporter: Aaron Beppu
>Assignee: Solomon Duskis
>Priority: Blocker
> Fix For: 1.0.0, 2.0.0, 1.1.0
>
>
> In previous versions of HBase, when use of HTablePool was encouraged, HTable
> instances were long-lived in that pool, and for that reason, if autoFlush was
> set to false, the table instance could accumulate a full buffer of writes
> before a flush was triggered. Writes from the client to the cluster could
> then be substantially larger and less frequent than without buffering.
> However, when HTablePool was deprecated, the primary justification seems to
> have been that creating HTable instances is cheap, so long as the connection
> and executor service being passed to it are pre-provided. A use pattern was
> encouraged where users should create a new HTable instance for every
> operation, using an existing connection and executor service, and then close
> the table. In this pattern, buffered writes are substantially less useful;
> writes are as small and as frequent as they would have been with
> autoflush=true, except the synchronous write is moved from the operation
> itself to the table close call which immediately follows.
> More concretely :
> ```
> // Given these two helpers ...
> private HTableInterface getAutoFlushTable(String tableName) throws
> IOException {
> // (autoflush is true by default)
> return storedConnection.getTable(tableName, executorService);
> }
> private HTableInterface getBufferedTable(String tableName) throws IOException
> {
> HTableInterface table = getAutoFlushTable(tableName);
> table.setAutoFlush(false);
> return table;
> }
> // it's my contention that these two methods would behave almost identically,
> // except the first will hit a synchronous flush during the put call,
> and the second will
> // flush during the (hidden) close call on table.
> private void writeAutoFlushed(Put somePut) throws IOException {
> try (HTableInterface table = getAutoFlushTable(tableName)) {
> table.put(somePut); // will do synchronous flush
> }
> }
> private void writeBuffered(Put somePut) throws IOException {
> try (HTableInterface table = getBufferedTable(tableName)) {
> table.put(somePut);
> } // auto-close will trigger synchronous flush
> }
> ```
> For buffered writes to actually provide a performance benefit to users, one
> of two things must happen:
> - The writeBuffer itself shouldn't live, flush and die with the lifecycle of
> it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
> lifespan, this could cease to be an issue. However, if the same writeBuffer
> is appended to by multiple tables, then some additional concurrency control
> will be needed around it.
> - Alternatively, there should be some pattern for having long-lived HTable
> instances. However, since HTable is not thread-safe, we'd need multiple
> instances, and a mechanism for leasing them out safely -- which sure sounds a
> lot like the old HTablePool to me.
> See discussion on mailing list here :
> http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12801) Failed to truncate a table while maintaing binary region boundaries
[
https://issues.apache.org/jira/browse/HBASE-12801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267264#comment-14267264
]
Liu Shaohui commented on HBASE-12801:
-
[~busbey]
try
{code}
create 't1', 'f1', {NUMREGIONS => 256, SPLITALGO => 'UniformSplit'}
truncate_preserve 't1'
{code}
If you test the trunk code, you should test with a Hmaster without
truncateTable api.
See: admin.rb #399
> Failed to truncate a table while maintaing binary region boundaries
> ---
>
> Key: HBASE-12801
> URL: https://issues.apache.org/jira/browse/HBASE-12801
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 0.94.11
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.28
>
> Attachments: HBASE-12801-0.94-v1.diff, HBASE-12801-trunk-v1.diff
>
>
> Binary region boundaries become wrong during
> converting byte array to normal string, and back to byte array in
> truncate_preserve of admin.rb, which makes the truncation of table failed.
> See: truncate_preserve method in admin.rb
> {code}
> splits = h_table.getRegionLocations().keys().map{|i|
> Bytes.toString(i.getStartKey)}.delete_if{|k| k == ""}.to_java :String
> splits = org.apache.hadoop.hbase.util.Bytes.toByteArrays(splits)
> {code}
> eg:
> {code}
> \xFA\x00\x00\x00\x00\x00\x00\x00 -> \xEF\xBF\xBD\x00\x00\x00\x00\x00\x00\x00
> \xFC\x00\x00\x00\x00\x00\x00\x00 -> \xEF\xBF\xBD\x00\x00\x00\x00\x00\x00\x00
> {code}
> Simple patch is using binary string instead of normal string.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11533) AsciiDoctor POC
[ https://issues.apache.org/jira/browse/HBASE-11533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267262#comment-14267262 ] Misty Stanley-Jones commented on HBASE-11533: - I pushed a branch HBASE-11533 which holds the asciidoc files in src/main/asciidoc, and includes the updated POM which will build the asciidoc during the site goal, alongside the Docbook. The asciidoc output is in target/docs. We would need to do a redirect from the docbkx to docs if we decide to do this. A few things to note: * Both HTML-single and HTML-multi get built, but there is no index for the HTML-multi. This is because I think that most people prefer the HTML-single, but I don't want to break anyone's legacy links. So if someone tries to get to a HTML-multi link for a CHAPTER, they will get there fine and it will render that chapter. In the footer is a link back to the main (HTML-single) doc, and one to the HBase website. If they have bookmarked a section they will get a 404. We could maybe fix that with a redirect too. * All existing anchors have been preserved with the same names, so cross-references that were there before are still there, and again, bookmarks will be less likely to break. * Currently, Disqus is not enabled. Nobody seems to be using it, even though we fixed it in Docbook HTML-single and HTML-multi. It would be trivial to put it back but I'm not sure it is worth the bother. We have not had a single question on it since we fixed it up. So please give it a try. Again, you can build as before. You can even skip the Javadoc builds by doing mvn clean pre-site. The hbase-default.adoc is created from the hbase-default.xml on the fly. I think everything should work as expected and (IMHO) the output is excellent. Oh also, look in the Contributing to the Documentation appendix, where I have done up a bit of an Asciidoc cheat sheet. > AsciiDoctor POC > --- > > Key: HBASE-11533 > URL: https://issues.apache.org/jira/browse/HBASE-11533 > Project: HBase > Issue Type: Sub-task > Components: build, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Minor > Fix For: 2.0.0 > > > Convert or create a subset of documentation in Asciidoc and integrate > Asciidoctor into the Maven build. http://asciidoctor.org/ > Get some folks to play around with it afterward. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12333) Add Integration Test Runner which is more friendly
[
https://issues.apache.org/jira/browse/HBASE-12333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267237#comment-14267237
]
Hadoop QA commented on HBASE-12333:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690469/HBASE-12333-v3.patch
against master branch at commit 45fc2bdd863ff238491f4941b3da04a38731d7e1.
ATTACHMENT ID: 12690469
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12336//console
This message is automatically generated.
> Add Integration Test Runner which is more friendly
> --
>
> Key: HBASE-12333
> URL: https://issues.apache.org/jira/browse/HBASE-12333
> Project: HBase
> Issue Type: New Feature
> Components: test
>Affects Versions: 2.0.0
>Reporter: Manukranth Kolloju
>Assignee: Manukranth Kolloju
> Fix For: 2.0.0
>
> Attachments:
> 0001-HBASE-12333-Add-Integration-Test-Runner-which-is-mor.patch,
> HBASE-12333-v2.patch, HBASE-12333-v3.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This Jira is intended to add a Driver class which would run a list of
> Integration tests on an actual hbase cluster. And generate a machine readable
> results file in JSON and put it on HDFS. The idea is to make it easy to run
> driver class using a long list of appropriate command line params and wait
> for the JSON file on the HDFS. This will help in plugging into external
> automation and makes it easier to maintain Continuous Integration Scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11144) Filter to support scan multiple row key ranges
[ https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267194#comment-14267194 ] Jiajia Li commented on HBASE-11144: --- hi, [~ram_krish], can you look at the comment on https://reviews.apache.org/r/21370/, and give your advise? Thanks > Filter to support scan multiple row key ranges > -- > > Key: HBASE-11144 > URL: https://issues.apache.org/jira/browse/HBASE-11144 > Project: HBase > Issue Type: Improvement > Components: Filters >Reporter: Jiajia Li >Assignee: Jiajia Li > Attachments: HBASE_11144_4.patch, HBASE_11144_V5.patch, > HBASE_11144_V6.patch, HBASE_11144_V7.patch, MultiRowRangeFilter.patch, > MultiRowRangeFilter2.patch, MultiRowRangeFilter3.patch, hbase_11144_V8.patch > > > HBase is quite efficient when scanning only one small row key range. If user > needs to specify multiple row key ranges in one scan, the typical solutions > are: 1. through FilterList which is a list of row key Filters, 2. using the > SQL layer over HBase to join with two table, such as hive, phoenix etc. > However, both solutions are inefficient. Both of them can’t utilize the range > info to perform fast forwarding during scan which is quite time consuming. If > the number of ranges are quite big (e.g. millions), join is a proper solution > though it is slow. However, there are cases that user wants to specify a > small number of ranges to scan (e.g. <1000 ranges). Both solutions can’t > provide satisfactory performance in such case. > We provide this filter (MultiRowRangeFilter) to support such use case (scan > multiple row key ranges), which can construct the row key ranges from user > specified list and perform fast-forwarding during scan. Thus, the scan will > be quite efficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267187#comment-14267187
]
Nick Dimiduk commented on HBASE-12728:
--
bq. I was thinking BufferedTable would be essentially what HTable is now.
[snip] Personally I'd favor an API where the have Connection and
BufferedConnection.
If BufferedTable is doing all the buffering that HTable currently does, but it
designed to be consumed from multiple threads, what is BufferedConnection
doing? It sounds like BufferedTable is pretty self-contained. Just add a
Connection.getBufferedTable method.
> buffered writes substantially less useful after removal of HTablePool
> -
>
> Key: HBASE-12728
> URL: https://issues.apache.org/jira/browse/HBASE-12728
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 0.98.0
>Reporter: Aaron Beppu
>Assignee: Solomon Duskis
>Priority: Blocker
> Fix For: 1.0.0, 2.0.0, 1.1.0
>
>
> In previous versions of HBase, when use of HTablePool was encouraged, HTable
> instances were long-lived in that pool, and for that reason, if autoFlush was
> set to false, the table instance could accumulate a full buffer of writes
> before a flush was triggered. Writes from the client to the cluster could
> then be substantially larger and less frequent than without buffering.
> However, when HTablePool was deprecated, the primary justification seems to
> have been that creating HTable instances is cheap, so long as the connection
> and executor service being passed to it are pre-provided. A use pattern was
> encouraged where users should create a new HTable instance for every
> operation, using an existing connection and executor service, and then close
> the table. In this pattern, buffered writes are substantially less useful;
> writes are as small and as frequent as they would have been with
> autoflush=true, except the synchronous write is moved from the operation
> itself to the table close call which immediately follows.
> More concretely :
> ```
> // Given these two helpers ...
> private HTableInterface getAutoFlushTable(String tableName) throws
> IOException {
> // (autoflush is true by default)
> return storedConnection.getTable(tableName, executorService);
> }
> private HTableInterface getBufferedTable(String tableName) throws IOException
> {
> HTableInterface table = getAutoFlushTable(tableName);
> table.setAutoFlush(false);
> return table;
> }
> // it's my contention that these two methods would behave almost identically,
> // except the first will hit a synchronous flush during the put call,
> and the second will
> // flush during the (hidden) close call on table.
> private void writeAutoFlushed(Put somePut) throws IOException {
> try (HTableInterface table = getAutoFlushTable(tableName)) {
> table.put(somePut); // will do synchronous flush
> }
> }
> private void writeBuffered(Put somePut) throws IOException {
> try (HTableInterface table = getBufferedTable(tableName)) {
> table.put(somePut);
> } // auto-close will trigger synchronous flush
> }
> ```
> For buffered writes to actually provide a performance benefit to users, one
> of two things must happen:
> - The writeBuffer itself shouldn't live, flush and die with the lifecycle of
> it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
> lifespan, this could cease to be an issue. However, if the same writeBuffer
> is appended to by multiple tables, then some additional concurrency control
> will be needed around it.
> - Alternatively, there should be some pattern for having long-lived HTable
> instances. However, since HTable is not thread-safe, we'd need multiple
> instances, and a mechanism for leasing them out safely -- which sure sounds a
> lot like the old HTablePool to me.
> See discussion on mailing list here :
> http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12815) Deprecate 0.89-fb specific Data Structures like KeyValue, WALEdit etc
[
https://issues.apache.org/jira/browse/HBASE-12815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267181#comment-14267181
]
Hadoop QA commented on HBASE-12815:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690474/0001-HBASE-12815-Remove-HBase-specific-Data-structures-li.patch
against master branch at commit 45fc2bdd863ff238491f4941b3da04a38731d7e1.
ATTACHMENT ID: 12690474
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 49 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12337//console
This message is automatically generated.
> Deprecate 0.89-fb specific Data Structures like KeyValue, WALEdit etc
> -
>
> Key: HBASE-12815
> URL: https://issues.apache.org/jira/browse/HBASE-12815
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: Rishit Shroff
>Assignee: Rishit Shroff
>Priority: Minor
> Attachments:
> 0001-HBASE-12815-Remove-HBase-specific-Data-structures-li.patch
>
>
> OSS HBase as different versions of data structures and the current module was
> retaining old ones from 0.89-fb.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12815) Deprecate 0.89-fb specific Data Structures like KeyValue, WALEdit etc
[ https://issues.apache.org/jira/browse/HBASE-12815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rishit Shroff updated HBASE-12815: -- Status: Patch Available (was: Open) > Deprecate 0.89-fb specific Data Structures like KeyValue, WALEdit etc > - > > Key: HBASE-12815 > URL: https://issues.apache.org/jira/browse/HBASE-12815 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Rishit Shroff >Assignee: Rishit Shroff >Priority: Minor > Attachments: > 0001-HBASE-12815-Remove-HBase-specific-Data-structures-li.patch > > > OSS HBase as different versions of data structures and the current module was > retaining old ones from 0.89-fb. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12815) Deprecate 0.89-fb specific Data Structures like KeyValue, WALEdit etc
[ https://issues.apache.org/jira/browse/HBASE-12815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rishit Shroff updated HBASE-12815: -- Attachment: 0001-HBASE-12815-Remove-HBase-specific-Data-structures-li.patch > Deprecate 0.89-fb specific Data Structures like KeyValue, WALEdit etc > - > > Key: HBASE-12815 > URL: https://issues.apache.org/jira/browse/HBASE-12815 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Rishit Shroff >Assignee: Rishit Shroff >Priority: Minor > Attachments: > 0001-HBASE-12815-Remove-HBase-specific-Data-structures-li.patch > > > OSS HBase as different versions of data structures and the current module was > retaining old ones from 0.89-fb. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12815) Deprecate 0.89-fb specific Data Structures like KeyValue, WALEdit etc
Rishit Shroff created HBASE-12815: - Summary: Deprecate 0.89-fb specific Data Structures like KeyValue, WALEdit etc Key: HBASE-12815 URL: https://issues.apache.org/jira/browse/HBASE-12815 Project: HBase Issue Type: Sub-task Reporter: Rishit Shroff Assignee: Rishit Shroff Priority: Minor OSS HBase as different versions of data structures and the current module was retaining old ones from 0.89-fb. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8410) Basic quota support for namespaces
[
https://issues.apache.org/jira/browse/HBASE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267172#comment-14267172
]
Enis Soztutar commented on HBASE-8410:
--
nvm. You had two RB links.
> Basic quota support for namespaces
> --
>
> Key: HBASE-8410
> URL: https://issues.apache.org/jira/browse/HBASE-8410
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Liu
>Assignee: Vandana Ayyalasomayajula
> Attachments: HBASE-8410-master.1.patch, HBASE-8410-master.4.patch,
> HBASE-8410-master.5.patch, HBASE-8410-master.6.patch,
> HBASE-8410-master.patch, HBASE-8410.docx, HBASE-8410_trunk_14.patch
>
>
> This task involves creating an observer which provides basic quota support to
> namespaces in terms of (1) number of tables and (2) number of regions. The
> quota support can be enabled by setting:
>
> hbase.coprocessor.region.classes
> org.apache.hadoop.hbase.namespace.NamespaceController
>
>
> hbase.coprocessor.master.classes
> org.apache.hadoop.hbase.namespace.NamespaceController
>
> in the hbase-site.xml.
> To add quotas to namespace, while creating namespace properties need to be
> added.
> Examples:
> 1. namespace_create 'ns1', {'hbase.namespace.quota.maxregion'=>'10'}
> 2. 1. namespace_create 'ns2', {'hbase.namespace.quota.maxtables'=>'2'},
> {'hbase.namespace.quota.maxregion'=>'5'}
> The quotas can be modified/added to namespace at any point of time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267171#comment-14267171 ] Lars Hofhansl commented on HBASE-10528: --- Wow... That sounds bad. +1 for 0.94 generally, lemme have a brief look at the patch tonight. Will commit unless I see a problem with it. > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 > > Attachments: 10528-1.0.addendum, HBASE-10528-0.94.patch, > HBASE-10528-0.98.patch, HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12071) Separate out thread pool for Master <-> RegionServer communication
[ https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267170#comment-14267170 ] Stephen Yuan Jiang commented on HBASE-12071: [~ndimiduk] To answer your comments, I increased the queues count to 2 to make it more scalable; this is an arbitrary number, ideally it should be tunable, but I am not going far for now. The major goal is to make admin requests using priorityExecutor - I stepped through the code in debugger and verified the results by running relevant Unit Tests. So funcational-wise, it works as expected. Since I have not run system tests (stress tests), whether this change solves customers problem is unknown. We will run system tests in a late cycle to see the results. At least it will not get worse comparing to pre-change. > Separate out thread pool for Master <-> RegionServer communication > -- > > Key: HBASE-12071 > URL: https://issues.apache.org/jira/browse/HBASE-12071 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Labels: reliablity > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-12071.v1-branch-1.patch, > HBASE-12071.v2-master.patch, HBASE-12071.v2-master.patch, > HBASE-12071.v3-master.patch, HBASE-12071.v4-master.patch, > HBASE-12071.v5-master.patch > > > Over in HBASE-12028, there is a discussion about the case of a RegionServer > still being alive despite all its handler threads being dead. One outcome of > this is that the Master is left hanging on the RS for completion of various > operations - such as region un-assignment when a table is disabled. Does it > make sense to create a separate thread pool for communication between the > Master and the RS? This addresses not just the case of the RPC handler > threads terminating but also long-running queries or co-processor executions > holding up master operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267164#comment-14267164
]
Lars Hofhansl edited comment on HBASE-12728 at 1/7/15 1:48 AM:
---
bq. Is your meaning for BufferedTable's to be lightweight like Tables? Or does
BufferedTable become the shared object, like the current Connection is? I think
we want the former, not the latter.
I think I meant the latter. The discussion started by saying that we need some
kind of shared space to buffer edits across threads of a web container.
It's possible I misunderstood, and in any case we can say the caller is
responsible for thread safety.
bq. The flush on BufferedTable is a noop, right?
Why? I was thinking BufferedTable would be essentially what HTable is now. And
HTable would have all buffering, autoflushing, and related huh-hah removed.
Personally I'd favor an API where the have Connection and BufferedConnection.
Both would have just a getTable(name) method. Connection.getTable(...) returns
an instance of HTable, BufferedConnection.getTable(...) an instance of
BufferedTable... I do not feel strongly about this, though. :)
was (Author: lhofhansl):
bq. Is your meaning for BufferedTable's to be lightweight like Tables? Or does
BufferedTable become the shared object, like the current Connection is? I think
we want the former, not the latter.
I think I meant the latter. The discussion started by saying that we need some
kind of shared space to buffer edits across threads of a web container.
It's possible I misunderstood, and in any case we can say the caller is
responsible for thread safety.
bq. The flush on BufferedTable is a noop, right?
Why? I was thinking BufferedTable would be essentially what HTable is now. And
HTable would have all buffering, autoflushing, and related huh-hah removed.
Personally I'd favor an API where the have Connection and BufferedConnection.
Both would have just a getTable(name) method. Connection.getTable(...) returns
an instance of HTable, BufferedConnection.getTable(...) an instance of
BufferedTable.
> buffered writes substantially less useful after removal of HTablePool
> -
>
> Key: HBASE-12728
> URL: https://issues.apache.org/jira/browse/HBASE-12728
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 0.98.0
>Reporter: Aaron Beppu
>Assignee: Solomon Duskis
>Priority: Blocker
> Fix For: 1.0.0, 2.0.0, 1.1.0
>
>
> In previous versions of HBase, when use of HTablePool was encouraged, HTable
> instances were long-lived in that pool, and for that reason, if autoFlush was
> set to false, the table instance could accumulate a full buffer of writes
> before a flush was triggered. Writes from the client to the cluster could
> then be substantially larger and less frequent than without buffering.
> However, when HTablePool was deprecated, the primary justification seems to
> have been that creating HTable instances is cheap, so long as the connection
> and executor service being passed to it are pre-provided. A use pattern was
> encouraged where users should create a new HTable instance for every
> operation, using an existing connection and executor service, and then close
> the table. In this pattern, buffered writes are substantially less useful;
> writes are as small and as frequent as they would have been with
> autoflush=true, except the synchronous write is moved from the operation
> itself to the table close call which immediately follows.
> More concretely :
> ```
> // Given these two helpers ...
> private HTableInterface getAutoFlushTable(String tableName) throws
> IOException {
> // (autoflush is true by default)
> return storedConnection.getTable(tableName, executorService);
> }
> private HTableInterface getBufferedTable(String tableName) throws IOException
> {
> HTableInterface table = getAutoFlushTable(tableName);
> table.setAutoFlush(false);
> return table;
> }
> // it's my contention that these two methods would behave almost identically,
> // except the first will hit a synchronous flush during the put call,
> and the second will
> // flush during the (hidden) close call on table.
> private void writeAutoFlushed(Put somePut) throws IOException {
> try (HTableInterface table = getAutoFlushTable(tableName)) {
> table.put(somePut); // will do synchronous flush
> }
> }
> private void writeBuffered(Put somePut) throws IOException {
> try (HTableInterface table = getBufferedTable(tableName)) {
> table.put(somePut);
> } // auto-close will trigger synchronous flush
> }
> ```
> For buffered writes to actually provide a performance benefit to users, one
> of two things must happen:
> - The writeBuffer itself sh
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267164#comment-14267164
]
Lars Hofhansl commented on HBASE-12728:
---
bq. Is your meaning for BufferedTable's to be lightweight like Tables? Or does
BufferedTable become the shared object, like the current Connection is? I think
we want the former, not the latter.
I think I meant the latter. The discussion started by saying that we need some
kind of shared space to buffer edits across threads of a web container.
It's possible I misunderstood, and in any case we can say the caller is
responsible for thread safety.
bq. The flush on BufferedTable is a noop, right?
Why? I was thinking BufferedTable would be essentially what HTable is now. And
HTable would have all buffering, autoflushing, and related huh-hah removed.
Personally I'd favor an API where the have Connection and BufferedConnection.
Both would have just a getTable(name) method. Connection.getTable(...) returns
an instance of HTable, BufferedConnection.getTable(...) an instance of
BufferedTable.
> buffered writes substantially less useful after removal of HTablePool
> -
>
> Key: HBASE-12728
> URL: https://issues.apache.org/jira/browse/HBASE-12728
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 0.98.0
>Reporter: Aaron Beppu
>Assignee: Solomon Duskis
>Priority: Blocker
> Fix For: 1.0.0, 2.0.0, 1.1.0
>
>
> In previous versions of HBase, when use of HTablePool was encouraged, HTable
> instances were long-lived in that pool, and for that reason, if autoFlush was
> set to false, the table instance could accumulate a full buffer of writes
> before a flush was triggered. Writes from the client to the cluster could
> then be substantially larger and less frequent than without buffering.
> However, when HTablePool was deprecated, the primary justification seems to
> have been that creating HTable instances is cheap, so long as the connection
> and executor service being passed to it are pre-provided. A use pattern was
> encouraged where users should create a new HTable instance for every
> operation, using an existing connection and executor service, and then close
> the table. In this pattern, buffered writes are substantially less useful;
> writes are as small and as frequent as they would have been with
> autoflush=true, except the synchronous write is moved from the operation
> itself to the table close call which immediately follows.
> More concretely :
> ```
> // Given these two helpers ...
> private HTableInterface getAutoFlushTable(String tableName) throws
> IOException {
> // (autoflush is true by default)
> return storedConnection.getTable(tableName, executorService);
> }
> private HTableInterface getBufferedTable(String tableName) throws IOException
> {
> HTableInterface table = getAutoFlushTable(tableName);
> table.setAutoFlush(false);
> return table;
> }
> // it's my contention that these two methods would behave almost identically,
> // except the first will hit a synchronous flush during the put call,
> and the second will
> // flush during the (hidden) close call on table.
> private void writeAutoFlushed(Put somePut) throws IOException {
> try (HTableInterface table = getAutoFlushTable(tableName)) {
> table.put(somePut); // will do synchronous flush
> }
> }
> private void writeBuffered(Put somePut) throws IOException {
> try (HTableInterface table = getBufferedTable(tableName)) {
> table.put(somePut);
> } // auto-close will trigger synchronous flush
> }
> ```
> For buffered writes to actually provide a performance benefit to users, one
> of two things must happen:
> - The writeBuffer itself shouldn't live, flush and die with the lifecycle of
> it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
> lifespan, this could cease to be an issue. However, if the same writeBuffer
> is appended to by multiple tables, then some additional concurrency control
> will be needed around it.
> - Alternatively, there should be some pattern for having long-lived HTable
> instances. However, since HTable is not thread-safe, we'd need multiple
> instances, and a mechanism for leasing them out safely -- which sure sounds a
> lot like the old HTablePool to me.
> See discussion on mailing list here :
> http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12333) Add Integration Test Runner which is more friendly
[ https://issues.apache.org/jira/browse/HBASE-12333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manukranth Kolloju updated HBASE-12333: --- Attachment: HBASE-12333-v3.patch Upon [~daviddengcn]'s comments, I made the modifications directly in the IntegrationTestDriver instead of extending it. > Add Integration Test Runner which is more friendly > -- > > Key: HBASE-12333 > URL: https://issues.apache.org/jira/browse/HBASE-12333 > Project: HBase > Issue Type: New Feature > Components: test >Affects Versions: 2.0.0 >Reporter: Manukranth Kolloju >Assignee: Manukranth Kolloju > Fix For: 2.0.0 > > Attachments: > 0001-HBASE-12333-Add-Integration-Test-Runner-which-is-mor.patch, > HBASE-12333-v2.patch, HBASE-12333-v3.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > This Jira is intended to add a Driver class which would run a list of > Integration tests on an actual hbase cluster. And generate a machine readable > results file in JSON and put it on HDFS. The idea is to make it easy to run > driver class using a long list of appropriate command line params and wait > for the JSON file on the HDFS. This will help in plugging into external > automation and makes it easier to maintain Continuous Integration Scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8410) Basic quota support for namespaces
[
https://issues.apache.org/jira/browse/HBASE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267129#comment-14267129
]
Enis Soztutar commented on HBASE-8410:
--
Vandana mind putting the latest patch to RB as well. I think what you have
there and here does not match.
> Basic quota support for namespaces
> --
>
> Key: HBASE-8410
> URL: https://issues.apache.org/jira/browse/HBASE-8410
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Liu
>Assignee: Vandana Ayyalasomayajula
> Attachments: HBASE-8410-master.1.patch, HBASE-8410-master.4.patch,
> HBASE-8410-master.5.patch, HBASE-8410-master.6.patch,
> HBASE-8410-master.patch, HBASE-8410.docx, HBASE-8410_trunk_14.patch
>
>
> This task involves creating an observer which provides basic quota support to
> namespaces in terms of (1) number of tables and (2) number of regions. The
> quota support can be enabled by setting:
>
> hbase.coprocessor.region.classes
> org.apache.hadoop.hbase.namespace.NamespaceController
>
>
> hbase.coprocessor.master.classes
> org.apache.hadoop.hbase.namespace.NamespaceController
>
> in the hbase-site.xml.
> To add quotas to namespace, while creating namespace properties need to be
> added.
> Examples:
> 1. namespace_create 'ns1', {'hbase.namespace.quota.maxregion'=>'10'}
> 2. 1. namespace_create 'ns2', {'hbase.namespace.quota.maxtables'=>'2'},
> {'hbase.namespace.quota.maxregion'=>'5'}
> The quotas can be modified/added to namespace at any point of time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12814) Zero downtime upgrade from 94 to 98
[
https://issues.apache.org/jira/browse/HBASE-12814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
churro morales updated HBASE-12814:
---
Summary: Zero downtime upgrade from 94 to 98 (was: Zero downtime upgrade
from 94 to 98 with replication)
> Zero downtime upgrade from 94 to 98
>
>
> Key: HBASE-12814
> URL: https://issues.apache.org/jira/browse/HBASE-12814
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.94.26, 0.98.10
>Reporter: churro morales
>Assignee: churro morales
> Attachments: HBASE-12814-0.94.patch, HBASE-12814-0.98.patch
>
>
> Here at Flurry we want to upgrade our HBase cluster from 94 to 98 while not
> having any downtime and maintaining master / master replication.
> Summary:
> Replication is done via thrift RPC between clusters. It is configurable on a
> peer by peer basis and the one caveat is that a thrift server starts up on
> every node which proxies the request to the ReplicationSink.
> For the upgrade process:
> * in hbase-site.xml two new configuration parameters are added:
> ** *Required*
> *** hbase.replication.sink.enable.thrift -> true
> *** hbase.replication.thrift.server.port ->
> ** *Optional*
> *** hbase.replication.thrift.protection {default: AUTHENTICATION}
> *** hbase.replication.thrift.framed {default: false}
> *** hbase.replication.thrift.compact {default: true}
> - All regionservers can be rolling restarted (no downtime), all clusters must
> have the respective patch for this to work.
> - the hbase shell add_peer command takes an additional parameter for rpc
> protocol
> - example: {code} add_peer '1' "hbase-101:2181:/hbase", "THRIFT" {code}
> Now comes the fun part when you want to upgrade your cluster from 94 to 98
> you simply pause replication to the cluster being upgraded, do the upgrade
> and un-pause replication. Once you have a pair of clusters only replicating
> inbound and outbound with the 98 release. You can start replicating via the
> native rpc protocol by adding the peer again without the _THRIFT_ parameter
> and subsequently deleting the peer with the thrift protocol. Because
> replication is idempotent I don't see any issues as long as you wait for the
> backlog to drain after un-pausing replication.
> Special thanks to Francis Liu at Yahoo for laying the groundwork and Mr. Dave
> Latham for his invaluable knowledge and assistance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12814) Zero downtime upgrade from 94 to 98 with replication
[
https://issues.apache.org/jira/browse/HBASE-12814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
churro morales updated HBASE-12814:
---
Attachment: HBASE-12814-0.94.patch
HBASE-12814-0.98.patch
The 94 patch is large mainly due to the fact we had to recompile the thrift
files with 0.9.1. If more people like this I can create a patch with the 0.8
versions for both branches.
> Zero downtime upgrade from 94 to 98 with replication
>
>
> Key: HBASE-12814
> URL: https://issues.apache.org/jira/browse/HBASE-12814
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.94.26, 0.98.10
>Reporter: churro morales
>Assignee: churro morales
> Attachments: HBASE-12814-0.94.patch, HBASE-12814-0.98.patch
>
>
> Here at Flurry we want to upgrade our HBase cluster from 94 to 98 while not
> having any downtime and maintaining master / master replication.
> Summary:
> Replication is done via thrift RPC between clusters. It is configurable on a
> peer by peer basis and the one caveat is that a thrift server starts up on
> every node which proxies the request to the ReplicationSink.
> For the upgrade process:
> * in hbase-site.xml two new configuration parameters are added:
> ** *Required*
> *** hbase.replication.sink.enable.thrift -> true
> *** hbase.replication.thrift.server.port ->
> ** *Optional*
> *** hbase.replication.thrift.protection {default: AUTHENTICATION}
> *** hbase.replication.thrift.framed {default: false}
> *** hbase.replication.thrift.compact {default: true}
> - All regionservers can be rolling restarted (no downtime), all clusters must
> have the respective patch for this to work.
> - the hbase shell add_peer command takes an additional parameter for rpc
> protocol
> - example: {code} add_peer '1' "hbase-101:2181:/hbase", "THRIFT" {code}
> Now comes the fun part when you want to upgrade your cluster from 94 to 98
> you simply pause replication to the cluster being upgraded, do the upgrade
> and un-pause replication. Once you have a pair of clusters only replicating
> inbound and outbound with the 98 release. You can start replicating via the
> native rpc protocol by adding the peer again without the _THRIFT_ parameter
> and subsequently deleting the peer with the thrift protocol. Because
> replication is idempotent I don't see any issues as long as you wait for the
> backlog to drain after un-pausing replication.
> Special thanks to Francis Liu at Yahoo for laying the groundwork and Mr. Dave
> Latham for his invaluable knowledge and assistance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267117#comment-14267117 ] stack commented on HBASE-12684: --- bq. Did you also try out the forced heapBuffer variant? I ran the patch with heapBuffer on the end bq. The actual selection of the default type of buffer happens on system characteristics so it could still be heap buffer on your system. Ok. Yeah, my change seemed to take buffer allocation out of the thread dumps (my thinking was they were no longer slow enough to show in thread dumps). bq. But maybe their usage is different. If you can't see anything in flight recorder to help w/ throughput (I didn't see anything obvious... the round trip w/ netty in the mix must just be taking a little longer), I could get traces for old client and new async client for a compare. > Add new AsyncRpcClient > -- > > Key: HBASE-12684 > URL: https://issues.apache.org/jira/browse/HBASE-12684 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, > HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, > HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, > HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, > HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v19.1.patch, > HBASE-12684-v19.patch, HBASE-12684-v19.patch, HBASE-12684-v2.patch, > HBASE-12684-v20-heapBuffer.patch, HBASE-12684-v20.patch, > HBASE-12684-v21-heapBuffer.1.patch, HBASE-12684-v21-heapBuffer.patch, > HBASE-12684-v21.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, > HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, > HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch, > myrecording.jfr, requests.png > > > With the changes in HBASE-12597 it is possible to add new RpcClients. This > issue is about adding a new Async RpcClient which would enable HBase to do > non blocking protobuf service communication. > Besides delivering a new AsyncRpcClient I would also like to ask the question > what it would take to replace the current RpcClient? This would enable to > simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12814) Zero downtime upgrade from 94 to 98 with replication
[
https://issues.apache.org/jira/browse/HBASE-12814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267112#comment-14267112
]
stack commented on HBASE-12814:
---
Sweet! Talk it up on dev list! Others interested.
> Zero downtime upgrade from 94 to 98 with replication
>
>
> Key: HBASE-12814
> URL: https://issues.apache.org/jira/browse/HBASE-12814
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.94.26, 0.98.10
>Reporter: churro morales
>Assignee: churro morales
>
> Here at Flurry we want to upgrade our HBase cluster from 94 to 98 while not
> having any downtime and maintaining master / master replication.
> Summary:
> Replication is done via thrift RPC between clusters. It is configurable on a
> peer by peer basis and the one caveat is that a thrift server starts up on
> every node which proxies the request to the ReplicationSink.
> For the upgrade process:
> * in hbase-site.xml two new configuration parameters are added:
> ** *Required*
> *** hbase.replication.sink.enable.thrift -> true
> *** hbase.replication.thrift.server.port ->
> ** *Optional*
> *** hbase.replication.thrift.protection {default: AUTHENTICATION}
> *** hbase.replication.thrift.framed {default: false}
> *** hbase.replication.thrift.compact {default: true}
> - All regionservers can be rolling restarted (no downtime), all clusters must
> have the respective patch for this to work.
> - the hbase shell add_peer command takes an additional parameter for rpc
> protocol
> - example: {code} add_peer '1' "hbase-101:2181:/hbase", "THRIFT" {code}
> Now comes the fun part when you want to upgrade your cluster from 94 to 98
> you simply pause replication to the cluster being upgraded, do the upgrade
> and un-pause replication. Once you have a pair of clusters only replicating
> inbound and outbound with the 98 release. You can start replicating via the
> native rpc protocol by adding the peer again without the _THRIFT_ parameter
> and subsequently deleting the peer with the thrift protocol. Because
> replication is idempotent I don't see any issues as long as you wait for the
> backlog to drain after un-pausing replication.
> Special thanks to Francis Liu at Yahoo for laying the groundwork and Mr. Dave
> Latham for his invaluable knowledge and assistance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12814) Zero downtime upgrade from 94 to 98 with replication
churro morales created HBASE-12814:
--
Summary: Zero downtime upgrade from 94 to 98 with replication
Key: HBASE-12814
URL: https://issues.apache.org/jira/browse/HBASE-12814
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.26, 0.98.10
Reporter: churro morales
Assignee: churro morales
Here at Flurry we want to upgrade our HBase cluster from 94 to 98 while not
having any downtime and maintaining master / master replication.
Summary:
Replication is done via thrift RPC between clusters. It is configurable on a
peer by peer basis and the one caveat is that a thrift server starts up on
every node which proxies the request to the ReplicationSink.
For the upgrade process:
* in hbase-site.xml two new configuration parameters are added:
** *Required*
*** hbase.replication.sink.enable.thrift -> true
*** hbase.replication.thrift.server.port ->
** *Optional*
*** hbase.replication.thrift.protection {default: AUTHENTICATION}
*** hbase.replication.thrift.framed {default: false}
*** hbase.replication.thrift.compact {default: true}
- All regionservers can be rolling restarted (no downtime), all clusters must
have the respective patch for this to work.
- the hbase shell add_peer command takes an additional parameter for rpc
protocol
- example: {code} add_peer '1' "hbase-101:2181:/hbase", "THRIFT" {code}
Now comes the fun part when you want to upgrade your cluster from 94 to 98 you
simply pause replication to the cluster being upgraded, do the upgrade and
un-pause replication. Once you have a pair of clusters only replicating
inbound and outbound with the 98 release. You can start replicating via the
native rpc protocol by adding the peer again without the _THRIFT_ parameter and
subsequently deleting the peer with the thrift protocol. Because replication
is idempotent I don't see any issues as long as you wait for the backlog to
drain after un-pausing replication.
Special thanks to Francis Liu at Yahoo for laying the groundwork and Mr. Dave
Latham for his invaluable knowledge and assistance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267096#comment-14267096
]
stack commented on HBASE-12728:
---
Feedback.
On AsyncPutter, will we ever want to buffer Increments or Appends or Deletes
(Increments we might want to aggregate client-side but that is something
different from AsyncPutter i suppose); if so, AsyncMutator instead? (An uglier
name!).
Async is probably good to have in the name since it conveys detachment between
invocation and when the op actually happens but on other hand, we could just
call it Buffer or recast the PutBuffer you have above and do away with an
extra moving part?
Ditto, do we even need to expose BufferedTable? Just add to Connection a
#getTable(TableName, PutBuffer) that takes a PutBuffer -- the implementation
behind the scenes would use the passed PutBuffer. I suppose this would mean
you'd have to ask Connection for a PutBuffer instance
(Connection#getPutBuffer(maxsize, executor) or maybe this is just where your
BufferedConnection comes in; it adds these methods.
Having the PutBuffer detached from Table is clean but what is to prevent me
adding a Put for TableA to TableB's PutBuffer? There'd be no protection
against this? (That'd probably be fine)
The flush on BufferedTable is a noop, right?
BufferedTable shouldn't have a close if its not going to do anything?
The ExceptionListener seems good but should it supply more info than just the
exception? For example, where we are in the processing of the buffer -- what
edits have succeeded and which have failed (or would that just be TMI?)
Good stuff [~carterpage]
> buffered writes substantially less useful after removal of HTablePool
> -
>
> Key: HBASE-12728
> URL: https://issues.apache.org/jira/browse/HBASE-12728
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 0.98.0
>Reporter: Aaron Beppu
>Assignee: Solomon Duskis
>Priority: Blocker
> Fix For: 1.0.0, 2.0.0, 1.1.0
>
>
> In previous versions of HBase, when use of HTablePool was encouraged, HTable
> instances were long-lived in that pool, and for that reason, if autoFlush was
> set to false, the table instance could accumulate a full buffer of writes
> before a flush was triggered. Writes from the client to the cluster could
> then be substantially larger and less frequent than without buffering.
> However, when HTablePool was deprecated, the primary justification seems to
> have been that creating HTable instances is cheap, so long as the connection
> and executor service being passed to it are pre-provided. A use pattern was
> encouraged where users should create a new HTable instance for every
> operation, using an existing connection and executor service, and then close
> the table. In this pattern, buffered writes are substantially less useful;
> writes are as small and as frequent as they would have been with
> autoflush=true, except the synchronous write is moved from the operation
> itself to the table close call which immediately follows.
> More concretely :
> ```
> // Given these two helpers ...
> private HTableInterface getAutoFlushTable(String tableName) throws
> IOException {
> // (autoflush is true by default)
> return storedConnection.getTable(tableName, executorService);
> }
> private HTableInterface getBufferedTable(String tableName) throws IOException
> {
> HTableInterface table = getAutoFlushTable(tableName);
> table.setAutoFlush(false);
> return table;
> }
> // it's my contention that these two methods would behave almost identically,
> // except the first will hit a synchronous flush during the put call,
> and the second will
> // flush during the (hidden) close call on table.
> private void writeAutoFlushed(Put somePut) throws IOException {
> try (HTableInterface table = getAutoFlushTable(tableName)) {
> table.put(somePut); // will do synchronous flush
> }
> }
> private void writeBuffered(Put somePut) throws IOException {
> try (HTableInterface table = getBufferedTable(tableName)) {
> table.put(somePut);
> } // auto-close will trigger synchronous flush
> }
> ```
> For buffered writes to actually provide a performance benefit to users, one
> of two things must happen:
> - The writeBuffer itself shouldn't live, flush and die with the lifecycle of
> it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
> lifespan, this could cease to be an issue. However, if the same writeBuffer
> is appended to by multiple tables, then some additional concurrency control
> will be needed around it.
> - Alternatively, there should be some pattern for having long-lived HTable
> instances. However, since HTable is not thread-safe, we'd need multiple
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267060#comment-14267060
]
Nick Dimiduk commented on HBASE-12728:
--
I agree with [~enis] that hiding AsyncPutter and PutBuffer behind
BufferedConnection makes better sense. Also that BufferedTable should be an
interface, and that all these API's should accept Mutation as the argument, not
Put.
[~lhofhansl]'s question is interesting. Is your meaning for BufferedTable's to
be lightweight like Tables? Or does BufferedTable become the shared object,
like the current Connection is? I think we want the former, not the latter.
AsyncPutter looks a lot like HTable's internal AsyncProcess. How are these
classes related/shared? Re-implementing all this work seems a waste, though
AsyncProcess jumps through some hoops to maintain some semblance of backward
compatibility with 0.94 semantics.
Calling BufferedTable.flush() forces a flush of the contents of the PutBuffer,
does it block the calling thread, or is it asynchronous?
In the event of a BufferedTable that does not share it's PutBuffer with any
other instances, can it just defer back to the current implementation in HTable?
Isn't PutBuffer just an implementation detail for communication between the
BufferedTableImpls and AsyncPutter? Shouldn't it be a BlockingQueue or some
such, that can be drained in the background and allow some rate-limiting?
> buffered writes substantially less useful after removal of HTablePool
> -
>
> Key: HBASE-12728
> URL: https://issues.apache.org/jira/browse/HBASE-12728
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 0.98.0
>Reporter: Aaron Beppu
>Assignee: Solomon Duskis
>Priority: Blocker
> Fix For: 1.0.0, 2.0.0, 1.1.0
>
>
> In previous versions of HBase, when use of HTablePool was encouraged, HTable
> instances were long-lived in that pool, and for that reason, if autoFlush was
> set to false, the table instance could accumulate a full buffer of writes
> before a flush was triggered. Writes from the client to the cluster could
> then be substantially larger and less frequent than without buffering.
> However, when HTablePool was deprecated, the primary justification seems to
> have been that creating HTable instances is cheap, so long as the connection
> and executor service being passed to it are pre-provided. A use pattern was
> encouraged where users should create a new HTable instance for every
> operation, using an existing connection and executor service, and then close
> the table. In this pattern, buffered writes are substantially less useful;
> writes are as small and as frequent as they would have been with
> autoflush=true, except the synchronous write is moved from the operation
> itself to the table close call which immediately follows.
> More concretely :
> ```
> // Given these two helpers ...
> private HTableInterface getAutoFlushTable(String tableName) throws
> IOException {
> // (autoflush is true by default)
> return storedConnection.getTable(tableName, executorService);
> }
> private HTableInterface getBufferedTable(String tableName) throws IOException
> {
> HTableInterface table = getAutoFlushTable(tableName);
> table.setAutoFlush(false);
> return table;
> }
> // it's my contention that these two methods would behave almost identically,
> // except the first will hit a synchronous flush during the put call,
> and the second will
> // flush during the (hidden) close call on table.
> private void writeAutoFlushed(Put somePut) throws IOException {
> try (HTableInterface table = getAutoFlushTable(tableName)) {
> table.put(somePut); // will do synchronous flush
> }
> }
> private void writeBuffered(Put somePut) throws IOException {
> try (HTableInterface table = getBufferedTable(tableName)) {
> table.put(somePut);
> } // auto-close will trigger synchronous flush
> }
> ```
> For buffered writes to actually provide a performance benefit to users, one
> of two things must happen:
> - The writeBuffer itself shouldn't live, flush and die with the lifecycle of
> it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
> lifespan, this could cease to be an issue. However, if the same writeBuffer
> is appended to by multiple tables, then some additional concurrency control
> will be needed around it.
> - Alternatively, there should be some pattern for having long-lived HTable
> instances. However, since HTable is not thread-safe, we'd need multiple
> instances, and a mechanism for leasing them out safely -- which sure sounds a
> lot like the old HTablePool to me.
> See discussion on mailing list here :
> http://mail-archives.apache.org/mod_mbox/hbase-user/
[jira] [Created] (HBASE-12813) Reporting region in transition shouldn't loop forever
Elliott Clark created HBASE-12813: - Summary: Reporting region in transition shouldn't loop forever Key: HBASE-12813 URL: https://issues.apache.org/jira/browse/HBASE-12813 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark We had an issue where a region server wasn't able to send the report region in transition request. Well after failing it just retries forever. At some point it would have been better to just abort the region server if it can't talk to master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12728:
--
Priority: Blocker (was: Major)
> buffered writes substantially less useful after removal of HTablePool
> -
>
> Key: HBASE-12728
> URL: https://issues.apache.org/jira/browse/HBASE-12728
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 0.98.0
>Reporter: Aaron Beppu
>Assignee: Solomon Duskis
>Priority: Blocker
> Fix For: 1.0.0, 2.0.0, 1.1.0
>
>
> In previous versions of HBase, when use of HTablePool was encouraged, HTable
> instances were long-lived in that pool, and for that reason, if autoFlush was
> set to false, the table instance could accumulate a full buffer of writes
> before a flush was triggered. Writes from the client to the cluster could
> then be substantially larger and less frequent than without buffering.
> However, when HTablePool was deprecated, the primary justification seems to
> have been that creating HTable instances is cheap, so long as the connection
> and executor service being passed to it are pre-provided. A use pattern was
> encouraged where users should create a new HTable instance for every
> operation, using an existing connection and executor service, and then close
> the table. In this pattern, buffered writes are substantially less useful;
> writes are as small and as frequent as they would have been with
> autoflush=true, except the synchronous write is moved from the operation
> itself to the table close call which immediately follows.
> More concretely :
> ```
> // Given these two helpers ...
> private HTableInterface getAutoFlushTable(String tableName) throws
> IOException {
> // (autoflush is true by default)
> return storedConnection.getTable(tableName, executorService);
> }
> private HTableInterface getBufferedTable(String tableName) throws IOException
> {
> HTableInterface table = getAutoFlushTable(tableName);
> table.setAutoFlush(false);
> return table;
> }
> // it's my contention that these two methods would behave almost identically,
> // except the first will hit a synchronous flush during the put call,
> and the second will
> // flush during the (hidden) close call on table.
> private void writeAutoFlushed(Put somePut) throws IOException {
> try (HTableInterface table = getAutoFlushTable(tableName)) {
> table.put(somePut); // will do synchronous flush
> }
> }
> private void writeBuffered(Put somePut) throws IOException {
> try (HTableInterface table = getBufferedTable(tableName)) {
> table.put(somePut);
> } // auto-close will trigger synchronous flush
> }
> ```
> For buffered writes to actually provide a performance benefit to users, one
> of two things must happen:
> - The writeBuffer itself shouldn't live, flush and die with the lifecycle of
> it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
> lifespan, this could cease to be an issue. However, if the same writeBuffer
> is appended to by multiple tables, then some additional concurrency control
> will be needed around it.
> - Alternatively, there should be some pattern for having long-lived HTable
> instances. However, since HTable is not thread-safe, we'd need multiple
> instances, and a mechanism for leasing them out safely -- which sure sounds a
> lot like the old HTablePool to me.
> See discussion on mailing list here :
> http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267002#comment-14267002 ] Jurriaan Mous commented on HBASE-12684: --- bq. Slower. Almost 8k requests a second as opposed to almost 10k with previous patch and almost 12k for old client. That is a big difference... Did you also try out the forced heapBuffer variant? bq. (not since I moved from directBuffer to buffer (though you are saying this was doing direct buffers...) The actual selection of the default type of buffer happens on system characteristics so it could still be heap buffer on your system. bq. Shows the skiplist in AsyncRpcChannel as hot. Not sure fixing this would impinge that much on throughput though – perhaps given it shows so prominently. RpcClientImpl its Connection class also uses a ConcurrentSkipListMap for the Call indexing so those should not be different between both clients. But maybe their usage is different. bq. Here is a flight recording of the latest patch. 60seconds sample. Open it w/ your jdk bin/jmc command. I will look into it tomorrow. Again thanks for your work! > Add new AsyncRpcClient > -- > > Key: HBASE-12684 > URL: https://issues.apache.org/jira/browse/HBASE-12684 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, > HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, > HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, > HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, > HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v19.1.patch, > HBASE-12684-v19.patch, HBASE-12684-v19.patch, HBASE-12684-v2.patch, > HBASE-12684-v20-heapBuffer.patch, HBASE-12684-v20.patch, > HBASE-12684-v21-heapBuffer.1.patch, HBASE-12684-v21-heapBuffer.patch, > HBASE-12684-v21.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, > HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, > HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch, > myrecording.jfr, requests.png > > > With the changes in HBASE-12597 it is possible to add new RpcClients. This > issue is about adding a new Async RpcClient which would enable HBase to do > non blocking protobuf service communication. > Besides delivering a new AsyncRpcClient I would also like to ask the question > what it would take to replace the current RpcClient? This would enable to > simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266961#comment-14266961
]
Enis Soztutar commented on HBASE-12728:
---
I think API wise, what Carter proposes above makes sense with a couple of
comments.
- I think {{BufferedTable}} should be an interface still with {{flush()}} or
{{flushCommits()}} method.
- {{ExceptionListener}} should also get the original {{Put}} so that it can
learn about which operation has failed (not just the exception).
- Do we want AsyncPutter, etc to be client-public API at all? I like the
{{BufferedConnection}} route where we can have getBufferedTable() method to
construct this and not worry about any class. However do we want to address
doing puts() with flush and doing async flush by the same client? Does the
client maintain two Connection objects?
- Should the interfaces be {{Put}} based or {{Mutation}} based? We can make it
generic in case of we add Delete's later on, but may not be worth the extra
cost.
bq. BufferedTable#close does not flush since we need to support batching across
multiple threads. AsyncPutter#close does flush.
Seems a bit unintuitive. If we do the BufferedConnection route, the only way to
flush everything is to
bq. Do we need a timeout-based flush?
It makes sense, but this can be added later I say.
> buffered writes substantially less useful after removal of HTablePool
> -
>
> Key: HBASE-12728
> URL: https://issues.apache.org/jira/browse/HBASE-12728
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 0.98.0
>Reporter: Aaron Beppu
>Assignee: Solomon Duskis
> Fix For: 1.0.0, 2.0.0, 1.1.0
>
>
> In previous versions of HBase, when use of HTablePool was encouraged, HTable
> instances were long-lived in that pool, and for that reason, if autoFlush was
> set to false, the table instance could accumulate a full buffer of writes
> before a flush was triggered. Writes from the client to the cluster could
> then be substantially larger and less frequent than without buffering.
> However, when HTablePool was deprecated, the primary justification seems to
> have been that creating HTable instances is cheap, so long as the connection
> and executor service being passed to it are pre-provided. A use pattern was
> encouraged where users should create a new HTable instance for every
> operation, using an existing connection and executor service, and then close
> the table. In this pattern, buffered writes are substantially less useful;
> writes are as small and as frequent as they would have been with
> autoflush=true, except the synchronous write is moved from the operation
> itself to the table close call which immediately follows.
> More concretely :
> ```
> // Given these two helpers ...
> private HTableInterface getAutoFlushTable(String tableName) throws
> IOException {
> // (autoflush is true by default)
> return storedConnection.getTable(tableName, executorService);
> }
> private HTableInterface getBufferedTable(String tableName) throws IOException
> {
> HTableInterface table = getAutoFlushTable(tableName);
> table.setAutoFlush(false);
> return table;
> }
> // it's my contention that these two methods would behave almost identically,
> // except the first will hit a synchronous flush during the put call,
> and the second will
> // flush during the (hidden) close call on table.
> private void writeAutoFlushed(Put somePut) throws IOException {
> try (HTableInterface table = getAutoFlushTable(tableName)) {
> table.put(somePut); // will do synchronous flush
> }
> }
> private void writeBuffered(Put somePut) throws IOException {
> try (HTableInterface table = getBufferedTable(tableName)) {
> table.put(somePut);
> } // auto-close will trigger synchronous flush
> }
> ```
> For buffered writes to actually provide a performance benefit to users, one
> of two things must happen:
> - The writeBuffer itself shouldn't live, flush and die with the lifecycle of
> it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
> lifespan, this could cease to be an issue. However, if the same writeBuffer
> is appended to by multiple tables, then some additional concurrency control
> will be needed around it.
> - Alternatively, there should be some pattern for having long-lived HTable
> instances. However, since HTable is not thread-safe, we'd need multiple
> instances, and a mechanism for leasing them out safely -- which sure sounds a
> lot like the old HTablePool to me.
> See discussion on mailing list here :
> http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This mess
[jira] [Commented] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266917#comment-14266917 ] Hudson commented on HBASE-10528: SUCCESS: Integrated in HBase-1.1 #64 (See [https://builds.apache.org/job/HBase-1.1/64/]) HBASE-10528 Addendum fixes TestRegionStates compilation (tedyu: rev c951238304d91b4b660babba58e4255b37560d2c) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRegionStates.java > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 > > Attachments: 10528-1.0.addendum, HBASE-10528-0.94.patch, > HBASE-10528-0.98.patch, HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8410) Basic quota support for namespaces
[
https://issues.apache.org/jira/browse/HBASE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266903#comment-14266903
]
Hadoop QA commented on HBASE-8410:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690406/HBASE-8410-master.6.patch
against master branch at commit 45fc2bdd863ff238491f4941b3da04a38731d7e1.
ATTACHMENT ID: 12690406
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
2087 checkstyle errors (more than the master's current 2084 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12335//console
This message is automatically generated.
> Basic quota support for namespaces
> --
>
> Key: HBASE-8410
> URL: https://issues.apache.org/jira/browse/HBASE-8410
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Liu
>Assignee: Vandana Ayyalasomayajula
> Attachments: HBASE-8410-master.1.patch, HBASE-8410-master.4.patch,
> HBASE-8410-master.5.patch, HBASE-8410-master.6.patch,
> HBASE-8410-master.patch, HBASE-8410.docx, HBASE-8410_trunk_14.patch
>
>
> This task involves creating an observer which provides basic quota support to
> namespaces in terms of (1) number of tables and (2) number of regions. The
> quota support can be enabled by setting:
>
> hbase.coprocessor.region.classes
> org.apache.hadoop.hbase.namespace.NamespaceController
>
>
> hbase.coprocessor.master.classes
> org.apache.hadoop.hbase.namespace.NamespaceController
>
> in the hbase-site.xml.
> To add quotas to namespace, while creating namespace properties need to be
> added.
> Examples:
> 1. namespace_create 'ns1', {'hbase.namespace.quota.maxregion'=>'10'}
> 2. 1. namespace_create 'ns2', {'hbase.namespace.quota.maxtables'=>'2'},
> {'hbase.namespace.quota.maxregion'=>'5'}
> The quotas can be modified/added to namespace at any point of time.
--
This message was sent by Atlas
[jira] [Commented] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266895#comment-14266895 ] Hudson commented on HBASE-10528: SUCCESS: Integrated in HBase-TRUNK #5999 (See [https://builds.apache.org/job/HBase-TRUNK/5999/]) HBASE-10528 DefaultBalancer selects plans to move regions onto draining nodes (Churro Morales) (tedyu: rev 45fc2bdd863ff238491f4941b3da04a38731d7e1) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRegionStates.java > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 > > Attachments: 10528-1.0.addendum, HBASE-10528-0.94.patch, > HBASE-10528-0.98.patch, HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12556) Create a golden file for testing client API source/binary compatibility
[ https://issues.apache.org/jira/browse/HBASE-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266893#comment-14266893 ] Dima Spivak commented on HBASE-12556: - Hey [~enis], I think passing the golden file generated the patch in this JIRA (after stripping out the method names and types) would be ideal for using the Java API Compliance Checker. Since I think the use of such a checker, and not just analyzing the list in a unit test (for the reasons detailed by [~busbey] above) is the way to go, would you be open to committing the golden file generation and then leaving the analysis to done by HBASE-12808? I've separately reached out to the developer of Java ACC about adding annotation support, but we could get this online in no time if we had a way to generate a file with the list of classes to analyze (since that's already a feature of the tool). > Create a golden file for testing client API source/binary compatibility > --- > > Key: HBASE-12556 > URL: https://issues.apache.org/jira/browse/HBASE-12556 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.1.0 > > Attachments: hbase-12556-wip.patch > > > [~lhofhansl] had a suggestion (in some other jira I forgot) for doing a > golden file for the HBase API so that we can compare between releases to > ensure that we are keeping source and binary compatibility as defined in this > document : > https://docs.google.com/document/d/1p5pP7v2OuzSSaomK2S2v7sfKky1Hex6OqwsJO0sZTUY/edit > > I think we can generate a file, commit it to the repo, and create a unit test > to check whether any API's are broken. Adding new InterfaceAudience.Public > interfaces has to modify this file so that it becomes an explicit decision. > The downside is that we have to modify the file every time we add a new API, > but it should be fine since it will force us to think more before committing > to supporting new interfaces within the same major versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12446) [list | abort] Compactions
[ https://issues.apache.org/jira/browse/HBASE-12446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266875#comment-14266875 ] Manukranth Kolloju commented on HBASE-12446: Thats great. We can merge both the patches and expose an Admin endpoint to provide shell commands. > [list | abort] Compactions > -- > > Key: HBASE-12446 > URL: https://issues.apache.org/jira/browse/HBASE-12446 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: Manukranth Kolloju > Fix For: 1.1.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > In some cases, we would need to quickly reduce load on a server without > killing it. Compactions is one of the critical processes which takes up a lot > of CPU and disk IOPS. We should have a way to list compactions given the > regionserver and abort compactions given regionserver and compaction id. And > additionally abort all compactions. > Pardon me if there was already a similar Jira, I'd be happy to merge this > there. > The current code handles interrupts. We should be able to interrupt the > thread that is performing the compaction and abort it from either the UI or > from the command line. This Jira is targeted to expose an admin function to > perform such a task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9000) Linear reseek in Memstore
[ https://issues.apache.org/jira/browse/HBASE-9000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-9000: -- Status: Open (was: Patch Available) > Linear reseek in Memstore > - > > Key: HBASE-9000 > URL: https://issues.apache.org/jira/browse/HBASE-9000 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.89-fb >Reporter: Shane Hogan >Priority: Minor > Fix For: 0.89-fb > > Attachments: hbase-9000-benchmark-program.patch, > hbase-9000-port-fb.patch, hbase-9000.patch > > > This is to address the linear reseek in MemStoreScanner. Currently reseek > iterates over the kvset and the snapshot linearly by just calling next > repeatedly. The new solution is to do this linear seek up to a configurable > maximum amount of times then if the seek is not yet complete fall back to > logarithmic seek. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12446) [list | abort] Compactions
[ https://issues.apache.org/jira/browse/HBASE-12446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266862#comment-14266862 ] Esteban Gutierrez commented on HBASE-12446: --- Nice work [~manukranthk]. I think we went into similar direction, will do some clean up of my patch and lets see if we can merge both changes. I really like the idea of the compactionIdGenerator to keep track of individual compactions and it makes easier to abort (or pause) specific compactions in my current implementation I already have the client code and the ACL code and also the extensions for pbuf. > [list | abort] Compactions > -- > > Key: HBASE-12446 > URL: https://issues.apache.org/jira/browse/HBASE-12446 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: Manukranth Kolloju > Fix For: 1.1.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > In some cases, we would need to quickly reduce load on a server without > killing it. Compactions is one of the critical processes which takes up a lot > of CPU and disk IOPS. We should have a way to list compactions given the > regionserver and abort compactions given regionserver and compaction id. And > additionally abort all compactions. > Pardon me if there was already a similar Jira, I'd be happy to merge this > there. > The current code handles interrupts. We should be able to interrupt the > thread that is performing the compaction and abort it from either the UI or > from the command line. This Jira is targeted to expose an admin function to > perform such a task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12446) [list | abort] Compactions
[ https://issues.apache.org/jira/browse/HBASE-12446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266835#comment-14266835 ] Manukranth Kolloju commented on HBASE-12446: This is the diff for the task : https://reviews.facebook.net/D30993 > [list | abort] Compactions > -- > > Key: HBASE-12446 > URL: https://issues.apache.org/jira/browse/HBASE-12446 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: Manukranth Kolloju > Fix For: 1.1.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > In some cases, we would need to quickly reduce load on a server without > killing it. Compactions is one of the critical processes which takes up a lot > of CPU and disk IOPS. We should have a way to list compactions given the > regionserver and abort compactions given regionserver and compaction id. And > additionally abort all compactions. > Pardon me if there was already a similar Jira, I'd be happy to merge this > there. > The current code handles interrupts. We should be able to interrupt the > thread that is performing the compaction and abort it from either the UI or > from the command line. This Jira is targeted to expose an admin function to > perform such a task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12574) Update replication metrics to not do so many map look ups.
[ https://issues.apache.org/jira/browse/HBASE-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266826#comment-14266826 ] Hudson commented on HBASE-12574: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #743 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/743/]) HBASE-12658 Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98 (apurtell: rev 7890d27eae9332557a6688b7cda56ac7c5f22fda) * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * hbase-hadoop2-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetricsReplicationSourceFactoryImpl.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceFactory.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceFactoryImpl.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceFactoryImpl.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSource.java * hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.replication.regionserver.MetricsReplicationSourceFactory * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetricsReplicationSourceImpl.java * hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.replication.regionserver.MetricsReplicationSourceFactory > Update replication metrics to not do so many map look ups. > -- > > Key: HBASE-12574 > URL: https://issues.apache.org/jira/browse/HBASE-12574 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.98.6.1 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.0.0, 2.0.0, 1.1.0 > > Attachments: HBASE-12574.patch > > > Replication is the last metrics area that still does a significant amount of > hash map lookups. Lets re-factor that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12658) Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266825#comment-14266825 ] Hudson commented on HBASE-12658: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #743 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/743/]) HBASE-12658 Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98 (apurtell: rev 7890d27eae9332557a6688b7cda56ac7c5f22fda) * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * hbase-hadoop2-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetricsReplicationSourceFactoryImpl.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceFactory.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceFactoryImpl.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceFactoryImpl.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSource.java * hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.replication.regionserver.MetricsReplicationSourceFactory * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetricsReplicationSourceImpl.java * hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.replication.regionserver.MetricsReplicationSourceFactory > Backport HBASE-12574 (Update replication metrics to not do so many map look > ups) to 0.98 > > > Key: HBASE-12658 > URL: https://issues.apache.org/jira/browse/HBASE-12658 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.10 > > Attachments: HBASE-12658-0.98.patch, HBASE-12658-0.98.patch > > > Backport HBASE-12574 (Update replication metrics to not do so many map look > ups) to 0.98. Nice enhancement. It's a refactor so need to check for compat > issues there, and we need to add Hadoop 1 compat module support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10528: --- Attachment: 10528-1.0.addendum Addendum for branch-1 > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 > > Attachments: 10528-1.0.addendum, HBASE-10528-0.94.patch, > HBASE-10528-0.98.patch, HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266806#comment-14266806 ] Hudson commented on HBASE-10528: FAILURE: Integrated in HBase-1.1 #63 (See [https://builds.apache.org/job/HBase-1.1/63/]) HBASE-10528 DefaultBalancer selects plans to move regions onto draining nodes (Churro Morales) (tedyu: rev c5b0a1b8a0a7db7c48473b8f3867146abfe076c1) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRegionStates.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 > > Attachments: HBASE-10528-0.94.patch, HBASE-10528-0.98.patch, > HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10528: --- Fix Version/s: 0.94.27 0.98.10 > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 > > Attachments: HBASE-10528-0.94.patch, HBASE-10528-0.98.patch, > HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reopened HBASE-10528: Reopening for 0.98 and 0.94 commit evaluation as requested. > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 > > Attachments: HBASE-10528-0.94.patch, HBASE-10528-0.98.patch, > HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266797#comment-14266797 ] Andrew Purtell commented on HBASE-10528: +1 for 0.98. The change is a bug fix. Please try to check the stability of the new tests before commit. If the change causes test failures in the Jenkins 0.98 builds we will have to revert it. > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-10528-0.94.patch, HBASE-10528-0.98.patch, > HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12574) Update replication metrics to not do so many map look ups.
[ https://issues.apache.org/jira/browse/HBASE-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266792#comment-14266792 ] Hudson commented on HBASE-12574: SUCCESS: Integrated in HBase-0.98 #778 (See [https://builds.apache.org/job/HBase-0.98/778/]) HBASE-12658 Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98 (apurtell: rev 7890d27eae9332557a6688b7cda56ac7c5f22fda) * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSource.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceFactoryImpl.java * hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.replication.regionserver.MetricsReplicationSourceFactory * hbase-hadoop2-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetricsReplicationSourceFactoryImpl.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceFactoryImpl.java * hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.replication.regionserver.MetricsReplicationSourceFactory * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceFactory.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java * hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetricsReplicationSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java > Update replication metrics to not do so many map look ups. > -- > > Key: HBASE-12574 > URL: https://issues.apache.org/jira/browse/HBASE-12574 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.98.6.1 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.0.0, 2.0.0, 1.1.0 > > Attachments: HBASE-12574.patch > > > Replication is the last metrics area that still does a significant amount of > hash map lookups. Lets re-factor that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12658) Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266791#comment-14266791 ] Hudson commented on HBASE-12658: SUCCESS: Integrated in HBase-0.98 #778 (See [https://builds.apache.org/job/HBase-0.98/778/]) HBASE-12658 Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98 (apurtell: rev 7890d27eae9332557a6688b7cda56ac7c5f22fda) * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSource.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceFactoryImpl.java * hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.replication.regionserver.MetricsReplicationSourceFactory * hbase-hadoop2-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetricsReplicationSourceFactoryImpl.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceFactoryImpl.java * hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.replication.regionserver.MetricsReplicationSourceFactory * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceFactory.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java * hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetricsReplicationSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java > Backport HBASE-12574 (Update replication metrics to not do so many map look > ups) to 0.98 > > > Key: HBASE-12658 > URL: https://issues.apache.org/jira/browse/HBASE-12658 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.10 > > Attachments: HBASE-12658-0.98.patch, HBASE-12658-0.98.patch > > > Backport HBASE-12574 (Update replication metrics to not do so many map look > ups) to 0.98. Nice enhancement. It's a refactor so need to check for compat > issues there, and we need to add Hadoop 1 compat module support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266790#comment-14266790 ] churro morales commented on HBASE-10528: Thanks Ted for getting this committed. [~lhofhansl] [~apurtell] any chance of getting the other patches included in this ticket in their respective branches? Thanks a lot! > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-10528-0.94.patch, HBASE-10528-0.98.patch, > HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12268) Add support for Scan.setRowPrefixFilter to shell
[ https://issues.apache.org/jira/browse/HBASE-12268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266789#comment-14266789 ] Hudson commented on HBASE-12268: SUCCESS: Integrated in HBase-TRUNK #5998 (See [https://builds.apache.org/job/HBase-TRUNK/5998/]) HBASE-12268 Add support for Scan.setRowPrefixFilter to shell (Niels Basjes) (tedyu: rev f57a9f5bd509a7e532723d0b0e4c97032a3d5482) * hbase-shell/src/main/ruby/shell/commands/set_visibility.rb * hbase-shell/src/main/ruby/shell/commands/scan.rb * hbase-shell/src/main/ruby/hbase/table.rb * hbase-shell/src/test/ruby/hbase/table_test.rb * hbase-shell/src/main/ruby/hbase.rb > Add support for Scan.setRowPrefixFilter to shell > > > Key: HBASE-12268 > URL: https://issues.apache.org/jira/browse/HBASE-12268 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Niels Basjes >Assignee: Niels Basjes > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-12268-2014-10-15-v1.patch, > HBASE-12268-2014-12-05-v2.patch > > > I think having the feature introduced in HBASE-11990 in the hbase shell would > be very useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12339) WAL performance evaluation tool doesn't roll logs
[ https://issues.apache.org/jira/browse/HBASE-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266750#comment-14266750 ] Hudson commented on HBASE-12339: SUCCESS: Integrated in HBase-1.0 #638 (See [https://builds.apache.org/job/HBase-1.0/638/]) HBASE-12339 WALPerformanceEvaluation should have a log roller. (enis: rev ca8846eeef1cc980194924038f831817eca67a0e) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/WALPerformanceEvaluation.java * hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java > WAL performance evaluation tool doesn't roll logs > - > > Key: HBASE-12339 > URL: https://issues.apache.org/jira/browse/HBASE-12339 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.99.0 >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.0.0, 2.0.0, 1.1.0 > > Attachments: HBASE-12339.1.patch.txt > > > the perf eval tool for the wal never sets up a log roller and instead used to > just call the rollWriter method directly. > Eventually it changed to call requestLogRoll instead of attempting to do the > roll itself. requestLogRoll is the same method used internal to the wal code > and it relies on there being a LogRoller to actually have rolls happen. (the > method just notifies all of the listeners for hte wal that one of them should > call the roll method.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-8410) Basic quota support for namespaces
[
https://issues.apache.org/jira/browse/HBASE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vandana Ayyalasomayajula updated HBASE-8410:
Attachment: HBASE-8410-master.6.patch
Patch for fixing javadoc and find bugs warnings.
> Basic quota support for namespaces
> --
>
> Key: HBASE-8410
> URL: https://issues.apache.org/jira/browse/HBASE-8410
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Liu
>Assignee: Vandana Ayyalasomayajula
> Attachments: HBASE-8410-master.1.patch, HBASE-8410-master.4.patch,
> HBASE-8410-master.5.patch, HBASE-8410-master.6.patch,
> HBASE-8410-master.patch, HBASE-8410.docx, HBASE-8410_trunk_14.patch
>
>
> This task involves creating an observer which provides basic quota support to
> namespaces in terms of (1) number of tables and (2) number of regions. The
> quota support can be enabled by setting:
>
> hbase.coprocessor.region.classes
> org.apache.hadoop.hbase.namespace.NamespaceController
>
>
> hbase.coprocessor.master.classes
> org.apache.hadoop.hbase.namespace.NamespaceController
>
> in the hbase-site.xml.
> To add quotas to namespace, while creating namespace properties need to be
> added.
> Examples:
> 1. namespace_create 'ns1', {'hbase.namespace.quota.maxregion'=>'10'}
> 2. 1. namespace_create 'ns2', {'hbase.namespace.quota.maxtables'=>'2'},
> {'hbase.namespace.quota.maxregion'=>'5'}
> The quotas can be modified/added to namespace at any point of time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12768) Support enable cache_data_on_write in Shell while creating table
[ https://issues.apache.org/jira/browse/HBASE-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266751#comment-14266751 ] Hudson commented on HBASE-12768: SUCCESS: Integrated in HBase-1.0 #638 (See [https://builds.apache.org/job/HBase-1.0/638/]) HBASE-12768 - Support enable cache_data_on_write in Shell while creating (enis: rev f92c2fbc03d486ff0c219e65c7c84a125d4ac83c) * hbase-shell/src/main/ruby/hbase/admin.rb > Support enable cache_data_on_write in Shell while creating table > > > Key: HBASE-12768 > URL: https://issues.apache.org/jira/browse/HBASE-12768 > Project: HBase > Issue Type: Improvement > Components: shell >Affects Versions: 1.0.0, 2.0.0, 0.94.27 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 > > Attachments: HBASE-12768.patch > > > A simple approach to support cache_data_on_write while creating table in > shell. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12268) Add support for Scan.setRowPrefixFilter to shell
[ https://issues.apache.org/jira/browse/HBASE-12268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266725#comment-14266725 ] Hudson commented on HBASE-12268: SUCCESS: Integrated in HBase-1.1 #62 (See [https://builds.apache.org/job/HBase-1.1/62/]) HBASE-12268 Add support for Scan.setRowPrefixFilter to shell (Niels Basjes) (tedyu: rev 06b386f2547b26fe9cd38d12bcb110b7c70b7853) * hbase-shell/src/main/ruby/hbase.rb * hbase-shell/src/main/ruby/hbase/table.rb * hbase-shell/src/main/ruby/shell/commands/scan.rb * hbase-shell/src/main/ruby/shell/commands/set_visibility.rb * hbase-shell/src/test/ruby/hbase/table_test.rb > Add support for Scan.setRowPrefixFilter to shell > > > Key: HBASE-12268 > URL: https://issues.apache.org/jira/browse/HBASE-12268 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Niels Basjes >Assignee: Niels Basjes > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-12268-2014-10-15-v1.patch, > HBASE-12268-2014-12-05-v2.patch > > > I think having the feature introduced in HBASE-11990 in the hbase shell would > be very useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12768) Support enable cache_data_on_write in Shell while creating table
[ https://issues.apache.org/jira/browse/HBASE-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266724#comment-14266724 ] Hudson commented on HBASE-12768: SUCCESS: Integrated in HBase-1.1 #62 (See [https://builds.apache.org/job/HBase-1.1/62/]) HBASE-12768 - Support enable cache_data_on_write in Shell while creating (enis: rev 141b33a630cb7bc11087bd0e09e28c4ac1cb21c4) * hbase-shell/src/main/ruby/hbase/admin.rb > Support enable cache_data_on_write in Shell while creating table > > > Key: HBASE-12768 > URL: https://issues.apache.org/jira/browse/HBASE-12768 > Project: HBase > Issue Type: Improvement > Components: shell >Affects Versions: 1.0.0, 2.0.0, 0.94.27 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 > > Attachments: HBASE-12768.patch > > > A simple approach to support cache_data_on_write while creating table in > shell. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10528: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch, Churro. > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-10528-0.94.patch, HBASE-10528-0.98.patch, > HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10528) DefaultBalancer selects plans to move regions onto draining nodes
[ https://issues.apache.org/jira/browse/HBASE-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10528: --- Fix Version/s: 1.1.0 2.0.0 Hadoop Flags: Reviewed > DefaultBalancer selects plans to move regions onto draining nodes > - > > Key: HBASE-10528 > URL: https://issues.apache.org/jira/browse/HBASE-10528 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-10528-0.94.patch, HBASE-10528-0.98.patch, > HBASE-10528.patch, HBASE-10528.v2.patch > > > We have quite a large cluster > 100k regions, and we needed to isolate a > region was very hot until we could push a patch. We put this region on its > own regionserver and set it in the draining state. The default balancer was > selecting regions to move to this cluster for its region plans. > It just so happened for other tables, the default load balancer was creating > plans for the draining servers, even though they were not available to move > regions to. Thus we were closing regions, then attempting to move them to > the draining server then finding out its draining. > We had to disable the balancer to resolve this issue. > There are some approaches we can take here. > 1. Exclude draining servers altogether, don't even pass those into the load > balancer from HMaster. > 2. We could exclude draining servers from ceiling and floor calculations > where we could potentially skip load balancing because those draining servers > wont be represented when deciding whether to balance. > 3. Along with #2 when assigning regions, we would skip plans to assign > regions to those draining servers. > I am in favor of #1 which is simply removes servers as candidates for > balancing if they are in the draining state. > But I would love to hear what everyone else thinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12794) Guidelines for filing JIRA issues
[ https://issues.apache.org/jira/browse/HBASE-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266678#comment-14266678 ] stack commented on HBASE-12794: --- Thanks [~apurtell] I like the way your characterization of types. Will steal it. > Guidelines for filing JIRA issues > - > > Key: HBASE-12794 > URL: https://issues.apache.org/jira/browse/HBASE-12794 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > > Following on from Andrew's JIRA year-end cleaning spree, lets get some > guidelines on filing issues e.g. fill out all pertinent fields, add context > and provenance, add value (i.e. triage), don't file issues that are nought > but repeat of info available elsewhere (build box or mailing list), be > reluctant filing issues that don't have a resource behind them, don't file > issues on behalf of others, don't split fixes across multiple issues (because > there are poor folks coming behind us trying to backport our mess and > piecemeal makes their jobs harder), and so on. > The guidelines are not meant to put a chill on the opening of issues when > problems are found, especially not for new contributors. They are more meant > for quoting to veteran contributors who continue to file issues in violation > of what was thought a common understanding; rather than explain each time why > an issue has been marked invalid, it would be better if we can quote chapter > and verse from the refguide. > Dump any suggestion in here and I'll wind them up as a patch that I'll run by > dev mailing list to get consensus before committing. > Here is a running google doc if you'd like to add comment: > https://docs.google.com/document/d/1p3ArVLcnQnifk6ZsF635qWBhMmfTUJsISyK15DXnam0/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5205) Delete handles deleteFamily incorrectly
[ https://issues.apache.org/jira/browse/HBASE-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266648#comment-14266648 ] Lars Hofhansl commented on HBASE-5205: -- Didn't seem important enough and it does change the behavior subtly. We can certainly "warm it up again". > Delete handles deleteFamily incorrectly > --- > > Key: HBASE-5205 > URL: https://issues.apache.org/jira/browse/HBASE-5205 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 5205.txt > > > Delete.deleteFamily clears all other markers for the same family. > That is not correct as some of these other markers might be for a later time. > That logic should be removed. > If (really) needed this can be slightly optimized by keeping track of the max > TS so far for each family. > If both the TS-so-far and the TS of a new deleteFamily request is < > LATEST_TIMESTAMP and the TS-so-far is < deleteFamily marker, or if both the > TS-so-far and the new TS equal LATEST_TIMESTAMP, then the previous delete > markerz for that family could be removed. > I think that might be overkill, as most deletes issued from clients are for > LATEST_TIMESTAMP (which the server translates to the current time). > I'll have a (one-line) patch soon. Unless folks insist on the optimization I > mentioned above. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12480) Regions in FAILED_OPEN/FAILED_CLOSE should be processed on master failover
[ https://issues.apache.org/jira/browse/HBASE-12480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266622#comment-14266622 ] Virag Kothari commented on HBASE-12480: --- Yes, I will put a patch by today > Regions in FAILED_OPEN/FAILED_CLOSE should be processed on master failover > --- > > Key: HBASE-12480 > URL: https://issues.apache.org/jira/browse/HBASE-12480 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Virag Kothari >Assignee: Virag Kothari > Fix For: 2.0.0, 0.98.10, 1.1.0 > > Attachments: HBASE-12480.patch > > > For zk assignment, we used to process this regions. For zk less assignment, > we should do the same -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12694) testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class leaves regions in transition
[ https://issues.apache.org/jira/browse/HBASE-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266616#comment-14266616 ] Virag Kothari commented on HBASE-12694: --- +1 on the new patches. Will commit soon to all branches > testTableExistsIfTheSpecifiedTableRegionIsSplitParent in > TestSplitTransactionOnCluster class leaves regions in transition > - > > Key: HBASE-12694 > URL: https://issues.apache.org/jira/browse/HBASE-12694 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.0, 0.98.9 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Fix For: 2.0.0, 0.98.10, 1.1.0 > > Attachments: HBASE-12694-98.1.patch, HBASE-12694-98.2.patch, > HBASE-12694-branch-1.1.patch, HBASE-12694-branch-1.2.patch, > HBASE-12694-branch-1.patch, HBASE-12694-master.2.patch, > HBASE-12694-master.patch, HBASE-12694-master.patch > > > There seems to a clean up issue with the unit test > "testTableExistsIfTheSpecifiedTableRegionIsSplitParent" in > TestSplitTransactionOnCluster class. It always has a region in transition. So > any unit test that executes after that test, will cause balance method to > return false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12728:
--
Fix Version/s: 1.1.0
2.0.0
1.0.0
> buffered writes substantially less useful after removal of HTablePool
> -
>
> Key: HBASE-12728
> URL: https://issues.apache.org/jira/browse/HBASE-12728
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 0.98.0
>Reporter: Aaron Beppu
>Assignee: Solomon Duskis
> Fix For: 1.0.0, 2.0.0, 1.1.0
>
>
> In previous versions of HBase, when use of HTablePool was encouraged, HTable
> instances were long-lived in that pool, and for that reason, if autoFlush was
> set to false, the table instance could accumulate a full buffer of writes
> before a flush was triggered. Writes from the client to the cluster could
> then be substantially larger and less frequent than without buffering.
> However, when HTablePool was deprecated, the primary justification seems to
> have been that creating HTable instances is cheap, so long as the connection
> and executor service being passed to it are pre-provided. A use pattern was
> encouraged where users should create a new HTable instance for every
> operation, using an existing connection and executor service, and then close
> the table. In this pattern, buffered writes are substantially less useful;
> writes are as small and as frequent as they would have been with
> autoflush=true, except the synchronous write is moved from the operation
> itself to the table close call which immediately follows.
> More concretely :
> ```
> // Given these two helpers ...
> private HTableInterface getAutoFlushTable(String tableName) throws
> IOException {
> // (autoflush is true by default)
> return storedConnection.getTable(tableName, executorService);
> }
> private HTableInterface getBufferedTable(String tableName) throws IOException
> {
> HTableInterface table = getAutoFlushTable(tableName);
> table.setAutoFlush(false);
> return table;
> }
> // it's my contention that these two methods would behave almost identically,
> // except the first will hit a synchronous flush during the put call,
> and the second will
> // flush during the (hidden) close call on table.
> private void writeAutoFlushed(Put somePut) throws IOException {
> try (HTableInterface table = getAutoFlushTable(tableName)) {
> table.put(somePut); // will do synchronous flush
> }
> }
> private void writeBuffered(Put somePut) throws IOException {
> try (HTableInterface table = getBufferedTable(tableName)) {
> table.put(somePut);
> } // auto-close will trigger synchronous flush
> }
> ```
> For buffered writes to actually provide a performance benefit to users, one
> of two things must happen:
> - The writeBuffer itself shouldn't live, flush and die with the lifecycle of
> it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
> lifespan, this could cease to be an issue. However, if the same writeBuffer
> is appended to by multiple tables, then some additional concurrency control
> will be needed around it.
> - Alternatively, there should be some pattern for having long-lived HTable
> instances. However, since HTable is not thread-safe, we'd need multiple
> instances, and a mechanism for leasing them out safely -- which sure sounds a
> lot like the old HTablePool to me.
> See discussion on mailing list here :
> http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12574) Update replication metrics to not do so many map look ups.
[ https://issues.apache.org/jira/browse/HBASE-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266609#comment-14266609 ] Elliott Clark commented on HBASE-12574: --- yeah my bad. This was in before the rc branch was cut. > Update replication metrics to not do so many map look ups. > -- > > Key: HBASE-12574 > URL: https://issues.apache.org/jira/browse/HBASE-12574 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.98.6.1 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.0.0, 2.0.0, 1.1.0 > > Attachments: HBASE-12574.patch > > > Replication is the last metrics area that still does a significant amount of > hash map lookups. Lets re-factor that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12339) WAL performance evaluation tool doesn't roll logs
[ https://issues.apache.org/jira/browse/HBASE-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266607#comment-14266607 ] Enis Soztutar commented on HBASE-12339: --- Pushed to branch-1.0 as well. Thanks Sean. > WAL performance evaluation tool doesn't roll logs > - > > Key: HBASE-12339 > URL: https://issues.apache.org/jira/browse/HBASE-12339 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.99.0 >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.0.0, 2.0.0, 1.1.0 > > Attachments: HBASE-12339.1.patch.txt > > > the perf eval tool for the wal never sets up a log roller and instead used to > just call the rollWriter method directly. > Eventually it changed to call requestLogRoll instead of attempting to do the > roll itself. requestLogRoll is the same method used internal to the wal code > and it relies on there being a LogRoller to actually have rolls happen. (the > method just notifies all of the listeners for hte wal that one of them should > call the roll method.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12339) WAL performance evaluation tool doesn't roll logs
[ https://issues.apache.org/jira/browse/HBASE-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12339: -- Fix Version/s: 1.0.0 > WAL performance evaluation tool doesn't roll logs > - > > Key: HBASE-12339 > URL: https://issues.apache.org/jira/browse/HBASE-12339 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 0.99.0 >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.0.0, 2.0.0, 1.1.0 > > Attachments: HBASE-12339.1.patch.txt > > > the perf eval tool for the wal never sets up a log roller and instead used to > just call the rollWriter method directly. > Eventually it changed to call requestLogRoll instead of attempting to do the > roll itself. requestLogRoll is the same method used internal to the wal code > and it relies on there being a LogRoller to actually have rolls happen. (the > method just notifies all of the listeners for hte wal that one of them should > call the roll method.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-3638) If a FS bootstrap, need to also ensure ZK is cleaned
[
https://issues.apache.org/jira/browse/HBASE-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-3638:
-
Labels: beginner beginners (was: delete)
> If a FS bootstrap, need to also ensure ZK is cleaned
>
>
> Key: HBASE-3638
> URL: https://issues.apache.org/jira/browse/HBASE-3638
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Minor
> Labels: beginner, beginners
>
> In a test environment where a cycle of start, operation, kill hbase (repeat),
> noticed that we were doing a bootstrap on startup but then we were picking up
> the previous cycles zk state. It made for a mess in the test.
> Last thing seen on previous cycle was:
> {code}
> 2011-03-11 06:33:36,708 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING, server=X.X.X.60020,1299853933073,
> region=1028785192/.META.
> {code}
> Then, in the messed up cycle I saw:
> {code}
> 2011-03-11 06:42:48,530 INFO org.apache.hadoop.hbase.master.MasterFileSystem:
> BOOTSTRAP: creating ROOT and first META regions
> .
> {code}
> Then after setting watcher on .META., we get a
> {code}
> 2011-03-11 06:42:58,301 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Processing region
> .META.,,1.1028785192 in state RS_ZK_REGION_OPENED
> 2011-03-11 06:42:58,302 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Region in transition
> 1028785192 references a server no longer up X.X.X; letting RIT timeout so
> will be assigned elsewhere
> {code}
> We're all confused.
> Should at least clear our zk if a bootstrap happened.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12268) Add support for Scan.setRowPrefixFilter to shell
[ https://issues.apache.org/jira/browse/HBASE-12268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12268: --- Resolution: Fixed Status: Resolved (was: Patch Available) Integrated to branch-1 and master Thanks for the patch, Niels > Add support for Scan.setRowPrefixFilter to shell > > > Key: HBASE-12268 > URL: https://issues.apache.org/jira/browse/HBASE-12268 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Niels Basjes >Assignee: Niels Basjes > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-12268-2014-10-15-v1.patch, > HBASE-12268-2014-12-05-v2.patch > > > I think having the feature introduced in HBASE-11990 in the hbase shell would > be very useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-3638) If a FS bootstrap, need to also ensure ZK is cleaned
[
https://issues.apache.org/jira/browse/HBASE-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266593#comment-14266593
]
stack commented on HBASE-3638:
--
It is. This is a constant irritant, a result of our keeping state in two
places. Let me mark as beginner.
> If a FS bootstrap, need to also ensure ZK is cleaned
>
>
> Key: HBASE-3638
> URL: https://issues.apache.org/jira/browse/HBASE-3638
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Minor
> Labels: delete
>
> In a test environment where a cycle of start, operation, kill hbase (repeat),
> noticed that we were doing a bootstrap on startup but then we were picking up
> the previous cycles zk state. It made for a mess in the test.
> Last thing seen on previous cycle was:
> {code}
> 2011-03-11 06:33:36,708 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING, server=X.X.X.60020,1299853933073,
> region=1028785192/.META.
> {code}
> Then, in the messed up cycle I saw:
> {code}
> 2011-03-11 06:42:48,530 INFO org.apache.hadoop.hbase.master.MasterFileSystem:
> BOOTSTRAP: creating ROOT and first META regions
> .
> {code}
> Then after setting watcher on .META., we get a
> {code}
> 2011-03-11 06:42:58,301 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: Processing region
> .META.,,1.1028785192 in state RS_ZK_REGION_OPENED
> 2011-03-11 06:42:58,302 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Region in transition
> 1028785192 references a server no longer up X.X.X; letting RIT timeout so
> will be assigned elsewhere
> {code}
> We're all confused.
> Should at least clear our zk if a bootstrap happened.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-12768) Support enable cache_data_on_write in Shell while creating table
[ https://issues.apache.org/jira/browse/HBASE-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar resolved HBASE-12768. --- Resolution: Fixed Fix Version/s: 1.1.0 Pushed to branch-1 and branch-1.0 as well. Thanks Ram. > Support enable cache_data_on_write in Shell while creating table > > > Key: HBASE-12768 > URL: https://issues.apache.org/jira/browse/HBASE-12768 > Project: HBase > Issue Type: Improvement > Components: shell >Affects Versions: 1.0.0, 2.0.0, 0.94.27 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 > > Attachments: HBASE-12768.patch > > > A simple approach to support cache_data_on_write while creating table in > shell. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-4352) Apply version of hbase-4015 to branch
[ https://issues.apache.org/jira/browse/HBASE-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-4352. -- Resolution: Invalid No longer valid. For 0.90. [~clehene] Just go ahead and move issue state as you see fit. Ask for second opinion only when it unclear I'd say. > Apply version of hbase-4015 to branch > - > > Key: HBASE-4352 > URL: https://issues.apache.org/jira/browse/HBASE-4352 > Project: HBase > Issue Type: Task >Reporter: stack >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Labels: delete > Attachments: HBASE-4352_0.90.patch, HBASE-4352_0.90_1.patch > > > Consider adding a version of hbase-4015 to 0.90. It changes HRegionInterface > so would need move change to end of the Interface and then test that it > doesn't break rolling restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-5079) DistributedLogSplitter interrupt can be hazardous to regionserver health
[
https://issues.apache.org/jira/browse/HBASE-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-5079.
--
Resolution: Not a Problem
Resolving as not a problem, something we have not seen for years (I believe).
Thanks [~clehene]
> DistributedLogSplitter interrupt can be hazardous to regionserver health
>
>
> Key: HBASE-5079
> URL: https://issues.apache.org/jira/browse/HBASE-5079
> Project: HBase
> Issue Type: Bug
>Reporter: stack
> Labels: delete
>
> The DLS interrupt can kill the regionserver if happens while conversation w/
> namenode is going on.
> The interrupt is used to end a task on regionserver when done whether
> successful or to interrupt an ongoing split since assumed by another server.
> I saw this issue testing because I was killing servers. I also was suffering
> "HBASE-5078 DistributedLogSplitter failing to split file because it has edits
> for lots of regions" which made it more likely to happen.
> Here is what it looks like on the regionserver that died:
> {code}
> 2011-12-20 17:54:58,009 INFO
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: task
> /hbase/splitlog/hdfs%3A%2F%2Fsv4r11s38%3A7000%2Fhbase%2F.logs%2Fsv4r31s44%2C7003%2C1324365396770-splitting%2Fsv4r31s44%252C7003%252C1324365396770.1324403495463
> preempted from sv4r13s38,7003,1324365396583, current task state and
> owner=owned sv4r27s44,7003,1324365396664
> 2011-12-20 17:54:58,009 INFO
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: Sending interrupt to
> stop the worker thread
> 2011-12-20 17:54:59,133 INFO
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: task
> /hbase/splitlog/hdfs%3A%2F%2Fsv4r11s38%3A7000%2Fhbase%2F.logs%2Fsv4r31s44%2C7003%2C1324365396770-splitting%2Fsv4r31s44%252C7003%252C1324365396770.1324403495463
> preempted from sv4r13s38,7003,1324365396583, current task state and
> owner=owned sv4r27s44,7003,1324365396664
> 2011-12-20 17:54:59,134 INFO
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: Sending interrupt to
> stop the worker thread
> ...
> 2011-12-20 17:55:25,505 INFO
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: task
> /hbase/splitlog/hdfs%3A%2F%2Fsv4r11s38%3A7000%2Fhbase%2F.logs%2Fsv4r31s44%2C7003%2C1324365396770-splitting%2Fsv4r31s44%252C7003%252C1324365396770.1324403495463
> preempted from sv4r13s38,7003,1324365396583, current task state and
> owner=unassigned sv4r11s38,7001,1324365395047
> 2011-12-20 17:55:25,505 INFO
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: Sending interrupt to
> stop the worker thread
> {code}
> Three interrupts are sent over period of 31 seconds or so.
> Eventually the interrupt has an effect and I get:
> {code}
> 2011-12-20 17:55:25,505 INFO
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: Sending interrupt to
> stop the worker thread
> 2011-12-20 17:55:48,022 DEBUG org.apache.hadoop.hbase.regionserver.LogRoller:
> HLog roll requested
> 2011-12-20 17:55:58,070 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
> Exception: java.io.IOException: Call to sv4r11s38/10.4.11.38:7000 failed on
> local exception: java.nio.channels.ClosedByInterruptException
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
> at org.apache.hadoop.ipc.Client.call(Client.java:1071)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at $Proxy9.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy9.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3507)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3370)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2700(DFSClient.java:2586)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2826)
> Caused by: java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:341)
> at
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.Socket
[jira] [Updated] (HBASE-12268) Add support for Scan.setRowPrefixFilter to shell
[ https://issues.apache.org/jira/browse/HBASE-12268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12268: --- Fix Version/s: 1.1.0 2.0.0 Assignee: Niels Basjes Hadoop Flags: Reviewed > Add support for Scan.setRowPrefixFilter to shell > > > Key: HBASE-12268 > URL: https://issues.apache.org/jira/browse/HBASE-12268 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Niels Basjes >Assignee: Niels Basjes > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-12268-2014-10-15-v1.patch, > HBASE-12268-2014-12-05-v2.patch > > > I think having the feature introduced in HBASE-11990 in the hbase shell would > be very useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-5250) readAndProcess threw exception java.lang.NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-5250.
--
Resolution: Not a Problem
Resolving because we no longer see this (Thanks [~clehene])
> readAndProcess threw exception java.lang.NullPointerException
> -
>
> Key: HBASE-5250
> URL: https://issues.apache.org/jira/browse/HBASE-5250
> Project: HBase
> Issue Type: Bug
>Reporter: stack
> Labels: delete
>
> From Benoit...
> {code}2012-01-20 17:13:20,634 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server Responder: doAsyncWrite threw exception java.io.IOException:
> Broken pipe
> 2012-01-20 17:13:20,635 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> Server listener on 52146: readAndProcess threw exception
> java.lang.NullPointerException. Count of bytes read: 0
> java.lang.NullPointerException
>at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1119)
>at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:703)
>at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:495)
>at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>at java.lang.Thread.run(Thread.java:680)
> This is when I unexpectedly close a connection to a RS. When the
> channel gets disconnected, some buffer is nulled out, but then if
> readAndProcess() is called again, it triggers the NPE above. Not a
> huge deal, but also indicates that maybe some logic in the RS's error
> handling is incorrect.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12574) Update replication metrics to not do so many map look ups.
[ https://issues.apache.org/jira/browse/HBASE-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12574: -- Fix Version/s: 1.1.0 > Update replication metrics to not do so many map look ups. > -- > > Key: HBASE-12574 > URL: https://issues.apache.org/jira/browse/HBASE-12574 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.98.6.1 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.0.0, 2.0.0, 1.1.0 > > Attachments: HBASE-12574.patch > > > Replication is the last metrics area that still does a significant amount of > hash map lookups. Lets re-factor that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12574) Update replication metrics to not do so many map look ups.
[ https://issues.apache.org/jira/browse/HBASE-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266581#comment-14266581 ] Enis Soztutar commented on HBASE-12574: --- Nvm. checked the commit logs. > Update replication metrics to not do so many map look ups. > -- > > Key: HBASE-12574 > URL: https://issues.apache.org/jira/browse/HBASE-12574 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.98.6.1 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.0.0, 2.0.0, 1.1.0 > > Attachments: HBASE-12574.patch > > > Replication is the last metrics area that still does a significant amount of > hash map lookups. Lets re-factor that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12801) Failed to truncate a table while maintaing binary region boundaries
[
https://issues.apache.org/jira/browse/HBASE-12801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266578#comment-14266578
]
Sean Busbey commented on HBASE-12801:
-
I'm trying to test manually and having some trouble generating a proper binary
split point in the first place.
> Failed to truncate a table while maintaing binary region boundaries
> ---
>
> Key: HBASE-12801
> URL: https://issues.apache.org/jira/browse/HBASE-12801
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 0.94.11
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.28
>
> Attachments: HBASE-12801-0.94-v1.diff, HBASE-12801-trunk-v1.diff
>
>
> Binary region boundaries become wrong during
> converting byte array to normal string, and back to byte array in
> truncate_preserve of admin.rb, which makes the truncation of table failed.
> See: truncate_preserve method in admin.rb
> {code}
> splits = h_table.getRegionLocations().keys().map{|i|
> Bytes.toString(i.getStartKey)}.delete_if{|k| k == ""}.to_java :String
> splits = org.apache.hadoop.hbase.util.Bytes.toByteArrays(splits)
> {code}
> eg:
> {code}
> \xFA\x00\x00\x00\x00\x00\x00\x00 -> \xEF\xBF\xBD\x00\x00\x00\x00\x00\x00\x00
> \xFC\x00\x00\x00\x00\x00\x00\x00 -> \xEF\xBF\xBD\x00\x00\x00\x00\x00\x00\x00
> {code}
> Simple patch is using binary string instead of normal string.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12574) Update replication metrics to not do so many map look ups.
[ https://issues.apache.org/jira/browse/HBASE-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266577#comment-14266577 ] Enis Soztutar commented on HBASE-12574: --- Sorry my +1 above stood for branch-1.0 as well. Is this pushed to branch-1 as well as branch-1.0? > Update replication metrics to not do so many map look ups. > -- > > Key: HBASE-12574 > URL: https://issues.apache.org/jira/browse/HBASE-12574 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.98.6.1 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-12574.patch > > > Replication is the last metrics area that still does a significant amount of > hash map lookups. Lets re-factor that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12071) Separate out thread pool for Master <-> RegionServer communication
[
https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266573#comment-14266573
]
Nick Dimiduk commented on HBASE-12071:
--
What's the thinking on increasing the number of queues to 2?
{noformat}
- this.priorityExecutor =
- priorityHandlerCount > 0 ? new BalancedQueueRpcExecutor("Priority",
priorityHandlerCount,
- 1, maxQueueLength, conf, abortable) : null;
+// Create 2 queues to help priorityExecutor be more scalable.
+this.priorityExecutor = priorityHandlerCount > 0 ?
+new BalancedQueueRpcExecutor("Priority", priorityHandlerCount, 2,
maxQueueLength) : null;
{noformat}
It's probably worth while turning priority into a protobuf enum. I don't know
if we can do that, given that RequestHeader.priority is already defined as a
uint32. The docs say valid enum value range is 32-bit signed int, so maybe we'd
have to deprecate the existing field? Someone who knows more about protobuf,
any thoughts here?
Other question: can we move the QosPriority annotations over to the
BlockingInterfaces? Maybe not since they're generated protobuf...
Otherwise, +1. You test all this out [~syuanjiang]? Working as expected?
> Separate out thread pool for Master <-> RegionServer communication
> --
>
> Key: HBASE-12071
> URL: https://issues.apache.org/jira/browse/HBASE-12071
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Sudarshan Kadambi
>Assignee: Stephen Yuan Jiang
> Labels: reliablity
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-12071.v1-branch-1.patch,
> HBASE-12071.v2-master.patch, HBASE-12071.v2-master.patch,
> HBASE-12071.v3-master.patch, HBASE-12071.v4-master.patch,
> HBASE-12071.v5-master.patch
>
>
> Over in HBASE-12028, there is a discussion about the case of a RegionServer
> still being alive despite all its handler threads being dead. One outcome of
> this is that the Master is left hanging on the RS for completion of various
> operations - such as region un-assignment when a table is disabled. Does it
> make sense to create a separate thread pool for communication between the
> Master and the RS? This addresses not just the case of the RPC handler
> threads terminating but also long-running queries or co-processor executions
> holding up master operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12071) Separate out thread pool for Master <-> RegionServer communication
[
https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266546#comment-14266546
]
Hadoop QA commented on HBASE-12071:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690375/HBASE-12071.v1-branch-1.patch
against master branch at commit 26e3e1b300670d6c3f9f4853aa937226592b9537.
ATTACHMENT ID: 12690375
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12333//console
This message is automatically generated.
> Separate out thread pool for Master <-> RegionServer communication
> --
>
> Key: HBASE-12071
> URL: https://issues.apache.org/jira/browse/HBASE-12071
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Sudarshan Kadambi
>Assignee: Stephen Yuan Jiang
> Labels: reliablity
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-12071.v1-branch-1.patch,
> HBASE-12071.v2-master.patch, HBASE-12071.v2-master.patch,
> HBASE-12071.v3-master.patch, HBASE-12071.v4-master.patch,
> HBASE-12071.v5-master.patch
>
>
> Over in HBASE-12028, there is a discussion about the case of a RegionServer
> still being alive despite all its handler threads being dead. One outcome of
> this is that the Master is left hanging on the RS for completion of various
> operations - such as region un-assignment when a table is disabled. Does it
> make sense to create a separate thread pool for communication between the
> Master and the RS? This addresses not just the case of the RPC handler
> threads terminating but also long-running queries or co-processor executions
> holding up master operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12658) Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12658: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to 0.98 > Backport HBASE-12574 (Update replication metrics to not do so many map look > ups) to 0.98 > > > Key: HBASE-12658 > URL: https://issues.apache.org/jira/browse/HBASE-12658 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.98.10 > > Attachments: HBASE-12658-0.98.patch, HBASE-12658-0.98.patch > > > Backport HBASE-12574 (Update replication metrics to not do so many map look > ups) to 0.98. Nice enhancement. It's a refactor so need to check for compat > issues there, and we need to add Hadoop 1 compat module support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11983) HRegion constructors should not create HLog
[
https://issues.apache.org/jira/browse/HBASE-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266482#comment-14266482
]
Nick Dimiduk commented on HBASE-11983:
--
bq. I'm happy to do my superset patch either here as a follow on jura.
Then let's get this change in as it meets the explicit requirement of the issue
subject. The followup can come when it's ready. Your reviews are humbly
requested.
> HRegion constructors should not create HLog
>
>
> Key: HBASE-11983
> URL: https://issues.apache.org/jira/browse/HBASE-11983
> Project: HBase
> Issue Type: Bug
> Components: wal
>Reporter: Enis Soztutar
>Assignee: Nick Dimiduk
> Labels: beginner
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-11983.00.patch, HBASE-11983.01.patch,
> HBASE-11983.02.patch
>
>
> We should get rid of HRegion creating its own HLog. It should ALWAYS get the
> log from outside.
> I think this was added for unit tests, but we should refrain from such
> practice in the future (adding UT constructors always leads to weird and
> critical bugs down the road). See recent: HBASE-11982, HBASE-11654.
> Get rid of weird things like ignoreHLog:
> {code}
> /**
>* @param ignoreHLog - true to skip generate new hlog if it is null, mostly
> for createTable
>*/
> public static HRegion createHRegion(final HRegionInfo info, final Path
> rootDir,
> final Configuration conf,
> final HTableDescriptor hTableDescriptor,
> final HLog hlog,
> final boolean initialize, final boolean
> ignoreHLog)
> {code}
> We can unify all the createXX and newXX methods and separate creating a
> region in the file system vs opening a region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12802) Remove unnecessary Table.flushCommits()
[
https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266480#comment-14266480
]
Nick Dimiduk commented on HBASE-12802:
--
[~stack] my mistake. For some reason I thought this change assumed put(List)
was sending a single RPC. Please disregard.
> Remove unnecessary Table.flushCommits()
> ---
>
> Key: HBASE-12802
> URL: https://issues.apache.org/jira/browse/HBASE-12802
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
> Fix For: 1.0.0, 2.0.0, 1.1.0
>
> Attachments: 12802-1.0.patch, HBASE-12802.patch, HBASE-12802B.patch
>
>
> It looks like there are a lot of places where flushCommits() are called on
> tables that do not have autoFlush turned on. That might be a legacy concern
> from pre-autoFlush days. HBASE-12728 will likely result in removing the
> flushCommits() method from Table. The patch for this issue should remove all
> unnecessary calls to flushCommit() to prepare for the full fix.
> flushCommits() are only necessary after a setAutoFlushTo(false) is called.
> Here's the relevant code from HTable.java:
> {code:title=HTable.java|borderStyle=solid}
> @Override
> public void put(final Put put)
> throws InterruptedIOException, RetriesExhaustedWithDetailsException {
> doPut(put);
> if (autoFlush) {
> flushCommits();
> }
> }
> /**
>* {@inheritDoc}
>*/
> @Override
> public void put(final List puts)
> throws InterruptedIOException, RetriesExhaustedWithDetailsException {
> for (Put put : puts) {
> doPut(put);
> }
> if (autoFlush) {
> flushCommits();
> }
> }
> {code}
> Puts have implicit flushCommits() calls when autoFlush is true. Deletes are
> not affected by autoFlush() and are not directly impacted by flushCommits()
> since deletes are not added to the writeAysncBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12811) [AccessController] NPE while scan a table with user associated with multiple groups.
[
https://issues.apache.org/jira/browse/HBASE-12811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266465#comment-14266465
]
Andrew Purtell commented on HBASE-12811:
Set fix versions
> [AccessController] NPE while scan a table with user associated with multiple
> groups.
>
>
> Key: HBASE-12811
> URL: https://issues.apache.org/jira/browse/HBASE-12811
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 0.98.9
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
>
>
> A user is associated with two groups.
> {noformat}
> /hbase/bin> groups ashish_test
> ashish_test : defaultgroup ashish_test_1420524824527
> {noformat}
> One of its group is granted permission on a table as shown by user_permission
> command.
> {noformat}
> hbase(main):005:0> user_permission 't1'
> User
> Table,Family,Qualifier:Permission
> @ashish_test_1420524824527 t1,,: [Permission:
> actions=EXEC,WRITE,CREATE]
> @ashish_test_1420524824527 t1,d,: [Permission:
> actions=EXEC,WRITE,CREATE]
> hbase t1,,: [Permission:
> actions=READ,WRITE,EXEC,CREATE,ADMIN]
> 3 row(s) in 0.3710 seconds
> {noformat}
> Now when this user try the scan the table, we get the following exception.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.security.access.TablePermission.implies(TablePermission.java:215)
> at
> org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:340)
> at
> org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:332)
> at
> org.apache.hadoop.hbase.security.access.TableAuthManager.authorizeGroup(TableAuthManager.java:473)
> at
> org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:490)
> at
> org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:500)
> at
> org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:415)
> at
> org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:484)
> at
> org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1504)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:2027)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1987)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3102)
> {noformat}
> *Note:* Line numbers may not match.
> Exception is coming because the other group of same user which has not been
> granted permission on the table will have the TablePermission's table(name)
> as null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12811) [AccessController] NPE while scan a table with user associated with multiple groups.
[
https://issues.apache.org/jira/browse/HBASE-12811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12811:
---
Fix Version/s: 1.1.0
0.98.10
2.0.0
1.0.0
Summary: [AccessController] NPE while scan a table with user
associated with multiple groups. (was: NPE while scan a table with user
associated with multiple groups.)
> [AccessController] NPE while scan a table with user associated with multiple
> groups.
>
>
> Key: HBASE-12811
> URL: https://issues.apache.org/jira/browse/HBASE-12811
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 0.98.9
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
>
>
> A user is associated with two groups.
> {noformat}
> /hbase/bin> groups ashish_test
> ashish_test : defaultgroup ashish_test_1420524824527
> {noformat}
> One of its group is granted permission on a table as shown by user_permission
> command.
> {noformat}
> hbase(main):005:0> user_permission 't1'
> User
> Table,Family,Qualifier:Permission
> @ashish_test_1420524824527 t1,,: [Permission:
> actions=EXEC,WRITE,CREATE]
> @ashish_test_1420524824527 t1,d,: [Permission:
> actions=EXEC,WRITE,CREATE]
> hbase t1,,: [Permission:
> actions=READ,WRITE,EXEC,CREATE,ADMIN]
> 3 row(s) in 0.3710 seconds
> {noformat}
> Now when this user try the scan the table, we get the following exception.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.security.access.TablePermission.implies(TablePermission.java:215)
> at
> org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:340)
> at
> org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:332)
> at
> org.apache.hadoop.hbase.security.access.TableAuthManager.authorizeGroup(TableAuthManager.java:473)
> at
> org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:490)
> at
> org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:500)
> at
> org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:415)
> at
> org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:484)
> at
> org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1504)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:2027)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1987)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3102)
> {noformat}
> *Note:* Line numbers may not match.
> Exception is coming because the other group of same user which has not been
> granted permission on the table will have the TablePermission's table(name)
> as null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12684: -- Attachment: myrecording.jfr Here is a flight recording of the latest patch. 60seconds sample. Open it w/ your jdk bin/jmc command. Shows the skiplist in AsyncRpcChannel as hot. Not sure fixing this would impinge that much on throughput though -- perhaps given it shows so prominently. GC has nice steady profile. > Add new AsyncRpcClient > -- > > Key: HBASE-12684 > URL: https://issues.apache.org/jira/browse/HBASE-12684 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Jurriaan Mous >Assignee: Jurriaan Mous > Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, > HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, > HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, > HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, > HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v19.1.patch, > HBASE-12684-v19.patch, HBASE-12684-v19.patch, HBASE-12684-v2.patch, > HBASE-12684-v20-heapBuffer.patch, HBASE-12684-v20.patch, > HBASE-12684-v21-heapBuffer.1.patch, HBASE-12684-v21-heapBuffer.patch, > HBASE-12684-v21.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, > HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, > HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch, > myrecording.jfr, requests.png > > > With the changes in HBASE-12597 it is possible to add new RpcClients. This > issue is about adding a new Async RpcClient which would enable HBase to do > non blocking protobuf service communication. > Besides delivering a new AsyncRpcClient I would also like to ask the question > what it would take to replace the current RpcClient? This would enable to > simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-5242) Provide link on Master UI to hide All Monitored Tasks
[ https://issues.apache.org/jira/browse/HBASE-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-5242. --- Resolution: Later > Provide link on Master UI to hide All Monitored Tasks > - > > Key: HBASE-5242 > URL: https://issues.apache.org/jira/browse/HBASE-5242 > Project: HBase > Issue Type: Wish >Reporter: Ted Yu > Labels: delete > > Currently there is no link on Master UI to hide All Monitored Tasks. > We should provide such link. > Same applies to 'non-RPC Tasks', 'RPC Handler Tasks', etc -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-5109) Fix TestAvroServer so that it waits properly for the modifyTable operation to complete
[
https://issues.apache.org/jira/browse/HBASE-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-5109.
---
Resolution: Not a Problem
TestAvroServer doesn't exist any more.
> Fix TestAvroServer so that it waits properly for the modifyTable operation to
> complete
> --
>
> Key: HBASE-5109
> URL: https://issues.apache.org/jira/browse/HBASE-5109
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HBASE-5109-0.92.patch
>
>
> TestAvroServer has the following issue
>
> impl.modifyTable(tableAname, tableA);
> // It can take a while for the change to take effect. Wait here a while.
> while(impl.describeTable(tableAname) == null ) {
> Threads.sleep(100);
> }
> assertTrue(impl.describeTable(tableAname).maxFileSize == 123456L);
>
> impl.describeTable(tableAname) returns the default maxSize 256M right away as
> modifyTable is async. Before HBASE-4328 is fixed, we can fix the test code to
> wait for say max of 5 seconds to check if
> impl.describeTable(tableAname).maxFileSize is uploaded to 123456L.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-5309) Hbase web-app /jmx throws an exception
[ https://issues.apache.org/jira/browse/HBASE-5309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene resolved HBASE-5309. -- Resolution: Cannot Reproduce > Hbase web-app /jmx throws an exception > -- > > Key: HBASE-5309 > URL: https://issues.apache.org/jira/browse/HBASE-5309 > Project: HBase > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Enis Soztutar > > hbasemaster:60010/jmx throws an NoSuchMethodError exception -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-5309) Hbase web-app /jmx throws an exception
[ https://issues.apache.org/jira/browse/HBASE-5309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene reopened HBASE-5309: -- > Hbase web-app /jmx throws an exception > -- > > Key: HBASE-5309 > URL: https://issues.apache.org/jira/browse/HBASE-5309 > Project: HBase > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Enis Soztutar > > hbasemaster:60010/jmx throws an NoSuchMethodError exception -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-5309) Hbase web-app /jmx throws an exception
[ https://issues.apache.org/jira/browse/HBASE-5309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene resolved HBASE-5309. -- Resolution: Fixed [~apurtell] I'm not sure what version this got fixed in, but it works for both master and regionserver now. > Hbase web-app /jmx throws an exception > -- > > Key: HBASE-5309 > URL: https://issues.apache.org/jira/browse/HBASE-5309 > Project: HBase > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Enis Soztutar > > hbasemaster:60010/jmx throws an NoSuchMethodError exception -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5242) Provide link on Master UI to hide All Monitored Tasks
[ https://issues.apache.org/jira/browse/HBASE-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266408#comment-14266408 ] Cosmin Lehene commented on HBASE-5242: -- [~te...@apache.org], [~apurtell] still want this? There's no target version > Provide link on Master UI to hide All Monitored Tasks > - > > Key: HBASE-5242 > URL: https://issues.apache.org/jira/browse/HBASE-5242 > Project: HBase > Issue Type: Wish >Reporter: Ted Yu > Labels: delete > > Currently there is no link on Master UI to hide All Monitored Tasks. > We should provide such link. > Same applies to 'non-RPC Tasks', 'RPC Handler Tasks', etc -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5242) Provide link on Master UI to hide All Monitored Tasks
[ https://issues.apache.org/jira/browse/HBASE-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-5242: - Issue Type: Wish (was: Bug) > Provide link on Master UI to hide All Monitored Tasks > - > > Key: HBASE-5242 > URL: https://issues.apache.org/jira/browse/HBASE-5242 > Project: HBase > Issue Type: Wish >Reporter: Ted Yu > Labels: delete > > Currently there is no link on Master UI to hide All Monitored Tasks. > We should provide such link. > Same applies to 'non-RPC Tasks', 'RPC Handler Tasks', etc -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5242) Provide link on Master UI to hide All Monitored Tasks
[ https://issues.apache.org/jira/browse/HBASE-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-5242: - Labels: delete (was: ) > Provide link on Master UI to hide All Monitored Tasks > - > > Key: HBASE-5242 > URL: https://issues.apache.org/jira/browse/HBASE-5242 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Labels: delete > > Currently there is no link on Master UI to hide All Monitored Tasks. > We should provide such link. > Same applies to 'non-RPC Tasks', 'RPC Handler Tasks', etc -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5250) readAndProcess threw exception java.lang.NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266401#comment-14266401
]
Cosmin Lehene commented on HBASE-5250:
--
[~saint@gmail.com], [~apurtell] review/close?
> readAndProcess threw exception java.lang.NullPointerException
> -
>
> Key: HBASE-5250
> URL: https://issues.apache.org/jira/browse/HBASE-5250
> Project: HBase
> Issue Type: Bug
>Reporter: stack
> Labels: delete
>
> From Benoit...
> {code}2012-01-20 17:13:20,634 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server Responder: doAsyncWrite threw exception java.io.IOException:
> Broken pipe
> 2012-01-20 17:13:20,635 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> Server listener on 52146: readAndProcess threw exception
> java.lang.NullPointerException. Count of bytes read: 0
> java.lang.NullPointerException
>at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1119)
>at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:703)
>at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:495)
>at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>at java.lang.Thread.run(Thread.java:680)
> This is when I unexpectedly close a connection to a RS. When the
> channel gets disconnected, some buffer is nulled out, but then if
> readAndProcess() is called again, it triggers the NPE above. Not a
> huge deal, but also indicates that maybe some logic in the RS's error
> handling is incorrect.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5250) readAndProcess threw exception java.lang.NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cosmin Lehene updated HBASE-5250:
-
Labels: delete (was: )
> readAndProcess threw exception java.lang.NullPointerException
> -
>
> Key: HBASE-5250
> URL: https://issues.apache.org/jira/browse/HBASE-5250
> Project: HBase
> Issue Type: Bug
>Reporter: stack
> Labels: delete
>
> From Benoit...
> {code}2012-01-20 17:13:20,634 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server Responder: doAsyncWrite threw exception java.io.IOException:
> Broken pipe
> 2012-01-20 17:13:20,635 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> Server listener on 52146: readAndProcess threw exception
> java.lang.NullPointerException. Count of bytes read: 0
> java.lang.NullPointerException
>at
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1119)
>at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:703)
>at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:495)
>at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:470)
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>at java.lang.Thread.run(Thread.java:680)
> This is when I unexpectedly close a connection to a RS. When the
> channel gets disconnected, some buffer is nulled out, but then if
> readAndProcess() is called again, it triggers the NPE above. Not a
> huge deal, but also indicates that maybe some logic in the RS's error
> handling is incorrect.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12071) Separate out thread pool for Master <-> RegionServer communication
[ https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12071: --- Labels: reliablity (was: ) > Separate out thread pool for Master <-> RegionServer communication > -- > > Key: HBASE-12071 > URL: https://issues.apache.org/jira/browse/HBASE-12071 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.1.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Labels: reliablity > Fix For: 2.0.0, 1.1.0 > > Attachments: HBASE-12071.v1-branch-1.patch, > HBASE-12071.v2-master.patch, HBASE-12071.v2-master.patch, > HBASE-12071.v3-master.patch, HBASE-12071.v4-master.patch, > HBASE-12071.v5-master.patch > > > Over in HBASE-12028, there is a discussion about the case of a RegionServer > still being alive despite all its handler threads being dead. One outcome of > this is that the Master is left hanging on the RS for completion of various > operations - such as region un-assignment when a table is disabled. Does it > make sense to create a separate thread pool for communication between the > Master and the RS? This addresses not just the case of the RPC handler > threads terminating but also long-running queries or co-processor executions > holding up master operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
