[jira] [Commented] (HBASE-6367) List backup masters in ui.
[ https://issues.apache.org/jira/browse/HBASE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507897#comment-13507897 ] Jeffrey Zhong commented on HBASE-6367: -- Hey Elliott, I can move the backup master section under region servers. Where do you think it's more appropriate(at the very bottom)? For the header hiding, I saw current region server section still keep the header even though there is no region servers. Leaving the backup master header even though there is no backup masters is more clear to users espeically when a user migrates to new hbase version from an old one. Thanks, -Jeffrey > List backup masters in ui. > -- > > Key: HBASE-6367 > URL: https://issues.apache.org/jira/browse/HBASE-6367 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Jeffrey Zhong >Priority: Minor > Labels: noob > Fix For: 0.96.0 > > Attachments: Has BackupMasters.png, hbase-6367.patch, No > BackupMasters.png > > > Right now only the active master shows any information on the web ui. It > would be nice to see that there are backup masters waiting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Remove Writable Interface from KeyValue
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507876#comment-13507876 ] Lars Hofhansl commented on HBASE-7233: -- Why not add a protobuf representation for KV? Could be just a byte[] (right?) The generated class will have this interface. (as said on the mailing list, if KVs keeps Writable interface I'll still be happy) > Remove Writable Interface from KeyValue > --- > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: 7233.txt, 7233-v2.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Remove Writable Interface from KeyValue
[
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507868#comment-13507868
]
stack commented on HBASE-7233:
--
So, new Interface to replace Writable:
Interface HSerializable {
// The serialization methods pb...
mergeFrom(byte [])
mergeFrom(InputStream)
mergeDelimitedFrom(byte [])
mergeDelimitedFrom(InputStream)
parseFrom(byte [])
parseFrom(InputStream)
...
etc?
}
> Remove Writable Interface from KeyValue
> ---
>
> Key: HBASE-7233
> URL: https://issues.apache.org/jira/browse/HBASE-7233
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Attachments: 7233.txt, 7233-v2.txt
>
>
> Undo KeyValue being a Writable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507862#comment-13507862 ] stack commented on HBASE-6423: -- I understand now. +1 on commit. > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Jimmy Xiang > Attachments: trunk-6423.patch, trunk-6423_v2.1.patch, > trunk-6423_v2.patch, trunk-6423_v3.2.patch, trunk-6423_v3.3.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6367) List backup masters in ui.
[ https://issues.apache.org/jira/browse/HBASE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507842#comment-13507842 ] Elliott Clark commented on HBASE-6367: -- Thanks for the patch. I think the placement's a little wrong. Backup masters should be below the region servers. Also if there are no backup servers we probably wouldn't want to put up the header. > List backup masters in ui. > -- > > Key: HBASE-6367 > URL: https://issues.apache.org/jira/browse/HBASE-6367 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Jeffrey Zhong >Priority: Minor > Labels: noob > Fix For: 0.96.0 > > Attachments: Has BackupMasters.png, hbase-6367.patch, No > BackupMasters.png > > > Right now only the active master shows any information on the web ui. It > would be nice to see that there are backup masters waiting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507833#comment-13507833 ] Hudson commented on HBASE-7204: --- Integrated in HBase-0.94 #609 (See [https://builds.apache.org/job/HBase-0.94/609/]) HBASE-7204 Make hbck ErrorReporter pluggable, ADDENDUM (Revision 1415898) Result = SUCCESS jxiang : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java > Make hbck ErrorReporter pluggable > - > > Key: HBASE-7204 > URL: https://issues.apache.org/jira/browse/HBASE-7204 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: 0.94-7204.patch, trunk-7204.addendum, trunk-7204.patch, > trunk-7204_v2.1.patch, trunk-7204_v2.patch > > > Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507832#comment-13507832 ] Hudson commented on HBASE-7204: --- Integrated in HBase-TRUNK #3585 (See [https://builds.apache.org/job/HBase-TRUNK/3585/]) HBASE-7204 Make hbck ErrorReporter pluggable, ADDENDUM (Revision 1415897) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java > Make hbck ErrorReporter pluggable > - > > Key: HBASE-7204 > URL: https://issues.apache.org/jira/browse/HBASE-7204 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: 0.94-7204.patch, trunk-7204.addendum, trunk-7204.patch, > trunk-7204_v2.1.patch, trunk-7204_v2.patch > > > Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6367) List backup masters in ui.
[ https://issues.apache.org/jira/browse/HBASE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-6367: - Attachment: hbase-6367.patch > List backup masters in ui. > -- > > Key: HBASE-6367 > URL: https://issues.apache.org/jira/browse/HBASE-6367 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Jeffrey Zhong >Priority: Minor > Labels: noob > Fix For: 0.96.0 > > Attachments: Has BackupMasters.png, hbase-6367.patch, No > BackupMasters.png > > > Right now only the active master shows any information on the web ui. It > would be nice to see that there are backup masters waiting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6367) List backup masters in ui.
[ https://issues.apache.org/jira/browse/HBASE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-6367: - Attachment: Has BackupMasters.png No BackupMasters.png Attached are screenshots after my fix. > List backup masters in ui. > -- > > Key: HBASE-6367 > URL: https://issues.apache.org/jira/browse/HBASE-6367 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Jeffrey Zhong >Priority: Minor > Labels: noob > Fix For: 0.96.0 > > Attachments: Has BackupMasters.png, No BackupMasters.png > > > Right now only the active master shows any information on the web ui. It > would be nice to see that there are backup masters waiting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6367) List backup masters in ui.
[ https://issues.apache.org/jira/browse/HBASE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-6367: - Fix Version/s: 0.96.0 Status: Patch Available (was: Open) > List backup masters in ui. > -- > > Key: HBASE-6367 > URL: https://issues.apache.org/jira/browse/HBASE-6367 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Jeffrey Zhong >Priority: Minor > Labels: noob > Fix For: 0.96.0 > > Attachments: Has BackupMasters.png, No BackupMasters.png > > > Right now only the active master shows any information on the web ui. It > would be nice to see that there are backup masters waiting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-1299) JSPs don't HTML escape literals (ie: table names, region names, start & end keys)
[ https://issues.apache.org/jira/browse/HBASE-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reassigned HBASE-1299: --- Assignee: Nick Dimiduk > JSPs don't HTML escape literals (ie: table names, region names, start & end > keys) > - > > Key: HBASE-1299 > URL: https://issues.apache.org/jira/browse/HBASE-1299 > Project: HBase > Issue Type: Bug >Affects Versions: 0.19.0, 0.19.1 >Reporter: Hoss Man >Assignee: Nick Dimiduk > Attachments: 1299.patch > > > similar to HBASE-1298, the various JSPs included with HBase for monitoring > the system don't seem to do any HTML escaping when displaying user entered > data which may contain special characters: table names, region names, start > Keys, or end Keys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
[
https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507822#comment-13507822
]
Ted Yu commented on HBASE-6580:
---
For HConnection.java, please add javadoc for tableName parameter.
{code}
+// thread executor shared by all HTableInterface instances created
+// by this connection
{code}
nit: 'created by' -> 'created from'
{code}
+ } catch (InterruptedException e) {
+ this.hTableExecutor.shutdownNow();
+ }
{code}
Please restore interrupt status above.
{code}
+ if (hTableExecutor == null) {
+ // shared HTable thread executor not yet initialized
+ synchronized (this) {
+ if (hTableExecutor == null) {
{code}
Why use double checked locking ? Connection would be used to create (at least)
one table, right ?
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService)
> constructor
> -
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.92.2, 0.94.2
>Reporter: Lars Hofhansl
>Priority: Minor
> Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each
> invocation of getTable(...) would create a new HTable and close() would just
> close it.
> In testing I find this more light weight than HTablePool and easier to
> monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3171) Drop ROOT and instead store META location(s) directly in ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Daniel Cryans updated HBASE-3171: -- Attachment: HBASE-3171.patch First pass on this based on latest trunk. Works on a cluster, haven't encountered any issue. Still need to: - Modify the unit tests - Fix the PB migration code - Write the migration for this jira - Finish fixing HBCK > Drop ROOT and instead store META location(s) directly in ZooKeeper > -- > > Key: HBASE-3171 > URL: https://issues.apache.org/jira/browse/HBASE-3171 > Project: HBase > Issue Type: Improvement > Components: Client, master, regionserver, Zookeeper >Reporter: Jonathan Gray > Attachments: HBASE-3171.patch > > > Rather than storing the ROOT region location in ZooKeeper, going to ROOT, and > reading the META location, we should just store the META location directly in > ZooKeeper. > The purpose of the root region from the bigtable paper was to support > multiple meta regions. Currently, we explicitly only support a single meta > region, so the translation from our current code of a single root location to > a single meta location will be very simple. Long-term, it seems reasonable > that we could store several meta region locations in ZK. There's been some > discussion in HBASE-1755 about actually moving META into ZK, but I think this > jira is a good step towards taking some of the complexity out of how we have > to deal with catalog tables everywhere. > As-is, a new client already requires ZK to get the root location, so this > would not change those requirements in any way. > The primary motivation for this is to simplify things like CatalogTracker. > The way we can handle root in that class is really simple but the tracking of > meta is difficulty and a bit hacky. This hack on tracking of the meta > location is what caused one of the bugs over in HBASE-3159. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1299) JSPs don't HTML escape literals (ie: table names, region names, start & end keys)
[
https://issues.apache.org/jira/browse/HBASE-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507799#comment-13507799
]
Nick Dimiduk commented on HBASE-1299:
-
steps to repro:
> create 't1', {NAME => 'f1'}
> put 't1', "alert('hello world');", 'f1:foo', 0
> split 't1', "alert('hello world');"
open http://localhost:60010/table.jsp?name=t1
alerts will pop.
> JSPs don't HTML escape literals (ie: table names, region names, start & end
> keys)
> -
>
> Key: HBASE-1299
> URL: https://issues.apache.org/jira/browse/HBASE-1299
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.19.0, 0.19.1
>Reporter: Hoss Man
> Attachments: 1299.patch
>
>
> similar to HBASE-1298, the various JSPs included with HBase for monitoring
> the system don't seem to do any HTML escaping when displaying user entered
> data which may contain special characters: table names, region names, start
> Keys, or end Keys
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-1299) JSPs don't HTML escape literals (ie: table names, region names, start & end keys)
[ https://issues.apache.org/jira/browse/HBASE-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-1299: Status: Patch Available (was: Open) This is a hacky kind of fix to the existing JSP pages. The proper solution should be the replacement of these pages with jamon templates, a la MasterStatus. Thoughts? > JSPs don't HTML escape literals (ie: table names, region names, start & end > keys) > - > > Key: HBASE-1299 > URL: https://issues.apache.org/jira/browse/HBASE-1299 > Project: HBase > Issue Type: Bug >Affects Versions: 0.19.1, 0.19.0 >Reporter: Hoss Man > Attachments: 1299.patch > > > similar to HBASE-1298, the various JSPs included with HBase for monitoring > the system don't seem to do any HTML escaping when displaying user entered > data which may contain special characters: table names, region names, start > Keys, or end Keys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-1299) JSPs don't HTML escape literals (ie: table names, region names, start & end keys)
[ https://issues.apache.org/jira/browse/HBASE-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-1299: Attachment: 1299.patch > JSPs don't HTML escape literals (ie: table names, region names, start & end > keys) > - > > Key: HBASE-1299 > URL: https://issues.apache.org/jira/browse/HBASE-1299 > Project: HBase > Issue Type: Bug >Affects Versions: 0.19.0, 0.19.1 >Reporter: Hoss Man > Attachments: 1299.patch > > > similar to HBASE-1298, the various JSPs included with HBase for monitoring > the system don't seem to do any HTML escaping when displaying user entered > data which may contain special characters: table names, region names, start > Keys, or end Keys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7204: --- Attachment: trunk-7204.addendum Attached an addendum to fix the jdk 1.7 test issue. > Make hbck ErrorReporter pluggable > - > > Key: HBASE-7204 > URL: https://issues.apache.org/jira/browse/HBASE-7204 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: 0.94-7204.patch, trunk-7204.addendum, trunk-7204.patch, > trunk-7204_v2.1.patch, trunk-7204_v2.patch > > > Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507790#comment-13507790 ] Hudson commented on HBASE-7204: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #282 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/282/]) HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415871) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java > Make hbck ErrorReporter pluggable > - > > Key: HBASE-7204 > URL: https://issues.apache.org/jira/browse/HBASE-7204 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: 0.94-7204.patch, trunk-7204.patch, > trunk-7204_v2.1.patch, trunk-7204_v2.patch > > > Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5069) [book] Document how to count rows
[ https://issues.apache.org/jira/browse/HBASE-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507791#comment-13507791 ] Hudson commented on HBASE-5069: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #282 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/282/]) HBASE-5069 [book] Document how to count rows (Revision 1415733) Result = FAILURE stack : Files : * /hbase/trunk/src/docbkx/ops_mgt.xml * /hbase/trunk/src/docbkx/shell.xml > [book] Document how to count rows > - > > Key: HBASE-5069 > URL: https://issues.apache.org/jira/browse/HBASE-5069 > Project: HBase > Issue Type: Improvement >Reporter: Jean-Daniel Cryans >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-5069.patch > > > I just saw that there's no documentation on how to count rows. Basically > there's: > - The "count" command in the shell, it's quite fast when configured with the > right CACHE. > - The "rowcounter" MR job, it's super fast but you need to have a MapReduce > setup in place in order to use it. > There's also a way to estimate the row count by using the HFile tool and > doing a bit of bash magic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7234) Remove long-deprecated HServerAddress and HServerInfo Writables
[ https://issues.apache.org/jira/browse/HBASE-7234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507789#comment-13507789 ] Hudson commented on HBASE-7234: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #282 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/282/]) HBASE-7234 Remove long-deprecated HServerAddress and HServerInfo Writables (Revision 1415766) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HRegionLocation.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HServerAddress.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HServerInfo.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHServerAddress.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHServerInfo.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/TestTableResource.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java > Remove long-deprecated HServerAddress and HServerInfo Writables > --- > > Key: HBASE-7234 > URL: https://issues.apache.org/jira/browse/HBASE-7234 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7234.txt, 7234v2.txt > > > These classes have been deprecated since 0.92 or before. Remove them. > Remove them too because they are Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507786#comment-13507786 ] Hudson commented on HBASE-7204: --- Integrated in HBase-0.94 #608 (See [https://builds.apache.org/job/HBase-0.94/608/]) HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415876) Result = SUCCESS jxiang : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java > Make hbck ErrorReporter pluggable > - > > Key: HBASE-7204 > URL: https://issues.apache.org/jira/browse/HBASE-7204 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: 0.94-7204.patch, trunk-7204.patch, > trunk-7204_v2.1.patch, trunk-7204_v2.patch > > > Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7250) create integration test for balancing regions and killing region servers - 2
[ https://issues.apache.org/jira/browse/HBASE-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7250: Attachment: HBASE-7250-v0.patch > create integration test for balancing regions and killing region servers - 2 > > > Key: HBASE-7250 > URL: https://issues.apache.org/jira/browse/HBASE-7250 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-7250-v0.patch > > > The original test is too general; need another one that would be more > targeted and would test master logic in particular (e.g. not kill master). I > re-discovered HBASE-6060 using it on the first run :) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7250) create integration test for balancing regions and killing region servers - 2
[ https://issues.apache.org/jira/browse/HBASE-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7250: Status: Patch Available (was: Open) > create integration test for balancing regions and killing region servers - 2 > > > Key: HBASE-7250 > URL: https://issues.apache.org/jira/browse/HBASE-7250 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-7250-v0.patch > > > The original test is too general; need another one that would be more > targeted and would test master logic in particular (e.g. not kill master). I > re-discovered HBASE-6060 using it on the first run :) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7250) create integration test for balancing regions and killing region servers - 2
Sergey Shelukhin created HBASE-7250: --- Summary: create integration test for balancing regions and killing region servers - 2 Key: HBASE-7250 URL: https://issues.apache.org/jira/browse/HBASE-7250 Project: HBase Issue Type: Improvement Components: test Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor The original test is too general; need another one that would be more targeted and would test master logic in particular (e.g. not kill master). I re-discovered HBASE-6060 using it on the first run :) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-6060: Affects Version/s: (was: 0.94.3) > Regions's in OPENING state from failed regionservers takes a long time to > recover > - > > Key: HBASE-6060 > URL: https://issues.apache.org/jira/browse/HBASE-6060 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Reporter: Enis Soztutar >Assignee: Jimmy Xiang > Fix For: 0.96.0 > > Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, > 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, > 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, > 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, > 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, > HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, > HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch, > trunk-6060.patch, trunk-6060_v2.patch, trunk-6060_v3.3.patch > > > we have seen a pattern in tests, that the regions are stuck in OPENING state > for a very long time when the region server who is opening the region fails. > My understanding of the process: > > - master calls rs to open the region. If rs is offline, a new plan is > generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in > master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), > HMaster.assign() > - RegionServer, starts opening a region, changes the state in znode. But > that znode is not ephemeral. (see ZkAssign) > - Rs transitions zk node from OFFLINE to OPENING. See > OpenRegionHandler.process() > - rs then opens the region, and changes znode from OPENING to OPENED > - when rs is killed between OPENING and OPENED states, then zk shows OPENING > state, and the master just waits for rs to change the region state, but since > rs is down, that wont happen. > - There is a AssignmentManager.TimeoutMonitor, which does exactly guard > against these kind of conditions. It periodically checks (every 10 sec by > default) the regions in transition to see whether they timedout > (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, > which explains what you and I are seeing. > - ServerShutdownHandler in Master does not reassign regions in OPENING > state, although it handles other states. > Lowering that threshold from the configuration is one option, but still I > think we can do better. > Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507766#comment-13507766 ] Sergey Shelukhin commented on HBASE-6060: - oh, didn't read all the comments, nevermind > Regions's in OPENING state from failed regionservers takes a long time to > recover > - > > Key: HBASE-6060 > URL: https://issues.apache.org/jira/browse/HBASE-6060 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Reporter: Enis Soztutar >Assignee: Jimmy Xiang > Fix For: 0.96.0 > > Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, > 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, > 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, > 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, > 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, > HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, > HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch, > trunk-6060.patch, trunk-6060_v2.patch, trunk-6060_v3.3.patch > > > we have seen a pattern in tests, that the regions are stuck in OPENING state > for a very long time when the region server who is opening the region fails. > My understanding of the process: > > - master calls rs to open the region. If rs is offline, a new plan is > generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in > master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), > HMaster.assign() > - RegionServer, starts opening a region, changes the state in znode. But > that znode is not ephemeral. (see ZkAssign) > - Rs transitions zk node from OFFLINE to OPENING. See > OpenRegionHandler.process() > - rs then opens the region, and changes znode from OPENING to OPENED > - when rs is killed between OPENING and OPENED states, then zk shows OPENING > state, and the master just waits for rs to change the region state, but since > rs is down, that wont happen. > - There is a AssignmentManager.TimeoutMonitor, which does exactly guard > against these kind of conditions. It periodically checks (every 10 sec by > default) the regions in transition to see whether they timedout > (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, > which explains what you and I are seeing. > - ServerShutdownHandler in Master does not reassign regions in OPENING > state, although it handles other states. > Lowering that threshold from the configuration is one option, but still I > think we can do better. > Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7249) add test name filter to IntegrationTestsDriver
[ https://issues.apache.org/jira/browse/HBASE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507764#comment-13507764 ] Sergey Shelukhin commented on HBASE-7249: - [~enis] Can you please take a look? Thanks. > add test name filter to IntegrationTestsDriver > -- > > Key: HBASE-7249 > URL: https://issues.apache.org/jira/browse/HBASE-7249 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.3, 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-7249-v0-094.patch, HBASE-7249-v0.patch > > > As the number of tests grows, one might want to just run one, or a subset -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507762#comment-13507762 ] Hudson commented on HBASE-7204: --- Integrated in HBase-TRUNK #3584 (See [https://builds.apache.org/job/HBase-TRUNK/3584/]) HBASE-7204 Make hbck ErrorReporter pluggable (Revision 1415871) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java > Make hbck ErrorReporter pluggable > - > > Key: HBASE-7204 > URL: https://issues.apache.org/jira/browse/HBASE-7204 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: 0.94-7204.patch, trunk-7204.patch, > trunk-7204_v2.1.patch, trunk-7204_v2.patch > > > Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507760#comment-13507760 ] Sergey Shelukhin commented on HBASE-6060: - I can easily(?) repro this on 0.94. > Regions's in OPENING state from failed regionservers takes a long time to > recover > - > > Key: HBASE-6060 > URL: https://issues.apache.org/jira/browse/HBASE-6060 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Affects Versions: 0.94.3 >Reporter: Enis Soztutar >Assignee: Jimmy Xiang > Fix For: 0.96.0 > > Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, > 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, > 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, > 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, > 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, > HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, > HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch, > trunk-6060.patch, trunk-6060_v2.patch, trunk-6060_v3.3.patch > > > we have seen a pattern in tests, that the regions are stuck in OPENING state > for a very long time when the region server who is opening the region fails. > My understanding of the process: > > - master calls rs to open the region. If rs is offline, a new plan is > generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in > master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), > HMaster.assign() > - RegionServer, starts opening a region, changes the state in znode. But > that znode is not ephemeral. (see ZkAssign) > - Rs transitions zk node from OFFLINE to OPENING. See > OpenRegionHandler.process() > - rs then opens the region, and changes znode from OPENING to OPENED > - when rs is killed between OPENING and OPENED states, then zk shows OPENING > state, and the master just waits for rs to change the region state, but since > rs is down, that wont happen. > - There is a AssignmentManager.TimeoutMonitor, which does exactly guard > against these kind of conditions. It periodically checks (every 10 sec by > default) the regions in transition to see whether they timedout > (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, > which explains what you and I are seeing. > - ServerShutdownHandler in Master does not reassign regions in OPENING > state, although it handles other states. > Lowering that threshold from the configuration is one option, but still I > think we can do better. > Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-6060: Affects Version/s: 0.94.3 > Regions's in OPENING state from failed regionservers takes a long time to > recover > - > > Key: HBASE-6060 > URL: https://issues.apache.org/jira/browse/HBASE-6060 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Affects Versions: 0.94.3 >Reporter: Enis Soztutar >Assignee: Jimmy Xiang > Fix For: 0.96.0 > > Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, > 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, > 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, > 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, > 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, > HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, > HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch, > trunk-6060.patch, trunk-6060_v2.patch, trunk-6060_v3.3.patch > > > we have seen a pattern in tests, that the regions are stuck in OPENING state > for a very long time when the region server who is opening the region fails. > My understanding of the process: > > - master calls rs to open the region. If rs is offline, a new plan is > generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in > master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), > HMaster.assign() > - RegionServer, starts opening a region, changes the state in znode. But > that znode is not ephemeral. (see ZkAssign) > - Rs transitions zk node from OFFLINE to OPENING. See > OpenRegionHandler.process() > - rs then opens the region, and changes znode from OPENING to OPENED > - when rs is killed between OPENING and OPENED states, then zk shows OPENING > state, and the master just waits for rs to change the region state, but since > rs is down, that wont happen. > - There is a AssignmentManager.TimeoutMonitor, which does exactly guard > against these kind of conditions. It periodically checks (every 10 sec by > default) the regions in transition to see whether they timedout > (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, > which explains what you and I are seeing. > - ServerShutdownHandler in Master does not reassign regions in OPENING > state, although it handles other states. > Lowering that threshold from the configuration is one option, but still I > think we can do better. > Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Remove Writable Interface from KeyValue
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507728#comment-13507728 ] Lars Hofhansl commented on HBASE-7233: -- [~mcorgan] Came here to write that... KVs are still used in HBase M/R, and a KV cannot currently be serialized by protobufs by itself (Mutation currently serialize the data as columns, not as contained KVs). At the very least we need to add a KV serializer for M/R. > Remove Writable Interface from KeyValue > --- > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: 7233.txt, 7233-v2.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Muraru updated HBASE-6580: - Attachment: HBASE-6580_v2.patch Added HConnection#getTable() > New HTable pool, based on HBase(byte[], HConnection, ExecutorService) > constructor > - > > Key: HBASE-6580 > URL: https://issues.apache.org/jira/browse/HBASE-6580 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.2, 0.94.2 >Reporter: Lars Hofhansl >Priority: Minor > Attachments: HBASE-6580_v1.patch, HBASE-6580_v2.patch > > > Here I propose a very simple TablePool. > It could be called LightHTablePool (or something - if you have a better name). > Internally it would maintain an HConnection and an Executor service and each > invocation of getTable(...) would create a new HTable and close() would just > close it. > In testing I find this more light weight than HTablePool and easier to > monitor in terms of resources used. > It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Muraru updated HBASE-6580: - Attachment: HBASE_v2.patch Added HConnection#getTable() > New HTable pool, based on HBase(byte[], HConnection, ExecutorService) > constructor > - > > Key: HBASE-6580 > URL: https://issues.apache.org/jira/browse/HBASE-6580 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.2, 0.94.2 >Reporter: Lars Hofhansl >Priority: Minor > Attachments: HBASE-6580_v1.patch > > > Here I propose a very simple TablePool. > It could be called LightHTablePool (or something - if you have a better name). > Internally it would maintain an HConnection and an Executor service and each > invocation of getTable(...) would create a new HTable and close() would just > close it. > In testing I find this more light weight than HTablePool and easier to > monitor in terms of resources used. > It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Muraru updated HBASE-6580: - Attachment: (was: HBASE_v2.patch) > New HTable pool, based on HBase(byte[], HConnection, ExecutorService) > constructor > - > > Key: HBASE-6580 > URL: https://issues.apache.org/jira/browse/HBASE-6580 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.2, 0.94.2 >Reporter: Lars Hofhansl >Priority: Minor > Attachments: HBASE-6580_v1.patch > > > Here I propose a very simple TablePool. > It could be called LightHTablePool (or something - if you have a better name). > Internally it would maintain an HConnection and an Executor service and each > invocation of getTable(...) would create a new HTable and close() would just > close it. > In testing I find this more light weight than HTablePool and easier to > monitor in terms of resources used. > It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7204: --- Resolution: Fixed Fix Version/s: 0.96.0 0.94.3 Status: Resolved (was: Patch Available) Ran tests locally. Integrated into trunk and 0.94. > Make hbck ErrorReporter pluggable > - > > Key: HBASE-7204 > URL: https://issues.apache.org/jira/browse/HBASE-7204 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: 0.94-7204.patch, trunk-7204.patch, > trunk-7204_v2.1.patch, trunk-7204_v2.patch > > > Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7249) add test name filter to IntegrationTestsDriver
[ https://issues.apache.org/jira/browse/HBASE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7249: Fix Version/s: 0.96.0 0.94.3 > add test name filter to IntegrationTestsDriver > -- > > Key: HBASE-7249 > URL: https://issues.apache.org/jira/browse/HBASE-7249 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.3, 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-7249-v0-094.patch, HBASE-7249-v0.patch > > > As the number of tests grows, one might want to just run one, or a subset -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7249) add test name filter to IntegrationTestsDriver
[ https://issues.apache.org/jira/browse/HBASE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7249: Affects Version/s: 0.96.0 0.94.3 > add test name filter to IntegrationTestsDriver > -- > > Key: HBASE-7249 > URL: https://issues.apache.org/jira/browse/HBASE-7249 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.3, 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-7249-v0-094.patch, HBASE-7249-v0.patch > > > As the number of tests grows, one might want to just run one, or a subset -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7249) add test name filter to IntegrationTestsDriver
[ https://issues.apache.org/jira/browse/HBASE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7249: Attachment: HBASE-7249-v0-094.patch > add test name filter to IntegrationTestsDriver > -- > > Key: HBASE-7249 > URL: https://issues.apache.org/jira/browse/HBASE-7249 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-7249-v0-094.patch, HBASE-7249-v0.patch > > > As the number of tests grows, one might want to just run one, or a subset -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7249) add test name filter to IntegrationTestsDriver
[ https://issues.apache.org/jira/browse/HBASE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507690#comment-13507690 ] Sergey Shelukhin commented on HBASE-7249: - 94 patch coming momentarily > add test name filter to IntegrationTestsDriver > -- > > Key: HBASE-7249 > URL: https://issues.apache.org/jira/browse/HBASE-7249 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-7249-v0.patch > > > As the number of tests grows, one might want to just run one, or a subset -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7249) add test name filter to IntegrationTestsDriver
[ https://issues.apache.org/jira/browse/HBASE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7249: Status: Patch Available (was: Open) > add test name filter to IntegrationTestsDriver > -- > > Key: HBASE-7249 > URL: https://issues.apache.org/jira/browse/HBASE-7249 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-7249-v0.patch > > > As the number of tests grows, one might want to just run one, or a subset -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7249) add test name filter to IntegrationTestsDriver
[ https://issues.apache.org/jira/browse/HBASE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7249: Attachment: HBASE-7249-v0.patch > add test name filter to IntegrationTestsDriver > -- > > Key: HBASE-7249 > URL: https://issues.apache.org/jira/browse/HBASE-7249 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-7249-v0.patch > > > As the number of tests grows, one might want to just run one, or a subset -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7245) Recovery on failed restore.
[ https://issues.apache.org/jira/browse/HBASE-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507680#comment-13507680 ] Ted Yu commented on HBASE-7245: --- +1 on option 1. > Recovery on failed restore. > --- > > Key: HBASE-7245 > URL: https://issues.apache.org/jira/browse/HBASE-7245 > Project: HBase > Issue Type: Sub-task > Components: Client, master, regionserver, snapshots, Zookeeper >Reporter: Jonathan Hsieh > Fix For: hbase-6055, 0.96.0 > > > Restore will do updates to the file system and to meta. it seems that an > inopportune failure before meta is completely updated could result in an > inconsistent state that would require hbck to fix. > We should define what the semantics are for recovering from this. Some > suggestions: > 1) Fail Forward (see some log saying restore's meta edits not completed, then > gather information necessary to build it all from fs, and complete meta > edits.). > 2) Fail backwards (see some log saying restore's meta edits not completed, > delete incomplete snapshot region entries from meta.) > I think I prefer 1 -- if two processes end somehow updating (somehow the > original master didn't die, and a new one started up) they would be > idempotent. If we used 2, we could still have a race and still be in a bad > place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7249) add test name filter to IntegrationTestsDriver
Sergey Shelukhin created HBASE-7249: --- Summary: add test name filter to IntegrationTestsDriver Key: HBASE-7249 URL: https://issues.apache.org/jira/browse/HBASE-7249 Project: HBase Issue Type: Improvement Components: test Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor As the number of tests grows, one might want to just run one, or a subset -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507643#comment-13507643 ] Jimmy Xiang commented on HBASE-6423: No, we don't need to do it in TestHRegion. We want TestHRegion to run without the setting. conf is a member variable now. TestHRegionBusyWait has all the tests in TestHRegion and its own tests, with the setting. > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Jimmy Xiang > Attachments: trunk-6423.patch, trunk-6423_v2.1.patch, > trunk-6423_v2.patch, trunk-6423_v3.2.patch, trunk-6423_v3.3.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507639#comment-13507639
]
Adrian Muraru commented on HBASE-6956:
--
There is an ongoing discussion on HBASE-6580 addressing this issue, a broader
scope but would resolve the broken HConnection discussed here.
The agreement there was to deprecate HTablePool and rely on lightweight (read
cheap) HTable (HBASE-4805) retrieved from a new HConnection#getTable()
I have a patch on HBASE-6580 that proposes a new HTableInterfaceFactory
:https://issues.apache.org/jira/secure/attachment/12554094/HBASE-6580_v1.patch
and can be used as a custom factory when creating the HTablePool
{noformat}org.apache.hadoop.hbase.client.HTablePool.HTablePool(Configuration
config, int maxSize, HTableInterfaceFactory tableFactory){noformat}
> Do not return back to HTablePool closed connections
> ---
>
> Key: HBASE-6956
> URL: https://issues.apache.org/jira/browse/HBASE-6956
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.90.6
>Reporter: Igor Yurinok
>
> Sometimes we see a lot of Exception about closed connections:
> {code}
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
> closed
> org.apache.hadoop.hbase.client.ClosedConnectionException:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
> closed
> {code}
> After investigation we assumed that it occurs because closed connection
> returns back into HTablePool.
> For our opinion best solution is check whether the table is closed in method
> HTablePool.putTable and if true don't add it into the queue and release such
> HTableInterface.
> But unfortunatly right now there are no access to HTable#closed field through
> HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507634#comment-13507634
]
Micah Whitacre commented on HBASE-6956:
---
Igor, can you explain how you are detecting the connection is closed? Your
isClosed() method only checks to see if the HTable boolean value has been
closed which is independent of the HConnection.
Additionally why do you avoid trying to close the HConnection by informing the
HConnectionManager that the connection is stale?
The last question is how does this solve the problem of evicting
HTableInterface instances from the pool that are no longer valid? Altering the
HTableInterfaceFactory + HTableInterface implementation only solves the problem
of new instances needed to be created.
> Do not return back to HTablePool closed connections
> ---
>
> Key: HBASE-6956
> URL: https://issues.apache.org/jira/browse/HBASE-6956
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.90.6
>Reporter: Igor Yurinok
>
> Sometimes we see a lot of Exception about closed connections:
> {code}
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
> closed
> org.apache.hadoop.hbase.client.ClosedConnectionException:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
> closed
> {code}
> After investigation we assumed that it occurs because closed connection
> returns back into HTablePool.
> For our opinion best solution is check whether the table is closed in method
> HTablePool.putTable and if true don't add it into the queue and release such
> HTableInterface.
> But unfortunatly right now there are no access to HTable#closed field through
> HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507623#comment-13507623 ] stack commented on HBASE-6423: -- You don't do it in TestHRegion. Should you? You used to set it if unset in previous patch. > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Jimmy Xiang > Attachments: trunk-6423.patch, trunk-6423_v2.1.patch, > trunk-6423_v2.patch, trunk-6423_v3.2.patch, trunk-6423_v3.3.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7247) Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening
[ https://issues.apache.org/jira/browse/HBASE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507621#comment-13507621 ] stack commented on HBASE-7247: -- bq. If one region server is opening a lot of regions, we just need one handler to tickle the opening. This would be a significant change for I'm-not-sure-what-benefit. The zk transactions are region level/scoped and their handling is done at this level in open/close exec handlers. Making it so RS does tracking and updating state for master to read regards CLOSING/OPENING would alter a bunch of code. I'm all for a reexamination of base operations. Stuff is this way because we would have issues where an open would stall for whatever reason... Master would intercept the open, take over the region and give it to someone else to open. More often than not, we'd just fail again for same reason on the new location but the odd time the new re-attempt would succeed. We could give up reattempt and just let everything hinge on whether a region server has a zk lease or not and let ServerShutdownHandler do it all. It'd be a pretty radical difference. Simplify code but also, in an odd case, it might mean we'd fail recover a region (I don't have stats on this). bq. We used to have the 'owernership' issue as Stack mentioned. Now, I think we are fine since AM should have a consistent view of region states. This is a similar type of leap-in-the-dark (smile). I love the notion that AM is now rock solid. It may be given the work expended (it certainly is a million times better) but I'd like us to run w/ the new AM a while in a few productions before making this ruling. Again, if we could undo the OPENING/CLOSING, etc., stuff would be cleaner/simpler (logs would be way less noisy) > Assignment performances decreased by 50% because of > regionserver.OpenRegionHandler#tickleOpening > > > Key: HBASE-7247 > URL: https://issues.apache.org/jira/browse/HBASE-7247 > Project: HBase > Issue Type: Improvement > Components: master, Region Assignment, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal >Priority: Critical > > The regionserver.OpenRegionHandler#tickleOpening updates the region znode as > "Do this so master doesn't timeout this region-in-transition.". > However, on the usual test, this makes the assignment time of 1500 regions > goes from 70s to 100s, that is, we're 50% slower because of this. > More generally, ZooKeper commits to disk all the data update, and this takes > time. Using it to provide a keep alive seems overkill. At the very list, it > could be made asynchronous. > I'm not sure how necessary these updates are required (I need to go deeper in > the internal, feedback welcome), but it seems very important to optimize > this... The trival fix would be to make this optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507612#comment-13507612 ] Jimmy Xiang commented on HBASE-6423: I do it only in the constructor(#TestHRegionBusyWait) now since I made conf a member variable. > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Jimmy Xiang > Attachments: trunk-6423.patch, trunk-6423_v2.1.patch, > trunk-6423_v2.patch, trunk-6423_v3.2.patch, trunk-6423_v3.3.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[
https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507608#comment-13507608
]
stack commented on HBASE-6423:
--
You don't this in your new patch?
{code}
+if (busyWaitDuration != null) {
+ conf.set("hbase.busy.wait.duration", busyWaitDuration);
+}
{code}
Should you be doing the above?
> Writes should not block reads on blocking updates to memstores
> --
>
> Key: HBASE-6423
> URL: https://issues.apache.org/jira/browse/HBASE-6423
> Project: HBase
> Issue Type: Bug
>Reporter: Karthik Ranganathan
>Assignee: Jimmy Xiang
> Attachments: trunk-6423.patch, trunk-6423_v2.1.patch,
> trunk-6423_v2.patch, trunk-6423_v3.2.patch, trunk-6423_v3.3.patch
>
>
> We have a big data use case where we turn off WAL and have a ton of reads and
> writes. We found that:
> 1. flushing a memstore takes a while (GZIP compression)
> 2. incoming writes cause the new memstore to grow in an unbounded fashion
> 3. this triggers blocking memstore updates
> 4. in turn, this causes all the RPC handler threads to block on writes to
> that memstore
> 5. we are not able to read during this time as RPC handlers are blocked
> At a higher level, we should not hold up the RPC threads while blocking
> updates, and we should build in some sort of rate control.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7247) Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening
[ https://issues.apache.org/jira/browse/HBASE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507607#comment-13507607 ] Jimmy Xiang commented on HBASE-7247: If one region server is opening a lot of regions, we just need one handler to tickle the opening. Master just needs to know the region server is still running so that it doesn't time out the assignment. To me, this is an overkill. We can combine this logic and something else, for example, to detect if a regionserver is dead. We used to have the 'owernership' issue as Stack mentioned. Now, I think we are fine since AM should have a consistent view of region states. > Assignment performances decreased by 50% because of > regionserver.OpenRegionHandler#tickleOpening > > > Key: HBASE-7247 > URL: https://issues.apache.org/jira/browse/HBASE-7247 > Project: HBase > Issue Type: Improvement > Components: master, Region Assignment, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal >Priority: Critical > > The regionserver.OpenRegionHandler#tickleOpening updates the region znode as > "Do this so master doesn't timeout this region-in-transition.". > However, on the usual test, this makes the assignment time of 1500 regions > goes from 70s to 100s, that is, we're 50% slower because of this. > More generally, ZooKeper commits to disk all the data update, and this takes > time. Using it to provide a keep alive seems overkill. At the very list, it > could be made asynchronous. > I'm not sure how necessary these updates are required (I need to go deeper in > the internal, feedback welcome), but it seems very important to optimize > this... The trival fix would be to make this optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7247) Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening
[ https://issues.apache.org/jira/browse/HBASE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507598#comment-13507598 ] stack commented on HBASE-7247: -- bq. ...the callback would write a variable that would be tested by tickleOpening, and we're done. And as the test cost is cheap, we can test more often. You mean you'd keep writing the znode, just async? bq. May be the master could ask to the region server if it's ok? So master would ping after every region that is opening? Would this be better/less costly than what we have now? In past, before we moved assign to all-zk, on the region server heartbeat to the master, the RS would include the list of OPENING regions. > Assignment performances decreased by 50% because of > regionserver.OpenRegionHandler#tickleOpening > > > Key: HBASE-7247 > URL: https://issues.apache.org/jira/browse/HBASE-7247 > Project: HBase > Issue Type: Improvement > Components: master, Region Assignment, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal >Priority: Critical > > The regionserver.OpenRegionHandler#tickleOpening updates the region znode as > "Do this so master doesn't timeout this region-in-transition.". > However, on the usual test, this makes the assignment time of 1500 regions > goes from 70s to 100s, that is, we're 50% slower because of this. > More generally, ZooKeper commits to disk all the data update, and this takes > time. Using it to provide a keep alive seems overkill. At the very list, it > could be made asynchronous. > I'm not sure how necessary these updates are required (I need to go deeper in > the internal, feedback welcome), but it seems very important to optimize > this... The trival fix would be to make this optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7236) add per-table/per-cf compaction configuration via metadata
[ https://issues.apache.org/jira/browse/HBASE-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507594#comment-13507594 ] Sergey Shelukhin commented on HBASE-7236: - [~stack] bq. IIUC, you are expanding CompoundConfiguration to do table and 'overrides'? Currently it already has metadata, I am adding overrides. The method name is special only because Java won't allow two method differing only by typename part of the parameter. It's no different than other "add"-s :) bq. How would this help us get to changing configuration on the fly? Looks like it doesn't change our current story. CompoundConfiguration is setup in HRegion or ColumnFamily setup still.. It doesn't (well, other than setting overrides after disabling the table, which is more dynamic than XML config replacement). For XML config, there's HBASE-3909 to solve that problem; for column updates, I'd prefer to make online-alter "bulletproof" - that way we'd get this config to be fully dynamic as a side benefit. Two ideas we had was tracking schema version on master and server, and validating on master what server has, but that's inconvenient right now given how it's stored, and may lead to other race conditions; or using new 2pc-ish barrier mechanism that is being introduced for snapshots, somehow. Regardless, in this JIRA I do not aim to solve this problem... bq. If we start to record metadata on a table, say column types, would we use this mechanism? You mean CompoundConfiguration mechanism, or overrides mechanism? Not overrides (they are intended for config), but as CompoundConfiguration is already used for cf metadata, I don't see why not use it also for table metadata if appropriate. bq. How would 'overrides' be specified in the shell say? (Patch doesn't seem to say) We have means of altering table and column descriptors. Where would 'overrides' go? Similarly to metadata. I am going to rename "CONFIG" to something indicating it's actually user metadata, and use name like CONFIGURATION_OVERRIDES - a separate Ruby hash with values. Didn't want to go thru it before confirmation on the viability of this approach :) bq. bq. however, making it explicit and separate from miscellaneous metadata would be cleaner imho. bq. Can you say more on above? See below. bq. HTableDescriptor and HColumnDescriptor dictionaries are key/value maps that get persisted into the filesystem when changed. We read them them throughout the codebase and we list them in master UIs, etc. Will they blow up under this new use case? HTD and HCD are mostly schema with a little config. This direction would seem to be using these descriptors to add table or column scoped configs. Should we be working to undo schema and config conflation rather than compound it? They may blow up (depending on how much config one wants to override), this is one of the reasons I want to keep it separate, to have finer grained control for things like UI or shell. Do you suggest we keep them even more separate, e.g. separate class/serialization mechanism? It seems it would add complexity, esp. given that some of the existing METADATA is config additions/overrides with custom names, essentially. Can you elaborate on example mechanism (e.g. meta-like table for tables/cfs, different file from tableinfo, ...)? The other thing about the current map is that it's already multi-purpose (see HBASE-7237; in the discussion in the shell issue when I wanted to keep CONFIG separate from system stuff I didn't realize they were still stored in the same map in HTD). Dumping config overrides there in addition to system metadata and user metadata is inelegant imho. Finally, the third thing is that if it's separate we can have stricter validation - e.g. whitelist what can and cannot be overridden, or even validate values. > add per-table/per-cf compaction configuration via metadata > -- > > Key: HBASE-7236 > URL: https://issues.apache.org/jira/browse/HBASE-7236 > Project: HBase > Issue Type: New Feature > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7236-PROTOTYPE.patch, HBASE-7236-PROTOTYPE.patch > > > Regardless of the compaction policy, it makes sense to have separate > configuration for compactions for different tables and column families, as > their access patterns and workloads can be different. In particular, for > tiered compactions that are being ported from 0.89-fb branch it is necessary > to have, to use it properly. > We might want to add support for compaction configuration via metadata on > table/cf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrator
[jira] [Commented] (HBASE-7124) typo in pom.xml with "exlude", no definition of "test.exclude.pattern"
[ https://issues.apache.org/jira/browse/HBASE-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507595#comment-13507595 ] Hudson commented on HBASE-7124: --- Integrated in HBase-0.94 #607 (See [https://builds.apache.org/job/HBase-0.94/607/]) HBASE-7124: typo in pom.xml with "exlude", no definition of "test.exclude.pattern" (Li Ping Zhang) (Revision 1415768) Result = SUCCESS jyates : Files : * /hbase/branches/0.94/pom.xml > typo in pom.xml with "exlude", no definition of "test.exclude.pattern" > -- > > Key: HBASE-7124 > URL: https://issues.apache.org/jira/browse/HBASE-7124 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Li Ping Zhang >Assignee: Li Ping Zhang >Priority: Minor > Labels: patch > Attachments: HBASE-7124-0.94.patch, HBASE-7124-v1.patch > > Original Estimate: 4h > Remaining Estimate: 4h > > There is a typo in pom.xml with "exlude", and there is no definition of > "test.exclude.pattern". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6423: --- Status: Patch Available (was: Open) > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Jimmy Xiang > Attachments: trunk-6423.patch, trunk-6423_v2.1.patch, > trunk-6423_v2.patch, trunk-6423_v3.2.patch, trunk-6423_v3.3.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6423: --- Attachment: trunk-6423_v3.3.patch > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Jimmy Xiang > Attachments: trunk-6423.patch, trunk-6423_v2.1.patch, > trunk-6423_v2.patch, trunk-6423_v3.2.patch, trunk-6423_v3.3.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7234) Remove long-deprecated HServerAddress and HServerInfo Writables
[ https://issues.apache.org/jira/browse/HBASE-7234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507586#comment-13507586 ] Hudson commented on HBASE-7234: --- Integrated in HBase-TRUNK #3583 (See [https://builds.apache.org/job/HBase-TRUNK/3583/]) HBASE-7234 Remove long-deprecated HServerAddress and HServerInfo Writables (Revision 1415766) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HRegionLocation.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HServerAddress.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HServerInfo.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedWithDetailsException.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHServerAddress.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHServerInfo.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/TestTableResource.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java > Remove long-deprecated HServerAddress and HServerInfo Writables > --- > > Key: HBASE-7234 > URL: https://issues.apache.org/jira/browse/HBASE-7234 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7234.txt, 7234v2.txt > > > These classes have been deprecated since 0.92 or before. Remove them. > Remove them too because they are Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6423: --- Status: Open (was: Patch Available) > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Jimmy Xiang > Attachments: trunk-6423.patch, trunk-6423_v2.1.patch, > trunk-6423_v2.patch, trunk-6423_v3.2.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7247) Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening
[ https://issues.apache.org/jira/browse/HBASE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507570#comment-13507570 ] nkeywal commented on HBASE-7247: For the regionserver side, the callback won't be very difficult to do I think: the callback would write a variable that would be tested by tickleOpening, and we're done. And as the test cost is cheap, we can test more often. For master, hum... there are different options. May be the master could ask to the region server if it's ok? This would be done if there's no feedback after a while and if the regionserver is still alive from a zookeeper point if view? This would allow to test multiple regions simultaneously (all the regions of this regionserver). > Assignment performances decreased by 50% because of > regionserver.OpenRegionHandler#tickleOpening > > > Key: HBASE-7247 > URL: https://issues.apache.org/jira/browse/HBASE-7247 > Project: HBase > Issue Type: Improvement > Components: master, Region Assignment, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal >Priority: Critical > > The regionserver.OpenRegionHandler#tickleOpening updates the region znode as > "Do this so master doesn't timeout this region-in-transition.". > However, on the usual test, this makes the assignment time of 1500 regions > goes from 70s to 100s, that is, we're 50% slower because of this. > More generally, ZooKeper commits to disk all the data update, and this takes > time. Using it to provide a keep alive seems overkill. At the very list, it > could be made asynchronous. > I'm not sure how necessary these updates are required (I need to go deeper in > the internal, feedback welcome), but it seems very important to optimize > this... The trival fix would be to make this optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507569#comment-13507569 ] Jimmy Xiang commented on HBASE-6423: The goal is to make busyWaitDuration a parameter for this test and its derived tests. It is okay to make configuration a parameter instead, if it is not changed during the tests. I will take a look and post a new patch. > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Jimmy Xiang > Attachments: trunk-6423.patch, trunk-6423_v2.1.patch, > trunk-6423_v2.patch, trunk-6423_v3.2.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically)
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507560#comment-13507560 ] Sergey Shelukhin commented on HBASE-7055: - Hi. Any comments on patch other than missing Javadoc? If not I can write a summary for now. > port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice > (not configurable by cf or dynamically) > - > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, > HBASE-6371-v3-refactor-only-squashed.patch, > HBASE-6371-v4-refactor-only-squashed.patch, > HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, > HBASE-7055-v1.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7248) Improve the resources usage in assignement
nkeywal created HBASE-7248: -- Summary: Improve the resources usage in assignement Key: HBASE-7248 URL: https://issues.apache.org/jira/browse/HBASE-7248 Project: HBase Issue Type: Improvement Components: master, Region Assignment, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Priority: Critical On top on HBASE-7246 and HBASE-7247, I've changed some parameters: hbase.regionserver.executor.openregion.threads from 3 to 300 hbase.assignment.zkevent.workers from 5 to 30 hbase.assignment.threads.max from 30 to 300 It makes the assignment time going from 70s to 8s. Seems there is something to dig here as well... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6990) Pretty print TTL
[
https://issues.apache.org/jira/browse/HBASE-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507540#comment-13507540
]
Jean-Daniel Cryans commented on HBASE-6990:
---
[~cody.mar...@gmail.com] TimeToLiveHFileCleaner is something completely
separate, the TTL in question here is the one in HColumnDescriptor. It's being
printed as part of {{getValues}}.
> Pretty print TTL
>
>
> Key: HBASE-6990
> URL: https://issues.apache.org/jira/browse/HBASE-6990
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.92.2, 0.94.2
>Reporter: Jean-Daniel Cryans
>Assignee: Kevin Odell
>Priority: Minor
> Fix For: 0.92.3, 0.96.0, 0.94.4
>
>
> I've seen a lot of users getting confused by the TTL configuration and I
> think that if we just pretty printed it it would solve most of the issues.
> For example, let's say a user wanted to set a TTL of 90 days. That would be
> 7776000. But let's say that it was typo'd to 7776 instead, it gives you
> 900 days!
> So when we print the TTL we could do something like "x days, x hours, x
> minutes, x seconds (real_ttl_value)". This would also help people when they
> use ms instead of seconds as they would see really big values in there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7247) Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening
[ https://issues.apache.org/jira/browse/HBASE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507536#comment-13507536 ] stack commented on HBASE-7247: -- Moving znode from OPENING to OPENING or CLOSING to CLOSING ensuring the sequence numbers are as we expect is how we check we still have 'ownership' of the region before we go about making alterations in the filesystem. The callback the master gets when we 'update' the znode is used to flag the master that the region server is still 'alive' and working on the opening so the master updates its opening timer when it gets the callback. It is imperfect in that we could lose the lease between the check and the fs operation. Alternatives would be to remove this mechanism and instead just rely on our getting a callback if the znode is taken from us? This would happen in another thread. Would have to be watching. This would seem to coarser than what we currently have widening the window during which we could do fs operations on a region though we've lost ownership. Master would not get notification that a region server is opening a region only its taking longer than usual. Any other suggestions? > Assignment performances decreased by 50% because of > regionserver.OpenRegionHandler#tickleOpening > > > Key: HBASE-7247 > URL: https://issues.apache.org/jira/browse/HBASE-7247 > Project: HBase > Issue Type: Improvement > Components: master, Region Assignment, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal >Priority: Critical > > The regionserver.OpenRegionHandler#tickleOpening updates the region znode as > "Do this so master doesn't timeout this region-in-transition.". > However, on the usual test, this makes the assignment time of 1500 regions > goes from 70s to 100s, that is, we're 50% slower because of this. > More generally, ZooKeper commits to disk all the data update, and this takes > time. Using it to provide a keep alive seems overkill. At the very list, it > could be made asynchronous. > I'm not sure how necessary these updates are required (I need to go deeper in > the internal, feedback welcome), but it seems very important to optimize > this... The trival fix would be to make this optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6990) Pretty print TTL
[ https://issues.apache.org/jira/browse/HBASE-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507532#comment-13507532 ] Cody Marcel commented on HBASE-6990: Do you know what class this is being logged from? The only place I have found that is logging ttl is TimeToLiveHFileCleaner. > Pretty print TTL > > > Key: HBASE-6990 > URL: https://issues.apache.org/jira/browse/HBASE-6990 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.2, 0.94.2 >Reporter: Jean-Daniel Cryans >Assignee: Kevin Odell >Priority: Minor > Fix For: 0.92.3, 0.96.0, 0.94.4 > > > I've seen a lot of users getting confused by the TTL configuration and I > think that if we just pretty printed it it would solve most of the issues. > For example, let's say a user wanted to set a TTL of 90 days. That would be > 7776000. But let's say that it was typo'd to 7776 instead, it gives you > 900 days! > So when we print the TTL we could do something like "x days, x hours, x > minutes, x seconds (real_ttl_value)". This would also help people when they > use ms instead of seconds as they would see really big values in there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Remove Writable Interface from KeyValue
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507517#comment-13507517 ] Matt Corgan commented on HBASE-7233: Nevermind about the ROOT/META comment. I was thinking the comparator might have an effect here but maybe not. > Remove Writable Interface from KeyValue > --- > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: 7233.txt, 7233-v2.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5069) [book] Document how to count rows
[ https://issues.apache.org/jira/browse/HBASE-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507506#comment-13507506 ] Hudson commented on HBASE-5069: --- Integrated in HBase-TRUNK #3582 (See [https://builds.apache.org/job/HBase-TRUNK/3582/]) HBASE-5069 [book] Document how to count rows (Revision 1415733) Result = FAILURE stack : Files : * /hbase/trunk/src/docbkx/ops_mgt.xml * /hbase/trunk/src/docbkx/shell.xml > [book] Document how to count rows > - > > Key: HBASE-5069 > URL: https://issues.apache.org/jira/browse/HBASE-5069 > Project: HBase > Issue Type: Improvement >Reporter: Jean-Daniel Cryans >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-5069.patch > > > I just saw that there's no documentation on how to count rows. Basically > there's: > - The "count" command in the shell, it's quite fast when configured with the > right CACHE. > - The "rowcounter" MR job, it's super fast but you need to have a MapReduce > setup in place in order to use it. > There's also a way to estimate the row count by using the HFile tool and > doing a bit of bash magic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7124) typo in pom.xml with "exlude", no definition of "test.exclude.pattern"
[ https://issues.apache.org/jira/browse/HBASE-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates resolved HBASE-7124. Resolution: Fixed > typo in pom.xml with "exlude", no definition of "test.exclude.pattern" > -- > > Key: HBASE-7124 > URL: https://issues.apache.org/jira/browse/HBASE-7124 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Li Ping Zhang >Assignee: Li Ping Zhang >Priority: Minor > Labels: patch > Attachments: HBASE-7124-0.94.patch, HBASE-7124-v1.patch > > Original Estimate: 4h > Remaining Estimate: 4h > > There is a typo in pom.xml with "exlude", and there is no definition of > "test.exclude.pattern". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7124) typo in pom.xml with "exlude", no definition of "test.exclude.pattern"
[ https://issues.apache.org/jira/browse/HBASE-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507502#comment-13507502 ] Jesse Yates commented on HBASE-7124: Committed. thanks [~michelle] > typo in pom.xml with "exlude", no definition of "test.exclude.pattern" > -- > > Key: HBASE-7124 > URL: https://issues.apache.org/jira/browse/HBASE-7124 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Li Ping Zhang >Assignee: Li Ping Zhang >Priority: Minor > Labels: patch > Attachments: HBASE-7124-0.94.patch, HBASE-7124-v1.patch > > Original Estimate: 4h > Remaining Estimate: 4h > > There is a typo in pom.xml with "exlude", and there is no definition of > "test.exclude.pattern". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7247) Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening
nkeywal created HBASE-7247: -- Summary: Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening Key: HBASE-7247 URL: https://issues.apache.org/jira/browse/HBASE-7247 Project: HBase Issue Type: Improvement Components: master, Region Assignment, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Critical The regionserver.OpenRegionHandler#tickleOpening updates the region znode as "Do this so master doesn't timeout this region-in-transition.". However, on the usual test, this makes the assignment time of 1500 regions goes from 70s to 100s, that is, we're 50% slower because of this. More generally, ZooKeper commits to disk all the data update, and this takes time. Using it to provide a keep alive seems overkill. At the very list, it could be made asynchronous. I'm not sure how necessary these updates are required (I need to go deeper in the internal, feedback welcome), but it seems very important to optimize this... The trival fix would be to make this optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7234) Remove long-deprecated HServerAddress and HServerInfo Writables
[ https://issues.apache.org/jira/browse/HBASE-7234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7234: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Elliott. Committed to trunk. > Remove long-deprecated HServerAddress and HServerInfo Writables > --- > > Key: HBASE-7234 > URL: https://issues.apache.org/jira/browse/HBASE-7234 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7234.txt, 7234v2.txt > > > These classes have been deprecated since 0.92 or before. Remove them. > Remove them too because they are Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7234) Remove long-deprecated HServerAddress and HServerInfo Writables
[ https://issues.apache.org/jira/browse/HBASE-7234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507486#comment-13507486 ] Elliott Clark commented on HBASE-7234: -- +1 looks great. Gotta love removing old slow code. > Remove long-deprecated HServerAddress and HServerInfo Writables > --- > > Key: HBASE-7234 > URL: https://issues.apache.org/jira/browse/HBASE-7234 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7234.txt, 7234v2.txt > > > These classes have been deprecated since 0.92 or before. Remove them. > Remove them too because they are Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7234) Remove long-deprecated HServerAddress and HServerInfo Writables
[ https://issues.apache.org/jira/browse/HBASE-7234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507485#comment-13507485 ] stack commented on HBASE-7234: -- OK if I commit this? Mostly just removals. The above test failure passes when I try it locally. > Remove long-deprecated HServerAddress and HServerInfo Writables > --- > > Key: HBASE-7234 > URL: https://issues.apache.org/jira/browse/HBASE-7234 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7234.txt, 7234v2.txt > > > These classes have been deprecated since 0.92 or before. Remove them. > Remove them too because they are Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7246) Assignment#nodeChildrenChanged calls listChildrenAndWatchForNewChildren, overloading master & zookeper needlessly
nkeywal created HBASE-7246: -- Summary: Assignment#nodeChildrenChanged calls listChildrenAndWatchForNewChildren, overloading master & zookeper needlessly Key: HBASE-7246 URL: https://issues.apache.org/jira/browse/HBASE-7246 Project: HBase Issue Type: Improvement Components: master, Region Assignment Affects Versions: 0.96.0 Reporter: nkeywal Assignment#nodeChildrenChanged listens to change event on hbase/unassigned, then calls listChildrenAndWatchForNewChildren. As its name says, this function gets the whole list of unnasigned regions from ZK. The internal comments says it's only for split creation. I tried to remove it, it seems to work. The point is: - ZK does not allow us to know upfront if it's a split or not - 50% of the events will be deletions, we don't care about them but we can't know in advance - for large assignment, we will have a lot of children (hence events). Common practise in ZK is to limit the number of subnodes (see the threads http://www.mail-archive.com/zookeeper-user@hadoop.apache.org/msg01954.html) Monitoring the master, it was spending 10% of it's time on this during assignment. It seems it was not on the critical path (yet...), so removing it did not change the overall performances on a test scenario. In real life however, this typically adds workload to ZK, and ZK workload is precious on large failures... Same as master actually. If it's used only for split, we should have a znode specific for split and not a global znode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7234) Remove long-deprecated HServerAddress and HServerInfo Writables
[
https://issues.apache.org/jira/browse/HBASE-7234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507467#comment-13507467
]
Hadoop QA commented on HBASE-7234:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1227/7234v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 31 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 26 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitTransaction
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3427//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3427//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3427//console
This message is automatically generated.
> Remove long-deprecated HServerAddress and HServerInfo Writables
> ---
>
> Key: HBASE-7234
> URL: https://issues.apache.org/jira/browse/HBASE-7234
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 0.96.0
>
> Attachments: 7234.txt, 7234v2.txt
>
>
> These classes have been deprecated since 0.92 or before. Remove them.
> Remove them too because they are Writable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7234) Remove long-deprecated HServerAddress and HServerInfo Writables
[ https://issues.apache.org/jira/browse/HBASE-7234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7234: - Attachment: 7234v2.txt Rebase > Remove long-deprecated HServerAddress and HServerInfo Writables > --- > > Key: HBASE-7234 > URL: https://issues.apache.org/jira/browse/HBASE-7234 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7234.txt, 7234v2.txt > > > These classes have been deprecated since 0.92 or before. Remove them. > Remove them too because they are Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7219) Make it so can connect remotely to a standalone hbase
[ https://issues.apache.org/jira/browse/HBASE-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507450#comment-13507450 ] Jean-Daniel Cryans commented on HBASE-7219: --- bq. I was thinking about checking if we write 127.0.0.1 or localhost as the server address in ZK Then alert somehow that the cluster isn't reachable remotely? Sounds good to me. > Make it so can connect remotely to a standalone hbase > - > > Key: HBASE-7219 > URL: https://issues.apache.org/jira/browse/HBASE-7219 > Project: HBase > Issue Type: Bug >Reporter: stack > > Should be able to connect from a remote client to a standalone instance. > HBase has 'localhost' in regionservers file and will write 'localhost' to > znode for master location which remote client can't use. Fix. This comes up > on mailing list w/ some frequency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Remove Writable Interface from KeyValue
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507441#comment-13507441 ] stack commented on HBASE-7233: -- [~mcorgan] How you mean (re: -ROOT- and .META.)? Should be ok given we are protobuf serializing here already. This patch comes after that work. Let me check KeyValueSortReducer. I don't see us specifying a Serializer on cursory glance. Will run the unit test. [~ted_yu] Thanks. > Remove Writable Interface from KeyValue > --- > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: 7233.txt, 7233-v2.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7245) Recovery on failed restore.
Jonathan Hsieh created HBASE-7245: - Summary: Recovery on failed restore. Key: HBASE-7245 URL: https://issues.apache.org/jira/browse/HBASE-7245 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Restore will do updates to the file system and to meta. it seems that an inopportune failure before meta is completely updated could result in an inconsistent state that would require hbck to fix. We should define what the semantics are for recovering from this. Some suggestions: 1) Fail Forward (see some log saying restore's meta edits not completed, then gather information necessary to build it all from fs, and complete meta edits.). 2) Fail backwards (see some log saying restore's meta edits not completed, delete incomplete snapshot region entries from meta.) I think I prefer 1 -- if two processes end somehow updating (somehow the original master didn't die, and a new one started up) they would be idempotent. If we used 2, we could still have a race and still be in a bad place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7155) Excessive usage of InterruptedException where it can't be thrown
[ https://issues.apache.org/jira/browse/HBASE-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7155: - Resolution: Won't Fix Status: Resolved (was: Patch Available) This is a problem but too much ugly work to fix. Thanks Daniel for bringing it up. > Excessive usage of InterruptedException where it can't be thrown > > > Key: HBASE-7155 > URL: https://issues.apache.org/jira/browse/HBASE-7155 > Project: HBase > Issue Type: Bug >Reporter: Daniel Gómez Ferro >Assignee: Daniel Gómez Ferro > Attachments: HBASE-7155.patch > > > RootRegionTracker.getRootRegionLocation() declares that it can throw an > InterruptedException, but it can't. This exception is rethrown by many other > functions reaching the HBaseAdmin API. > If we remove the throws statement from the HBaseAdmin API libraries already > compiled will work fine, but if the user is trying to catch an > InterruptedException around one of those methods the compiler will complain. > Should we clean this up? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5069) [book] Document how to count rows
[ https://issues.apache.org/jira/browse/HBASE-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-5069. -- Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Committed to trunk. Thank you for the doc Daisuke. > [book] Document how to count rows > - > > Key: HBASE-5069 > URL: https://issues.apache.org/jira/browse/HBASE-5069 > Project: HBase > Issue Type: Improvement >Reporter: Jean-Daniel Cryans >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-5069.patch > > > I just saw that there's no documentation on how to count rows. Basically > there's: > - The "count" command in the shell, it's quite fast when configured with the > right CACHE. > - The "rowcounter" MR job, it's super fast but you need to have a MapReduce > setup in place in order to use it. > There's also a way to estimate the row count by using the HFile tool and > doing a bit of bash magic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7155) Excessive usage of InterruptedException where it can't be thrown
[ https://issues.apache.org/jira/browse/HBASE-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507422#comment-13507422 ] Daniel Gómez Ferro commented on HBASE-7155: --- [~stack] I guess we can just close this ticket, no? > Excessive usage of InterruptedException where it can't be thrown > > > Key: HBASE-7155 > URL: https://issues.apache.org/jira/browse/HBASE-7155 > Project: HBase > Issue Type: Bug >Reporter: Daniel Gómez Ferro >Assignee: Daniel Gómez Ferro > Attachments: HBASE-7155.patch > > > RootRegionTracker.getRootRegionLocation() declares that it can throw an > InterruptedException, but it can't. This exception is rethrown by many other > functions reaching the HBaseAdmin API. > If we remove the throws statement from the HBaseAdmin API libraries already > compiled will work fine, but if the user is trying to catch an > InterruptedException around one of those methods the compiler will complain. > Should we clean this up? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507413#comment-13507413
]
stack commented on HBASE-6956:
--
This seems like an ugly enough oversight on our part. Any chance of a patch to
address it? Could we change HConnection Interface to make it easier getting a
new one [~amuraru]? (I like the name of your class [~iyurinok])
> Do not return back to HTablePool closed connections
> ---
>
> Key: HBASE-6956
> URL: https://issues.apache.org/jira/browse/HBASE-6956
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.90.6
>Reporter: Igor Yurinok
>
> Sometimes we see a lot of Exception about closed connections:
> {code}
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
> closed
> org.apache.hadoop.hbase.client.ClosedConnectionException:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
> closed
> {code}
> After investigation we assumed that it occurs because closed connection
> returns back into HTablePool.
> For our opinion best solution is check whether the table is closed in method
> HTablePool.putTable and if true don't add it into the queue and release such
> HTableInterface.
> But unfortunatly right now there are no access to HTable#closed field through
> HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3375) Move away from jruby; build our shell elsewise either on another foundation or build up our own
[ https://issues.apache.org/jira/browse/HBASE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-3375. -- Resolution: Won't Fix As per [~vicaya], JRuby fixed the licensing so we just stuck w/ jruby. > Move away from jruby; build our shell elsewise either on another foundation > or build up our own > --- > > Key: HBASE-3375 > URL: https://issues.apache.org/jira/browse/HBASE-3375 > Project: HBase > Issue Type: Task > Components: shell >Reporter: stack > > JRuby has been sullied; its been shipping *GPL jars with a while now. A hack > up to remove these jars is being done elsewhere (HBASE-3374). This issue is > about casting our shell anew atop a foundation that is other than JRuby or > writing a shell of our own from scratch. > JRuby has gotten us this far. It provides a shell and it also was used > scripting HBase. It would be nice if we could get scripting and shell in the > redo. > Apart from the licensing issue above and that the fix will be reverting our > JRuby to a version that is no longer supported and that is old, other reasons > to move off JRuby are that while its nice having ruby to hand when scripting, > the JRuby complete jar is 10 or more MB in size. Its bloated at least from > our small shell perspective. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507353#comment-13507353
]
Igor Yurinok commented on HBASE-6956:
-
We implemented our own HTableFactory and HTable which able to check whether
connection closed:
{code}
public class HumbleHTableFactory implements HTableInterfaceFactory {
@Override
public HTableInterface createHTableInterface(Configuration config, byte[]
tableName) {
try {
return new HumbleHTable(config, tableName);
} catch (IOException ioe) {
throw new RuntimeException(ioe);
}
}
@Override
public void releaseHTableInterface(HTableInterface table) {
try {
table.close();
} catch (IOException ioe) {
throw new RuntimeException(ioe);
}
}
}
{code}
{code}
public class HumbleHTable extends HTable {
private static final Log log = LogFactory.getLog(HumbleHTable.class);
public HumbleHTable(String tableName) throws IOException {
super(tableName);
}
public HumbleHTable(byte[] tableName) throws IOException {
super(tableName);
}
public HumbleHTable(Configuration conf, String tableName) throws
IOException {
super(conf, tableName);
}
public HumbleHTable(Configuration conf, byte[] tableName) throws
IOException {
super(conf, tableName);
}
/**
* Harmless clean-up - HConnection isn't touched. Only the executor pool is
shut down
*
* @throws IOException
*/
@Override
public void close() throws IOException {
if (isClosed()) {
return;
}
flushCommits();
ExecutorService pool = getPool();
if (pool != null) {
pool.shutdown();
}
setClosed();
}
private boolean isClosed() {
try {
return getSuperField("closed").getBoolean(this);
} catch (Exception e) {
log.error(e);
return false;
}
}
private void setClosed() {
try {
getSuperField("closed").setBoolean(this, true);
} catch (Exception e) {
log.error(e);
}
}
private ExecutorService getPool() {
try {
return (ExecutorService) getSuperField("pool").get(this);
} catch (Exception e) {
log.error(e);
return null;
}
}
/**
* Loads stuff from the parent class via reflection
*/
private Field getSuperField(String fieldName) throws NoSuchFieldException {
Field field = HTable.class.getDeclaredField(fieldName);
field.setAccessible(true);
return field;
}
}
{code}
> Do not return back to HTablePool closed connections
> ---
>
> Key: HBASE-6956
> URL: https://issues.apache.org/jira/browse/HBASE-6956
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.90.6
>Reporter: Igor Yurinok
>
> Sometimes we see a lot of Exception about closed connections:
> {code}
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
> closed
> org.apache.hadoop.hbase.client.ClosedConnectionException:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
> closed
> {code}
> After investigation we assumed that it occurs because closed connection
> returns back into HTablePool.
> For our opinion best solution is check whether the table is closed in method
> HTablePool.putTable and if true don't add it into the queue and release such
> HTableInterface.
> But unfortunatly right now there are no access to HTable#closed field through
> HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3375) Move away from jruby; build our shell elsewise either on another foundation or build up our own
[ https://issues.apache.org/jira/browse/HBASE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507343#comment-13507343 ] Luke Lu commented on HBASE-3375: License issue of jruby runtime jars has been fixed: http://jira.codehaus.org/browse/JRUBY-5410. Let's close this as won't fix :) > Move away from jruby; build our shell elsewise either on another foundation > or build up our own > --- > > Key: HBASE-3375 > URL: https://issues.apache.org/jira/browse/HBASE-3375 > Project: HBase > Issue Type: Task > Components: shell >Reporter: stack > > JRuby has been sullied; its been shipping *GPL jars with a while now. A hack > up to remove these jars is being done elsewhere (HBASE-3374). This issue is > about casting our shell anew atop a foundation that is other than JRuby or > writing a shell of our own from scratch. > JRuby has gotten us this far. It provides a shell and it also was used > scripting HBase. It would be nice if we could get scripting and shell in the > redo. > Apart from the licensing issue above and that the fix will be reverting our > JRuby to a version that is no longer supported and that is old, other reasons > to move off JRuby are that while its nice having ruby to hand when scripting, > the JRuby complete jar is 10 or more MB in size. Its bloated at least from > our small shell perspective. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507345#comment-13507345
]
Bryan Baugher commented on HBASE-6956:
--
We have also ran into this issue specifically during major failure of the
cluster (i.e. hmaster, zookeeper, regionservers are down).
[~amuraru] did you manage to find a good way to flush a bad table from your
pool when the connection becomes closed?
> Do not return back to HTablePool closed connections
> ---
>
> Key: HBASE-6956
> URL: https://issues.apache.org/jira/browse/HBASE-6956
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.90.6
>Reporter: Igor Yurinok
>
> Sometimes we see a lot of Exception about closed connections:
> {code}
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
> closed
> org.apache.hadoop.hbase.client.ClosedConnectionException:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
> closed
> {code}
> After investigation we assumed that it occurs because closed connection
> returns back into HTablePool.
> For our opinion best solution is check whether the table is closed in method
> HTablePool.putTable and if true don't add it into the queue and release such
> HTableInterface.
> But unfortunatly right now there are no access to HTable#closed field through
> HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7232) Remove HbaseMapWritable
[ https://issues.apache.org/jira/browse/HBASE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507279#comment-13507279 ] Hudson commented on HBASE-7232: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #281 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/281/]) HBASE-7232 Remove HbaseMapWritable (Revision 1415507) Result = FAILURE stack : Files : * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HFileProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/HFile.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HbaseMapWritable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/InlineBlockWriter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilterWriter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestSerialization.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java > Remove HbaseMapWritable > --- > > Key: HBASE-7232 > URL: https://issues.apache.org/jira/browse/HBASE-7232 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 0.96.0 > > Attachments: 7232.txt, 7232.txt, 7232v2.txt, 7232v3.txt, 7232v4.txt > > > Its used by hfile fileinfo only so need to convert fileinfo to remove this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7243) Test for creating a large number of regions
[ https://issues.apache.org/jira/browse/HBASE-7243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507251#comment-13507251 ] nkeywal commented on HBASE-7243: +1. You don't mention it, but it should be be an integration test, no? > Test for creating a large number of regions > --- > > Key: HBASE-7243 > URL: https://issues.apache.org/jira/browse/HBASE-7243 > Project: HBase > Issue Type: Bug > Components: Region Assignment, regionserver, test >Reporter: Enis Soztutar > Labels: noob > Fix For: 0.96.0 > > > After HBASE-7220, I think it will be good to write a unit test/IT to create a > large number of regions. We can put a reasonable timeout to the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507250#comment-13507250 ] nkeywal commented on HBASE-7213: To me, there are some stuff that are nice to have in multiwal but very useful here: - it's better to have only one table in the wal with ".meta." - it's would be great to be able to configure the replication factor for this wal - having a separate wal for meta should be the only option (it could make sense short term to have it as an option for safety, but it would be temporary) - there is no real split of the .meta. wal (at least conceptually: all regions ends up on the same server..). There is some room for (premature?) optimization there. > Have HLog files for .META. edits only > - > > Key: HBASE-7213 > URL: https://issues.apache.org/jira/browse/HBASE-7213 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > Attachments: 7213-in-progress.patch > > > Over on HBASE-6774, there is a discussion on separating out the edits for > .META. regions from the other regions' edits w.r.t where the edits are > written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5069) [book] Document how to count rows
[ https://issues.apache.org/jira/browse/HBASE-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daisuke Kobayashi updated HBASE-5069: - Attachment: HBASE-5069.patch Can you check the format and my English because it isn't my primary language. > [book] Document how to count rows > - > > Key: HBASE-5069 > URL: https://issues.apache.org/jira/browse/HBASE-5069 > Project: HBase > Issue Type: Improvement >Reporter: Jean-Daniel Cryans >Priority: Minor > Attachments: HBASE-5069.patch > > > I just saw that there's no documentation on how to count rows. Basically > there's: > - The "count" command in the shell, it's quite fast when configured with the > right CACHE. > - The "rowcounter" MR job, it's super fast but you need to have a MapReduce > setup in place in order to use it. > There's also a way to estimate the row count by using the HFile tool and > doing a bit of bash magic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5877) When a query fails because the region has moved, let the regionserver return the new address to the client
[ https://issues.apache.org/jira/browse/HBASE-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507245#comment-13507245 ] nkeywal commented on HBASE-5877: @jd Yep, you're right. > When a query fails because the region has moved, let the regionserver return > the new address to the client > -- > > Key: HBASE-5877 > URL: https://issues.apache.org/jira/browse/HBASE-5877 > Project: HBase > Issue Type: Improvement > Components: Client, master, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal >Priority: Minor > Fix For: 0.96.0 > > Attachments: 5877.v12.patch, 5877.v15.patch, 5877-v16.txt, > 5877-v17.txt, 5877-v17.txt, 5877.v18.patch, 5877.v18.patch, 5877.v18.patch, > 5877.v1.patch, 5877.v6.patch > > > This is mainly useful when we do a rolling restart. This will decrease the > load on the master and the network load. > Note that a region is not immediately opened after a close. So: > - it seems preferable to wait before retrying on the other server. An > optimisation would be to have an heuristic depending on when the region was > closed. > - during a rolling restart, the server moves the regions then stops. So we > may have failures when the server is stopped, and this patch won't help. > The implementation in the first patch does: > - on the region move, there is an added parameter on the regionserver#close > to say where we are sending the region > - the regionserver keeps a list of what was moved. Each entry is kept 100 > seconds. > - the regionserver sends a specific exception when it receives a query on a > moved region. This exception contains the new address. > - the client analyses the exeptions and update its cache accordingly... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7219) Make it so can connect remotely to a standalone hbase
[ https://issues.apache.org/jira/browse/HBASE-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507242#comment-13507242 ] nkeywal commented on HBASE-7219: When it was discussed on the mailing list, I was thinking about checking if we write 127.0.0.1 or localhost as the server address in ZK, i.e. something that would work only on the local machine. > Make it so can connect remotely to a standalone hbase > - > > Key: HBASE-7219 > URL: https://issues.apache.org/jira/browse/HBASE-7219 > Project: HBase > Issue Type: Bug >Reporter: stack > > Should be able to connect from a remote client to a standalone instance. > HBase has 'localhost' in regionservers file and will write 'localhost' to > znode for master location which remote client can't use. Fix. This comes up > on mailing list w/ some frequency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7233) Remove Writable Interface from KeyValue
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7233: -- Attachment: 7233-v2.txt Change in TestSerialization.java didn't apply cleanly. Attached diff which fixes the above. > Remove Writable Interface from KeyValue > --- > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: 7233.txt, 7233-v2.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
