[jira] [Commented] (HBASE-5143) Fix config typo in pluggable load balancer factory

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182214#comment-13182214
 ] 

Hadoop QA commented on HBASE-5143:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12509839/HBASE-5143.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 javadoc.  The javadoc tool appears to have generated -151 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 79 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.mapreduce.TestImportTsv
  org.apache.hadoop.hbase.mapred.TestTableMapReduce
  org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/701//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/701//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/701//console

This message is automatically generated.

> Fix config typo in pluggable load balancer factory
> --
>
> Key: HBASE-5143
> URL: https://issues.apache.org/jira/browse/HBASE-5143
> Project: HBase
>  Issue Type: Sub-task
>  Components: master
>Reporter: Harsh J
>Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5143.patch
>
>
> HBASE-4240 made LoadBalancer pluggable.
> Configuration it loads seems to be wrongly named and carries a typo: 
> "hbase.maser.loadBalancer.class"
> Could rather be "hbase.master.loadbalancer.class"
> Luckily 0.92 is not out yet and we should fix it asap, before folks start 
> using it. Attaching patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5052) The path where a dynamically loaded coprocessor jar is copied on the local file system depends on the region name (and implicitly, the start key)

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182216#comment-13182216
 ] 

Hudson commented on HBASE-5052:
---

Integrated in HBase-0.92-security #67 (See 
[https://builds.apache.org/job/HBase-0.92-security/67/])
HBASE-5052 The path where a dynamically loaded coprocessor jar is copied on 
the local file system depends on the region name (and implicitly, the start key)

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java


> The path where a dynamically loaded coprocessor jar is copied on the local 
> file system depends on the region name (and implicitly, the start key)
> -
>
> Key: HBASE-5052
> URL: https://issues.apache.org/jira/browse/HBASE-5052
> Project: HBase
>  Issue Type: Bug
>  Components: coprocessors
>Affects Versions: 0.92.0
>Reporter: Andrei Dragomir
>Assignee: Andrei Dragomir
> Fix For: 0.92.0
>
> Attachments: HBASE-5052.patch
>
>
> When loading a coprocessor from hdfs, the jar file gets copied to a path on 
> the local filesystem, which depends on the region name, and the region start 
> key. The name is "cleaned", but not enough, so when you have filesystem 
> unfriendly characters (/?:, etc), the coprocessor is not loaded, and an error 
> is thrown

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5141) Memory leak in MonitoredRPCHandlerImpl

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182215#comment-13182215
 ] 

Hudson commented on HBASE-5141:
---

Integrated in HBase-0.92-security #67 (See 
[https://builds.apache.org/job/HBase-0.92-security/67/])
HBASE-5141 Memory leak in MonitoredRPCHandlerImpl

stack : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java


> Memory leak in MonitoredRPCHandlerImpl
> --
>
> Key: HBASE-5141
> URL: https://issues.apache.org/jira/browse/HBASE-5141
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
>Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5141-v2.patch, HBASE-5141.patch, Screen Shot 
> 2012-01-06 at 3.03.09 PM.png
>
>
> I got a pretty reliable way of OOME'ing my region servers. Using a big 
> payload (64MB in my case), a default heap and default number of handlers, 
> it's not too long that all the MonitoredRPCHandlerImpl hold on a 64MB 
> reference and once a compaction kicks in it kills everything.
> The issue is that even after the RPC call is done, the packet still lives in 
> MonitoredRPCHandlerImpl.
> Will attach a screen shot of jprofiler's analysis in a moment.
> This is a blocker for 0.92.0, anyone using a high number of handlers and 
> bigish values will kill themselves.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5088) A concurrency issue on SoftValueSortedMap

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182217#comment-13182217
 ] 

Hudson commented on HBASE-5088:
---

Integrated in HBase-0.92-security #67 (See 
[https://builds.apache.org/job/HBase-0.92-security/67/])
HBASE-5088  addendum

larsh : 
Files : 
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/SoftValueSortedMap.java


> A concurrency issue on SoftValueSortedMap
> -
>
> Key: HBASE-5088
> URL: https://issues.apache.org/jira/browse/HBASE-5088
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 0.92.0, 0.94.0, 0.90.6
>
> Attachments: 5088-0.90.txt, 5088-0.92-trunk-addendum.txt, 
> 5088-final3.txt, HBase-5088-90.patch, HBase-5088-trunk.patch, 
> HBase5088-90-replaceSoftValueSortedMap.patch, 
> HBase5088-90-replaceTreeMap.patch, HBase5088-trunk-replaceTreeMap.patch, 
> HBase5088Reproduce.java, PerformanceTestResults.png
>
>
> SoftValueSortedMap is backed by a TreeMap. All the methods in this class are 
> synchronized. If we use this method to add/delete elements, it's ok.
> But in HConnectionManager#getCachedLocation, it use headMap to get a view 
> from SoftValueSortedMap#internalMap. Once we operate 
> on this view map(like add/delete) in other threads, a concurrency issue may 
> occur.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5143) Fix config typo in pluggable load balancer factory

[Harsh J (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182219#comment-13182219
 ] 

Harsh J commented on HBASE-5143:


Test failures appear unrelated. Also, an {{ack/grep}} of the old config string 
yields no result across code base.

> Fix config typo in pluggable load balancer factory
> --
>
> Key: HBASE-5143
> URL: https://issues.apache.org/jira/browse/HBASE-5143
> Project: HBase
>  Issue Type: Sub-task
>  Components: master
>Reporter: Harsh J
>Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5143.patch
>
>
> HBASE-4240 made LoadBalancer pluggable.
> Configuration it loads seems to be wrongly named and carries a typo: 
> "hbase.maser.loadBalancer.class"
> Could rather be "hbase.master.loadbalancer.class"
> Luckily 0.92 is not out yet and we should fix it asap, before folks start 
> using it. Attaching patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5143) Fix config typo in pluggable load balancer factory

[Harsh J (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated HBASE-5143:
---

Priority: Blocker  (was: Critical)

> Fix config typo in pluggable load balancer factory
> --
>
> Key: HBASE-5143
> URL: https://issues.apache.org/jira/browse/HBASE-5143
> Project: HBase
>  Issue Type: Sub-task
>  Components: master
>Reporter: Harsh J
>Priority: Blocker
> Fix For: 0.92.0, 0.94.0
>
> Attachments: HBASE-5143.patch
>
>
> HBASE-4240 made LoadBalancer pluggable.
> Configuration it loads seems to be wrongly named and carries a typo: 
> "hbase.maser.loadBalancer.class"
> Could rather be "hbase.master.loadbalancer.class"
> Luckily 0.92 is not out yet and we should fix it asap, before folks start 
> using it. Attaching patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4224) Need a flush by regionserver rather than by table option

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182231#comment-13182231
 ] 

jirapos...@reviews.apache.org commented on HBASE-4224:
--



bq.  On 2012-01-07 17:27:49, Ted Yu wrote:
bq.  > /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, 
line 2794
bq.  > 
bq.  >
bq.  > Do we need to place a try/catch block around line 2795 ?
bq.  > Currently the first failure would stop subsequent flushes.

Ted this sounds good and we'd want to surround it with a try catch. But then 
we'd be masking those errors right? 

If I were the user and called flush through the shell, I'd not only wanna know 
that all flushes didn't go successful but I'd also want to know how many of 
them were unsuccessful and which region flushes failed due to what reason. So 
do you think it would be a good idea to return a status object which gives the 
client more information regarding the flushes ?


bq.  On 2012-01-07 17:27:49, Ted Yu wrote:
bq.  > /src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java, line 
486
bq.  > 
bq.  >
bq.  > This and flushAllRegions() are similar.
bq.  > Can we introduce just one new method which checks whether the list 
is empty to decide what to do ?
bq.  > i.e. move the check @ line 1403 of HBaseAdmin to the implementation 
of the new method.

This is rather a flag defined on the client which means when the regions == 
null then u flush the entire RegionServer. IF we move this to the region server 
side then we'd be making this an API level definition. I think flushAllRegions 
is more intuitive as a RegionServer API than an API which flushes the Region 
Server when the regions parameter is null. 

Assume a new person wants to flushAllRegions. flush(List regions) 
tells him that hey I can flush a list of regions and unless he looks at the 
implementation or the javadoc he'd miss the fact that this API can also flush 
all the regions. On the other hand flushAllRegions invites the person with open 
arms saying that hey i'll flush all the regions for you. 
 


bq.  On 2012-01-07 17:27:49, Ted Yu wrote:
bq.  > /src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java, line 1409
bq.  > 
bq.  >
bq.  > regions could be empty, right ?

The only condition when regions could be empty is when the name to be flushed 
is a RegionServer and not a single Region or table.

This is based on the presumption that MetaReader.getTableRegionsAndLocations 
doesn't return null regions and isRegionsName is null then it never executes 
the flush. 

So this is fine ?


- Akash


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3308/#review4236
---


On 2012-01-06 18:48:11, Akash  Ashok wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3308/
bq.  ---
bq.  
bq.  (Updated 2012-01-06 18:48:11)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Flush by RegionServer
bq.  
bq.  
bq.  This addresses bug HBase-4224.
bq.  https://issues.apache.org/jira/browse/HBase-4224
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1226330 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1226330 
bq./src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 1226330 
bq./src/main/java/org/apache/hadoop/hbase/ServerName.java 1226330 
bq.  
bq.  Diff: https://reviews.apache.org/r/3308/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Akash
bq.  
bq.



> Need a flush by regionserver rather than by table option
> 
>
> Key: HBASE-4224
> URL: https://issues.apache.org/jira/browse/HBASE-4224
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Reporter: stack
>Assignee: Akash Ashok
> Attachments: HBase-4224-v2.patch, HBase-4224.patch
>
>
> This evening needed to clean out logs on the cluster.  logs are by 
> regionserver.  to let go of logs, we need to have all edits emptied from 
> memory.  only flush is by table or region.  We need to be able to flush the 
> regionserver.  Need to add this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrect

[jira] [Commented] (HBASE-4224) Need a flush by regionserver rather than by table option

[Zhihong Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182234#comment-13182234
 ] 

Zhihong Yu commented on HBASE-4224:
---

w.r.t. whether flushAllRegions() should be introduced into HRegionInterface. In 
your patch HBaseAdmin has this method:
{ocde}
private boolean flush(ServerName serverName, List regions)
{ocde}
which decides the scope of flushing by checking regions. I think this check can 
be performed on region server side.

Your argument about javadoc makes sense. We should clearly document the 
functionality of the (single) new method added to HRegionInterface.

bq. But then we'd be masking those errors right?
Please refer to HRegion.bulkLoadHFiles() for example of buffering multiple 
IOE's:
{code}
  if (ioes.size() != 0) {
LOG.error("There were IO errors when checking if bulk load is ok.  " +
"throwing exception!");
throw MultipleIOException.createIOException(ioes);
  }
{code}

> Need a flush by regionserver rather than by table option
> 
>
> Key: HBASE-4224
> URL: https://issues.apache.org/jira/browse/HBASE-4224
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Reporter: stack
>Assignee: Akash Ashok
> Attachments: HBase-4224-v2.patch, HBase-4224.patch
>
>
> This evening needed to clean out logs on the cluster.  logs are by 
> regionserver.  to let go of logs, we need to have all edits emptied from 
> memory.  only flush is by table or region.  We need to be able to flush the 
> regionserver.  Need to add this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4224) Need a flush by regionserver rather than by table option

[Akash Ashok (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182262#comment-13182262
 ] 

Akash Ashok commented on HBASE-4224:


"Your argument about javadoc makes sense. We should clearly document the 
functionality of the (single) new method added to HRegionInterface."

What I meant to say was that it's easy enough to over look this functionality 
even with good amount of javadoc. I think it would make a lot of difference if 
we have this extra new method flushAllRegions(). 

I am just trying to understand if you see any issue with this new extra API 
being added on the Region Server ?

+1 on the MultipleIOException. Thanks :). I like that class :)

> Need a flush by regionserver rather than by table option
> 
>
> Key: HBASE-4224
> URL: https://issues.apache.org/jira/browse/HBASE-4224
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Reporter: stack
>Assignee: Akash Ashok
> Attachments: HBase-4224-v2.patch, HBase-4224.patch
>
>
> This evening needed to clean out logs on the cluster.  logs are by 
> regionserver.  to let go of logs, we need to have all edits emptied from 
> memory.  only flush is by table or region.  We need to be able to flush the 
> regionserver.  Need to add this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4224) Need a flush by regionserver rather than by table option

[Zhihong Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182266#comment-13182266
 ] 

Zhihong Yu commented on HBASE-4224:
---

I am not strongly opposed to adding a new method. 
Comment from other committers is welcome. 

> Need a flush by regionserver rather than by table option
> 
>
> Key: HBASE-4224
> URL: https://issues.apache.org/jira/browse/HBASE-4224
> Project: HBase
>  Issue Type: Bug
>  Components: shell
>Reporter: stack
>Assignee: Akash Ashok
> Attachments: HBase-4224-v2.patch, HBase-4224.patch
>
>
> This evening needed to clean out logs on the cluster.  logs are by 
> regionserver.  to let go of logs, we need to have all edits emptied from 
> memory.  only flush is by table or region.  We need to be able to flush the 
> regionserver.  Need to add this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5146) Hbase Shell - allow setting config properties

[Mikael Sitruk (Created) (JIRA)]

Hbase Shell - allow setting config properties
-

 Key: HBASE-5146
 URL: https://issues.apache.org/jira/browse/HBASE-5146
 Project: HBase
  Issue Type: New Feature
  Components: shell
Affects Versions: 0.90.2, 0.90.1
Reporter: Mikael Sitruk


It should be possible to set config properties of the cluster/table/column 
family via the shell and not only via API
For example the 'hbase.hregion.majorcompaction' property cannot be set via the 
shell.




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5146) Hbase Shell - allow setting config properties

[Mikael Sitruk (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikael Sitruk updated HBASE-5146:
-

Affects Version/s: 0.92.0

> Hbase Shell - allow setting config properties
> -
>
> Key: HBASE-5146
> URL: https://issues.apache.org/jira/browse/HBASE-5146
> Project: HBase
>  Issue Type: New Feature
>  Components: shell
>Affects Versions: 0.90.1, 0.90.2, 0.92.0
>Reporter: Mikael Sitruk
>
> It should be possible to set config properties of the cluster/table/column 
> family via the shell and not only via API
> For example the 'hbase.hregion.majorcompaction' property cannot be set via 
> the shell.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5146) Hbase Shell - allow setting config properties

[Zhihong Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182301#comment-13182301
 ] 

Zhihong Yu commented on HBASE-5146:
---

Table / column family attribute change is related to HBASE-4213.

Cluster wide property update depends on HBASE-3909.

> Hbase Shell - allow setting config properties
> -
>
> Key: HBASE-5146
> URL: https://issues.apache.org/jira/browse/HBASE-5146
> Project: HBase
>  Issue Type: New Feature
>  Components: shell
>Affects Versions: 0.90.1, 0.90.2, 0.92.0
>Reporter: Mikael Sitruk
>
> It should be possible to set config properties of the cluster/table/column 
> family via the shell and not only via API
> For example the 'hbase.hregion.majorcompaction' property cannot be set via 
> the shell.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5147) Compaction/Major compaction operation from shell/API/JMX

[Mikael Sitruk (Created) (JIRA)]

Compaction/Major compaction operation from shell/API/JMX


 Key: HBASE-5147
 URL: https://issues.apache.org/jira/browse/HBASE-5147
 Project: HBase
  Issue Type: New Feature
  Components: master, regionserver, shell
Affects Versions: 0.90.2, 0.90.1, 0.92.0
Reporter: Mikael Sitruk


Compaction handling is important internal process nevertheless when it occurs 
it can have dramatic impact on cluster performance (especially when the cluster 
growth uniformly). 
Secondly current operations enabled are just to trigger a compaction process 
without possibility to know what is its status. Making very unconvenient to 
manage compaction manually.
This JIRA suggests to allow management of compaction from different interfaces 
- Shell, API, JMX

Following is the list of requested operations
1. getting compaction status - which server/region is under compaction and what 
is compacted (currently this info is only via log)
2. % of uncompacted files
3. Possibility to stop major compaction
4. Starting compaction from JMX (other interface already support this)

Note - part of these requirements are also suggested in HBASE-3965



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5148) Compaction property at the server level are not propagated at the table level.

[Mikael Sitruk (Created) (JIRA)]

Compaction property at the server level are not propagated at the table level.
--

 Key: HBASE-5148
 URL: https://issues.apache.org/jira/browse/HBASE-5148
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 0.90.1
Reporter: Mikael Sitruk


In case you do not override compaction parameter on the table/cf level, the 
values returned by the table descriptor will not reflect the value configured 
in the cluster.

For example - let assume you disabled major compaction by setting 
"hbase.hregion.majorcompaction" in the config to "0", prior starting the 
cluster. Let also assume that you have a table that in which you didn't set at 
all this parameter.
Then invoking 
HTableDescriptor hTableDescriptor = conn.getHTableDescriptor(Bytes.toBytes("my 
table"));
hTableDescriptor.getValue("hbase.hregion.majorcompaction")
should return the cluster property (currently returns the default, ignoring the 
cluster prop.)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5142) Compute standard deviation using AggregateProtocol

[Zhihong Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182308#comment-13182308
 ] 

Zhihong Yu commented on HBASE-5142:
---

AggregationClient.std() already provides this capability.

> Compute standard deviation using AggregateProtocol
> --
>
> Key: HBASE-5142
> URL: https://issues.apache.org/jira/browse/HBASE-5142
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Zhihong Yu
>
> Looking at http://en.wikipedia.org/wiki/Standard_deviation, I think two 
> passes are involved in computing std:
> 1. compute average, u
> 2. tuple of (sum of (xi-u)*(xi-u), number of rows) is returned from each 
> region
> 3. client computes std based on results from step 2

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5149) getConfiguration() implementation is misleading

[Mikael Sitruk (Created) (JIRA)]

getConfiguration() implementation is misleading
---

 Key: HBASE-5149
 URL: https://issues.apache.org/jira/browse/HBASE-5149
 Project: HBase
  Issue Type: Improvement
  Components: master, regionserver
Affects Versions: 0.90.4, 0.90.2, 0.90.1
Reporter: Mikael Sitruk


The following code will not return the cluster configuration but the local one 
which is somewhat misleading.

{code}
conn = (HConnection) HConnectionManager.getConnection(m_hbConfig); //here the 
configuration is local
Configuration conf = conn.getConfiguration()
conf.getString("hbase.hregion.majorcompaction"); // will return the parameter 
from the local config instead of the cluster the connection is connected to.
{code}

It is suggested that once a connection has been acquired the configuration 
object should be the one of the cluster.

As a general observation it is not possible to retrieve the used configuration 
on the cluster
It is suggested to add API at {{HRegionServerInterface}}, {{HMasterInterface}} 
to get the configuration used by the component appropriately (note in 0.90.4 
the getConfiguration exist on the Server interface implemented by HRegionServer 
and HMaster classes) but this interface is not visible/extended by 
HRegionServerInterface/HMasterInterface, therefore not accessible from client 
code.

Also an API like {{HashMap 
getClusterConfigurations()}} can be added on the HConnection object.


Additional notes:
Since servers can have different properties values (like disk, tmp dir,...) it 
can be acceptable that the configuration object returned by the connection 
returns special value to indicate - conflict between value or that multiple 
values exist.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3584) We need to atomically put/delete/increment in one call

[Lars Hofhansl (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182327#comment-13182327
 ] 

Lars Hofhansl commented on HBASE-3584:
--

Should we do this before "client 2.0"?

What should the client side API look like?
Something like: AtomicRowOperation (which implements Row, so can itself be 
batched), which wraps a set Put/Delete/Increment/Append (all for the same 
rowkey)?

Compared to a straight put(List), this would probably be slower as we 
cannot do the same mini batching (not with a *lot* of refactoring).

Also it is not entirely clear that this could even work with Increment/Append 
of the special immediate-visibility that those need. So maybe start with 
Put/Delete.


> We need to atomically put/delete/increment in one call
> --
>
> Key: HBASE-3584
> URL: https://issues.apache.org/jira/browse/HBASE-3584
> Project: HBase
>  Issue Type: Bug
>Reporter: ryan rawson
>
> Right now we have the following calls:
> put(Put)
> delete(Delete)
> increment(Increments)
> But we cannot combine all of the above in a single call, complete with a 
> single row lock.  It would be nice to do that.
> It would also allow us to do a CAS where we could do a put/increment if the 
> check succeeded.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5142) Compute standard deviation using AggregateProtocol

[Zhihong Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5142:
--

Comment: was deleted

(was: AggregationClient.std() already provides this capability.)

> Compute standard deviation using AggregateProtocol
> --
>
> Key: HBASE-5142
> URL: https://issues.apache.org/jira/browse/HBASE-5142
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Zhihong Yu
>
> Looking at http://en.wikipedia.org/wiki/Standard_deviation, I think two 
> passes are involved in computing std:
> 1. compute average, u
> 2. tuple of (sum of (xi-u)*(xi-u), number of rows) is returned from each 
> region
> 3. client computes std based on results from step 2

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-5123) Provide more aggregate functions for Aggregations Protocol

[Zhihong Yu (Issue Comment Edited) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182337#comment-13182337
 ] 

Zhihong Yu edited comment on HBASE-5123 at 1/9/12 12:32 AM:


AggregationClient.std() already provides support for standard deviation.

  was (Author: zhi...@ebaysf.com):
AggregationClient.std() already provides this capability.
  
> Provide more aggregate functions for Aggregations Protocol
> --
>
> Key: HBASE-5123
> URL: https://issues.apache.org/jira/browse/HBASE-5123
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zhihong Yu
>
> Royston requested the following aggregates on top of what we already have:
> Median, Weighted Median, Mult
> See discussion entitled 'AggregateProtocol Help' on user list

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5123) Provide more aggregate functions for Aggregations Protocol

[Zhihong Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182337#comment-13182337
 ] 

Zhihong Yu commented on HBASE-5123:
---

AggregationClient.std() already provides this capability.

> Provide more aggregate functions for Aggregations Protocol
> --
>
> Key: HBASE-5123
> URL: https://issues.apache.org/jira/browse/HBASE-5123
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zhihong Yu
>
> Royston requested the following aggregates on top of what we already have:
> Median, Weighted Median, Mult
> See discussion entitled 'AggregateProtocol Help' on user list

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5142) Compute quantile/ntile using AggregateProtocol

[Zhihong Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5142:
--

Description: (was: Looking at 
http://en.wikipedia.org/wiki/Standard_deviation, I think two passes are 
involved in computing std:
1. compute average, u
2. tuple of (sum of (xi-u)*(xi-u), number of rows) is returned from each region
3. client computes std based on results from step 2)
Summary: Compute quantile/ntile using AggregateProtocol  (was: Compute 
standard deviation using AggregateProtocol)

> Compute quantile/ntile using AggregateProtocol
> --
>
> Key: HBASE-5142
> URL: https://issues.apache.org/jira/browse/HBASE-5142
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Zhihong Yu
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5150) Fail in a thread may not fail a test, clean up log splitting test

[Jimmy Xiang (Created) (JIRA)]

Fail in a thread may not fail a test, clean up log splitting test
-

 Key: HBASE-5150
 URL: https://issues.apache.org/jira/browse/HBASE-5150
 Project: HBase
  Issue Type: Test
Affects Versions: 0.94.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


This is to clean up some tests for HBASE-5081.  The Assert.fail method in a 
separate thread will terminate the thread, but may not fail the test.
We can use callable, so that we can get the error in getting the result. 

Some documentation to explain the test will be helpful too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5150) Fail in a thread may not fail a test, clean up log splitting test

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182342#comment-13182342
 ] 

jirapos...@reviews.apache.org commented on HBASE-5150:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3433/
---

Review request for hbase, Todd Lipcon and Michael Stack.


Summary
---

Cleaned up the tests introduced in HBASE-5081, added some documentation.


This addresses bug HBASE-5150.
https://issues.apache.org/jira/browse/HBASE-5150


Diffs
-

  src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 0ef0e33 
  src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java 
b0487f1 
  src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 0974b56 

Diff: https://reviews.apache.org/r/3433/diff


Testing
---

Affected tests: TestDistributedLogSplitting and TestSplitLogManager, are passed


Thanks,

Jimmy



> Fail in a thread may not fail a test, clean up log splitting test
> -
>
> Key: HBASE-5150
> URL: https://issues.apache.org/jira/browse/HBASE-5150
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
>
> This is to clean up some tests for HBASE-5081.  The Assert.fail method in a 
> separate thread will terminate the thread, but may not fail the test.
> We can use callable, so that we can get the error in getting the result. 
> Some documentation to explain the test will be helpful too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5150) Fail in a thread may not fail a test, clean up log splitting test

[Jimmy Xiang (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-5150:
---

Attachment: hbase-5150.txt

> Fail in a thread may not fail a test, clean up log splitting test
> -
>
> Key: HBASE-5150
> URL: https://issues.apache.org/jira/browse/HBASE-5150
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: hbase-5150.txt
>
>
> This is to clean up some tests for HBASE-5081.  The Assert.fail method in a 
> separate thread will terminate the thread, but may not fail the test.
> We can use callable, so that we can get the error in getting the result. 
> Some documentation to explain the test will be helpful too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5150) Fail in a thread may not fail a test, clean up log splitting test

[Jimmy Xiang (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-5150:
---

Status: Patch Available  (was: Open)

> Fail in a thread may not fail a test, clean up log splitting test
> -
>
> Key: HBASE-5150
> URL: https://issues.apache.org/jira/browse/HBASE-5150
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: hbase-5150.txt
>
>
> This is to clean up some tests for HBASE-5081.  The Assert.fail method in a 
> separate thread will terminate the thread, but may not fail the test.
> We can use callable, so that we can get the error in getting the result. 
> Some documentation to explain the test will be helpful too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5121) MajorCompaction may affect scan's correctness

[chunhui shen (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182345#comment-13182345
 ] 

chunhui shen commented on HBASE-5121:
-

@Lars
I agree with the suggested patch which is simpler and available.

> MajorCompaction may affect scan's correctness
> -
>
> Key: HBASE-5121
> URL: https://issues.apache.org/jira/browse/HBASE-5121
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
>Reporter: chunhui shen
>Assignee: chunhui shen
>Priority: Critical
> Fix For: 0.94.0, 0.92.1, 0.90.6
>
> Attachments: 5121-suggest.txt, 5121-trunk-combined.txt, 5121.90, 
> hbase-5121-testcase.patch, hbase-5121.patch, hbase-5121v2.patch
>
>
> In our test, there are two families' keyvalue for one row.
> But we could find a infrequent problem when doing scan's next if 
> majorCompaction happens concurrently.
> In the client's two continuous doing scan.next():
> 1.First time, scan's next returns the result where family A is null.
> 2.Second time, scan's next returns the result where family B is null.
> The two next()'s result have the same row.
> If there are more families, I think the scenario will be more strange...
> We find the reason is that storescanner.peek() is changed after 
> majorCompaction if there are delete type KeyValue.
> This change causes the PriorityQueue of RegionScanner's heap 
> is not sure to be sorted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5150) Fail in a thread may not fail a test, clean up log splitting test

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182346#comment-13182346
 ] 

jirapos...@reviews.apache.org commented on HBASE-5150:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3433/#review4256
---



src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java


Should this read ', or until it is ...' ?



src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java


Typo: exception



src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java


So this is no longer a failure case ?


- Ted


On 2012-01-09 01:32:36, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3433/
bq.  ---
bq.  
bq.  (Updated 2012-01-09 01:32:36)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Cleaned up the tests introduced in HBASE-5081, added some documentation.
bq.  
bq.  
bq.  This addresses bug HBASE-5150.
bq.  https://issues.apache.org/jira/browse/HBASE-5150
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 
0ef0e33 
bq.
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java 
b0487f1 
bq.src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 
0974b56 
bq.  
bq.  Diff: https://reviews.apache.org/r/3433/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Affected tests: TestDistributedLogSplitting and TestSplitLogManager, are 
passed
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



> Fail in a thread may not fail a test, clean up log splitting test
> -
>
> Key: HBASE-5150
> URL: https://issues.apache.org/jira/browse/HBASE-5150
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: hbase-5150.txt
>
>
> This is to clean up some tests for HBASE-5081.  The Assert.fail method in a 
> separate thread will terminate the thread, but may not fail the test.
> We can use callable, so that we can get the error in getting the result. 
> Some documentation to explain the test will be helpful too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5150) Fail in a thread may not fail a test, clean up log splitting test

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182347#comment-13182347
 ] 

Hadoop QA commented on HBASE-5150:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12509870/hbase-5150.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 6 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -151 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 79 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.mapreduce.TestImportTsv
  org.apache.hadoop.hbase.mapred.TestTableMapReduce
  org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/702//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/702//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/702//console

This message is automatically generated.

> Fail in a thread may not fail a test, clean up log splitting test
> -
>
> Key: HBASE-5150
> URL: https://issues.apache.org/jira/browse/HBASE-5150
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: hbase-5150.txt
>
>
> This is to clean up some tests for HBASE-5081.  The Assert.fail method in a 
> separate thread will terminate the thread, but may not fail the test.
> We can use callable, so that we can get the error in getting the result. 
> Some documentation to explain the test will be helpful too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5150) Fail in a thread may not fail a test, clean up log splitting test

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182350#comment-13182350
 ] 

jirapos...@reviews.apache.org commented on HBASE-5150:
--



bq.  On 2012-01-09 01:56:13, Ted Yu wrote:
bq.  > 
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java, 
line 302
bq.  > 
bq.  >
bq.  > Should this read ', or until it is ...' ?

It will keep waiting until it is interrupted.  The task won't be removed from 
the task map in this test case actually because slm.ignoreZKDeleteForTesting is 
set to true.
I will add this to the doc.


bq.  On 2012-01-09 01:56:13, Ted Yu wrote:
bq.  > 
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java, 
line 322
bq.  > 
bq.  >
bq.  > Typo: exception

right.


bq.  On 2012-01-09 01:56:13, Ted Yu wrote:
bq.  > src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java, 
line 481
bq.  > 
bq.  >
bq.  > So this is no longer a failure case ?

It doesn't fail here. It tests a scenario when a znode is removed by others.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3433/#review4256
---


On 2012-01-09 01:32:36, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3433/
bq.  ---
bq.  
bq.  (Updated 2012-01-09 01:32:36)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Cleaned up the tests introduced in HBASE-5081, added some documentation.
bq.  
bq.  
bq.  This addresses bug HBASE-5150.
bq.  https://issues.apache.org/jira/browse/HBASE-5150
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 
0ef0e33 
bq.
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java 
b0487f1 
bq.src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 
0974b56 
bq.  
bq.  Diff: https://reviews.apache.org/r/3433/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Affected tests: TestDistributedLogSplitting and TestSplitLogManager, are 
passed
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



> Fail in a thread may not fail a test, clean up log splitting test
> -
>
> Key: HBASE-5150
> URL: https://issues.apache.org/jira/browse/HBASE-5150
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: hbase-5150.txt
>
>
> This is to clean up some tests for HBASE-5081.  The Assert.fail method in a 
> separate thread will terminate the thread, but may not fail the test.
> We can use callable, so that we can get the error in getting the result. 
> Some documentation to explain the test will be helpful too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5150) Fail in a thread may not fail a test, clean up log splitting test

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182351#comment-13182351
 ] 

jirapos...@reviews.apache.org commented on HBASE-5150:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3433/
---

(Updated 2012-01-09 02:39:28.394063)


Review request for hbase, Todd Lipcon and Michael Stack.


Changes
---

Updated per comments.


Summary
---

Cleaned up the tests introduced in HBASE-5081, added some documentation.


This addresses bug HBASE-5150.
https://issues.apache.org/jira/browse/HBASE-5150


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 0ef0e33 
  src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java 
b0487f1 
  src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 0974b56 

Diff: https://reviews.apache.org/r/3433/diff


Testing
---

Affected tests: TestDistributedLogSplitting and TestSplitLogManager, are passed


Thanks,

Jimmy



> Fail in a thread may not fail a test, clean up log splitting test
> -
>
> Key: HBASE-5150
> URL: https://issues.apache.org/jira/browse/HBASE-5150
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: hbase-5150.txt
>
>
> This is to clean up some tests for HBASE-5081.  The Assert.fail method in a 
> separate thread will terminate the thread, but may not fail the test.
> We can use callable, so that we can get the error in getting the result. 
> Some documentation to explain the test will be helpful too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5150) Fail in a thread may not fail a test, clean up log splitting test

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182355#comment-13182355
 ] 

jirapos...@reviews.apache.org commented on HBASE-5150:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3433/#review4259
---



src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java


There is no reason to wait for 3 seconds over here



src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java


Since fs is from master.getMasterFileSystem().getFileSystem(), I am pretty 
sure it gets cleaned up at the end of the test when the cluster is stopped. Did 
you see the files hanging around?



src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java


The test should fail if there is an exception.


- Prakash


On 2012-01-09 02:39:28, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3433/
bq.  ---
bq.  
bq.  (Updated 2012-01-09 02:39:28)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Cleaned up the tests introduced in HBASE-5081, added some documentation.
bq.  
bq.  
bq.  This addresses bug HBASE-5150.
bq.  https://issues.apache.org/jira/browse/HBASE-5150
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 
0ef0e33 
bq.
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java 
b0487f1 
bq.src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 
0974b56 
bq.  
bq.  Diff: https://reviews.apache.org/r/3433/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Affected tests: TestDistributedLogSplitting and TestSplitLogManager, are 
passed
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



> Fail in a thread may not fail a test, clean up log splitting test
> -
>
> Key: HBASE-5150
> URL: https://issues.apache.org/jira/browse/HBASE-5150
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: hbase-5150.txt
>
>
> This is to clean up some tests for HBASE-5081.  The Assert.fail method in a 
> separate thread will terminate the thread, but may not fail the test.
> We can use callable, so that we can get the error in getting the result. 
> Some documentation to explain the test will be helpful too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5150) Fail in a thread may not fail a test, clean up log splitting test

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182359#comment-13182359
 ] 

jirapos...@reviews.apache.org commented on HBASE-5150:
--



bq.  On 2012-01-09 03:01:16, Prakash Khemani wrote:
bq.  > 
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java, 
line 316
bq.  > 
bq.  >
bq.  > There is no reason to wait for 3 seconds over here

If everything works as expected, we won't have to wait here, since those two 
fail() calls won't be called.  I'd like to give it some time to make sure the 
second splitLogDistributed() is really blocked there.


bq.  On 2012-01-09 03:01:16, Prakash Khemani wrote:
bq.  > 
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java, 
line 326
bq.  > 
bq.  >
bq.  > Since fs is from master.getMasterFileSystem().getFileSystem(), I am 
pretty sure it gets cleaned up at the end of the test when the cluster is 
stopped. Did you see the files hanging around?

ShutdownMiniCluster will try to clean up the test dir.  Just want to be safe.


bq.  On 2012-01-09 03:01:16, Prakash Khemani wrote:
bq.  > src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java, 
line 481
bq.  > 
bq.  >
bq.  > The test should fail if there is an exception.

That fail() doesn't fail the test.   If there is an exception, those 
waitForCounter() will actually fail the test.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3433/#review4259
---


On 2012-01-09 02:39:28, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3433/
bq.  ---
bq.  
bq.  (Updated 2012-01-09 02:39:28)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Cleaned up the tests introduced in HBASE-5081, added some documentation.
bq.  
bq.  
bq.  This addresses bug HBASE-5150.
bq.  https://issues.apache.org/jira/browse/HBASE-5150
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 
0ef0e33 
bq.
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java 
b0487f1 
bq.src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 
0974b56 
bq.  
bq.  Diff: https://reviews.apache.org/r/3433/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Affected tests: TestDistributedLogSplitting and TestSplitLogManager, are 
passed
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



> Fail in a thread may not fail a test, clean up log splitting test
> -
>
> Key: HBASE-5150
> URL: https://issues.apache.org/jira/browse/HBASE-5150
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.94.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: hbase-5150.txt
>
>
> This is to clean up some tests for HBASE-5081.  The Assert.fail method in a 
> separate thread will terminate the thread, but may not fail the test.
> We can use callable, so that we can get the error in getting the result. 
> Some documentation to explain the test will be helpful too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5137) MasterFileSystem.splitLog() should abort even if waitOnSafeMode() throws IOException

[ramkrishna.s.vasudevan (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182365#comment-13182365
 ] 

ramkrishna.s.vasudevan commented on HBASE-5137:
---

@Ted
Patch for trunk is fine.  I still have a doubt on that sleep part. If sleep is 
really needed then the fix for trunk is needed. 

> MasterFileSystem.splitLog() should abort even if waitOnSafeMode() throws 
> IOException
> 
>
> Key: HBASE-5137
> URL: https://issues.apache.org/jira/browse/HBASE-5137
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: 5137-trunk.txt, HBASE-5137.patch, HBASE-5137.patch
>
>
> I am not sure if this bug was already raised in JIRA.
> In our test cluster we had a scenario where the RS had gone down and 
> ServerShutDownHandler started with splitLog.
> But as the HDFS was down the check waitOnSafeMode throws IOException.
> {code}
> try {
> // If FS is in safe mode, just wait till out of it.
> FSUtils.waitOnSafeMode(conf,
>   conf.getInt(HConstants.THREAD_WAKE_FREQUENCY, 1000));  
> splitter.splitLog();
>   } catch (OrphanHLogAfterSplitException e) {
> {code}
> We catch the exception
> {code}
> } catch (IOException e) {
>   checkFileSystem();
>   LOG.error("Failed splitting " + logDir.toString(), e);
> }
> {code}
> So the HLog split itself did not happen. We encontered like 4 regions that 
> was recently splitted in the crashed RS was lost.
> Can we abort the Master in such scenarios? Pls suggest.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5151) Rename "hbase.skip.errors" in HRegion as it is too general-sounding.

[Harsh J (Created) (JIRA)]

Rename "hbase.skip.errors" in HRegion as it is too general-sounding.


 Key: HBASE-5151
 URL: https://issues.apache.org/jira/browse/HBASE-5151
 Project: HBase
  Issue Type: Sub-task
  Components: documentation
Affects Versions: 0.94.0
Reporter: Harsh J


We should rename "hbase.skip.errors", used in HRegion.java for skipping errors 
when replaying edits. It should probably be something more like 
"hbase.hregion.edits.replay.skip.errors" or so.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3949) Add "Master" link to RegionServer pages

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182379#comment-13182379
 ] 

jirapos...@reviews.apache.org commented on HBASE-3949:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3398/#review4261
---

Ship it!


looks good to me.

- Todd


On 2012-01-05 01:28:49, Gregory Chanan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3398/
bq.  ---
bq.  
bq.  (Updated 2012-01-05 01:28:49)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This patch addresses HBASE-3949: Add "Master" link to RegionServer pages
bq.  
bq.  I used the first suggestion in the patch, "use the RS local config setting"
bq.  
bq.  
bq.  This addresses bug HBASE-3949.
bq.  https://issues.apache.org/jira/browse/HBASE-3949
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon 
24caabd 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
6ac23d6 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 
64e61bb 
bq.  
bq.  Diff: https://reviews.apache.org/r/3398/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  - Ran unit test TestRSStatusServlet.java {Passed}
bq.  
bq.  - Ran a few simple configurations:
bq.  - No hbase.master.info.port specified (default) {Link worked}
bq.  - hbase.master.info.port specified and > 0 {Link worked}
bq.  - hbase.master.info.port specified and < 0 {Printed proper message}
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Gregory
bq.  
bq.



> Add "Master" link to RegionServer pages
> ---
>
> Key: HBASE-3949
> URL: https://issues.apache.org/jira/browse/HBASE-3949
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.90.3, 0.92.0
>Reporter: Lars George
>Assignee: Gregory Chanan
>Priority: Minor
>  Labels: noob
>
> Use the ZK info where the master is to add a UI link on the top of each 
> RegionServer page. Currently you cannot navigate directly to the Master UI 
> once you are on a RS page.
> Not sure if the info port is exposed OTTOMH, but we could either use the RS 
> local config setting for that or add it to ZK to enable lookup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5152) Region is on service before completing initialized when doing rollback of split, it will affect readcorrectness

[chunhui shen (Created) (JIRA)]

Region is on service before completing initialized when doing rollback of 
split, it will affect readcorrectness 


 Key: HBASE-5152
 URL: https://issues.apache.org/jira/browse/HBASE-5152
 Project: HBase
  Issue Type: Bug
Reporter: chunhui shen




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-3949) Add "Master" link to RegionServer pages

[Todd Lipcon (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HBASE-3949.


   Resolution: Fixed
Fix Version/s: 0.94.0

Committed to trunk. The patch didn't apply to the 92 branch as is - if we want 
this in 92, we'll have to prepare a separate patch.

> Add "Master" link to RegionServer pages
> ---
>
> Key: HBASE-3949
> URL: https://issues.apache.org/jira/browse/HBASE-3949
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.90.3, 0.92.0
>Reporter: Lars George
>Assignee: Gregory Chanan
>Priority: Minor
>  Labels: noob
> Fix For: 0.94.0
>
>
> Use the ZK info where the master is to add a UI link on the top of each 
> RegionServer page. Currently you cannot navigate directly to the Master UI 
> once you are on a RS page.
> Not sure if the info port is exposed OTTOMH, but we could either use the RS 
> local config setting for that or add it to ZK to enable lookup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5152) Region is on service before completing initialized when doing rollback of split, it will affect readcorrectness

[chunhui shen (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182380#comment-13182380
 ] 

chunhui shen commented on HBASE-5152:
-

When doing rollback of split, it will first add region to onlineRegions,and 
then do this.parent.initialize();

Let's see the code of this.parent.initialize()
{code}
...
 // A region can be reopened if failed a split; reset flags
this.closing.set(false);
this.closed.set(false);
...
Store store = instantiateHStore(this.tableDir, c);
...
{code}

So, after this.closing.set(false);this.closed.set(false);
this region is on service, but it is not initialized where store is not 
initialized,
Therefore, in this period, we can't read data from this region

> Region is on service before completing initialized when doing rollback of 
> split, it will affect readcorrectness 
> 
>
> Key: HBASE-5152
> URL: https://issues.apache.org/jira/browse/HBASE-5152
> Project: HBase
>  Issue Type: Bug
>Reporter: chunhui shen
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira