[jira] [Commented] (HBASE-4669) Add an option of using round-robin assignment for enabling table

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140544#comment-13140544
 ] 

Ted Yu commented on HBASE-4669:
---

The test failures were due to 'Too many open files'

> Add an option of using round-robin assignment for enabling table
> 
>
> Key: HBASE-4669
> URL: https://issues.apache.org/jira/browse/HBASE-4669
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: HBASE-4669-90-V2.patch, HBASE-4669-90.patch, 
> HBASE-4669-Trunk-V2.patch, HBASE-4669-Trunk.patch
>
>
> Under some scenarios, we use the function of disable/enable HTable. But 
> currently, enable HTable uses the random-assignment. We hope all the regions 
> show a better distribution, no matter how many regions and how many 
> regionservers.
> So I suggest to add an option of using round-robin assignment on 
> enable-table. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140546#comment-13140546
 ] 

jirapos...@reviews.apache.org commented on HBASE-4377:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2126/
---

(Updated 2011-10-31 20:55:03.327832)


Review request for hbase, Michael Stack and Andrew Purtell.


Changes
---

Addressed Stack's comments 


Summary
---

commit fbf82c17be6b3ecca5a981f5270cf93aac26e479
Author: Jonathan Hsieh 
Date:   Wed Sep 28 10:18:11 2011 -0700

HBASE-4377 [hbck] Offline rebuild .META. from fs data only


This patch rebuilds a new .META. table by reading all the .regioninfo files in 
the hbase main directory.  It depends on the yet to be committed HBASE-4515 
(either my verison or Gary's version), HBASE-4509, and HBASE-4506.  

Some follow on work includes backporting to 0.90, auto-patching true holes, and 
adding documentation.


This addresses bug HBASE-4377.
https://issues.apache.org/jira/browse/HBASE-4377


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/HRegionInfo.java ae068c7 
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 46ca765 
  src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 9e9e07b 
  src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java ca6dd4b 
  src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java 
PRE-CREATION 
  
src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java 
PRE-CREATION 
  
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java 
PRE-CREATION 
  
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java 
PRE-CREATION 
  
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/2126/diff


Testing
---

An earlier version of this code (backported to 0.90) was used to diagnose and 
repair a cluster that had 2700 inconsistencies due to failed splits (the 
cluster was underprovisioned memory-wise, and on restart, the some regions 
would start splitting and then die due to oome's).  This was not actually used 
on a live cluster -- it was used to reconstruct a .META. from .regioninfo's 
laid out in hbase's directory structure.

Note also that this is not an automatic fix -- whenever any problems are found, 
this bails out but dumps info on holes, suggests some fixes, and displays sets 
of overlapping regions.  It is up to the user to merge regions, to create 
.regioninfo files to plug hole, and to do any potential data loosing operations.

The tests demonstrate current expected behavior -- rebuild meta if things line 
up, and fail without making modifications if holes or overlaps exist.


Thanks,

jmhsieh



> [hbck] Offline rebuild .META. from fs data only.
> 
>
> Key: HBASE-4377
> URL: https://issues.apache.org/jira/browse/HBASE-4377
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.0.90-v4.patch, 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.0.90.v3.patch, 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.patch, 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.trunk.v3.patch, 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data.0.92.v1.patch, 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data.0.92.v2.patch, 
> EXT_AC.regioninfo, EXT_ATU_05f84d32cbc0bdabf00e00bc2f3570f0.regioninfo, 
> hbase-4377-trunk.v2.patch, hbase-4377.trunk.v3.txt, hbase-4377.trunk.v4.txt, 
> hbase-4377.trunk.v5.txt
>
>
> In a worst case situation, it may be helpful to have an offline .META. 
> rebuilder that just looks at the file system's .regioninfos and rebuilds meta 
> from scratch.  Users could move bad regions out until there is a clean 
> rebuild.  
> It would likely fill in region split holes.  Follow on work could given 
> options to merge or select regions that overlap, or do online rebuilds.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4669) Add an option of using round-robin assignment for enabling table

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140541#comment-13140541
 ] 

Hadoop QA commented on HBASE-4669:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12501649/HBASE-4669-Trunk-V2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -166 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
  org.apache.hadoop.hbase.master.TestMasterFailover

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/112//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/112//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/112//console

This message is automatically generated.

> Add an option of using round-robin assignment for enabling table
> 
>
> Key: HBASE-4669
> URL: https://issues.apache.org/jira/browse/HBASE-4669
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: HBASE-4669-90-V2.patch, HBASE-4669-90.patch, 
> HBASE-4669-Trunk-V2.patch, HBASE-4669-Trunk.patch
>
>
> Under some scenarios, we use the function of disable/enable HTable. But 
> currently, enable HTable uses the random-assignment. We hope all the regions 
> show a better distribution, no matter how many regions and how many 
> regionservers.
> So I suggest to add an option of using round-robin assignment on 
> enable-table. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.

[Jonathan Hsieh (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140533#comment-13140533
 ] 

Jonathan Hsieh commented on HBASE-4377:
---

Addressed most of stack's comments:

* Removed try-catch from deleteTable.
* Updated comment related issues.
* Renamed splits in populateTable to values (splits is for region splits, the 
latter is for creating values.)
* Have separate patch for filling in holes.
* Removed setTableName and added internal check code to getTableName().
* Refactored the sidelining function to check rename returns.

I'm going to punt on these two.

* HRegion creation was done manually because the version that existed attempted 
to open stores and I didn't want or need that.
* MetaReader was not used because at the time I was trying to figure out the 
different table existence semantics in 0.90 vs trunk.   


> [hbck] Offline rebuild .META. from fs data only.
> 
>
> Key: HBASE-4377
> URL: https://issues.apache.org/jira/browse/HBASE-4377
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.0.90-v4.patch, 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.0.90.v3.patch, 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.patch, 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.trunk.v3.patch, 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data.0.92.v1.patch, 
> 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data.0.92.v2.patch, 
> EXT_AC.regioninfo, EXT_ATU_05f84d32cbc0bdabf00e00bc2f3570f0.regioninfo, 
> hbase-4377-trunk.v2.patch, hbase-4377.trunk.v3.txt, hbase-4377.trunk.v4.txt, 
> hbase-4377.trunk.v5.txt
>
>
> In a worst case situation, it may be helpful to have an offline .META. 
> rebuilder that just looks at the file system's .regioninfos and rebuilds meta 
> from scratch.  Users could move bad regions out until there is a clean 
> rebuild.  
> It would likely fill in region split holes.  Follow on work could given 
> options to merge or select regions that overlap, or do online rebuilds.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4605) Constraints

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140503#comment-13140503
 ] 

jirapos...@reviews.apache.org commented on HBASE-4605:
--



bq.  On 2011-10-29 02:11:25, Gary Helmling wrote:
bq.  > Jessie, this looks like a pretty good start.  In addition to the 
individual comments, a couple of general questions:
bq.  > 
bq.  > 1) Should ConstraintProcessor support some standard way of mapping 
Constraint implementations to the families/qualifiers they should apply to?
bq.  > 2) Should we ship a base set of Constraint implementations for common 
cases?
bq.  > 
bq.  > Even if the answer to both is "yes", they could be addressed as 
follow-up JIRAs.  But it would be good to think through the end goal here.
bq.  
bq.  Jesse Yates wrote:
bq.  1) I was thinking an individual constriant could check any/all of the 
cf/cq being used. We can do it via (2) where we have an abstract Constraint 
that makes it easy to check a given cf/cq.
bq.  
bq.  So, no (kinda) and yes. For (2), I think it will be community 
usage/findings as to what people find useful - we can decide on a case-by-case 
basis if it is a valid contribution.

Okay, seems fine to me.  Just wanted to raise the questions.  


bq.  On 2011-10-29 02:11:25, Gary Helmling wrote:
bq.  > 
src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java, line 
80
bq.  > 
bq.  >
bq.  > If we use a defined exception class for constraint violations (see 
comment above), I think we can omit this try/catch block.  Expected exceptions 
would already subclass DoNotRetryIOException, so no need to wrap it here.  
Other (non-expected) Throwables would be handled higher up by 
RegionCoprocessorHost.
bq.  
bq.  Jesse Yates wrote:
bq.  I don't think that runtime exceptions on a constraint should be 
allowed to propagate up the CPHost  - if the constraint fails then that put 
should fail, but not anything else.

Some runtime exceptions could indicate programming errors or bugs, in which 
case I think it's best to handle them the same way we handle unexpected errors 
in coprocessors -- by either unloading or aborting, depending on configuration. 
 We need to be careful for what we allow from user code running in process on 
region servers.


bq.  On 2011-10-29 02:11:25, Gary Helmling wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/constraint/IntegerConstraint.java, 
line 13
bq.  > 
bq.  >
bq.  > Is this intended as a test class or as a usable implementation?  If 
a test class, should be under src/test/java/...
bq.  > 
bq.  > If a usable implementation, then, looking at it, I wonder how useful 
it is without a way of mapping constraints to individual qualifiers.  Do you 
think that's something ConstraintProcessor should provide in general?
bq.  > 
bq.  > It could be left up to end user implementations of Constraint to 
match which KeyValues they should apply to.  But then I don't think any 
implementations we can bundle will have much value, unless they provide for 
their own qualifier matching via configuration values.
bq.  
bq.  Jesse Yates wrote:
bq.  Its meant to be both an example and a test util. I can move it into 
test, but I can see cases where people might want to just check to make sure 
they only store values. Yes, it a really simple use case, but it also helps 
people to build more complex ones

Maybe it should just go in the documentation then?  Either in package level 
javadoc or a section of the HBase book?  Maybe longer term we need a separate 
"examples" maven module for this and other extensions points.  But the current 
implementation doesn't seem useful out of the box, since it forces integers to 
be stored as strings...  in order to check that they're integers.  Seems odd.


bq.  On 2011-10-29 02:11:25, Gary Helmling wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/constraint/Constraint.java, line 39
bq.  > 
bq.  >
bq.  > Should we define a base exception here that is thrown?  I'm thinking 
something like:
bq.  > 
bq.  > ConstraintViolationException extends DoNotRetryIOException
bq.  > 
bq.  > This way we don't need to wrap in DoNotRetryIOException later and we 
give implementers a bit more guidance on what to do.
bq.  
bq.  Jesse Yates wrote:
bq.  I wanted to let people throw whatever exception they wanted to so they 
can be specific as to what happened with minimal overhead. For instance, the 
IntegerConstraint lets the constraint just check to and then throw a 
NumberFormatException if it fails. However, the code for the user (the actual 
constraint implementation) is super concise and easy.

I disagree here.  Treating

[jira] [Commented] (HBASE-4532) Avoid top row seek by dedicated bloom filter for delete family bloom filter

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140463#comment-13140463
 ] 

Ted Yu commented on HBASE-4532:
---

Looks like CHANGES.txt wasn't updated to include this JIRA.

> Avoid top row seek by dedicated bloom filter for delete family bloom filter
> ---
>
> Key: HBASE-4532
> URL: https://issues.apache.org/jira/browse/HBASE-4532
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Fix For: 0.94.0
>
> Attachments: D27.1.patch, D27.1.patch, HBASE-4532-apache-trunk.patch, 
> hbase-4532-89-fb.patch, hbase-4532-remove-system.out.println.patch
>
>
> The previous jira, HBASE-4469, is to avoid the top row seek operation if 
> row-col bloom filter is enabled. 
> This jira tries to avoid top row seek for all the cases by creating a 
> dedicated bloom filter only for delete family
> The only subtle use case is when we are interested in the top row with empty 
> column.
> For example, 
> we are interested in row1/cf1:/1/put.
> So we seek to the top row: row1/cf1:/MAX_TS/MAXIMUM. And the delete family 
> bloom filter will say there is NO delete family.
> Then it will avoid the top row seek and return a fake kv, which is the last 
> kv for this row (createLastOnRowCol).
> In this way, we have already missed the real kv we are interested in.
> The solution for the above problem is to disable this optimization if we are 
> trying to GET/SCAN a row with empty column.
> Evaluation from TestSeekOptimization:
> Previously:
> For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
> optimization: 1714 (68.40%), savings: 31.60%
> For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
> optimization: 1714 (68.40%), savings: 31.60%
> For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
> optimization: 1458 (58.18%), savings: 41.82%
> For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
> optimization: 1714 (68.40%), savings: 31.60%
> For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
> optimization: 1714 (68.40%), savings: 31.60%
> For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
> optimization: 1458 (58.18%), savings: 41.82%
> So we can get about 10% more seek savings ONLY if the ROWCOL bloom filter is 
> enabled.[HBASE-4469]
> 
> After this change:
> For bloom=NONE, compr=NONE total seeks without optimization: 2506, with 
> optimization: 1458 (58.18%), savings: 41.82%
> For bloom=ROW, compr=NONE total seeks without optimization: 2506, with 
> optimization: 1458 (58.18%), savings: 41.82%
> For bloom=ROWCOL, compr=NONE total seeks without optimization: 2506, with 
> optimization: 1458 (58.18%), savings: 41.82%
> For bloom=NONE, compr=GZ total seeks without optimization: 2506, with 
> optimization: 1458 (58.18%), savings: 41.82%
> For bloom=ROW, compr=GZ total seeks without optimization: 2506, with 
> optimization: 1458 (58.18%), savings: 41.82%
> For bloom=ROWCOL, compr=GZ total seeks without optimization: 2506, with 
> optimization: 1458 (58.18%), savings: 41.82%
> So we can get about 10% more seek savings for ALL kinds of bloom filter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4298) Support to drain RS nodes through ZK

[Aravind Gottipati (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140455#comment-13140455
 ] 

Aravind Gottipati commented on HBASE-4298:
--

I haven't had the time to figure out how to write unit tests for this, but here 
are the test cases I ran through.

Setup a cluster with 5 or more region servers and a bunch of regions.

1.  Mark a server (server A) as draining (using zkCli) and then shutdown a 
different server (server B) - check to make sure that server A does not get any 
more new regions.
2.  Mark a server (server A) as draining (using zkCli) and then go through all 
the regions on that server and assign them (from the cli), they should all end 
up on different servers and by the end, server A should not have any regions 
left on it.
3.  Mark a server (server A) as draining (using zkCli), disable the balancer 
and drain server A (as in step 2).  Then enable the balancer again and verify 
that server A does not get any new regions.
4.  Remove the server (server A) from the draining list, and check that it gets 
regions when the balancer next runs.

I ran through other scenarios/tests like this, but with multipls servers in the 
draining list.  We could probably repeat the above tests, but with two servers 
in the drain list..


> Support to drain RS nodes through ZK
> 
>
> Key: HBASE-4298
> URL: https://issues.apache.org/jira/browse/HBASE-4298
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.90.4
> Environment: all
>Reporter: Aravind Gottipati
>Priority: Critical
>  Labels: patch
> Fix For: 0.92.0, 0.90.5
>
> Attachments: 4298-trunk-v2.txt, 90_hbase.patch, trunk_hbase.patch
>
>
> HDFS currently has a way to exclude certain datanodes and prevent them from 
> getting new blocks.  HDFS goes one step further and even drains these nodes 
> for you.  This enhancement is a step in that direction.
> The idea is that we mark nodes in zookeeper as draining nodes.  This means 
> that they don't get any more new regions.  These draining nodes look exactly 
> the same as the corresponding nodes in /rs, except they live under /draining.
> Eventually, support for draining them can be added.  I am submitting two 
> patches for review - one for the 0.90 branch and one for trunk (in git).
> Here are the two patches
> 0.90 - 
> https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2
> trunk - 
> https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5
> I have tested both these patches and they work as advertised.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4680) FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have permissions

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4680:
-

Priority: Major  (was: Critical)

> FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have 
> permissions
> -
>
> Key: HBASE-4680
> URL: https://issues.apache.org/jira/browse/HBASE-4680
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 0.92.0
>
> Attachments: HBASE-4680.patch
>
>
> The HDFS safe mode check workaround introduced by HBASE-4510 performs a 
> {{FileSystem.setPermission()}} operation on the root directory ("/") when 
> attempting to trigger a {{SafeModeException}}.  As a result, it requires 
> superuser privileges when running with DFS permission checking enabled.  
> Changing the operations to act on the HBase root directory should be safe, 
> since the master process must have write access to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure

[Ted Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4690:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

TRUNK build 2390 has passed.

Will open new issue if test(s) fail in the future.

> Intermittent 
> TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut
>  failure
> 
>
> Key: HBASE-4690
> URL: https://issues.apache.org/jira/browse/HBASE-4690
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4690-trunk.txt
>
>
> See 
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/
> Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region 
> server.
> One fix for this issue is to spin up MiniCluster with 1 region server so that 
> we don't need to search for the region server where first region is hosted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4669) Add an option of using round-robin assignment for enabling table

[Ted Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4669:
--

Attachment: HBASE-4669-Trunk-V2.patch

> Add an option of using round-robin assignment for enabling table
> 
>
> Key: HBASE-4669
> URL: https://issues.apache.org/jira/browse/HBASE-4669
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: HBASE-4669-90-V2.patch, HBASE-4669-90.patch, 
> HBASE-4669-Trunk-V2.patch, HBASE-4669-Trunk.patch
>
>
> Under some scenarios, we use the function of disable/enable HTable. But 
> currently, enable HTable uses the random-assignment. We hope all the regions 
> show a better distribution, no matter how many regions and how many 
> regionservers.
> So I suggest to add an option of using round-robin assignment on 
> enable-table. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4669) Add an option of using round-robin assignment for enabling table

[Ted Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4669:
--

Status: Patch Available  (was: Open)

Patch testing.

> Add an option of using round-robin assignment for enabling table
> 
>
> Key: HBASE-4669
> URL: https://issues.apache.org/jira/browse/HBASE-4669
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: HBASE-4669-90-V2.patch, HBASE-4669-90.patch, 
> HBASE-4669-Trunk-V2.patch, HBASE-4669-Trunk.patch
>
>
> Under some scenarios, we use the function of disable/enable HTable. But 
> currently, enable HTable uses the random-assignment. We hope all the regions 
> show a better distribution, no matter how many regions and how many 
> regionservers.
> So I suggest to add an option of using round-robin assignment on 
> enable-table. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4669) Add an option of using round-robin assignment for enabling table

[Ted Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4669:
--

Attachment: (was: HBASE-4669-Trunk-V2.patch)

> Add an option of using round-robin assignment for enabling table
> 
>
> Key: HBASE-4669
> URL: https://issues.apache.org/jira/browse/HBASE-4669
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: HBASE-4669-90-V2.patch, HBASE-4669-90.patch, 
> HBASE-4669-Trunk-V2.patch, HBASE-4669-Trunk.patch
>
>
> Under some scenarios, we use the function of disable/enable HTable. But 
> currently, enable HTable uses the random-assignment. We hope all the regions 
> show a better distribution, no matter how many regions and how many 
> regionservers.
> So I suggest to add an option of using round-robin assignment on 
> enable-table. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4336) Convert source tree into maven modules

[Jesse Yates (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140413#comment-13140413
 ] 

Jesse Yates commented on HBASE-4336:


It _seems_ like the number of developed patches has gone down, making it feel 
like the time for this modularization stuff is coming soon. How about post 
branching for the fb stuff?

> Convert source tree into maven modules
> --
>
> Key: HBASE-4336
> URL: https://issues.apache.org/jira/browse/HBASE-4336
> Project: HBase
>  Issue Type: Task
>  Components: build
>Reporter: Gary Helmling
>
> When we originally converted the build to maven we had a single "core" module 
> defined, but later reverted this to a module-less build for the sake of 
> simplicity.
> It now looks like it's time to re-address this, as we have an actual need for 
> modules to:
> * provide a trimmed down "client" library that applications can make use of
> * more cleanly support building against different versions of Hadoop, in 
> place of some of the reflection machinations currently required
> * incorporate the secure RPC engine that depends on some secure Hadoop classes
> I propose we start simply by refactoring into two initial modules:
> * core - common classes and utilities, and client-side code and interfaces
> * server - master and region server implementations and supporting code
> This would also lay the groundwork for incorporating the HBase security 
> features that have been developed.  Once the module structure is in place, 
> security-related features could then be incorporated into a third module -- 
> "security" -- after normal review and approval.  The security module could 
> then depend on secure Hadoop, without modifying the dependencies of the rest 
> of the HBase code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4669) Add an option of using round-robin assignment for enabling table

[Ted Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4669:
--

Status: Open  (was: Patch Available)

> Add an option of using round-robin assignment for enabling table
> 
>
> Key: HBASE-4669
> URL: https://issues.apache.org/jira/browse/HBASE-4669
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: HBASE-4669-90-V2.patch, HBASE-4669-90.patch, 
> HBASE-4669-Trunk-V2.patch, HBASE-4669-Trunk.patch
>
>
> Under some scenarios, we use the function of disable/enable HTable. But 
> currently, enable HTable uses the random-assignment. We hope all the regions 
> show a better distribution, no matter how many regions and how many 
> regionservers.
> So I suggest to add an option of using round-robin assignment on 
> enable-table. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4669) Add an option of using round-robin assignment for enabling table

[ramkrishna.s.vasudevan (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140401#comment-13140401
 ] 

ramkrishna.s.vasudevan commented on HBASE-4669:
---

+1 on patch if tests passes.

> Add an option of using round-robin assignment for enabling table
> 
>
> Key: HBASE-4669
> URL: https://issues.apache.org/jira/browse/HBASE-4669
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.90.4, 0.94.0
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: HBASE-4669-90-V2.patch, HBASE-4669-90.patch, 
> HBASE-4669-Trunk-V2.patch, HBASE-4669-Trunk.patch
>
>
> Under some scenarios, we use the function of disable/enable HTable. But 
> currently, enable HTable uses the random-assignment. We hope all the regions 
> show a better distribution, no matter how many regions and how many 
> regionservers.
> So I suggest to add an option of using round-robin assignment on 
> enable-table. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4708) Revert safemode related pieces of hbase-4510

[Harsh J (Assigned) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J reassigned HBASE-4708:
--

Assignee: Harsh J

> Revert safemode related pieces of hbase-4510
> 
>
> Key: HBASE-4708
> URL: https://issues.apache.org/jira/browse/HBASE-4708
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Assignee: Harsh J
>Priority: Critical
> Fix For: 0.92.0
>
>
> This thread in dev has us backing out the safemode related portions of 
> hbase-4510 commit: 
> http://search-hadoop.com/m/7WOjpVyG5F/Hmaster+can%2527t+start+for+the+latest+trunk+version&subj=Hmaster+can+t+start+for+the+latest+trunk+version

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4708) Revert safemode related pieces of hbase-4510

[stack (Created) (JIRA)]

Revert safemode related pieces of hbase-4510


 Key: HBASE-4708
 URL: https://issues.apache.org/jira/browse/HBASE-4708
 Project: HBase
  Issue Type: Task
Reporter: stack


This thread in dev has us backing out the safemode related portions of 
hbase-4510 commit: 
http://search-hadoop.com/m/7WOjpVyG5F/Hmaster+can%2527t+start+for+the+latest+trunk+version&subj=Hmaster+can+t+start+for+the+latest+trunk+version

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4708) Revert safemode related pieces of hbase-4510

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4708:
-

 Priority: Critical  (was: Major)
Fix Version/s: 0.92.0

Marking critical against 0.92.

> Revert safemode related pieces of hbase-4510
> 
>
> Key: HBASE-4708
> URL: https://issues.apache.org/jira/browse/HBASE-4708
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>Priority: Critical
> Fix For: 0.92.0
>
>
> This thread in dev has us backing out the safemode related portions of 
> hbase-4510 commit: 
> http://search-hadoop.com/m/7WOjpVyG5F/Hmaster+can%2527t+start+for+the+latest+trunk+version&subj=Hmaster+can+t+start+for+the+latest+trunk+version

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4680) FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have permissions

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140391#comment-13140391
 ] 

stack commented on HBASE-4680:
--

I reverted this change.

> FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have 
> permissions
> -
>
> Key: HBASE-4680
> URL: https://issues.apache.org/jira/browse/HBASE-4680
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: HBASE-4680.patch
>
>
> The HDFS safe mode check workaround introduced by HBASE-4510 performs a 
> {{FileSystem.setPermission()}} operation on the root directory ("/") when 
> attempting to trigger a {{SafeModeException}}.  As a result, it requires 
> superuser privileges when running with DFS permission checking enabled.  
> Changing the operations to act on the HBase root directory should be safe, 
> since the master process must have write access to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4649) Add atomic bulk load function to region server

[ramkrishna.s.vasudevan (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140388#comment-13140388
 ] 

ramkrishna.s.vasudevan commented on HBASE-4649:
---

{code}
  // were gathered and before the HReiogn's write lock was taken.  We need
{code}
Typo error - it should be HRegion
{code}
+Store store = getStore(familyName);
+if (store == null) {
+  IOException ioe = new DoNotRetryIOException(
+  "No such column family " + Bytes.toStringBinary(familyName));
+  ioes.add(ioe);
+  failures.add(p);
+}
+
+try {
+  store.assertBulkLoadHFileOk(new Path(path));
+} catch (IOException ioe) {
{code}

What if store is null..invoking assertBulkLoadHFileOk() may cause exception 
right ?

> Add atomic bulk load function to region server
> --
>
> Key: HBASE-4649
> URL: https://issues.apache.org/jira/browse/HBASE-4649
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: 
> 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch, 
> hbase-4649.v2.patch, hbase-4649.v3.patch
>
>
> Add a method that atomically bulk load multiple hfiles.  Row atomicity 
> guarantees for multi-column family rows require this functionality.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HBASE-4680) FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have permissions

[stack (Reopened) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reopened HBASE-4680:
--


Reopening

> FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have 
> permissions
> -
>
> Key: HBASE-4680
> URL: https://issues.apache.org/jira/browse/HBASE-4680
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: HBASE-4680.patch
>
>
> The HDFS safe mode check workaround introduced by HBASE-4510 performs a 
> {{FileSystem.setPermission()}} operation on the root directory ("/") when 
> attempting to trigger a {{SafeModeException}}.  As a result, it requires 
> superuser privileges when running with DFS permission checking enabled.  
> Changing the operations to act on the HBase root directory should be safe, 
> since the master process must have write access to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4680) FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have permissions

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140386#comment-13140386
 ] 

stack commented on HBASE-4680:
--

Reverting because of this discussion: 
http://search-hadoop.com/m/7WOjpVyG5F/Hmaster+can%2527t+start+for+the+latest+trunk+version&subj=Hmaster+can+t+start+for+the+latest+trunk+version



> FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have 
> permissions
> -
>
> Key: HBASE-4680
> URL: https://issues.apache.org/jira/browse/HBASE-4680
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: HBASE-4680.patch
>
>
> The HDFS safe mode check workaround introduced by HBASE-4510 performs a 
> {{FileSystem.setPermission()}} operation on the root directory ("/") when 
> attempting to trigger a {{SafeModeException}}.  As a result, it requires 
> superuser privileges when running with DFS permission checking enabled.  
> Changing the operations to act on the HBase root directory should be safe, 
> since the master process must have write access to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4698) Let the HFile Pretty Printer print all the key values for a specific row.

[Phabricator (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140377#comment-13140377
 ] 

Phabricator commented on HBASE-4698:


nspiegelberg has commented on the revision "[jira] [HBASE-4698] Let the HFile 
Pretty Printer print all the key values for a specific row.".

INLINE COMMENTS
  
src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:136-137 
I would remove this line.  We should only print out information we inferred 
from user input versus this is direct input.
  
src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:266-267 
remove these 2 println() as well

REVISION DETAIL
  https://reviews.facebook.net/D111


> Let the HFile Pretty Printer print all the key values for a specific row.
> -
>
> Key: HBASE-4698
> URL: https://issues.apache.org/jira/browse/HBASE-4698
> Project: HBase
>  Issue Type: New Feature
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D111.1.patch, D111.1.patch, D111.1.patch, D111.2.patch, 
> D111.3.patch
>
>
> When using HFile Pretty Printer to debug HBase issues, 
> it would very nice to allow the Pretty Printer to seek to a specific row, and 
> only print all the key values for this row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Differential] [Commented On] D111: [jira] [HBASE-4698] Let the HFile Pretty Printer print all the key values for a specific row.

[nspiegelberg (Nicolas Spiegelberg)]

nspiegelberg has commented on the revision "[jira] [HBASE-4698] Let the HFile 
Pretty Printer print all the key values for a specific row.".

INLINE COMMENTS
  
src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:136-137 
I would remove this line.  We should only print out information we inferred 
from user input versus this is direct input.
  
src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:266-267 
remove these 2 println() as well

REVISION DETAIL
  https://reviews.facebook.net/D111

[jira] [Updated] (HBASE-4698) Let the HFile Pretty Printer print all the key values for a specific row.

[Phabricator (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4698:
---

Attachment: D111.3.patch

Liyin updated the revision "[jira] [HBASE-4698] Let the HFile Pretty Printer 
print all the key values for a specific row.".
Reviewers: mbautin, Kannan, jgray, gqchen, nspiegelberg, JIRA

  Address Mikhail's Comments

REVISION DETAIL
  https://reviews.facebook.net/D111

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java


> Let the HFile Pretty Printer print all the key values for a specific row.
> -
>
> Key: HBASE-4698
> URL: https://issues.apache.org/jira/browse/HBASE-4698
> Project: HBase
>  Issue Type: New Feature
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D111.1.patch, D111.1.patch, D111.1.patch, D111.2.patch, 
> D111.3.patch
>
>
> When using HFile Pretty Printer to debug HBase issues, 
> it would very nice to allow the Pretty Printer to seek to a specific row, and 
> only print all the key values for this row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Differential] [Updated, 54 lines] D111: [jira] [HBASE-4698] Let the HFile Pretty Printer print all the key values for a specific row.

[Liyin (Liyin Tang)]

Liyin updated the revision "[jira] [HBASE-4698] Let the HFile Pretty Printer 
print all the key values for a specific row.".
Reviewers: mbautin, Kannan, jgray, gqchen, nspiegelberg, JIRA

  Address Mikhail's Comments

REVISION DETAIL
  https://reviews.facebook.net/D111

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
Index: src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
===
--- src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
+++ src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
@@ -41,7 +41,6 @@
 import org.apache.hadoop.hbase.HRegionInfo;
 import org.apache.hadoop.hbase.KeyValue;
 import org.apache.hadoop.hbase.io.hfile.HFile.FileInfo;
-import org.apache.hadoop.hbase.io.hfile.HFileBlockIndex.BlockIndexReader;
 import org.apache.hadoop.hbase.regionserver.TimeRangeTracker;
 import org.apache.hadoop.hbase.util.BloomFilter;
 import org.apache.hadoop.hbase.util.BloomFilterFactory;
@@ -68,7 +67,11 @@
   private boolean printBlocks;
   private boolean checkRow;
   private boolean checkFamily;
-
+  private boolean isSeekToRow = false;
+  /**
+   * The row that the user want to specify and only print all the kvs for.
+   */
+  private byte[] row = null;
   private Configuration conf;
 
   private List files = new ArrayList();
@@ -92,6 +95,8 @@
 options.addOption("a", "checkfamily", false, "Enable family check");
 options.addOption("f", "file", true,
 "File to scan. Pass full-path; e.g. hdfs://a:9000/hbase/.META./12/34");
+options.addOption("s", "seekToRow", true,
+"Seek to this row and print all the kvs for this row only");
 options.addOption("r", "region", true,
 "Region to scan. Pass region name; e.g. '.META.,,1'");
   }
@@ -125,6 +130,19 @@
   files.add(new Path(cmd.getOptionValue("f")));
 }
 
+if (cmd.hasOption("s")) {
+  String key = cmd.getOptionValue("s");
+  if (key != null && key.length() != 0) {
+row = key.getBytes();
+isSeekToRow = true;
+System.out.println("Only print all the kvs for the row: " +
+	Bytes.toString(row));
+  } else {
+System.err.println("Invalid row is specified.");
+System.exit(-1);
+  }
+}
+
 if (cmd.hasOption("r")) {
   String regionName = cmd.getOptionValue("r");
   byte[] rn = Bytes.toBytes(regionName);
@@ -203,8 +221,13 @@
 if (printKey || checkRow || checkFamily) {
   // scan over file and read key/value's, performing any requested checks
   HFileScanner scanner = reader.getScanner(false, false, false);
-  scanner.seekTo();
-  scanKeyValues(file, scanner);
+  if (this.isSeekToRow) {
+// seek to the first kv on this row
+scanner.seekTo(KeyValue.createFirstOnRow(this.row).getKey());
+  } else {
+scanner.seekTo();
+  }
+  scanKeyValues(file, scanner, row);
 }
 
 // print meta data
@@ -220,7 +243,16 @@
 reader.close();
   }
 
-  private void scanKeyValues(Path file, HFileScanner scanner)
+  /**
+   * 
+   * @param file The path of the file
+   * @param scanner The HFileScanner for the file
+   * @param row Seek to the specific row and print all the kvs for this row. 
+   * if Row is null, it means no row is specified and it will 
+   * print all the rows in this file.
+   * @throws IOException
+   */
+  private void scanKeyValues(Path file, HFileScanner scanner, byte[] row)
   throws IOException {
 KeyValue pkv = null;
 boolean first = true;
@@ -230,6 +262,18 @@
   System.out.print("[");
 do {
   KeyValue kv = scanner.getKeyValue();
+  if (row != null && row.length != 0) {
+int result = Bytes.compareTo(kv.getRow(), row);
+if (result > 0) {
+  System.out.println("All the data for row " +
+  Bytes.toString(row) + " has been printed out");
+  break;
+} else if (result < 0) {
+  System.out.println("Start to scan for the row: " +
+  Bytes.toString(row));
+  continue;
+}
+  }
   // check if rows are in order
   if (checkRow && pkv != null) {
 if (Bytes.compareTo(pkv.getRow(), kv.getRow()) > 0) {

[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure

[Eugene Koontz (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140371#comment-13140371
 ] 

Eugene Koontz commented on HBASE-4690:
--

Thanks for the fix Ted; it looks good to me too. Will be watching trunk builds; 
hope this fixes it.

> Intermittent 
> TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut
>  failure
> 
>
> Key: HBASE-4690
> URL: https://issues.apache.org/jira/browse/HBASE-4690
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4690-trunk.txt
>
>
> See 
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/
> Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region 
> server.
> One fix for this issue is to spin up MiniCluster with 1 region server so that 
> we don't need to search for the region server where first region is hosted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4552) multi-CF bulk load is not atomic across column families

[ramkrishna.s.vasudevan (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140367#comment-13140367
 ] 

ramkrishna.s.vasudevan commented on HBASE-4552:
---

@Jon
Yes.. i was wrong...Sorry for that. So may be will chk the reason for failure 
once again..Thanks Jon

> multi-CF bulk load is not atomic across column families
> ---
>
> Key: HBASE-4552
> URL: https://issues.apache.org/jira/browse/HBASE-4552
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Jonathan Hsieh
> Fix For: 0.92.0
>
> Attachments: hbase-4552.consolidated.patch, 
> hbase-4552.consolidated.v2.patch, hbase-4552.consolidated.v3.patch, 
> hbase-4552.consolidated.v4.patch
>
>
> Currently the bulk load API simply imports one HFile at a time. With 
> multi-column-family support, this is inappropriate, since different CFs show 
> up separately. Instead, the IPC endpoint should take a of CF -> HFiles, so we 
> can online them all under a single region-wide lock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4698) Let the HFile Pretty Printer print all the key values for a specific row.

[Phabricator (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140369#comment-13140369
 ] 

Phabricator commented on HBASE-4698:


mbautin has accepted the revision "[jira] [HBASE-4698] Let the HFile Pretty 
Printer print all the key values for a specific row.".

  Looks good (a few very minor comments inline).

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:71 
Nit: it would be nice to use Javadoc-style comments when referring to a 
specific field.
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:133 
Coding style: space between key and !=
  
src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:136-137 
Nit: "the kv" seems to imply there is only one kv for the row, which is most 
often not the case

REVISION DETAIL
  https://reviews.facebook.net/D111


> Let the HFile Pretty Printer print all the key values for a specific row.
> -
>
> Key: HBASE-4698
> URL: https://issues.apache.org/jira/browse/HBASE-4698
> Project: HBase
>  Issue Type: New Feature
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D111.1.patch, D111.1.patch, D111.1.patch, D111.2.patch
>
>
> When using HFile Pretty Printer to debug HBase issues, 
> it would very nice to allow the Pretty Printer to seek to a specific row, and 
> only print all the key values for this row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[Differential] [Accepted] D111: [jira] [HBASE-4698] Let the HFile Pretty Printer print all the key values for a specific row.

[mbautin (Mikhail Bautin)]

mbautin has accepted the revision "[jira] [HBASE-4698] Let the HFile Pretty 
Printer print all the key values for a specific row.".

  Looks good (a few very minor comments inline).

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:71 
Nit: it would be nice to use Javadoc-style comments when referring to a 
specific field.
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:133 
Coding style: space between key and !=
  
src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java:136-137 
Nit: "the kv" seems to imply there is only one kv for the row, which is most 
often not the case

REVISION DETAIL
  https://reviews.facebook.net/D111

[jira] [Updated] (HBASE-4610) Port HBASE-3380 (Master failover can split logs of live servers) to 92/trunk (definitely bring in config params, decide if we need to do more to fix the bug)

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4610:
-

Priority: Major  (was: Critical)

No movement on this; making it major.  Not going to let it get in way of 
cutting a 0.92RC

> Port HBASE-3380 (Master failover can split logs of live servers) to 92/trunk 
> (definitely bring in config params, decide if we need to do more to fix the 
> bug)
> -
>
> Key: HBASE-4610
> URL: https://issues.apache.org/jira/browse/HBASE-4610
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Jonathan Gray
>Assignee: Jonathan Gray
> Fix For: 0.92.0
>
>
> Over in HBASE-3380 we were having some TestMasterFailover flakiness.  We 
> added some more config parameters to better control the master startup loop 
> where it waits for RS to heartbeat in.  We had thought at the time that 92 
> would have a different solution but it is still relying on heartbeats to 
> learn about RSs.
> For now, we should definitely bring these config params into 92/trunk.  
> Otherwise this is an incompatible regression and adding these will also make 
> things like what was just reported over in HBASE-4603 trivial to fix in an 
> optimal way.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4677) Remove old single bulkLoadHFile method

[Ted Yu (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4677:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

This patch has been integrated to 0.92 as part of HBASE-4552.

> Remove old single bulkLoadHFile method
> --
>
> Key: HBASE-4677
> URL: https://issues.apache.org/jira/browse/HBASE-4677
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.92.0
>
> Attachments: hbase-4677.patch
>
>
> In review for HBASE-4649, there is some debate as whether to remove, 
> deprecate, or leave the HRegionServer.bulkLoadHFile method. 
> https://reviews.apache.org/r/2545/ .   This jira will take care of that for 
> the 0.92 and trunk releases, and allow the same patch to remain for an 
> optional 0.90.x patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4703) Improvements in tests

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140343#comment-13140343
 ] 

stack commented on HBASE-4703:
--

I will apply this patch this afternoon unless objection.

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch, 20111030_4703_all.v4.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4552) multi-CF bulk load is not atomic across column families

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140344#comment-13140344
 ] 

Ted Yu commented on HBASE-4552:
---

Integrated to 0.92 and TRUNK.

Thanks for the patch Jonathan.

Thanks for the review Todd.

> multi-CF bulk load is not atomic across column families
> ---
>
> Key: HBASE-4552
> URL: https://issues.apache.org/jira/browse/HBASE-4552
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Jonathan Hsieh
> Fix For: 0.92.0
>
> Attachments: hbase-4552.consolidated.patch, 
> hbase-4552.consolidated.v2.patch, hbase-4552.consolidated.v3.patch, 
> hbase-4552.consolidated.v4.patch
>
>
> Currently the bulk load API simply imports one HFile at a time. With 
> multi-column-family support, this is inappropriate, since different CFs show 
> up separately. Instead, the IPC endpoint should take a of CF -> HFiles, so we 
> can online them all under a single region-wide lock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4703) Improvements in tests

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4703:
-

Fix Version/s: 0.92.0

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch, 20111030_4703_all.v4.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4703) Improvements in tests

[nkeywal (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140335#comment-13140335
 ] 

nkeywal commented on HBASE-4703:


The two errors in the report above are caused by a "java.io.IOException: Too 
many open files" (even if it's hidden in the logs for testWorkerAbort). I tend 
to think that this patch is now ok.

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch, 20111030_4703_all.v4.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4552) multi-CF bulk load is not atomic across column families

[Jonathan Hsieh (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140317#comment-13140317
 ] 

Jonathan Hsieh commented on HBASE-4552:
---

@Ram, on trunk or 0.92 branches, HTableDescriptor(conf,tablename) doesn't seem 
to be in the api.  In patch v4, it seems like all the HTable constructors have 
been updated to explicitly take a the configuration reference.

I'm assuming you meant HTable? 

> multi-CF bulk load is not atomic across column families
> ---
>
> Key: HBASE-4552
> URL: https://issues.apache.org/jira/browse/HBASE-4552
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Jonathan Hsieh
> Fix For: 0.92.0
>
> Attachments: hbase-4552.consolidated.patch, 
> hbase-4552.consolidated.v2.patch, hbase-4552.consolidated.v3.patch, 
> hbase-4552.consolidated.v4.patch
>
>
> Currently the bulk load API simply imports one HFile at a time. With 
> multi-column-family support, this is inappropriate, since different CFs show 
> up separately. Instead, the IPC endpoint should take a of CF -> HFiles, so we 
> can online them all under a single region-wide lock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4703) Improvements in tests

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140306#comment-13140306
 ] 

Hadoop QA commented on HBASE-4703:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12501620/20111030_4703_all.v4.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 177 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -166 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.master.TestDistributedLogSplitting

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/111//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/111//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/111//console

This message is automatically generated.

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch, 20111030_4703_all.v4.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4674) splitLog silently fails

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140302#comment-13140302
 ] 

stack commented on HBASE-4674:
--

@Prakash Thanks

> splitLog silently fails
> ---
>
> Key: HBASE-4674
> URL: https://issues.apache.org/jira/browse/HBASE-4674
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
> Environment: splitLog() can fail silently and region can open w/o its 
> edits getting replayed.
>Reporter: Prakash Khemani
>Assignee: Prakash Khemani
>Priority: Blocker
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4695) WAL logs get deleted before region server can fully flush

[stack (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4695:
-

Fix Version/s: 0.92.0

> WAL logs get deleted before region server can fully flush
> -
>
> Key: HBASE-4695
> URL: https://issues.apache.org/jira/browse/HBASE-4695
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.90.4
>Reporter: jack levin
>Assignee: gaojinchao
>Priority: Blocker
> Fix For: 0.92.0, 0.90.5
>
> Attachments: HBASE-4695_Trunk_V2.patch, 
> HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt
>
>
> To replicate the problem do the following:
> 1. check /hbase/.logs/ directory to see if you have WAL logs for the 
> region server you are shutting down.
> 2. executing kill  (where pid is a regionserver pid)
> 3. Watch the regionserver log to start flushing, you will see how many 
> regions are left to flush:
> 09:36:54,665 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 489 regions to close
> 09:56:35,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 116 regions to close
> 4. Check /hbase/.logs/ -- you will notice that it has dissapeared.
> 5. Check namenode logs:
> 09:26:41,607 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: 
> ugi=root ip=/10.101.1.5 cmd=delete 
> src=/hbase/.logs/rdaa5.prod.imageshack.com,60020,1319749
> Note that, if you kill -9 the RS now, and it crashes on flush, you won't have 
> any WAL logs to replay.  We need to make sure that logs are deleted or moved 
> out only when RS has fully flushed. Otherwise its possible to lose data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4695) WAL logs get deleted before region server can fully flush

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140297#comment-13140297
 ] 

stack commented on HBASE-4695:
--

+1 on patch.  Looks good to me.

> WAL logs get deleted before region server can fully flush
> -
>
> Key: HBASE-4695
> URL: https://issues.apache.org/jira/browse/HBASE-4695
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.90.4
>Reporter: jack levin
>Assignee: gaojinchao
>Priority: Blocker
> Fix For: 0.92.0, 0.90.5
>
> Attachments: HBASE-4695_Trunk_V2.patch, 
> HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt
>
>
> To replicate the problem do the following:
> 1. check /hbase/.logs/ directory to see if you have WAL logs for the 
> region server you are shutting down.
> 2. executing kill  (where pid is a regionserver pid)
> 3. Watch the regionserver log to start flushing, you will see how many 
> regions are left to flush:
> 09:36:54,665 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 489 regions to close
> 09:56:35,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 116 regions to close
> 4. Check /hbase/.logs/ -- you will notice that it has dissapeared.
> 5. Check namenode logs:
> 09:26:41,607 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: 
> ugi=root ip=/10.101.1.5 cmd=delete 
> src=/hbase/.logs/rdaa5.prod.imageshack.com,60020,1319749
> Note that, if you kill -9 the RS now, and it crashes on flush, you won't have 
> any WAL logs to replay.  We need to make sure that logs are deleted or moved 
> out only when RS has fully flushed. Otherwise its possible to lose data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4695) WAL logs get deleted before region server can fully flush

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140299#comment-13140299
 ] 

stack commented on HBASE-4695:
--

@Jack You want to try this patch?

> WAL logs get deleted before region server can fully flush
> -
>
> Key: HBASE-4695
> URL: https://issues.apache.org/jira/browse/HBASE-4695
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.90.4
>Reporter: jack levin
>Assignee: gaojinchao
>Priority: Blocker
> Fix For: 0.92.0, 0.90.5
>
> Attachments: HBASE-4695_Trunk_V2.patch, 
> HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt
>
>
> To replicate the problem do the following:
> 1. check /hbase/.logs/ directory to see if you have WAL logs for the 
> region server you are shutting down.
> 2. executing kill  (where pid is a regionserver pid)
> 3. Watch the regionserver log to start flushing, you will see how many 
> regions are left to flush:
> 09:36:54,665 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 489 regions to close
> 09:56:35,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 116 regions to close
> 4. Check /hbase/.logs/ -- you will notice that it has dissapeared.
> 5. Check namenode logs:
> 09:26:41,607 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: 
> ugi=root ip=/10.101.1.5 cmd=delete 
> src=/hbase/.logs/rdaa5.prod.imageshack.com,60020,1319749
> Note that, if you kill -9 the RS now, and it crashes on flush, you won't have 
> any WAL logs to replay.  We need to make sure that logs are deleted or moved 
> out only when RS has fully flushed. Otherwise its possible to lose data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4704) A JRuby script for identifying active master

[Phabricator (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140290#comment-13140290
 ] 

Phabricator commented on HBASE-4704:


stack has commented on the revision "HBASE-4704 [jira] A JRuby script for 
identifying active master".

  +1 but add licenses when you add your patch to JIRA

REVISION DETAIL
  https://reviews.facebook.net/D117


> A JRuby script for identifying active master
> 
>
> Key: HBASE-4704
> URL: https://issues.apache.org/jira/browse/HBASE-4704
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Trivial
> Fix For: 0.94.0
>
> Attachments: D117.1.patch
>
>
> This simple script reads the HBase master ZK node and outputs the hostname of 
> the active master. This is needed so that operational scripts can decide 
> where the primary master is running. I am also including a one-line 
> hbase-jruby script so we can make our jruby scripts proper UNIX executables 
> by including an "#!/usr/bin/env hbase-jruby" at the top.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4677) Remove old single bulkLoadHFile method

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140275#comment-13140275
 ] 

stack commented on HBASE-4677:
--

Hmmm.  I'm a little lost as to what is to be applied.  I'll steer clear of this 
("Patch should not be applied until other two are applied." -- I'm not sure 
what the two referred to are).

> Remove old single bulkLoadHFile method
> --
>
> Key: HBASE-4677
> URL: https://issues.apache.org/jira/browse/HBASE-4677
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.92.0
>
> Attachments: hbase-4677.patch
>
>
> In review for HBASE-4649, there is some debate as whether to remove, 
> deprecate, or leave the HRegionServer.bulkLoadHFile method. 
> https://reviews.apache.org/r/2545/ .   This jira will take care of that for 
> the 0.92 and trunk releases, and allow the same patch to remain for an 
> optional 0.90.x patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4677) Remove old single bulkLoadHFile method

[Ted Yu (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140269#comment-13140269
 ] 

Ted Yu commented on HBASE-4677:
---

@Stack:
For TRUNK, this change is contained in hbase-4552.consolidated.v4.patch up for 
HBASE-4552.


> Remove old single bulkLoadHFile method
> --
>
> Key: HBASE-4677
> URL: https://issues.apache.org/jira/browse/HBASE-4677
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.92.0
>
> Attachments: hbase-4677.patch
>
>
> In review for HBASE-4649, there is some debate as whether to remove, 
> deprecate, or leave the HRegionServer.bulkLoadHFile method. 
> https://reviews.apache.org/r/2545/ .   This jira will take care of that for 
> the 0.92 and trunk releases, and allow the same patch to remain for an 
> optional 0.90.x patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4277) HRS.closeRegion should be able to close regions with only the encoded name

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140270#comment-13140270
 ] 

stack commented on HBASE-4277:
--

+1 on commit

> HRS.closeRegion should be able to close regions with only the encoded name
> --
>
> Key: HBASE-4277
> URL: https://issues.apache.org/jira/browse/HBASE-4277
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 0.90.5
>
> Attachments: HBASE-4277_0.90.patch
>
>
> As suggested by Stack in HBASE-4217 creating a new issue to provide a patch 
> for 0.90.x version.
> We had some sort of an outage this morning due to a few racks losing power, 
> and some regions were left in the following state:
> ERROR: Region UNKNOWN_REGION on sv4r17s9:60020, 
> key=e32bbe1f48c9b3633c557dc0291b90a3, not on HDFS or in META but deployed on 
> sv4r17s9:60020
> That region was deleted by the master but the region server never got the 
> memo. Right now there's no way to force close it because HRS.closeRegion 
> requires an HRI and the only way to create one is to get it from .META. which 
> in our case doesn't contain a row for that region. Basically we have to wait 
> until that server is dead to get rid of the region and make hbck happy.
> The required change is to have closeRegion accept an encoded name in both HBA 
> (when the RS address is provided) and HRS since it's able to find it anyways 
> from it's list of live regions.
> bq.If a 0.90 version, we maybe should do that in another issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4677) Remove old single bulkLoadHFile method

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140265#comment-13140265
 ] 

stack commented on HBASE-4677:
--

@Jon OK.  Want to attach your last patch up on RB here so I can apply it?  
Thanks.

> Remove old single bulkLoadHFile method
> --
>
> Key: HBASE-4677
> URL: https://issues.apache.org/jira/browse/HBASE-4677
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.92.0
>
> Attachments: hbase-4677.patch
>
>
> In review for HBASE-4649, there is some debate as whether to remove, 
> deprecate, or leave the HRegionServer.bulkLoadHFile method. 
> https://reviews.apache.org/r/2545/ .   This jira will take care of that for 
> the 0.92 and trunk releases, and allow the same patch to remain for an 
> optional 0.90.x patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4298) Support to drain RS nodes through ZK

[stack (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140251#comment-13140251
 ] 

stack commented on HBASE-4298:
--

Thanks Ming.  Let me chase Aravind today.

> Support to drain RS nodes through ZK
> 
>
> Key: HBASE-4298
> URL: https://issues.apache.org/jira/browse/HBASE-4298
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.90.4
> Environment: all
>Reporter: Aravind Gottipati
>Priority: Critical
>  Labels: patch
> Fix For: 0.92.0, 0.90.5
>
> Attachments: 4298-trunk-v2.txt, 90_hbase.patch, trunk_hbase.patch
>
>
> HDFS currently has a way to exclude certain datanodes and prevent them from 
> getting new blocks.  HDFS goes one step further and even drains these nodes 
> for you.  This enhancement is a step in that direction.
> The idea is that we mark nodes in zookeeper as draining nodes.  This means 
> that they don't get any more new regions.  These draining nodes look exactly 
> the same as the corresponding nodes in /rs, except they live under /draining.
> Eventually, support for draining them can be added.  I am submitting two 
> patches for review - one for the 0.90 branch and one for trunk (in git).
> Here are the two patches
> 0.90 - 
> https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2
> trunk - 
> https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5
> I have tested both these patches and they work as advertised.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4298) Support to drain RS nodes through ZK

[Ming Ma (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140233#comment-13140233
 ] 

Ming Ma commented on HBASE-4298:


Just in case, there was another code review at 
https://reviews.apache.org/r/2064. It seems some of the questions have been 
answered by Aravind.

> Support to drain RS nodes through ZK
> 
>
> Key: HBASE-4298
> URL: https://issues.apache.org/jira/browse/HBASE-4298
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.90.4
> Environment: all
>Reporter: Aravind Gottipati
>Priority: Critical
>  Labels: patch
> Fix For: 0.92.0, 0.90.5
>
> Attachments: 4298-trunk-v2.txt, 90_hbase.patch, trunk_hbase.patch
>
>
> HDFS currently has a way to exclude certain datanodes and prevent them from 
> getting new blocks.  HDFS goes one step further and even drains these nodes 
> for you.  This enhancement is a step in that direction.
> The idea is that we mark nodes in zookeeper as draining nodes.  This means 
> that they don't get any more new regions.  These draining nodes look exactly 
> the same as the corresponding nodes in /rs, except they live under /draining.
> Eventually, support for draining them can be added.  I am submitting two 
> patches for review - one for the 0.90 branch and one for trunk (in git).
> Here are the two patches
> 0.90 - 
> https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2
> trunk - 
> https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5
> I have tested both these patches and they work as advertised.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4703) Improvements in tests

[nkeywal (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Status: Open  (was: Patch Available)

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch, 20111030_4703_all.v4.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4703) Improvements in tests

[nkeywal (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Status: Patch Available  (was: Open)

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch, 20111030_4703_all.v4.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4677) Remove old single bulkLoadHFile method

[Jonathan Hsieh (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140217#comment-13140217
 ] 

Jonathan Hsieh commented on HBASE-4677:
---

@Stack

I lean slightly towards removing instead of deprecating.  From those reviews, I 
was initially leaning towards deprecating until it became clear we'd need to 
bump the rpc version numbers in both cases.

The patch is broken out so it is easy to pick one path or the other.


> Remove old single bulkLoadHFile method
> --
>
> Key: HBASE-4677
> URL: https://issues.apache.org/jira/browse/HBASE-4677
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.92.0
>
> Attachments: hbase-4677.patch
>
>
> In review for HBASE-4649, there is some debate as whether to remove, 
> deprecate, or leave the HRegionServer.bulkLoadHFile method. 
> https://reviews.apache.org/r/2545/ .   This jira will take care of that for 
> the 0.92 and trunk releases, and allow the same patch to remain for an 
> optional 0.90.x patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4552) multi-CF bulk load is not atomic across column families

[ramkrishna.s.vasudevan (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140212#comment-13140212
 ] 

ramkrishna.s.vasudevan commented on HBASE-4552:
---

In test case TestLoadIncrementalHFilesSplitRecovery
new HTable(tableName) is getting used.

{code}
+  LOG.info("Creating table " + table);
+  HTableDescriptor htd = new HTableDescriptor(table);
{code}
As part of HBASE-4253 it was found that using new HTableDescriptor(conf, 
tablename) is the best way.  Also check HBASE-4138(comment 25/Aug/11 09:19) for 
reference.
This will prevent the failure that happened in 
https://builds.apache.org/job/PreCommit-HBASE-Build/99/testReport/org.apache.hadoop.hbase.mapreduce/TestLoadIncrementalHFilesSplitRecovery/testBulkLoadPhaseRecovery/

Correct me if am wrong. 


> multi-CF bulk load is not atomic across column families
> ---
>
> Key: HBASE-4552
> URL: https://issues.apache.org/jira/browse/HBASE-4552
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Jonathan Hsieh
> Fix For: 0.92.0
>
> Attachments: hbase-4552.consolidated.patch, 
> hbase-4552.consolidated.v2.patch, hbase-4552.consolidated.v3.patch, 
> hbase-4552.consolidated.v4.patch
>
>
> Currently the bulk load API simply imports one HFile at a time. With 
> multi-column-family support, this is inappropriate, since different CFs show 
> up separately. Instead, the IPC endpoint should take a of CF -> HFiles, so we 
> can online them all under a single region-wide lock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4703) Improvements in tests

[nkeywal (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Attachment: 20111030_4703_all.v4.patch

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch, 20111030_4703_all.v4.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4703) Improvements in tests

[nkeywal (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140210#comment-13140210
 ] 

nkeywal commented on HBASE-4703:


I've managed to reproduce the stack on 
testRowRange(org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol; 
after more than 10 tries. I am going to revert my change on this file. I 
haven't been able to reproduce the others (two are caused by a 
java.io.IOException: Too many open files).

I had a stack on 
testWorkerAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting), 
but it's a stack I've already seen without this patch, so I don't think it's 
related.
{noformat}
testWorkerAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting)  
Time elapsed: 231.598 sec  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at 
org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testWorkerAbort(TestDistributedLogSplitting.java:294)
{noformat}


> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch, 20111030_4703_all.v4.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4707) Avoid unnecessary DNS lookups for locality computation in HMaster

[Karthik Ranganathan (Created) (JIRA)]

Avoid unnecessary DNS lookups for locality computation in HMaster
-

 Key: HBASE-4707
 URL: https://issues.apache.org/jira/browse/HBASE-4707
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Karthik Ranganathan
Assignee: Karthik Ranganathan


We recently hit an issue where upon each RS heartbeat we were looking up and 
resolving DNS name + address of the RS in the master, but needed it only for 
locality based assignment on startup.

Some flakiness in the DNS subsystem cause one of the threads to get stuck in 
the lookup and the synchronized call at:

ServerManager.java:528
processMsgs() {
...
synchronized (this.master.getRegionManager()) {
// does dns lookup
}
}

The offending stack trace was:

"IPC Server handler 232 on 6" daemon prio=10 tid=0x7fcb64164000 
nid=0x7d16 runnable [0x52e7f000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:849)
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1200)
at java.net.InetAddress.getAllByName0(InetAddress.java:1153)
at java.net.InetAddress.getAllByName0(InetAddress.java:1128)
at java.net.InetAddress.getHostFromNameService(InetAddress.java:550)
at java.net.InetAddress.getHostName(InetAddress.java:476)
at java.net.InetAddress.getHostName(InetAddress.java:448)
at java.net.InetSocketAddress.getHostName(InetSocketAddress.java:210)
at org.apache.hadoop.hbase.HServerAddress.getHostname(HServerAddress.java:117)
at 
org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:469)
at 
org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:263)
at 
org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:500)
- locked <0x7fcb985b2030> (a org.apache.hadoop.hbase.master.RegionManager)
at 
org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:425)
at 
org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:335)
at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:841)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:585)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:933)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4277) HRS.closeRegion should be able to close regions with only the encoded name

[ramkrishna.s.vasudevan (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140135#comment-13140135
 ] 

ramkrishna.s.vasudevan commented on HBASE-4277:
---

The patch applies as is.. If it is ok I can go ahead and commit it.  Pls share 
your comments.

> HRS.closeRegion should be able to close regions with only the encoded name
> --
>
> Key: HBASE-4277
> URL: https://issues.apache.org/jira/browse/HBASE-4277
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 0.90.5
>
> Attachments: HBASE-4277_0.90.patch
>
>
> As suggested by Stack in HBASE-4217 creating a new issue to provide a patch 
> for 0.90.x version.
> We had some sort of an outage this morning due to a few racks losing power, 
> and some regions were left in the following state:
> ERROR: Region UNKNOWN_REGION on sv4r17s9:60020, 
> key=e32bbe1f48c9b3633c557dc0291b90a3, not on HDFS or in META but deployed on 
> sv4r17s9:60020
> That region was deleted by the master but the region server never got the 
> memo. Right now there's no way to force close it because HRS.closeRegion 
> requires an HRI and the only way to create one is to get it from .META. which 
> in our case doesn't contain a row for that region. Basically we have to wait 
> until that server is dead to get rid of the region and make hbck happy.
> The required change is to have closeRegion accept an encoded name in both HBA 
> (when the RS address is provided) and HRS since it's able to find it anyways 
> from it's list of live regions.
> bq.If a 0.90 version, we maybe should do that in another issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4703) Improvements in tests

[nkeywal (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Status: Open  (was: Patch Available)

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4703) Improvements in tests

[nkeywal (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Status: Patch Available  (was: Open)

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4695) WAL logs get deleted before region server can fully flush

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140044#comment-13140044
 ] 

Hadoop QA commented on HBASE-4695:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12501582/HBASE-4695_Trunk_V2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 javadoc.  The javadoc tool appears to have generated -166 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
  
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
  org.apache.hadoop.hbase.master.TestMasterFailover

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/110//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/110//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/110//console

This message is automatically generated.

> WAL logs get deleted before region server can fully flush
> -
>
> Key: HBASE-4695
> URL: https://issues.apache.org/jira/browse/HBASE-4695
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.90.4
>Reporter: jack levin
>Assignee: gaojinchao
>Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4695_Trunk_V2.patch, 
> HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt
>
>
> To replicate the problem do the following:
> 1. check /hbase/.logs/ directory to see if you have WAL logs for the 
> region server you are shutting down.
> 2. executing kill  (where pid is a regionserver pid)
> 3. Watch the regionserver log to start flushing, you will see how many 
> regions are left to flush:
> 09:36:54,665 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 489 regions to close
> 09:56:35,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 116 regions to close
> 4. Check /hbase/.logs/ -- you will notice that it has dissapeared.
> 5. Check namenode logs:
> 09:26:41,607 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: 
> ugi=root ip=/10.101.1.5 cmd=delete 
> src=/hbase/.logs/rdaa5.prod.imageshack.com,60020,1319749
> Note that, if you kill -9 the RS now, and it crashes on flush, you won't have 
> any WAL logs to replay.  We need to make sure that logs are deleted or moved 
> out only when RS has fully flushed. Otherwise its possible to lose data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4703) Improvements in tests

[nkeywal (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Status: Patch Available  (was: Open)

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4703) Improvements in tests

[nkeywal (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Status: Open  (was: Patch Available)

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4695) WAL logs get deleted before region server can fully flush

[gaojinchao (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140009#comment-13140009
 ] 

gaojinchao commented on HBASE-4695:
---

@Ted
Our laboratory network failure, I verify the newest patch tomorrow.


> WAL logs get deleted before region server can fully flush
> -
>
> Key: HBASE-4695
> URL: https://issues.apache.org/jira/browse/HBASE-4695
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.90.4
>Reporter: jack levin
>Assignee: gaojinchao
>Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4695_Trunk_V2.patch, 
> HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt
>
>
> To replicate the problem do the following:
> 1. check /hbase/.logs/ directory to see if you have WAL logs for the 
> region server you are shutting down.
> 2. executing kill  (where pid is a regionserver pid)
> 3. Watch the regionserver log to start flushing, you will see how many 
> regions are left to flush:
> 09:36:54,665 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 489 regions to close
> 09:56:35,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 116 regions to close
> 4. Check /hbase/.logs/ -- you will notice that it has dissapeared.
> 5. Check namenode logs:
> 09:26:41,607 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: 
> ugi=root ip=/10.101.1.5 cmd=delete 
> src=/hbase/.logs/rdaa5.prod.imageshack.com,60020,1319749
> Note that, if you kill -9 the RS now, and it crashes on flush, you won't have 
> any WAL logs to replay.  We need to make sure that logs are deleted or moved 
> out only when RS has fully flushed. Otherwise its possible to lose data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4703) Improvements in tests

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140007#comment-13140007
 ] 

Hadoop QA commented on HBASE-4703:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12501578/20111030_4703_all.v3.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 180 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -166 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.master.TestMasterFailover
  org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint
  org.apache.hadoop.hbase.master.TestDistributedLogSplitting

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/109//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/109//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/109//console

This message is automatically generated.

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4695) WAL logs get deleted before region server can fully flush

[gaojinchao (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

gaojinchao updated HBASE-4695:
--

Attachment: HBASE-4695_Trunk_V2.patch

> WAL logs get deleted before region server can fully flush
> -
>
> Key: HBASE-4695
> URL: https://issues.apache.org/jira/browse/HBASE-4695
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.90.4
>Reporter: jack levin
>Assignee: gaojinchao
>Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4695_Trunk_V2.patch, 
> HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt
>
>
> To replicate the problem do the following:
> 1. check /hbase/.logs/ directory to see if you have WAL logs for the 
> region server you are shutting down.
> 2. executing kill  (where pid is a regionserver pid)
> 3. Watch the regionserver log to start flushing, you will see how many 
> regions are left to flush:
> 09:36:54,665 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 489 regions to close
> 09:56:35,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 116 regions to close
> 4. Check /hbase/.logs/ -- you will notice that it has dissapeared.
> 5. Check namenode logs:
> 09:26:41,607 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: 
> ugi=root ip=/10.101.1.5 cmd=delete 
> src=/hbase/.logs/rdaa5.prod.imageshack.com,60020,1319749
> Note that, if you kill -9 the RS now, and it crashes on flush, you won't have 
> any WAL logs to replay.  We need to make sure that logs are deleted or moved 
> out only when RS has fully flushed. Otherwise its possible to lose data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4695) WAL logs get deleted before region server can fully flush

[gaojinchao (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139987#comment-13139987
 ] 

gaojinchao commented on HBASE-4695:
---

@Ted
I think your patch has some problem. when this.fsOk is false, we clean the 
Hlog, the data will lose. 


> WAL logs get deleted before region server can fully flush
> -
>
> Key: HBASE-4695
> URL: https://issues.apache.org/jira/browse/HBASE-4695
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.90.4
>Reporter: jack levin
>Assignee: gaojinchao
>Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt
>
>
> To replicate the problem do the following:
> 1. check /hbase/.logs/ directory to see if you have WAL logs for the 
> region server you are shutting down.
> 2. executing kill  (where pid is a regionserver pid)
> 3. Watch the regionserver log to start flushing, you will see how many 
> regions are left to flush:
> 09:36:54,665 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 489 regions to close
> 09:56:35,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 116 regions to close
> 4. Check /hbase/.logs/ -- you will notice that it has dissapeared.
> 5. Check namenode logs:
> 09:26:41,607 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: 
> ugi=root ip=/10.101.1.5 cmd=delete 
> src=/hbase/.logs/rdaa5.prod.imageshack.com,60020,1319749
> Note that, if you kill -9 the RS now, and it crashes on flush, you won't have 
> any WAL logs to replay.  We need to make sure that logs are deleted or moved 
> out only when RS has fully flushed. Otherwise its possible to lose data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4695) WAL logs get deleted before region server can fully flush

[gaojinchao (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139986#comment-13139986
 ] 

gaojinchao commented on HBASE-4695:
---

Latest Trunk version, test passed in a real cluster:

Region Server logs:
2011-10-31 03:32:42,922 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server 
C3S31,20020,1320034091400
2011-10-31 03:32:46,974 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server 
C3S31,20020,1320034091400; all regions closed.
2011-10-31 03:32:48,633 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: 
Moved 7 log files to /hbase/.oldlogs
2011-10-31 03:32:49,200 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server 
C3S31,20020,1320034091400; zookeeper connection closed.

Namenode logs:
2011-10-31 03:32:46,988 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditEvent(192)) - ugi=root,root,sfcbip=/158.1.130.31 
   cmd=listStatus  src=/hbase/.logs/C3S31,20020,1320034091400  
perm=root:supergroup:rwxr-xr-x
2011-10-31 03:32:46,991 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditEvent(177)) - ugi=root,root,sfcbip=/158.1.130.31 
   cmd=rename  
src=/hbase/.logs/C3S31,20020,1320034091400/C3S31%2C20020%2C1320034091400.1320045179340
  dst=/hbase/.oldlogs/C3S31%2C20020%2C1320034091400.1320045179340 
perm=root:supergroup:rw-r--r--
2011-10-31 03:32:46,992 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditEvent(177)) - ugi=root,root,sfcbip=/158.1.130.31 
   cmd=rename  
src=/hbase/.logs/C3S31,20020,1320034091400/C3S31%2C20020%2C1320034091400.1320046155808
  dst=/hbase/.oldlogs/C3S31%2C20020%2C1320034091400.1320046155808 
perm=root:supergroup:rw-r--r--
2011-10-31 03:32:46,994 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditEvent(177)) - ugi=root,root,sfcbip=/158.1.130.31 
   cmd=rename  
src=/hbase/.logs/C3S31,20020,1320034091400/C3S31%2C20020%2C1320034091400.1320046186294
  dst=/hbase/.oldlogs/C3S31%2C20020%2C1320034091400.1320046186294 
perm=root:supergroup:rw-r--r--
2011-10-31 03:32:46,996 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditEvent(177)) - ugi=root,root,sfcbip=/158.1.130.31 
   cmd=rename  
src=/hbase/.logs/C3S31,20020,1320034091400/C3S31%2C20020%2C1320034091400.1320046216288
  dst=/hbase/.oldlogs/C3S31%2C20020%2C1320034091400.1320046216288 
perm=root:supergroup:rw-r--r--
2011-10-31 03:32:46,998 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditEvent(177)) - ugi=root,root,sfcbip=/158.1.130.31 
   cmd=rename  
src=/hbase/.logs/C3S31,20020,1320034091400/C3S31%2C20020%2C1320034091400.1320046255166
  dst=/hbase/.oldlogs/C3S31%2C20020%2C1320034091400.1320046255166 
perm=root:supergroup:rw-r--r--
2011-10-31 03:32:47,206 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditEvent(192)) - ugi=webuser,webgroup  ip=/158.1.130.33 
   cmd=listStatus  src=/hbase/.logs/C3S31,20020,1320034091400  
perm=root:supergroup:rwxr-xr-x
2011-10-31 03:32:48,518 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditEvent(177)) - ugi=root,root,sfcbip=/158.1.130.31 
   cmd=rename  
src=/hbase/.logs/C3S31,20020,1320034091400/C3S31%2C20020%2C1320034091400.1320046295501
  dst=/hbase/.oldlogs/C3S31%2C20020%2C1320034091400.1320046295501 
perm=root:supergroup:rw-r--r--
2011-10-31 03:32:48,633 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditEvent(177)) - ugi=root,root,sfcbip=/158.1.130.31 
   cmd=rename  
src=/hbase/.logs/C3S31,20020,1320034091400/C3S31%2C20020%2C1320034091400.1320046325013
  dst=/hbase/.oldlogs/C3S31%2C20020%2C1320034091400.1320046325013 
perm=root:supergroup:rw-r--r--
2011-10-31 03:32:48,650 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditEvent(206)) - ugi=root,root,sfcbip=/158.1.130.31 
   cmd=delete  src=/hbase/.logs/C3S31,20020,1320034091400  
2011-10-31 03:32:49,389 INFO  FSNamesystem.audit 
(FSNamesystem.java:logAuditEvent(206)) - ugi=root,root,sfcbip=/158.1.130.32 
   cmd=delete  src=/hbase/.META./1028785192/.tmp   



> WAL logs get deleted before region server can fully flush
> -
>
> Key: HBASE-4695
> URL: https://issues.apache.org/jira/browse/HBASE-4695
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.90.4
>Reporter: jack levin
>Assignee: gaojinchao
>Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt
>
>
> To replicate the problem do the following:
> 1. check /hbase/.logs/ directory to see if you have WAL logs for the 
> region server you are shutting down.
> 2. executing kill  (where pid is a regionserver pid)
> 3. Watch the regionserver log to start flushing, you will see how many 
> regions are 

[jira] [Updated] (HBASE-4703) Improvements in tests

[nkeywal (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Status: Patch Available  (was: Open)

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4703) Improvements in tests

[nkeywal (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Status: Open  (was: Patch Available)

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4703) Improvements in tests

[nkeywal (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Attachment: 20111030_4703_all.v3.patch

Should fix the points above.

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch, 
> 20111030_4703_all.v3.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira