[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Hadoop QA (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139936#comment-13139936
 ] 

Hadoop QA commented on HBASE-4583:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12501559/4583-v4.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -166 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.master.TestMasterFailover
  org.apache.hadoop.hbase.master.TestDistributedLogSplitting
  org.apache.hadoop.hbase.regionserver.TestSplitLogWorker

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/108//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/108//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/108//console

This message is automatically generated.

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583-v2.txt, 4583-v3.txt, 4583-v4.txt, 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4706) HBASE-4120 Create shell or portal tool for user to manage priority and group

2011-10-30 Thread sinfox (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sinfox updated HBASE-4706:
--

Attachment: TablePriorityJamon.patch

 portal tool for this issue.

> HBASE-4120 Create shell or portal tool for user to manage priority and group
> 
>
> Key: HBASE-4706
> URL: https://issues.apache.org/jira/browse/HBASE-4706
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, shell
>Affects Versions: 0.92.0
>Reporter: Liu Jia
>Assignee: Liu Jia
> Fix For: 0.94.0
>
> Attachments: TablePriorityJamon.patch
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Add a tool for user to manage the functions provided by HBase-4120

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4277) HRS.closeRegion should be able to close regions with only the encoded name

2011-10-30 Thread ramkrishna.s.vasudevan (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139933#comment-13139933
 ] 

ramkrishna.s.vasudevan commented on HBASE-4277:
---

I will see if the patch still applies if not will prepare an updated on and 
then commit it.
Thanks for your review Stack

> HRS.closeRegion should be able to close regions with only the encoded name
> --
>
> Key: HBASE-4277
> URL: https://issues.apache.org/jira/browse/HBASE-4277
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.4
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 0.90.5
>
> Attachments: HBASE-4277_0.90.patch
>
>
> As suggested by Stack in HBASE-4217 creating a new issue to provide a patch 
> for 0.90.x version.
> We had some sort of an outage this morning due to a few racks losing power, 
> and some regions were left in the following state:
> ERROR: Region UNKNOWN_REGION on sv4r17s9:60020, 
> key=e32bbe1f48c9b3633c557dc0291b90a3, not on HDFS or in META but deployed on 
> sv4r17s9:60020
> That region was deleted by the master but the region server never got the 
> memo. Right now there's no way to force close it because HRS.closeRegion 
> requires an HRI and the only way to create one is to get it from .META. which 
> in our case doesn't contain a row for that region. Basically we have to wait 
> until that server is dead to get rid of the region and make hbck happy.
> The required change is to have closeRegion accept an encoded name in both HBA 
> (when the RS address is provided) and HRS since it's able to find it anyways 
> from it's list of live regions.
> bq.If a 0.90 version, we maybe should do that in another issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Created] (HBASE-4706) HBASE-4120 Create shell or portal tool for user to manage priority and group

2011-10-30 Thread Liu Jia (Created) (JIRA)
HBASE-4120 Create shell or portal tool for user to manage priority and group


 Key: HBASE-4706
 URL: https://issues.apache.org/jira/browse/HBASE-4706
 Project: HBase
  Issue Type: Sub-task
  Components: ipc, shell
Affects Versions: 0.92.0
Reporter: Liu Jia
Assignee: Liu Jia
 Fix For: 0.94.0


Add a tool for user to manage the functions provided by HBase-4120

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Assigned] (HBASE-4120) isolation and allocation

2011-10-30 Thread Liu Jia (Assigned) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Jia reassigned HBASE-4120:
--

Assignee: Liu Jia

> isolation and allocation
> 
>
> Key: HBASE-4120
> URL: https://issues.apache.org/jira/browse/HBASE-4120
> Project: HBase
>  Issue Type: New Feature
>  Components: master, regionserver
>Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0
>Reporter: Liu Jia
>Assignee: Liu Jia
> Fix For: 0.94.0
>
> Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, 
> Design_document_for_HBase_isolation_and_allocation_Revised.pdf, 
> HBase_isolation_and_allocation_user_guide.pdf, 
> Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch
>
>
> The HBase isolation and allocation tool is designed to help users manage 
> cluster resource among different application and tables.
> When we have a large scale of HBase cluster with many applications running on 
> it, there will be lots of problems. In Taobao there is a cluster for many 
> departments to test their applications performance, these applications are 
> based on HBase. With one cluster which has 12 servers, there will be only one 
> application running exclusively on this server, and many other applications 
> must wait until the previous test finished.
> After we add allocation manage function to the cluster, applications can 
> share the cluster and run concurrently. Also if the Test Engineer wants to 
> make sure there is no interference, he/she can move out other tables from 
> this group.
> In groups we use table priority to allocate resource, when system is busy; we 
> can make sure high-priority tables are not affected lower-priority tables
> Different groups can have different region server configurations, some groups 
> optimized for reading can have large block cache size, and others optimized 
> for writing can have large memstore size. 
> Tables and region servers can be moved easily between groups; after changing 
> the configuration, a group can be restarted alone instead of restarting the 
> whole cluster.
> git entry : https://github.com/ICT-Ope/HBase_allocation .
> We hope our work is helpful.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4120) isolation and allocation

2011-10-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139929#comment-13139929
 ] 

jirapos...@reviews.apache.org commented on HBASE-4120:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1421/
---

(Updated 2011-10-31 05:22:05.235409)


Review request for hbase.


Changes
---

rewrite test cases.


Summary
---

Patch used for table priority alone,In this patch, not only tables can have 
different priorities but also the different actions like "get","scan","put" and 
"delete" can have priorities.


This addresses bug HBase-4120.
https://issues.apache.org/jira/browse/HBase-4120


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java
 1189169 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
 1189169 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 1189169 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForActionPriority.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForPriorityJobQueue.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForTablePriority.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/1421/diff


Testing
---

Tested with test cases in  TestCase_For_TablePriority_trunk_v1.patch 
please apply the patch of HBASE-4181 first,in some circumstances this bug will 
affect the performance of client.


Thanks,

Jia



> isolation and allocation
> 
>
> Key: HBASE-4120
> URL: https://issues.apache.org/jira/browse/HBASE-4120
> Project: HBase
>  Issue Type: New Feature
>  Components: master, regionserver
>Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0
>Reporter: Liu Jia
> Fix For: 0.94.0
>
> Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, 
> Design_document_for_HBase_isolation_and_allocation_Revised.pdf, 
> HBase_isolation_and_allocation_user_guide.pdf, 
> Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch
>
>
> The HBase isolation and allocation tool is designed to help users manage 
> cluster resource among different application and tables.
> When we have a large scale of HBase cluster with many applications running on 
> it, there will be lots of problems. In Taobao there is a cluster for many 
> departments to test their applications performance, these applications are 
> based on HBase. With one cluster which has 12 servers, there will be only one 
> application running exclusively on this server, and many other applications 
> must wait until the previous test finished.
> After we add allocation manage function to the cluster, applications can 
> share the cluster and run concurrently. Also if the Test Engineer wants to 
> make sure there is no interference, he/she can move out other tables from 
> this group.
> In groups we use table priority to allocate resource, when system is busy; we 
> can make sure high-priority tables are not affected lower-priority tables
> Different groups can have different region server configurations, some groups 
> optimized for reading can have large block cache size, and others optimized 
> for writing can have large memstore size. 
> Tables and region servers can be moved easily between groups; after changing 
> the configuration, a group can be restarted alone instead of restarting the 
> whole cluster.
> git entry : https://github.com/ICT-Ope/HBase_allocation .
> We hope our work is helpful.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4705) HBase won't initialize if /hbase is not present

2011-10-30 Thread Harsh J (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139918#comment-13139918
 ] 

Harsh J commented on HBASE-4705:


I apologize for missing the whole activity on HBASE-4680.

I think a better fix would have been, instead of the HBASE-4680, to just catch 
other exceptions (which is something I missed despite commenting).

This change is fine too, but has lead to HBASE-4705 scenarios. Could we revert 
HBASE-4680 and instead add a:

{code}
catch (Safemode) {
 return true;
} catch (Others) {
 ignore;
}
{code}

Kinda block, while still operating on the only dir guaranteed to exist (/ - 
root)? This is so cause safemode is always checked first on the server side 
before anything else is returned. The superuser check is not client side.

> HBase won't initialize if /hbase is not present
> ---
>
> Key: HBASE-4705
> URL: https://issues.apache.org/jira/browse/HBASE-4705
> Project: HBase
>  Issue Type: Bug
>Reporter: Harsh J
>
> {code}
> 2011-10-31 00:09:09,549 FATAL org.apache.hadoop.hbase.master.HMaster: 
> Unhandled exception. Starting shutdown.
> java.io.FileNotFoundException: File does not exist: hdfs://C3S31:9000/hbase
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:731)
> at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:163)
> at 
> org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:458)
> at 
> org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:301)
> at 
> org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
> at 
> org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:112)
> at 
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:426)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> at java.lang.Thread.run(Thread.java:662)
> 2011-10-31 00:09:09,551 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
> 2011-10-31 00:09:09,551 DEBUG org.apache.hadoop.hbase.master.HMaster: 
> Stopping service threads
> {code}
> Trunk won't start HBase unless /hbase is already present, after HBASE-4680 
> (and the silly error I made in HBASE-4510).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4680) FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have permissions

2011-10-30 Thread Harsh J (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139915#comment-13139915
 ] 

Harsh J commented on HBASE-4680:


I apologize for missing the whole activity on this (didn't see a comment on 
HBASE-4510, oops).

I think a better fix would have been to just catch other exceptions (which is 
something I missed despite commenting).

This change is fine too, but has lead to HBASE-4705 scenarios. Could we revert 
this and instead add a:

{code}
catch (Safemode) {
 return true;
} catch (Others) {
 ignore;
}
{code}

Kinda block, while still operating on the only dir guaranteed to exist (/ - 
root)?

> FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have 
> permissions
> -
>
> Key: HBASE-4680
> URL: https://issues.apache.org/jira/browse/HBASE-4680
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: HBASE-4680.patch
>
>
> The HDFS safe mode check workaround introduced by HBASE-4510 performs a 
> {{FileSystem.setPermission()}} operation on the root directory ("/") when 
> attempting to trigger a {{SafeModeException}}.  As a result, it requires 
> superuser privileges when running with DFS permission checking enabled.  
> Changing the operations to act on the HBase root directory should be safe, 
> since the master process must have write access to it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Hadoop QA (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139907#comment-13139907
 ] 

Hadoop QA commented on HBASE-4583:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12501555/4583-v3.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -166 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
  org.apache.hadoop.hbase.master.TestMasterFailover

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/107//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/107//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/107//console

This message is automatically generated.

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583-v2.txt, 4583-v3.txt, 4583-v4.txt, 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Created] (HBASE-4705) HBase won't initialize if /hbase is not present

2011-10-30 Thread Harsh J (Created) (JIRA)
HBase won't initialize if /hbase is not present
---

 Key: HBASE-4705
 URL: https://issues.apache.org/jira/browse/HBASE-4705
 Project: HBase
  Issue Type: Bug
Reporter: Harsh J


{code}
2011-10-31 00:09:09,549 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled 
exception. Starting shutdown.
java.io.FileNotFoundException: File does not exist: hdfs://C3S31:9000/hbase
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:731)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:163)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:458)
at 
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:301)
at 
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
at 
org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:112)
at 
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:426)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
at java.lang.Thread.run(Thread.java:662)
2011-10-31 00:09:09,551 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2011-10-31 00:09:09,551 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping 
service threads
{code}

Trunk won't start HBase unless /hbase is already present, after HBASE-4680 (and 
the silly error I made in HBASE-4510).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4583:
-

Attachment: 4583-v4.txt

Latest patch (also on RB, but needs to be attached so that I can trigger a 
patch build).

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583-v2.txt, 4583-v3.txt, 4583-v4.txt, 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4583:
-

Status: Patch Available  (was: Open)

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583-v2.txt, 4583-v3.txt, 4583-v4.txt, 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4704) A JRuby script for identifying active master

2011-10-30 Thread Phabricator (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-4704:
---

Attachment: D117.1.patch

mbautin requested code review of "HBASE-4704 [jira] A JRuby script for 
identifying active master".
Reviewers: Kannan, Liyin, JIRA

  This simple script reads the HBase master ZK node and outputs the hostname of 
the active master. This is needed so that operational scripts can decide where 
the primary master is running. I am also adding a one-line hbase-jruby script 
so we can make our jruby scripts proper UNIX executables by including an 
"#!/usr/bin/env hbase-jruby" line at the top.

TEST PLAN
  Run it. Kill the master. Wait for znode to expire. Run again.

REVISION DETAIL
  https://reviews.facebook.net/D117

AFFECTED FILES
  bin/get-active-master.rb
  bin/hbase-jruby

MANAGE HERALD DIFFERENTIAL RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/249/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.


> A JRuby script for identifying active master
> 
>
> Key: HBASE-4704
> URL: https://issues.apache.org/jira/browse/HBASE-4704
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mikhail Bautin
>Assignee: Mikhail Bautin
>Priority: Trivial
> Fix For: 0.94.0
>
> Attachments: D117.1.patch
>
>
> This simple script reads the HBase master ZK node and outputs the hostname of 
> the active master. This is needed so that operational scripts can decide 
> where the primary master is running. I am also including a one-line 
> hbase-jruby script so we can make our jruby scripts proper UNIX executables 
> by including an "#!/usr/bin/env hbase-jruby" at the top.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4605) Constraints

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139887#comment-13139887
 ] 

Ted Yu commented on HBASE-4605:
---

@Jesse:
I only see synchronized keyword on Constraints.add().
Have you tried using synchronization on other methods ?

Also, HTableDescriptor.values is protected field. We can change its actual 
implementation to ConcurrentHashMap, etc to accommodate for the concurrency you 
described.

If we store metadata about constraints in the Configuration object as I 
described @ 29/Oct/11 04:20, we utilize the available serialization mechanism.
The current approach deals with serialization itself. This is not as flexible 
as the above approach.

> Constraints
> ---
>
> Key: HBASE-4605
> URL: https://issues.apache.org/jira/browse/HBASE-4605
> Project: HBase
>  Issue Type: Improvement
>  Components: client, coprocessors
>Affects Versions: 0.94.0
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
>
>
> From Jesse's comment on dev:
> {quote}
> What I would like to propose is a simple interface that people can use to 
> implement a 'constraint' (matching the classic database definition). This 
> would help ease of adoption by helping HBase more easily check that box, help 
> minimize code duplication across organizations, and lead to easier adoption.
> Essentially, people would implement a 'Constraint' interface for checking 
> keys before they are put into a table. Puts that are valid get written to the 
> table, but if not people can will throw an exception that gets propagated 
> back to the client explaining why the put was invalid.
> Constraints would be set on a per-table basis and the user would be expected 
> to ensure the jars containing the constraint are present on the machines 
> serving that table.
> Yes, people could roll their own mechanism for doing this via coprocessors 
> each time, but this would make it easier to do so, so you only have to 
> implement a very minimal interface and not worry about the specifics.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Created] (HBASE-4704) A JRuby script for identifying active master

2011-10-30 Thread Mikhail Bautin (Created) (JIRA)
A JRuby script for identifying active master


 Key: HBASE-4704
 URL: https://issues.apache.org/jira/browse/HBASE-4704
 Project: HBase
  Issue Type: Improvement
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Priority: Trivial
 Fix For: 0.94.0


This simple script reads the HBase master ZK node and outputs the hostname of 
the active master. This is needed so that operational scripts can decide where 
the primary master is running. I am also including a one-line hbase-jruby 
script so we can make our jruby scripts proper UNIX executables by including an 
"#!/usr/bin/env hbase-jruby" at the top.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4583:
-

Status: Open  (was: Patch Available)

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583-v2.txt, 4583-v3.txt, 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4583:
-

Attachment: 4583-v3.txt

Much cleaned up patch. Unifies the atomicUpsert (the called must acquire the 
locks, atomicUpsert releases them).

Also includes rollback logic in case the sync of the WAL fails.

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583-v2.txt, 4583-v3.txt, 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4583:
-

Status: Patch Available  (was: Open)

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583-v2.txt, 4583-v3.txt, 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4583:
-

Status: Open  (was: Patch Available)

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583-v2.txt, 4583-v3.txt, 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4677) Remove old single bulkLoadHFile method

2011-10-30 Thread Todd Lipcon (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139866#comment-13139866
 ] 

Todd Lipcon commented on HBASE-4677:


I don't think we need to deprecate this before removal, because IMO 
HRegionInterface is not a public API. Users should only be using the public 
interfaces like HTable, LoadIncrementalHFiles, etc. There is no real legitimate 
use to directly access the HRI API.

> Remove old single bulkLoadHFile method
> --
>
> Key: HBASE-4677
> URL: https://issues.apache.org/jira/browse/HBASE-4677
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.92.0
>
> Attachments: hbase-4677.patch
>
>
> In review for HBASE-4649, there is some debate as whether to remove, 
> deprecate, or leave the HRegionServer.bulkLoadHFile method. 
> https://reviews.apache.org/r/2545/ .   This jira will take care of that for 
> the 0.92 and trunk releases, and allow the same patch to remain for an 
> optional 0.90.x patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4064) Two concurrent unassigning of the same region caused the endless loop of "Region has been PENDING_CLOSE for too long..."

2011-10-30 Thread gaojinchao (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139860#comment-13139860
 ] 

gaojinchao commented on HBASE-4064:
---

I found enable table issue was fixed in HBASE-4395. 
I think this issue should use the same way to fix.

> Two concurrent unassigning of the same region caused the endless loop of 
> "Region has been PENDING_CLOSE for too long..."
> 
>
> Key: HBASE-4064
> URL: https://issues.apache.org/jira/browse/HBASE-4064
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
> Fix For: 0.90.5
>
> Attachments: HBASE-4064-v1.patch, HBASE-4064_branch90V2.patch, 
> disableflow.png
>
>
> 1. If there is a "rubbish" RegionState object with "PENDING_CLOSE" in 
> regionsInTransition(The RegionState was remained by some exception which 
> should be removed, that's why I called it as "rubbish" object), but the 
> region is not currently assigned anywhere, TimeoutMonitor will fall into an 
> endless loop:
> 2011-06-27 10:32:21,326 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed 
> out:  test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. 
> state=PENDING_CLOSE, ts=1309141555301
> 2011-06-27 10:32:21,326 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been 
> PENDING_CLOSE for too long, running forced unassign again on 
> region=test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f.
> 2011-06-27 10:32:21,438 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of 
> region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. 
> (offlining)
> 2011-06-27 10:32:21,441 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign 
> region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. but it is 
> not currently assigned anywhere
> 2011-06-27 10:32:31,207 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed 
> out:  test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. 
> state=PENDING_CLOSE, ts=1309141555301
> 2011-06-27 10:32:31,207 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been 
> PENDING_CLOSE for too long, running forced unassign again on 
> region=test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f.
> 2011-06-27 10:32:31,215 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of 
> region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. 
> (offlining)
> 2011-06-27 10:32:31,215 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign 
> region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. but it is 
> not currently assigned anywhere
> 2011-06-27 10:32:41,164 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed 
> out:  test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. 
> state=PENDING_CLOSE, ts=1309141555301
> 2011-06-27 10:32:41,164 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Region has been 
> PENDING_CLOSE for too long, running forced unassign again on 
> region=test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f.
> 2011-06-27 10:32:41,172 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of 
> region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. 
> (offlining)
> 2011-06-27 10:32:41,172 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign 
> region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. but it is 
> not currently assigned anywhere
> .
> 2  In the following scenario, two concurrent unassigning call of the same 
> region may lead to the above problem:
> the first unassign call send rpc call success, the master watched the event 
> of "RS_ZK_REGION_CLOSED", process this event, will create a 
> ClosedRegionHandler to remove the state of the region in master.eg.
> while ClosedRegionHandler is running in  
> "hbase.master.executor.closeregion.threads" thread (A), another unassign call 
> of same region run in another thread(B).
> while thread B  run "if (!regions.containsKey(region))", this.regions have 
> the region info, now  cpu switch to thread A.
> The thread A will remove the region from the sets of "this.regions" and 
> "regionsInTransition", then switch to thread B. the thread B run continue, 
> will throw an exception with the msg of "Server null returned 
> java.lang.NullPointerException: Passed server is null for 
> 9a6e26d40293663a79523c58315b930f", but without removing the new-adding 
> RegionState from "regionsInTransition",and it can not be 

[jira] [Commented] (HBASE-4605) Constraints

2011-10-30 Thread Jesse Yates (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139859#comment-13139859
 ] 

Jesse Yates commented on HBASE-4605:


@Ted: concurrent modification exception - an inherent problem of using the 
iterator on the hash map (actually tried that first though :))

> Constraints
> ---
>
> Key: HBASE-4605
> URL: https://issues.apache.org/jira/browse/HBASE-4605
> Project: HBase
>  Issue Type: Improvement
>  Components: client, coprocessors
>Affects Versions: 0.94.0
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
>
>
> From Jesse's comment on dev:
> {quote}
> What I would like to propose is a simple interface that people can use to 
> implement a 'constraint' (matching the classic database definition). This 
> would help ease of adoption by helping HBase more easily check that box, help 
> minimize code duplication across organizations, and lead to easier adoption.
> Essentially, people would implement a 'Constraint' interface for checking 
> keys before they are put into a table. Puts that are valid get written to the 
> table, but if not people can will throw an exception that gets propagated 
> back to the client explaining why the put was invalid.
> Constraints would be set on a per-table basis and the user would be expected 
> to ensure the jars containing the constraint are present on the machines 
> serving that table.
> Yes, people could roll their own mechanism for doing this via coprocessors 
> each time, but this would make it easier to do so, so you only have to 
> implement a very minimal interface and not worry about the specifics.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4674) splitLog silently fails

2011-10-30 Thread Prakash Khemani (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139857#comment-13139857
 ] 

Prakash Khemani commented on HBASE-4674:


Stack, it is pretty obvious in the code. And yes, I have had seen lost
edits a number of times.

A simple way to reproduce this issue

Create a table
Kill namenode. That kills all region servers.
Master doesn't die.
Master tries to split logs and fails. But it ignores the failure and moves
on to assign regions.
Start namenode.
Start regionservers
The regions get assigned w/o their logs getting replayed.

==

BTW, the fix to this is being posted by Nicolas in HBASE-2312







> splitLog silently fails
> ---
>
> Key: HBASE-4674
> URL: https://issues.apache.org/jira/browse/HBASE-4674
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
> Environment: splitLog() can fail silently and region can open w/o its 
> edits getting replayed.
>Reporter: Prakash Khemani
>Assignee: Prakash Khemani
>Priority: Blocker
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4605) Constraints

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139858#comment-13139858
 ] 

Ted Yu commented on HBASE-4605:
---

I think the name of table should be added to the following exception:
{code}
  throw new IllegalArgumentException("Constraint: " + clazz.getName()
  + " is not associated with this table.");
{code}
Constraints.getKeyValueForClass() may return null. I think we should check for 
null in remove(HTableDescriptor, Class):
{code}
 desc.remove(e.getKey().get());
{code}
and other methods.

> Constraints
> ---
>
> Key: HBASE-4605
> URL: https://issues.apache.org/jira/browse/HBASE-4605
> Project: HBase
>  Issue Type: Improvement
>  Components: client, coprocessors
>Affects Versions: 0.94.0
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
>
>
> From Jesse's comment on dev:
> {quote}
> What I would like to propose is a simple interface that people can use to 
> implement a 'constraint' (matching the classic database definition). This 
> would help ease of adoption by helping HBase more easily check that box, help 
> minimize code duplication across organizations, and lead to easier adoption.
> Essentially, people would implement a 'Constraint' interface for checking 
> keys before they are put into a table. Puts that are valid get written to the 
> table, but if not people can will throw an exception that gets propagated 
> back to the client explaining why the put was invalid.
> Constraints would be set on a per-table basis and the user would be expected 
> to ensure the jars containing the constraint are present on the machines 
> serving that table.
> Yes, people could roll their own mechanism for doing this via coprocessors 
> each time, but this would make it easier to do so, so you only have to 
> implement a very minimal interface and not worry about the specifics.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4605) Constraints

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139854#comment-13139854
 ] 

Ted Yu commented on HBASE-4605:
---

In the above patch, why do we need two loops in 
Constraints.remove(HTableDescriptor) ?
Can we call desc.remove(e.getKey()) in the first loop ?

> Constraints
> ---
>
> Key: HBASE-4605
> URL: https://issues.apache.org/jira/browse/HBASE-4605
> Project: HBase
>  Issue Type: Improvement
>  Components: client, coprocessors
>Affects Versions: 0.94.0
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
>
>
> From Jesse's comment on dev:
> {quote}
> What I would like to propose is a simple interface that people can use to 
> implement a 'constraint' (matching the classic database definition). This 
> would help ease of adoption by helping HBase more easily check that box, help 
> minimize code duplication across organizations, and lead to easier adoption.
> Essentially, people would implement a 'Constraint' interface for checking 
> keys before they are put into a table. Puts that are valid get written to the 
> table, but if not people can will throw an exception that gets propagated 
> back to the client explaining why the put was invalid.
> Constraints would be set on a per-table basis and the user would be expected 
> to ensure the jars containing the constraint are present on the machines 
> serving that table.
> Yes, people could roll their own mechanism for doing this via coprocessors 
> each time, but this would make it easier to do so, so you only have to 
> implement a very minimal interface and not worry about the specifics.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4605) Constraints

2011-10-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139843#comment-13139843
 ] 

jirapos...@reviews.apache.org commented on HBASE-4605:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2579/#review2934
---



src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java


Whoops -  missed that one.


- Jesse


On 2011-10-31 00:48:49, Jesse Yates wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2579/
bq.  ---
bq.  
bq.  (Updated 2011-10-31 00:48:49)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Most of the implementation for adding constraints as a coprocessor. 
bq.  
bq.  Looking for general comments on style/structure, though nitpicks are ok 
too. 
bq.  
bq.  Currently missing implementation for disableConstraints() since that will 
require adding removeCoprocessor() to HTD (also comments on if this is worth it 
would be good). 
bq.  
bq.  
bq.  This addresses bug HBASE-4605.
bq.  https://issues.apache.org/jira/browse/HBASE-4605
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 99875b8 
bq.src/main/java/org/apache/hadoop/hbase/constraint/BaseConstraint.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/constraint/Constraint.java 
PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/constraint/Constraints.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/constraint/IntegerConstraint.java 
PRE-CREATION 
bq.
src/test/java/org/apache/hadoop/hbase/constraint/IntegrationTestConstraint.java 
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/2579/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Adding IntegrationTestConstraint and unit tests for Constraints and 
IntegerConstraint. All of those pass.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jesse
bq.  
bq.



> Constraints
> ---
>
> Key: HBASE-4605
> URL: https://issues.apache.org/jira/browse/HBASE-4605
> Project: HBase
>  Issue Type: Improvement
>  Components: client, coprocessors
>Affects Versions: 0.94.0
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
>
>
> From Jesse's comment on dev:
> {quote}
> What I would like to propose is a simple interface that people can use to 
> implement a 'constraint' (matching the classic database definition). This 
> would help ease of adoption by helping HBase more easily check that box, help 
> minimize code duplication across organizations, and lead to easier adoption.
> Essentially, people would implement a 'Constraint' interface for checking 
> keys before they are put into a table. Puts that are valid get written to the 
> table, but if not people can will throw an exception that gets propagated 
> back to the client explaining why the put was invalid.
> Constraints would be set on a per-table basis and the user would be expected 
> to ensure the jars containing the constraint are present on the machines 
> serving that table.
> Yes, people could roll their own mechanism for doing this via coprocessors 
> each time, but this would make it easier to do so, so you only have to 
> implement a very minimal interface and not worry about the specifics.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4605) Constraints

2011-10-30 Thread jirapos...@reviews.apache.org (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139842#comment-13139842
 ] 

jirapos...@reviews.apache.org commented on HBASE-4605:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2579/
---

(Updated 2011-10-31 00:48:49.738577)


Review request for hbase.


Changes
---

Updated based on comments on the last patch and in HBASE-4605 (corresponding 
JIRA).
Added feautures:
- enable
- disable
- remove
- enable constraint
- disable constraint
- remove constraint
- actually enforcing ordering of constraints based on add
- update constraint
- removal of CPs in HTD
Also, updated tests to cover new cases (which all pass locally).

Still TODO: tie constraints into shell


Summary
---

Most of the implementation for adding constraints as a coprocessor. 

Looking for general comments on style/structure, though nitpicks are ok too. 

Currently missing implementation for disableConstraints() since that will 
require adding removeCoprocessor() to HTD (also comments on if this is worth it 
would be good). 


This addresses bug HBASE-4605.
https://issues.apache.org/jira/browse/HBASE-4605


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 99875b8 
  src/main/java/org/apache/hadoop/hbase/constraint/BaseConstraint.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/constraint/Constraint.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/constraint/Constraints.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/constraint/IntegerConstraint.java 
PRE-CREATION 
  
src/test/java/org/apache/hadoop/hbase/constraint/IntegrationTestConstraint.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/2579/diff


Testing
---

Adding IntegrationTestConstraint and unit tests for Constraints and 
IntegerConstraint. All of those pass.


Thanks,

Jesse



> Constraints
> ---
>
> Key: HBASE-4605
> URL: https://issues.apache.org/jira/browse/HBASE-4605
> Project: HBase
>  Issue Type: Improvement
>  Components: client, coprocessors
>Affects Versions: 0.94.0
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
>
>
> From Jesse's comment on dev:
> {quote}
> What I would like to propose is a simple interface that people can use to 
> implement a 'constraint' (matching the classic database definition). This 
> would help ease of adoption by helping HBase more easily check that box, help 
> minimize code duplication across organizations, and lead to easier adoption.
> Essentially, people would implement a 'Constraint' interface for checking 
> keys before they are put into a table. Puts that are valid get written to the 
> table, but if not people can will throw an exception that gets propagated 
> back to the client explaining why the put was invalid.
> Constraints would be set on a per-table basis and the user would be expected 
> to ensure the jars containing the constraint are present on the machines 
> serving that table.
> Yes, people could roll their own mechanism for doing this via coprocessors 
> each time, but this would make it easier to do so, so you only have to 
> implement a very minimal interface and not worry about the specifics.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure

2011-10-30 Thread Hudson (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139839#comment-13139839
 ] 

Hudson commented on HBASE-4690:
---

Integrated in HBase-TRUNK #2388 (See 
[https://builds.apache.org/job/HBase-TRUNK/2388/])
HBASE-4690  Intermittent TestRegionServerCoprocessorExceptionWithAbort 
failure

tedyu : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java


> Intermittent 
> TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut
>  failure
> 
>
> Key: HBASE-4690
> URL: https://issues.apache.org/jira/browse/HBASE-4690
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4690-trunk.txt
>
>
> See 
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/
> Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region 
> server.
> One fix for this issue is to spin up MiniCluster with 1 region server so that 
> we don't need to search for the region server where first region is hosted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Hadoop QA (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139837#comment-13139837
 ] 

Hadoop QA commented on HBASE-4583:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12501538/4583-v2.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -166 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.master.TestMasterFailover
  org.apache.hadoop.hbase.master.TestDistributedLogSplitting

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/106//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/106//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/106//console

This message is automatically generated.

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583-v2.txt, 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4603) Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers

2011-10-30 Thread Hudson (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139820#comment-13139820
 ] 

Hudson commented on HBASE-4603:
---

Integrated in HBase-TRUNK #2387 (See 
[https://builds.apache.org/job/HBase-TRUNK/2387/])
HBASE-4603 Uneeded sleep time for tests in 
hbase.master.ServerManager#waitForRegionServers

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/test/resources/hbase-site.xml


> Uneeded sleep time for tests in 
> hbase.master.ServerManager#waitForRegionServers
> ---
>
> Key: HBASE-4603
> URL: https://issues.apache.org/jira/browse/HBASE-4603
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all.
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 20111017_4603_MiniHBaseCluster.patch, 4603-v2.txt
>
>
> This functions waits for at least 2 times 
> hbase.master.wait.on.regionservers.interval, defaulted at 3 seconds, i.e. 6 
> seconds for every mini hbase cluster starts.
> In the context of a mini cluster, it's not useful, as the regions servers are 
> created locally.
> Changing this to a lower value such as 100ms gives 5.8 second per HBase 
> cluser start. It should lower the build time on the apache server by more 
> than 8%.
> Beeing more aggressive (removing all the wait time) could be possible as 
> well. To be studied later.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4703) Improvements in tests

2011-10-30 Thread Hadoop QA (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139817#comment-13139817
 ] 

Hadoop QA commented on HBASE-4703:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12501528/20111030_4703_all.v2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 186 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -166 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
  org.apache.hadoop.hbase.TestFullLogReconstruction
  org.apache.hadoop.hbase.master.TestMasterFailover

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/105//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/105//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/105//console

This message is automatically generated.

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4703) Improvements in tests

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139816#comment-13139816
 ] 

Ted Yu commented on HBASE-4703:
---

I got the following test failures for the attached patch.
These two failed both in test suite and individually.
{code}
testFlushCommitsWithAbort(org.apache.hadoop.hbase.client.TestMultiParallel)  
Time elapsed: 45.25 sec  <<< ERROR!
java.io.IOException: HRegionInfo was null or empty in .META., 
row=keyvalues={multi_test_table,ddd,1320006349447.09bf3a7042e4b84494f49b21d8e6f771./info:server/1320006349957/Put/vlen=19,
 
multi_test_table,ddd,1320006349447.09bf3a7042e4b84494f49b21d8e6f771./info:serverstartcode/1320006349957/Put/vlen=8}
  at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:908)
  at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:769)
  at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1429)
  at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1314)
  at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:878)
  at 
org.apache.hadoop.hbase.client.TestMultiParallel.doTestFlushCommits(TestMultiParallel.java:240)
  at 
org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsWithAbort(TestMultiParallel.java:207)
{code}
{code}
testThreeRSAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting)  
Time elapsed: 91.605 sec  <<< FAILURE!
java.lang.AssertionError:
  at org.junit.Assert.fail(Assert.java:91)
  at org.junit.Assert.assertTrue(Assert.java:43)
  at org.junit.Assert.assertTrue(Assert.java:54)
  at 
org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testThreeRSAbort(TestDistributedLogSplitting.java:139)
{code}
The following failed in test suite but succeeded individually:
{code}
testMasterFailoverWithMockedRITOnDeadRS(org.apache.hadoop.hbase.master.TestMasterFailover)
  Time elapsed: 180.011 sec  <<< ERROR!
java.lang.Exception: test timed out after 18 milliseconds
  at java.lang.Thread.sleep(Native Method)
  at 
org.apache.hadoop.hbase.MiniHBaseCluster.waitForActiveAndReadyMaster(MiniHBaseCluster.java:385)
  at 
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:923)
{code}
{code}
testRowRange(org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol)  
Time elapsed: 0.037 sec  <<< FAILURE!
java.lang.AssertionError: Results should contain region 
test,ccc,1320009243019.ffa363645a679c2c4caf69ede87bdfe3. for row 'ccc'
  at org.junit.Assert.fail(Assert.java:91)
  at org.junit.Assert.assertTrue(Assert.java:43)
  at 
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.verifyRegionResults(TestServerCustomProtocol.java:335)
  at 
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.verifyRegionResults(TestServerCustomProtocol.java:327)
  at 
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.testRowRange(TestServerCustomProtocol.java:213)
{code}


> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead o

[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4583:
-

Status: Patch Available  (was: Open)

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583-v2.txt, 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4583:
-

Attachment: 4583-v2.txt

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583-v2.txt, 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139804#comment-13139804
 ] 

Lars Hofhansl commented on HBASE-4583:
--

Sure, that is more to the point.

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139788#comment-13139788
 ] 

Ted Yu commented on HBASE-4583:
---

bq. and then when the rowlock and the updatesLock.readLock are released, but 
after the rwcc is forwarded, the old rows are removed from the memstore.
I think the above description would be more readable if expressed this way:

after the rwcc is forwarded and, the rowlock and the updatesLock.readLock are 
released, the old rows are removed from the memstore.

Correct me if the above is wrong.


> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139779#comment-13139779
 ] 

Ted Yu commented on HBASE-4690:
---

Integrate to 0.92 and TRUNK.

Thanks for the review Stack.

> Intermittent 
> TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut
>  failure
> 
>
> Key: HBASE-4690
> URL: https://issues.apache.org/jira/browse/HBASE-4690
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4690-trunk.txt
>
>
> See 
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/
> Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region 
> server.
> One fix for this issue is to spin up MiniCluster with 1 region server so that 
> we don't need to search for the region server where first region is hosted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4603) Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers

2011-10-30 Thread stack (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4603:
-

   Resolution: Fixed
Fix Version/s: 0.92.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Applied to trunk and 0.92. Thanks for patch N.

> Uneeded sleep time for tests in 
> hbase.master.ServerManager#waitForRegionServers
> ---
>
> Key: HBASE-4603
> URL: https://issues.apache.org/jira/browse/HBASE-4603
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all.
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.92.0
>
> Attachments: 20111017_4603_MiniHBaseCluster.patch, 4603-v2.txt
>
>
> This functions waits for at least 2 times 
> hbase.master.wait.on.regionservers.interval, defaulted at 3 seconds, i.e. 6 
> seconds for every mini hbase cluster starts.
> In the context of a mini cluster, it's not useful, as the regions servers are 
> created locally.
> Changing this to a lower value such as 100ms gives 5.8 second per HBase 
> cluser start. It should lower the build time on the apache server by more 
> than 8%.
> Beeing more aggressive (removing all the wait time) could be possible as 
> well. To be studied later.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4703) Improvements in tests

2011-10-30 Thread nkeywal (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139774#comment-13139774
 ] 

nkeywal commented on HBASE-4703:


I don't have a generic solution. In some cases, it's possible to replace 
something like this:
{noformat}
for (i<1; i Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-3680) Publish more metrics about mslab

2011-10-30 Thread stack (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139768#comment-13139768
 ] 

stack commented on HBASE-3680:
--

+1

> Publish more metrics about mslab
> 
>
> Key: HBASE-3680
> URL: https://issues.apache.org/jira/browse/HBASE-3680
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.1
>Reporter: Jean-Daniel Cryans
>Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-3680.txt
>
>
> We have been using mslab on all our clusters for a while now and it seems it 
> tends to OOME or send us into GC loops of death a lot more than it used to. 
> For example, one RS with mslab enabled and 7GB of heap died out of OOME this 
> afternoon; it had .55GB in the block cache and 2.03GB in the memstores which 
> doesn't account for much... but it could be that because of mslab a lot of 
> space was lost in those incomplete 2MB blocks and without metrics we can't 
> really tell. Compactions were running at the time of the OOME and I see block 
> cache activity. The average load on that cluster is 531.
> We should at least publish the total size of all those blocks and maybe even 
> take actions based on that (like force flushing).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-3939) Some crossports of Hadoop IPC fixes

2011-10-30 Thread stack (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-3939:
-

Status: Patch Available  (was: Open)

> Some crossports of Hadoop IPC fixes
> ---
>
> Key: HBASE-3939
> URL: https://issues.apache.org/jira/browse/HBASE-3939
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939.txt
>
>
> A few fixes from Hadoop IPC that we should probably cross-port into our copy:
> - HADOOP-7227: remove the protocol version check at call time
> - HADOOP-7146: fix a socket leak in server
> - HADOOP-7121: fix behavior when response serialization throws an exception
> - HADOOP-7346: send back nicer error response when client is using an out of 
> date IPC version

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-3680) Publish more metrics about mslab

2011-10-30 Thread stack (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-3680:
-

Status: Open  (was: Patch Available)

> Publish more metrics about mslab
> 
>
> Key: HBASE-3680
> URL: https://issues.apache.org/jira/browse/HBASE-3680
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.1
>Reporter: Jean-Daniel Cryans
>Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-3680.txt
>
>
> We have been using mslab on all our clusters for a while now and it seems it 
> tends to OOME or send us into GC loops of death a lot more than it used to. 
> For example, one RS with mslab enabled and 7GB of heap died out of OOME this 
> afternoon; it had .55GB in the block cache and 2.03GB in the memstores which 
> doesn't account for much... but it could be that because of mslab a lot of 
> space was lost in those incomplete 2MB blocks and without metrics we can't 
> really tell. Compactions were running at the time of the OOME and I see block 
> cache activity. The average load on that cluster is 531.
> We should at least publish the total size of all those blocks and maybe even 
> take actions based on that (like force flushing).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-3680) Publish more metrics about mslab

2011-10-30 Thread stack (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-3680:
-

Status: Patch Available  (was: Open)

Trying patch build.

> Publish more metrics about mslab
> 
>
> Key: HBASE-3680
> URL: https://issues.apache.org/jira/browse/HBASE-3680
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.1
>Reporter: Jean-Daniel Cryans
>Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-3680.txt
>
>
> We have been using mslab on all our clusters for a while now and it seems it 
> tends to OOME or send us into GC loops of death a lot more than it used to. 
> For example, one RS with mslab enabled and 7GB of heap died out of OOME this 
> afternoon; it had .55GB in the block cache and 2.03GB in the memstores which 
> doesn't account for much... but it could be that because of mslab a lot of 
> space was lost in those incomplete 2MB blocks and without metrics we can't 
> really tell. Compactions were running at the time of the OOME and I see block 
> cache activity. The average load on that cluster is 531.
> We should at least publish the total size of all those blocks and maybe even 
> take actions based on that (like force flushing).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-3939) Some crossports of Hadoop IPC fixes

2011-10-30 Thread stack (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-3939:
-

Status: Open  (was: Patch Available)

> Some crossports of Hadoop IPC fixes
> ---
>
> Key: HBASE-3939
> URL: https://issues.apache.org/jira/browse/HBASE-3939
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939.txt
>
>
> A few fixes from Hadoop IPC that we should probably cross-port into our copy:
> - HADOOP-7227: remove the protocol version check at call time
> - HADOOP-7146: fix a socket leak in server
> - HADOOP-7121: fix behavior when response serialization throws an exception
> - HADOOP-7346: send back nicer error response when client is using an out of 
> date IPC version

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4603) Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers

2011-10-30 Thread nkeywal (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139762#comment-13139762
 ] 

nkeywal commented on HBASE-4603:


Yes, it should decrease the total test duration by 5 minutes.

> Uneeded sleep time for tests in 
> hbase.master.ServerManager#waitForRegionServers
> ---
>
> Key: HBASE-4603
> URL: https://issues.apache.org/jira/browse/HBASE-4603
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all.
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111017_4603_MiniHBaseCluster.patch, 4603-v2.txt
>
>
> This functions waits for at least 2 times 
> hbase.master.wait.on.regionservers.interval, defaulted at 3 seconds, i.e. 6 
> seconds for every mini hbase cluster starts.
> In the context of a mini cluster, it's not useful, as the regions servers are 
> created locally.
> Changing this to a lower value such as 100ms gives 5.8 second per HBase 
> cluser start. It should lower the build time on the apache server by more 
> than 8%.
> Beeing more aggressive (removing all the wait time) could be possible as 
> well. To be studied later.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4703) Improvements in tests

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139763#comment-13139763
 ] 

Ted Yu commented on HBASE-4703:
---

bq. 'sleep': lower or remove the sleep based synchronisation in the tests (in 
HBaseTestingUtility
I am not sure if we can remove sleep based synchronisation.
As the analysis in HBASE-4690 showed, the timing of operations in cluster for 
different hardware / OS combinations would be different.
I guess we don't want to make test code too complicated so that test code can 
be perfectly synchronous with minicluster.

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4677) Remove old single bulkLoadHFile method

2011-10-30 Thread stack (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139760#comment-13139760
 ] 

stack commented on HBASE-4677:
--

Comment from RB: "+1 on patch Jon but we didn't deprecate this in 0.90 so 
removing it would be a little sudden?  Should we deprecate it in 0.92 and apply 
this patch to TRUNK (0.94)?"


> Remove old single bulkLoadHFile method
> --
>
> Key: HBASE-4677
> URL: https://issues.apache.org/jira/browse/HBASE-4677
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.92.0
>
> Attachments: hbase-4677.patch
>
>
> In review for HBASE-4649, there is some debate as whether to remove, 
> deprecate, or leave the HRegionServer.bulkLoadHFile method. 
> https://reviews.apache.org/r/2545/ .   This jira will take care of that for 
> the 0.92 and trunk releases, and allow the same patch to remain for an 
> optional 0.90.x patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4603) Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers

2011-10-30 Thread stack (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139758#comment-13139758
 ] 

stack commented on HBASE-4603:
--

@nkeywal Want me to commit this?  Its committable.  The TDLS failure was 
because of too many open files.

> Uneeded sleep time for tests in 
> hbase.master.ServerManager#waitForRegionServers
> ---
>
> Key: HBASE-4603
> URL: https://issues.apache.org/jira/browse/HBASE-4603
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all.
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111017_4603_MiniHBaseCluster.patch, 4603-v2.txt
>
>
> This functions waits for at least 2 times 
> hbase.master.wait.on.regionservers.interval, defaulted at 3 seconds, i.e. 6 
> seconds for every mini hbase cluster starts.
> In the context of a mini cluster, it's not useful, as the regions servers are 
> created locally.
> Changing this to a lower value such as 100ms gives 5.8 second per HBase 
> cluser start. It should lower the build time on the apache server by more 
> than 8%.
> Beeing more aggressive (removing all the wait time) could be possible as 
> well. To be studied later.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4298) Support to drain RS nodes through ZK

2011-10-30 Thread stack (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139757#comment-13139757
 ] 

stack commented on HBASE-4298:
--

The TDLS failed because of:

{code}
2011-10-30 00:16:16,142 ERROR 
[org.apache.hadoop.hdfs.server.datanode.DataXceiver@15bc53] 
datanode.DataXceiver(131): DatanodeRegistration(127.0.0.1:50070, 
storageID=DS-1489265021-67.195.138.20-50070-1319933763131, infoPort=37761, 
ipcPort=47077):DataXceiver
java.net.SocketException: Too many open files
{code}


> Support to drain RS nodes through ZK
> 
>
> Key: HBASE-4298
> URL: https://issues.apache.org/jira/browse/HBASE-4298
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Affects Versions: 0.90.4
> Environment: all
>Reporter: Aravind Gottipati
>Priority: Critical
>  Labels: patch
> Fix For: 0.92.0, 0.90.5
>
> Attachments: 4298-trunk-v2.txt, 90_hbase.patch, trunk_hbase.patch
>
>
> HDFS currently has a way to exclude certain datanodes and prevent them from 
> getting new blocks.  HDFS goes one step further and even drains these nodes 
> for you.  This enhancement is a step in that direction.
> The idea is that we mark nodes in zookeeper as draining nodes.  This means 
> that they don't get any more new regions.  These draining nodes look exactly 
> the same as the corresponding nodes in /rs, except they live under /draining.
> Eventually, support for draining them can be added.  I am submitting two 
> patches for review - one for the 0.90 branch and one for trunk (in git).
> Here are the two patches
> 0.90 - 
> https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2
> trunk - 
> https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5
> I have tested both these patches and they work as advertised.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4703) Improvements in tests

2011-10-30 Thread nkeywal (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139755#comment-13139755
 ] 

nkeywal commented on HBASE-4703:


Yes, I was thinking about writing a small doc, I will have a look at the
existing section and propose some modifications.




> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure

2011-10-30 Thread stack (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139753#comment-13139753
 ] 

stack commented on HBASE-4690:
--

+1 on trying the patch.  Commit it I'd say.  We can resolve later if we go a 
period w/o these failures.

> Intermittent 
> TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut
>  failure
> 
>
> Key: HBASE-4690
> URL: https://issues.apache.org/jira/browse/HBASE-4690
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4690-trunk.txt
>
>
> See 
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/
> Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region 
> server.
> One fix for this issue is to spin up MiniCluster with 1 region server so that 
> we don't need to search for the region server where first region is hosted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139751#comment-13139751
 ] 

Lars Hofhansl commented on HBASE-4583:
--

Some amount of boilerplate is necessary, because of the locking requirements 
(i.e. cannot have an upsert method in HRegion, since the setup and the first 
part of upsert need to have the same locks held).
Could have a boilerplate method and pass in a command object, but that would 
not necessarily make it more readable.


> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139750#comment-13139750
 ] 

Lars Hofhansl commented on HBASE-4583:
--

New patch on rb: https://reviews.apache.org/r/2633/

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4703) Improvements in tests

2011-10-30 Thread stack (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139747#comment-13139747
 ] 

stack commented on HBASE-4703:
--

I'm not sure there is much to discuss about the recommendations made above by 
N; they seem commonsensical.  N, we should make sure your patterns make it out 
into documentation -- there is a dev section in the manual that we should add 
them to.

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4703) Improvements in tests

2011-10-30 Thread stack (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4703:
-

Status: Patch Available  (was: Open)

Submitting test to run it against patch build

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[Differential] [Updated, 52 lines] D111: [jira] [HBASE-4698] Let the HFile Pretty Printer print all the key values for a specific row.

2011-10-30 Thread Liyin (Liyin Tang)
Liyin updated the revision "[jira] [HBASE-4698] Let the HFile Pretty Printer 
print all the key values for a specific row.".
Reviewers: mbautin, Kannan, jgray, gqchen, nspiegelberg, JIRA

  Address Mikhail's comments.

REVISION DETAIL
  https://reviews.facebook.net/D111

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
Index: src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
===
--- src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
+++ src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
@@ -41,7 +41,6 @@
 import org.apache.hadoop.hbase.HRegionInfo;
 import org.apache.hadoop.hbase.KeyValue;
 import org.apache.hadoop.hbase.io.hfile.HFile.FileInfo;
-import org.apache.hadoop.hbase.io.hfile.HFileBlockIndex.BlockIndexReader;
 import org.apache.hadoop.hbase.regionserver.TimeRangeTracker;
 import org.apache.hadoop.hbase.util.BloomFilter;
 import org.apache.hadoop.hbase.util.BloomFilterFactory;
@@ -68,7 +67,9 @@
   private boolean printBlocks;
   private boolean checkRow;
   private boolean checkFamily;
-
+  private boolean isSeekToRow = false;
+  // the row that the user want to specify and only print kv for.
+  private byte[] row = null;
   private Configuration conf;
 
   private List files = new ArrayList();
@@ -92,6 +93,8 @@
 options.addOption("a", "checkfamily", false, "Enable family check");
 options.addOption("f", "file", true,
 "File to scan. Pass full-path; e.g. hdfs://a:9000/hbase/.META./12/34");
+options.addOption("s", "seekToRow", true,
+"Seek to this row and print all the kvs for this row only");
 options.addOption("r", "region", true,
 "Region to scan. Pass region name; e.g. '.META.,,1'");
   }
@@ -125,6 +128,19 @@
   files.add(new Path(cmd.getOptionValue("f")));
 }
 
+if (cmd.hasOption("s")) {
+  String key = cmd.getOptionValue("s");
+  if (key!= null && key.length() != 0) {
+row = key.getBytes();
+isSeekToRow = true;
+System.out.println("Only print the kv for the row: " +
+	Bytes.toString(row));
+  } else {
+System.err.println("Invalid row is specified.");
+System.exit(-1);
+  }
+}
+
 if (cmd.hasOption("r")) {
   String regionName = cmd.getOptionValue("r");
   byte[] rn = Bytes.toBytes(regionName);
@@ -203,8 +219,13 @@
 if (printKey || checkRow || checkFamily) {
   // scan over file and read key/value's, performing any requested checks
   HFileScanner scanner = reader.getScanner(false, false, false);
-  scanner.seekTo();
-  scanKeyValues(file, scanner);
+  if (this.isSeekToRow) {
+// seek to the first kv on this row
+scanner.seekTo(KeyValue.createFirstOnRow(this.row).getKey());
+  } else {
+scanner.seekTo();
+  }
+  scanKeyValues(file, scanner, row);
 }
 
 // print meta data
@@ -220,7 +241,16 @@
 reader.close();
   }
 
-  private void scanKeyValues(Path file, HFileScanner scanner)
+  /**
+   * 
+   * @param file The path of the file
+   * @param scanner The HFileScanner for the file
+   * @param row Seek to the specific row and print all the kvs for this row. 
+   * if Row is null, it means no row is specified and it will 
+   * print all the rows in this file.
+   * @throws IOException
+   */
+  private void scanKeyValues(Path file, HFileScanner scanner, byte[] row)
   throws IOException {
 KeyValue pkv = null;
 boolean first = true;
@@ -230,6 +260,18 @@
   System.out.print("[");
 do {
   KeyValue kv = scanner.getKeyValue();
+  if (row != null && row.length != 0) {
+int result = Bytes.compareTo(kv.getRow(), row);
+if (result > 0) {
+  System.out.println("All the data for row " +
+  Bytes.toString(row) + " has been printed out");
+  break;
+} else if (result < 0) {
+  System.out.println("Start to scan for the row: " +
+  Bytes.toString(row));
+  continue;
+}
+  }
   // check if rows are in order
   if (checkRow && pkv != null) {
 if (Bytes.compareTo(pkv.getRow(), kv.getRow()) > 0) {


[jira] [Updated] (HBASE-4703) Improvements in tests

2011-10-30 Thread nkeywal (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Attachment: 20111030_4703_all.v2.patch

taking into account ted's comment (with --no-prefix).

Ok for your other comments, I will write a general mail.

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139734#comment-13139734
 ] 

Ted Yu commented on HBASE-4583:
---

@Lars:
removeDupsInMemstore() isn't declared to throw exception. So I didn't expect 
failure in that part of the code.
If runtime exception comes out of it, would requestFlush() still be executed ?

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4703) Improvements in tests

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139730#comment-13139730
 ] 

Ted Yu commented on HBASE-4703:
---

Currently patch testing uses -p0 to apply patches.

>From Todd:
Try "git show --no-prefix" or "git diff --no-prefix" instead. That way it will 
apply with patch -p0 and should work in test-patch

Also, the observations in this JIRA should be broadcast and receive wider 
discussion so that we can establish well-accepted guidelines.
Otherwise the issues corrected by this effort would get back into hbase 
codebase.

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139721#comment-13139721
 ] 

Lars Hofhansl commented on HBASE-4583:
--

The reason why I am calling isFlushSize again is because the duplicate removal 
might fail and that should not impact the operation. So I call it first in the 
required part and then again in the part that might fail. If the 2nd part 
fails, the first value should be used.

Agree on the boiler plate removal.

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4605) Constraints

2011-10-30 Thread Jesse Yates (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139709#comment-13139709
 ] 

Jesse Yates commented on HBASE-4605:


@Ted: ok, fair enough. I'm just trying to work through the possibilities and 
get feedback (thanks btw!).

> Constraints
> ---
>
> Key: HBASE-4605
> URL: https://issues.apache.org/jira/browse/HBASE-4605
> Project: HBase
>  Issue Type: Improvement
>  Components: client, coprocessors
>Affects Versions: 0.94.0
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
>
>
> From Jesse's comment on dev:
> {quote}
> What I would like to propose is a simple interface that people can use to 
> implement a 'constraint' (matching the classic database definition). This 
> would help ease of adoption by helping HBase more easily check that box, help 
> minimize code duplication across organizations, and lead to easier adoption.
> Essentially, people would implement a 'Constraint' interface for checking 
> keys before they are put into a table. Puts that are valid get written to the 
> table, but if not people can will throw an exception that gets propagated 
> back to the client explaining why the put was invalid.
> Constraints would be set on a per-table basis and the user would be expected 
> to ensure the jars containing the constraint are present on the machines 
> serving that table.
> Yes, people could roll their own mechanism for doing this via coprocessors 
> each time, but this would make it easier to do so, so you only have to 
> implement a very minimal interface and not worry about the specifics.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4703) Improvements in tests

2011-10-30 Thread nkeywal (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Attachment: 20111030_4703_all.patch

> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 20111030_4703_all.patch
>
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one 
> server, should start the number of missing server instead.
> - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4703) Improvements in tests

2011-10-30 Thread nkeywal (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nkeywal updated HBASE-4703:
---

Description: 
Global:
 - when possible, make the test using the default cluster configuration for the 
number of region (1 instead of 2 or 3). This allows a faster stop/start, and is 
a step toward a shared cluster configuration.
 - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
TestServerCustomProtocol, TestReplicationSink)
  - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or 
in a loop. Not done for tests that rely on the WAL.
 
Local issues:
- TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
tearDown, that makes it impossible to use in // with another test using this 
directory
- TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
seconds to make it a part of the small subset
- TestMemoryBoundedLogMessageBuffer useless System.out.println
- io.hfile.TestReseekTo useless System.out.println
- TestTableInputFormat does not shutdown the cluster
- testGlobalMemStore does not shutdown the cluster
- rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
test instead of two.
- HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, 
should start the number of missing server instead.
- TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

  was:
Global:
 - when possible, make the test using the default cluster configuration for the 
number of region (1 instead of 2 or 3). This allows a faster stop/start, and is 
a step toward a shared cluster configuration.
 - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
TestServerCustomProtocol, TestReplicationSink)
  - tmp dir management: to allow multiple tests using dir to run on the same 
machine without conflicting, the tmp dir is set once per JVM. This should fix 
HBASE-3672 as well, because the map reduce cluster is configured to use the 
hadoop tmp dir
  - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or 
in a loop. Not done for tests that rely on the WAL.
 
Local issues:
- TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
tearDown, that makes it impossible to use in // with another test using this 
directory
- TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
seconds to make it a part of the small subset
- TestMemoryBoundedLogMessageBuffer useless System.out.println
- io.hfile.TestReseekTo useless System.out.println
- TestTableInputFormat does not shutdown the cluster
- testGlobalMemStore does not shutdown the cluster
- rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
test instead of two.
- HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, 
should start the number of missing server instead.
- TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility


> Improvements in tests
> -
>
> Key: HBASE-4703
> URL: https://issues.apache.org/jira/browse/HBASE-4703
> Project: HBase
>  Issue Type: Improvement
>  Components: test
>Affects Versions: 0.92.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
>
> Global:
>  - when possible, make the test using the default cluster configuration for 
> the number of region (1 instead of 2 or 3). This allows a faster stop/start, 
> and is a step toward a shared cluster configuration.
>  - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
> HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
> TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
> TestServerCustomProtocol, TestReplicationSink)
>   - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big 
> or in a loop. Not done for tests that rely on the WAL.
>  
> Local issues:
> - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
> tearDown, that makes it impossible to use in // with another test using this 
> directory
> - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
> seconds to make it a part of the small subset
> - TestMemoryBoundedLogMessageBuffer useless System.out.println
> - io.hfile.TestReseekTo useless System.out.println
> - TestTableInputFormat does not shutdown the cluster
> - testGlobalMemStore does not shutdown the cluster
> - rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
> test instead of two.
> - HBaseTestingUtility#e

[jira] [Created] (HBASE-4703) Improvements in tests

2011-10-30 Thread nkeywal (Created) (JIRA)
Improvements in tests
-

 Key: HBASE-4703
 URL: https://issues.apache.org/jira/browse/HBASE-4703
 Project: HBase
  Issue Type: Improvement
  Components: test
Affects Versions: 0.92.0
 Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor


Global:
 - when possible, make the test using the default cluster configuration for the 
number of region (1 instead of 2 or 3). This allows a faster stop/start, and is 
a step toward a shared cluster configuration.
 - 'sleep': lower or remove the sleep based synchronisation in the tests (in 
HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, 
TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, 
TestServerCustomProtocol, TestReplicationSink)
  - tmp dir management: to allow multiple tests using dir to run on the same 
machine without conflicting, the tmp dir is set once per JVM. This should fix 
HBASE-3672 as well, because the map reduce cluster is configured to use the 
hadoop tmp dir
  - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or 
in a loop. Not done for tests that rely on the WAL.
 
Local issues:
- TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on 
tearDown, that makes it impossible to use in // with another test using this 
directory
- TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 
seconds to make it a part of the small subset
- TestMemoryBoundedLogMessageBuffer useless System.out.println
- io.hfile.TestReseekTo useless System.out.println
- TestTableInputFormat does not shutdown the cluster
- testGlobalMemStore does not shutdown the cluster
- rest.client.TestRemoteAdmin: simplified, does not use local admin, single 
test instead of two.
- HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, 
should start the number of missing server instead.
- TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139660#comment-13139660
 ] 

Ted Yu commented on HBASE-4583:
---

bq. Still a lot of boilerplate shared between Append and Increment.
I think we should take this opportunity to reduce duplicate code.

> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4519) 25s sleep when expiring sessions in tests

2011-10-30 Thread nkeywal (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139643#comment-13139643
 ] 

nkeywal commented on HBASE-4519:


TestZooKeeper#testClientSessionExpired() has a very similar piece of code, with 
a 15s timeout 'only'. 
But when i tried this value, I had a failure in TestFromClientSide... to be 
continued

> 25s sleep when expiring sessions in tests
> -
>
> Key: HBASE-4519
> URL: https://issues.apache.org/jira/browse/HBASE-4519
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.90.4
>Reporter: Jean-Daniel Cryans
>Assignee: nkeywal
> Fix For: 0.92.0
>
>
> There's a hardcoded 25 seconds sleep in HBaseTestingUtility.expireSession: 
> {code}
> int sessionTimeout = 5 * 1000; // 5 seconds
> ...
> final long sleep = sessionTimeout * 5L;
> LOG.info("ZK Closed Session 0x" + Long.toHexString(sessionID) +
>   "; sleeping=" + sleep);
> Thread.sleep(sleep);
> {code}
> I'm pretty sure this can be lowered at lot, and it would speed up a couple of 
> tests. The only thing I'm afraid of is if this was made to accomodate flaky 
> tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139641#comment-13139641
 ] 

Ted Yu commented on HBASE-4583:
---

Publishing on reviewboard would make reviewing much easier.

Patch looks good overall.

A few minor comments.
newFamilyMap.asMap() is called more than once in append() and increment(). We 
can use a variable to hold its return and reuse the variable in place of the 
second call.
Line 3948 can be omitted:
{code}
   flush = isFlushSize(size);
{code}
because we have the same call at line 3964.
Same argument can be made for the call at line 3824.

In MemStore.java:
{code}
   * For each KeyValue if the keyValue did already exist, with a
{code}
Should read 'if the KeyValue does already ...'

This comment in Store.java can be refined:
{code}
   * qualifier exists in MemStore with a memstoreTS < the passed KV, it will be 
removed.
{code}
because we only pass keyvalues to this.memstore.removeDups().


> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4695) WAL logs get deleted before region server can fully flush

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139630#comment-13139630
 ] 

Ted Yu commented on HBASE-4695:
---

The test failures reported by HadoopQA were due to 'Too many open files'

> WAL logs get deleted before region server can fully flush
> -
>
> Key: HBASE-4695
> URL: https://issues.apache.org/jira/browse/HBASE-4695
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.90.4
>Reporter: jack levin
>Assignee: gaojinchao
>Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt
>
>
> To replicate the problem do the following:
> 1. check /hbase/.logs/ directory to see if you have WAL logs for the 
> region server you are shutting down.
> 2. executing kill  (where pid is a regionserver pid)
> 3. Watch the regionserver log to start flushing, you will see how many 
> regions are left to flush:
> 09:36:54,665 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 489 regions to close
> 09:56:35,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 116 regions to close
> 4. Check /hbase/.logs/ -- you will notice that it has dissapeared.
> 5. Check namenode logs:
> 09:26:41,607 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: 
> ugi=root ip=/10.101.1.5 cmd=delete 
> src=/hbase/.logs/rdaa5.prod.imageshack.com,60020,1319749
> Note that, if you kill -9 the RS now, and it crashes on flush, you won't have 
> any WAL logs to replay.  We need to make sure that logs are deleted or moved 
> out only when RS has fully flushed. Otherwise its possible to lose data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure

2011-10-30 Thread Ted Yu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139627#comment-13139627
 ] 

Ted Yu commented on HBASE-4690:
---

Except for TestAdmin#testEnableDisableAddColumnDeleteColumn, the other test 
failures were due to 'Too many open files'

I ran TestAdmin#testEnableDisableAddColumnDeleteColumn a few times and didn't 
encounter test failure.

> Intermittent 
> TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut
>  failure
> 
>
> Key: HBASE-4690
> URL: https://issues.apache.org/jira/browse/HBASE-4690
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4690-trunk.txt
>
>
> See 
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/
> Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region 
> server.
> One fix for this issue is to spin up MiniCluster with 1 region server so that 
> we don't need to search for the region server where first region is hosted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4696) HRegionThriftServer' might have to indefinitely do redirtects

2011-10-30 Thread dhruba borthakur (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139564#comment-13139564
 ] 

dhruba borthakur commented on HBASE-4696:
-

+1 code looks good to me.

> HRegionThriftServer' might have to indefinitely do redirtects
> -
>
> Key: HBASE-4696
> URL: https://issues.apache.org/jira/browse/HBASE-4696
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.94.0
>Reporter: Prakash Khemani
>Assignee: Jonathan Gray
> Fix For: 0.94.0
>
> Attachments: HBASE-4696-v1.patch
>
>
> HRegionThriftServer.getRowWithColumnsTs() redirects the request to the 
> correct region server if it has landed on the wrong region-server. With this 
> approach the smart-client will never get a NotServingRegionException and it 
> will never be able to invalidate its cache. It will indefinitely send the 
> request to the wrong region server and the wrong region server will always be 
> redirecting it.
> Either redirects should be turned off and the client should react to 
> NotServingRegionExceptions.
> Or somehow a flag should be set in the response telling the client to refresh 
> its cache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure

2011-10-30 Thread Hadoop QA (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139554#comment-13139554
 ] 

Hadoop QA commented on HBASE-4690:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12501496/4690-trunk.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -166 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.master.TestMasterFailover
  org.apache.hadoop.hbase.client.TestAdmin
  org.apache.hadoop.hbase.master.TestDistributedLogSplitting

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/104//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/104//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/104//console

This message is automatically generated.

> Intermittent 
> TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut
>  failure
> 
>
> Key: HBASE-4690
> URL: https://issues.apache.org/jira/browse/HBASE-4690
> Project: HBase
>  Issue Type: Test
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4690-trunk.txt
>
>
> See 
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/
> Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region 
> server.
> One fix for this issue is to spin up MiniCluster with 1 region server so that 
> we don't need to search for the region server where first region is hosted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Issue Comment Edited] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Issue Comment Edited) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139550#comment-13139550
 ] 

Lars Hofhansl edited comment on HBASE-4583 at 10/30/11 7:06 AM:


Here's a *sketch* of a patch.
May need some clean up but should work.

Basically the KVs of Append and Increment are applied to memstore through a 
call to the "standard" applyFamilyMapToMemstore(...); then duplicates are 
removed after the rwcc is moved forward.
Still a lot of boilerplate shared between Append and Increment.

I think the dup removal does not need to hold the regions updatesLock.readLock 
(just the lock.readLock), but I am not entirely sure.

Comments/suggestions/flames/praise are welcome. :)


  was (Author: lhofhansl):
Here a *sketch* of a patch.
May need some clean up but should work.

Basically the KVs of Append and Increment are applied to memstore through a 
call to the "standard" applyFamilyMapToMemstore(...); then duplicates are 
removed after the rwcc is moved forward.
Still a lot of boilerplate shared between Append and Increment.

I think the dup removal does not need to hold the regions updatesLock.readLock 
(just the lock.readLock), but I am not entirely sure.

Comments/suggestions/flames/praise are welcome. :)

  
> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Issue Comment Edited] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Issue Comment Edited) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139551#comment-13139551
 ] 

Lars Hofhansl edited comment on HBASE-4583 at 10/30/11 7:08 AM:


Also changes some calls to take  "? extends Collection"  rather than List, so 
that I can use guava's MultiMap. If all callers would be changed the  "? 
extends"  could be removed.


  was (Author: lhofhansl):
Also changes some calls to take  ? extends Collection  rather than List, so 
that I can use guava's MultiMap. If all callers would be changed the  ? extends 
 could be removed.

  
> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139551#comment-13139551
 ] 

Lars Hofhansl commented on HBASE-4583:
--

Also changes some calls to take  ? extends Collection  rather than List, so 
that I can use guava's MultiMap. If all callers would be changed the  ? extends 
 could be removed.


> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (HBASE-4695) WAL logs get deleted before region server can fully flush

2011-10-30 Thread gaojinchao (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139553#comment-13139553
 ] 

gaojinchao commented on HBASE-4695:
---

@Ted
Thanks for your reveiw.  
I considered if this.fsOk is false , calling closeWAL() makes no sense.
Your approach is same as original logic, that is ok, I will verify next monday.

> WAL logs get deleted before region server can fully flush
> -
>
> Key: HBASE-4695
> URL: https://issues.apache.org/jira/browse/HBASE-4695
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.90.4
>Reporter: jack levin
>Assignee: gaojinchao
>Priority: Blocker
> Fix For: 0.90.5
>
> Attachments: HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt
>
>
> To replicate the problem do the following:
> 1. check /hbase/.logs/ directory to see if you have WAL logs for the 
> region server you are shutting down.
> 2. executing kill  (where pid is a regionserver pid)
> 3. Watch the regionserver log to start flushing, you will see how many 
> regions are left to flush:
> 09:36:54,665 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 489 regions to close
> 09:56:35,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting 
> on 116 regions to close
> 4. Check /hbase/.logs/ -- you will notice that it has dissapeared.
> 5. Check namenode logs:
> 09:26:41,607 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: 
> ugi=root ip=/10.101.1.5 cmd=delete 
> src=/hbase/.logs/rdaa5.prod.imageshack.com,60020,1319749
> Note that, if you kill -9 the RS now, and it crashes on flush, you won't have 
> any WAL logs to replay.  We need to make sure that logs are deleted or moved 
> out only when RS has fully flushed. Otherwise its possible to lose data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations

2011-10-30 Thread Lars Hofhansl (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-4583:
-

Attachment: 4583.txt

Here a *sketch* of a patch.
May need some clean up but should work.

Basically the KVs of Append and Increment are applied to memstore through a 
call to the "standard" applyFamilyMapToMemstore(...); then duplicates are 
removed after the rwcc is moved forward.
Still a lot of boilerplate shared between Append and Increment.

I think the dup removal does not need to hold the regions updatesLock.readLock 
(just the lock.readLock), but I am not entirely sure.

Comments/suggestions/flames/praise are welcome. :)


> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
>  Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 4583.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a 
> client could see the results of multiple such operation mixed in the same 
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes 
> to and from the memstore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira